2024-09-21T09:23:45.670 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T09:23:45.696 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T09:23:45.790 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913651 branch: main description: upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} email: ceph-qa@ceph.com first-half-sequence: - cephadm.shell: env: - sha1 mon.a: - ceph config set mgr mgr/cephadm/daemon_cache_timeout 60 - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps - echo wait for minority of mons to upgrade - while ! ceph mon versions | grep $sha1 && ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error; do sleep 2 ; done - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for majority of mons to upgrade - 'while ! ceph mon versions | grep $sha1 | egrep '': [23]'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for all mons to upgrade - 'while ! ceph mon versions | grep $sha1 | grep '': 3'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for half of osds to upgrade - 'while ! ceph osd versions | grep $sha1 | egrep '': [45678]'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch upgrade pause - ceph orch ps - ceph orch ps - ceph versions first-half-tasks: - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - print: '**** done end radosbench.yaml' first_in_suite: false job_id: '7913651' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' - desc: 'run basic cls tests for rbd ' - desc: 'run basic import/export cli tests for rbd ' - desc: 'librbd C and C++ api tests ' - desc: 'randomized correctness test for rados operations on a replicated pool, using only reads, writes, and deletes ' - desc: 'randomized correctness test for rados operations on a replicated pool with snapshot operations ' - desc: 'run basic import/export cli tests for rbd ' name: teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true create_rbd_pool: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - OSD_DOWN - mons down - mon down - MON_DOWN - out of quorum - PG_DEGRADED - Reduced data availability - Degraded data redundancy - OSDMAP_FLAGS - PG_AVAILABILITY sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 owner: scheduled_teuthology@teuthology priority: 850 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b second-half-sequence: sequential: - cephadm.shell: env: - sha1 mon.a: - ceph orch upgrade resume - sleep 60 - echo wait for upgrade to complete - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; sleep 30 ; done - echo upgrade complete - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 second-half-tasks: - workunit: branch: quincy clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' seed: 5357 sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 sleep_before_teardown: 0 stress-tasks: - thrashosds: aggressive_pg_num_changes: false chance_force_recovery: 0 chance_pgnum_grow: 1 chance_pgpnum_fix: 1 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 chance_thrash_pg_upmap_items: 0 disable_objectstore_tool_tests: true timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - print: '**** done end radosbench.yaml' - workunit: branch: quincy clients: client.0: - cls/test_cls_rbd.sh - print: '**** done cls/test_cls_rbd.sh 5-workload' - workunit: branch: quincy clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' - workunit: branch: quincy clients: client.0: - rbd/test_librbd.sh env: RBD_FEATURES: '61' - print: '**** done rbd/test_librbd.sh 7-workload' - full_sequential: - rados: clients: - client.0 objects: 500 op_weights: delete: 10 read: 45 write: 45 ops: 4000 write_append_excl: false - print: '**** done rados/readwrite 5-workload' - full_sequential: - rados: clients: - client.0 objects: 50 op_weights: delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 write_append_excl: false - print: '**** done rados/snaps-few-objects 5-workload' subset: 31/32 suite: upgrade suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 targets: smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEnS5fv5NKFsN+Hd9Nbg3CgjGoPf8E7sQp2Yu34cy6sLFM4kLW3oNCsFl67qIZdN5C9rxBpndsxx3bBWeYaC2wc= smithi129.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM4IEgpUJA7TgWeQTMQyJrTcBK8QFLIFwBtaUs2/0ktb9XNeKYPHulrpKY92TmdWXzvxs6UJZnEVKozieJYBZCc= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install runc nvmetcli nvme-cli -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: branch: quincy exclude_packages: - ceph-volume - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy - cephadm.shell: mon.a: - ceph fs volume create foo - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph.healthy: null - print: '**** upgrading first half of cluster, with stress ****' - parallel: - first-half-tasks - first-half-sequence - print: '**** done upgrading first half of cluster ****' - ceph.healthy: null - print: '**** applying stress + thrashing to mixed-version cluster ****' - parallel: - stress-tasks - ceph.healthy: null - print: '**** finishing upgrade ****' - parallel: - second-half-tasks - second-half-sequence - ceph.healthy: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_03:08:26 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T09:23:45.790 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa; will attempt to use it 2024-09-21T09:23:45.791 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks 2024-09-21T09:23:45.791 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T09:23:45.793 INFO:teuthology.task.internal:Checking packages... 2024-09-21T09:23:45.812 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '62da54b4b83a96ca9976b7e1d0c8272a564a2208' 2024-09-21T09:23:45.813 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:23:45.813 INFO:teuthology.packaging:ref: None 2024-09-21T09:23:45.813 INFO:teuthology.packaging:tag: None 2024-09-21T09:23:45.813 INFO:teuthology.packaging:branch: main 2024-09-21T09:23:45.813 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:23:45.813 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-09-21T09:23:46.051 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5092.g62da54b4 2024-09-21T09:23:46.053 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T09:23:46.059 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T09:23:46.059 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T09:23:46.087 INFO:teuthology.task.internal:Saving configuration 2024-09-21T09:23:46.109 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T09:23:46.125 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T09:23:46.150 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913651', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 09:18:39.521821', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEnS5fv5NKFsN+Hd9Nbg3CgjGoPf8E7sQp2Yu34cy6sLFM4kLW3oNCsFl67qIZdN5C9rxBpndsxx3bBWeYaC2wc='} 2024-09-21T09:23:46.174 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi129.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913651', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 09:18:39.523299', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM4IEgpUJA7TgWeQTMQyJrTcBK8QFLIFwBtaUs2/0ktb9XNeKYPHulrpKY92TmdWXzvxs6UJZnEVKozieJYBZCc='} 2024-09-21T09:23:46.174 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T09:23:46.185 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-09-21T09:23:46.185 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-09-21T09:23:46.185 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T09:23:46.263 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb9b2d7c160>, signals=[15]) 2024-09-21T09:23:46.264 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T09:23:46.272 INFO:teuthology.task.internal:Opening connections... 2024-09-21T09:23:46.273 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2024-09-21T09:23:46.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:23:46.348 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi129.front.sepia.ceph.com 2024-09-21T09:23:46.348 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:23:46.423 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T09:23:46.435 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2024-09-21T09:23:46.453 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2024-09-21T09:23:46.453 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-09-21T09:23:46.512 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-09-21T09:23:46.512 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="9" 2024-09-21T09:23:46.512 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-09-21T09:23:46.512 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-09-21T09:23:46.512 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="9" 2024-09-21T09:23:46.512 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el9" 2024-09-21T09:23:46.512 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T09:23:46.512 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-09-21T09:23:46.512 INFO:teuthology.orchestra.run.smithi107.stdout:LOGO="fedora-logo-icon" 2024-09-21T09:23:46.512 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T09:23:46.512 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-09-21T09:23:46.513 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T09:23:46.513 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T09:23:46.513 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T09:23:46.513 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-09-21T09:23:46.540 DEBUG:teuthology.orchestra.run.smithi129:> uname -m 2024-09-21T09:23:46.558 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2024-09-21T09:23:46.559 DEBUG:teuthology.orchestra.run.smithi129:> cat /etc/os-release 2024-09-21T09:23:46.616 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="CentOS Stream" 2024-09-21T09:23:46.616 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="9" 2024-09-21T09:23:46.616 INFO:teuthology.orchestra.run.smithi129.stdout:ID="centos" 2024-09-21T09:23:46.616 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE="rhel fedora" 2024-09-21T09:23:46.616 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="9" 2024-09-21T09:23:46.616 INFO:teuthology.orchestra.run.smithi129.stdout:PLATFORM_ID="platform:el9" 2024-09-21T09:23:46.617 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T09:23:46.617 INFO:teuthology.orchestra.run.smithi129.stdout:ANSI_COLOR="0;31" 2024-09-21T09:23:46.617 INFO:teuthology.orchestra.run.smithi129.stdout:LOGO="fedora-logo-icon" 2024-09-21T09:23:46.617 INFO:teuthology.orchestra.run.smithi129.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T09:23:46.617 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://centos.org/" 2024-09-21T09:23:46.617 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T09:23:46.617 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T09:23:46.617 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T09:23:46.617 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2024-09-21T09:23:46.640 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T09:23:46.653 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T09:23:46.702 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T09:23:46.702 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2024-09-21T09:23:46.705 DEBUG:teuthology.orchestra.run.smithi129:> test '!' -e /home/ubuntu/cephtest 2024-09-21T09:23:46.722 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T09:23:46.732 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T09:23:46.733 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2024-09-21T09:23:46.762 DEBUG:teuthology.orchestra.run.smithi129:> test -z $(ls -A /var/lib/ceph) 2024-09-21T09:23:46.869 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T09:23:46.927 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T09:23:47.024 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T09:23:47.024 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T09:23:47.024 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T09:23:47.024 DEBUG:teuthology.orchestra.run.smithi129:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T09:23:47.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T09:23:47.041 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-09-21T09:23:47.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T09:23:47.042 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-09-21T09:23:47.099 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-503.el9.x86_64 2024-09-21T09:23:47.099 INFO:teuthology.task.kernel:Running kernel on smithi107: 5.14.0-503.el9.x86_64 2024-09-21T09:23:47.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum install -y kernel 2024-09-21T09:23:47.100 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-503.el9.x86_64 2024-09-21T09:23:47.100 INFO:teuthology.task.kernel:Running kernel on smithi129: 5.14.0-503.el9.x86_64 2024-09-21T09:23:47.101 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum install -y kernel 2024-09-21T09:23:49.445 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.3 MB 00:00 2024-09-21T09:23:51.121 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 34 MB/s | 20 MB 00:00 2024-09-21T09:23:51.811 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 2.9 MB/s | 8.3 MB 00:02 2024-09-21T09:23:53.920 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 20 MB/s | 20 MB 00:01 2024-09-21T09:23:55.239 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 18 MB/s | 6.5 MB 00:00 2024-09-21T09:23:57.706 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 19 kB 00:01 2024-09-21T09:23:58.537 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 8.2 MB/s | 6.5 MB 00:00 2024-09-21T09:23:58.793 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-21T09:24:00.029 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 19 kB 00:00 2024-09-21T09:24:00.775 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 36 MB/s | 23 MB 00:00 2024-09-21T09:24:04.158 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-21T09:24:06.064 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T09:24:06.064 INFO:teuthology.orchestra.run.smithi129.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T09:24:06.164 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:24:06.173 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:24:06.173 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-09-21T09:24:06.173 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:24:06.173 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T09:24:06.173 INFO:teuthology.orchestra.run.smithi129.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T09:24:06.173 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-21T09:24:06.173 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T09:24:06.174 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T09:24:06.174 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T09:24:06.174 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:24:06.174 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T09:24:06.174 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:24:06.174 INFO:teuthology.orchestra.run.smithi129.stdout:Install 4 Packages 2024-09-21T09:24:06.174 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:24:06.175 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 79 M 2024-09-21T09:24:06.175 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 126 M 2024-09-21T09:24:06.175 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T09:24:06.263 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-21T09:24:06.326 INFO:teuthology.orchestra.run.smithi129.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 339 kB/s | 36 kB 00:00 2024-09-21T09:24:06.726 INFO:teuthology.orchestra.run.smithi129.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-09-21T09:24:07.126 INFO:teuthology.orchestra.run.smithi129.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 36 MB/s | 29 MB 00:00 2024-09-21T09:24:07.451 INFO:teuthology.orchestra.run.smithi129.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 28 MB/s | 35 MB 00:01 2024-09-21T09:24:07.452 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:24:07.452 INFO:teuthology.orchestra.run.smithi129.stdout:Total 62 MB/s | 79 MB 00:01 2024-09-21T09:24:08.127 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T09:24:08.186 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T09:24:08.186 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T09:24:08.270 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:24:08.276 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:24:08.276 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-09-21T09:24:08.276 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:24:08.277 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T09:24:08.277 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T09:24:08.277 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-21T09:24:08.277 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T09:24:08.277 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T09:24:08.277 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T09:24:08.277 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:24:08.277 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T09:24:08.277 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:24:08.277 INFO:teuthology.orchestra.run.smithi107.stdout:Install 4 Packages 2024-09-21T09:24:08.277 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:24:08.278 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 79 M 2024-09-21T09:24:08.278 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 126 M 2024-09-21T09:24:08.278 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T09:24:08.482 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T09:24:08.482 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T09:24:08.565 INFO:teuthology.orchestra.run.smithi107.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 136 kB/s | 36 kB 00:00 2024-09-21T09:24:09.453 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T09:24:09.453 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T09:24:11.887 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T09:24:12.789 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T09:24:12.868 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:24:12.889 INFO:teuthology.orchestra.run.smithi107.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 3.4 MB/s | 16 MB 00:04 2024-09-21T09:24:15.624 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:24:15.747 INFO:teuthology.orchestra.run.smithi107.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 4.0 MB/s | 29 MB 00:07 2024-09-21T09:24:15.752 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:24:20.564 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:24:20.730 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:24:25.563 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:24:26.412 INFO:teuthology.orchestra.run.smithi107.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 1.9 MB/s | 35 MB 00:18 2024-09-21T09:24:26.412 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:24:26.413 INFO:teuthology.orchestra.run.smithi107.stdout:Total 4.3 MB/s | 79 MB 00:18 2024-09-21T09:24:27.098 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T09:24:27.453 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T09:24:27.453 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T09:24:28.417 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T09:24:28.417 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T09:24:30.846 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T09:24:31.545 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T09:24:31.646 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:24:34.386 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:24:34.531 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:24:39.378 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:24:39.422 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:24:44.266 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:25:03.588 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:25:03.628 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:25:03.983 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:25:03.983 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T09:25:03.983 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:25:03.983 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:25:04.849 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:25:04.850 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:25:04.850 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T09:25:04.850 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T09:25:04.850 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T09:25:04.850 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T09:25:04.850 INFO:teuthology.orchestra.run.smithi129.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T09:25:04.850 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:25:04.850 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:25:05.199 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T09:25:05.284 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T09:25:05.285 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T09:25:05.285 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T09:25:05.285 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T09:25:05.285 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T09:25:05.285 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-21T09:25:05.285 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi129.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T09:25:05.285 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T09:25:05.343 INFO:teuthology.orchestra.run.smithi129.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T09:25:05.343 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T09:25:05.344 DEBUG:teuthology.orchestra.run.smithi129:> sudo rpm -qi grub2-tools 2024-09-21T09:25:05.376 INFO:teuthology.orchestra.run.smithi129.stdout:Name : grub2-tools 2024-09-21T09:25:05.376 INFO:teuthology.orchestra.run.smithi129.stdout:Epoch : 1 2024-09-21T09:25:05.376 INFO:teuthology.orchestra.run.smithi129.stdout:Version : 2.06 2024-09-21T09:25:05.376 INFO:teuthology.orchestra.run.smithi129.stdout:Release : 82.el9 2024-09-21T09:25:05.376 INFO:teuthology.orchestra.run.smithi129.stdout:Architecture: x86_64 2024-09-21T09:25:05.376 INFO:teuthology.orchestra.run.smithi129.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T09:25:05.376 INFO:teuthology.orchestra.run.smithi129.stdout:Group : Unspecified 2024-09-21T09:25:05.376 INFO:teuthology.orchestra.run.smithi129.stdout:Size : 8274670 2024-09-21T09:25:05.376 INFO:teuthology.orchestra.run.smithi129.stdout:License : GPLv3+ 2024-09-21T09:25:05.376 INFO:teuthology.orchestra.run.smithi129.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T09:25:05.376 INFO:teuthology.orchestra.run.smithi129.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T09:25:05.376 INFO:teuthology.orchestra.run.smithi129.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T09:25:05.376 INFO:teuthology.orchestra.run.smithi129.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T09:25:05.377 INFO:teuthology.orchestra.run.smithi129.stdout:Packager : builder@centos.org 2024-09-21T09:25:05.377 INFO:teuthology.orchestra.run.smithi129.stdout:Vendor : CentOS 2024-09-21T09:25:05.377 INFO:teuthology.orchestra.run.smithi129.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T09:25:05.377 INFO:teuthology.orchestra.run.smithi129.stdout:Summary : Support tools for GRUB. 2024-09-21T09:25:05.377 INFO:teuthology.orchestra.run.smithi129.stdout:Description : 2024-09-21T09:25:05.377 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:25:05.377 INFO:teuthology.orchestra.run.smithi129.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T09:25:05.377 INFO:teuthology.orchestra.run.smithi129.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T09:25:05.377 INFO:teuthology.orchestra.run.smithi129.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T09:25:05.377 INFO:teuthology.orchestra.run.smithi129.stdout:hardware devices. 2024-09-21T09:25:05.377 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:25:05.377 INFO:teuthology.orchestra.run.smithi129.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T09:25:05.378 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T09:25:05.378 INFO:teuthology.task.kernel:Updating grub on smithi129 to boot 5.14.0-508.el9.x86_64 2024-09-21T09:25:05.378 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T09:25:06.053 INFO:teuthology.orchestra.run.smithi129.stderr:Generating grub configuration file ... 2024-09-21T09:25:07.328 INFO:teuthology.orchestra.run.smithi129.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T09:25:07.362 INFO:teuthology.orchestra.run.smithi129.stderr:done 2024-09-21T09:25:07.364 DEBUG:teuthology.orchestra.run.smithi129:> mktemp 2024-09-21T09:25:07.379 INFO:teuthology.orchestra.run.smithi129.stdout:/tmp/tmp.GoJTVELLlm 2024-09-21T09:25:07.379 DEBUG:teuthology.orchestra.run.smithi129:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.GoJTVELLlm 2024-09-21T09:25:07.445 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0666 /tmp/tmp.GoJTVELLlm 2024-09-21T09:25:07.583 DEBUG:teuthology.orchestra.remote:smithi129:/tmp/tmp.GoJTVELLlm is 6KB 2024-09-21T09:25:07.603 DEBUG:teuthology.orchestra.run.smithi129:> rm -fr /tmp/tmp.GoJTVELLlm 2024-09-21T09:25:07.617 DEBUG:teuthology.orchestra.run.smithi129:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T09:25:07.681 INFO:teuthology.orchestra.run.smithi129.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T09:25:07.681 INFO:teuthology.orchestra.run.smithi129.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T09:25:07.681 INFO:teuthology.orchestra.run.smithi129.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T09:25:07.681 INFO:teuthology.orchestra.run.smithi129.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T09:25:07.683 DEBUG:teuthology.orchestra.run.smithi129:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T09:25:07.830 DEBUG:teuthology.orchestra.run.smithi129:> sudo shutdown -r now 2024-09-21T09:25:22.419 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:25:22.458 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:25:22.805 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:25:22.805 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T09:25:22.805 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:25:22.805 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:25:23.633 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:25:23.633 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:25:23.634 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T09:25:23.634 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T09:25:23.634 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T09:25:23.634 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T09:25:23.634 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T09:25:23.634 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:25:23.634 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:25:24.005 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T09:25:24.087 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T09:25:24.088 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T09:25:24.088 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T09:25:24.088 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T09:25:24.088 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T09:25:24.088 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T09:25:24.088 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T09:25:24.088 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T09:25:24.147 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T09:25:24.147 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T09:25:24.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo rpm -qi grub2-tools 2024-09-21T09:25:24.178 INFO:teuthology.orchestra.run.smithi107.stdout:Name : grub2-tools 2024-09-21T09:25:24.178 INFO:teuthology.orchestra.run.smithi107.stdout:Epoch : 1 2024-09-21T09:25:24.178 INFO:teuthology.orchestra.run.smithi107.stdout:Version : 2.06 2024-09-21T09:25:24.178 INFO:teuthology.orchestra.run.smithi107.stdout:Release : 82.el9 2024-09-21T09:25:24.178 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: x86_64 2024-09-21T09:25:24.178 INFO:teuthology.orchestra.run.smithi107.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T09:25:24.179 INFO:teuthology.orchestra.run.smithi107.stdout:Group : Unspecified 2024-09-21T09:25:24.179 INFO:teuthology.orchestra.run.smithi107.stdout:Size : 8274670 2024-09-21T09:25:24.179 INFO:teuthology.orchestra.run.smithi107.stdout:License : GPLv3+ 2024-09-21T09:25:24.179 INFO:teuthology.orchestra.run.smithi107.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T09:25:24.179 INFO:teuthology.orchestra.run.smithi107.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T09:25:24.179 INFO:teuthology.orchestra.run.smithi107.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T09:25:24.179 INFO:teuthology.orchestra.run.smithi107.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T09:25:24.179 INFO:teuthology.orchestra.run.smithi107.stdout:Packager : builder@centos.org 2024-09-21T09:25:24.179 INFO:teuthology.orchestra.run.smithi107.stdout:Vendor : CentOS 2024-09-21T09:25:24.179 INFO:teuthology.orchestra.run.smithi107.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T09:25:24.179 INFO:teuthology.orchestra.run.smithi107.stdout:Summary : Support tools for GRUB. 2024-09-21T09:25:24.179 INFO:teuthology.orchestra.run.smithi107.stdout:Description : 2024-09-21T09:25:24.179 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:25:24.180 INFO:teuthology.orchestra.run.smithi107.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T09:25:24.180 INFO:teuthology.orchestra.run.smithi107.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T09:25:24.180 INFO:teuthology.orchestra.run.smithi107.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T09:25:24.180 INFO:teuthology.orchestra.run.smithi107.stdout:hardware devices. 2024-09-21T09:25:24.180 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:25:24.180 INFO:teuthology.orchestra.run.smithi107.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T09:25:24.180 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T09:25:24.181 INFO:teuthology.task.kernel:Updating grub on smithi107 to boot 5.14.0-508.el9.x86_64 2024-09-21T09:25:24.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T09:25:24.789 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2024-09-21T09:25:26.146 INFO:teuthology.orchestra.run.smithi107.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T09:25:26.184 INFO:teuthology.orchestra.run.smithi107.stderr:done 2024-09-21T09:25:26.186 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2024-09-21T09:25:26.202 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.n3m8YWBm4n 2024-09-21T09:25:26.202 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.n3m8YWBm4n 2024-09-21T09:25:26.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.n3m8YWBm4n 2024-09-21T09:25:26.407 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.n3m8YWBm4n is 6KB 2024-09-21T09:25:26.427 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.n3m8YWBm4n 2024-09-21T09:25:26.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T09:25:26.511 INFO:teuthology.orchestra.run.smithi107.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T09:25:26.511 INFO:teuthology.orchestra.run.smithi107.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T09:25:26.511 INFO:teuthology.orchestra.run.smithi107.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T09:25:26.511 INFO:teuthology.orchestra.run.smithi107.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T09:25:26.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T09:25:26.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2024-09-21T09:25:37.842 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-21T09:25:37.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-21T09:25:37.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:25:56.690 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T09:25:56.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-09-21T09:25:56.691 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:26:31.151 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-09-21T09:26:37.851 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T09:26:40.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-09-21T09:26:40.154 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:26:46.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-21T09:26:46.858 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:26:50.553 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-09-21T09:26:50.947 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-21T09:26:50.947 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-21T09:26:50.947 DEBUG:teuthology.orchestra.run.smithi129:> uname -r 2024-09-21T09:26:50.961 INFO:teuthology.orchestra.run.smithi129.stdout:5.14.0-508.el9.x86_64 2024-09-21T09:26:50.962 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T09:26:50.962 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T09:26:50.962 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T09:26:51.963 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-21T09:26:51.963 DEBUG:teuthology.orchestra.run.smithi129:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T09:26:52.039 INFO:teuthology.orchestra.run.smithi129.stdout:ttyS1 2024-09-21T09:26:52.068 DEBUG:teuthology.parallel:result is None 2024-09-21T09:26:58.543 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-09-21T09:27:10.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-09-21T09:27:10.544 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:27:11.110 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-09-21T09:27:11.454 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-09-21T09:27:11.454 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-21T09:27:11.455 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-09-21T09:27:11.469 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-508.el9.x86_64 2024-09-21T09:27:11.469 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T09:27:11.470 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T09:27:11.470 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T09:27:12.471 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T09:27:12.471 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T09:27:12.540 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2024-09-21T09:27:12.573 DEBUG:teuthology.parallel:result is None 2024-09-21T09:27:12.573 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T09:27:12.580 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T09:27:12.580 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T09:27:12.582 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T09:27:12.603 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T09:27:12.609 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T09:27:12.636 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T09:27:12.636 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T09:27:12.638 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T09:27:12.774 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T09:27:12.781 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T09:27:12.781 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T09:27:12.783 DEBUG:teuthology.orchestra.run.smithi129:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T09:27:12.813 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T09:27:12.815 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T09:27:12.823 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T09:27:12.826 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T09:27:12.828 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T09:27:12.835 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T09:27:12.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T09:27:12.864 DEBUG:teuthology.orchestra.run.smithi129:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T09:27:12.897 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T09:27:12.905 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T09:27:12.905 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T09:27:12.926 DEBUG:teuthology.orchestra.run.smithi129:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T09:27:12.954 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T09:27:13.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T09:27:13.069 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T09:27:13.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T09:27:13.184 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T09:27:13.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T09:27:13.247 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T09:27:13.292 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T09:27:13.340 DEBUG:teuthology.orchestra.run.smithi129:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T09:27:13.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T09:27:13.468 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T09:27:13.468 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T09:27:13.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2024-09-21T09:27:13.537 DEBUG:teuthology.orchestra.run.smithi129:> sudo service rsyslog restart 2024-09-21T09:27:13.579 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T09:27:13.624 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T09:27:13.958 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T09:27:13.971 INFO:teuthology.task.internal:Starting timer... 2024-09-21T09:27:13.971 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T09:27:14.031 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T09:27:14.063 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-21T09:27:14.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2024-09-21T09:27:14.139 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2024-09-21T09:27:14.141 DEBUG:teuthology.orchestra.run.smithi129:> sudo service auditd rotate 2024-09-21T09:27:14.209 INFO:teuthology.orchestra.run.smithi129.stdout:Rotating logs: 2024-09-21T09:27:14.211 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T09:27:14.211 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2024-09-21T09:27:14.244 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2024-09-21T09:27:14.244 DEBUG:teuthology.orchestra.run.smithi129:> /usr/sbin/getenforce 2024-09-21T09:27:14.267 INFO:teuthology.orchestra.run.smithi129.stdout:Permissive 2024-09-21T09:27:14.267 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi107.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi129.front.sepia.ceph.com': 'permissive'} 2024-09-21T09:27:14.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T09:27:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T09:27:14.309 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T09:27:14.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T09:27:14.336 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T09:27:14.336 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2024-09-21T09:27:14.414 DEBUG:teuthology.orchestra.run.smithi129:> sudo /usr/sbin/setenforce permissive 2024-09-21T09:27:14.445 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T09:27:14.455 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-21T09:27:14.457 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-21T09:27:14.563 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T09:27:14.573 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T09:27:14.574 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi107.front.sepia.ceph.com,smithi129.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T09:30:56.700 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi107.front.sepia.ceph.com'), Remote(name='ubuntu@smithi129.front.sepia.ceph.com')] 2024-09-21T09:30:56.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-09-21T09:30:56.701 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:30:56.778 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-09-21T09:30:56.850 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-09-21T09:30:56.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-21T09:30:56.851 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi129.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:30:56.926 DEBUG:teuthology.orchestra.run.smithi129:> true 2024-09-21T09:30:57.007 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi129.front.sepia.ceph.com' 2024-09-21T09:30:57.007 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T09:30:57.016 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T09:30:57.016 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T09:30:57.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T09:30:57.019 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T09:30:57.019 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T09:30:57.046 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T09:30:57.061 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T09:30:57.078 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T09:30:57.085 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: ntpd: command not found 2024-09-21T09:30:57.094 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T09:30:57.096 INFO:teuthology.orchestra.run.smithi107.stdout:506 Cannot talk to daemon 2024-09-21T09:30:57.110 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T09:30:57.120 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: ntpd: command not found 2024-09-21T09:30:57.124 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T09:30:57.131 INFO:teuthology.orchestra.run.smithi129.stdout:506 Cannot talk to daemon 2024-09-21T09:30:57.146 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T09:30:57.160 INFO:teuthology.orchestra.run.smithi129.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T09:30:57.185 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-09-21T09:30:57.188 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T09:30:57.188 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-09-21T09:30:57.188 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:30:57.188 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:30:57.188 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:30:57.189 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:30:57.218 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-09-21T09:30:57.221 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T09:30:57.222 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-09-21T09:30:57.222 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:30:57.222 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:30:57.222 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:30:57.222 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:30:57.223 INFO:teuthology.run_tasks:Running task pexec... 2024-09-21T09:30:57.234 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-21T09:30:57.234 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi107.front.sepia.ceph.com 2024-09-21T09:30:57.235 DEBUG:teuthology.orchestra.run.smithi107:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-21T09:30:57.235 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi129.front.sepia.ceph.com 2024-09-21T09:30:57.235 DEBUG:teuthology.orchestra.run.smithi129:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-21T09:30:57.586 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:30:57.594 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:30:57.594 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-09-21T09:30:57.595 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:30:57.595 INFO:teuthology.orchestra.run.smithi107.stdout:Removing: 2024-09-21T09:30:57.595 INFO:teuthology.orchestra.run.smithi107.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-21T09:30:57.595 INFO:teuthology.orchestra.run.smithi107.stdout:Removing unused dependencies: 2024-09-21T09:30:57.595 INFO:teuthology.orchestra.run.smithi107.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-21T09:30:57.595 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:30:57.595 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T09:30:57.595 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:30:57.595 INFO:teuthology.orchestra.run.smithi107.stdout:Remove 2 Packages 2024-09-21T09:30:57.595 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:30:57.595 INFO:teuthology.orchestra.run.smithi107.stdout:Freed space: 5.5 M 2024-09-21T09:30:57.595 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T09:30:57.599 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T09:30:57.599 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T09:30:57.617 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:30:57.625 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:30:57.625 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-09-21T09:30:57.625 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:30:57.625 INFO:teuthology.orchestra.run.smithi129.stdout:Removing: 2024-09-21T09:30:57.625 INFO:teuthology.orchestra.run.smithi129.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-21T09:30:57.625 INFO:teuthology.orchestra.run.smithi129.stdout:Removing unused dependencies: 2024-09-21T09:30:57.625 INFO:teuthology.orchestra.run.smithi129.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-21T09:30:57.625 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:30:57.626 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T09:30:57.626 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:30:57.626 INFO:teuthology.orchestra.run.smithi129.stdout:Remove 2 Packages 2024-09-21T09:30:57.626 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:30:57.626 INFO:teuthology.orchestra.run.smithi129.stdout:Freed space: 5.5 M 2024-09-21T09:30:57.626 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T09:30:57.630 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T09:30:57.630 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T09:30:57.670 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T09:30:57.670 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T09:30:57.684 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T09:30:57.684 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T09:30:57.804 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T09:30:57.822 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T09:30:57.886 INFO:teuthology.orchestra.run.smithi107.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-21T09:30:57.916 INFO:teuthology.orchestra.run.smithi129.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-21T09:30:57.943 INFO:teuthology.orchestra.run.smithi107.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T09:30:57.969 INFO:teuthology.orchestra.run.smithi129.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T09:30:58.362 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T09:30:58.362 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-21T09:30:58.422 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T09:30:58.422 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-21T09:30:58.601 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-21T09:30:58.602 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:30:58.602 INFO:teuthology.orchestra.run.smithi107.stdout:Removed: 2024-09-21T09:30:58.602 INFO:teuthology.orchestra.run.smithi107.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T09:30:58.602 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:30:58.602 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:30:58.669 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-21T09:30:58.669 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:30:58.670 INFO:teuthology.orchestra.run.smithi129.stdout:Removed: 2024-09-21T09:30:58.670 INFO:teuthology.orchestra.run.smithi129.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T09:30:58.670 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:30:58.670 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:30:59.415 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:49 ago on Sat 21 Sep 2024 09:30:10 AM UTC. 2024-09-21T09:30:59.515 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:48 ago on Sat 21 Sep 2024 09:30:11 AM UTC. 2024-09-21T09:30:59.549 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:30:59.549 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:30:59.550 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T09:30:59.550 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:30:59.550 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T09:30:59.550 INFO:teuthology.orchestra.run.smithi107.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-21T09:30:59.550 INFO:teuthology.orchestra.run.smithi107.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-21T09:30:59.550 INFO:teuthology.orchestra.run.smithi107.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-21T09:30:59.550 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-21T09:30:59.550 INFO:teuthology.orchestra.run.smithi107.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-21T09:30:59.550 INFO:teuthology.orchestra.run.smithi107.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-21T09:30:59.550 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-21T09:30:59.550 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-21T09:30:59.550 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:30:59.550 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T09:30:59.551 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:30:59.551 INFO:teuthology.orchestra.run.smithi107.stdout:Install 7 Packages 2024-09-21T09:30:59.551 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:30:59.560 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 5.2 M 2024-09-21T09:30:59.560 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 19 M 2024-09-21T09:30:59.560 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T09:30:59.650 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:30:59.651 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:30:59.651 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T09:30:59.651 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:30:59.651 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T09:30:59.651 INFO:teuthology.orchestra.run.smithi129.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-21T09:30:59.652 INFO:teuthology.orchestra.run.smithi129.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-21T09:30:59.652 INFO:teuthology.orchestra.run.smithi129.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-21T09:30:59.652 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-21T09:30:59.652 INFO:teuthology.orchestra.run.smithi129.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-21T09:30:59.652 INFO:teuthology.orchestra.run.smithi129.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-21T09:30:59.652 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-21T09:30:59.652 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-21T09:30:59.652 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:30:59.652 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T09:30:59.652 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:30:59.652 INFO:teuthology.orchestra.run.smithi129.stdout:Install 7 Packages 2024-09-21T09:30:59.652 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:30:59.685 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 5.2 M 2024-09-21T09:30:59.685 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 19 M 2024-09-21T09:30:59.685 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T09:30:59.924 INFO:teuthology.orchestra.run.smithi129.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 471 kB/s | 44 kB 00:00 2024-09-21T09:30:59.958 INFO:teuthology.orchestra.run.smithi129.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 826 kB/s | 106 kB 00:00 2024-09-21T09:30:59.983 INFO:teuthology.orchestra.run.smithi129.stdout:(3/7): python3-configshell-1.1.30-1.el9.noarch. 1.2 MB/s | 72 kB 00:00 2024-09-21T09:31:00.033 INFO:teuthology.orchestra.run.smithi129.stdout:(4/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 4.3 MB/s | 906 kB 00:00 2024-09-21T09:31:00.059 INFO:teuthology.orchestra.run.smithi129.stdout:(5/7): python3-kmod-0.9-32.el9.x86_64.rpm 837 kB/s | 84 kB 00:00 2024-09-21T09:31:00.117 INFO:teuthology.orchestra.run.smithi129.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 6.1 MB/s | 837 kB 00:00 2024-09-21T09:31:00.181 INFO:teuthology.orchestra.run.smithi107.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 131 kB/s | 44 kB 00:00 2024-09-21T09:31:00.290 INFO:teuthology.orchestra.run.smithi107.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 237 kB/s | 106 kB 00:00 2024-09-21T09:31:00.292 INFO:teuthology.orchestra.run.smithi129.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 12 MB/s | 3.2 MB 00:00 2024-09-21T09:31:00.293 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:31:00.293 INFO:teuthology.orchestra.run.smithi129.stdout:Total 8.5 MB/s | 5.2 MB 00:00 2024-09-21T09:31:00.401 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T09:31:00.416 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T09:31:00.416 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T09:31:00.599 INFO:teuthology.orchestra.run.smithi107.stdout:(3/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 1.2 MB/s | 906 kB 00:00 2024-09-21T09:31:00.624 INFO:teuthology.orchestra.run.smithi107.stdout:(4/7): python3-configshell-1.1.30-1.el9.noarch. 162 kB/s | 72 kB 00:00 2024-09-21T09:31:00.641 INFO:teuthology.orchestra.run.smithi107.stdout:(5/7): python3-kmod-0.9-32.el9.x86_64.rpm 240 kB/s | 84 kB 00:00 2024-09-21T09:31:00.722 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T09:31:00.723 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T09:31:01.191 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T09:31:01.274 INFO:teuthology.orchestra.run.smithi107.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 1.2 MB/s | 837 kB 00:00 2024-09-21T09:31:01.278 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-21T09:31:01.313 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-21T09:31:01.342 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-21T09:31:01.358 INFO:teuthology.orchestra.run.smithi107.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 4.3 MB/s | 3.2 MB 00:00 2024-09-21T09:31:01.358 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:31:01.358 INFO:teuthology.orchestra.run.smithi107.stdout:Total 2.9 MB/s | 5.2 MB 00:01 2024-09-21T09:31:01.466 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T09:31:01.481 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T09:31:01.481 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T09:31:01.552 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-21T09:31:01.581 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T09:31:01.805 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T09:31:01.805 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T09:31:02.017 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T09:31:02.052 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T09:31:02.158 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T09:31:02.230 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T09:31:02.280 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T09:31:02.368 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-21T09:31:02.407 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-21T09:31:02.432 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-21T09:31:02.638 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-21T09:31:02.671 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T09:31:03.105 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T09:31:03.132 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T09:31:03.149 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T09:31:03.149 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-21T09:31:03.150 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-21T09:31:03.150 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-21T09:31:03.150 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-21T09:31:03.150 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-21T09:31:03.150 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-21T09:31:03.250 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T09:31:03.310 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T09:31:03.488 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T09:31:03.488 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:31:03.488 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T09:31:03.488 INFO:teuthology.orchestra.run.smithi129.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T09:31:03.488 INFO:teuthology.orchestra.run.smithi129.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-21T09:31:03.488 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-21T09:31:03.488 INFO:teuthology.orchestra.run.smithi129.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-21T09:31:03.488 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:31:03.488 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:31:03.860 DEBUG:teuthology.parallel:result is None 2024-09-21T09:31:04.311 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T09:31:04.311 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-21T09:31:04.311 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-21T09:31:04.311 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-21T09:31:04.311 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-21T09:31:04.311 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-21T09:31:04.311 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-21T09:31:04.665 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T09:31:04.665 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:31:04.665 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T09:31:04.665 INFO:teuthology.orchestra.run.smithi107.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T09:31:04.665 INFO:teuthology.orchestra.run.smithi107.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-21T09:31:04.665 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-21T09:31:04.665 INFO:teuthology.orchestra.run.smithi107.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-21T09:31:04.666 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:31:04.666 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:31:05.062 DEBUG:teuthology.parallel:result is None 2024-09-21T09:31:05.063 INFO:teuthology.run_tasks:Running task install... 2024-09-21T09:31:05.071 DEBUG:teuthology.task.install:project ceph 2024-09-21T09:31:05.071 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}} 2024-09-21T09:31:05.071 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'} 2024-09-21T09:31:05.071 INFO:teuthology.task.install:Using flavor: default 2024-09-21T09:31:05.077 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T09:31:05.077 INFO:teuthology.task.install:extra packages: [] 2024-09-21T09:31:05.077 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'tag': None, 'wait_for_package': False} 2024-09-21T09:31:05.077 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:31:05.078 INFO:teuthology.packaging:ref: None 2024-09-21T09:31:05.078 INFO:teuthology.packaging:tag: None 2024-09-21T09:31:05.078 INFO:teuthology.packaging:branch: quincy 2024-09-21T09:31:05.078 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:31:05.078 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-21T09:31:05.079 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'tag': None, 'wait_for_package': False} 2024-09-21T09:31:05.080 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:31:05.080 INFO:teuthology.packaging:ref: None 2024-09-21T09:31:05.080 INFO:teuthology.packaging:tag: None 2024-09-21T09:31:05.080 INFO:teuthology.packaging:branch: quincy 2024-09-21T09:31:05.080 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:31:05.080 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-21T09:31:05.292 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/ 2024-09-21T09:31:05.292 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1649.gfbd1a6ab 2024-09-21T09:31:05.324 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/ 2024-09-21T09:31:05.324 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1649.gfbd1a6ab 2024-09-21T09:31:05.400 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T09:31:05.400 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T09:31:05.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T09:31:05.427 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T09:31:05.427 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:31:05.427 INFO:teuthology.packaging:ref: None 2024-09-21T09:31:05.428 INFO:teuthology.packaging:tag: None 2024-09-21T09:31:05.428 INFO:teuthology.packaging:branch: quincy 2024-09-21T09:31:05.428 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:31:05.428 DEBUG:teuthology.orchestra.run.smithi107:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T09:31:05.438 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T09:31:05.438 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T09:31:05.438 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T09:31:05.469 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T09:31:05.469 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:31:05.469 INFO:teuthology.packaging:ref: None 2024-09-21T09:31:05.469 INFO:teuthology.packaging:tag: None 2024-09-21T09:31:05.469 INFO:teuthology.packaging:branch: quincy 2024-09-21T09:31:05.469 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:31:05.469 DEBUG:teuthology.orchestra.run.smithi129:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T09:31:05.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T09:31:05.539 DEBUG:teuthology.orchestra.run.smithi129:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T09:31:05.570 DEBUG:teuthology.orchestra.run.smithi107:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T09:31:05.619 DEBUG:teuthology.orchestra.run.smithi129:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T09:31:05.634 INFO:teuthology.orchestra.run.smithi107.stdout:check_obsoletes = 1 2024-09-21T09:31:05.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum clean all 2024-09-21T09:31:05.685 INFO:teuthology.orchestra.run.smithi129.stdout:check_obsoletes = 1 2024-09-21T09:31:05.687 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum clean all 2024-09-21T09:31:05.876 INFO:teuthology.orchestra.run.smithi107.stdout:45 files removed 2024-09-21T09:31:05.903 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw 2024-09-21T09:31:05.924 INFO:teuthology.orchestra.run.smithi129.stdout:45 files removed 2024-09-21T09:31:05.952 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-radosgw 2024-09-21T09:31:06.455 INFO:teuthology.orchestra.run.smithi107.stdout:ceph packages for x86_64 284 kB/s | 77 kB 00:00 2024-09-21T09:31:06.502 INFO:teuthology.orchestra.run.smithi129.stdout:ceph packages for x86_64 289 kB/s | 77 kB 00:00 2024-09-21T09:31:06.739 INFO:teuthology.orchestra.run.smithi107.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-21T09:31:06.778 INFO:teuthology.orchestra.run.smithi129.stdout:ceph noarch packages 46 kB/s | 11 kB 00:00 2024-09-21T09:31:07.006 INFO:teuthology.orchestra.run.smithi107.stdout:ceph source packages 7.7 kB/s | 1.8 kB 00:00 2024-09-21T09:31:07.046 INFO:teuthology.orchestra.run.smithi129.stdout:ceph source packages 7.7 kB/s | 1.8 kB 00:00 2024-09-21T09:31:07.739 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.3 MB 00:00 2024-09-21T09:31:07.762 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.3 MB 00:00 2024-09-21T09:31:10.999 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 9.2 MB/s | 20 MB 00:02 2024-09-21T09:31:15.218 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-21T09:31:16.396 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 20 MB 00:07 2024-09-21T09:31:16.860 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 19 kB 00:00 2024-09-21T09:31:17.756 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-21T09:31:21.781 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.5 MB 00:01 2024-09-21T09:31:23.345 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-21T09:31:23.490 INFO:teuthology.orchestra.run.smithi129.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 19 kB 00:00 2024-09-21T09:31:24.261 INFO:teuthology.orchestra.run.smithi129.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-21T09:31:25.575 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:31:25.576 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:31:25.576 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T09:31:25.577 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:31:25.577 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T09:31:25.577 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 14 M 2024-09-21T09:31:25.577 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2024-09-21T09:31:25.577 INFO:teuthology.orchestra.run.smithi107.stdout: librados2 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 3.1 M 2024-09-21T09:31:25.577 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 2.9 M 2024-09-21T09:31:25.577 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-21T09:31:25.577 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 5.0 M 2024-09-21T09:31:25.577 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 22 M 2024-09-21T09:31:25.577 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 25 k 2024-09-21T09:31:25.577 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T09:31:25.577 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 636 k 2024-09-21T09:31:25.577 INFO:teuthology.orchestra.run.smithi107.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T09:31:25.578 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T09:31:25.578 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T09:31:25.578 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 457 k 2024-09-21T09:31:25.578 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T09:31:25.578 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 4.9 M 2024-09-21T09:31:25.578 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T09:31:25.578 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T09:31:25.578 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 45 k 2024-09-21T09:31:25.578 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 120 k 2024-09-21T09:31:25.578 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 156 k 2024-09-21T09:31:25.578 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T09:31:25.578 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 320 k 2024-09-21T09:31:25.578 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 295 k 2024-09-21T09:31:25.579 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 99 k 2024-09-21T09:31:25.579 INFO:teuthology.orchestra.run.smithi107.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T09:31:25.579 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:31:25.579 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T09:31:25.579 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:31:25.579 INFO:teuthology.orchestra.run.smithi107.stdout:Install 22 Packages 2024-09-21T09:31:25.579 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 2 Packages 2024-09-21T09:31:25.579 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:31:25.579 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 58 M 2024-09-21T09:31:25.580 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T09:31:26.296 INFO:teuthology.orchestra.run.smithi107.stdout:(1/24): ceph-base-17.2.7-1649.gfbd1a6ab.el9.x86 11 MB/s | 5.0 MB 00:00 2024-09-21T09:31:26.354 INFO:teuthology.orchestra.run.smithi107.stdout:(2/24): ceph-selinux-17.2.7-1649.gfbd1a6ab.el9. 432 kB/s | 25 kB 00:00 2024-09-21T09:31:26.438 INFO:teuthology.orchestra.run.smithi107.stdout:(3/24): libcephfs2-17.2.7-1649.gfbd1a6ab.el9.x8 7.5 MB/s | 636 kB 00:00 2024-09-21T09:31:26.538 INFO:teuthology.orchestra.run.smithi107.stdout:(4/24): libradosstriper1-17.2.7-1649.gfbd1a6ab. 4.5 MB/s | 457 kB 00:00 2024-09-21T09:31:26.864 INFO:teuthology.orchestra.run.smithi107.stdout:(5/24): ceph-radosgw-17.2.7-1649.gfbd1a6ab.el9. 14 MB/s | 14 MB 00:01 2024-09-21T09:31:26.955 INFO:teuthology.orchestra.run.smithi107.stdout:(6/24): librgw2-17.2.7-1649.gfbd1a6ab.el9.x86_6 12 MB/s | 4.9 MB 00:00 2024-09-21T09:31:26.972 INFO:teuthology.orchestra.run.smithi107.stdout:(7/24): python3-ceph-argparse-17.2.7-1649.gfbd1 415 kB/s | 45 kB 00:00 2024-09-21T09:31:27.006 INFO:teuthology.orchestra.run.smithi107.stdout:(8/24): python3-ceph-common-17.2.7-1649.gfbd1a6 2.3 MB/s | 120 kB 00:00 2024-09-21T09:31:27.031 INFO:teuthology.orchestra.run.smithi107.stdout:(9/24): python3-cephfs-17.2.7-1649.gfbd1a6ab.el 2.6 MB/s | 156 kB 00:00 2024-09-21T09:31:27.065 INFO:teuthology.orchestra.run.smithi107.stdout:(10/24): python3-rados-17.2.7-1649.gfbd1a6ab.el 5.3 MB/s | 320 kB 00:00 2024-09-21T09:31:27.090 INFO:teuthology.orchestra.run.smithi107.stdout:(11/24): python3-rbd-17.2.7-1649.gfbd1a6ab.el9. 4.9 MB/s | 295 kB 00:00 2024-09-21T09:31:27.116 INFO:teuthology.orchestra.run.smithi107.stdout:(12/24): python3-rgw-17.2.7-1649.gfbd1a6ab.el9. 1.9 MB/s | 99 kB 00:00 2024-09-21T09:31:27.324 INFO:teuthology.orchestra.run.smithi107.stdout:(13/24): ceph-common-17.2.7-1649.gfbd1a6ab.el9. 15 MB/s | 22 MB 00:01 2024-09-21T09:31:27.833 INFO:teuthology.orchestra.run.smithi107.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 63 kB/s | 45 kB 00:00 2024-09-21T09:31:27.967 INFO:teuthology.orchestra.run.smithi107.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 183 kB/s | 160 kB 00:00 2024-09-21T09:31:28.250 INFO:teuthology.orchestra.run.smithi107.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 716 kB/s | 662 kB 00:00 2024-09-21T09:31:28.309 INFO:teuthology.orchestra.run.smithi107.stdout:(17/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 5.2 MB/s | 308 kB 00:00 2024-09-21T09:31:28.334 INFO:teuthology.orchestra.run.smithi107.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 2.0 MB/s | 49 kB 00:00 2024-09-21T09:31:28.359 INFO:teuthology.orchestra.run.smithi107.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-21T09:31:28.418 INFO:teuthology.orchestra.run.smithi107.stdout:(20/24): thrift-0.15.0-2.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-09-21T09:31:28.668 INFO:teuthology.orchestra.run.smithi107.stdout:(21/24): librados2-17.2.7-1649.gfbd1a6ab.el9.x8 12 MB/s | 3.1 MB 00:00 2024-09-21T09:31:28.693 INFO:teuthology.orchestra.run.smithi107.stdout:(22/24): python3-prettytable-0.7.2-27.el9.noarc 58 kB/s | 42 kB 00:00 2024-09-21T09:31:28.718 INFO:teuthology.orchestra.run.smithi107.stdout:(23/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 330 kB/s | 292 kB 00:00 2024-09-21T09:31:28.785 INFO:teuthology.orchestra.run.smithi107.stdout:(24/24): librbd1-17.2.7-1649.gfbd1a6ab.el9.x86_ 25 MB/s | 2.9 MB 00:00 2024-09-21T09:31:28.791 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:31:28.791 INFO:teuthology.orchestra.run.smithi107.stdout:Total 18 MB/s | 58 MB 00:03 2024-09-21T09:31:28.854 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T09:31:28.888 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T09:31:28.888 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T09:31:29.364 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T09:31:29.364 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T09:31:29.575 INFO:teuthology.orchestra.run.smithi129.stdout:lab-extras 44 kB/s | 1.7 kB 00:00 2024-09-21T09:31:30.554 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T09:31:30.921 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-21T09:31:30.988 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/26 2024-09-21T09:31:31.072 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/26 2024-09-21T09:31:31.150 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/26 2024-09-21T09:31:31.181 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/26 2024-09-21T09:31:31.252 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/26 2024-09-21T09:31:31.298 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-21T09:31:31.324 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-21T09:31:31.772 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-21T09:31:31.797 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 8/26 2024-09-21T09:31:31.842 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:31:31.843 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 8/26 2024-09-21T09:31:31.844 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:31:31.844 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T09:31:31.844 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:31:31.844 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T09:31:31.844 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 14 M 2024-09-21T09:31:31.844 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrading: 2024-09-21T09:31:31.844 INFO:teuthology.orchestra.run.smithi129.stdout: librados2 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 3.1 M 2024-09-21T09:31:31.844 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 2.9 M 2024-09-21T09:31:31.844 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-21T09:31:31.845 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 5.0 M 2024-09-21T09:31:31.845 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 22 M 2024-09-21T09:31:31.845 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 25 k 2024-09-21T09:31:31.845 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T09:31:31.845 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 636 k 2024-09-21T09:31:31.845 INFO:teuthology.orchestra.run.smithi129.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T09:31:31.845 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T09:31:31.845 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T09:31:31.845 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 457 k 2024-09-21T09:31:31.845 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T09:31:31.845 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 4.9 M 2024-09-21T09:31:31.845 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T09:31:31.845 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T09:31:31.846 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 45 k 2024-09-21T09:31:31.846 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 120 k 2024-09-21T09:31:31.846 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 156 k 2024-09-21T09:31:31.846 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T09:31:31.846 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 320 k 2024-09-21T09:31:31.846 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 295 k 2024-09-21T09:31:31.846 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 99 k 2024-09-21T09:31:31.846 INFO:teuthology.orchestra.run.smithi129.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T09:31:31.846 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:31:31.846 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T09:31:31.846 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:31:31.846 INFO:teuthology.orchestra.run.smithi129.stdout:Install 22 Packages 2024-09-21T09:31:31.846 INFO:teuthology.orchestra.run.smithi129.stdout:Upgrade 2 Packages 2024-09-21T09:31:31.846 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:31:31.847 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 58 M 2024-09-21T09:31:31.847 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T09:31:31.871 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-21T09:31:31.920 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-argparse-2:17.2.7-1649.gfbd1a6ab.el9. 10/26 2024-09-21T09:31:31.977 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 11/26 2024-09-21T09:31:32.046 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 12/26 2024-09-21T09:31:32.075 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_6 13/26 2024-09-21T09:31:32.141 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_6 13/26 2024-09-21T09:31:32.181 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-21T09:31:32.233 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-21T09:31:32.271 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-21T09:31:32.561 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-21T09:31:32.595 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 18/26 2024-09-21T09:31:32.684 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 18/26 2024-09-21T09:31:32.759 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 19/26 2024-09-21T09:31:32.789 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x8 20/26 2024-09-21T09:31:32.868 INFO:teuthology.orchestra.run.smithi129.stdout:(1/24): ceph-base-17.2.7-1649.gfbd1a6ab.el9.x86 12 MB/s | 5.0 MB 00:00 2024-09-21T09:31:32.927 INFO:teuthology.orchestra.run.smithi129.stdout:(2/24): ceph-selinux-17.2.7-1649.gfbd1a6ab.el9. 434 kB/s | 25 kB 00:00 2024-09-21T09:31:33.003 INFO:teuthology.orchestra.run.smithi129.stdout:(3/24): libcephfs2-17.2.7-1649.gfbd1a6ab.el9.x8 8.3 MB/s | 636 kB 00:00 2024-09-21T09:31:33.086 INFO:teuthology.orchestra.run.smithi129.stdout:(4/24): libradosstriper1-17.2.7-1649.gfbd1a6ab. 5.4 MB/s | 457 kB 00:00 2024-09-21T09:31:33.420 INFO:teuthology.orchestra.run.smithi129.stdout:(5/24): ceph-radosgw-17.2.7-1649.gfbd1a6ab.el9. 15 MB/s | 14 MB 00:00 2024-09-21T09:31:33.503 INFO:teuthology.orchestra.run.smithi129.stdout:(6/24): librgw2-17.2.7-1649.gfbd1a6ab.el9.x86_6 12 MB/s | 4.9 MB 00:00 2024-09-21T09:31:33.520 INFO:teuthology.orchestra.run.smithi129.stdout:(7/24): python3-ceph-argparse-17.2.7-1649.gfbd1 451 kB/s | 45 kB 00:00 2024-09-21T09:31:33.554 INFO:teuthology.orchestra.run.smithi129.stdout:(8/24): python3-ceph-common-17.2.7-1649.gfbd1a6 2.3 MB/s | 120 kB 00:00 2024-09-21T09:31:33.579 INFO:teuthology.orchestra.run.smithi129.stdout:(9/24): python3-cephfs-17.2.7-1649.gfbd1a6ab.el 2.6 MB/s | 156 kB 00:00 2024-09-21T09:31:33.605 INFO:teuthology.orchestra.run.smithi129.stdout:(10/24): python3-rados-17.2.7-1649.gfbd1a6ab.el 6.2 MB/s | 320 kB 00:00 2024-09-21T09:31:33.638 INFO:teuthology.orchestra.run.smithi129.stdout:(11/24): python3-rbd-17.2.7-1649.gfbd1a6ab.el9. 4.9 MB/s | 295 kB 00:00 2024-09-21T09:31:33.663 INFO:teuthology.orchestra.run.smithi129.stdout:(12/24): python3-rgw-17.2.7-1649.gfbd1a6ab.el9. 1.7 MB/s | 99 kB 00:00 2024-09-21T09:31:33.755 INFO:teuthology.orchestra.run.smithi129.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 493 kB/s | 45 kB 00:00 2024-09-21T09:31:33.981 INFO:teuthology.orchestra.run.smithi129.stdout:(14/24): ceph-common-17.2.7-1649.gfbd1a6ab.el9. 15 MB/s | 22 MB 00:01 2024-09-21T09:31:34.006 INFO:teuthology.orchestra.run.smithi129.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 436 kB/s | 160 kB 00:00 2024-09-21T09:31:34.040 INFO:teuthology.orchestra.run.smithi129.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.3 MB/s | 662 kB 00:00 2024-09-21T09:31:34.082 INFO:teuthology.orchestra.run.smithi129.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 559 kB/s | 42 kB 00:00 2024-09-21T09:31:34.124 INFO:teuthology.orchestra.run.smithi129.stdout:(18/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.6 MB/s | 308 kB 00:00 2024-09-21T09:31:34.149 INFO:teuthology.orchestra.run.smithi129.stdout:(19/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 292 kB 00:00 2024-09-21T09:31:34.175 INFO:teuthology.orchestra.run.smithi129.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 537 kB/s | 49 kB 00:00 2024-09-21T09:31:34.208 INFO:teuthology.orchestra.run.smithi129.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 817 kB/s | 67 kB 00:00 2024-09-21T09:31:34.267 INFO:teuthology.orchestra.run.smithi129.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-21T09:31:34.376 INFO:teuthology.orchestra.run.smithi129.stdout:(23/24): librados2-17.2.7-1649.gfbd1a6ab.el9.x8 16 MB/s | 3.1 MB 00:00 2024-09-21T09:31:34.442 INFO:teuthology.orchestra.run.smithi129.stdout:(24/24): librbd1-17.2.7-1649.gfbd1a6ab.el9.x86_ 13 MB/s | 2.9 MB 00:00 2024-09-21T09:31:34.447 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:31:34.448 INFO:teuthology.orchestra.run.smithi129.stdout:Total 22 MB/s | 58 MB 00:02 2024-09-21T09:31:34.505 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T09:31:34.537 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T09:31:34.537 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T09:31:35.012 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T09:31:35.013 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T09:31:35.220 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 21/26 2024-09-21T09:31:35.253 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 21/26 2024-09-21T09:31:35.821 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 21/26 2024-09-21T09:31:35.865 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 22/26 2024-09-21T09:31:35.948 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 22/26 2024-09-21T09:31:35.948 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T09:31:35.948 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T09:31:35.948 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:31:35.974 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 23/26 2024-09-21T09:31:36.193 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T09:31:36.571 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-21T09:31:36.645 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/26 2024-09-21T09:31:36.737 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/26 2024-09-21T09:31:36.815 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/26 2024-09-21T09:31:36.863 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/26 2024-09-21T09:31:36.960 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/26 2024-09-21T09:31:37.012 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-21T09:31:37.044 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-21T09:31:37.492 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-21T09:31:37.517 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 8/26 2024-09-21T09:31:37.580 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 8/26 2024-09-21T09:31:37.616 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-21T09:31:37.661 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-argparse-2:17.2.7-1649.gfbd1a6ab.el9. 10/26 2024-09-21T09:31:37.697 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 11/26 2024-09-21T09:31:37.758 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 12/26 2024-09-21T09:31:37.795 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_6 13/26 2024-09-21T09:31:37.861 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_6 13/26 2024-09-21T09:31:37.905 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-21T09:31:37.958 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-21T09:31:37.998 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-21T09:31:38.293 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-21T09:31:38.329 INFO:teuthology.orchestra.run.smithi129.stdout: Upgrading : librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 18/26 2024-09-21T09:31:38.402 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 18/26 2024-09-21T09:31:38.476 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 19/26 2024-09-21T09:31:38.507 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x8 20/26 2024-09-21T09:31:41.023 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 21/26 2024-09-21T09:31:41.062 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 21/26 2024-09-21T09:31:41.620 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 21/26 2024-09-21T09:31:41.657 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 22/26 2024-09-21T09:31:41.766 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 22/26 2024-09-21T09:31:41.767 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T09:31:41.767 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T09:31:41.767 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:31:41.809 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 23/26 2024-09-21T09:31:45.769 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 23/26 2024-09-21T09:31:45.769 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /sys 2024-09-21T09:31:45.769 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /proc 2024-09-21T09:31:45.769 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /mnt 2024-09-21T09:31:45.769 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /var/tmp 2024-09-21T09:31:45.769 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /home 2024-09-21T09:31:45.769 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /root 2024-09-21T09:31:45.769 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /tmp 2024-09-21T09:31:45.769 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:31:47.074 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 24/26 2024-09-21T09:31:47.150 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 24/26 2024-09-21T09:31:47.151 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:31:47.151 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T09:31:47.151 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T09:31:47.151 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T09:31:47.151 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:31:47.152 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-21T09:31:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-21T09:31:47.235 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-21T09:31:48.532 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-21T09:31:48.532 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/26 2024-09-21T09:31:48.532 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/26 2024-09-21T09:31:48.532 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/26 2024-09-21T09:31:48.532 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/26 2024-09-21T09:31:48.532 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 5/26 2024-09-21T09:31:48.532 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_6 6/26 2024-09-21T09:31:48.532 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 7/26 2024-09-21T09:31:48.532 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1649.gfbd1a6ab.el9. 8/26 2024-09-21T09:31:48.532 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x8 9/26 2024-09-21T09:31:48.532 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 10/26 2024-09-21T09:31:48.532 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 11/26 2024-09-21T09:31:48.532 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 12/26 2024-09-21T09:31:48.533 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 13/26 2024-09-21T09:31:48.533 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-21T09:31:48.533 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-21T09:31:48.533 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-21T09:31:48.533 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-21T09:31:48.533 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-21T09:31:48.533 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-21T09:31:48.533 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-21T09:31:48.533 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-21T09:31:48.533 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-21T09:31:48.533 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 23/26 2024-09-21T09:31:48.533 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-21T09:31:48.533 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 25/26 2024-09-21T09:31:49.381 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-21T09:31:49.382 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:31:49.382 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2024-09-21T09:31:49.382 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:49.382 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:49.382 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T09:31:49.382 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:49.382 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:49.382 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:49.382 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:49.382 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T09:31:49.382 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:49.383 INFO:teuthology.orchestra.run.smithi107.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T09:31:49.383 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T09:31:49.383 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T09:31:49.383 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:49.383 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T09:31:49.383 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:49.383 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T09:31:49.383 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T09:31:49.383 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:49.383 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:49.383 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:49.383 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T09:31:49.383 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:49.383 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:49.383 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:49.384 INFO:teuthology.orchestra.run.smithi107.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T09:31:49.384 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:31:49.384 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:31:49.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-test 2024-09-21T09:31:50.336 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:27 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:31:50.467 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:31:50.467 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:31:50.468 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T09:31:50.468 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:31:50.468 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T09:31:50.468 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 48 M 2024-09-21T09:31:50.468 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-21T09:31:50.468 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 163 k 2024-09-21T09:31:50.468 INFO:teuthology.orchestra.run.smithi107.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T09:31:50.468 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T09:31:50.468 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:31:50.468 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T09:31:50.468 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:31:50.468 INFO:teuthology.orchestra.run.smithi107.stdout:Install 4 Packages 2024-09-21T09:31:50.468 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:31:50.469 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 48 M 2024-09-21T09:31:50.469 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 193 M 2024-09-21T09:31:50.469 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T09:31:50.662 INFO:teuthology.orchestra.run.smithi107.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.8 MB/s | 304 kB 00:00 2024-09-21T09:31:50.704 INFO:teuthology.orchestra.run.smithi107.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-21T09:31:50.729 INFO:teuthology.orchestra.run.smithi107.stdout:(3/4): libcephsqlite-17.2.7-1649.gfbd1a6ab.el9. 688 kB/s | 163 kB 00:00 2024-09-21T09:31:51.493 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 23/26 2024-09-21T09:31:51.493 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /sys 2024-09-21T09:31:51.493 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /proc 2024-09-21T09:31:51.493 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /mnt 2024-09-21T09:31:51.493 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /var/tmp 2024-09-21T09:31:51.493 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /home 2024-09-21T09:31:51.493 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /root 2024-09-21T09:31:51.493 INFO:teuthology.orchestra.run.smithi129.stdout:skipping the directory /tmp 2024-09-21T09:31:51.493 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:31:52.004 INFO:teuthology.orchestra.run.smithi107.stdout:(4/4): ceph-test-17.2.7-1649.gfbd1a6ab.el9.x86_ 32 MB/s | 48 MB 00:01 2024-09-21T09:31:52.007 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:31:52.007 INFO:teuthology.orchestra.run.smithi107.stdout:Total 32 MB/s | 48 MB 00:01 2024-09-21T09:31:52.034 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T09:31:52.053 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T09:31:52.053 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T09:31:52.420 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T09:31:52.421 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T09:31:52.828 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 24/26 2024-09-21T09:31:52.907 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 24/26 2024-09-21T09:31:52.907 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:31:52.907 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T09:31:52.907 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T09:31:52.907 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T09:31:52.907 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:31:52.909 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-21T09:31:52.983 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T09:31:52.998 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-21T09:31:52.998 INFO:teuthology.orchestra.run.smithi129.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-21T09:31:53.065 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T09:31:53.104 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T09:31:53.140 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/4 2024-09-21T09:31:54.286 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-21T09:31:54.287 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/26 2024-09-21T09:31:54.287 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/26 2024-09-21T09:31:54.287 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/26 2024-09-21T09:31:54.287 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/26 2024-09-21T09:31:54.287 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 5/26 2024-09-21T09:31:54.287 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_6 6/26 2024-09-21T09:31:54.287 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 7/26 2024-09-21T09:31:54.287 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1649.gfbd1a6ab.el9. 8/26 2024-09-21T09:31:54.287 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x8 9/26 2024-09-21T09:31:54.287 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 10/26 2024-09-21T09:31:54.287 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 11/26 2024-09-21T09:31:54.287 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 12/26 2024-09-21T09:31:54.287 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 13/26 2024-09-21T09:31:54.287 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-21T09:31:54.288 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-21T09:31:54.288 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-21T09:31:54.288 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-21T09:31:54.288 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-21T09:31:54.288 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-21T09:31:54.288 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-21T09:31:54.288 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-21T09:31:54.288 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-21T09:31:54.288 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 23/26 2024-09-21T09:31:54.288 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-21T09:31:54.288 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 25/26 2024-09-21T09:31:55.175 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-21T09:31:55.175 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:31:55.175 INFO:teuthology.orchestra.run.smithi129.stdout:Upgraded: 2024-09-21T09:31:55.175 INFO:teuthology.orchestra.run.smithi129.stdout: librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:55.175 INFO:teuthology.orchestra.run.smithi129.stdout: librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:55.175 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T09:31:55.175 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:55.176 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:55.176 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:55.176 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:55.176 INFO:teuthology.orchestra.run.smithi129.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T09:31:55.176 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:55.176 INFO:teuthology.orchestra.run.smithi129.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T09:31:55.176 INFO:teuthology.orchestra.run.smithi129.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T09:31:55.176 INFO:teuthology.orchestra.run.smithi129.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T09:31:55.176 INFO:teuthology.orchestra.run.smithi129.stdout: libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:55.176 INFO:teuthology.orchestra.run.smithi129.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T09:31:55.176 INFO:teuthology.orchestra.run.smithi129.stdout: librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:55.176 INFO:teuthology.orchestra.run.smithi129.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T09:31:55.176 INFO:teuthology.orchestra.run.smithi129.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T09:31:55.177 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-argparse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:55.177 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:55.177 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:55.177 INFO:teuthology.orchestra.run.smithi129.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T09:31:55.177 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:55.177 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:55.177 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:31:55.177 INFO:teuthology.orchestra.run.smithi129.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T09:31:55.177 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:31:55.177 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:31:55.523 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-test 2024-09-21T09:31:56.104 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:27 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:31:56.236 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:31:56.237 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:31:56.237 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T09:31:56.237 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:31:56.237 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T09:31:56.237 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 48 M 2024-09-21T09:31:56.237 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-21T09:31:56.237 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 163 k 2024-09-21T09:31:56.237 INFO:teuthology.orchestra.run.smithi129.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T09:31:56.237 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T09:31:56.238 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:31:56.238 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T09:31:56.238 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:31:56.238 INFO:teuthology.orchestra.run.smithi129.stdout:Install 4 Packages 2024-09-21T09:31:56.238 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:31:56.238 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 48 M 2024-09-21T09:31:56.238 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 193 M 2024-09-21T09:31:56.238 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T09:31:56.419 INFO:teuthology.orchestra.run.smithi129.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.0 MB/s | 304 kB 00:00 2024-09-21T09:31:56.461 INFO:teuthology.orchestra.run.smithi129.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-21T09:31:56.528 INFO:teuthology.orchestra.run.smithi129.stdout:(3/4): libcephsqlite-17.2.7-1649.gfbd1a6ab.el9. 630 kB/s | 163 kB 00:00 2024-09-21T09:31:57.803 INFO:teuthology.orchestra.run.smithi129.stdout:(4/4): ceph-test-17.2.7-1649.gfbd1a6ab.el9.x86_ 31 MB/s | 48 MB 00:01 2024-09-21T09:31:57.806 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:31:57.806 INFO:teuthology.orchestra.run.smithi129.stdout:Total 31 MB/s | 48 MB 00:01 2024-09-21T09:31:57.815 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/4 2024-09-21T09:31:57.834 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T09:31:57.853 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T09:31:57.853 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T09:31:57.901 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/4 2024-09-21T09:31:58.222 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T09:31:58.222 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T09:31:58.762 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T09:31:58.852 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T09:31:58.895 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T09:31:58.922 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/4 2024-09-21T09:32:00.088 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/4 2024-09-21T09:32:00.088 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/4 2024-09-21T09:32:00.088 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/4 2024-09-21T09:32:00.088 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T09:32:00.378 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T09:32:00.378 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:00.378 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T09:32:00.379 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:32:00.379 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:32:00.379 INFO:teuthology.orchestra.run.smithi107.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T09:32:00.379 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T09:32:00.379 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:00.379 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:32:00.662 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph 2024-09-21T09:32:01.244 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:38 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:32:01.383 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:32:01.385 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:01.385 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T09:32:01.386 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:01.386 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T09:32:01.386 INFO:teuthology.orchestra.run.smithi107.stdout: ceph x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 6.5 k 2024-09-21T09:32:01.386 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-21T09:32:01.386 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 2.1 M 2024-09-21T09:32:01.386 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 1.4 M 2024-09-21T09:32:01.386 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 240 k 2024-09-21T09:32:01.386 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 3.9 M 2024-09-21T09:32:01.386 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 15 M 2024-09-21T09:32:01.386 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T09:32:01.387 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T09:32:01.387 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T09:32:01.387 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T09:32:01.387 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T09:32:01.387 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T09:32:01.387 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T09:32:01.387 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections 2024-09-21T09:32:01.387 INFO:teuthology.orchestra.run.smithi107.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-21T09:32:01.387 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T09:32:01.387 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T09:32:01.387 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T09:32:01.387 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T09:32:01.388 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T09:32:01.388 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T09:32:01.388 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T09:32:01.388 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T09:32:01.388 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T09:32:01.388 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T09:32:01.388 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T09:32:01.388 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T09:32:01.388 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T09:32:01.388 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T09:32:01.388 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T09:32:01.388 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T09:32:01.388 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T09:32:01.389 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T09:32:01.389 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T09:32:01.389 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T09:32:01.389 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:01.389 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T09:32:01.389 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:01.389 INFO:teuthology.orchestra.run.smithi107.stdout:Install 34 Packages 2024-09-21T09:32:01.389 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:01.390 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 27 M 2024-09-21T09:32:01.390 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 96 M 2024-09-21T09:32:01.390 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T09:32:01.739 INFO:teuthology.orchestra.run.smithi107.stdout:(1/34): ceph-17.2.7-1649.gfbd1a6ab.el9.x86_64.r 53 kB/s | 6.5 kB 00:00 2024-09-21T09:32:01.927 INFO:teuthology.orchestra.run.smithi107.stdout:(2/34): ceph-mgr-17.2.7-1649.gfbd1a6ab.el9.x86_ 4.5 MB/s | 1.4 MB 00:00 2024-09-21T09:32:01.977 INFO:teuthology.orchestra.run.smithi107.stdout:(3/34): ceph-mds-17.2.7-1649.gfbd1a6ab.el9.x86_ 5.7 MB/s | 2.1 MB 00:00 2024-09-21T09:32:02.069 INFO:teuthology.orchestra.run.smithi107.stdout:(4/34): ceph-mon-17.2.7-1649.gfbd1a6ab.el9.x86_ 12 MB/s | 3.9 MB 00:00 2024-09-21T09:32:02.094 INFO:teuthology.orchestra.run.smithi107.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1649.gfbd1 2.0 MB/s | 240 kB 00:00 2024-09-21T09:32:02.245 INFO:teuthology.orchestra.run.smithi107.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-09-21T09:32:02.286 INFO:teuthology.orchestra.run.smithi107.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-21T09:32:02.445 INFO:teuthology.orchestra.run.smithi107.stdout:(8/34): ceph-osd-17.2.7-1649.gfbd1a6ab.el9.x86_ 30 MB/s | 15 MB 00:00 2024-09-21T09:32:02.462 INFO:teuthology.orchestra.run.smithi107.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 771 kB/s | 135 kB 00:00 2024-09-21T09:32:02.504 INFO:teuthology.orchestra.run.smithi107.stdout:(10/34): python3-cryptography-36.0.1-4.el9.x86_ 3.0 MB/s | 1.2 MB 00:00 2024-09-21T09:32:02.529 INFO:teuthology.orchestra.run.smithi107.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-21T09:32:02.629 INFO:teuthology.orchestra.run.smithi107.stdout:(12/34): python3-requests-2.25.1-8.el9.noarch.r 679 kB/s | 125 kB 00:00 2024-09-21T09:32:02.680 INFO:teuthology.orchestra.run.smithi107.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 231 kB/s | 35 kB 00:00 2024-09-21T09:32:02.780 INFO:teuthology.orchestra.run.smithi107.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 624 kB/s | 172 kB 00:00 2024-09-21T09:32:02.831 INFO:teuthology.orchestra.run.smithi107.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 339 kB/s | 51 kB 00:00 2024-09-21T09:32:02.856 INFO:teuthology.orchestra.run.smithi107.stdout:(16/34): python3-toml-0.10.2-6.el9.noarch.rpm 553 kB/s | 42 kB 00:00 2024-09-21T09:32:02.881 INFO:teuthology.orchestra.run.smithi107.stdout:(17/34): python3-packaging-20.9-5.el9.noarch.rp 307 kB/s | 77 kB 00:00 2024-09-21T09:32:02.931 INFO:teuthology.orchestra.run.smithi107.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-21T09:32:02.957 INFO:teuthology.orchestra.run.smithi107.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 459 kB/s | 11 kB 00:00 2024-09-21T09:32:02.982 INFO:teuthology.orchestra.run.smithi107.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-21T09:32:03.007 INFO:teuthology.orchestra.run.smithi107.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-09-21T09:32:03.032 INFO:teuthology.orchestra.run.smithi107.stdout:(22/34): python3-jaraco-functools-3.5.0-2.el9.n 780 kB/s | 19 kB 00:00 2024-09-21T09:32:03.058 INFO:teuthology.orchestra.run.smithi107.stdout:(23/34): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-21T09:32:03.083 INFO:teuthology.orchestra.run.smithi107.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-21T09:32:03.109 INFO:teuthology.orchestra.run.smithi107.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 919 kB/s | 46 kB 00:00 2024-09-21T09:32:03.150 INFO:teuthology.orchestra.run.smithi107.stdout:(26/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-09-21T09:32:03.175 INFO:teuthology.orchestra.run.smithi107.stdout:(27/34): python3-more-itertools-8.12.0-2.el9.no 853 kB/s | 79 kB 00:00 2024-09-21T09:32:03.201 INFO:teuthology.orchestra.run.smithi107.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-21T09:32:03.226 INFO:teuthology.orchestra.run.smithi107.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-21T09:32:03.243 INFO:teuthology.orchestra.run.smithi107.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 855 kB/s | 36 kB 00:00 2024-09-21T09:32:03.268 INFO:teuthology.orchestra.run.smithi107.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-21T09:32:03.285 INFO:teuthology.orchestra.run.smithi107.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-21T09:32:03.310 INFO:teuthology.orchestra.run.smithi107.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-09-21T09:32:03.336 INFO:teuthology.orchestra.run.smithi107.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-21T09:32:03.340 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:32:03.340 INFO:teuthology.orchestra.run.smithi107.stdout:Total 14 MB/s | 27 MB 00:01 2024-09-21T09:32:03.352 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/4 2024-09-21T09:32:03.432 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/4 2024-09-21T09:32:03.473 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T09:32:03.510 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T09:32:03.510 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T09:32:03.887 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T09:32:03.887 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T09:32:05.122 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T09:32:05.205 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-21T09:32:05.237 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-21T09:32:05.278 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-21T09:32:05.371 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-21T09:32:05.448 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-21T09:32:05.498 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-21T09:32:05.533 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-21T09:32:05.571 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-21T09:32:05.606 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-21T09:32:05.639 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-21T09:32:05.682 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-21T09:32:05.706 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/4 2024-09-21T09:32:05.706 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/4 2024-09-21T09:32:05.706 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/4 2024-09-21T09:32:05.706 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T09:32:05.715 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-21T09:32:05.749 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-21T09:32:05.797 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-21T09:32:05.830 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-21T09:32:05.902 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-21T09:32:06.056 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-21T09:32:06.062 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T09:32:06.062 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:06.062 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T09:32:06.062 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:32:06.062 INFO:teuthology.orchestra.run.smithi129.stdout: libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:32:06.062 INFO:teuthology.orchestra.run.smithi129.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T09:32:06.062 INFO:teuthology.orchestra.run.smithi129.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T09:32:06.063 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:06.063 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:32:06.137 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-21T09:32:06.190 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-21T09:32:06.230 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-21T09:32:06.278 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-21T09:32:06.318 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph 2024-09-21T09:32:06.333 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-21T09:32:06.518 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-21T09:32:06.568 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-21T09:32:06.663 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-21T09:32:06.750 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-21T09:32:06.783 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-21T09:32:06.896 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-21T09:32:06.903 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:37 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:32:07.045 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1649.gfbd1a6ab.el9. 29/34 2024-09-21T09:32:07.046 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:32:07.048 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:07.048 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T09:32:07.048 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:07.048 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T09:32:07.048 INFO:teuthology.orchestra.run.smithi129.stdout: ceph x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 6.5 k 2024-09-21T09:32:07.048 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-21T09:32:07.048 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 2.1 M 2024-09-21T09:32:07.049 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 1.4 M 2024-09-21T09:32:07.049 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 240 k 2024-09-21T09:32:07.049 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 3.9 M 2024-09-21T09:32:07.049 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 15 M 2024-09-21T09:32:07.049 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T09:32:07.050 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T09:32:07.050 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T09:32:07.050 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T09:32:07.050 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T09:32:07.050 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T09:32:07.050 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T09:32:07.051 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections 2024-09-21T09:32:07.051 INFO:teuthology.orchestra.run.smithi129.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-21T09:32:07.051 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T09:32:07.051 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T09:32:07.051 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T09:32:07.051 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T09:32:07.052 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T09:32:07.052 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T09:32:07.052 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T09:32:07.052 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T09:32:07.052 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T09:32:07.052 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T09:32:07.052 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T09:32:07.053 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T09:32:07.053 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T09:32:07.053 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T09:32:07.053 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T09:32:07.053 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T09:32:07.053 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T09:32:07.054 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T09:32:07.054 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T09:32:07.054 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T09:32:07.054 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:07.054 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T09:32:07.054 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:07.055 INFO:teuthology.orchestra.run.smithi129.stdout:Install 34 Packages 2024-09-21T09:32:07.055 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:07.056 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 27 M 2024-09-21T09:32:07.056 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 96 M 2024-09-21T09:32:07.056 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T09:32:07.069 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 30/34 2024-09-21T09:32:07.094 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 30/34 2024-09-21T09:32:07.094 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:32:07.095 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T09:32:07.095 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T09:32:07.095 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T09:32:07.095 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:07.617 INFO:teuthology.orchestra.run.smithi129.stdout:(1/34): ceph-17.2.7-1649.gfbd1a6ab.el9.x86_64.r 55 kB/s | 6.5 kB 00:00 2024-09-21T09:32:07.818 INFO:teuthology.orchestra.run.smithi129.stdout:(2/34): ceph-mds-17.2.7-1649.gfbd1a6ab.el9.x86_ 6.5 MB/s | 2.1 MB 00:00 2024-09-21T09:32:07.852 INFO:teuthology.orchestra.run.smithi129.stdout:(3/34): ceph-mgr-17.2.7-1649.gfbd1a6ab.el9.x86_ 4.0 MB/s | 1.4 MB 00:00 2024-09-21T09:32:07.935 INFO:teuthology.orchestra.run.smithi129.stdout:(4/34): ceph-mon-17.2.7-1649.gfbd1a6ab.el9.x86_ 12 MB/s | 3.9 MB 00:00 2024-09-21T09:32:07.960 INFO:teuthology.orchestra.run.smithi129.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1649.gfbd1 2.2 MB/s | 240 kB 00:00 2024-09-21T09:32:08.086 INFO:teuthology.orchestra.run.smithi129.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-09-21T09:32:08.128 INFO:teuthology.orchestra.run.smithi129.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-21T09:32:08.170 INFO:teuthology.orchestra.run.smithi129.stdout:(8/34): python3-cryptography-36.0.1-4.el9.x86_6 5.9 MB/s | 1.2 MB 00:00 2024-09-21T09:32:08.195 INFO:teuthology.orchestra.run.smithi129.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-09-21T09:32:08.220 INFO:teuthology.orchestra.run.smithi129.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-21T09:32:08.254 INFO:teuthology.orchestra.run.smithi129.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-21T09:32:08.404 INFO:teuthology.orchestra.run.smithi129.stdout:(12/34): ceph-osd-17.2.7-1649.gfbd1a6ab.el9.x86 26 MB/s | 15 MB 00:00 2024-09-21T09:32:08.463 INFO:teuthology.orchestra.run.smithi129.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 167 kB/s | 35 kB 00:00 2024-09-21T09:32:08.470 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 31/34 2024-09-21T09:32:08.488 INFO:teuthology.orchestra.run.smithi129.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 643 kB/s | 172 kB 00:00 2024-09-21T09:32:08.500 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 31/34 2024-09-21T09:32:08.500 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:32:08.500 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T09:32:08.500 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T09:32:08.500 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T09:32:08.500 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:08.513 INFO:teuthology.orchestra.run.smithi129.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 710 kB/s | 77 kB 00:00 2024-09-21T09:32:08.539 INFO:teuthology.orchestra.run.smithi129.stdout:(16/34): python3-pytz-2021.1-5.el9.noarch.rpm 673 kB/s | 51 kB 00:00 2024-09-21T09:32:08.564 INFO:teuthology.orchestra.run.smithi129.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 550 kB/s | 42 kB 00:00 2024-09-21T09:32:08.614 INFO:teuthology.orchestra.run.smithi129.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-21T09:32:08.645 INFO:teuthology.orchestra.run.smithi129.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 378 kB/s | 11 kB 00:00 2024-09-21T09:32:08.678 INFO:teuthology.orchestra.run.smithi129.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-21T09:32:08.704 INFO:teuthology.orchestra.run.smithi129.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 302 kB/s | 18 kB 00:00 2024-09-21T09:32:08.745 INFO:teuthology.orchestra.run.smithi129.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 559 kB/s | 23 kB 00:00 2024-09-21T09:32:08.771 INFO:teuthology.orchestra.run.smithi129.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 292 kB/s | 19 kB 00:00 2024-09-21T09:32:08.796 INFO:teuthology.orchestra.run.smithi129.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-21T09:32:08.821 INFO:teuthology.orchestra.run.smithi129.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 920 kB/s | 46 kB 00:00 2024-09-21T09:32:08.846 INFO:teuthology.orchestra.run.smithi129.stdout:(26/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-21T09:32:08.872 INFO:teuthology.orchestra.run.smithi129.stdout:(27/34): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-21T09:32:08.880 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 32/34 2024-09-21T09:32:08.897 INFO:teuthology.orchestra.run.smithi129.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-21T09:32:08.904 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 32/34 2024-09-21T09:32:08.904 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:32:08.904 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T09:32:08.905 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T09:32:08.905 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T09:32:08.905 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:08.923 INFO:teuthology.orchestra.run.smithi129.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-21T09:32:08.948 INFO:teuthology.orchestra.run.smithi129.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-21T09:32:08.981 INFO:teuthology.orchestra.run.smithi129.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-21T09:32:09.007 INFO:teuthology.orchestra.run.smithi129.stdout:(32/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-21T09:32:09.032 INFO:teuthology.orchestra.run.smithi129.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-09-21T09:32:09.057 INFO:teuthology.orchestra.run.smithi129.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-21T09:32:09.062 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:32:09.062 INFO:teuthology.orchestra.run.smithi129.stdout:Total 14 MB/s | 27 MB 00:02 2024-09-21T09:32:09.105 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 33/34 2024-09-21T09:32:09.129 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 33/34 2024-09-21T09:32:09.129 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:32:09.130 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T09:32:09.130 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T09:32:09.130 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T09:32:09.130 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:09.191 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T09:32:09.194 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 34/34 2024-09-21T09:32:09.228 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T09:32:09.228 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T09:32:09.596 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T09:32:09.597 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T09:32:10.824 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T09:32:10.902 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-21T09:32:10.943 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-21T09:32:10.977 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 34/34 2024-09-21T09:32:10.977 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/34 2024-09-21T09:32:10.977 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/34 2024-09-21T09:32:10.977 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/34 2024-09-21T09:32:10.977 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/34 2024-09-21T09:32:10.977 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 5/34 2024-09-21T09:32:10.977 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1649.gfbd1a6ab.el9. 6/34 2024-09-21T09:32:10.977 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-21T09:32:10.977 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-21T09:32:10.978 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-21T09:32:10.978 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-21T09:32:10.978 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-21T09:32:10.978 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-21T09:32:10.978 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-21T09:32:10.978 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-21T09:32:10.978 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-21T09:32:10.978 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-21T09:32:10.978 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-21T09:32:10.978 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-21T09:32:10.978 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-21T09:32:10.978 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-21T09:32:10.978 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-21T09:32:10.979 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-21T09:32:10.979 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-21T09:32:10.979 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-21T09:32:10.979 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-21T09:32:10.979 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-21T09:32:10.979 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-21T09:32:10.979 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-21T09:32:10.980 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-21T09:32:10.980 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-21T09:32:10.980 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-21T09:32:10.980 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-21T09:32:10.980 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-21T09:32:10.981 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-21T09:32:11.067 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-21T09:32:11.145 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-21T09:32:11.187 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-21T09:32:11.239 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-21T09:32:11.277 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-21T09:32:11.311 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-21T09:32:11.345 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-21T09:32:11.385 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-21T09:32:11.420 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-21T09:32:11.455 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-21T09:32:11.503 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-21T09:32:11.544 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-21T09:32:11.617 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-21T09:32:11.780 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-21T09:32:11.852 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-21T09:32:11.905 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-21T09:32:11.952 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-21T09:32:11.966 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-21T09:32:11.966 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:11.967 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T09:32:11.967 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:32:11.967 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:32:11.967 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:32:11.967 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:32:11.967 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:32:11.967 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:32:11.967 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T09:32:11.967 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T09:32:11.967 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T09:32:11.967 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T09:32:11.967 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T09:32:11.967 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T09:32:11.968 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T09:32:11.968 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T09:32:11.968 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T09:32:11.968 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T09:32:11.968 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T09:32:11.968 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T09:32:11.968 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T09:32:11.968 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T09:32:11.968 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T09:32:11.968 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T09:32:11.968 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T09:32:11.968 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T09:32:11.968 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T09:32:11.969 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T09:32:11.969 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T09:32:11.969 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T09:32:11.969 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T09:32:11.969 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T09:32:11.969 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T09:32:11.969 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T09:32:11.969 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T09:32:11.969 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T09:32:11.969 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:11.969 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:32:12.001 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-21T09:32:12.056 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-21T09:32:12.242 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-21T09:32:12.298 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-21T09:32:12.368 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-21T09:32:12.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-base 2024-09-21T09:32:12.457 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-21T09:32:12.498 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-21T09:32:12.614 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-21T09:32:12.761 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1649.gfbd1a6ab.el9. 29/34 2024-09-21T09:32:12.784 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 30/34 2024-09-21T09:32:12.810 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 30/34 2024-09-21T09:32:12.810 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:32:12.810 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T09:32:12.811 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T09:32:12.811 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T09:32:12.811 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:12.963 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:49 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:32:13.054 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:32:13.094 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:32:13.095 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-21T09:32:13.095 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:32:13.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install cephadm 2024-09-21T09:32:13.747 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:50 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:32:13.877 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:32:13.878 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:13.878 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T09:32:13.878 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:13.878 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T09:32:13.878 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 83 k 2024-09-21T09:32:13.879 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:13.879 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T09:32:13.879 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:13.879 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-21T09:32:13.879 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:13.879 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 83 k 2024-09-21T09:32:13.879 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 375 k 2024-09-21T09:32:13.879 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T09:32:14.080 INFO:teuthology.orchestra.run.smithi107.stdout:cephadm-17.2.7-1649.gfbd1a6ab.el9.noarch.rpm 412 kB/s | 83 kB 00:00 2024-09-21T09:32:14.080 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:32:14.080 INFO:teuthology.orchestra.run.smithi107.stdout:Total 409 kB/s | 83 kB 00:00 2024-09-21T09:32:14.080 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T09:32:14.086 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T09:32:14.086 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T09:32:14.088 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T09:32:14.088 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T09:32:14.157 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T09:32:14.178 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 31/34 2024-09-21T09:32:14.207 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 31/34 2024-09-21T09:32:14.208 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:32:14.208 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T09:32:14.208 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T09:32:14.208 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T09:32:14.208 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:14.587 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 32/34 2024-09-21T09:32:14.612 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 32/34 2024-09-21T09:32:14.612 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:32:14.612 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T09:32:14.612 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T09:32:14.612 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T09:32:14.612 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:14.629 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:32:14.725 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:32:14.813 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 33/34 2024-09-21T09:32:14.836 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 33/34 2024-09-21T09:32:14.837 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:32:14.837 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T09:32:14.837 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T09:32:14.837 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T09:32:14.837 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:14.910 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 34/34 2024-09-21T09:32:15.099 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:32:15.358 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:32:15.359 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:15.359 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T09:32:15.359 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:32:15.359 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:15.359 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:32:15.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T09:32:16.175 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:53 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:32:16.311 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:32:16.312 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:16.312 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-09-21T09:32:16.312 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:16.312 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T09:32:16.312 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 137 k 2024-09-21T09:32:16.312 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:16.312 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T09:32:16.312 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:16.312 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-21T09:32:16.312 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:16.313 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 137 k 2024-09-21T09:32:16.313 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 379 k 2024-09-21T09:32:16.313 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T09:32:16.522 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-immutable-object-cache-17.2.7-1649.gfbd1a6 660 kB/s | 137 kB 00:00 2024-09-21T09:32:16.522 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:32:16.522 INFO:teuthology.orchestra.run.smithi107.stdout:Total 655 kB/s | 137 kB 00:00 2024-09-21T09:32:16.523 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T09:32:16.530 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T09:32:16.530 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T09:32:16.587 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T09:32:16.587 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T09:32:16.697 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 34/34 2024-09-21T09:32:16.697 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/34 2024-09-21T09:32:16.698 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/34 2024-09-21T09:32:16.698 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/34 2024-09-21T09:32:16.698 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/34 2024-09-21T09:32:16.698 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 5/34 2024-09-21T09:32:16.698 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1649.gfbd1a6ab.el9. 6/34 2024-09-21T09:32:16.698 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-21T09:32:16.698 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-21T09:32:16.698 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-21T09:32:16.698 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-21T09:32:16.698 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-21T09:32:16.698 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-21T09:32:16.698 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-21T09:32:16.698 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-21T09:32:16.699 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-21T09:32:16.699 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-21T09:32:16.699 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-21T09:32:16.699 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-21T09:32:16.699 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-21T09:32:16.699 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-21T09:32:16.699 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-21T09:32:16.699 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-21T09:32:16.699 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-21T09:32:16.699 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-21T09:32:16.699 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-21T09:32:16.699 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-21T09:32:16.699 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-21T09:32:16.699 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-21T09:32:16.699 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-21T09:32:16.700 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-21T09:32:16.700 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-21T09:32:16.700 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-21T09:32:16.700 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-21T09:32:16.737 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T09:32:16.798 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab. 1/1 2024-09-21T09:32:16.824 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab. 1/1 2024-09-21T09:32:16.824 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:32:16.824 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T09:32:16.824 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:17.627 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-21T09:32:17.627 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:17.628 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T09:32:17.628 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:32:17.628 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:32:17.628 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:32:17.628 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-modules-core-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:32:17.628 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:32:17.628 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:32:17.628 INFO:teuthology.orchestra.run.smithi129.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T09:32:17.628 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T09:32:17.628 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T09:32:17.628 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T09:32:17.628 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T09:32:17.628 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T09:32:17.629 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T09:32:17.629 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T09:32:17.629 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T09:32:17.629 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T09:32:17.629 INFO:teuthology.orchestra.run.smithi129.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T09:32:17.629 INFO:teuthology.orchestra.run.smithi129.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T09:32:17.629 INFO:teuthology.orchestra.run.smithi129.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T09:32:17.629 INFO:teuthology.orchestra.run.smithi129.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T09:32:17.629 INFO:teuthology.orchestra.run.smithi129.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T09:32:17.629 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T09:32:17.629 INFO:teuthology.orchestra.run.smithi129.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T09:32:17.629 INFO:teuthology.orchestra.run.smithi129.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T09:32:17.629 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T09:32:17.629 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T09:32:17.629 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T09:32:17.630 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T09:32:17.630 INFO:teuthology.orchestra.run.smithi129.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T09:32:17.630 INFO:teuthology.orchestra.run.smithi129.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T09:32:17.630 INFO:teuthology.orchestra.run.smithi129.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T09:32:17.630 INFO:teuthology.orchestra.run.smithi129.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T09:32:17.630 INFO:teuthology.orchestra.run.smithi129.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T09:32:17.630 INFO:teuthology.orchestra.run.smithi129.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T09:32:17.630 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:17.630 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:32:17.746 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab. 1/1 2024-09-21T09:32:17.746 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:17.746 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T09:32:17.746 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:32:17.746 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:17.747 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:32:17.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr 2024-09-21T09:32:18.024 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-base 2024-09-21T09:32:18.575 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:55 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:32:18.597 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:49 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:32:18.669 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:32:18.686 INFO:teuthology.orchestra.run.smithi129.stdout:Package ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:32:18.710 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:32:18.711 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-21T09:32:18.711 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:32:18.729 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:32:18.729 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-21T09:32:18.730 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:32:18.779 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T09:32:18.793 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install cephadm 2024-09-21T09:32:19.351 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:56 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:32:19.370 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:50 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:32:19.481 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:32:19.482 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:19.482 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T09:32:19.482 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:19.482 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T09:32:19.482 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 3.4 M 2024-09-21T09:32:19.482 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-21T09:32:19.482 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 24 k 2024-09-21T09:32:19.482 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 15 k 2024-09-21T09:32:19.483 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-21T09:32:19.483 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T09:32:19.483 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T09:32:19.483 INFO:teuthology.orchestra.run.smithi107.stdout:Installing weak dependencies: 2024-09-21T09:32:19.483 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-21T09:32:19.483 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:19.483 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T09:32:19.483 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:19.483 INFO:teuthology.orchestra.run.smithi107.stdout:Install 7 Packages 2024-09-21T09:32:19.483 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:19.484 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 3.7 M 2024-09-21T09:32:19.484 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 80 M 2024-09-21T09:32:19.484 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T09:32:19.502 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:32:19.502 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:19.503 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T09:32:19.503 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:19.503 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T09:32:19.503 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 83 k 2024-09-21T09:32:19.503 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:19.503 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T09:32:19.503 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:19.503 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-21T09:32:19.503 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:19.504 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 83 k 2024-09-21T09:32:19.504 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 375 k 2024-09-21T09:32:19.504 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T09:32:19.706 INFO:teuthology.orchestra.run.smithi129.stdout:cephadm-17.2.7-1649.gfbd1a6ab.el9.noarch.rpm 409 kB/s | 83 kB 00:00 2024-09-21T09:32:19.707 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:32:19.707 INFO:teuthology.orchestra.run.smithi129.stdout:Total 405 kB/s | 83 kB 00:00 2024-09-21T09:32:19.707 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T09:32:19.709 INFO:teuthology.orchestra.run.smithi107.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1649.gfbd 149 kB/s | 24 kB 00:00 2024-09-21T09:32:19.712 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T09:32:19.713 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T09:32:19.714 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T09:32:19.715 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T09:32:19.734 INFO:teuthology.orchestra.run.smithi107.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1649.gfbd1 77 kB/s | 15 kB 00:00 2024-09-21T09:32:19.776 INFO:teuthology.orchestra.run.smithi107.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 136 kB/s | 9.0 kB 00:00 2024-09-21T09:32:19.795 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T09:32:19.801 INFO:teuthology.orchestra.run.smithi107.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-21T09:32:19.827 INFO:teuthology.orchestra.run.smithi107.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 443 kB/s | 41 kB 00:00 2024-09-21T09:32:19.852 INFO:teuthology.orchestra.run.smithi107.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-21T09:32:19.969 INFO:teuthology.orchestra.run.smithi107.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1649.gfbd1a6ab 7.9 MB/s | 3.4 MB 00:00 2024-09-21T09:32:19.973 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:32:19.973 INFO:teuthology.orchestra.run.smithi107.stdout:Total 7.5 MB/s | 3.7 MB 00:00 2024-09-21T09:32:19.973 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T09:32:19.981 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T09:32:19.981 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T09:32:20.080 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T09:32:20.080 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T09:32:20.264 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:32:20.361 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:32:20.384 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T09:32:20.482 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-21T09:32:20.521 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-21T09:32:20.575 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-21T09:32:20.616 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-21T09:32:20.646 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1649.gfbd1a6ab.el9.n 5/7 2024-09-21T09:32:20.686 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:32:20.960 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:32:20.960 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:20.960 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T09:32:20.960 INFO:teuthology.orchestra.run.smithi129.stdout: cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:32:20.960 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:20.960 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:32:21.180 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T09:32:21.649 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1649.gfbd1a6ab.el9. 6/7 2024-09-21T09:32:21.683 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarc 7/7 2024-09-21T09:32:21.770 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:52 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:32:21.905 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:32:21.905 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:21.906 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-09-21T09:32:21.906 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:21.906 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T09:32:21.906 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 137 k 2024-09-21T09:32:21.906 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:21.906 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T09:32:21.906 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:21.906 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-21T09:32:21.906 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:21.906 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 137 k 2024-09-21T09:32:21.907 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 379 k 2024-09-21T09:32:21.907 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T09:32:22.114 INFO:teuthology.orchestra.run.smithi129.stdout:ceph-immutable-object-cache-17.2.7-1649.gfbd1a6 662 kB/s | 137 kB 00:00 2024-09-21T09:32:22.114 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:32:22.115 INFO:teuthology.orchestra.run.smithi129.stdout:Total 658 kB/s | 137 kB 00:00 2024-09-21T09:32:22.115 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T09:32:22.122 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T09:32:22.122 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T09:32:22.180 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T09:32:22.180 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T09:32:22.341 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T09:32:22.415 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab. 1/1 2024-09-21T09:32:22.439 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab. 1/1 2024-09-21T09:32:22.439 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:32:22.439 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T09:32:22.440 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:22.684 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarc 7/7 2024-09-21T09:32:22.684 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1649.gfbd1a6ab.el9. 1/7 2024-09-21T09:32:22.684 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarc 2/7 2024-09-21T09:32:22.684 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1649.gfbd1a6ab.el9.n 3/7 2024-09-21T09:32:22.684 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-21T09:32:22.685 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-21T09:32:22.685 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-21T09:32:22.971 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-21T09:32:22.971 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:22.971 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T09:32:22.972 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:32:22.972 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:32:22.972 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:32:22.972 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-21T09:32:22.972 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-21T09:32:22.972 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T09:32:22.972 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T09:32:22.972 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:22.972 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:32:23.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T09:32:23.369 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab. 1/1 2024-09-21T09:32:23.369 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:23.369 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T09:32:23.369 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:32:23.369 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:23.370 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:32:23.609 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr 2024-09-21T09:32:23.782 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:00 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:32:23.917 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:32:23.918 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2024-09-21T09:32:23.918 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T09:32:23.918 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2024-09-21T09:32:23.919 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T09:32:23.919 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 7.4 M 2024-09-21T09:32:23.919 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-21T09:32:23.919 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T09:32:23.919 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T09:32:23.919 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T09:32:23.919 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T09:32:23.919 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T09:32:23.919 INFO:teuthology.orchestra.run.smithi107.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T09:32:23.919 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T09:32:23.919 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T09:32:23.919 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T09:32:23.920 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T09:32:23.920 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T09:32:23.920 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:23.920 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T09:32:23.920 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2024-09-21T09:32:23.920 INFO:teuthology.orchestra.run.smithi107.stdout:Install 12 Packages 2024-09-21T09:32:23.920 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:23.921 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 38 M 2024-09-21T09:32:23.921 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 212 M 2024-09-21T09:32:23.921 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T09:32:24.219 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:55 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:32:24.311 INFO:teuthology.orchestra.run.smithi129.stdout:Package ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:32:24.352 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:32:24.353 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-21T09:32:24.353 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:32:24.426 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T09:32:24.444 INFO:teuthology.orchestra.run.smithi107.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-21T09:32:24.495 INFO:teuthology.orchestra.run.smithi107.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.5 MB/s | 801 kB 00:00 2024-09-21T09:32:24.570 INFO:teuthology.orchestra.run.smithi107.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 261 kB/s | 33 kB 00:00 2024-09-21T09:32:24.612 INFO:teuthology.orchestra.run.smithi107.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 437 kB/s | 18 kB 00:00 2024-09-21T09:32:24.654 INFO:teuthology.orchestra.run.smithi107.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 928 kB/s | 38 kB 00:00 2024-09-21T09:32:24.737 INFO:teuthology.orchestra.run.smithi107.stdout:(6/12): ceph-mgr-diskprediction-local-17.2.7-16 16 MB/s | 7.4 MB 00:00 2024-09-21T09:32:24.896 INFO:teuthology.orchestra.run.smithi107.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-09-21T09:32:25.006 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:00:56 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:32:25.139 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:32:25.139 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T09:32:25.140 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:25.140 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T09:32:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 3.4 M 2024-09-21T09:32:25.140 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-21T09:32:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 24 k 2024-09-21T09:32:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 15 k 2024-09-21T09:32:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-21T09:32:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T09:32:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T09:32:25.140 INFO:teuthology.orchestra.run.smithi129.stdout:Installing weak dependencies: 2024-09-21T09:32:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-21T09:32:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:25.141 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T09:32:25.141 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:25.141 INFO:teuthology.orchestra.run.smithi129.stdout:Install 7 Packages 2024-09-21T09:32:25.141 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:25.141 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 3.7 M 2024-09-21T09:32:25.141 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 80 M 2024-09-21T09:32:25.141 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T09:32:25.247 INFO:teuthology.orchestra.run.smithi107.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.0 MB/s | 3.0 MB 00:00 2024-09-21T09:32:25.327 INFO:teuthology.orchestra.run.smithi129.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1649.gfbd 158 kB/s | 24 kB 00:00 2024-09-21T09:32:25.352 INFO:teuthology.orchestra.run.smithi129.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1649.gfbd1 81 kB/s | 15 kB 00:00 2024-09-21T09:32:25.355 INFO:teuthology.orchestra.run.smithi107.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 250 kB 00:00 2024-09-21T09:32:25.394 INFO:teuthology.orchestra.run.smithi129.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-21T09:32:25.420 INFO:teuthology.orchestra.run.smithi129.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-21T09:32:25.445 INFO:teuthology.orchestra.run.smithi129.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 443 kB/s | 41 kB 00:00 2024-09-21T09:32:25.470 INFO:teuthology.orchestra.run.smithi129.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-21T09:32:25.604 INFO:teuthology.orchestra.run.smithi129.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1649.gfbd1a6ab 7.8 MB/s | 3.4 MB 00:00 2024-09-21T09:32:25.608 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:32:25.608 INFO:teuthology.orchestra.run.smithi129.stdout:Total 7.8 MB/s | 3.7 MB 00:00 2024-09-21T09:32:25.608 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T09:32:25.616 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T09:32:25.616 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T09:32:25.712 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T09:32:25.713 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T09:32:25.839 INFO:teuthology.orchestra.run.smithi107.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.4 MB/s | 5.1 MB 00:00 2024-09-21T09:32:25.914 INFO:teuthology.orchestra.run.smithi107.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.9 MB/s | 4.9 MB 00:01 2024-09-21T09:32:26.025 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T09:32:26.089 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-21T09:32:26.123 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-21T09:32:26.169 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-21T09:32:26.210 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-21T09:32:26.239 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1649.gfbd1a6ab.el9.n 5/7 2024-09-21T09:32:27.232 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1649.gfbd1a6ab.el9. 6/7 2024-09-21T09:32:27.276 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarc 7/7 2024-09-21T09:32:28.105 INFO:teuthology.orchestra.run.smithi107.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 5.9 MB/s | 16 MB 00:02 2024-09-21T09:32:28.107 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:32:28.107 INFO:teuthology.orchestra.run.smithi107.stdout:Total 9.1 MB/s | 38 MB 00:04 2024-09-21T09:32:28.358 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarc 7/7 2024-09-21T09:32:28.358 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1649.gfbd1a6ab.el9. 1/7 2024-09-21T09:32:28.358 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarc 2/7 2024-09-21T09:32:28.358 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1649.gfbd1a6ab.el9.n 3/7 2024-09-21T09:32:28.358 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-21T09:32:28.358 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-21T09:32:28.359 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-21T09:32:28.450 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T09:32:28.476 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T09:32:28.476 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T09:32:28.676 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-21T09:32:28.676 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:28.676 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T09:32:28.677 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-grafana-dashboards-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:32:28.677 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:32:28.677 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-prometheus-alerts-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:32:28.677 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-21T09:32:28.677 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-21T09:32:28.677 INFO:teuthology.orchestra.run.smithi129.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T09:32:28.677 INFO:teuthology.orchestra.run.smithi129.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T09:32:28.677 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:28.677 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:32:28.908 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T09:32:28.986 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T09:32:28.986 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T09:32:29.486 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:00 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:32:29.619 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:32:29.620 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-09-21T09:32:29.620 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T09:32:29.620 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-09-21T09:32:29.620 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T09:32:29.620 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 7.4 M 2024-09-21T09:32:29.620 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-21T09:32:29.621 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T09:32:29.621 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T09:32:29.621 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T09:32:29.621 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T09:32:29.621 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T09:32:29.621 INFO:teuthology.orchestra.run.smithi129.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T09:32:29.621 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T09:32:29.621 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T09:32:29.621 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T09:32:29.621 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T09:32:29.621 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T09:32:29.621 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:29.621 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T09:32:29.622 INFO:teuthology.orchestra.run.smithi129.stdout:======================================================================================= 2024-09-21T09:32:29.622 INFO:teuthology.orchestra.run.smithi129.stdout:Install 12 Packages 2024-09-21T09:32:29.622 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:29.622 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 38 M 2024-09-21T09:32:29.623 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 212 M 2024-09-21T09:32:29.623 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T09:32:29.743 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T09:32:29.812 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T09:32:29.845 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T09:32:29.905 INFO:teuthology.orchestra.run.smithi129.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.6 MB/s | 191 kB 00:00 2024-09-21T09:32:29.945 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T09:32:29.980 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T09:32:29.980 INFO:teuthology.orchestra.run.smithi129.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.0 MB/s | 801 kB 00:00 2024-09-21T09:32:30.164 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T09:32:30.195 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T09:32:30.272 INFO:teuthology.orchestra.run.smithi129.stdout:(3/12): ceph-mgr-diskprediction-local-17.2.7-16 15 MB/s | 7.4 MB 00:00 2024-09-21T09:32:30.272 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T09:32:30.297 INFO:teuthology.orchestra.run.smithi129.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 83 kB/s | 33 kB 00:00 2024-09-21T09:32:30.539 INFO:teuthology.orchestra.run.smithi129.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 68 kB/s | 18 kB 00:00 2024-09-21T09:32:30.556 INFO:teuthology.orchestra.run.smithi129.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 149 kB/s | 38 kB 00:00 2024-09-21T09:32:30.907 INFO:teuthology.orchestra.run.smithi129.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.2 MB/s | 3.0 MB 00:00 2024-09-21T09:32:30.925 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T09:32:31.042 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T09:32:31.323 INFO:teuthology.orchestra.run.smithi129.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 319 kB/s | 245 kB 00:00 2024-09-21T09:32:31.407 INFO:teuthology.orchestra.run.smithi129.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 5.7 MB/s | 4.9 MB 00:00 2024-09-21T09:32:31.866 INFO:teuthology.orchestra.run.smithi129.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.3 MB/s | 5.1 MB 00:00 2024-09-21T09:32:32.291 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T09:32:32.349 INFO:teuthology.orchestra.run.smithi129.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 244 kB/s | 250 kB 00:01 2024-09-21T09:32:32.683 INFO:teuthology.orchestra.run.smithi129.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-09-21T09:32:32.684 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:32:32.684 INFO:teuthology.orchestra.run.smithi129.stdout:Total 12 MB/s | 38 MB 00:03 2024-09-21T09:32:33.045 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T09:32:33.072 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T09:32:33.072 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T09:32:33.317 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T09:32:33.346 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a 12/12 2024-09-21T09:32:33.581 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T09:32:33.582 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T09:32:34.357 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T09:32:34.448 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T09:32:34.489 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T09:32:34.597 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T09:32:34.641 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T09:32:34.825 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T09:32:34.856 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T09:32:34.942 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T09:32:35.599 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T09:32:35.626 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a 12/12 2024-09-21T09:32:35.626 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a 1/12 2024-09-21T09:32:35.626 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T09:32:35.626 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T09:32:35.627 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T09:32:35.627 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T09:32:35.627 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T09:32:35.627 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T09:32:35.627 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T09:32:35.627 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T09:32:35.627 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T09:32:35.627 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T09:32:35.711 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T09:32:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T09:32:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:36.076 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T09:32:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:32:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T09:32:36.076 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T09:32:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T09:32:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T09:32:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T09:32:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T09:32:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T09:32:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T09:32:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T09:32:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T09:32:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T09:32:36.077 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:36.077 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:32:36.460 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-rook 2024-09-21T09:32:36.962 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T09:32:37.049 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:14 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:32:37.182 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:32:37.183 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:37.184 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T09:32:37.184 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:37.184 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T09:32:37.184 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 47 k 2024-09-21T09:32:37.184 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-21T09:32:37.184 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T09:32:37.184 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T09:32:37.184 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T09:32:37.184 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T09:32:37.184 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T09:32:37.184 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T09:32:37.184 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T09:32:37.184 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T09:32:37.184 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T09:32:37.184 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T09:32:37.185 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T09:32:37.185 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T09:32:37.185 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:37.185 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T09:32:37.185 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:37.185 INFO:teuthology.orchestra.run.smithi107.stdout:Install 13 Packages 2024-09-21T09:32:37.185 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:37.185 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 2.2 M 2024-09-21T09:32:37.186 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 26 M 2024-09-21T09:32:37.186 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T09:32:37.439 INFO:teuthology.orchestra.run.smithi107.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 226 kB/s | 19 kB 00:00 2024-09-21T09:32:37.464 INFO:teuthology.orchestra.run.smithi107.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 232 kB/s | 26 kB 00:00 2024-09-21T09:32:37.523 INFO:teuthology.orchestra.run.smithi107.stdout:(3/13): ceph-mgr-rook-17.2.7-1649.gfbd1a6ab.el9 281 kB/s | 47 kB 00:00 2024-09-21T09:32:37.574 INFO:teuthology.orchestra.run.smithi107.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 222 kB 00:00 2024-09-21T09:32:37.598 INFO:teuthology.orchestra.run.smithi107.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-21T09:32:37.631 INFO:teuthology.orchestra.run.smithi107.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 929 kB/s | 54 kB 00:00 2024-09-21T09:32:37.674 INFO:teuthology.orchestra.run.smithi107.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 428 kB/s | 32 kB 00:00 2024-09-21T09:32:37.699 INFO:teuthology.orchestra.run.smithi107.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 279 kB 00:00 2024-09-21T09:32:37.725 INFO:teuthology.orchestra.run.smithi107.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 152 kB/s | 14 kB 00:00 2024-09-21T09:32:37.758 INFO:teuthology.orchestra.run.smithi107.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.2 MB/s | 59 kB 00:00 2024-09-21T09:32:37.784 INFO:teuthology.orchestra.run.smithi107.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.0 MB/s | 220 kB 00:00 2024-09-21T09:32:37.829 INFO:teuthology.orchestra.run.smithi107.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-09-21T09:32:37.876 INFO:teuthology.orchestra.run.smithi107.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.8 MB/s | 1.0 MB 00:00 2024-09-21T09:32:37.879 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:32:37.879 INFO:teuthology.orchestra.run.smithi107.stdout:Total 3.2 MB/s | 2.2 MB 00:00 2024-09-21T09:32:37.944 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T09:32:37.959 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T09:32:37.960 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T09:32:37.996 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T09:32:38.040 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a 12/12 2024-09-21T09:32:38.050 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T09:32:38.051 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T09:32:38.556 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T09:32:38.717 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T09:32:38.777 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T09:32:38.833 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T09:32:38.870 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T09:32:38.912 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T09:32:39.014 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T09:32:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T09:32:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T09:32:39.903 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T09:32:39.953 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T09:32:39.986 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T09:32:40.032 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T09:32:40.056 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 13/13 2024-09-21T09:32:40.547 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a 12/12 2024-09-21T09:32:40.547 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a 1/12 2024-09-21T09:32:40.548 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T09:32:40.548 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T09:32:40.548 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T09:32:40.548 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T09:32:40.548 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T09:32:40.548 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T09:32:40.548 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T09:32:40.548 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T09:32:40.548 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T09:32:40.548 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T09:32:40.800 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 13/13 2024-09-21T09:32:40.800 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/13 2024-09-21T09:32:40.801 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T09:32:40.801 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T09:32:40.801 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T09:32:40.801 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T09:32:40.801 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T09:32:40.801 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T09:32:40.801 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T09:32:40.801 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T09:32:40.801 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T09:32:40.801 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T09:32:40.801 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T09:32:40.961 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T09:32:40.961 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:40.961 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T09:32:40.961 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:32:40.961 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T09:32:40.961 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T09:32:40.962 INFO:teuthology.orchestra.run.smithi129.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T09:32:40.962 INFO:teuthology.orchestra.run.smithi129.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T09:32:40.962 INFO:teuthology.orchestra.run.smithi129.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T09:32:40.962 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T09:32:40.962 INFO:teuthology.orchestra.run.smithi129.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T09:32:40.962 INFO:teuthology.orchestra.run.smithi129.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T09:32:40.962 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T09:32:40.962 INFO:teuthology.orchestra.run.smithi129.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T09:32:40.963 INFO:teuthology.orchestra.run.smithi129.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T09:32:40.963 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:40.963 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:32:41.132 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T09:32:41.132 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:41.132 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T09:32:41.132 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:32:41.132 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T09:32:41.132 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T09:32:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T09:32:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T09:32:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T09:32:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T09:32:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T09:32:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T09:32:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T09:32:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T09:32:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T09:32:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T09:32:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:41.133 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:32:41.418 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T09:32:41.419 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-rook 2024-09-21T09:32:42.002 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:19 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:32:42.010 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:13 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:32:42.137 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:32:42.137 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:42.138 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T09:32:42.138 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:42.138 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T09:32:42.138 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 124 k 2024-09-21T09:32:42.138 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-21T09:32:42.138 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T09:32:42.138 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T09:32:42.138 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T09:32:42.138 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T09:32:42.138 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T09:32:42.138 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:42.138 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T09:32:42.139 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:42.139 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2024-09-21T09:32:42.139 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:42.139 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 7.0 M 2024-09-21T09:32:42.139 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 33 M 2024-09-21T09:32:42.139 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T09:32:42.143 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:32:42.145 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:42.145 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T09:32:42.145 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:42.145 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T09:32:42.145 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 47 k 2024-09-21T09:32:42.145 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-21T09:32:42.145 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T09:32:42.145 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T09:32:42.145 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T09:32:42.146 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T09:32:42.146 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T09:32:42.146 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T09:32:42.146 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T09:32:42.146 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T09:32:42.146 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T09:32:42.146 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T09:32:42.146 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T09:32:42.146 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T09:32:42.146 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:42.146 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T09:32:42.147 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:42.147 INFO:teuthology.orchestra.run.smithi129.stdout:Install 13 Packages 2024-09-21T09:32:42.147 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:42.148 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 2.2 M 2024-09-21T09:32:42.148 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 26 M 2024-09-21T09:32:42.148 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T09:32:42.462 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-09-21T09:32:42.487 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1649.gfbd1a6ab.e 606 kB/s | 124 kB 00:00 2024-09-21T09:32:42.538 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-21T09:32:42.563 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 767 kB/s | 58 kB 00:00 2024-09-21T09:32:42.577 INFO:teuthology.orchestra.run.smithi129.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 202 kB/s | 19 kB 00:00 2024-09-21T09:32:42.588 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-21T09:32:42.602 INFO:teuthology.orchestra.run.smithi129.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 213 kB/s | 26 kB 00:00 2024-09-21T09:32:42.661 INFO:teuthology.orchestra.run.smithi129.stdout:(3/13): ceph-mgr-rook-17.2.7-1649.gfbd1a6ab.el9 266 kB/s | 47 kB 00:00 2024-09-21T09:32:42.686 INFO:teuthology.orchestra.run.smithi129.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 159 kB 00:00 2024-09-21T09:32:42.711 INFO:teuthology.orchestra.run.smithi129.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 222 kB 00:00 2024-09-21T09:32:42.737 INFO:teuthology.orchestra.run.smithi129.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-21T09:32:42.779 INFO:teuthology.orchestra.run.smithi129.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-21T09:32:42.804 INFO:teuthology.orchestra.run.smithi129.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 211 kB/s | 14 kB 00:00 2024-09-21T09:32:42.829 INFO:teuthology.orchestra.run.smithi129.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.6 MB/s | 279 kB 00:00 2024-09-21T09:32:42.863 INFO:teuthology.orchestra.run.smithi129.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-21T09:32:42.889 INFO:teuthology.orchestra.run.smithi129.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-21T09:32:42.913 INFO:teuthology.orchestra.run.smithi129.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 707 kB/s | 59 kB 00:00 2024-09-21T09:32:42.947 INFO:teuthology.orchestra.run.smithi129.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-21T09:32:42.950 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:32:42.950 INFO:teuthology.orchestra.run.smithi129.stdout:Total 2.8 MB/s | 2.2 MB 00:00 2024-09-21T09:32:43.014 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T09:32:43.030 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T09:32:43.030 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T09:32:43.118 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T09:32:43.120 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T09:32:43.625 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T09:32:43.806 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T09:32:43.857 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T09:32:43.921 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T09:32:43.966 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T09:32:44.025 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T09:32:44.127 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T09:32:44.221 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.1 MB/s | 6.0 MB 00:01 2024-09-21T09:32:44.224 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:32:44.224 INFO:teuthology.orchestra.run.smithi107.stdout:Total 3.4 MB/s | 7.0 MB 00:02 2024-09-21T09:32:44.237 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T09:32:44.289 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T09:32:44.295 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T09:32:44.306 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T09:32:44.306 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T09:32:44.405 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T09:32:44.406 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T09:32:44.722 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T09:32:44.887 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T09:32:44.931 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T09:32:45.021 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T09:32:45.066 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T09:32:45.108 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T09:32:45.153 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T09:32:45.177 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 13/13 2024-09-21T09:32:45.450 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T09:32:45.543 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T09:32:45.598 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T09:32:45.636 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 6/6 2024-09-21T09:32:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 13/13 2024-09-21T09:32:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/13 2024-09-21T09:32:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T09:32:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T09:32:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T09:32:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T09:32:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T09:32:46.024 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T09:32:46.025 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T09:32:46.025 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T09:32:46.025 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T09:32:46.025 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T09:32:46.025 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T09:32:46.385 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T09:32:46.385 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:46.385 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T09:32:46.385 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:32:46.385 INFO:teuthology.orchestra.run.smithi129.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T09:32:46.385 INFO:teuthology.orchestra.run.smithi129.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T09:32:46.385 INFO:teuthology.orchestra.run.smithi129.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T09:32:46.385 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T09:32:46.385 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T09:32:46.386 INFO:teuthology.orchestra.run.smithi129.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T09:32:46.386 INFO:teuthology.orchestra.run.smithi129.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T09:32:46.386 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T09:32:46.386 INFO:teuthology.orchestra.run.smithi129.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T09:32:46.386 INFO:teuthology.orchestra.run.smithi129.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T09:32:46.386 INFO:teuthology.orchestra.run.smithi129.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T09:32:46.386 INFO:teuthology.orchestra.run.smithi129.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T09:32:46.386 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:46.386 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:32:46.434 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 6/6 2024-09-21T09:32:46.434 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/6 2024-09-21T09:32:46.434 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T09:32:46.434 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T09:32:46.434 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T09:32:46.434 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T09:32:46.682 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T09:32:46.698 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T09:32:46.698 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:46.699 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T09:32:46.699 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:32:46.699 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T09:32:46.699 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T09:32:46.699 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T09:32:46.699 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T09:32:46.699 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T09:32:46.699 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:46.699 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:32:46.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-fuse 2024-09-21T09:32:47.266 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:18 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:32:47.399 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:32:47.400 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:47.400 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T09:32:47.400 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:47.400 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T09:32:47.400 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 124 k 2024-09-21T09:32:47.400 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-21T09:32:47.400 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T09:32:47.400 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T09:32:47.400 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T09:32:47.400 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T09:32:47.400 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T09:32:47.401 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:47.401 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T09:32:47.401 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:47.401 INFO:teuthology.orchestra.run.smithi129.stdout:Install 6 Packages 2024-09-21T09:32:47.401 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:47.401 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 7.0 M 2024-09-21T09:32:47.401 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 33 M 2024-09-21T09:32:47.401 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T09:32:47.555 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:24 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:32:47.688 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:32:47.688 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:47.688 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T09:32:47.689 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:47.689 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T09:32:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 813 k 2024-09-21T09:32:47.689 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-21T09:32:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T09:32:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:47.689 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T09:32:47.689 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:47.689 INFO:teuthology.orchestra.run.smithi107.stdout:Install 2 Packages 2024-09-21T09:32:47.689 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:47.690 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 893 k 2024-09-21T09:32:47.690 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 2.6 M 2024-09-21T09:32:47.690 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T09:32:47.706 INFO:teuthology.orchestra.run.smithi129.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1649.gfbd1a6ab.e 557 kB/s | 124 kB 00:00 2024-09-21T09:32:47.768 INFO:teuthology.orchestra.run.smithi129.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.8 MB/s | 548 kB 00:00 2024-09-21T09:32:47.793 INFO:teuthology.orchestra.run.smithi129.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-09-21T09:32:47.818 INFO:teuthology.orchestra.run.smithi129.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-21T09:32:47.990 INFO:teuthology.orchestra.run.smithi107.stdout:(1/2): ceph-fuse-17.2.7-1649.gfbd1a6ab.el9.x86_ 3.1 MB/s | 813 kB 00:00 2024-09-21T09:32:48.069 INFO:teuthology.orchestra.run.smithi129.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 424 kB/s | 248 kB 00:00 2024-09-21T09:32:48.124 INFO:teuthology.orchestra.run.smithi107.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 206 kB/s | 80 kB 00:00 2024-09-21T09:32:48.125 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:32:48.125 INFO:teuthology.orchestra.run.smithi107.stdout:Total 2.0 MB/s | 893 kB 00:00 2024-09-21T09:32:48.136 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T09:32:48.158 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T09:32:48.158 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T09:32:48.239 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T09:32:48.239 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T09:32:48.448 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T09:32:48.578 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T09:32:48.664 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/2 2024-09-21T09:32:49.292 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/2 2024-09-21T09:32:49.292 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/2 2024-09-21T09:32:49.542 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T09:32:49.542 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:49.542 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T09:32:49.543 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T09:32:49.543 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:49.543 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:32:49.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados-devel 2024-09-21T09:32:50.438 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:27 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:32:50.576 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:32:50.577 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:50.577 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-09-21T09:32:50.577 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:50.577 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T09:32:50.577 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 126 k 2024-09-21T09:32:50.577 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:50.577 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T09:32:50.577 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:50.577 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-21T09:32:50.577 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:50.578 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 126 k 2024-09-21T09:32:50.578 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 448 k 2024-09-21T09:32:50.578 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T09:32:50.777 INFO:teuthology.orchestra.run.smithi129.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 1.8 MB/s | 6.0 MB 00:03 2024-09-21T09:32:50.779 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:32:50.780 INFO:teuthology.orchestra.run.smithi107.stdout:librados-devel-17.2.7-1649.gfbd1a6ab.el9.x86_64 625 kB/s | 126 kB 00:00 2024-09-21T09:32:50.780 INFO:teuthology.orchestra.run.smithi129.stdout:Total 2.1 MB/s | 7.0 MB 00:03 2024-09-21T09:32:50.781 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:32:50.781 INFO:teuthology.orchestra.run.smithi107.stdout:Total 620 kB/s | 126 kB 00:00 2024-09-21T09:32:50.781 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T09:32:50.786 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T09:32:50.786 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T09:32:50.849 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T09:32:50.850 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T09:32:50.855 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T09:32:50.867 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T09:32:50.867 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T09:32:50.967 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T09:32:50.968 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T09:32:50.996 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T09:32:51.102 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:32:51.328 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T09:32:51.507 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T09:32:51.534 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:32:51.562 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T09:32:51.754 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:32:51.754 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:51.755 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T09:32:51.755 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:32:51.755 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:51.755 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:32:52.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs2 2024-09-21T09:32:52.080 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T09:32:52.165 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T09:32:52.221 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T09:32:52.250 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 6/6 2024-09-21T09:32:52.667 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:29 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:32:52.758 INFO:teuthology.orchestra.run.smithi107.stdout:Package libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:32:52.801 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:32:52.803 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-21T09:32:52.803 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:32:52.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs-devel 2024-09-21T09:32:52.976 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 6/6 2024-09-21T09:32:52.976 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/6 2024-09-21T09:32:52.976 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T09:32:52.976 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T09:32:52.976 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T09:32:52.976 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T09:32:53.247 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T09:32:53.248 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:53.248 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T09:32:53.248 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:32:53.248 INFO:teuthology.orchestra.run.smithi129.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T09:32:53.248 INFO:teuthology.orchestra.run.smithi129.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T09:32:53.248 INFO:teuthology.orchestra.run.smithi129.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T09:32:53.248 INFO:teuthology.orchestra.run.smithi129.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T09:32:53.248 INFO:teuthology.orchestra.run.smithi129.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T09:32:53.248 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:53.248 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:32:53.462 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:30 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:32:53.510 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install ceph-fuse 2024-09-21T09:32:53.592 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:32:53.592 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:53.592 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-09-21T09:32:53.592 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:53.592 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T09:32:53.592 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 25 k 2024-09-21T09:32:53.593 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:53.593 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T09:32:53.593 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:32:53.593 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-21T09:32:53.593 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:53.593 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 25 k 2024-09-21T09:32:53.594 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 110 k 2024-09-21T09:32:53.594 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T09:32:53.750 INFO:teuthology.orchestra.run.smithi107.stdout:libcephfs-devel-17.2.7-1649.gfbd1a6ab.el9.x86_6 159 kB/s | 25 kB 00:00 2024-09-21T09:32:53.751 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:32:53.751 INFO:teuthology.orchestra.run.smithi107.stdout:Total 158 kB/s | 25 kB 00:00 2024-09-21T09:32:53.751 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T09:32:53.754 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T09:32:53.755 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T09:32:53.756 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T09:32:53.756 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T09:32:53.842 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T09:32:53.954 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:32:54.099 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:25 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:32:54.231 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:32:54.232 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:54.232 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repository Size 2024-09-21T09:32:54.232 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:54.232 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T09:32:54.232 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 813 k 2024-09-21T09:32:54.232 INFO:teuthology.orchestra.run.smithi129.stdout:Installing dependencies: 2024-09-21T09:32:54.232 INFO:teuthology.orchestra.run.smithi129.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T09:32:54.232 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:54.232 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T09:32:54.232 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:54.233 INFO:teuthology.orchestra.run.smithi129.stdout:Install 2 Packages 2024-09-21T09:32:54.233 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:54.233 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 893 k 2024-09-21T09:32:54.233 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 2.6 M 2024-09-21T09:32:54.233 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T09:32:54.336 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:32:54.521 INFO:teuthology.orchestra.run.smithi129.stdout:(1/2): ceph-fuse-17.2.7-1649.gfbd1a6ab.el9.x86_ 3.0 MB/s | 813 kB 00:00 2024-09-21T09:32:54.573 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:32:54.573 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:54.573 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T09:32:54.573 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:32:54.573 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:32:54.573 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:32:54.713 INFO:teuthology.orchestra.run.smithi129.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 174 kB/s | 80 kB 00:00 2024-09-21T09:32:54.714 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:32:54.714 INFO:teuthology.orchestra.run.smithi129.stdout:Total 1.8 MB/s | 893 kB 00:00 2024-09-21T09:32:54.726 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T09:32:54.748 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T09:32:54.748 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T09:32:54.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados2 2024-09-21T09:32:54.829 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T09:32:54.829 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T09:32:55.023 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T09:32:55.154 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T09:32:55.229 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/2 2024-09-21T09:32:55.370 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:32 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:32:55.459 INFO:teuthology.orchestra.run.smithi107.stdout:Package librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:32:55.502 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:32:55.503 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-21T09:32:55.503 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:32:55.586 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librbd1 2024-09-21T09:32:55.901 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/2 2024-09-21T09:32:55.901 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/2 2024-09-21T09:32:56.163 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:33 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:32:56.181 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T09:32:56.182 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:56.182 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T09:32:56.182 INFO:teuthology.orchestra.run.smithi129.stdout: ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T09:32:56.182 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:56.182 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:32:56.253 INFO:teuthology.orchestra.run.smithi107.stdout:Package librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:32:56.296 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:32:56.297 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-21T09:32:56.297 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:32:56.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rados 2024-09-21T09:32:56.494 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librados-devel 2024-09-21T09:32:56.958 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:33 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:32:57.046 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:32:57.089 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:32:57.091 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-21T09:32:57.091 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:32:57.113 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:28 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:32:57.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rgw 2024-09-21T09:32:57.249 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:32:57.249 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:57.250 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-09-21T09:32:57.250 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:57.250 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T09:32:57.250 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 126 k 2024-09-21T09:32:57.250 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:57.250 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T09:32:57.250 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:32:57.250 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-21T09:32:57.250 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:57.251 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 126 k 2024-09-21T09:32:57.251 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 448 k 2024-09-21T09:32:57.251 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T09:32:57.461 INFO:teuthology.orchestra.run.smithi129.stdout:librados-devel-17.2.7-1649.gfbd1a6ab.el9.x86_64 601 kB/s | 126 kB 00:00 2024-09-21T09:32:57.461 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:32:57.462 INFO:teuthology.orchestra.run.smithi129.stdout:Total 597 kB/s | 126 kB 00:00 2024-09-21T09:32:57.462 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T09:32:57.467 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T09:32:57.467 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T09:32:57.532 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T09:32:57.533 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T09:32:57.681 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T09:32:57.747 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:34 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:32:57.767 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:32:57.834 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:32:57.877 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:32:57.878 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-21T09:32:57.878 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:32:57.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-cephfs 2024-09-21T09:32:58.242 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:32:58.487 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:32:58.487 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:58.488 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T09:32:58.488 INFO:teuthology.orchestra.run.smithi129.stdout: librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:32:58.488 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:32:58.488 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:32:58.542 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:35 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:32:58.632 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:32:58.674 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:32:58.675 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-21T09:32:58.675 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:32:58.713 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install libcephfs2 2024-09-21T09:32:58.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rbd 2024-09-21T09:32:59.301 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:30 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:32:59.354 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:36 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:32:59.392 INFO:teuthology.orchestra.run.smithi129.stdout:Package libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:32:59.437 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:32:59.438 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-21T09:32:59.438 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:32:59.444 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:32:59.487 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:32:59.488 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-21T09:32:59.488 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:32:59.509 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install libcephfs-devel 2024-09-21T09:32:59.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-fuse 2024-09-21T09:33:00.089 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:31 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:33:00.153 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:37 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:33:00.220 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:33:00.221 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:33:00.221 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-09-21T09:33:00.221 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:33:00.221 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T09:33:00.221 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 25 k 2024-09-21T09:33:00.221 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:33:00.221 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T09:33:00.221 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:33:00.221 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-21T09:33:00.222 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:33:00.222 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 25 k 2024-09-21T09:33:00.222 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 110 k 2024-09-21T09:33:00.222 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T09:33:00.287 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:33:00.287 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:33:00.287 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-09-21T09:33:00.288 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:33:00.288 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T09:33:00.288 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 82 k 2024-09-21T09:33:00.288 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:33:00.288 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T09:33:00.288 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:33:00.288 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-21T09:33:00.288 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:33:00.289 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 82 k 2024-09-21T09:33:00.289 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 222 k 2024-09-21T09:33:00.289 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T09:33:00.374 INFO:teuthology.orchestra.run.smithi129.stdout:libcephfs-devel-17.2.7-1649.gfbd1a6ab.el9.x86_6 164 kB/s | 25 kB 00:00 2024-09-21T09:33:00.375 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:33:00.375 INFO:teuthology.orchestra.run.smithi129.stdout:Total 162 kB/s | 25 kB 00:00 2024-09-21T09:33:00.375 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T09:33:00.378 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T09:33:00.378 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T09:33:00.380 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T09:33:00.380 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T09:33:00.467 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T09:33:00.478 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-fuse-17.2.7-1649.gfbd1a6ab.el9.x86_64.rpm 432 kB/s | 82 kB 00:00 2024-09-21T09:33:00.479 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:33:00.479 INFO:teuthology.orchestra.run.smithi107.stdout:Total 428 kB/s | 82 kB 00:00 2024-09-21T09:33:00.479 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T09:33:00.486 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T09:33:00.486 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T09:33:00.549 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T09:33:00.550 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T09:33:00.597 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:00.689 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T09:33:00.783 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:01.023 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:01.143 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:01.270 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:01.270 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:33:01.270 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T09:33:01.270 INFO:teuthology.orchestra.run.smithi129.stdout: libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:33:01.271 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:33:01.271 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:33:01.385 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:01.385 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:33:01.385 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T09:33:01.385 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:33:01.386 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:33:01.386 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:33:01.492 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librados2 2024-09-21T09:33:01.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-mirror 2024-09-21T09:33:02.072 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:33 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:33:02.162 INFO:teuthology.orchestra.run.smithi129.stdout:Package librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:33:02.205 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:33:02.206 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-21T09:33:02.206 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:33:02.248 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:39 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:33:02.279 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install librbd1 2024-09-21T09:33:02.382 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:33:02.383 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:33:02.383 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-09-21T09:33:02.383 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:33:02.383 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T09:33:02.383 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 2.9 M 2024-09-21T09:33:02.383 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:33:02.383 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T09:33:02.383 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:33:02.384 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-21T09:33:02.384 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:33:02.384 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 2.9 M 2024-09-21T09:33:02.384 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 12 M 2024-09-21T09:33:02.384 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T09:33:02.707 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-mirror-17.2.7-1649.gfbd1a6ab.el9.x86_64.rpm 9.1 MB/s | 2.9 MB 00:00 2024-09-21T09:33:02.708 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:33:02.708 INFO:teuthology.orchestra.run.smithi107.stdout:Total 9.0 MB/s | 2.9 MB 00:00 2024-09-21T09:33:02.709 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T09:33:02.717 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T09:33:02.717 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T09:33:02.797 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T09:33:02.797 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T09:33:02.860 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:33 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:33:02.948 INFO:teuthology.orchestra.run.smithi129.stdout:Package librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:33:02.991 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:33:02.992 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-21T09:33:02.992 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:33:03.066 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rados 2024-09-21T09:33:03.215 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T09:33:03.275 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:03.299 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:03.299 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:33:03.300 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T09:33:03.300 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T09:33:03.300 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T09:33:03.300 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:33:03.647 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:34 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:33:03.736 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:33:03.779 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:33:03.780 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-21T09:33:03.780 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:33:03.859 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rgw 2024-09-21T09:33:04.343 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:04.343 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:33:04.343 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T09:33:04.343 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:33:04.343 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:33:04.343 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:33:04.434 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:35 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:33:04.523 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:33:04.566 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:33:04.567 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-21T09:33:04.567 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:33:04.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-nbd 2024-09-21T09:33:04.646 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-cephfs 2024-09-21T09:33:05.195 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:42 ago on Sat 21 Sep 2024 09:31:23 AM UTC. 2024-09-21T09:33:05.220 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:36 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:33:05.309 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:33:05.327 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T09:33:05.328 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:33:05.328 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-09-21T09:33:05.328 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:33:05.328 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T09:33:05.328 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 165 k 2024-09-21T09:33:05.328 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:33:05.328 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T09:33:05.329 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T09:33:05.329 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-21T09:33:05.329 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:33:05.329 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 165 k 2024-09-21T09:33:05.329 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 477 k 2024-09-21T09:33:05.329 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T09:33:05.353 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:33:05.354 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-21T09:33:05.354 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:33:05.428 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install python3-rbd 2024-09-21T09:33:05.545 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-nbd-17.2.7-1649.gfbd1a6ab.el9.x86_64.rpm 764 kB/s | 165 kB 00:00 2024-09-21T09:33:05.546 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:33:05.546 INFO:teuthology.orchestra.run.smithi107.stdout:Total 760 kB/s | 165 kB 00:00 2024-09-21T09:33:05.546 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T09:33:05.553 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T09:33:05.554 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T09:33:05.617 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T09:33:05.617 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T09:33:05.772 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T09:33:05.859 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:06.004 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:37 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:33:06.094 INFO:teuthology.orchestra.run.smithi129.stdout:Package python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:33:06.137 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:33:06.138 INFO:teuthology.orchestra.run.smithi129.stdout:Nothing to do. 2024-09-21T09:33:06.138 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:33:06.212 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-fuse 2024-09-21T09:33:06.310 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:06.577 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:06.577 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:33:06.577 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T09:33:06.577 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:33:06.578 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:33:06.578 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T09:33:06.792 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:37 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:33:06.857 DEBUG:teuthology.parallel:result is None 2024-09-21T09:33:06.924 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:33:06.925 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:33:06.925 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-09-21T09:33:06.925 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:33:06.925 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T09:33:06.925 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 82 k 2024-09-21T09:33:06.925 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:33:06.926 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T09:33:06.926 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:33:06.926 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-21T09:33:06.926 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:33:06.926 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 82 k 2024-09-21T09:33:06.926 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 222 k 2024-09-21T09:33:06.926 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T09:33:07.128 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-fuse-17.2.7-1649.gfbd1a6ab.el9.x86_64.rpm 405 kB/s | 82 kB 00:00 2024-09-21T09:33:07.128 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:33:07.129 INFO:teuthology.orchestra.run.smithi129.stdout:Total 403 kB/s | 82 kB 00:00 2024-09-21T09:33:07.129 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T09:33:07.135 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T09:33:07.135 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T09:33:07.198 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T09:33:07.198 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T09:33:07.339 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T09:33:07.424 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:07.776 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:08.038 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:08.038 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:33:08.038 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T09:33:08.038 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:33:08.038 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:33:08.038 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:33:08.254 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-mirror 2024-09-21T09:33:08.852 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:39 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:33:08.989 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:33:08.989 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:33:08.989 INFO:teuthology.orchestra.run.smithi129.stdout: Package Arch Version Repo Size 2024-09-21T09:33:08.989 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:33:08.990 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T09:33:08.990 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 2.9 M 2024-09-21T09:33:08.990 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:33:08.990 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T09:33:08.990 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:33:08.990 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-21T09:33:08.990 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:33:08.990 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 2.9 M 2024-09-21T09:33:08.990 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 12 M 2024-09-21T09:33:08.990 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T09:33:09.323 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-mirror-17.2.7-1649.gfbd1a6ab.el9.x86_64.rpm 8.8 MB/s | 2.9 MB 00:00 2024-09-21T09:33:09.324 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:33:09.324 INFO:teuthology.orchestra.run.smithi129.stdout:Total 8.8 MB/s | 2.9 MB 00:00 2024-09-21T09:33:09.324 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T09:33:09.333 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T09:33:09.333 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T09:33:09.412 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T09:33:09.412 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T09:33:09.831 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T09:33:09.899 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:09.924 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:09.924 INFO:teuthology.orchestra.run.smithi129.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:33:09.925 INFO:teuthology.orchestra.run.smithi129.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T09:33:09.925 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T09:33:09.925 INFO:teuthology.orchestra.run.smithi129.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T09:33:09.925 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:33:11.054 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:11.054 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:33:11.054 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T09:33:11.054 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:33:11.054 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:33:11.054 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:33:11.304 DEBUG:teuthology.orchestra.run.smithi129:> sudo yum -y install rbd-nbd 2024-09-21T09:33:11.909 INFO:teuthology.orchestra.run.smithi129.stdout:Last metadata expiration check: 0:01:42 ago on Sat 21 Sep 2024 09:31:29 AM UTC. 2024-09-21T09:33:12.043 INFO:teuthology.orchestra.run.smithi129.stdout:Dependencies resolved. 2024-09-21T09:33:12.043 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:33:12.043 INFO:teuthology.orchestra.run.smithi129.stdout: Package Architecture Version Repository Size 2024-09-21T09:33:12.043 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:33:12.043 INFO:teuthology.orchestra.run.smithi129.stdout:Installing: 2024-09-21T09:33:12.043 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 165 k 2024-09-21T09:33:12.044 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:33:12.044 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction Summary 2024-09-21T09:33:12.044 INFO:teuthology.orchestra.run.smithi129.stdout:================================================================================ 2024-09-21T09:33:12.044 INFO:teuthology.orchestra.run.smithi129.stdout:Install 1 Package 2024-09-21T09:33:12.044 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:33:12.044 INFO:teuthology.orchestra.run.smithi129.stdout:Total download size: 165 k 2024-09-21T09:33:12.044 INFO:teuthology.orchestra.run.smithi129.stdout:Installed size: 477 k 2024-09-21T09:33:12.044 INFO:teuthology.orchestra.run.smithi129.stdout:Downloading Packages: 2024-09-21T09:33:12.261 INFO:teuthology.orchestra.run.smithi129.stdout:rbd-nbd-17.2.7-1649.gfbd1a6ab.el9.x86_64.rpm 762 kB/s | 165 kB 00:00 2024-09-21T09:33:12.262 INFO:teuthology.orchestra.run.smithi129.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:33:12.262 INFO:teuthology.orchestra.run.smithi129.stdout:Total 756 kB/s | 165 kB 00:00 2024-09-21T09:33:12.262 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction check 2024-09-21T09:33:12.270 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction check succeeded. 2024-09-21T09:33:12.270 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction test 2024-09-21T09:33:12.334 INFO:teuthology.orchestra.run.smithi129.stdout:Transaction test succeeded. 2024-09-21T09:33:12.334 INFO:teuthology.orchestra.run.smithi129.stdout:Running transaction 2024-09-21T09:33:12.498 INFO:teuthology.orchestra.run.smithi129.stdout: Preparing : 1/1 2024-09-21T09:33:12.592 INFO:teuthology.orchestra.run.smithi129.stdout: Installing : rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:13.041 INFO:teuthology.orchestra.run.smithi129.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:13.304 INFO:teuthology.orchestra.run.smithi129.stdout: Verifying : rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:33:13.304 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:33:13.304 INFO:teuthology.orchestra.run.smithi129.stdout:Installed: 2024-09-21T09:33:13.304 INFO:teuthology.orchestra.run.smithi129.stdout: rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:33:13.304 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:33:13.304 INFO:teuthology.orchestra.run.smithi129.stdout:Complete! 2024-09-21T09:33:13.586 DEBUG:teuthology.parallel:result is None 2024-09-21T09:33:13.586 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:33:13.586 INFO:teuthology.packaging:ref: None 2024-09-21T09:33:13.586 INFO:teuthology.packaging:tag: None 2024-09-21T09:33:13.586 INFO:teuthology.packaging:branch: quincy 2024-09-21T09:33:13.586 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:33:13.586 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-21T09:33:13.815 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T09:33:13.843 INFO:teuthology.orchestra.run.smithi107.stdout:17.2.7-1649.gfbd1a6ab.el9 2024-09-21T09:33:13.843 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1649.gfbd1a6ab.el9 2024-09-21T09:33:13.843 INFO:teuthology.task.install:The correct ceph version 17.2.7-1649.gfbd1a6ab is installed. 2024-09-21T09:33:13.845 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:33:13.845 INFO:teuthology.packaging:ref: None 2024-09-21T09:33:13.845 INFO:teuthology.packaging:tag: None 2024-09-21T09:33:13.845 INFO:teuthology.packaging:branch: quincy 2024-09-21T09:33:13.845 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:33:13.845 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-21T09:33:14.068 DEBUG:teuthology.orchestra.run.smithi129:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T09:33:14.091 INFO:teuthology.orchestra.run.smithi129.stdout:17.2.7-1649.gfbd1a6ab.el9 2024-09-21T09:33:14.091 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1649.gfbd1a6ab.el9 2024-09-21T09:33:14.091 INFO:teuthology.task.install:The correct ceph version 17.2.7-1649.gfbd1a6ab is installed. 2024-09-21T09:33:14.093 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T09:33:14.093 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T09:33:14.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T09:33:14.125 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T09:33:14.125 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T09:33:14.156 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T09:33:14.156 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T09:33:14.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T09:33:14.195 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T09:33:14.263 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T09:33:14.263 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T09:33:14.303 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T09:33:14.368 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T09:33:14.368 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T09:33:14.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T09:33:14.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T09:33:14.462 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T09:33:14.462 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T09:33:14.486 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T09:33:14.552 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T09:33:14.552 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T09:33:14.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T09:33:14.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T09:33:14.645 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T09:33:14.645 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T09:33:14.670 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T09:33:14.734 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-21T09:33:14.826 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'OSD_DOWN', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSDMAP_FLAGS', 'PG_AVAILABILITY'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'} 2024-09-21T09:33:14.826 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-09-21T09:33:14.826 INFO:tasks.cephadm:Cluster fsid is 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:33:14.827 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-21T09:33:14.827 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.107', 'mon.c': '[v2:172.21.15.107:3301,v1:172.21.15.107:6790]', 'mon.b': '172.21.15.129'} 2024-09-21T09:33:14.827 INFO:tasks.cephadm:First mon is mon.a on smithi107 2024-09-21T09:33:14.827 INFO:tasks.cephadm:First mgr is y 2024-09-21T09:33:14.827 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-21T09:33:14.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo hostname $(hostname -s) 2024-09-21T09:33:14.855 DEBUG:teuthology.orchestra.run.smithi129:> sudo hostname $(hostname -s) 2024-09-21T09:33:14.879 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-09-21T09:33:14.880 DEBUG:teuthology.orchestra.run.smithi107:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T09:33:15.081 INFO:teuthology.orchestra.run.smithi107.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Sep 21 09:33 /home/ubuntu/cephtest/cephadm 2024-09-21T09:33:15.082 DEBUG:teuthology.orchestra.run.smithi129:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T09:33:15.210 INFO:teuthology.orchestra.run.smithi129.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Sep 21 09:33 /home/ubuntu/cephtest/cephadm 2024-09-21T09:33:15.211 DEBUG:teuthology.orchestra.run.smithi107:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T09:33:15.229 DEBUG:teuthology.orchestra.run.smithi129:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T09:33:15.256 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-09-21T09:33:15.257 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-09-21T09:33:15.272 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-09-21T09:33:16.775 INFO:teuthology.orchestra.run.smithi107.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-21T09:33:16.853 INFO:teuthology.orchestra.run.smithi129.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-21T09:33:50.386 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T09:33:50.387 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_version": "ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)", 2024-09-21T09:33:50.387 INFO:teuthology.orchestra.run.smithi107.stdout: "image_id": "fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9", 2024-09-21T09:33:50.387 INFO:teuthology.orchestra.run.smithi107.stdout: "repo_digests": [ 2024-09-21T09:33:50.387 INFO:teuthology.orchestra.run.smithi107.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2" 2024-09-21T09:33:50.387 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T09:33:50.387 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T09:33:54.552 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T09:33:54.553 INFO:teuthology.orchestra.run.smithi129.stdout: "ceph_version": "ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)", 2024-09-21T09:33:54.553 INFO:teuthology.orchestra.run.smithi129.stdout: "image_id": "fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9", 2024-09-21T09:33:54.553 INFO:teuthology.orchestra.run.smithi129.stdout: "repo_digests": [ 2024-09-21T09:33:54.553 INFO:teuthology.orchestra.run.smithi129.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2" 2024-09-21T09:33:54.553 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T09:33:54.553 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T09:33:54.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph 2024-09-21T09:33:54.603 DEBUG:teuthology.orchestra.run.smithi129:> sudo mkdir -p /etc/ceph 2024-09-21T09:33:54.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /etc/ceph 2024-09-21T09:33:54.671 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 777 /etc/ceph 2024-09-21T09:33:54.695 INFO:tasks.cephadm:Writing seed config... 2024-09-21T09:33:54.696 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-09-21T09:33:54.696 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-09-21T09:33:54.696 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-21T09:33:54.696 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-21T09:33:54.696 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-21T09:33:54.696 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-09-21T09:33:54.696 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-21T09:33:54.696 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-21T09:33:54.696 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-21T09:33:54.696 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-21T09:33:54.696 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-21T09:33:54.697 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T09:33:54.697 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-21T09:33:54.728 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 86c49514-77fc-11ef-baf6-efdc52797490 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-21T09:33:54.728 DEBUG:teuthology.orchestra.run.smithi107:mon.a> sudo journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.a.service 2024-09-21T09:33:54.770 DEBUG:teuthology.orchestra.run.smithi107:mgr.y> sudo journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@mgr.y.service 2024-09-21T09:33:54.812 INFO:tasks.cephadm:Bootstrapping... 2024-09-21T09:33:54.812 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 86c49514-77fc-11ef-baf6-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.107 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-21T09:33:55.013 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:33:55.014 INFO:teuthology.orchestra.run.smithi107.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '86c49514-77fc-11ef-baf6-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.107', '--skip-admin-label'] 2024-09-21T09:33:55.037 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stdout 5.2.2 2024-09-21T09:33:55.037 INFO:teuthology.orchestra.run.smithi107.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-21T09:33:55.037 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying podman|docker is present... 2024-09-21T09:33:55.060 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stdout 5.2.2 2024-09-21T09:33:55.060 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying lvm2 is present... 2024-09-21T09:33:55.060 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying time synchronization is in place... 2024-09-21T09:33:55.068 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T09:33:55.068 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T09:33:55.075 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T09:33:55.075 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-09-21T09:33:55.081 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout enabled 2024-09-21T09:33:55.088 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout active 2024-09-21T09:33:55.088 INFO:teuthology.orchestra.run.smithi107.stdout:Unit chronyd.service is enabled and running 2024-09-21T09:33:55.088 INFO:teuthology.orchestra.run.smithi107.stdout:Repeating the final host check... 2024-09-21T09:33:55.110 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stdout 5.2.2 2024-09-21T09:33:55.110 INFO:teuthology.orchestra.run.smithi107.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-21T09:33:55.110 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl is present 2024-09-21T09:33:55.110 INFO:teuthology.orchestra.run.smithi107.stdout:lvcreate is present 2024-09-21T09:33:55.117 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T09:33:55.117 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T09:33:55.123 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T09:33:55.123 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-09-21T09:33:55.130 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout enabled 2024-09-21T09:33:55.137 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout active 2024-09-21T09:33:55.137 INFO:teuthology.orchestra.run.smithi107.stdout:Unit chronyd.service is enabled and running 2024-09-21T09:33:55.137 INFO:teuthology.orchestra.run.smithi107.stdout:Host looks OK 2024-09-21T09:33:55.137 INFO:teuthology.orchestra.run.smithi107.stdout:Cluster fsid: 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:33:55.137 INFO:teuthology.orchestra.run.smithi107.stdout:Acquiring lock 140396910306208 on /run/cephadm/86c49514-77fc-11ef-baf6-efdc52797490.lock 2024-09-21T09:33:55.137 INFO:teuthology.orchestra.run.smithi107.stdout:Lock 140396910306208 acquired on /run/cephadm/86c49514-77fc-11ef-baf6-efdc52797490.lock 2024-09-21T09:33:55.137 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying IP 172.21.15.107 port 3300 ... 2024-09-21T09:33:55.138 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying IP 172.21.15.107 port 6789 ... 2024-09-21T09:33:55.138 INFO:teuthology.orchestra.run.smithi107.stdout:Base mon IP(s) is [172.21.15.107:3300, 172.21.15.107:6789], mon addrv is [v2:172.21.15.107:3300,v1:172.21.15.107:6789] 2024-09-21T09:33:55.141 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.107 metric 100 2024-09-21T09:33:55.142 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.107 metric 100 2024-09-21T09:33:55.144 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-21T09:33:55.144 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-21T09:33:55.144 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-21T09:33:55.147 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-21T09:33:55.147 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-21T09:33:55.147 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T09:33:55.147 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-21T09:33:55.147 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link noprefixroute 2024-09-21T09:33:55.148 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T09:33:55.148 INFO:teuthology.orchestra.run.smithi107.stdout:Mon IP `172.21.15.107` is in CIDR network `172.21.0.0/20` 2024-09-21T09:33:55.148 INFO:teuthology.orchestra.run.smithi107.stdout:Mon IP `172.21.15.107` is in CIDR network `172.21.0.0/20` 2024-09-21T09:33:55.148 INFO:teuthology.orchestra.run.smithi107.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-21T09:33:55.149 INFO:teuthology.orchestra.run.smithi107.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-21T09:33:55.149 INFO:teuthology.orchestra.run.smithi107.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-21T09:33:55.627 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stdout fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 2024-09-21T09:33:55.628 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-21T09:33:55.628 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Getting image source signatures 2024-09-21T09:33:55.628 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Copying blob sha256:3ba00a516579693832719224d909a7cd5333511394d4f35f95e899e50f0bfaeb 2024-09-21T09:33:55.628 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-21T09:33:55.628 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Copying config sha256:fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 2024-09-21T09:33:55.628 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-21T09:33:56.507 INFO:teuthology.orchestra.run.smithi107.stdout:ceph: stdout ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable) 2024-09-21T09:33:56.507 INFO:teuthology.orchestra.run.smithi107.stdout:Ceph version: ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable) 2024-09-21T09:33:56.507 INFO:teuthology.orchestra.run.smithi107.stdout:Extracting ceph user uid/gid from container image... 2024-09-21T09:33:57.521 INFO:teuthology.orchestra.run.smithi107.stdout:stat: stdout 167 167 2024-09-21T09:33:57.521 INFO:teuthology.orchestra.run.smithi107.stdout:Creating initial keys... 2024-09-21T09:33:58.493 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph-authtool: stdout AQAFk+5m5ppDNBAAj5GB23gHK7GV46Va/TNc0g== 2024-09-21T09:33:59.498 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph-authtool: stdout AQAGk+5mZzeYMhAAeVgl9maLB+ATUhzJ9WXChQ== 2024-09-21T09:34:00.429 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph-authtool: stdout AQAHk+5mxnduMRAAlC3q1PoncQEVb1GhqB7PvQ== 2024-09-21T09:34:00.430 INFO:teuthology.orchestra.run.smithi107.stdout:Creating initial monmap... 2024-09-21T09:34:01.385 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T09:34:01.385 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-09-21T09:34:01.385 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:34:01.385 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T09:34:01.385 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool for a [v2:172.21.15.107:3300,v1:172.21.15.107:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T09:34:01.385 INFO:teuthology.orchestra.run.smithi107.stdout:setting min_mon_release = octopus 2024-09-21T09:34:01.385 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: set fsid to 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:34:01.385 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T09:34:01.385 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:34:01.385 INFO:teuthology.orchestra.run.smithi107.stdout:Creating mon... 2024-09-21T09:34:02.382 INFO:teuthology.orchestra.run.smithi107.stdout:create mon.a on 2024-09-21T09:34:02.586 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-21T09:34:02.724 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-21T09:34:02.871 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-86c49514-77fc-11ef-baf6-efdc52797490.target → /etc/systemd/system/ceph-86c49514-77fc-11ef-baf6-efdc52797490.target. 2024-09-21T09:34:02.871 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-86c49514-77fc-11ef-baf6-efdc52797490.target → /etc/systemd/system/ceph-86c49514-77fc-11ef-baf6-efdc52797490.target. 2024-09-21T09:34:03.035 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.a 2024-09-21T09:34:03.035 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Failed to reset failed state of unit ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.a.service: Unit ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.a.service not loaded. 2024-09-21T09:34:03.191 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-86c49514-77fc-11ef-baf6-efdc52797490.target.wants/ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.a.service → /etc/systemd/system/ceph-86c49514-77fc-11ef-baf6-efdc52797490@.service. 2024-09-21T09:34:03.700 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:03 smithi107 ceph-mon[28913]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T09:34:03.713 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T09:34:03.714 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout disabled 2024-09-21T09:34:03.720 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T09:34:03.721 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-09-21T09:34:03.721 INFO:teuthology.orchestra.run.smithi107.stdout:firewalld.service is not enabled 2024-09-21T09:34:03.721 INFO:teuthology.orchestra.run.smithi107.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T09:34:03.721 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mon to start... 2024-09-21T09:34:03.721 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mon... 2024-09-21T09:34:04.036 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:03 smithi107 bash[28876]: 550a023cc8a66773d43758e616dae429f0aba291f1e78fe45c493a84ad8a98bc 2024-09-21T09:34:04.036 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:03 smithi107 systemd[1]: Started Ceph mon.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:34:04.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout cluster: 2024-09-21T09:34:04.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout id: 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:34:04.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-21T09:34:04.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-09-21T09:34:04.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout services: 2024-09-21T09:34:04.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.541356s) 2024-09-21T09:34:04.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-21T09:34:04.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-21T09:34:04.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-09-21T09:34:04.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout data: 2024-09-21T09:34:04.678 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-21T09:34:04.678 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-21T09:34:04.678 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-21T09:34:04.678 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout pgs: 2024-09-21T09:34:04.678 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-09-21T09:34:04.678 INFO:teuthology.orchestra.run.smithi107.stdout:mon is available 2024-09-21T09:34:04.678 INFO:teuthology.orchestra.run.smithi107.stdout:Assimilating anything we can from ceph.conf... 2024-09-21T09:34:04.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:04 smithi107 ceph-mon[28913]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T09:34:04.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:04 smithi107 ceph-mon[28913]: monmap e1: 1 mons at {a=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:34:04.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:04 smithi107 ceph-mon[28913]: fsmap 2024-09-21T09:34:04.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:04 smithi107 ceph-mon[28913]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T09:34:04.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:04 smithi107 ceph-mon[28913]: mgrmap e1: no daemons active 2024-09-21T09:34:04.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:04 smithi107 ceph-mon[28913]: from='client.? 172.21.15.107:0/4006998727' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-21T09:34:05.720 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:05 smithi107 ceph-mon[28913]: from='client.? 172.21.15.107:0/1051557050' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-21T09:34:05.720 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:05 smithi107 ceph-mon[28913]: from='client.? 172.21.15.107:0/1051557050' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-21T09:34:05.724 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-09-21T09:34:05.724 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T09:34:05.724 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout fsid = 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:34:05.724 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T09:34:05.724 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.107:3300,v1:172.21.15.107:6789] 2024-09-21T09:34:05.724 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T09:34:05.724 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T09:34:05.724 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T09:34:05.725 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T09:34:05.725 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-09-21T09:34:05.725 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T09:34:05.725 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T09:34:05.725 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-09-21T09:34:05.725 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T09:34:05.725 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T09:34:05.725 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T09:34:05.725 INFO:teuthology.orchestra.run.smithi107.stdout:Generating new minimal ceph.conf... 2024-09-21T09:34:06.695 INFO:teuthology.orchestra.run.smithi107.stdout:Restarting the monitor... 2024-09-21T09:34:06.712 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:06 smithi107 ceph-mon[28913]: from='client.? 172.21.15.107:0/3948180141' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:34:06.712 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:06 smithi107 systemd[1]: Stopping Ceph mon.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:34:07.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:06 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a[28890]: 2024-09-21T09:34:06.979+0000 7ff56fdff640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T09:34:07.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:06 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a[28890]: 2024-09-21T09:34:06.979+0000 7ff56fdff640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-21T09:34:07.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:07 smithi107 podman[29230]: 2024-09-21 09:34:07.164019389 +0000 UTC m=+0.299860829 container died 550a023cc8a66773d43758e616dae429f0aba291f1e78fe45c493a84ad8a98bc (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=quincy-fbd1a6a, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-21T09:34:07.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:07 smithi107 podman[29230]: 2024-09-21 09:34:07.518525712 +0000 UTC m=+0.654367153 container cleanup 550a023cc8a66773d43758e616dae429f0aba291f1e78fe45c493a84ad8a98bc (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-fbd1a6a, CEPH_POINT_RELEASE=) 2024-09-21T09:34:07.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:07 smithi107 bash[29230]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a 2024-09-21T09:34:08.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:07 smithi107 podman[29247]: 2024-09-21 09:34:07.777865142 +0000 UTC m=+0.609109053 container remove 550a023cc8a66773d43758e616dae429f0aba291f1e78fe45c493a84ad8a98bc (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=quincy-fbd1a6a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T09:34:08.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:07 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-21T09:34:08.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:07 smithi107 systemd[1]: Stopped Ceph mon.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:34:08.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:07 smithi107 systemd[1]: Starting Ceph mon.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:34:08.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 podman[29343]: 2024-09-21 09:34:08.056702796 +0000 UTC m=+0.015100234 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph:quincy 2024-09-21T09:34:08.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 podman[29343]: 2024-09-21 09:34:08.162397424 +0000 UTC m=+0.120794905 container create 7135d1946aead717a053d91222f0f9fd6d96e6a9ec78be14381becd8db0beac9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-21T09:34:08.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 podman[29343]: 2024-09-21 09:34:08.355996487 +0000 UTC m=+0.314393930 container init 7135d1946aead717a053d91222f0f9fd6d96e6a9ec78be14381becd8db0beac9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T09:34:08.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 podman[29343]: 2024-09-21 09:34:08.360274277 +0000 UTC m=+0.318671712 container start 7135d1946aead717a053d91222f0f9fd6d96e6a9ec78be14381becd8db0beac9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS) 2024-09-21T09:34:08.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T09:34:08.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable), process ceph-mon, pid 7 2024-09-21T09:34:08.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: pidfile_write: ignore empty --pid-file 2024-09-21T09:34:08.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: load: jerasure load: lrc 2024-09-21T09:34:08.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: RocksDB version: 6.15.5 2024-09-21T09:34:08.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-21T09:34:08.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Compile date Sep 20 2024 2024-09-21T09:34:08.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: DB SUMMARY 2024-09-21T09:34:08.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: DB Session ID: NEJ24H66ZCVWK93P9ZBM 2024-09-21T09:34:08.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: CURRENT file: CURRENT 2024-09-21T09:34:08.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T09:34:08.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-09-21T09:34:08.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-21T09:34:08.434 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 83718 ; 2024-09-21T09:34:08.434 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.error_if_exists: 0 2024-09-21T09:34:08.434 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.create_if_missing: 0 2024-09-21T09:34:08.434 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.paranoid_checks: 1 2024-09-21T09:34:08.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T09:34:08.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.env: 0x5640ffdd8440 2024-09-21T09:34:08.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.fs: Posix File System 2024-09-21T09:34:08.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.info_log: 0x5641006b4ec0 2024-09-21T09:34:08.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T09:34:08.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.statistics: (nil) 2024-09-21T09:34:08.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.use_fsync: 0 2024-09-21T09:34:08.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_log_file_size: 0 2024-09-21T09:34:08.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T09:34:08.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T09:34:08.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T09:34:08.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T09:34:08.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.allow_fallocate: 1 2024-09-21T09:34:08.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T09:34:08.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T09:34:08.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.use_direct_reads: 0 2024-09-21T09:34:08.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T09:34:08.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T09:34:08.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.db_log_dir: 2024-09-21T09:34:08.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-09-21T09:34:08.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T09:34:08.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T09:34:08.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T09:34:08.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T09:34:08.437 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T09:34:08.437 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T09:34:08.437 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T09:34:08.437 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T09:34:08.437 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.write_buffer_manager: 0x564100772780 2024-09-21T09:34:08.437 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T09:34:08.437 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-21T09:34:08.437 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T09:34:08.437 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T09:34:08.437 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T09:34:08.437 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T09:34:08.437 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T09:34:08.437 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T09:34:08.437 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T09:34:08.438 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.unordered_write: 0 2024-09-21T09:34:08.438 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T09:34:08.438 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T09:34:08.438 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T09:34:08.438 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T09:34:08.438 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.row_cache: None 2024-09-21T09:34:08.438 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.wal_filter: None 2024-09-21T09:34:08.438 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T09:34:08.438 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T09:34:08.438 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.preserve_deletes: 0 2024-09-21T09:34:08.438 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.two_write_queues: 0 2024-09-21T09:34:08.438 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T09:34:08.438 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.atomic_flush: 0 2024-09-21T09:34:08.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T09:34:08.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T09:34:08.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T09:34:08.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.log_readahead_size: 0 2024-09-21T09:34:08.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T09:34:08.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T09:34:08.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T09:34:08.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T09:34:08.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T09:34:08.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T09:34:08.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_background_jobs: 2 2024-09-21T09:34:08.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_background_compactions: -1 2024-09-21T09:34:08.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_subcompactions: 1 2024-09-21T09:34:08.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T09:34:08.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T09:34:08.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T09:34:08.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T09:34:08.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T09:34:08.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T09:34:08.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T09:34:08.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T09:34:08.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_open_files: -1 2024-09-21T09:34:08.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T09:34:08.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T09:34:08.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T09:34:08.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T09:34:08.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_background_flushes: -1 2024-09-21T09:34:08.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Compression algorithms supported: 2024-09-21T09:34:08.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: kZSTD supported: 0 2024-09-21T09:34:08.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: kXpressCompression supported: 0 2024-09-21T09:34:08.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: kBZip2Compression supported: 0 2024-09-21T09:34:08.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T09:34:08.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: kLZ4Compression supported: 1 2024-09-21T09:34:08.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: kZlibCompression supported: 1 2024-09-21T09:34:08.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T09:34:08.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: kSnappyCompression supported: 1 2024-09-21T09:34:08.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T09:34:08.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-09-21T09:34:08.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-21T09:34:08.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T09:34:08.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.merge_operator: 2024-09-21T09:34:08.442 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compaction_filter: None 2024-09-21T09:34:08.442 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T09:34:08.442 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T09:34:08.442 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T09:34:08.442 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T09:34:08.442 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56410065ba38) 2024-09-21T09:34:08.442 INFO:journalctl@ceph.mon.a.smithi107.stdout: cache_index_and_filter_blocks: 1 2024-09-21T09:34:08.442 INFO:journalctl@ceph.mon.a.smithi107.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T09:34:08.442 INFO:journalctl@ceph.mon.a.smithi107.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T09:34:08.442 INFO:journalctl@ceph.mon.a.smithi107.stdout: pin_top_level_index_and_filter: 1 2024-09-21T09:34:08.442 INFO:journalctl@ceph.mon.a.smithi107.stdout: index_type: 0 2024-09-21T09:34:08.442 INFO:journalctl@ceph.mon.a.smithi107.stdout: data_block_index_type: 0 2024-09-21T09:34:08.442 INFO:journalctl@ceph.mon.a.smithi107.stdout: index_shortening: 1 2024-09-21T09:34:08.443 INFO:journalctl@ceph.mon.a.smithi107.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T09:34:08.443 INFO:journalctl@ceph.mon.a.smithi107.stdout: hash_index_allow_collision: 1 2024-09-21T09:34:08.443 INFO:journalctl@ceph.mon.a.smithi107.stdout: checksum: 1 2024-09-21T09:34:08.443 INFO:journalctl@ceph.mon.a.smithi107.stdout: no_block_cache: 0 2024-09-21T09:34:08.443 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache: 0x5641006911f0 2024-09-21T09:34:08.443 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache_name: BinnedLRUCache 2024-09-21T09:34:08.443 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache_options: 2024-09-21T09:34:08.443 INFO:journalctl@ceph.mon.a.smithi107.stdout: capacity : 536870912 2024-09-21T09:34:08.443 INFO:journalctl@ceph.mon.a.smithi107.stdout: num_shard_bits : 4 2024-09-21T09:34:08.443 INFO:journalctl@ceph.mon.a.smithi107.stdout: strict_capacity_limit : 0 2024-09-21T09:34:08.443 INFO:journalctl@ceph.mon.a.smithi107.stdout: high_pri_pool_ratio: 0.000 2024-09-21T09:34:08.443 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache_compressed: (nil) 2024-09-21T09:34:08.443 INFO:journalctl@ceph.mon.a.smithi107.stdout: persistent_cache: (nil) 2024-09-21T09:34:08.444 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_size: 4096 2024-09-21T09:34:08.444 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_size_deviation: 10 2024-09-21T09:34:08.444 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_restart_interval: 16 2024-09-21T09:34:08.444 INFO:journalctl@ceph.mon.a.smithi107.stdout: index_block_restart_interval: 1 2024-09-21T09:34:08.444 INFO:journalctl@ceph.mon.a.smithi107.stdout: metadata_block_size: 4096 2024-09-21T09:34:08.444 INFO:journalctl@ceph.mon.a.smithi107.stdout: partition_filters: 0 2024-09-21T09:34:08.444 INFO:journalctl@ceph.mon.a.smithi107.stdout: use_delta_encoding: 1 2024-09-21T09:34:08.444 INFO:journalctl@ceph.mon.a.smithi107.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-21T09:34:08.444 INFO:journalctl@ceph.mon.a.smithi107.stdout: whole_key_filtering: 1 2024-09-21T09:34:08.444 INFO:journalctl@ceph.mon.a.smithi107.stdout: verify_compression: 0 2024-09-21T09:34:08.444 INFO:journalctl@ceph.mon.a.smithi107.stdout: read_amp_bytes_per_bit: 0 2024-09-21T09:34:08.444 INFO:journalctl@ceph.mon.a.smithi107.stdout: format_version: 4 2024-09-21T09:34:08.444 INFO:journalctl@ceph.mon.a.smithi107.stdout: enable_index_compression: 1 2024-09-21T09:34:08.444 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_align: 0 2024-09-21T09:34:08.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T09:34:08.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T09:34:08.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compression: NoCompression 2024-09-21T09:34:08.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T09:34:08.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T09:34:08.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T09:34:08.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.num_levels: 7 2024-09-21T09:34:08.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T09:34:08.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T09:34:08.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T09:34:08.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T09:34:08.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T09:34:08.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T09:34:08.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T09:34:08.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:34:08.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T09:34:08.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T09:34:08.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T09:34:08.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T09:34:08.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T09:34:08.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T09:34:08.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:34:08.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T09:34:08.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T09:34:08.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T09:34:08.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T09:34:08.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T09:34:08.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T09:34:08.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T09:34:08.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T09:34:08.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T09:34:08.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T09:34:08.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T09:34:08.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T09:34:08.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T09:34:08.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T09:34:08.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T09:34:08.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T09:34:08.447 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T09:34:08.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T09:34:08.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T09:34:08.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.arena_block_size: 4194304 2024-09-21T09:34:08.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T09:34:08.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T09:34:08.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-21T09:34:08.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T09:34:08.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T09:34:08.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T09:34:08.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T09:34:08.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T09:34:08.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T09:34:08.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T09:34:08.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T09:34:08.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T09:34:08.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T09:34:08.449 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T09:34:08.449 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T09:34:08.449 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.inplace_update_support: 0 2024-09-21T09:34:08.449 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T09:34:08.449 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T09:34:08.449 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T09:34:08.449 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T09:34:08.449 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.bloom_locality: 0 2024-09-21T09:34:08.449 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.max_successive_merges: 0 2024-09-21T09:34:08.449 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T09:34:08.449 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T09:34:08.449 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T09:34:08.449 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T09:34:08.450 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.ttl: 2592000 2024-09-21T09:34:08.450 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T09:34:08.450 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.enable_blob_files: false 2024-09-21T09:34:08.450 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.min_blob_size: 0 2024-09-21T09:34:08.450 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T09:34:08.450 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T09:34:08.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T09:34:08.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T09:34:08.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T09:34:08.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-09-21T09:34:08.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-09-21T09:34:08.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726911248383590, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-09-21T09:34:08.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-09-21T09:34:08.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-21T09:34:08.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726911248384336, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 80291, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 78533, "index_size": 220, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10130, "raw_average_key_size": 47, "raw_value_size": 72746, "raw_average_value_size": 341, "num_data_blocks": 10, "num_entries": 213, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726911248, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "34211761-7902-49b4-b259-2c8224153f0f", "db_session_id": "NEJ24H66ZCVWK93P9ZBM"}} 2024-09-21T09:34:08.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-09-21T09:34:08.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726911248384889, "job": 1, "event": "recovery_finished"} 2024-09-21T09:34:08.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T09:34:08.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x5641006c4700 2024-09-21T09:34:08.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: DB pointer 0x564100694000 2024-09-21T09:34:08.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-21T09:34:08.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-21T09:34:08.452 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** DB Stats ** 2024-09-21T09:34:08.452 INFO:journalctl@ceph.mon.a.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:34:08.452 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T09:34:08.452 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:34:08.452 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:34:08.452 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T09:34:08.452 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-21T09:34:08.452 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:34:08.452 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-09-21T09:34:08.452 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** Compaction Stats [default] ** 2024-09-21T09:34:08.452 INFO:journalctl@ceph.mon.a.smithi107.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:34:08.452 INFO:journalctl@ceph.mon.a.smithi107.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:34:08.452 INFO:journalctl@ceph.mon.a.smithi107.stdout: L0 2/0 80.19 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 152.2 0.00 0.00 1 0.001 0 0 2024-09-21T09:34:08.452 INFO:journalctl@ceph.mon.a.smithi107.stdout: Sum 2/0 80.19 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 152.2 0.00 0.00 1 0.001 0 0 2024-09-21T09:34:08.453 INFO:journalctl@ceph.mon.a.smithi107.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 152.2 0.00 0.00 1 0.001 0 0 2024-09-21T09:34:08.453 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-09-21T09:34:08.453 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** Compaction Stats [default] ** 2024-09-21T09:34:08.453 INFO:journalctl@ceph.mon.a.smithi107.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:34:08.453 INFO:journalctl@ceph.mon.a.smithi107.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:34:08.453 INFO:journalctl@ceph.mon.a.smithi107.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 152.2 0.00 0.00 1 0.001 0 0 2024-09-21T09:34:08.453 INFO:journalctl@ceph.mon.a.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:34:08.453 INFO:journalctl@ceph.mon.a.smithi107.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:34:08.453 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:34:08.453 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:34:08.453 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:34:08.453 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:34:08.453 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative compaction: 0.00 GB write, 23.80 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:34:08.453 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval compaction: 0.00 GB write, 23.80 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:34:08.453 INFO:journalctl@ceph.mon.a.smithi107.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:34:08.453 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-09-21T09:34:08.454 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:34:08.454 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-09-21T09:34:08.454 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** Compaction Stats [default] ** 2024-09-21T09:34:08.454 INFO:journalctl@ceph.mon.a.smithi107.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:34:08.454 INFO:journalctl@ceph.mon.a.smithi107.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:34:08.454 INFO:journalctl@ceph.mon.a.smithi107.stdout: L0 2/0 80.19 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 152.2 0.00 0.00 1 0.001 0 0 2024-09-21T09:34:08.454 INFO:journalctl@ceph.mon.a.smithi107.stdout: Sum 2/0 80.19 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 152.2 0.00 0.00 1 0.001 0 0 2024-09-21T09:34:08.454 INFO:journalctl@ceph.mon.a.smithi107.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-21T09:34:08.454 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-09-21T09:34:08.454 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** Compaction Stats [default] ** 2024-09-21T09:34:08.454 INFO:journalctl@ceph.mon.a.smithi107.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:34:08.454 INFO:journalctl@ceph.mon.a.smithi107.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:34:08.454 INFO:journalctl@ceph.mon.a.smithi107.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 152.2 0.00 0.00 1 0.001 0 0 2024-09-21T09:34:08.454 INFO:journalctl@ceph.mon.a.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:34:08.455 INFO:journalctl@ceph.mon.a.smithi107.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:34:08.455 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:34:08.455 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:34:08.455 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:34:08.455 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:34:08.455 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative compaction: 0.00 GB write, 23.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:34:08.455 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:34:08.455 INFO:journalctl@ceph.mon.a.smithi107.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:34:08.455 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-09-21T09:34:08.455 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:34:08.455 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: starting mon.a rank 0 at public addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] at bind addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:34:08.455 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: mon.a@-1(???) e1 preinit fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:34:08.455 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: mon.a@-1(???).mds e1 new map 2024-09-21T09:34:08.455 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: mon.a@-1(???).mds e1 print_map 2024-09-21T09:34:08.455 INFO:journalctl@ceph.mon.a.smithi107.stdout: e1 2024-09-21T09:34:08.456 INFO:journalctl@ceph.mon.a.smithi107.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T09:34:08.456 INFO:journalctl@ceph.mon.a.smithi107.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:34:08.456 INFO:journalctl@ceph.mon.a.smithi107.stdout: legacy client fscid: -1 2024-09-21T09:34:08.456 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-09-21T09:34:08.456 INFO:journalctl@ceph.mon.a.smithi107.stdout: No filesystems configured 2024-09-21T09:34:08.456 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T09:34:08.456 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:34:08.456 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:34:08.456 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:34:08.456 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-21T09:34:08.456 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 bash[29343]: 7135d1946aead717a053d91222f0f9fd6d96e6a9ec78be14381becd8db0beac9 2024-09-21T09:34:08.456 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 systemd[1]: Started Ceph mon.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:34:08.456 INFO:teuthology.orchestra.run.smithi107.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-21T09:34:08.685 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T09:34:08.685 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: monmap e1: 1 mons at {a=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:34:08.685 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: fsmap 2024-09-21T09:34:08.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T09:34:08.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:08 smithi107 ceph-mon[29382]: mgrmap e1: no daemons active 2024-09-21T09:34:09.507 INFO:teuthology.orchestra.run.smithi107.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-21T09:34:09.507 INFO:teuthology.orchestra.run.smithi107.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-21T09:34:09.508 INFO:teuthology.orchestra.run.smithi107.stdout:Creating mgr... 2024-09-21T09:34:09.508 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-21T09:34:09.508 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-21T09:34:09.676 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-86c49514-77fc-11ef-baf6-efdc52797490@mgr.y 2024-09-21T09:34:09.677 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Failed to reset failed state of unit ceph-86c49514-77fc-11ef-baf6-efdc52797490@mgr.y.service: Unit ceph-86c49514-77fc-11ef-baf6-efdc52797490@mgr.y.service not loaded. 2024-09-21T09:34:09.819 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-86c49514-77fc-11ef-baf6-efdc52797490.target.wants/ceph-86c49514-77fc-11ef-baf6-efdc52797490@mgr.y.service → /etc/systemd/system/ceph-86c49514-77fc-11ef-baf6-efdc52797490@.service. 2024-09-21T09:34:10.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:09 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2867922488' entity='client.admin' 2024-09-21T09:34:10.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:10 smithi107 podman[29618]: 2024-09-21 09:34:09.951133612 +0000 UTC m=+0.015483918 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph:quincy 2024-09-21T09:34:10.338 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T09:34:10.338 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout disabled 2024-09-21T09:34:10.347 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T09:34:10.347 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-09-21T09:34:10.347 INFO:teuthology.orchestra.run.smithi107.stdout:firewalld.service is not enabled 2024-09-21T09:34:10.347 INFO:teuthology.orchestra.run.smithi107.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T09:34:10.356 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T09:34:10.356 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout disabled 2024-09-21T09:34:10.365 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T09:34:10.365 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-09-21T09:34:10.365 INFO:teuthology.orchestra.run.smithi107.stdout:firewalld.service is not enabled 2024-09-21T09:34:10.365 INFO:teuthology.orchestra.run.smithi107.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-21T09:34:10.365 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mgr to start... 2024-09-21T09:34:10.365 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mgr... 2024-09-21T09:34:10.489 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:10 smithi107 podman[29618]: 2024-09-21 09:34:10.231999279 +0000 UTC m=+0.296349594 container init 5ab1b97eb209e0bad7b9ccee96ed698fbe53313670bf52acb80a62e23bef52d2 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y, RELEASE=quincy-fbd1a6a, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T09:34:10.490 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:10 smithi107 podman[29618]: 2024-09-21 09:34:10.237019838 +0000 UTC m=+0.301370145 container start 5ab1b97eb209e0bad7b9ccee96ed698fbe53313670bf52acb80a62e23bef52d2 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-21T09:34:10.490 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:10 smithi107 bash[29618]: 5ab1b97eb209e0bad7b9ccee96ed698fbe53313670bf52acb80a62e23bef52d2 2024-09-21T09:34:10.490 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:10 smithi107 systemd[1]: Started Ceph mgr.y for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:34:10.490 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:10 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:10.354+0000 7fc02e422ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:34:10.490 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:10 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:10.414+0000 7fc02e422ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:34:10.490 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:10 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:10.478+0000 7fc02e422ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:34:10.747 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:10 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:10.605+0000 7fc02e422ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:34:10.747 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:10 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:10.745+0000 7fc02e422ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:34:11.013 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:10 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/214260906' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-21T09:34:11.358 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:11.011+0000 7fc02e422ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:34:11.358 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:11.071+0000 7fc02e422ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:34:11.359 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:11.234+0000 7fc02e422ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:34:11.636 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-09-21T09:34:11.637 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:34:11.637 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsid": "86c49514-77fc-11ef-baf6-efdc52797490", 2024-09-21T09:34:11.637 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T09:34:11.637 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T09:34:11.637 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T09:34:11.637 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T09:34:11.637 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:11.637 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T09:34:11.637 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T09:34:11.637 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 0 2024-09-21T09:34:11.637 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:34:11.637 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T09:34:11.638 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T09:34:11.638 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:34:11.638 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-21T09:34:11.638 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T09:34:11.638 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:34:11.638 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-21T09:34:11.638 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T09:34:11.638 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:11.638 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T09:34:11.638 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:34:11.638 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T09:34:11.638 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T09:34:11.638 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T09:34:11.642 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T09:34:11.642 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T09:34:11.642 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T09:34:11.642 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:11.642 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T09:34:11.642 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T09:34:11.642 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T09:34:11.642 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T09:34:11.642 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T09:34:11.642 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T09:34:11.642 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T09:34:11.642 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T09:34:11.642 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T09:34:11.643 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:11.643 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T09:34:11.643 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:34:11.643 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T09:34:11.643 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T09:34:11.643 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:11.643 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T09:34:11.643 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T09:34:11.643 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T09:34:11.643 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T09:34:11.643 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T09:34:11.643 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T09:34:11.643 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T09:34:11.643 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:34:11.644 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:34:11.644 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:11.644 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T09:34:11.644 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:34:11.644 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T09:34:03.636450+0000", 2024-09-21T09:34:11.644 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:34:11.644 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:11.644 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T09:34:11.644 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:34:11.644 INFO:teuthology.orchestra.run.smithi107.stdout:mgr not available, waiting (1/15)... 2024-09-21T09:34:12.358 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:12 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:12.069+0000 7fc02e422ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:34:12.358 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:12 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:12.131+0000 7fc02e422ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:34:13.108 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:12 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:12.823+0000 7fc02e422ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:34:13.108 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:12 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:12.932+0000 7fc02e422ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:34:13.108 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:13.049+0000 7fc02e422ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:34:13.358 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:13.106+0000 7fc02e422ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:34:13.358 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:13.228+0000 7fc02e422ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:34:13.359 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:13.293+0000 7fc02e422ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:34:13.359 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:13.349+0000 7fc02e422ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:34:13.748 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:13.481+0000 7fc02e422ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:34:13.749 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:13.543+0000 7fc02e422ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:34:13.749 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:13.606+0000 7fc02e422ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:34:13.749 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:13.690+0000 7fc02e422ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:34:13.749 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:13.746+0000 7fc02e422ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:34:14.059 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:13.804+0000 7fc02e422ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:34:14.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:14 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3901395149' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-21T09:34:14.705 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:14 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:14.565+0000 7fc02e422ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:34:14.709 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-09-21T09:34:14.709 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:34:14.709 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsid": "86c49514-77fc-11ef-baf6-efdc52797490", 2024-09-21T09:34:14.709 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T09:34:14.709 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T09:34:14.709 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T09:34:14.709 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T09:34:14.709 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:14.710 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T09:34:14.710 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T09:34:14.710 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 0 2024-09-21T09:34:14.710 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:34:14.710 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T09:34:14.710 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T09:34:14.710 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:34:14.710 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-21T09:34:14.710 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T09:34:14.710 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:34:14.710 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-21T09:34:14.710 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T09:34:14.710 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:14.710 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T09:34:14.711 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:34:14.711 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T09:34:14.711 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T09:34:14.711 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T09:34:14.711 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T09:34:14.711 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T09:34:14.711 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T09:34:14.714 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:14.714 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T09:34:14.714 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T09:34:14.714 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T09:34:14.714 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T09:34:14.714 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T09:34:14.714 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T09:34:14.714 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T09:34:14.714 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T09:34:14.714 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T09:34:14.714 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:14.714 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T09:34:14.715 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:34:14.715 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T09:34:14.715 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T09:34:14.715 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:14.715 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T09:34:14.715 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T09:34:14.715 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T09:34:14.715 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T09:34:14.715 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T09:34:14.715 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T09:34:14.715 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T09:34:14.715 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:34:14.715 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:34:14.715 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:14.715 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T09:34:14.715 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:34:14.716 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T09:34:03.636450+0000", 2024-09-21T09:34:14.716 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:34:14.716 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:14.716 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T09:34:14.716 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:34:14.716 INFO:teuthology.orchestra.run.smithi107.stdout:mgr not available, waiting (2/15)... 2024-09-21T09:34:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:15 smithi107 ceph-mon[29382]: Activating manager daemon y 2024-09-21T09:34:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:15 smithi107 ceph-mon[29382]: mgrmap e2: y(active, starting, since 0.00282671s) 2024-09-21T09:34:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:15 smithi107 ceph-mon[29382]: from='mgr.14100 172.21.15.107:0/3355093174' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:34:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:15 smithi107 ceph-mon[29382]: from='mgr.14100 172.21.15.107:0/3355093174' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:34:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:15 smithi107 ceph-mon[29382]: from='mgr.14100 172.21.15.107:0/3355093174' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:34:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:15 smithi107 ceph-mon[29382]: from='mgr.14100 172.21.15.107:0/3355093174' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:34:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:15 smithi107 ceph-mon[29382]: from='mgr.14100 172.21.15.107:0/3355093174' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:34:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:15 smithi107 ceph-mon[29382]: Manager daemon y is now available 2024-09-21T09:34:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:15 smithi107 ceph-mon[29382]: from='mgr.14100 172.21.15.107:0/3355093174' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:34:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:15 smithi107 ceph-mon[29382]: from='mgr.14100 172.21.15.107:0/3355093174' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:34:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:15 smithi107 ceph-mon[29382]: from='mgr.14100 172.21.15.107:0/3355093174' entity='mgr.y' 2024-09-21T09:34:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:15 smithi107 ceph-mon[29382]: from='mgr.14100 172.21.15.107:0/3355093174' entity='mgr.y' 2024-09-21T09:34:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:15 smithi107 ceph-mon[29382]: from='mgr.14100 172.21.15.107:0/3355093174' entity='mgr.y' 2024-09-21T09:34:16.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:16 smithi107 ceph-mon[29382]: mgrmap e3: y(active, since 1.00714s) 2024-09-21T09:34:17.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:17 smithi107 ceph-mon[29382]: mgrmap e4: y(active, since 2s) 2024-09-21T09:34:18.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:18 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2406224979' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-21T09:34:24.307 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-09-21T09:34:24.307 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:34:24.307 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsid": "86c49514-77fc-11ef-baf6-efdc52797490", 2024-09-21T09:34:24.307 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T09:34:24.307 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T09:34:24.307 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T09:34:24.307 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T09:34:24.307 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:24.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T09:34:24.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T09:34:24.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 0 2024-09-21T09:34:24.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:34:24.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T09:34:24.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T09:34:24.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:34:24.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-21T09:34:24.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T09:34:24.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:34:24.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-21T09:34:24.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T09:34:24.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:24.308 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T09:34:24.309 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:34:24.309 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T09:34:24.309 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T09:34:24.309 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T09:34:24.309 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T09:34:24.309 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T09:34:24.309 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T09:34:24.312 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:24.312 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T09:34:24.312 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T09:34:24.312 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T09:34:24.312 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T09:34:24.312 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T09:34:24.312 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T09:34:24.312 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T09:34:24.312 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T09:34:24.313 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T09:34:24.313 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:24.313 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T09:34:24.313 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:34:24.313 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T09:34:24.313 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T09:34:24.313 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:24.313 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T09:34:24.313 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T09:34:24.313 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T09:34:24.313 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T09:34:24.313 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T09:34:24.313 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T09:34:24.313 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T09:34:24.314 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:34:24.314 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:34:24.314 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:24.314 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T09:34:24.314 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:34:24.314 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T09:34:03.636450+0000", 2024-09-21T09:34:24.314 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:34:24.314 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:34:24.315 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T09:34:24.315 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:34:24.315 INFO:teuthology.orchestra.run.smithi107.stdout:mgr is available 2024-09-21T09:34:28.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:27 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3088293721' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-21T09:34:29.620 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-09-21T09:34:29.620 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T09:34:29.620 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout fsid = 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:34:29.620 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T09:34:29.620 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T09:34:29.620 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T09:34:29.620 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T09:34:29.620 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T09:34:29.620 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-09-21T09:34:29.621 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T09:34:29.621 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T09:34:29.621 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-09-21T09:34:29.621 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T09:34:29.621 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T09:34:29.621 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T09:34:29.621 INFO:teuthology.orchestra.run.smithi107.stdout:Enabling cephadm module... 2024-09-21T09:34:30.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:30 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1754276548' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-21T09:34:30.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:30 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ignoring --setuser ceph since I am not root 2024-09-21T09:34:30.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:30 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ignoring --setgroup ceph since I am not root 2024-09-21T09:34:30.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:30 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:30.423+0000 7f5dfca89ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:34:30.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:30 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:30.477+0000 7f5dfca89ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:34:30.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:30 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:30.541+0000 7f5dfca89ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:34:30.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:30 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:30.666+0000 7f5dfca89ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:34:30.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:30 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:30.802+0000 7f5dfca89ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:34:31.193 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:31.066+0000 7f5dfca89ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:34:31.193 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:31.129+0000 7f5dfca89ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:34:31.608 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:31.297+0000 7f5dfca89ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:34:31.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:31 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1754276548' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-21T09:34:31.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:31 smithi107 ceph-mon[29382]: mgrmap e5: y(active, since 15s) 2024-09-21T09:34:32.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:32 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/472459254' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-21T09:34:32.608 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:32.184+0000 7f5dfca89ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:34:32.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:32.248+0000 7f5dfca89ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:34:32.639 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:34:32.639 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-21T09:34:32.639 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T09:34:32.640 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-21T09:34:32.640 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T09:34:32.640 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:34:32.640 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for the mgr to restart... 2024-09-21T09:34:32.640 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mgr epoch 5... 2024-09-21T09:34:33.264 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:32.968+0000 7f5dfca89ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:34:33.264 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:33.085+0000 7f5dfca89ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:34:33.264 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:33.204+0000 7f5dfca89ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:34:33.264 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:33.262+0000 7f5dfca89ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:34:33.608 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:33.386+0000 7f5dfca89ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:34:33.608 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:33.452+0000 7f5dfca89ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:34:33.608 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:33.509+0000 7f5dfca89ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:34:33.907 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:33.641+0000 7f5dfca89ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:34:33.907 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:33.704+0000 7f5dfca89ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:34:33.907 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:33.769+0000 7f5dfca89ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:34:33.907 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:33.848+0000 7f5dfca89ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:34:33.907 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:33.905+0000 7f5dfca89ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:34:34.358 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:33.961+0000 7f5dfca89ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:34:35.108 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:34.695+0000 7f5dfca89ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:34:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:34 smithi107 ceph-mon[29382]: Active manager daemon y restarted 2024-09-21T09:34:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:34 smithi107 ceph-mon[29382]: Activating manager daemon y 2024-09-21T09:34:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:34 smithi107 ceph-mon[29382]: osdmap e2: 0 total, 0 up, 0 in 2024-09-21T09:34:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:34 smithi107 ceph-mon[29382]: mgrmap e6: y(active, starting, since 0.00301612s) 2024-09-21T09:34:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:34 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:34:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:34 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:34:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:34 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:34:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:34 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:34:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:34 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:34:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:34 smithi107 ceph-mon[29382]: Manager daemon y is now available 2024-09-21T09:34:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:34 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' 2024-09-21T09:34:35.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:34 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' 2024-09-21T09:34:35.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:34 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:34:35.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:34 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:34:35.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:34 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:34:35.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:34 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:34:36.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:35 smithi107 ceph-mon[29382]: Found migration_current of "None". Setting to last migration. 2024-09-21T09:34:36.042 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:35 smithi107 ceph-mon[29382]: mgrmap e7: y(active, since 1.00485s) 2024-09-21T09:34:36.236 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:34:36.236 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-21T09:34:36.236 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T09:34:36.236 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:34:36.236 INFO:teuthology.orchestra.run.smithi107.stdout:mgr epoch 5 is available 2024-09-21T09:34:36.236 INFO:teuthology.orchestra.run.smithi107.stdout:Setting orchestrator backend to cephadm... 2024-09-21T09:34:36.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:36.378+0000 7f5daee78640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:34:36] ENGINE Error in HTTPServer.serve 2024-09-21T09:34:36.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: Traceback (most recent call last): 2024-09-21T09:34:36.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:34:36.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: self._connections.run(self.expiration_interval) 2024-09-21T09:34:36.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:34:36.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: self._run(expiration_interval) 2024-09-21T09:34:36.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:34:36.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:34:36.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:34:36.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:34:36.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:34:36.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: s = self.context.wrap_socket( 2024-09-21T09:34:36.610 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:34:36.610 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: return self.sslsocket_class._create( 2024-09-21T09:34:36.610 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:34:36.610 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: self.do_handshake() 2024-09-21T09:34:36.610 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:34:36.610 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: self._sslobj.do_handshake() 2024-09-21T09:34:36.610 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:34:36.610 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:37.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-mon[29382]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T09:34:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-mon[29382]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T09:34:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-mon[29382]: [21/Sep/2024:09:34:36] ENGINE Bus STARTING 2024-09-21T09:34:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' 2024-09-21T09:34:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:36 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:34:37.994 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:37 smithi107 ceph-mon[29382]: [21/Sep/2024:09:34:36] ENGINE Serving on https://172.21.15.107:7150 2024-09-21T09:34:37.994 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:37 smithi107 ceph-mon[29382]: [21/Sep/2024:09:34:36] ENGINE Bus STARTED 2024-09-21T09:34:37.994 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:37 smithi107 ceph-mon[29382]: [21/Sep/2024:09:34:36] ENGINE Error in HTTPServer.serve 2024-09-21T09:34:37.994 INFO:journalctl@ceph.mon.a.smithi107.stdout: Traceback (most recent call last): 2024-09-21T09:34:37.994 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:34:37.994 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:34:37.994 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:34:37.994 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._run(expiration_interval) 2024-09-21T09:34:37.994 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:34:37.994 INFO:journalctl@ceph.mon.a.smithi107.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:34:37.994 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:34:37.994 INFO:journalctl@ceph.mon.a.smithi107.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:34:37.995 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:34:37.995 INFO:journalctl@ceph.mon.a.smithi107.stdout: s = self.context.wrap_socket( 2024-09-21T09:34:37.995 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:34:37.995 INFO:journalctl@ceph.mon.a.smithi107.stdout: return self.sslsocket_class._create( 2024-09-21T09:34:37.995 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:34:37.995 INFO:journalctl@ceph.mon.a.smithi107.stdout: self.do_handshake() 2024-09-21T09:34:37.995 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:34:37.995 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._sslobj.do_handshake() 2024-09-21T09:34:37.995 INFO:journalctl@ceph.mon.a.smithi107.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:34:37.995 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:37 smithi107 ceph-mon[29382]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:34:37.995 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:37 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' 2024-09-21T09:34:37.995 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:37 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:34:37.995 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:37 smithi107 ceph-mon[29382]: mgrmap e8: y(active, since 2s) 2024-09-21T09:34:38.531 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-21T09:34:38.531 INFO:teuthology.orchestra.run.smithi107.stdout:Generating ssh key... 2024-09-21T09:34:39.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:38 smithi107 ceph-mon[29382]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:34:39.551 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: Generating public/private rsa key pair. 2024-09-21T09:34:39.551 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: Your identification has been saved in /tmp/tmp70gtlr54/key 2024-09-21T09:34:39.551 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: Your public key has been saved in /tmp/tmp70gtlr54/key.pub 2024-09-21T09:34:39.551 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: The key fingerprint is: 2024-09-21T09:34:39.552 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: SHA256:YhRtvtbBupLM8OpcgJPSHyH8iumsx/zP1geNVvEpdB4 ceph-86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:34:39.552 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: The key's randomart image is: 2024-09-21T09:34:39.552 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: +---[RSA 3072]----+ 2024-09-21T09:34:39.552 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: | .. | 2024-09-21T09:34:39.552 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: | . .o o E | 2024-09-21T09:34:39.552 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: | o . .o o = o | 2024-09-21T09:34:39.552 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: | . = o . = + | 2024-09-21T09:34:39.552 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: |. = + o SB o | 2024-09-21T09:34:39.552 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: | + +.+ .B o | 2024-09-21T09:34:39.552 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: |oo. .=.= o | 2024-09-21T09:34:39.552 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: |o +. oB o . | 2024-09-21T09:34:39.552 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: |o+ o=+o. . | 2024-09-21T09:34:39.552 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: +----[SHA256]-----+ 2024-09-21T09:34:40.438 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:40 smithi107 ceph-mon[29382]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:34:40.438 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:40 smithi107 ceph-mon[29382]: Generating ssh key... 2024-09-21T09:34:40.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:40 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' 2024-09-21T09:34:40.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:40 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' 2024-09-21T09:34:40.943 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCd2RbKBwQPD27iss2rrkfOfOs+mNazD9XXjrOQLK+9F1l5gzSMdOlUB9rBaiM6NAhhG44wgblVIU31+LtJQQQCS6XmyRa8hRG0Q3XZkuXb/JJ8OsZmldgFXkGRhuir/PUwqvWrABlZvNU9EzyFRVI44u9CcadmMJ4CkCLPfbHWAFo0WNzQEy2Rr524Byve+j4/AEzRMqZbvVtqhnhDWN5jG1d+GdaigAg3G/x3HGtOKlul1AQhUvI2mHKh5nR72/7qKO0/dN0YXQ3L3Fi+png1jK18AfDGbZCib2Dp3pOkgctog7WqaRrpDx4UejdzEPcd28q3Fma2R/BnyCby105G6KRkc3f5KD3GuqKL0YmInFmooBty/IXH8vDM1BJq7KEg69NNASlq0z7xyGnxcYmlxFPg1QlpvNsWkHxeamsjmxKcFFBwDL7qilT6W4f2a9SXSF54KfABToUOcjtWIHHsGTPZB7o70rAowiIrTjtUDzxNMXiNf78laGJ3Gju9uQk= ceph-86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:34:40.943 INFO:teuthology.orchestra.run.smithi107.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-21T09:34:40.943 INFO:teuthology.orchestra.run.smithi107.stdout:Adding key to root@localhost authorized_keys... 2024-09-21T09:34:40.947 INFO:teuthology.orchestra.run.smithi107.stdout:Adding host smithi107... 2024-09-21T09:34:41.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:41 smithi107 ceph-mon[29382]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:34:42.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:42 smithi107 ceph-mon[29382]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi107", "addr": "172.21.15.107", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:34:43.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:43 smithi107 ceph-mon[29382]: Deploying cephadm binary to smithi107 2024-09-21T09:34:44.833 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout Added host 'smithi107' with addr '172.21.15.107' 2024-09-21T09:34:44.834 INFO:teuthology.orchestra.run.smithi107.stdout:Deploying unmanaged mon service... 2024-09-21T09:34:45.456 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:45 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' 2024-09-21T09:34:45.456 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:45 smithi107 ceph-mon[29382]: Added host smithi107 2024-09-21T09:34:45.456 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:45 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:34:46.178 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-21T09:34:46.178 INFO:teuthology.orchestra.run.smithi107.stdout:Deploying unmanaged mgr service... 2024-09-21T09:34:46.688 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:46 smithi107 ceph-mon[29382]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:34:46.688 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:46 smithi107 ceph-mon[29382]: Saving service mon spec with placement count:5 2024-09-21T09:34:46.688 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:46 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' 2024-09-21T09:34:48.223 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-21T09:34:48.342 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:48 smithi107 ceph-mon[29382]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:34:48.343 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:48 smithi107 ceph-mon[29382]: Saving service mgr spec with placement count:2 2024-09-21T09:34:48.343 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:48 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' 2024-09-21T09:34:48.343 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:48 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' 2024-09-21T09:34:49.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:49 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' 2024-09-21T09:34:49.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:49 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1373592922' entity='client.admin' 2024-09-21T09:34:50.880 INFO:teuthology.orchestra.run.smithi107.stdout:Enabling the dashboard module... 2024-09-21T09:34:51.390 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:51 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2403680042' entity='client.admin' 2024-09-21T09:34:51.390 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:51 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' 2024-09-21T09:34:52.499 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:52 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2495436022' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-21T09:34:52.500 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:52 smithi107 ceph-mon[29382]: from='mgr.14118 172.21.15.107:0/3058675632' entity='mgr.y' 2024-09-21T09:34:52.500 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:52 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ignoring --setuser ceph since I am not root 2024-09-21T09:34:52.500 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:52 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ignoring --setgroup ceph since I am not root 2024-09-21T09:34:52.500 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:52 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:52.379+0000 7f094597fec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:34:52.500 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:52 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:52.433+0000 7f094597fec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:34:52.500 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:52 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:52.497+0000 7f094597fec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:34:52.761 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:52 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:52.620+0000 7f094597fec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:34:52.762 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:52 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:52.759+0000 7f094597fec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:34:53.018 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:53 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:53.016+0000 7f094597fec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:34:53.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:53 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2495436022' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-21T09:34:53.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:53 smithi107 ceph-mon[29382]: mgrmap e9: y(active, since 17s) 2024-09-21T09:34:53.281 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:53 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:53.077+0000 7f094597fec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:34:53.281 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:53 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:53.241+0000 7f094597fec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:34:54.153 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:54 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:54.113+0000 7f094597fec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:34:54.456 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:54 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1387912080' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-21T09:34:54.456 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:54 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:54.180+0000 7f094597fec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:34:54.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:34:54.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-21T09:34:54.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T09:34:54.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-21T09:34:54.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T09:34:54.677 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:34:54.677 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for the mgr to restart... 2024-09-21T09:34:54.677 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mgr epoch 9... 2024-09-21T09:34:55.108 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:54 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:54.900+0000 7f094597fec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:34:55.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:55 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:55.013+0000 7f094597fec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:34:55.392 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:55 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:55.139+0000 7f094597fec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:34:55.392 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:55 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:55.197+0000 7f094597fec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:34:55.392 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:55 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:55.323+0000 7f094597fec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:34:55.392 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:55 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:55.390+0000 7f094597fec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:34:55.645 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:55 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:55.447+0000 7f094597fec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:34:55.645 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:55 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:55.580+0000 7f094597fec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:34:55.645 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:55 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:55.643+0000 7f094597fec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:34:55.902 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:55 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:55.707+0000 7f094597fec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:34:55.902 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:55 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:55.787+0000 7f094597fec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:34:55.902 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:55 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:55.844+0000 7f094597fec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:34:55.902 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:55 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:55.900+0000 7f094597fec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:34:57.108 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:56 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:56.641+0000 7f094597fec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:34:57.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:56 smithi107 ceph-mon[29382]: Active manager daemon y restarted 2024-09-21T09:34:57.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:56 smithi107 ceph-mon[29382]: Activating manager daemon y 2024-09-21T09:34:57.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:56 smithi107 ceph-mon[29382]: osdmap e3: 0 total, 0 up, 0 in 2024-09-21T09:34:57.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:56 smithi107 ceph-mon[29382]: mgrmap e10: y(active, starting, since 0.00306522s) 2024-09-21T09:34:57.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:56 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:34:57.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:56 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:34:57.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:56 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:34:57.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:56 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:34:57.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:56 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:34:57.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:56 smithi107 ceph-mon[29382]: Manager daemon y is now available 2024-09-21T09:34:57.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:56 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:34:57.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:56 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:34:57.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:56 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:34:58.335 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:34:58.335 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-21T09:34:58.336 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T09:34:58.336 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:34:58.336 INFO:teuthology.orchestra.run.smithi107.stdout:mgr epoch 9 is available 2024-09-21T09:34:58.336 INFO:teuthology.orchestra.run.smithi107.stdout:Generating a dashboard self-signed certificate... 2024-09-21T09:34:58.588 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:58.417+0000 7f08f64dd640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:34:58] ENGINE Error in HTTPServer.serve 2024-09-21T09:34:58.588 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: Traceback (most recent call last): 2024-09-21T09:34:58.588 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:34:58.588 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: self._connections.run(self.expiration_interval) 2024-09-21T09:34:58.588 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:34:58.588 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: self._run(expiration_interval) 2024-09-21T09:34:58.588 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:34:58.588 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:34:58.589 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:34:58.589 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:34:58.589 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:34:58.589 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: s = self.context.wrap_socket( 2024-09-21T09:34:58.589 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:34:58.589 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: return self.sslsocket_class._create( 2024-09-21T09:34:58.589 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:34:58.589 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: self.do_handshake() 2024-09-21T09:34:58.589 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:34:58.589 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: self._sslobj.do_handshake() 2024-09-21T09:34:58.589 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:34:58.589 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:34:58.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-mon[29382]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T09:34:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-mon[29382]: mgrmap e11: y(active, since 1.00562s) 2024-09-21T09:34:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-mon[29382]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T09:34:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:34:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:58 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:00.010 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:59 smithi107 ceph-mon[29382]: [21/Sep/2024:09:34:58] ENGINE Bus STARTING 2024-09-21T09:35:00.010 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:59 smithi107 ceph-mon[29382]: [21/Sep/2024:09:34:58] ENGINE Serving on https://172.21.15.107:7150 2024-09-21T09:35:00.010 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:59 smithi107 ceph-mon[29382]: [21/Sep/2024:09:34:58] ENGINE Bus STARTED 2024-09-21T09:35:00.010 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:59 smithi107 ceph-mon[29382]: [21/Sep/2024:09:34:58] ENGINE Error in HTTPServer.serve 2024-09-21T09:35:00.011 INFO:journalctl@ceph.mon.a.smithi107.stdout: Traceback (most recent call last): 2024-09-21T09:35:00.011 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:35:00.011 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:35:00.011 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:35:00.011 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._run(expiration_interval) 2024-09-21T09:35:00.011 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:35:00.011 INFO:journalctl@ceph.mon.a.smithi107.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:35:00.011 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:35:00.011 INFO:journalctl@ceph.mon.a.smithi107.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:35:00.011 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:35:00.011 INFO:journalctl@ceph.mon.a.smithi107.stdout: s = self.context.wrap_socket( 2024-09-21T09:35:00.011 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:35:00.011 INFO:journalctl@ceph.mon.a.smithi107.stdout: return self.sslsocket_class._create( 2024-09-21T09:35:00.011 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:35:00.012 INFO:journalctl@ceph.mon.a.smithi107.stdout: self.do_handshake() 2024-09-21T09:35:00.012 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:35:00.012 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._sslobj.do_handshake() 2024-09-21T09:35:00.012 INFO:journalctl@ceph.mon.a.smithi107.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:35:00.012 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:59 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:00.012 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:59 smithi107 ceph-mon[29382]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:35:00.012 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:59 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:00.012 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:59 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:00.012 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:34:59 smithi107 ceph-mon[29382]: mgrmap e12: y(active, since 2s) 2024-09-21T09:35:00.698 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-21T09:35:00.698 INFO:teuthology.orchestra.run.smithi107.stdout:Creating initial admin user... 2024-09-21T09:35:02.162 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$6ve2VWxZHf9m58ZeO5BWj.Ec/OHmkVTbEhXr4ccccFMyxOm4Lrw4C", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726911301, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-21T09:35:02.163 INFO:teuthology.orchestra.run.smithi107.stdout:Fetching dashboard port number... 2024-09-21T09:35:02.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:02 smithi107 ceph-mon[29382]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:35:02.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:02 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:03.319 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 8443 2024-09-21T09:35:03.329 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T09:35:03.329 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout disabled 2024-09-21T09:35:03.336 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T09:35:03.336 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-09-21T09:35:03.336 INFO:teuthology.orchestra.run.smithi107.stdout:firewalld.service is not enabled 2024-09-21T09:35:03.336 INFO:teuthology.orchestra.run.smithi107.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-21T09:35:03.338 INFO:teuthology.orchestra.run.smithi107.stdout:Ceph Dashboard is now available at: 2024-09-21T09:35:03.338 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:35:03.338 INFO:teuthology.orchestra.run.smithi107.stdout: URL: https://smithi107.front.sepia.ceph.com:8443/ 2024-09-21T09:35:03.339 INFO:teuthology.orchestra.run.smithi107.stdout: User: admin 2024-09-21T09:35:03.339 INFO:teuthology.orchestra.run.smithi107.stdout: Password: d28mo805ie 2024-09-21T09:35:03.339 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:35:03.339 INFO:teuthology.orchestra.run.smithi107.stdout:Saving cluster configuration to /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config directory 2024-09-21T09:35:03.339 INFO:teuthology.orchestra.run.smithi107.stdout:Enabling autotune for osd_memory_target 2024-09-21T09:35:03.604 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:03 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3279168934' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-21T09:35:04.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:04 smithi107 ceph-mon[29382]: mgrmap e13: y(active, since 6s) 2024-09-21T09:35:05.807 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-21T09:35:05.807 INFO:teuthology.orchestra.run.smithi107.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-21T09:35:05.807 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:35:05.807 INFO:teuthology.orchestra.run.smithi107.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-21T09:35:05.808 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:35:05.808 INFO:teuthology.orchestra.run.smithi107.stdout:Or, if you are only running a single cluster on this host: 2024-09-21T09:35:05.808 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:35:05.808 INFO:teuthology.orchestra.run.smithi107.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-21T09:35:05.808 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:35:05.808 INFO:teuthology.orchestra.run.smithi107.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-21T09:35:05.808 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:35:05.808 INFO:teuthology.orchestra.run.smithi107.stdout: ceph telemetry on 2024-09-21T09:35:05.808 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:35:05.808 INFO:teuthology.orchestra.run.smithi107.stdout:For more information see: 2024-09-21T09:35:05.808 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:35:05.808 INFO:teuthology.orchestra.run.smithi107.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-09-21T09:35:05.808 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:35:05.809 INFO:teuthology.orchestra.run.smithi107.stdout:Bootstrap complete. 2024-09-21T09:35:05.842 INFO:tasks.cephadm:Fetching config... 2024-09-21T09:35:05.842 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T09:35:05.842 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-21T09:35:05.858 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-21T09:35:05.858 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T09:35:05.858 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-21T09:35:05.915 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-21T09:35:05.915 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T09:35:05.915 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/keyring of=/dev/stdout 2024-09-21T09:35:05.979 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-21T09:35:05.979 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T09:35:05.979 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-21T09:35:06.035 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-21T09:35:06.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCd2RbKBwQPD27iss2rrkfOfOs+mNazD9XXjrOQLK+9F1l5gzSMdOlUB9rBaiM6NAhhG44wgblVIU31+LtJQQQCS6XmyRa8hRG0Q3XZkuXb/JJ8OsZmldgFXkGRhuir/PUwqvWrABlZvNU9EzyFRVI44u9CcadmMJ4CkCLPfbHWAFo0WNzQEy2Rr524Byve+j4/AEzRMqZbvVtqhnhDWN5jG1d+GdaigAg3G/x3HGtOKlul1AQhUvI2mHKh5nR72/7qKO0/dN0YXQ3L3Fi+png1jK18AfDGbZCib2Dp3pOkgctog7WqaRrpDx4UejdzEPcd28q3Fma2R/BnyCby105G6KRkc3f5KD3GuqKL0YmInFmooBty/IXH8vDM1BJq7KEg69NNASlq0z7xyGnxcYmlxFPg1QlpvNsWkHxeamsjmxKcFFBwDL7qilT6W4f2a9SXSF54KfABToUOcjtWIHHsGTPZB7o70rAowiIrTjtUDzxNMXiNf78laGJ3Gju9uQk= ceph-86c49514-77fc-11ef-baf6-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T09:35:06.109 INFO:teuthology.orchestra.run.smithi107.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCd2RbKBwQPD27iss2rrkfOfOs+mNazD9XXjrOQLK+9F1l5gzSMdOlUB9rBaiM6NAhhG44wgblVIU31+LtJQQQCS6XmyRa8hRG0Q3XZkuXb/JJ8OsZmldgFXkGRhuir/PUwqvWrABlZvNU9EzyFRVI44u9CcadmMJ4CkCLPfbHWAFo0WNzQEy2Rr524Byve+j4/AEzRMqZbvVtqhnhDWN5jG1d+GdaigAg3G/x3HGtOKlul1AQhUvI2mHKh5nR72/7qKO0/dN0YXQ3L3Fi+png1jK18AfDGbZCib2Dp3pOkgctog7WqaRrpDx4UejdzEPcd28q3Fma2R/BnyCby105G6KRkc3f5KD3GuqKL0YmInFmooBty/IXH8vDM1BJq7KEg69NNASlq0z7xyGnxcYmlxFPg1QlpvNsWkHxeamsjmxKcFFBwDL7qilT6W4f2a9SXSF54KfABToUOcjtWIHHsGTPZB7o70rAowiIrTjtUDzxNMXiNf78laGJ3Gju9uQk= ceph-86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:35:06.121 DEBUG:teuthology.orchestra.run.smithi129:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCd2RbKBwQPD27iss2rrkfOfOs+mNazD9XXjrOQLK+9F1l5gzSMdOlUB9rBaiM6NAhhG44wgblVIU31+LtJQQQCS6XmyRa8hRG0Q3XZkuXb/JJ8OsZmldgFXkGRhuir/PUwqvWrABlZvNU9EzyFRVI44u9CcadmMJ4CkCLPfbHWAFo0WNzQEy2Rr524Byve+j4/AEzRMqZbvVtqhnhDWN5jG1d+GdaigAg3G/x3HGtOKlul1AQhUvI2mHKh5nR72/7qKO0/dN0YXQ3L3Fi+png1jK18AfDGbZCib2Dp3pOkgctog7WqaRrpDx4UejdzEPcd28q3Fma2R/BnyCby105G6KRkc3f5KD3GuqKL0YmInFmooBty/IXH8vDM1BJq7KEg69NNASlq0z7xyGnxcYmlxFPg1QlpvNsWkHxeamsjmxKcFFBwDL7qilT6W4f2a9SXSF54KfABToUOcjtWIHHsGTPZB7o70rAowiIrTjtUDzxNMXiNf78laGJ3Gju9uQk= ceph-86c49514-77fc-11ef-baf6-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T09:35:06.174 INFO:teuthology.orchestra.run.smithi129.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCd2RbKBwQPD27iss2rrkfOfOs+mNazD9XXjrOQLK+9F1l5gzSMdOlUB9rBaiM6NAhhG44wgblVIU31+LtJQQQCS6XmyRa8hRG0Q3XZkuXb/JJ8OsZmldgFXkGRhuir/PUwqvWrABlZvNU9EzyFRVI44u9CcadmMJ4CkCLPfbHWAFo0WNzQEy2Rr524Byve+j4/AEzRMqZbvVtqhnhDWN5jG1d+GdaigAg3G/x3HGtOKlul1AQhUvI2mHKh5nR72/7qKO0/dN0YXQ3L3Fi+png1jK18AfDGbZCib2Dp3pOkgctog7WqaRrpDx4UejdzEPcd28q3Fma2R/BnyCby105G6KRkc3f5KD3GuqKL0YmInFmooBty/IXH8vDM1BJq7KEg69NNASlq0z7xyGnxcYmlxFPg1QlpvNsWkHxeamsjmxKcFFBwDL7qilT6W4f2a9SXSF54KfABToUOcjtWIHHsGTPZB7o70rAowiIrTjtUDzxNMXiNf78laGJ3Gju9uQk= ceph-86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:35:06.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-21T09:35:06.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:06 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2782021886' entity='client.admin' 2024-09-21T09:35:06.429 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:35:08.965 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:08 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:08.965 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:08 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:08.965 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:08 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:35:08.966 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:08 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:08.966 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:08 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:35:08.966 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:08 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:10.306 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-21T09:35:10.306 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-21T09:35:10.550 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:35:10.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:10 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/439992343' entity='client.admin' 2024-09-21T09:35:10.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:10 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:35:10.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:10 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:12.863 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi129 2024-09-21T09:35:12.863 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T09:35:12.863 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/etc/ceph/ceph.conf 2024-09-21T09:35:12.884 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T09:35:12.884 DEBUG:teuthology.orchestra.run.smithi129:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:35:12.941 INFO:tasks.cephadm:Adding host smithi129 to orchestrator... 2024-09-21T09:35:12.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph orch host add smithi129 2024-09-21T09:35:13.190 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:35:13.415 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:13 smithi107 ceph-mon[29382]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:35:13.415 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:13.415 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:35:13.415 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:13.415 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:35:13.415 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:13 smithi107 ceph-mon[29382]: Updating smithi107:/etc/ceph/ceph.conf 2024-09-21T09:35:13.415 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:13.415 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:13.415 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:14.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:14 smithi107 ceph-mon[29382]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:35:14.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:14 smithi107 ceph-mon[29382]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:35:14.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:14 smithi107 ceph-mon[29382]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:35:15.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:15 smithi107 ceph-mon[29382]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi129", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:35:16.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:16 smithi107 ceph-mon[29382]: Deploying cephadm binary to smithi129 2024-09-21T09:35:17.549 INFO:teuthology.orchestra.run.smithi107.stdout:Added host 'smithi129' with addr '172.21.15.129' 2024-09-21T09:35:17.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:17 smithi107 ceph-mon[29382]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:17.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:17 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:17.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:17 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:35:18.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph orch host ls --format=json 2024-09-21T09:35:18.377 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:35:19.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:18 smithi107 ceph-mon[29382]: Added host smithi129 2024-09-21T09:35:19.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:18 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:19.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:18 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:20.099 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:35:20.099 INFO:teuthology.orchestra.run.smithi107.stdout:[{"addr": "172.21.15.107", "hostname": "smithi107", "labels": [], "status": ""}, {"addr": "172.21.15.129", "hostname": "smithi129", "labels": [], "status": ""}] 2024-09-21T09:35:20.100 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:19 smithi107 ceph-mon[29382]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:20.100 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:19 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:20.696 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-21T09:35:20.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd crush tunables default 2024-09-21T09:35:20.936 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:35:21.062 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:20 smithi107 ceph-mon[29382]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:35:22.070 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:21 smithi107 ceph-mon[29382]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:22.070 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:21 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:22.070 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:21 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:22.070 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:21 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:22.070 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:21 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:22.070 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:21 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi129", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:35:22.070 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:21 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:22.070 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:21 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:35:23.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:22 smithi107 ceph-mon[29382]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T09:35:23.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:22 smithi107 ceph-mon[29382]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:35:23.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:22 smithi107 ceph-mon[29382]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:35:23.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:22 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:23.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:22 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:23.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:22 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:23.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:22 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1428770317' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-21T09:35:23.515 INFO:teuthology.orchestra.run.smithi107.stderr:adjusted tunables profile to default 2024-09-21T09:35:24.085 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:23 smithi107 ceph-mon[29382]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:35:24.085 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:23 smithi107 ceph-mon[29382]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:24.085 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:23 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1428770317' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T09:35:24.086 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:23 smithi107 ceph-mon[29382]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:35:24.115 INFO:tasks.cephadm:Adding mon.a on smithi107 2024-09-21T09:35:24.115 INFO:tasks.cephadm:Adding mon.c on smithi107 2024-09-21T09:35:24.115 INFO:tasks.cephadm:Adding mon.b on smithi129 2024-09-21T09:35:24.115 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph orch apply mon '3;smithi107:172.21.15.107=a;smithi107:[v2:172.21.15.107:3301,v1:172.21.15.107:6790]=c;smithi129:172.21.15.129=b' 2024-09-21T09:35:24.355 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:35:25.951 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled mon update... 2024-09-21T09:35:26.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:25 smithi107 ceph-mon[29382]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:26.495 DEBUG:teuthology.orchestra.run.smithi107:mon.c> sudo journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.c.service 2024-09-21T09:35:26.498 DEBUG:teuthology.orchestra.run.smithi129:mon.b> sudo journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.b.service 2024-09-21T09:35:26.501 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T09:35:26.501 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T09:35:27.200 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:35:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:26 smithi107 ceph-mon[29382]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi107:172.21.15.107=a;smithi107:[v2:172.21.15.107:3301,v1:172.21.15.107:6790]=c;smithi129:172.21.15.129=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:35:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:26 smithi107 ceph-mon[29382]: Saving service mon spec with placement smithi107:172.21.15.107=a;smithi107:[v2:172.21.15.107:3301,v1:172.21.15.107:6790]=c;smithi129:172.21.15.129=b;count:3 2024-09-21T09:35:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:26 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:26 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:35:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:26 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:26 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:35:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:26 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:26 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:35:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:26 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:27.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:26 smithi107 ceph-mon[29382]: Deploying daemon mon.b on smithi129 2024-09-21T09:35:28.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:27 smithi107 ceph-mon[29382]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:29.121 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:35:29.121 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":1,"fsid":"86c49514-77fc-11ef-baf6-efdc52797490","modified":"2024-09-21T09:34:00.747992Z","created":"2024-09-21T09:34:00.747992Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T09:35:29.121 INFO:teuthology.orchestra.run.smithi129.stderr:dumped monmap epoch 1 2024-09-21T09:35:30.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:29 smithi107 ceph-mon[29382]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:30.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:29 smithi107 ceph-mon[29382]: from='client.? 172.21.15.129:0/830271865' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:35:31.294 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 podman[32692]: 2024-09-21 09:35:31.091191434 +0000 UTC m=+0.015568846 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T09:35:31.607 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T09:35:31.608 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T09:35:31.855 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:35:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 podman[32692]: 2024-09-21 09:35:31.430881863 +0000 UTC m=+0.355259257 container init 589a4d4f8d6b5cf8cc4f00d3c6e5428d27a4737214535881eda37d7d74850081 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-21T09:35:31.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 podman[32692]: 2024-09-21 09:35:31.436188929 +0000 UTC m=+0.360566323 container start 589a4d4f8d6b5cf8cc4f00d3c6e5428d27a4737214535881eda37d7d74850081 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-fbd1a6a, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-21T09:35:31.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T09:35:31.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable), process ceph-mon, pid 7 2024-09-21T09:35:31.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: pidfile_write: ignore empty --pid-file 2024-09-21T09:35:31.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: load: jerasure load: lrc 2024-09-21T09:35:31.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: RocksDB version: 6.15.5 2024-09-21T09:35:31.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-21T09:35:31.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Compile date Sep 20 2024 2024-09-21T09:35:31.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: DB SUMMARY 2024-09-21T09:35:31.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: DB Session ID: 4I0C24ADCB1RRHREHU70 2024-09-21T09:35:31.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: CURRENT file: CURRENT 2024-09-21T09:35:31.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T09:35:31.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-09-21T09:35:31.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-21T09:35:31.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-09-21T09:35:31.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.error_if_exists: 0 2024-09-21T09:35:31.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.create_if_missing: 0 2024-09-21T09:35:31.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.paranoid_checks: 1 2024-09-21T09:35:31.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T09:35:31.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.env: 0x55b52700a440 2024-09-21T09:35:31.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.fs: Posix File System 2024-09-21T09:35:31.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.info_log: 0x55b527f02ec0 2024-09-21T09:35:31.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T09:35:31.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.statistics: (nil) 2024-09-21T09:35:31.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.use_fsync: 0 2024-09-21T09:35:31.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_log_file_size: 0 2024-09-21T09:35:31.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T09:35:31.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T09:35:31.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T09:35:31.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T09:35:31.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.allow_fallocate: 1 2024-09-21T09:35:31.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T09:35:31.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T09:35:31.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.use_direct_reads: 0 2024-09-21T09:35:31.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T09:35:31.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T09:35:31.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.db_log_dir: 2024-09-21T09:35:31.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-09-21T09:35:31.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T09:35:31.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T09:35:31.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T09:35:31.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T09:35:31.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T09:35:31.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T09:35:31.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T09:35:31.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T09:35:31.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.write_buffer_manager: 0x55b527fc0780 2024-09-21T09:35:31.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T09:35:31.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-21T09:35:31.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T09:35:31.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T09:35:31.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T09:35:31.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T09:35:31.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T09:35:31.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T09:35:31.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T09:35:31.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.unordered_write: 0 2024-09-21T09:35:31.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T09:35:31.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T09:35:31.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T09:35:31.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T09:35:31.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.row_cache: None 2024-09-21T09:35:31.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.wal_filter: None 2024-09-21T09:35:31.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T09:35:31.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T09:35:31.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.preserve_deletes: 0 2024-09-21T09:35:31.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.two_write_queues: 0 2024-09-21T09:35:31.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T09:35:31.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.atomic_flush: 0 2024-09-21T09:35:31.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T09:35:31.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T09:35:31.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T09:35:31.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.log_readahead_size: 0 2024-09-21T09:35:31.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T09:35:31.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T09:35:31.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T09:35:31.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T09:35:31.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T09:35:31.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T09:35:31.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_background_jobs: 2 2024-09-21T09:35:31.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_background_compactions: -1 2024-09-21T09:35:31.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_subcompactions: 1 2024-09-21T09:35:31.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T09:35:31.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T09:35:31.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T09:35:31.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T09:35:31.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T09:35:31.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T09:35:31.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T09:35:31.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T09:35:31.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_open_files: -1 2024-09-21T09:35:31.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T09:35:31.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T09:35:31.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T09:35:31.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T09:35:31.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_background_flushes: -1 2024-09-21T09:35:31.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Compression algorithms supported: 2024-09-21T09:35:31.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: kZSTD supported: 0 2024-09-21T09:35:31.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: kXpressCompression supported: 0 2024-09-21T09:35:31.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: kBZip2Compression supported: 0 2024-09-21T09:35:31.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T09:35:31.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: kLZ4Compression supported: 1 2024-09-21T09:35:31.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: kZlibCompression supported: 1 2024-09-21T09:35:31.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T09:35:31.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: kSnappyCompression supported: 1 2024-09-21T09:35:31.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T09:35:31.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-09-21T09:35:31.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-21T09:35:31.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T09:35:31.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.merge_operator: 2024-09-21T09:35:31.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compaction_filter: None 2024-09-21T09:35:31.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T09:35:31.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T09:35:31.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T09:35:31.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T09:35:31.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b527ea9a38) 2024-09-21T09:35:31.931 INFO:journalctl@ceph.mon.b.smithi129.stdout: cache_index_and_filter_blocks: 1 2024-09-21T09:35:31.931 INFO:journalctl@ceph.mon.b.smithi129.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T09:35:31.931 INFO:journalctl@ceph.mon.b.smithi129.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T09:35:31.931 INFO:journalctl@ceph.mon.b.smithi129.stdout: pin_top_level_index_and_filter: 1 2024-09-21T09:35:31.931 INFO:journalctl@ceph.mon.b.smithi129.stdout: index_type: 0 2024-09-21T09:35:31.931 INFO:journalctl@ceph.mon.b.smithi129.stdout: data_block_index_type: 0 2024-09-21T09:35:31.931 INFO:journalctl@ceph.mon.b.smithi129.stdout: index_shortening: 1 2024-09-21T09:35:31.931 INFO:journalctl@ceph.mon.b.smithi129.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T09:35:31.931 INFO:journalctl@ceph.mon.b.smithi129.stdout: hash_index_allow_collision: 1 2024-09-21T09:35:31.931 INFO:journalctl@ceph.mon.b.smithi129.stdout: checksum: 1 2024-09-21T09:35:31.931 INFO:journalctl@ceph.mon.b.smithi129.stdout: no_block_cache: 0 2024-09-21T09:35:31.931 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_cache: 0x55b527edf1f0 2024-09-21T09:35:31.931 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_cache_name: BinnedLRUCache 2024-09-21T09:35:31.931 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_cache_options: 2024-09-21T09:35:31.932 INFO:journalctl@ceph.mon.b.smithi129.stdout: capacity : 536870912 2024-09-21T09:35:31.932 INFO:journalctl@ceph.mon.b.smithi129.stdout: num_shard_bits : 4 2024-09-21T09:35:31.932 INFO:journalctl@ceph.mon.b.smithi129.stdout: strict_capacity_limit : 0 2024-09-21T09:35:31.932 INFO:journalctl@ceph.mon.b.smithi129.stdout: high_pri_pool_ratio: 0.000 2024-09-21T09:35:31.932 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_cache_compressed: (nil) 2024-09-21T09:35:31.932 INFO:journalctl@ceph.mon.b.smithi129.stdout: persistent_cache: (nil) 2024-09-21T09:35:31.932 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_size: 4096 2024-09-21T09:35:31.932 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_size_deviation: 10 2024-09-21T09:35:31.932 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_restart_interval: 16 2024-09-21T09:35:31.932 INFO:journalctl@ceph.mon.b.smithi129.stdout: index_block_restart_interval: 1 2024-09-21T09:35:31.932 INFO:journalctl@ceph.mon.b.smithi129.stdout: metadata_block_size: 4096 2024-09-21T09:35:31.932 INFO:journalctl@ceph.mon.b.smithi129.stdout: partition_filters: 0 2024-09-21T09:35:31.932 INFO:journalctl@ceph.mon.b.smithi129.stdout: use_delta_encoding: 1 2024-09-21T09:35:31.932 INFO:journalctl@ceph.mon.b.smithi129.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-21T09:35:31.933 INFO:journalctl@ceph.mon.b.smithi129.stdout: whole_key_filtering: 1 2024-09-21T09:35:31.933 INFO:journalctl@ceph.mon.b.smithi129.stdout: verify_compression: 0 2024-09-21T09:35:31.933 INFO:journalctl@ceph.mon.b.smithi129.stdout: read_amp_bytes_per_bit: 0 2024-09-21T09:35:31.933 INFO:journalctl@ceph.mon.b.smithi129.stdout: format_version: 4 2024-09-21T09:35:31.933 INFO:journalctl@ceph.mon.b.smithi129.stdout: enable_index_compression: 1 2024-09-21T09:35:31.933 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_align: 0 2024-09-21T09:35:31.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T09:35:31.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T09:35:31.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compression: NoCompression 2024-09-21T09:35:31.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T09:35:31.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T09:35:31.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T09:35:31.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.num_levels: 7 2024-09-21T09:35:31.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T09:35:31.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T09:35:31.934 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T09:35:31.934 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T09:35:31.934 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T09:35:31.934 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T09:35:31.934 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T09:35:31.934 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:35:31.934 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T09:35:31.934 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T09:35:31.934 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T09:35:31.935 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T09:35:31.935 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T09:35:31.935 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T09:35:31.935 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:35:31.935 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T09:35:31.935 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T09:35:31.935 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T09:35:31.935 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T09:35:31.935 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T09:35:31.935 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T09:35:31.935 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T09:35:31.935 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T09:35:31.935 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T09:35:31.935 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T09:35:31.935 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T09:35:31.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T09:35:31.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T09:35:31.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T09:35:31.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T09:35:31.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T09:35:31.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T09:35:31.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T09:35:31.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T09:35:31.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.arena_block_size: 4194304 2024-09-21T09:35:31.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T09:35:31.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T09:35:31.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-21T09:35:31.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T09:35:31.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T09:35:31.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T09:35:31.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T09:35:31.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T09:35:31.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T09:35:31.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T09:35:31.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T09:35:31.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T09:35:31.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T09:35:31.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T09:35:31.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T09:35:31.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.inplace_update_support: 0 2024-09-21T09:35:31.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T09:35:31.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T09:35:31.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T09:35:31.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T09:35:31.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.bloom_locality: 0 2024-09-21T09:35:31.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.max_successive_merges: 0 2024-09-21T09:35:31.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T09:35:31.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T09:35:31.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T09:35:31.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T09:35:31.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.ttl: 2592000 2024-09-21T09:35:31.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T09:35:31.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.enable_blob_files: false 2024-09-21T09:35:31.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.min_blob_size: 0 2024-09-21T09:35:31.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T09:35:31.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T09:35:31.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T09:35:31.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T09:35:31.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T09:35:31.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-09-21T09:35:31.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-09-21T09:35:31.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726911331460460, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T09:35:31.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-09-21T09:35:31.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-21T09:35:31.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726911331460762, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1571, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726911331, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "ce344556-834e-436c-bce9-a300bc6eb641", "db_session_id": "4I0C24ADCB1RRHREHU70"}} 2024-09-21T09:35:31.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-09-21T09:35:31.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726911331461076, "job": 1, "event": "recovery_finished"} 2024-09-21T09:35:31.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T09:35:31.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55b527f12700 2024-09-21T09:35:31.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: DB pointer 0x55b527ee2000 2024-09-21T09:35:31.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-21T09:35:31.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-21T09:35:31.940 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** DB Stats ** 2024-09-21T09:35:31.940 INFO:journalctl@ceph.mon.b.smithi129.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:35:31.940 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T09:35:31.940 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:35:31.940 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:35:31.940 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T09:35:31.940 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-21T09:35:31.940 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:35:31.940 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T09:35:31.940 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** Compaction Stats [default] ** 2024-09-21T09:35:31.941 INFO:journalctl@ceph.mon.b.smithi129.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:35:31.941 INFO:journalctl@ceph.mon.b.smithi129.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:35:31.941 INFO:journalctl@ceph.mon.b.smithi129.stdout: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:35:31.941 INFO:journalctl@ceph.mon.b.smithi129.stdout: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:35:31.941 INFO:journalctl@ceph.mon.b.smithi129.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:35:31.941 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T09:35:31.941 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** Compaction Stats [default] ** 2024-09-21T09:35:31.941 INFO:journalctl@ceph.mon.b.smithi129.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:35:31.941 INFO:journalctl@ceph.mon.b.smithi129.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:35:31.941 INFO:journalctl@ceph.mon.b.smithi129.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:35:31.941 INFO:journalctl@ceph.mon.b.smithi129.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:35:31.941 INFO:journalctl@ceph.mon.b.smithi129.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:35:31.941 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:35:31.941 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:35:31.941 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:35:31.942 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:35:31.942 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:35:31.942 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:35:31.942 INFO:journalctl@ceph.mon.b.smithi129.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:35:31.942 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T09:35:31.942 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:35:31.942 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T09:35:31.942 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** Compaction Stats [default] ** 2024-09-21T09:35:31.942 INFO:journalctl@ceph.mon.b.smithi129.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:35:31.942 INFO:journalctl@ceph.mon.b.smithi129.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:35:31.942 INFO:journalctl@ceph.mon.b.smithi129.stdout: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:35:31.942 INFO:journalctl@ceph.mon.b.smithi129.stdout: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:35:31.942 INFO:journalctl@ceph.mon.b.smithi129.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-21T09:35:31.943 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T09:35:31.943 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** Compaction Stats [default] ** 2024-09-21T09:35:31.943 INFO:journalctl@ceph.mon.b.smithi129.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:35:31.943 INFO:journalctl@ceph.mon.b.smithi129.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:35:31.943 INFO:journalctl@ceph.mon.b.smithi129.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:35:31.943 INFO:journalctl@ceph.mon.b.smithi129.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:35:31.943 INFO:journalctl@ceph.mon.b.smithi129.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:35:31.943 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:35:31.943 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:35:31.943 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:35:31.943 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:35:31.943 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:35:31.943 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:35:31.943 INFO:journalctl@ceph.mon.b.smithi129.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:35:31.944 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T09:35:31.944 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:35:31.944 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-21T09:35:31.944 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: using public_addr v2:172.21.15.129:0/0 -> [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] 2024-09-21T09:35:31.944 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: starting mon.b rank -1 at public addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] at bind addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:35:31.944 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: mon.b@-1(???) e0 preinit fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:35:31.944 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: mon.b@-1(synchronizing).mds e1 new map 2024-09-21T09:35:31.944 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-21T09:35:31.944 INFO:journalctl@ceph.mon.b.smithi129.stdout: e1 2024-09-21T09:35:31.944 INFO:journalctl@ceph.mon.b.smithi129.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T09:35:31.944 INFO:journalctl@ceph.mon.b.smithi129.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:35:31.944 INFO:journalctl@ceph.mon.b.smithi129.stdout: legacy client fscid: -1 2024-09-21T09:35:31.944 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T09:35:31.945 INFO:journalctl@ceph.mon.b.smithi129.stdout: No filesystems configured 2024-09-21T09:35:31.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T09:35:31.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T09:35:31.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T09:35:31.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T09:35:31.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T09:35:31.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T09:35:31.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T09:35:31.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:35:31.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:35:31.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:35:31.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:31.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:31.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:31.945 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:31.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:31.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi129", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:35:31.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:31.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:35:31.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T09:35:31.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:35:31.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:35:31.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:31.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:31.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:31.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1428770317' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-21T09:35:31.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:35:31.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:31.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1428770317' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T09:35:31.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:35:31.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:31.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi107:172.21.15.107=a;smithi107:[v2:172.21.15.107:3301,v1:172.21.15.107:6790]=c;smithi129:172.21.15.129=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:35:31.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: Saving service mon spec with placement smithi107:172.21.15.107=a;smithi107:[v2:172.21.15.107:3301,v1:172.21.15.107:6790]=c;smithi129:172.21.15.129=b;count:3 2024-09-21T09:35:31.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:31.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:35:31.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:31.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:35:31.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:31.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:35:31.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:31.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: Deploying daemon mon.b on smithi129 2024-09-21T09:35:31.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:31.948 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:31.948 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: from='client.? 172.21.15.129:0/830271865' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:35:31.948 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 ceph-mon[32729]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-21T09:35:31.948 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 bash[32692]: 589a4d4f8d6b5cf8cc4f00d3c6e5428d27a4737214535881eda37d7d74850081 2024-09-21T09:35:31.948 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:31 smithi129 systemd[1]: Started Ceph mon.b for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:35:36.737 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:35:36.737 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":2,"fsid":"86c49514-77fc-11ef-baf6-efdc52797490","modified":"2024-09-21T09:35:31.485332Z","created":"2024-09-21T09:34:00.747992Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:3300","nonce":0},{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T09:35:36.737 INFO:teuthology.orchestra.run.smithi129.stderr:dumped monmap epoch 2 2024-09-21T09:35:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:35:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: mon.a calling monitor election 2024-09-21T09:35:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: mon.b calling monitor election 2024-09-21T09:35:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:36.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T09:35:36.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: monmap e2: 2 mons at {a=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:35:36.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: fsmap 2024-09-21T09:35:36.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:35:36.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: mgrmap e13: y(active, since 39s) 2024-09-21T09:35:36.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: overall HEALTH_OK 2024-09-21T09:35:36.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:36.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:36.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:36.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:35:36.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:36 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:35:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: mon.a calling monitor election 2024-09-21T09:35:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: mon.b calling monitor election 2024-09-21T09:35:36.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:36.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:36.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:36.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:36.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T09:35:36.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: monmap e2: 2 mons at {a=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:35:36.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: fsmap 2024-09-21T09:35:36.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:35:36.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: mgrmap e13: y(active, since 39s) 2024-09-21T09:35:36.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: overall HEALTH_OK 2024-09-21T09:35:36.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:36.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:36.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:36.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:35:36.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:36 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:37.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:37 smithi107 ceph-mon[29382]: Deploying daemon mon.c on smithi107 2024-09-21T09:35:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:37 smithi107 ceph-mon[29382]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:37 smithi107 ceph-mon[29382]: from='client.? 172.21.15.129:0/43827006' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:35:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:37 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:37 smithi129 ceph-mon[32729]: Deploying daemon mon.c on smithi107 2024-09-21T09:35:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:37 smithi129 ceph-mon[32729]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:37 smithi129 ceph-mon[32729]: from='client.? 172.21.15.129:0/43827006' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:35:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:37 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:38.293 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T09:35:38.293 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T09:35:38.550 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:35:39.697 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 podman[41147]: 2024-09-21 09:35:39.597463276 +0000 UTC m=+0.130679973 container create 4d41a86eb0a87952e40ca59b886a0acb256292d85a3655133e60b4a0b8fca615 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-c, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-fbd1a6a, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-21T09:35:39.870 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:39 smithi129 ceph-mon[32729]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:40.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[29382]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:40.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 podman[41147]: 2024-09-21 09:35:39.799797409 +0000 UTC m=+0.333014061 container init 4d41a86eb0a87952e40ca59b886a0acb256292d85a3655133e60b4a0b8fca615 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-c, RELEASE=quincy-fbd1a6a, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-21T09:35:40.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 podman[41147]: 2024-09-21 09:35:39.804296172 +0000 UTC m=+0.337512830 container start 4d41a86eb0a87952e40ca59b886a0acb256292d85a3655133e60b4a0b8fca615 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=quincy-fbd1a6a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T09:35:40.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T09:35:40.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable), process ceph-mon, pid 7 2024-09-21T09:35:40.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: pidfile_write: ignore empty --pid-file 2024-09-21T09:35:40.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: load: jerasure load: lrc 2024-09-21T09:35:40.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: RocksDB version: 6.15.5 2024-09-21T09:35:40.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-21T09:35:40.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Compile date Sep 20 2024 2024-09-21T09:35:40.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: DB SUMMARY 2024-09-21T09:35:40.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: DB Session ID: FJ1OGH83JU3HUHPEC9BI 2024-09-21T09:35:40.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: CURRENT file: CURRENT 2024-09-21T09:35:40.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T09:35:40.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-09-21T09:35:40.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-21T09:35:40.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-21T09:35:40.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.error_if_exists: 0 2024-09-21T09:35:40.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.create_if_missing: 0 2024-09-21T09:35:40.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.paranoid_checks: 1 2024-09-21T09:35:40.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T09:35:40.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.env: 0x55c168699440 2024-09-21T09:35:40.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.fs: Posix File System 2024-09-21T09:35:40.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.info_log: 0x55c16909aec0 2024-09-21T09:35:40.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T09:35:40.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.statistics: (nil) 2024-09-21T09:35:40.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.use_fsync: 0 2024-09-21T09:35:40.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_log_file_size: 0 2024-09-21T09:35:40.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T09:35:40.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T09:35:40.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T09:35:40.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T09:35:40.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.allow_fallocate: 1 2024-09-21T09:35:40.113 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T09:35:40.113 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T09:35:40.113 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.use_direct_reads: 0 2024-09-21T09:35:40.113 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T09:35:40.113 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T09:35:40.113 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.db_log_dir: 2024-09-21T09:35:40.114 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-09-21T09:35:40.114 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T09:35:40.114 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T09:35:40.114 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T09:35:40.114 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T09:35:40.115 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T09:35:40.115 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T09:35:40.115 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T09:35:40.115 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T09:35:40.115 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.write_buffer_manager: 0x55c1691587b0 2024-09-21T09:35:40.115 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T09:35:40.116 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-21T09:35:40.116 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T09:35:40.116 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T09:35:40.116 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T09:35:40.116 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T09:35:40.116 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T09:35:40.117 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T09:35:40.117 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T09:35:40.117 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.unordered_write: 0 2024-09-21T09:35:40.117 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T09:35:40.117 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T09:35:40.117 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T09:35:40.118 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T09:35:40.118 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.row_cache: None 2024-09-21T09:35:40.118 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.wal_filter: None 2024-09-21T09:35:40.118 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T09:35:40.120 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T09:35:40.120 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.preserve_deletes: 0 2024-09-21T09:35:40.120 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.two_write_queues: 0 2024-09-21T09:35:40.120 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T09:35:40.120 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.atomic_flush: 0 2024-09-21T09:35:40.120 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T09:35:40.120 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T09:35:40.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T09:35:40.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.log_readahead_size: 0 2024-09-21T09:35:40.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T09:35:40.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T09:35:40.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T09:35:40.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T09:35:40.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T09:35:40.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T09:35:40.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_background_jobs: 2 2024-09-21T09:35:40.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_background_compactions: -1 2024-09-21T09:35:40.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_subcompactions: 1 2024-09-21T09:35:40.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T09:35:40.121 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T09:35:40.122 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T09:35:40.122 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T09:35:40.122 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T09:35:40.122 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T09:35:40.122 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T09:35:40.122 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T09:35:40.122 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_open_files: -1 2024-09-21T09:35:40.122 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T09:35:40.122 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T09:35:40.122 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T09:35:40.122 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T09:35:40.122 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_background_flushes: -1 2024-09-21T09:35:40.123 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Compression algorithms supported: 2024-09-21T09:35:40.123 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: kZSTD supported: 0 2024-09-21T09:35:40.123 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: kXpressCompression supported: 0 2024-09-21T09:35:40.123 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: kBZip2Compression supported: 0 2024-09-21T09:35:40.123 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T09:35:40.123 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: kLZ4Compression supported: 1 2024-09-21T09:35:40.123 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: kZlibCompression supported: 1 2024-09-21T09:35:40.123 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T09:35:40.123 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: kSnappyCompression supported: 1 2024-09-21T09:35:40.123 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T09:35:40.124 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-09-21T09:35:40.124 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-21T09:35:40.124 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T09:35:40.124 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.merge_operator: 2024-09-21T09:35:40.124 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compaction_filter: None 2024-09-21T09:35:40.124 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T09:35:40.124 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T09:35:40.124 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T09:35:40.124 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T09:35:40.124 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c169041a38) 2024-09-21T09:35:40.124 INFO:journalctl@ceph.mon.c.smithi107.stdout: cache_index_and_filter_blocks: 1 2024-09-21T09:35:40.124 INFO:journalctl@ceph.mon.c.smithi107.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T09:35:40.124 INFO:journalctl@ceph.mon.c.smithi107.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T09:35:40.124 INFO:journalctl@ceph.mon.c.smithi107.stdout: pin_top_level_index_and_filter: 1 2024-09-21T09:35:40.124 INFO:journalctl@ceph.mon.c.smithi107.stdout: index_type: 0 2024-09-21T09:35:40.124 INFO:journalctl@ceph.mon.c.smithi107.stdout: data_block_index_type: 0 2024-09-21T09:35:40.125 INFO:journalctl@ceph.mon.c.smithi107.stdout: index_shortening: 1 2024-09-21T09:35:40.125 INFO:journalctl@ceph.mon.c.smithi107.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T09:35:40.125 INFO:journalctl@ceph.mon.c.smithi107.stdout: hash_index_allow_collision: 1 2024-09-21T09:35:40.125 INFO:journalctl@ceph.mon.c.smithi107.stdout: checksum: 1 2024-09-21T09:35:40.125 INFO:journalctl@ceph.mon.c.smithi107.stdout: no_block_cache: 0 2024-09-21T09:35:40.125 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_cache: 0x55c1690771f0 2024-09-21T09:35:40.125 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_cache_name: BinnedLRUCache 2024-09-21T09:35:40.125 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_cache_options: 2024-09-21T09:35:40.125 INFO:journalctl@ceph.mon.c.smithi107.stdout: capacity : 536870912 2024-09-21T09:35:40.125 INFO:journalctl@ceph.mon.c.smithi107.stdout: num_shard_bits : 4 2024-09-21T09:35:40.125 INFO:journalctl@ceph.mon.c.smithi107.stdout: strict_capacity_limit : 0 2024-09-21T09:35:40.125 INFO:journalctl@ceph.mon.c.smithi107.stdout: high_pri_pool_ratio: 0.000 2024-09-21T09:35:40.125 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_cache_compressed: (nil) 2024-09-21T09:35:40.125 INFO:journalctl@ceph.mon.c.smithi107.stdout: persistent_cache: (nil) 2024-09-21T09:35:40.125 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_size: 4096 2024-09-21T09:35:40.125 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_size_deviation: 10 2024-09-21T09:35:40.126 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_restart_interval: 16 2024-09-21T09:35:40.126 INFO:journalctl@ceph.mon.c.smithi107.stdout: index_block_restart_interval: 1 2024-09-21T09:35:40.126 INFO:journalctl@ceph.mon.c.smithi107.stdout: metadata_block_size: 4096 2024-09-21T09:35:40.126 INFO:journalctl@ceph.mon.c.smithi107.stdout: partition_filters: 0 2024-09-21T09:35:40.126 INFO:journalctl@ceph.mon.c.smithi107.stdout: use_delta_encoding: 1 2024-09-21T09:35:40.126 INFO:journalctl@ceph.mon.c.smithi107.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-21T09:35:40.126 INFO:journalctl@ceph.mon.c.smithi107.stdout: whole_key_filtering: 1 2024-09-21T09:35:40.126 INFO:journalctl@ceph.mon.c.smithi107.stdout: verify_compression: 0 2024-09-21T09:35:40.126 INFO:journalctl@ceph.mon.c.smithi107.stdout: read_amp_bytes_per_bit: 0 2024-09-21T09:35:40.126 INFO:journalctl@ceph.mon.c.smithi107.stdout: format_version: 4 2024-09-21T09:35:40.126 INFO:journalctl@ceph.mon.c.smithi107.stdout: enable_index_compression: 1 2024-09-21T09:35:40.126 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_align: 0 2024-09-21T09:35:40.126 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T09:35:40.126 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T09:35:40.127 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compression: NoCompression 2024-09-21T09:35:40.127 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T09:35:40.127 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T09:35:40.127 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T09:35:40.127 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.num_levels: 7 2024-09-21T09:35:40.127 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T09:35:40.127 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T09:35:40.127 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T09:35:40.127 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T09:35:40.127 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T09:35:40.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T09:35:40.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T09:35:40.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:35:40.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T09:35:40.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T09:35:40.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T09:35:40.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T09:35:40.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T09:35:40.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T09:35:40.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:35:40.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T09:35:40.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T09:35:40.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T09:35:40.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T09:35:40.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T09:35:40.129 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T09:35:40.129 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T09:35:40.129 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T09:35:40.129 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T09:35:40.129 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T09:35:40.129 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T09:35:40.129 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T09:35:40.130 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T09:35:40.130 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T09:35:40.130 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T09:35:40.130 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T09:35:40.130 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T09:35:40.131 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T09:35:40.131 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T09:35:40.131 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.arena_block_size: 4194304 2024-09-21T09:35:40.131 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T09:35:40.131 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T09:35:40.131 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-21T09:35:40.131 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T09:35:40.131 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T09:35:40.131 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T09:35:40.131 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T09:35:40.131 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T09:35:40.131 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T09:35:40.131 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T09:35:40.131 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T09:35:40.132 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T09:35:40.132 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T09:35:40.132 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T09:35:40.132 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T09:35:40.132 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.inplace_update_support: 0 2024-09-21T09:35:40.132 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T09:35:40.132 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T09:35:40.132 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T09:35:40.132 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T09:35:40.132 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.bloom_locality: 0 2024-09-21T09:35:40.132 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.max_successive_merges: 0 2024-09-21T09:35:40.132 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T09:35:40.132 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T09:35:40.132 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T09:35:40.132 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T09:35:40.133 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.ttl: 2592000 2024-09-21T09:35:40.133 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T09:35:40.133 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.enable_blob_files: false 2024-09-21T09:35:40.133 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.min_blob_size: 0 2024-09-21T09:35:40.133 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T09:35:40.133 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T09:35:40.133 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T09:35:40.133 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T09:35:40.133 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T09:35:40.133 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-09-21T09:35:40.133 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-09-21T09:35:40.133 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726911339828005, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T09:35:40.133 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-09-21T09:35:40.133 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-21T09:35:40.133 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726911339828291, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1696, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726911339, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "a739d123-6959-410f-bf81-740a833f3d5b", "db_session_id": "FJ1OGH83JU3HUHPEC9BI"}} 2024-09-21T09:35:40.134 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-09-21T09:35:40.134 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726911339828601, "job": 1, "event": "recovery_finished"} 2024-09-21T09:35:40.134 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T09:35:40.134 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55c1690aa700 2024-09-21T09:35:40.134 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: DB pointer 0x55c16907a000 2024-09-21T09:35:40.134 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-21T09:35:40.134 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-21T09:35:40.134 INFO:journalctl@ceph.mon.c.smithi107.stdout: ** DB Stats ** 2024-09-21T09:35:40.134 INFO:journalctl@ceph.mon.c.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:35:40.134 INFO:journalctl@ceph.mon.c.smithi107.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T09:35:40.134 INFO:journalctl@ceph.mon.c.smithi107.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:35:40.134 INFO:journalctl@ceph.mon.c.smithi107.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:35:40.134 INFO:journalctl@ceph.mon.c.smithi107.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T09:35:40.134 INFO:journalctl@ceph.mon.c.smithi107.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-21T09:35:40.135 INFO:journalctl@ceph.mon.c.smithi107.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:35:40.135 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-09-21T09:35:40.135 INFO:journalctl@ceph.mon.c.smithi107.stdout: ** Compaction Stats [default] ** 2024-09-21T09:35:40.135 INFO:journalctl@ceph.mon.c.smithi107.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:35:40.135 INFO:journalctl@ceph.mon.c.smithi107.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:35:40.135 INFO:journalctl@ceph.mon.c.smithi107.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.9 0.00 0.00 1 0.000 0 0 2024-09-21T09:35:40.135 INFO:journalctl@ceph.mon.c.smithi107.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.9 0.00 0.00 1 0.000 0 0 2024-09-21T09:35:40.135 INFO:journalctl@ceph.mon.c.smithi107.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.9 0.00 0.00 1 0.000 0 0 2024-09-21T09:35:40.135 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-09-21T09:35:40.135 INFO:journalctl@ceph.mon.c.smithi107.stdout: ** Compaction Stats [default] ** 2024-09-21T09:35:40.135 INFO:journalctl@ceph.mon.c.smithi107.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:35:40.135 INFO:journalctl@ceph.mon.c.smithi107.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:35:40.135 INFO:journalctl@ceph.mon.c.smithi107.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.9 0.00 0.00 1 0.000 0 0 2024-09-21T09:35:40.135 INFO:journalctl@ceph.mon.c.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:35:40.135 INFO:journalctl@ceph.mon.c.smithi107.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:35:40.136 INFO:journalctl@ceph.mon.c.smithi107.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:35:40.136 INFO:journalctl@ceph.mon.c.smithi107.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:35:40.136 INFO:journalctl@ceph.mon.c.smithi107.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:35:40.136 INFO:journalctl@ceph.mon.c.smithi107.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:35:40.136 INFO:journalctl@ceph.mon.c.smithi107.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:35:40.136 INFO:journalctl@ceph.mon.c.smithi107.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:35:40.136 INFO:journalctl@ceph.mon.c.smithi107.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:35:40.136 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-09-21T09:35:40.136 INFO:journalctl@ceph.mon.c.smithi107.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:35:40.136 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-09-21T09:35:40.136 INFO:journalctl@ceph.mon.c.smithi107.stdout: ** Compaction Stats [default] ** 2024-09-21T09:35:40.136 INFO:journalctl@ceph.mon.c.smithi107.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:35:40.136 INFO:journalctl@ceph.mon.c.smithi107.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:35:40.136 INFO:journalctl@ceph.mon.c.smithi107.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.9 0.00 0.00 1 0.000 0 0 2024-09-21T09:35:40.137 INFO:journalctl@ceph.mon.c.smithi107.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.9 0.00 0.00 1 0.000 0 0 2024-09-21T09:35:40.137 INFO:journalctl@ceph.mon.c.smithi107.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-21T09:35:40.137 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-09-21T09:35:40.137 INFO:journalctl@ceph.mon.c.smithi107.stdout: ** Compaction Stats [default] ** 2024-09-21T09:35:40.137 INFO:journalctl@ceph.mon.c.smithi107.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:35:40.137 INFO:journalctl@ceph.mon.c.smithi107.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:35:40.137 INFO:journalctl@ceph.mon.c.smithi107.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.9 0.00 0.00 1 0.000 0 0 2024-09-21T09:35:40.137 INFO:journalctl@ceph.mon.c.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:35:40.137 INFO:journalctl@ceph.mon.c.smithi107.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:35:40.137 INFO:journalctl@ceph.mon.c.smithi107.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:35:40.137 INFO:journalctl@ceph.mon.c.smithi107.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:35:40.137 INFO:journalctl@ceph.mon.c.smithi107.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:35:40.137 INFO:journalctl@ceph.mon.c.smithi107.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:35:40.138 INFO:journalctl@ceph.mon.c.smithi107.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:35:40.138 INFO:journalctl@ceph.mon.c.smithi107.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:35:40.138 INFO:journalctl@ceph.mon.c.smithi107.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:35:40.138 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-09-21T09:35:40.138 INFO:journalctl@ceph.mon.c.smithi107.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:35:40.138 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-21T09:35:40.138 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: using public_addrv [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2024-09-21T09:35:40.138 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: starting mon.c rank -1 at public addrs [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] at bind addrs [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:35:40.138 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: mon.c@-1(???) e0 preinit fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:35:40.138 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: mon.c@-1(synchronizing).mds e1 new map 2024-09-21T09:35:40.138 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-21T09:35:40.138 INFO:journalctl@ceph.mon.c.smithi107.stdout: e1 2024-09-21T09:35:40.138 INFO:journalctl@ceph.mon.c.smithi107.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T09:35:40.139 INFO:journalctl@ceph.mon.c.smithi107.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:35:40.139 INFO:journalctl@ceph.mon.c.smithi107.stdout: legacy client fscid: -1 2024-09-21T09:35:40.139 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-09-21T09:35:40.139 INFO:journalctl@ceph.mon.c.smithi107.stdout: No filesystems configured 2024-09-21T09:35:40.139 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T09:35:40.139 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T09:35:40.139 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T09:35:40.139 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T09:35:40.139 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T09:35:40.139 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T09:35:40.139 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T09:35:40.139 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:35:40.139 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:35:40.139 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:35:40.139 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:40.140 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:40.140 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:40.140 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:40.140 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:40.140 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi129", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:35:40.140 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:40.140 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:35:40.140 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T09:35:40.140 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:35:40.140 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:35:40.140 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:40.140 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:40.140 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:40.140 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1428770317' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-21T09:35:40.140 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:35:40.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:40.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1428770317' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T09:35:40.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:35:40.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:40.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi107:172.21.15.107=a;smithi107:[v2:172.21.15.107:3301,v1:172.21.15.107:6790]=c;smithi129:172.21.15.129=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:35:40.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: Saving service mon spec with placement smithi107:172.21.15.107=a;smithi107:[v2:172.21.15.107:3301,v1:172.21.15.107:6790]=c;smithi129:172.21.15.129=b;count:3 2024-09-21T09:35:40.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:40.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:35:40.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:40.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:35:40.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:40.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:35:40.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:40.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: Deploying daemon mon.b on smithi129 2024-09-21T09:35:40.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:40.141 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:40.142 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='client.? 172.21.15.129:0/830271865' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:35:40.142 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:35:40.142 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: mon.a calling monitor election 2024-09-21T09:35:40.142 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:40.142 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:40.142 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:40.142 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:40.142 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: mon.b calling monitor election 2024-09-21T09:35:40.142 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:40.142 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:40.142 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:40.142 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:40.142 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T09:35:40.142 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: monmap e2: 2 mons at {a=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:35:40.142 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: fsmap 2024-09-21T09:35:40.143 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:35:40.143 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: mgrmap e13: y(active, since 39s) 2024-09-21T09:35:40.143 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: overall HEALTH_OK 2024-09-21T09:35:40.143 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:40.143 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:40.143 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:40.143 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:35:40.143 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:40.143 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: Deploying daemon mon.c on smithi107 2024-09-21T09:35:40.143 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:40.143 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='client.? 172.21.15.129:0/43827006' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:35:40.143 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:40.143 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:40.143 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 ceph-mon[41185]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-21T09:35:40.143 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 bash[41147]: 4d41a86eb0a87952e40ca59b886a0acb256292d85a3655133e60b4a0b8fca615 2024-09-21T09:35:40.143 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:39 smithi107 systemd[1]: Started Ceph mon.c for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:35:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:35:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: mon.a calling monitor election 2024-09-21T09:35:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: mon.b calling monitor election 2024-09-21T09:35:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: mon.c calling monitor election 2024-09-21T09:35:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:45.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:45.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:45.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:45.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:35:45.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: monmap e3: 3 mons at {a=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:35:45.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: fsmap 2024-09-21T09:35:45.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:35:45.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: mgrmap e13: y(active, since 48s) 2024-09-21T09:35:45.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: overall HEALTH_OK 2024-09-21T09:35:45.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:45.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:45.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:45.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:45.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:35:45.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: mon.a calling monitor election 2024-09-21T09:35:45.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: mon.b calling monitor election 2024-09-21T09:35:45.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:45.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:45.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:45.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:45.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:45.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: mon.c calling monitor election 2024-09-21T09:35:45.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:45.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:45.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:45.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:45.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:35:45.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: monmap e3: 3 mons at {a=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:35:45.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: fsmap 2024-09-21T09:35:45.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:35:45.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: mgrmap e13: y(active, since 48s) 2024-09-21T09:35:45.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: overall HEALTH_OK 2024-09-21T09:35:45.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:45.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:45.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:45.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:44 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:35:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: mon.a calling monitor election 2024-09-21T09:35:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: mon.b calling monitor election 2024-09-21T09:35:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:35:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: mon.c calling monitor election 2024-09-21T09:35:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:45.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:45.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:45.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:45.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:35:45.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: monmap e3: 3 mons at {a=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],b=[v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0],c=[v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:35:45.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: fsmap 2024-09-21T09:35:45.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:35:45.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: mgrmap e13: y(active, since 48s) 2024-09-21T09:35:45.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: overall HEALTH_OK 2024-09-21T09:35:45.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:45.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:45.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:45.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:44 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:46.178 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:35:46.328 INFO:teuthology.orchestra.run.smithi129.stdout:{"epoch":3,"fsid":"86c49514-77fc-11ef-baf6-efdc52797490","modified":"2024-09-21T09:35:39.850868Z","created":"2024-09-21T09:34:00.747992Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:3300","nonce":0},{"type":"v1","addr":"172.21.15.129:6789","nonce":0}]},"addr":"172.21.15.129:6789/0","public_addr":"172.21.15.129:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3301","nonce":0},{"type":"v1","addr":"172.21.15.107:6790","nonce":0}]},"addr":"172.21.15.107:6790/0","public_addr":"172.21.15.107:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-21T09:35:46.329 INFO:teuthology.orchestra.run.smithi129.stderr:dumped monmap epoch 3 2024-09-21T09:35:46.329 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:45 smithi129 ceph-mon[32729]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:46.329 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:45 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:35:46.329 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:45 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:46.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:45 smithi107 ceph-mon[29382]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:46.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:45 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:35:46.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:45 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:46.358 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:45 smithi107 ceph-mon[41185]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:46.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:45 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:35:46.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:45 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:35:46.860 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-21T09:35:46.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph config generate-minimal-conf 2024-09-21T09:35:47.199 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.200 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.200 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.200 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.200 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:47.200 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:35:47.200 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[29382]: Updating smithi107:/etc/ceph/ceph.conf 2024-09-21T09:35:47.200 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[29382]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T09:35:47.200 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[29382]: from='client.? 172.21.15.129:0/2714770472' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:35:47.200 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.200 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.200 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.200 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.201 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.201 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:35:47.201 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:35:47.201 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:47.201 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.201 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.201 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.201 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.201 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.201 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:47.201 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:35:47.202 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[41185]: Updating smithi107:/etc/ceph/ceph.conf 2024-09-21T09:35:47.202 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[41185]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T09:35:47.202 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[41185]: from='client.? 172.21.15.129:0/2714770472' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:35:47.202 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.202 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.202 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.202 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.202 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.202 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:35:47.202 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:35:47.202 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:47.202 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:46 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:46 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:46 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:46 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:46 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:46 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:47.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:46 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:35:47.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:46 smithi129 ceph-mon[32729]: Updating smithi107:/etc/ceph/ceph.conf 2024-09-21T09:35:47.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:46 smithi129 ceph-mon[32729]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T09:35:47.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:46 smithi129 ceph-mon[32729]: from='client.? 172.21.15.129:0/2714770472' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:35:47.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:46 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:46 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:46 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:46 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:46 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:46 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:35:47.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:46 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:35:47.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:46 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:47.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:46 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:47.901 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:35:48.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:47 smithi107 ceph-mon[29382]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:35:48.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:47 smithi107 ceph-mon[29382]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:35:48.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:47 smithi107 ceph-mon[29382]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T09:35:48.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:47 smithi107 ceph-mon[29382]: Reconfiguring daemon mon.a on smithi107 2024-09-21T09:35:48.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:47 smithi107 ceph-mon[29382]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:48.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:47 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:48.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:47 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:48.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:47 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:35:48.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:47 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:35:48.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:47 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:48.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:47 smithi107 ceph-mon[41185]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:35:48.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:47 smithi107 ceph-mon[41185]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:35:48.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:47 smithi107 ceph-mon[41185]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T09:35:48.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:47 smithi107 ceph-mon[41185]: Reconfiguring daemon mon.a on smithi107 2024-09-21T09:35:48.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:47 smithi107 ceph-mon[41185]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:48.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:47 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:48.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:47 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:48.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:47 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:35:48.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:47 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:35:48.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:47 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:48.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:47 smithi129 ceph-mon[32729]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:35:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:47 smithi129 ceph-mon[32729]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:35:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:47 smithi129 ceph-mon[32729]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T09:35:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:47 smithi129 ceph-mon[32729]: Reconfiguring daemon mon.a on smithi107 2024-09-21T09:35:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:47 smithi129 ceph-mon[32729]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:47 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:47 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:47 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:35:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:47 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:35:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:47 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:49.276 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:48 smithi107 ceph-mon[29382]: Reconfiguring mon.c (monmap changed)... 2024-09-21T09:35:49.276 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:48 smithi107 ceph-mon[29382]: Reconfiguring daemon mon.c on smithi107 2024-09-21T09:35:49.276 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:48 smithi107 ceph-mon[41185]: Reconfiguring mon.c (monmap changed)... 2024-09-21T09:35:49.276 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:48 smithi107 ceph-mon[41185]: Reconfiguring daemon mon.c on smithi107 2024-09-21T09:35:49.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:48 smithi129 ceph-mon[32729]: Reconfiguring mon.c (monmap changed)... 2024-09-21T09:35:49.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:48 smithi129 ceph-mon[32729]: Reconfiguring daemon mon.c on smithi107 2024-09-21T09:35:49.775 INFO:teuthology.orchestra.run.smithi107.stdout:# minimal ceph.conf for 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:35:49.775 INFO:teuthology.orchestra.run.smithi107.stdout:[global] 2024-09-21T09:35:49.775 INFO:teuthology.orchestra.run.smithi107.stdout: fsid = 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:35:49.775 INFO:teuthology.orchestra.run.smithi107.stdout: mon_host = [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] 2024-09-21T09:35:50.108 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:49 smithi107 ceph-mon[41185]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:50.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:49 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/4273391395' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:50.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:49 smithi107 ceph-mon[29382]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:50.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:49 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/4273391395' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:50.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:49 smithi129 ceph-mon[32729]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:50.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:49 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/4273391395' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:50.578 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-21T09:35:50.578 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T09:35:50.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T09:35:50.607 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T09:35:50.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:35:50.672 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T09:35:50.672 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T09:35:50.705 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T09:35:50.705 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:35:50.774 INFO:tasks.cephadm:Adding mgr.y on smithi107 2024-09-21T09:35:50.774 INFO:tasks.cephadm:Adding mgr.x on smithi129 2024-09-21T09:35:50.774 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph orch apply mgr '2;smithi107=y;smithi129=x' 2024-09-21T09:35:51.059 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:35:52.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:52 smithi107 ceph-mon[41185]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:52.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:52 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:52.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:52 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:52.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:52 smithi107 ceph-mon[41185]: Reconfiguring mon.b (monmap changed)... 2024-09-21T09:35:52.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:52 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:35:52.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:52 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:35:52.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:52 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:52.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:52 smithi107 ceph-mon[41185]: Reconfiguring daemon mon.b on smithi129 2024-09-21T09:35:52.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:52 smithi107 ceph-mon[29382]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:52.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:52 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:52.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:52 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:52.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:52 smithi107 ceph-mon[29382]: Reconfiguring mon.b (monmap changed)... 2024-09-21T09:35:52.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:52 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:35:52.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:52 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:35:52.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:52 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:52.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:52 smithi107 ceph-mon[29382]: Reconfiguring daemon mon.b on smithi129 2024-09-21T09:35:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:52 smithi129 ceph-mon[32729]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:52 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:52 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:52 smithi129 ceph-mon[32729]: Reconfiguring mon.b (monmap changed)... 2024-09-21T09:35:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:52 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:35:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:52 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:35:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:52 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:52 smithi129 ceph-mon[32729]: Reconfiguring daemon mon.b on smithi129 2024-09-21T09:35:53.192 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled mgr update... 2024-09-21T09:35:54.044 DEBUG:teuthology.orchestra.run.smithi129:mgr.x> sudo journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@mgr.x.service 2024-09-21T09:35:54.046 DEBUG:tasks.cephadm:set 0 configs 2024-09-21T09:35:54.046 INFO:tasks.cephadm:Deploying OSDs... 2024-09-21T09:35:54.046 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T09:35:54.046 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T09:35:54.066 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T09:35:54.066 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 2024-09-21T09:35:54.125 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T09:35:54.125 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:35:54.125 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 981 Links: 1 2024-09-21T09:35:54.125 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:35:54.125 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:35:54.125 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-09-21 09:35:39.310512222 +0000 2024-09-21T09:35:54.125 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-09-21 09:31:03.224337589 +0000 2024-09-21T09:35:54.125 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-09-21 09:31:03.224337589 +0000 2024-09-21T09:35:54.125 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-09-21 09:31:03.224337589 +0000 2024-09-21T09:35:54.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T09:35:54.193 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-09-21T09:35:54.193 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-09-21T09:35:54.194 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000202887 s, 2.5 MB/s 2024-09-21T09:35:54.195 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T09:35:54.254 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 2024-09-21T09:35:54.311 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T09:35:54.311 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:35:54.311 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 977 Links: 1 2024-09-21T09:35:54.311 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:35:54.311 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:35:54.311 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-09-21 09:35:39.310512222 +0000 2024-09-21T09:35:54.311 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-09-21 09:31:03.222337624 +0000 2024-09-21T09:35:54.311 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-09-21 09:31:03.222337624 +0000 2024-09-21T09:35:54.311 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-09-21 09:31:03.222337624 +0000 2024-09-21T09:35:54.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T09:35:54.326 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:54 smithi129 ceph-mon[32729]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:54.327 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:54 smithi129 ceph-mon[32729]: from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi107=y;smithi129=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:35:54.327 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:54 smithi129 ceph-mon[32729]: Saving service mgr spec with placement smithi107=y;smithi129=x;count:2 2024-09-21T09:35:54.327 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:54 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:54.327 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:54 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:54.327 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:54 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:54.327 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:54 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:35:54.327 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:54 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:54.327 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:54 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:35:54.327 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:54 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:54.327 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:54 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:35:54.327 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:54 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T09:35:54.327 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:54 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:35:54.327 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:54 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:54.376 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-09-21T09:35:54.376 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-09-21T09:35:54.376 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000232084 s, 2.2 MB/s 2024-09-21T09:35:54.377 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T09:35:54.434 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 2024-09-21T09:35:54.490 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T09:35:54.490 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:35:54.491 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 974 Links: 1 2024-09-21T09:35:54.491 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:35:54.491 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:35:54.491 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-09-21 09:35:39.310512222 +0000 2024-09-21T09:35:54.491 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-09-21 09:31:03.222337624 +0000 2024-09-21T09:35:54.491 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-09-21 09:31:03.222337624 +0000 2024-09-21T09:35:54.491 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-09-21 09:31:03.222337624 +0000 2024-09-21T09:35:54.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T09:35:54.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[41185]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:54.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[41185]: from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi107=y;smithi129=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:35:54.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[41185]: Saving service mgr spec with placement smithi107=y;smithi129=x;count:2 2024-09-21T09:35:54.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:54.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:54.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:54.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:35:54.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:54.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:35:54.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:54.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:35:54.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T09:35:54.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:35:54.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:54.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[29382]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:54.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[29382]: from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi107=y;smithi129=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:35:54.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[29382]: Saving service mgr spec with placement smithi107=y;smithi129=x;count:2 2024-09-21T09:35:54.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:54.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:54.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:54.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:35:54.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:54.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:35:54.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:54.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:35:54.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T09:35:54.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:35:54.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:54 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:54.555 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-09-21T09:35:54.555 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-09-21T09:35:54.555 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000210739 s, 2.4 MB/s 2024-09-21T09:35:54.556 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T09:35:54.613 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 2024-09-21T09:35:54.669 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T09:35:54.669 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:35:54.669 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 967 Links: 1 2024-09-21T09:35:54.669 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:35:54.669 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:35:54.669 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-09-21 09:35:39.311512205 +0000 2024-09-21T09:35:54.669 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-09-21 09:31:03.221337641 +0000 2024-09-21T09:35:54.669 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-09-21 09:31:03.221337641 +0000 2024-09-21T09:35:54.669 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-09-21 09:31:03.221337641 +0000 2024-09-21T09:35:54.670 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T09:35:54.733 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-09-21T09:35:54.733 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-09-21T09:35:54.733 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.00019959 s, 2.6 MB/s 2024-09-21T09:35:54.734 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T09:35:54.790 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T09:35:54.790 DEBUG:teuthology.orchestra.run.smithi129:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T09:35:54.808 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T09:35:54.808 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_1 2024-09-21T09:35:54.864 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T09:35:54.865 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:35:54.865 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 983 Links: 1 2024-09-21T09:35:54.865 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:35:54.865 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:35:54.865 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-21 09:35:30.898785801 +0000 2024-09-21T09:35:54.865 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-21 09:31:02.120887896 +0000 2024-09-21T09:35:54.865 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-21 09:31:02.120887896 +0000 2024-09-21T09:35:54.865 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-21 09:31:02.120887896 +0000 2024-09-21T09:35:54.865 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T09:35:54.930 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-21T09:35:54.930 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-21T09:35:54.930 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000370507 s, 1.4 MB/s 2024-09-21T09:35:54.932 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T09:35:54.988 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_2 2024-09-21T09:35:55.044 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T09:35:55.044 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:35:55.044 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 965 Links: 1 2024-09-21T09:35:55.044 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:35:55.045 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:35:55.045 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-21 09:35:30.898785801 +0000 2024-09-21T09:35:55.045 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-21 09:31:02.118888042 +0000 2024-09-21T09:35:55.045 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-21 09:31:02.118888042 +0000 2024-09-21T09:35:55.045 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-21 09:31:02.118888042 +0000 2024-09-21T09:35:55.045 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T09:35:55.108 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-21T09:35:55.108 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-21T09:35:55.108 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000284013 s, 1.8 MB/s 2024-09-21T09:35:55.109 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T09:35:55.166 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_3 2024-09-21T09:35:55.222 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T09:35:55.223 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:35:55.223 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 975 Links: 1 2024-09-21T09:35:55.223 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:35:55.223 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:35:55.223 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-21 09:35:30.899785778 +0000 2024-09-21T09:35:55.223 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-21 09:31:02.118888042 +0000 2024-09-21T09:35:55.223 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-21 09:31:02.118888042 +0000 2024-09-21T09:35:55.223 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-21 09:31:02.118888042 +0000 2024-09-21T09:35:55.223 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T09:35:55.301 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-21T09:35:55.301 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-21T09:35:55.301 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000200735 s, 2.6 MB/s 2024-09-21T09:35:55.302 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T09:35:55.359 DEBUG:teuthology.orchestra.run.smithi129:> stat /dev/vg_nvme/lv_4 2024-09-21T09:35:55.414 INFO:teuthology.orchestra.run.smithi129.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T09:35:55.414 INFO:teuthology.orchestra.run.smithi129.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:35:55.414 INFO:teuthology.orchestra.run.smithi129.stdout:Device: 5h/5d Inode: 963 Links: 1 2024-09-21T09:35:55.414 INFO:teuthology.orchestra.run.smithi129.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:35:55.414 INFO:teuthology.orchestra.run.smithi129.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:35:55.414 INFO:teuthology.orchestra.run.smithi129.stdout:Access: 2024-09-21 09:35:55.253237471 +0000 2024-09-21T09:35:55.414 INFO:teuthology.orchestra.run.smithi129.stdout:Modify: 2024-09-21 09:31:02.118888042 +0000 2024-09-21T09:35:55.415 INFO:teuthology.orchestra.run.smithi129.stdout:Change: 2024-09-21 09:31:02.118888042 +0000 2024-09-21T09:35:55.415 INFO:teuthology.orchestra.run.smithi129.stdout: Birth: 2024-09-21 09:31:02.118888042 +0000 2024-09-21T09:35:55.415 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T09:35:55.476 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:55 smithi129 ceph-mon[32729]: Deploying daemon mgr.x on smithi129 2024-09-21T09:35:55.486 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records in 2024-09-21T09:35:55.486 INFO:teuthology.orchestra.run.smithi129.stderr:1+0 records out 2024-09-21T09:35:55.486 INFO:teuthology.orchestra.run.smithi129.stderr:512 bytes copied, 0.000176807 s, 2.9 MB/s 2024-09-21T09:35:55.487 DEBUG:teuthology.orchestra.run.smithi129:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T09:35:55.543 INFO:tasks.cephadm:Deploying osd.0 on smithi107 with /dev/vg_nvme/lv_4... 2024-09-21T09:35:55.543 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T09:35:55.566 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:55 smithi107 ceph-mon[29382]: Deploying daemon mgr.x on smithi129 2024-09-21T09:35:55.566 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:55 smithi107 ceph-mon[41185]: Deploying daemon mgr.x on smithi129 2024-09-21T09:35:55.781 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:35:55.823 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:55 smithi129 podman[34763]: 2024-09-21 09:35:55.614073423 +0000 UTC m=+0.015605952 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T09:35:56.184 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:55 smithi129 podman[34763]: 2024-09-21 09:35:55.930440196 +0000 UTC m=+0.331972705 container init 1f321f7d8eff1220945e51d5357db00e290120c2ad9d9b1a13007900297080f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-21T09:35:56.184 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:55 smithi129 podman[34763]: 2024-09-21 09:35:55.935355121 +0000 UTC m=+0.336887617 container start 1f321f7d8eff1220945e51d5357db00e290120c2ad9d9b1a13007900297080f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=quincy-fbd1a6a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-21T09:35:56.184 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:55 smithi129 ceph-mgr[34801]: -- 172.21.15.129:0/3423033296 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55756f3b6340 con 0x55756f3c3000 2024-09-21T09:35:56.184 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:56 smithi129 bash[34763]: 1f321f7d8eff1220945e51d5357db00e290120c2ad9d9b1a13007900297080f3 2024-09-21T09:35:56.184 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:56 smithi129 systemd[1]: Started Ceph mgr.x for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:35:56.185 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:56.053+0000 7f5fb24deec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:35:56.185 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:56.116+0000 7f5fb24deec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:35:56.185 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:56.182+0000 7f5fb24deec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:35:56.466 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:56.313+0000 7f5fb24deec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:35:56.466 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:56.462+0000 7f5fb24deec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:35:56.466 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:56 smithi129 ceph-mon[32729]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:56.467 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:56 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:56.467 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:56 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:56.467 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:56 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:56.467 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:56 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:56.467 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:56 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:35:56.519 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:56 smithi107 ceph-mon[41185]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:56.519 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:56 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:56.519 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:56 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:56.519 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:56 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:56.519 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:56 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:56.519 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:56 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:35:56.519 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:56 smithi107 ceph-mon[29382]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:56.520 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:56 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:56.520 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:56 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:56.520 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:56 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:56.520 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:56 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:56.520 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:56 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:35:56.736 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:56.734+0000 7f5fb24deec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:35:57.043 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:56.798+0000 7f5fb24deec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:35:57.043 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:56.966+0000 7f5fb24deec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:35:58.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:57 smithi107 ceph-mon[41185]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:58.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:57 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:58.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:57 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:58.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:57 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:58.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:57 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:58.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:57 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:35:58.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:57 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:58.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:57 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:35:58.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:57 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:35:58.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:57 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:58.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:57 smithi107 ceph-mon[29382]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:58.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:57 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:58.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:57 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:58.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:57 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:58.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:57 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:58.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:57 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:35:58.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:57 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:58.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:57 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:35:58.111 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:57 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:35:58.111 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:57 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:58.165 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:35:58.168 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:57.817+0000 7f5fb24deec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:35:58.168 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:57.880+0000 7f5fb24deec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:35:58.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:57 smithi129 ceph-mon[32729]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:58.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:57 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:58.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:57 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:58.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:57 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:58.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:57 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:58.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:57 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:35:58.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:57 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:58.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:57 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:35:58.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:57 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:35:58.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:57 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:35:58.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi107:vg_nvme/lv_4 2024-09-21T09:35:58.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:58 smithi129 ceph-mon[32729]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T09:35:58.864 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:58 smithi129 ceph-mon[32729]: Reconfiguring daemon mgr.y on smithi107 2024-09-21T09:35:58.864 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:58 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:58.577+0000 7f5fb24deec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:35:58.864 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:58 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:58.687+0000 7f5fb24deec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:35:58.864 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:58 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:58.804+0000 7f5fb24deec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:35:58.879 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:35:58.998 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:58 smithi107 ceph-mon[29382]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T09:35:58.998 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:58 smithi107 ceph-mon[29382]: Reconfiguring daemon mgr.y on smithi107 2024-09-21T09:35:58.999 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:58 smithi107 ceph-mon[41185]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T09:35:58.999 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:58 smithi107 ceph-mon[41185]: Reconfiguring daemon mgr.y on smithi107 2024-09-21T09:35:59.168 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:58 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:58.862+0000 7f5fb24deec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:35:59.169 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:58 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:58.986+0000 7f5fb24deec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:35:59.169 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:59.051+0000 7f5fb24deec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:35:59.169 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:59.107+0000 7f5fb24deec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:35:59.498 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:59.238+0000 7f5fb24deec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:35:59.499 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:59.300+0000 7f5fb24deec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:35:59.499 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:59.362+0000 7f5fb24deec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:35:59.499 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:59.441+0000 7f5fb24deec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:35:59.499 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:59.497+0000 7f5fb24deec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:35:59.794 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:59 smithi107 ceph-mon[41185]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:59.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:59 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:59.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:59 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:59.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:35:59 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:35:59.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:59 smithi107 ceph-mon[29382]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:59.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:59 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:59.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:59 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:59.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:35:59 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:35:59.918 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:35:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:35:59.553+0000 7f5fb24deec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:35:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:59 smithi129 ceph-mon[32729]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:35:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:59 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:59 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:35:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:35:59 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:36:00.668 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:36:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:36:00.288+0000 7f5fb24deec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:36:00.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:00 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:00.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:00 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:36:00.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:00 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:00.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:00 smithi107 ceph-mon[29382]: Standby manager daemon x started 2024-09-21T09:36:00.811 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:00 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.129:0/756171400' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:36:00.811 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:00 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.129:0/756171400' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:36:00.811 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:00 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.129:0/756171400' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:36:00.811 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:00 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.129:0/756171400' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:36:00.811 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:00 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:00.811 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:00 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:36:00.811 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:00 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:00.811 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:00 smithi107 ceph-mon[41185]: Standby manager daemon x started 2024-09-21T09:36:00.811 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:00 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/756171400' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:36:00.811 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:00 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/756171400' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:36:00.811 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:00 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/756171400' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:36:00.811 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:00 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/756171400' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:36:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:00 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:00 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:36:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:00 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:00 smithi129 ceph-mon[32729]: Standby manager daemon x started 2024-09-21T09:36:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:00 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/756171400' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:36:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:00 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/756171400' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:36:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:00 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/756171400' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:36:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:00 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/756171400' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:36:02.067 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:01 smithi107 ceph-mon[29382]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:02.067 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:01 smithi107 ceph-mon[29382]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:36:02.067 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:01 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:36:02.068 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:01 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:36:02.068 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:01 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:02.068 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:01 smithi107 ceph-mon[29382]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-21T09:36:02.068 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:01 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:36:02.068 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:01 smithi107 ceph-mon[41185]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:02.068 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:01 smithi107 ceph-mon[41185]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:36:02.068 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:01 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:36:02.068 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:01 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:36:02.068 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:01 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:02.068 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:01 smithi107 ceph-mon[41185]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-21T09:36:02.068 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:01 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:36:02.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:01 smithi129 ceph-mon[32729]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:01 smithi129 ceph-mon[32729]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:36:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:01 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:36:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:01 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:36:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:01 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:01 smithi129 ceph-mon[32729]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-21T09:36:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:01 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:36:03.832 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:03 smithi107 ceph-mon[29382]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:03.832 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:03 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1739429208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "635c1eb0-29b1-4fdd-bd47-881ef05e49bd"}]: dispatch 2024-09-21T09:36:03.832 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:03 smithi107 ceph-mon[29382]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "635c1eb0-29b1-4fdd-bd47-881ef05e49bd"}]: dispatch 2024-09-21T09:36:03.832 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:03 smithi107 ceph-mon[29382]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "635c1eb0-29b1-4fdd-bd47-881ef05e49bd"}]': finished 2024-09-21T09:36:03.832 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:03 smithi107 ceph-mon[29382]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T09:36:03.832 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:03 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:36:03.832 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:03 smithi107 ceph-mon[41185]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:03.832 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:03 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1739429208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "635c1eb0-29b1-4fdd-bd47-881ef05e49bd"}]: dispatch 2024-09-21T09:36:03.832 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:03 smithi107 ceph-mon[41185]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "635c1eb0-29b1-4fdd-bd47-881ef05e49bd"}]: dispatch 2024-09-21T09:36:03.832 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:03 smithi107 ceph-mon[41185]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "635c1eb0-29b1-4fdd-bd47-881ef05e49bd"}]': finished 2024-09-21T09:36:03.832 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:03 smithi107 ceph-mon[41185]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T09:36:03.832 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:03 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:36:04.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:03 smithi129 ceph-mon[32729]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:03 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1739429208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "635c1eb0-29b1-4fdd-bd47-881ef05e49bd"}]: dispatch 2024-09-21T09:36:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:03 smithi129 ceph-mon[32729]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "635c1eb0-29b1-4fdd-bd47-881ef05e49bd"}]: dispatch 2024-09-21T09:36:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:03 smithi129 ceph-mon[32729]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "635c1eb0-29b1-4fdd-bd47-881ef05e49bd"}]': finished 2024-09-21T09:36:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:03 smithi129 ceph-mon[32729]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T09:36:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:03 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:36:05.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:04 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3195288862' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:36:05.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:04 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3195288862' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:36:05.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:04 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3195288862' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:36:05.966 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:05 smithi107 ceph-mon[29382]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:05.967 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:05 smithi107 ceph-mon[41185]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:06.070 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:05 smithi129 ceph-mon[32729]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:07.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:07 smithi107 ceph-mon[29382]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:07.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:07 smithi107 ceph-mon[41185]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:08.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:07 smithi129 ceph-mon[32729]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:09.960 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:09 smithi107 ceph-mon[29382]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:09.960 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:09 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-21T09:36:09.960 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:09 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:09.960 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:09 smithi107 ceph-mon[41185]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:09.960 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:09 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-21T09:36:09.960 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:09 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:10.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:09 smithi129 ceph-mon[32729]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:10.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:09 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-21T09:36:10.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:09 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:10.778 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:10 smithi107 ceph-mon[29382]: Deploying daemon osd.0 on smithi107 2024-09-21T09:36:10.778 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:10 smithi107 ceph-mon[41185]: Deploying daemon osd.0 on smithi107 2024-09-21T09:36:11.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:10 smithi129 ceph-mon[32729]: Deploying daemon osd.0 on smithi107 2024-09-21T09:36:11.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:11 smithi107 ceph-mon[29382]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:11.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:11 smithi107 ceph-mon[41185]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:12.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:11 smithi129 ceph-mon[32729]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:13.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:13 smithi107 ceph-mon[29382]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:13.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:13 smithi107 ceph-mon[41185]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:14.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:13 smithi129 ceph-mon[32729]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:14.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:14 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:36:14.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:14 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:14 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:14 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:36:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:14 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:14 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:15.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:14 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:36:15.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:14 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:15.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:14 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:16.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:16 smithi107 ceph-mon[29382]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:16.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:16 smithi107 ceph-mon[41185]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:16.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:16 smithi129 ceph-mon[32729]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:17.712 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 0 on host 'smithi107' 2024-09-21T09:36:18.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:17 smithi107 ceph-mon[41185]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:18.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:17 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:18.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:17 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:18.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:17 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:18.333 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:17 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:36:18.333 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:17 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:18.333 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:17 smithi107 ceph-mon[41185]: from='osd.0 [v2:172.21.15.107:6802/174831156,v1:172.21.15.107:6803/174831156]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T09:36:18.333 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:17 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:36:18.333 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:17 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:18.333 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:17 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:18.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:17 smithi107 ceph-mon[29382]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:18.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:17 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:18.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:17 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:18.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:17 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:18.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:17 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:36:18.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:17 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:18.334 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:17 smithi107 ceph-mon[29382]: from='osd.0 [v2:172.21.15.107:6802/174831156,v1:172.21.15.107:6803/174831156]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T09:36:18.335 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:17 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:36:18.335 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:17 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:18.335 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:17 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:18.356 DEBUG:teuthology.orchestra.run.smithi107:osd.0> sudo journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.0.service 2024-09-21T09:36:18.359 INFO:tasks.cephadm:Deploying osd.1 on smithi107 with /dev/vg_nvme/lv_3... 2024-09-21T09:36:18.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T09:36:18.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:17 smithi129 ceph-mon[32729]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:18.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:17 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:18.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:17 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:18.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:17 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:18.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:17 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:36:18.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:17 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:18.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:17 smithi129 ceph-mon[32729]: from='osd.0 [v2:172.21.15.107:6802/174831156,v1:172.21.15.107:6803/174831156]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T09:36:18.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:17 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:36:18.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:17 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:18.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:17 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:19.142 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:36:19.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:19 smithi107 ceph-mon[29382]: from='osd.0 [v2:172.21.15.107:6802/174831156,v1:172.21.15.107:6803/174831156]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T09:36:19.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:19 smithi107 ceph-mon[29382]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T09:36:19.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:19 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:36:19.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:19 smithi107 ceph-mon[29382]: from='osd.0 [v2:172.21.15.107:6802/174831156,v1:172.21.15.107:6803/174831156]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-09-21T09:36:19.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:19 smithi107 ceph-mon[41185]: from='osd.0 [v2:172.21.15.107:6802/174831156,v1:172.21.15.107:6803/174831156]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T09:36:19.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:19 smithi107 ceph-mon[41185]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T09:36:19.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:19 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:36:19.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:19 smithi107 ceph-mon[41185]: from='osd.0 [v2:172.21.15.107:6802/174831156,v1:172.21.15.107:6803/174831156]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-09-21T09:36:19.441 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:36:19 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0[45557]: 2024-09-21T09:36:19.189+0000 7ff80bb1e640 -1 osd.0 0 waiting for initial osdmap 2024-09-21T09:36:19.441 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:36:19 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0[45557]: 2024-09-21T09:36:19.195+0000 7ff806955640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:36:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:19 smithi129 ceph-mon[32729]: from='osd.0 [v2:172.21.15.107:6802/174831156,v1:172.21.15.107:6803/174831156]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T09:36:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:19 smithi129 ceph-mon[32729]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T09:36:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:19 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:36:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:19 smithi129 ceph-mon[32729]: from='osd.0 [v2:172.21.15.107:6802/174831156,v1:172.21.15.107:6803/174831156]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-09-21T09:36:20.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:20 smithi107 ceph-mon[29382]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:20.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:20 smithi107 ceph-mon[29382]: from='osd.0 [v2:172.21.15.107:6802/174831156,v1:172.21.15.107:6803/174831156]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-09-21T09:36:20.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:20 smithi107 ceph-mon[29382]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T09:36:20.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:20 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:36:20.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:20 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:36:20.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:20 smithi107 ceph-mon[41185]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:20.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:20 smithi107 ceph-mon[41185]: from='osd.0 [v2:172.21.15.107:6802/174831156,v1:172.21.15.107:6803/174831156]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-09-21T09:36:20.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:20 smithi107 ceph-mon[41185]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T09:36:20.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:20 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:36:20.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:20 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:36:20.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:20 smithi129 ceph-mon[32729]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:20.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:20 smithi129 ceph-mon[32729]: from='osd.0 [v2:172.21.15.107:6802/174831156,v1:172.21.15.107:6803/174831156]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-09-21T09:36:20.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:20 smithi129 ceph-mon[32729]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T09:36:20.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:20 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:36:20.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:20 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:36:21.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:21 smithi107 ceph-mon[29382]: purged_snaps scrub starts 2024-09-21T09:36:21.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:21 smithi107 ceph-mon[29382]: purged_snaps scrub ok 2024-09-21T09:36:21.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:21 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:36:21.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:21 smithi107 ceph-mon[29382]: osd.0 [v2:172.21.15.107:6802/174831156,v1:172.21.15.107:6803/174831156] boot 2024-09-21T09:36:21.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:21 smithi107 ceph-mon[29382]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T09:36:21.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:21 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:36:21.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:21 smithi107 ceph-mon[41185]: purged_snaps scrub starts 2024-09-21T09:36:21.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:21 smithi107 ceph-mon[41185]: purged_snaps scrub ok 2024-09-21T09:36:21.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:21 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:36:21.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:21 smithi107 ceph-mon[41185]: osd.0 [v2:172.21.15.107:6802/174831156,v1:172.21.15.107:6803/174831156] boot 2024-09-21T09:36:21.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:21 smithi107 ceph-mon[41185]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T09:36:21.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:21 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:36:21.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:21 smithi129 ceph-mon[32729]: purged_snaps scrub starts 2024-09-21T09:36:21.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:21 smithi129 ceph-mon[32729]: purged_snaps scrub ok 2024-09-21T09:36:21.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:21 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:36:21.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:21 smithi129 ceph-mon[32729]: osd.0 [v2:172.21.15.107:6802/174831156,v1:172.21.15.107:6803/174831156] boot 2024-09-21T09:36:21.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:21 smithi129 ceph-mon[32729]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T09:36:21.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:21 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:36:22.065 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:36:22.084 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi107:vg_nvme/lv_3 2024-09-21T09:36:22.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:22 smithi107 ceph-mon[29382]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:22.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:22 smithi107 ceph-mon[29382]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T09:36:22.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:22 smithi107 ceph-mon[41185]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:22.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:22 smithi107 ceph-mon[41185]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T09:36:22.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:22 smithi129 ceph-mon[32729]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:36:22.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:22 smithi129 ceph-mon[32729]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T09:36:22.720 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:36:24.358 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:24 smithi107 ceph-mon[41185]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:24.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:24 smithi107 ceph-mon[29382]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:24.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:24 smithi129 ceph-mon[32729]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:25.838 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[29382]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:25.838 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[29382]: Detected new or changed devices on smithi107 2024-09-21T09:36:25.838 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:25.838 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:25.838 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:36:25.839 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[29382]: Adjusting osd_memory_target on smithi107 to 16063M 2024-09-21T09:36:25.839 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:25.839 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:25.839 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:36:25.839 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:25.839 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[29382]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:36:25.839 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:36:25.839 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:36:25.839 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:25.839 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[41185]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:25.840 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[41185]: Detected new or changed devices on smithi107 2024-09-21T09:36:25.840 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:25.840 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:25.840 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:36:25.840 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[41185]: Adjusting osd_memory_target on smithi107 to 16063M 2024-09-21T09:36:25.840 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:25.840 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:25.840 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:36:25.840 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:25.840 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[41185]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:36:25.840 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:36:25.840 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:36:25.840 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:25 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:25 smithi129 ceph-mon[32729]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:25 smithi129 ceph-mon[32729]: Detected new or changed devices on smithi107 2024-09-21T09:36:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:25 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:25 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:25 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:36:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:25 smithi129 ceph-mon[32729]: Adjusting osd_memory_target on smithi107 to 16063M 2024-09-21T09:36:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:25 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:25 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:25 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:36:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:25 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:25 smithi129 ceph-mon[32729]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:36:26.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:25 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:36:26.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:25 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:36:26.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:25 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:27.927 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:27 smithi107 ceph-mon[29382]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:27.927 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:27 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1516237886' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a39b6a18-939d-4999-89b2-1126c1b2d1f0"}]: dispatch 2024-09-21T09:36:27.927 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:27 smithi107 ceph-mon[29382]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a39b6a18-939d-4999-89b2-1126c1b2d1f0"}]: dispatch 2024-09-21T09:36:27.927 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:27 smithi107 ceph-mon[29382]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a39b6a18-939d-4999-89b2-1126c1b2d1f0"}]': finished 2024-09-21T09:36:27.927 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:27 smithi107 ceph-mon[29382]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T09:36:27.927 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:27 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:36:27.927 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:27 smithi107 ceph-mon[41185]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:27.928 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:27 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1516237886' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a39b6a18-939d-4999-89b2-1126c1b2d1f0"}]: dispatch 2024-09-21T09:36:27.928 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:27 smithi107 ceph-mon[41185]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a39b6a18-939d-4999-89b2-1126c1b2d1f0"}]: dispatch 2024-09-21T09:36:27.928 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:27 smithi107 ceph-mon[41185]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a39b6a18-939d-4999-89b2-1126c1b2d1f0"}]': finished 2024-09-21T09:36:27.928 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:27 smithi107 ceph-mon[41185]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T09:36:27.928 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:27 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:36:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:27 smithi129 ceph-mon[32729]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:27 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1516237886' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a39b6a18-939d-4999-89b2-1126c1b2d1f0"}]: dispatch 2024-09-21T09:36:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:27 smithi129 ceph-mon[32729]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a39b6a18-939d-4999-89b2-1126c1b2d1f0"}]: dispatch 2024-09-21T09:36:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:27 smithi129 ceph-mon[32729]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a39b6a18-939d-4999-89b2-1126c1b2d1f0"}]': finished 2024-09-21T09:36:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:27 smithi129 ceph-mon[32729]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T09:36:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:27 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:36:29.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:28 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/585313498' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:36:29.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:28 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/585313498' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:36:29.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:28 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/585313498' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:36:30.054 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:29 smithi107 ceph-mon[29382]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:30.054 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:29 smithi107 ceph-mon[41185]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:30.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:29 smithi129 ceph-mon[32729]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:32.015 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:31 smithi107 ceph-mon[29382]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:32.015 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:31 smithi107 ceph-mon[41185]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:32.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:31 smithi129 ceph-mon[32729]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:34.091 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:33 smithi107 ceph-mon[41185]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:34.091 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:33 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-21T09:36:34.091 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:33 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:34.091 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:33 smithi107 ceph-mon[29382]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:34.091 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:33 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-21T09:36:34.091 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:33 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:34.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:33 smithi129 ceph-mon[32729]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:34.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:33 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-21T09:36:34.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:33 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:35.028 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:34 smithi107 ceph-mon[29382]: Deploying daemon osd.1 on smithi107 2024-09-21T09:36:35.028 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:34 smithi107 ceph-mon[41185]: Deploying daemon osd.1 on smithi107 2024-09-21T09:36:35.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:34 smithi129 ceph-mon[32729]: Deploying daemon osd.1 on smithi107 2024-09-21T09:36:35.874 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:35 smithi107 ceph-mon[29382]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:35.875 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:35 smithi107 ceph-mon[41185]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:36.070 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:35 smithi129 ceph-mon[32729]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:37.983 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:37 smithi107 ceph-mon[29382]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:37.983 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:37 smithi107 ceph-mon[41185]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:38.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:37 smithi129 ceph-mon[32729]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:38.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:38 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:36:38.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:38 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:38.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:38 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:38.847 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:38 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:36:38.847 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:38 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:38.847 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:38 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:39.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:38 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:36:39.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:38 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:39.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:38 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:40.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:40 smithi107 ceph-mon[29382]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:40.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:40 smithi107 ceph-mon[41185]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:40.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:40 smithi129 ceph-mon[32729]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:41.435 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 1 on host 'smithi107' 2024-09-21T09:36:41.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:41 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:41.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:41 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:41.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:41 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:41.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:41 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:36:41.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:41 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:41.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:41 smithi107 ceph-mon[29382]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:41.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:41 smithi107 ceph-mon[29382]: from='osd.1 [v2:172.21.15.107:6810/1227448003,v1:172.21.15.107:6811/1227448003]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T09:36:41.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:41 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:36:41.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:41 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:41.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:41 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:41.748 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:41 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:41.748 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:41 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:41.748 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:41 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:41.748 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:41 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:36:41.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:41 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:41.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:41 smithi107 ceph-mon[41185]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:41.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:41 smithi107 ceph-mon[41185]: from='osd.1 [v2:172.21.15.107:6810/1227448003,v1:172.21.15.107:6811/1227448003]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T09:36:41.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:41 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:36:41.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:41 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:41.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:41 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:41 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:41 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:41 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:41 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:36:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:41 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:41 smithi129 ceph-mon[32729]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:41 smithi129 ceph-mon[32729]: from='osd.1 [v2:172.21.15.107:6810/1227448003,v1:172.21.15.107:6811/1227448003]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T09:36:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:41 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:36:41.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:41 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:41.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:41 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:42.047 DEBUG:teuthology.orchestra.run.smithi107:osd.1> sudo journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.1.service 2024-09-21T09:36:42.049 INFO:tasks.cephadm:Deploying osd.2 on smithi107 with /dev/vg_nvme/lv_2... 2024-09-21T09:36:42.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T09:36:42.801 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:36:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:42 smithi129 ceph-mon[32729]: from='osd.1 [v2:172.21.15.107:6810/1227448003,v1:172.21.15.107:6811/1227448003]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T09:36:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:42 smithi129 ceph-mon[32729]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T09:36:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:42 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:36:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:42 smithi129 ceph-mon[32729]: from='osd.1 [v2:172.21.15.107:6810/1227448003,v1:172.21.15.107:6811/1227448003]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-09-21T09:36:42.921 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:42 smithi107 ceph-mon[29382]: from='osd.1 [v2:172.21.15.107:6810/1227448003,v1:172.21.15.107:6811/1227448003]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T09:36:42.922 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:42 smithi107 ceph-mon[29382]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T09:36:42.922 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:42 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:36:42.922 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:42 smithi107 ceph-mon[29382]: from='osd.1 [v2:172.21.15.107:6810/1227448003,v1:172.21.15.107:6811/1227448003]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-09-21T09:36:42.922 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:42 smithi107 ceph-mon[41185]: from='osd.1 [v2:172.21.15.107:6810/1227448003,v1:172.21.15.107:6811/1227448003]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T09:36:42.922 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:42 smithi107 ceph-mon[41185]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T09:36:42.922 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:42 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:36:42.922 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:42 smithi107 ceph-mon[41185]: from='osd.1 [v2:172.21.15.107:6810/1227448003,v1:172.21.15.107:6811/1227448003]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-09-21T09:36:42.922 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:36:42 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1[51795]: 2024-09-21T09:36:42.658+0000 7fec6d396640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T09:36:42.923 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:36:42 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1[51795]: 2024-09-21T09:36:42.663+0000 7fec6a1d1640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:36:43.728 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:43 smithi107 ceph-mon[41185]: from='osd.1 [v2:172.21.15.107:6810/1227448003,v1:172.21.15.107:6811/1227448003]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-09-21T09:36:43.728 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:43 smithi107 ceph-mon[41185]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T09:36:43.728 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:43 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:36:43.728 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:43 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:36:43.728 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:43 smithi107 ceph-mon[41185]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:43.728 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:43 smithi107 ceph-mon[29382]: from='osd.1 [v2:172.21.15.107:6810/1227448003,v1:172.21.15.107:6811/1227448003]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-09-21T09:36:43.728 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:43 smithi107 ceph-mon[29382]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T09:36:43.729 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:43 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:36:43.729 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:43 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:36:43.729 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:43 smithi107 ceph-mon[29382]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:43 smithi129 ceph-mon[32729]: from='osd.1 [v2:172.21.15.107:6810/1227448003,v1:172.21.15.107:6811/1227448003]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-09-21T09:36:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:43 smithi129 ceph-mon[32729]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T09:36:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:43 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:36:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:43 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:36:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:43 smithi129 ceph-mon[32729]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:36:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:44 smithi129 ceph-mon[32729]: purged_snaps scrub starts 2024-09-21T09:36:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:44 smithi129 ceph-mon[32729]: purged_snaps scrub ok 2024-09-21T09:36:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:44 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:36:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:44 smithi129 ceph-mon[32729]: osd.1 [v2:172.21.15.107:6810/1227448003,v1:172.21.15.107:6811/1227448003] boot 2024-09-21T09:36:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:44 smithi129 ceph-mon[32729]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T09:36:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:44 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:36:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:44 smithi107 ceph-mon[29382]: purged_snaps scrub starts 2024-09-21T09:36:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:44 smithi107 ceph-mon[29382]: purged_snaps scrub ok 2024-09-21T09:36:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:44 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:36:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:44 smithi107 ceph-mon[29382]: osd.1 [v2:172.21.15.107:6810/1227448003,v1:172.21.15.107:6811/1227448003] boot 2024-09-21T09:36:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:44 smithi107 ceph-mon[29382]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T09:36:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:44 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:36:45.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:44 smithi107 ceph-mon[41185]: purged_snaps scrub starts 2024-09-21T09:36:45.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:44 smithi107 ceph-mon[41185]: purged_snaps scrub ok 2024-09-21T09:36:45.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:44 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:36:45.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:44 smithi107 ceph-mon[41185]: osd.1 [v2:172.21.15.107:6810/1227448003,v1:172.21.15.107:6811/1227448003] boot 2024-09-21T09:36:45.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:44 smithi107 ceph-mon[41185]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T09:36:45.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:44 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:36:45.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:45 smithi107 ceph-mon[29382]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:45.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:45 smithi107 ceph-mon[29382]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T09:36:45.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:45 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:45.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:45 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:45.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:45 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:36:45.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:45 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:36:45.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:45 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:45.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:45 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:45.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:45 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:36:45.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:45 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:45.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:45 smithi107 ceph-mon[41185]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:45.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:45 smithi107 ceph-mon[41185]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T09:36:45.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:45 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:45.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:45 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:45.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:45 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:36:45.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:45 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:36:45.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:45 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:45.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:45 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:45.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:45 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:36:45.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:45 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:46.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:45 smithi129 ceph-mon[32729]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:46.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:45 smithi129 ceph-mon[32729]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T09:36:46.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:45 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:46.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:45 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:46.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:45 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:36:46.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:45 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:36:46.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:45 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:46.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:45 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:46.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:45 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:36:46.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:45 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:36:46.833 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:46 smithi107 ceph-mon[29382]: Detected new or changed devices on smithi107 2024-09-21T09:36:46.833 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:46 smithi107 ceph-mon[29382]: Adjusting osd_memory_target on smithi107 to 8031M 2024-09-21T09:36:46.833 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:46 smithi107 ceph-mon[41185]: Detected new or changed devices on smithi107 2024-09-21T09:36:46.834 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:46 smithi107 ceph-mon[41185]: Adjusting osd_memory_target on smithi107 to 8031M 2024-09-21T09:36:46.838 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:36:46.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi107:vg_nvme/lv_2 2024-09-21T09:36:47.096 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:36:47.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:46 smithi129 ceph-mon[32729]: Detected new or changed devices on smithi107 2024-09-21T09:36:47.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:46 smithi129 ceph-mon[32729]: Adjusting osd_memory_target on smithi107 to 8031M 2024-09-21T09:36:47.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:47 smithi107 ceph-mon[29382]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:47.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:47 smithi107 ceph-mon[41185]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:48.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:47 smithi129 ceph-mon[32729]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:49.974 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:49 smithi107 ceph-mon[29382]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:49.974 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:49 smithi107 ceph-mon[29382]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:36:49.974 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:49 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:36:49.974 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:49 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:36:49.974 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:49 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:49.974 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:49 smithi107 ceph-mon[41185]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:49.974 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:49 smithi107 ceph-mon[41185]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:36:49.974 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:49 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:36:49.975 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:49 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:36:49.975 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:49 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:50.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:49 smithi129 ceph-mon[32729]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:50.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:49 smithi129 ceph-mon[32729]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:36:50.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:49 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:36:50.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:49 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:36:50.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:49 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:52.018 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:51 smithi107 ceph-mon[41185]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:52.018 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:51 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3395568385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b01c5f1b-77e4-462f-80d0-fd1d3afb786b"}]: dispatch 2024-09-21T09:36:52.019 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:51 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3395568385' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b01c5f1b-77e4-462f-80d0-fd1d3afb786b"}]': finished 2024-09-21T09:36:52.019 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:51 smithi107 ceph-mon[41185]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T09:36:52.019 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:51 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:36:52.019 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:51 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/2517390226' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:36:52.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:51 smithi107 ceph-mon[29382]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:52.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:51 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3395568385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b01c5f1b-77e4-462f-80d0-fd1d3afb786b"}]: dispatch 2024-09-21T09:36:52.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:51 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3395568385' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b01c5f1b-77e4-462f-80d0-fd1d3afb786b"}]': finished 2024-09-21T09:36:52.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:51 smithi107 ceph-mon[29382]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T09:36:52.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:51 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:36:52.019 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:51 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2517390226' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:36:52.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:51 smithi129 ceph-mon[32729]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:52.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:51 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3395568385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b01c5f1b-77e4-462f-80d0-fd1d3afb786b"}]: dispatch 2024-09-21T09:36:52.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:51 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3395568385' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b01c5f1b-77e4-462f-80d0-fd1d3afb786b"}]': finished 2024-09-21T09:36:52.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:51 smithi129 ceph-mon[32729]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T09:36:52.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:51 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:36:52.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:51 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2517390226' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:36:53.939 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:53 smithi107 ceph-mon[29382]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:53.939 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:53 smithi107 ceph-mon[41185]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:54.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:53 smithi129 ceph-mon[32729]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:55.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:55 smithi107 ceph-mon[41185]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:55 smithi107 ceph-mon[29382]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:56.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:55 smithi129 ceph-mon[32729]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:57.837 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:57 smithi107 ceph-mon[41185]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:57.837 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:57 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-21T09:36:57.837 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:57 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:57.837 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:57 smithi107 ceph-mon[41185]: Deploying daemon osd.2 on smithi107 2024-09-21T09:36:57.837 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:57 smithi107 ceph-mon[29382]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:57.837 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:57 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-21T09:36:57.838 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:57 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:57.838 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:57 smithi107 ceph-mon[29382]: Deploying daemon osd.2 on smithi107 2024-09-21T09:36:58.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:57 smithi129 ceph-mon[32729]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:58.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:57 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-21T09:36:58.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:57 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:36:58.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:57 smithi129 ceph-mon[32729]: Deploying daemon osd.2 on smithi107 2024-09-21T09:36:59.742 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:36:59 smithi107 ceph-mon[29382]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:36:59.742 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:36:59 smithi107 ceph-mon[41185]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:37:00.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:36:59 smithi129 ceph-mon[32729]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:37:01.812 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:01 smithi107 ceph-mon[29382]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:37:01.812 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:01 smithi107 ceph-mon[41185]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:37:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:01 smithi129 ceph-mon[32729]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:37:02.915 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:02 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:37:02.915 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:02 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:02.915 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:02 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:02.915 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:02 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:37:02.915 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:02 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:02.915 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:02 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:03.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:02 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:37:03.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:02 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:03.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:02 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:04.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:04 smithi107 ceph-mon[29382]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:37:04.290 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:04 smithi107 ceph-mon[41185]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:37:04.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:04 smithi129 ceph-mon[32729]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:37:05.454 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 2 on host 'smithi107' 2024-09-21T09:37:06.029 DEBUG:teuthology.orchestra.run.smithi107:osd.2> sudo journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.2.service 2024-09-21T09:37:06.032 INFO:tasks.cephadm:Deploying osd.3 on smithi107 with /dev/vg_nvme/lv_1... 2024-09-21T09:37:06.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T09:37:06.324 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:37:06.358 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:06 smithi107 ceph-mon[41185]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:37:06.358 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:06 smithi107 ceph-mon[41185]: from='osd.2 [v2:172.21.15.107:6818/3613749103,v1:172.21.15.107:6819/3613749103]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T09:37:06.358 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:06 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:06.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:06 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:06.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:06 smithi107 ceph-mon[29382]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:37:06.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:06 smithi107 ceph-mon[29382]: from='osd.2 [v2:172.21.15.107:6818/3613749103,v1:172.21.15.107:6819/3613749103]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T09:37:06.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:06 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:06.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:06 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:06.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:06 smithi129 ceph-mon[32729]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:37:06.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:06 smithi129 ceph-mon[32729]: from='osd.2 [v2:172.21.15.107:6818/3613749103,v1:172.21.15.107:6819/3613749103]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T09:37:06.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:06 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:06.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:06 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:07.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:07 smithi107 ceph-mon[41185]: from='osd.2 [v2:172.21.15.107:6818/3613749103,v1:172.21.15.107:6819/3613749103]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T09:37:07.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:07 smithi107 ceph-mon[41185]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T09:37:07.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:07 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:37:07.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:07 smithi107 ceph-mon[41185]: from='osd.2 [v2:172.21.15.107:6818/3613749103,v1:172.21.15.107:6819/3613749103]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-09-21T09:37:07.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:07 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:07.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:07 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:07.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:07 smithi107 ceph-mon[29382]: from='osd.2 [v2:172.21.15.107:6818/3613749103,v1:172.21.15.107:6819/3613749103]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T09:37:07.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:07 smithi107 ceph-mon[29382]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T09:37:07.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:07 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:37:07.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:07 smithi107 ceph-mon[29382]: from='osd.2 [v2:172.21.15.107:6818/3613749103,v1:172.21.15.107:6819/3613749103]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-09-21T09:37:07.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:07 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:07.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:07 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:07.361 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:37:07 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2[57889]: 2024-09-21T09:37:07.068+0000 7f0fb9fc8640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T09:37:07.361 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:37:07 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2[57889]: 2024-09-21T09:37:07.074+0000 7f0fb6e03640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:37:07.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:07 smithi129 ceph-mon[32729]: from='osd.2 [v2:172.21.15.107:6818/3613749103,v1:172.21.15.107:6819/3613749103]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T09:37:07.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:07 smithi129 ceph-mon[32729]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T09:37:07.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:07 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:37:07.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:07 smithi129 ceph-mon[32729]: from='osd.2 [v2:172.21.15.107:6818/3613749103,v1:172.21.15.107:6819/3613749103]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-09-21T09:37:07.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:07 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:07.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:07 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:08.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:08 smithi107 ceph-mon[29382]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:37:08.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:08 smithi107 ceph-mon[29382]: from='osd.2 [v2:172.21.15.107:6818/3613749103,v1:172.21.15.107:6819/3613749103]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-09-21T09:37:08.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:08 smithi107 ceph-mon[29382]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T09:37:08.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:08 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:37:08.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:08 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:37:08.429 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:08 smithi107 ceph-mon[41185]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:37:08.429 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:08 smithi107 ceph-mon[41185]: from='osd.2 [v2:172.21.15.107:6818/3613749103,v1:172.21.15.107:6819/3613749103]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-09-21T09:37:08.429 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:08 smithi107 ceph-mon[41185]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T09:37:08.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:08 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:37:08.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:08 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:37:08.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:08 smithi129 ceph-mon[32729]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:37:08.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:08 smithi129 ceph-mon[32729]: from='osd.2 [v2:172.21.15.107:6818/3613749103,v1:172.21.15.107:6819/3613749103]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-09-21T09:37:08.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:08 smithi129 ceph-mon[32729]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T09:37:08.430 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:08 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:37:08.431 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:08 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:37:09.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:09 smithi107 ceph-mon[29382]: purged_snaps scrub starts 2024-09-21T09:37:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:09 smithi107 ceph-mon[29382]: purged_snaps scrub ok 2024-09-21T09:37:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:09 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:37:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:09 smithi107 ceph-mon[29382]: osd.2 [v2:172.21.15.107:6818/3613749103,v1:172.21.15.107:6819/3613749103] boot 2024-09-21T09:37:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:09 smithi107 ceph-mon[29382]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T09:37:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:09 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:37:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:09 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-21T09:37:09.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:09 smithi107 ceph-mon[41185]: purged_snaps scrub starts 2024-09-21T09:37:09.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:09 smithi107 ceph-mon[41185]: purged_snaps scrub ok 2024-09-21T09:37:09.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:09 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:37:09.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:09 smithi107 ceph-mon[41185]: osd.2 [v2:172.21.15.107:6818/3613749103,v1:172.21.15.107:6819/3613749103] boot 2024-09-21T09:37:09.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:09 smithi107 ceph-mon[41185]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T09:37:09.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:09 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:37:09.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:09 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-21T09:37:09.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:09 smithi129 ceph-mon[32729]: purged_snaps scrub starts 2024-09-21T09:37:09.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:09 smithi129 ceph-mon[32729]: purged_snaps scrub ok 2024-09-21T09:37:09.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:09 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:37:09.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:09 smithi129 ceph-mon[32729]: osd.2 [v2:172.21.15.107:6818/3613749103,v1:172.21.15.107:6819/3613749103] boot 2024-09-21T09:37:09.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:09 smithi129 ceph-mon[32729]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T09:37:09.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:09 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:37:09.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:09 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-21T09:37:09.615 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:37:09.635 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi107:vg_nvme/lv_1 2024-09-21T09:37:10.234 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:37:10.355 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:10 smithi107 ceph-mon[29382]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:37:10.355 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:10 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-21T09:37:10.356 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:10 smithi107 ceph-mon[29382]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T09:37:10.356 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:10 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:37:10.356 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:10 smithi107 ceph-mon[41185]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:37:10.356 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:10 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-21T09:37:10.356 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:10 smithi107 ceph-mon[41185]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T09:37:10.356 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:10 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:37:10.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:10 smithi129 ceph-mon[32729]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:37:10.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:10 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-21T09:37:10.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:10 smithi129 ceph-mon[32729]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T09:37:10.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:10 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:37:10.608 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:37:10 smithi107 sudo[59685]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:37:10.609 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:37:10 smithi107 sudo[59685]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:37:10.609 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:37:10 smithi107 sudo[59685]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:37:10.609 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:37:10 smithi107 sudo[59685]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:37:10.935 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:37:10 smithi107 sudo[62103]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:37:10.935 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:37:10 smithi107 sudo[62103]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:37:10.936 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:37:10 smithi107 sudo[62103]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:37:10.936 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:37:10 smithi107 sudo[62103]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:37:11.259 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:11 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:37:11.259 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:11 smithi107 ceph-mon[41185]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T09:37:11.259 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:11 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:11.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:11 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:37:11.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:11 smithi107 ceph-mon[29382]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T09:37:11.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:11 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:11.260 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:37:10 smithi107 sudo[62106]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:37:11.260 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:37:10 smithi107 sudo[62106]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:37:11.260 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:37:10 smithi107 sudo[62106]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:37:11.260 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:37:11 smithi107 sudo[62106]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:37:11.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:11 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:37:11.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:11 smithi129 ceph-mon[32729]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T09:37:11.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:11 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:11.523 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:37:11 smithi107 sudo[62115]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:37:11.523 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:37:11 smithi107 sudo[62115]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:37:11.523 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:37:11 smithi107 sudo[62115]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:37:11.523 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:37:11 smithi107 sudo[62115]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:37:11.523 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:37:11 smithi107 sudo[62120]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:37:11.523 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:37:11 smithi107 sudo[62120]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:37:11.523 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:37:11 smithi107 sudo[62120]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:37:11.523 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:37:11 smithi107 sudo[62120]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:37:11.773 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:37:11 smithi107 sudo[62167]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:37:11.773 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:37:11 smithi107 sudo[62167]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:37:11.774 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:37:11 smithi107 sudo[62167]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:37:11.774 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:37:11 smithi107 sudo[62167]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:37:12.033 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 sudo[62198]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:37:12.034 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 sudo[62198]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:37:12.034 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 sudo[62198]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:37:12.034 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 sudo[62198]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:37:12.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 sudo[62221]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:37:12.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 sudo[62221]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:37:12.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 sudo[62221]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:37:12.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 sudo[62221]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:37:12.541 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 ceph-mon[32729]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:12.541 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 ceph-mon[32729]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T09:37:12.541 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 ceph-mon[32729]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T09:37:12.542 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:12.542 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:12.542 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:12.542 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:12.542 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:12.542 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:12.542 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:12.542 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:37:12.542 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:12.542 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:37:12.542 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:12.542 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:37:12.542 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:12.543 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 ceph-mon[32729]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:37:12.543 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 sudo[35464]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:37:12.543 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 sudo[35464]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:37:12.543 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 sudo[35464]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:37:12.543 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 sudo[35464]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:37:12.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[29382]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:12.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[29382]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T09:37:12.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[29382]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T09:37:12.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:12.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:12.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:12.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:12.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:12.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:12.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:12.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:37:12.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:12.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:37:12.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:12.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:37:12.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:12.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[29382]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:37:12.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[41185]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:12.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[41185]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T09:37:12.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[41185]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T09:37:12.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:12.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:12.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:12.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:12.546 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:12.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:12.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:12.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:37:12.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:12.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:37:12.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:12.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:37:12.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:12.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:12 smithi107 ceph-mon[41185]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:37:12.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 sudo[35467]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:37:12.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 sudo[35467]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:37:12.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 sudo[35467]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:37:12.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:12 smithi129 sudo[35467]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:37:13.318 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 sudo[62391]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:37:13.318 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 sudo[62391]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:37:13.319 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 sudo[62391]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:37:13.319 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 sudo[62391]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:37:13.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 sudo[62413]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:37:13.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 sudo[62413]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:37:13.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 sudo[62413]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:37:13.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 sudo[62413]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:37:13.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[41185]: Detected new or changed devices on smithi107 2024-09-21T09:37:13.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[41185]: Adjusting osd_memory_target on smithi107 to 5354M 2024-09-21T09:37:13.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[41185]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:37:13.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:37:13.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:37:13.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:37:13.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[41185]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:37:13.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:37:13.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:37:13.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:37:13.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:37:13.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:37:13.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:13.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[41185]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:37:13.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:13.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:37:13.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:37:13.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:37:13.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[29382]: Detected new or changed devices on smithi107 2024-09-21T09:37:13.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[29382]: Adjusting osd_memory_target on smithi107 to 5354M 2024-09-21T09:37:13.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[29382]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:37:13.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:37:13.612 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:37:13.612 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:37:13.612 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[29382]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:37:13.612 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:37:13.612 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:37:13.612 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:37:13.612 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:37:13.612 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:37:13.612 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:13.612 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[29382]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:37:13.612 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:13.613 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:37:13.613 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:37:13.613 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:37:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:13 smithi129 ceph-mon[32729]: Detected new or changed devices on smithi107 2024-09-21T09:37:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:13 smithi129 ceph-mon[32729]: Adjusting osd_memory_target on smithi107 to 5354M 2024-09-21T09:37:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:13 smithi129 ceph-mon[32729]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:37:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:13 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:37:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:13 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:37:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:13 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:37:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:13 smithi129 ceph-mon[32729]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:37:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:13 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:37:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:13 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:37:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:13 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:37:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:13 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:37:13.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:13 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:37:13.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:13 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:13.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:13 smithi129 ceph-mon[32729]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:37:13.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:13 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:13.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:13 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:37:13.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:13 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:37:13.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:13 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:37:14.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:14 smithi107 ceph-mon[29382]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:14 smithi107 ceph-mon[29382]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:37:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:14 smithi107 ceph-mon[29382]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:37:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:14 smithi107 ceph-mon[29382]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:37:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:14 smithi107 ceph-mon[41185]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:14 smithi107 ceph-mon[41185]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:37:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:14 smithi107 ceph-mon[41185]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:37:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:14 smithi107 ceph-mon[41185]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:37:14.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:14 smithi129 ceph-mon[32729]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:14 smithi129 ceph-mon[32729]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:37:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:14 smithi129 ceph-mon[32729]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:37:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:14 smithi129 ceph-mon[32729]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:37:15.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:15 smithi107 ceph-mon[29382]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:15.565 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:15 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/4236638213' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5135d269-b6c4-4a9d-bcf9-ffe19bc5c9ad"}]: dispatch 2024-09-21T09:37:15.565 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:15 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/4236638213' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5135d269-b6c4-4a9d-bcf9-ffe19bc5c9ad"}]': finished 2024-09-21T09:37:15.565 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:15 smithi107 ceph-mon[29382]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T09:37:15.565 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:15 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:37:15.565 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:15 smithi107 ceph-mon[41185]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:15.565 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:15 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/4236638213' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5135d269-b6c4-4a9d-bcf9-ffe19bc5c9ad"}]: dispatch 2024-09-21T09:37:15.565 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:15 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/4236638213' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5135d269-b6c4-4a9d-bcf9-ffe19bc5c9ad"}]': finished 2024-09-21T09:37:15.565 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:15 smithi107 ceph-mon[41185]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T09:37:15.565 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:15 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:37:15.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:15 smithi129 ceph-mon[32729]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:15.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:15 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/4236638213' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5135d269-b6c4-4a9d-bcf9-ffe19bc5c9ad"}]: dispatch 2024-09-21T09:37:15.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:15 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/4236638213' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5135d269-b6c4-4a9d-bcf9-ffe19bc5c9ad"}]': finished 2024-09-21T09:37:15.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:15 smithi129 ceph-mon[32729]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T09:37:15.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:15 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:37:16.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:16 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2306967154' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:37:16.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:16 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/2306967154' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:37:16.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:16 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2306967154' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:37:17.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:17 smithi107 ceph-mon[29382]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:17.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:17 smithi107 ceph-mon[41185]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:17.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:17 smithi129 ceph-mon[32729]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:19.973 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:19 smithi107 ceph-mon[29382]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:19.974 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:19 smithi107 ceph-mon[41185]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:20.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:19 smithi129 ceph-mon[32729]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:21.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:21 smithi107 ceph-mon[29382]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:21.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:21 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-21T09:37:21.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:21 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:21.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:21 smithi107 ceph-mon[29382]: Deploying daemon osd.3 on smithi107 2024-09-21T09:37:21.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:21 smithi107 ceph-mon[41185]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:21.831 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:21 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-21T09:37:21.831 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:21 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:21.831 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:21 smithi107 ceph-mon[41185]: Deploying daemon osd.3 on smithi107 2024-09-21T09:37:22.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:21 smithi129 ceph-mon[32729]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:22.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:21 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-21T09:37:22.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:21 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:22.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:21 smithi129 ceph-mon[32729]: Deploying daemon osd.3 on smithi107 2024-09-21T09:37:24.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:23 smithi107 ceph-mon[29382]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:24.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:23 smithi107 ceph-mon[41185]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:24.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:23 smithi129 ceph-mon[32729]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:26.015 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:25 smithi107 ceph-mon[29382]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:26.015 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:25 smithi107 ceph-mon[41185]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:26.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:25 smithi129 ceph-mon[32729]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:26.976 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:26 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:37:26.976 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:26 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:26.976 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:26 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:26.976 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:26 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:37:26.976 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:26 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:26.976 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:26 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:27.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:26 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:37:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:26 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:26 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:28.533 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:28 smithi107 ceph-mon[29382]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:28.533 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:28 smithi107 ceph-mon[41185]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:28.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:28 smithi129 ceph-mon[32729]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:29.403 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 3 on host 'smithi107' 2024-09-21T09:37:30.030 DEBUG:teuthology.orchestra.run.smithi107:osd.3> sudo journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.3.service 2024-09-21T09:37:30.033 INFO:tasks.cephadm:Deploying osd.4 on smithi129 with /dev/vg_nvme/lv_4... 2024-09-21T09:37:30.033 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T09:37:30.289 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:37:30.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:30 smithi129 ceph-mon[32729]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:30.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:30 smithi129 ceph-mon[32729]: from='osd.3 [v2:172.21.15.107:6826/3206618601,v1:172.21.15.107:6827/3206618601]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T09:37:30.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:30 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:30.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:30 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:30.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:30 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:30.397 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:30 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:30.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:30 smithi107 ceph-mon[41185]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:30.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:30 smithi107 ceph-mon[41185]: from='osd.3 [v2:172.21.15.107:6826/3206618601,v1:172.21.15.107:6827/3206618601]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T09:37:30.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:30 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:30.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:30 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:30.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:30 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:30.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:30 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:30.607 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:30 smithi107 ceph-mon[29382]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:30.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:30 smithi107 ceph-mon[29382]: from='osd.3 [v2:172.21.15.107:6826/3206618601,v1:172.21.15.107:6827/3206618601]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T09:37:30.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:30 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:30.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:30 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:30.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:30 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:30.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:30 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:31.310 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:31 smithi129 ceph-mon[32729]: from='osd.3 [v2:172.21.15.107:6826/3206618601,v1:172.21.15.107:6827/3206618601]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T09:37:31.310 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:31 smithi129 ceph-mon[32729]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T09:37:31.310 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:31 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:37:31.310 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:31 smithi129 ceph-mon[32729]: from='osd.3 [v2:172.21.15.107:6826/3206618601,v1:172.21.15.107:6827/3206618601]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-09-21T09:37:31.505 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:31 smithi107 ceph-mon[29382]: from='osd.3 [v2:172.21.15.107:6826/3206618601,v1:172.21.15.107:6827/3206618601]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T09:37:31.505 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:31 smithi107 ceph-mon[29382]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T09:37:31.505 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:31 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:37:31.505 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:31 smithi107 ceph-mon[29382]: from='osd.3 [v2:172.21.15.107:6826/3206618601,v1:172.21.15.107:6827/3206618601]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-09-21T09:37:31.505 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:31 smithi107 ceph-mon[41185]: from='osd.3 [v2:172.21.15.107:6826/3206618601,v1:172.21.15.107:6827/3206618601]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T09:37:31.506 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:31 smithi107 ceph-mon[41185]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T09:37:31.506 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:31 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:37:31.506 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:31 smithi107 ceph-mon[41185]: from='osd.3 [v2:172.21.15.107:6826/3206618601,v1:172.21.15.107:6827/3206618601]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-09-21T09:37:31.506 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:37:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3[64322]: 2024-09-21T09:37:31.272+0000 7f897f94b640 -1 osd.3 0 waiting for initial osdmap 2024-09-21T09:37:31.506 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:37:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3[64322]: 2024-09-21T09:37:31.276+0000 7f897bf85640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:37:32.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:32 smithi107 ceph-mon[41185]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:32.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:32 smithi107 ceph-mon[41185]: from='osd.3 [v2:172.21.15.107:6826/3206618601,v1:172.21.15.107:6827/3206618601]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-09-21T09:37:32.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:32 smithi107 ceph-mon[41185]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T09:37:32.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:32 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:37:32.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:32 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:37:32.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:32 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:37:32.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:32 smithi107 ceph-mon[29382]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:32.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:32 smithi107 ceph-mon[29382]: from='osd.3 [v2:172.21.15.107:6826/3206618601,v1:172.21.15.107:6827/3206618601]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-09-21T09:37:32.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:32 smithi107 ceph-mon[29382]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T09:37:32.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:32 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:37:32.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:32 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:37:32.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:32 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:37:32.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:32 smithi129 ceph-mon[32729]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:32.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:32 smithi129 ceph-mon[32729]: from='osd.3 [v2:172.21.15.107:6826/3206618601,v1:172.21.15.107:6827/3206618601]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-09-21T09:37:32.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:32 smithi129 ceph-mon[32729]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T09:37:32.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:32 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:37:32.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:32 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:37:32.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:32 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:37:32.655 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:37:32.676 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi129:vg_nvme/lv_4 2024-09-21T09:37:32.912 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:37:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[41185]: purged_snaps scrub starts 2024-09-21T09:37:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[41185]: purged_snaps scrub ok 2024-09-21T09:37:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[41185]: osd.3 [v2:172.21.15.107:6826/3206618601,v1:172.21.15.107:6827/3206618601] boot 2024-09-21T09:37:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[41185]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T09:37:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:37:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:33.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:33.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:33.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:37:33.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:33.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:37:33.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:33.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:37:33.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:33.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[29382]: purged_snaps scrub starts 2024-09-21T09:37:33.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[29382]: purged_snaps scrub ok 2024-09-21T09:37:33.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[29382]: osd.3 [v2:172.21.15.107:6826/3206618601,v1:172.21.15.107:6827/3206618601] boot 2024-09-21T09:37:33.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[29382]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T09:37:33.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:37:33.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:33.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:33.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:33.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:33.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:33.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:33.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:33.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:33.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:37:33.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:33.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:37:33.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:33.612 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:37:33.612 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:33 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:33.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:33 smithi129 ceph-mon[32729]: purged_snaps scrub starts 2024-09-21T09:37:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:33 smithi129 ceph-mon[32729]: purged_snaps scrub ok 2024-09-21T09:37:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:33 smithi129 ceph-mon[32729]: osd.3 [v2:172.21.15.107:6826/3206618601,v1:172.21.15.107:6827/3206618601] boot 2024-09-21T09:37:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:33 smithi129 ceph-mon[32729]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T09:37:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:33 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:37:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:33 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:33 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:33 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:33 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:33 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:33 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:33 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:33 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:33 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:37:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:33 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:33.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:33 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:37:33.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:33 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:33.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:33 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:37:33.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:33 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:34.517 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:34 smithi129 ceph-mon[32729]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:34.517 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:34 smithi129 ceph-mon[32729]: Detected new or changed devices on smithi107 2024-09-21T09:37:34.518 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:34 smithi129 ceph-mon[32729]: Adjusting osd_memory_target on smithi107 to 4015M 2024-09-21T09:37:34.518 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:34 smithi129 ceph-mon[32729]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T09:37:34.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:34 smithi107 ceph-mon[29382]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:34.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:34 smithi107 ceph-mon[29382]: Detected new or changed devices on smithi107 2024-09-21T09:37:34.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:34 smithi107 ceph-mon[29382]: Adjusting osd_memory_target on smithi107 to 4015M 2024-09-21T09:37:34.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:34 smithi107 ceph-mon[29382]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T09:37:34.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:34 smithi107 ceph-mon[41185]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:37:34.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:34 smithi107 ceph-mon[41185]: Detected new or changed devices on smithi107 2024-09-21T09:37:34.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:34 smithi107 ceph-mon[41185]: Adjusting osd_memory_target on smithi107 to 4015M 2024-09-21T09:37:34.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:34 smithi107 ceph-mon[41185]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T09:37:35.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:35 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:37:35.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:35 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:37:35.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:35 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:35.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:35 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:37:35.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:35 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:37:35.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:35 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:35.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:35 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:37:35.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:35 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:37:35.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:35 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:36.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:36 smithi107 ceph-mon[29382]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:36.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:36 smithi107 ceph-mon[29382]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:37:36.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:36 smithi107 ceph-mon[41185]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:36.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:36 smithi107 ceph-mon[41185]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:37:36.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:36 smithi129 ceph-mon[32729]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:36.662 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:36 smithi129 ceph-mon[32729]: from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:37:37.526 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:37 smithi129 ceph-mon[32729]: from='client.? 172.21.15.129:0/3509612955' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23242972-e341-4773-809f-299aabeacd56"}]: dispatch 2024-09-21T09:37:37.526 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:37 smithi129 ceph-mon[32729]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23242972-e341-4773-809f-299aabeacd56"}]: dispatch 2024-09-21T09:37:37.526 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:37 smithi129 ceph-mon[32729]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23242972-e341-4773-809f-299aabeacd56"}]': finished 2024-09-21T09:37:37.526 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:37 smithi129 ceph-mon[32729]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T09:37:37.526 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:37 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:37:37.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:37 smithi107 ceph-mon[29382]: from='client.? 172.21.15.129:0/3509612955' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23242972-e341-4773-809f-299aabeacd56"}]: dispatch 2024-09-21T09:37:37.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:37 smithi107 ceph-mon[29382]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23242972-e341-4773-809f-299aabeacd56"}]: dispatch 2024-09-21T09:37:37.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:37 smithi107 ceph-mon[29382]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23242972-e341-4773-809f-299aabeacd56"}]': finished 2024-09-21T09:37:37.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:37 smithi107 ceph-mon[29382]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T09:37:37.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:37 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:37:37.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:37 smithi107 ceph-mon[41185]: from='client.? 172.21.15.129:0/3509612955' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23242972-e341-4773-809f-299aabeacd56"}]: dispatch 2024-09-21T09:37:37.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:37 smithi107 ceph-mon[41185]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23242972-e341-4773-809f-299aabeacd56"}]: dispatch 2024-09-21T09:37:37.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:37 smithi107 ceph-mon[41185]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23242972-e341-4773-809f-299aabeacd56"}]': finished 2024-09-21T09:37:37.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:37 smithi107 ceph-mon[41185]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T09:37:37.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:37 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:37:38.586 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:38 smithi129 ceph-mon[32729]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:38.587 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:38 smithi129 ceph-mon[32729]: from='client.? 172.21.15.129:0/938895072' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:37:38.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:38 smithi107 ceph-mon[29382]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:38.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:38 smithi107 ceph-mon[29382]: from='client.? 172.21.15.129:0/938895072' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:37:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:38 smithi107 ceph-mon[41185]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:38 smithi107 ceph-mon[41185]: from='client.? 172.21.15.129:0/938895072' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:37:39.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:39 smithi107 ceph-mon[29382]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:39.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:39 smithi107 ceph-mon[41185]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:39.664 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:39 smithi129 ceph-mon[32729]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:41.992 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:41 smithi129 ceph-mon[32729]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:42.108 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:41 smithi107 ceph-mon[41185]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:42.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:41 smithi107 ceph-mon[29382]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:43.818 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:43 smithi129 ceph-mon[32729]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:43.818 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:43 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-21T09:37:43.818 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:43 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:43.818 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:43 smithi129 ceph-mon[32729]: Deploying daemon osd.4 on smithi129 2024-09-21T09:37:44.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:43 smithi107 ceph-mon[29382]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:44.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:43 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-21T09:37:44.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:43 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:44.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:43 smithi107 ceph-mon[29382]: Deploying daemon osd.4 on smithi129 2024-09-21T09:37:44.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:43 smithi107 ceph-mon[41185]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:44.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:43 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-21T09:37:44.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:43 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:44.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:43 smithi107 ceph-mon[41185]: Deploying daemon osd.4 on smithi129 2024-09-21T09:37:46.045 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:45 smithi129 ceph-mon[32729]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:46.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:45 smithi107 ceph-mon[29382]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:46.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:45 smithi107 ceph-mon[41185]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:48.041 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:47 smithi129 ceph-mon[32729]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:48.041 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:47 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:37:48.041 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:47 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:48.042 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:47 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:48.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:47 smithi107 ceph-mon[29382]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:48.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:47 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:37:48.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:47 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:48.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:47 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:48.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:47 smithi107 ceph-mon[41185]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:48.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:47 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:37:48.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:47 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:48.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:47 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:49 smithi129 ceph-mon[32729]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:50.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:49 smithi107 ceph-mon[29382]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:50.108 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:49 smithi107 ceph-mon[41185]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:50.407 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 4 on host 'smithi129' 2024-09-21T09:37:50.902 DEBUG:teuthology.orchestra.run.smithi129:osd.4> sudo journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.4.service 2024-09-21T09:37:50.904 INFO:tasks.cephadm:Deploying osd.5 on smithi129 with /dev/vg_nvme/lv_3... 2024-09-21T09:37:50.905 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T09:37:51.219 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:51 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:51.219 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:51 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:51.219 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:51 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:51.219 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:51 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:51.220 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:51 smithi129 ceph-mon[32729]: from='osd.4 [v2:172.21.15.129:6800/4101249335,v1:172.21.15.129:6801/4101249335]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:37:51.220 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:51 smithi129 ceph-mon[32729]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:37:51.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:51 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:51.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:51 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:51.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:51 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:51.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:51 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:51.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:51 smithi107 ceph-mon[29382]: from='osd.4 [v2:172.21.15.129:6800/4101249335,v1:172.21.15.129:6801/4101249335]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:37:51.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:51 smithi107 ceph-mon[29382]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:37:51.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:51 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:51.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:51 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:51.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:51 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:51.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:51 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:51.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:51 smithi107 ceph-mon[41185]: from='osd.4 [v2:172.21.15.129:6800/4101249335,v1:172.21.15.129:6801/4101249335]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:37:51.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:51 smithi107 ceph-mon[41185]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:37:51.679 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:37:52.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:52 smithi107 ceph-mon[29382]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:52.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:52 smithi107 ceph-mon[29382]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T09:37:52.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:52 smithi107 ceph-mon[29382]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T09:37:52.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:52 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:37:52.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:52 smithi107 ceph-mon[29382]: from='osd.4 [v2:172.21.15.129:6800/4101249335,v1:172.21.15.129:6801/4101249335]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:37:52.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:52 smithi107 ceph-mon[29382]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:37:52.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:52 smithi107 ceph-mon[41185]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:52.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:52 smithi107 ceph-mon[41185]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T09:37:52.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:52 smithi107 ceph-mon[41185]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T09:37:52.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:52 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:37:52.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:52 smithi107 ceph-mon[41185]: from='osd.4 [v2:172.21.15.129:6800/4101249335,v1:172.21.15.129:6801/4101249335]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:37:52.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:52 smithi107 ceph-mon[41185]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:37:52.379 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:52 smithi129 ceph-mon[32729]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:52.380 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:52 smithi129 ceph-mon[32729]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T09:37:52.380 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:52 smithi129 ceph-mon[32729]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T09:37:52.380 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:52 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:37:52.380 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:52 smithi129 ceph-mon[32729]: from='osd.4 [v2:172.21.15.129:6800/4101249335,v1:172.21.15.129:6801/4101249335]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:37:52.380 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:52 smithi129 ceph-mon[32729]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:37:52.669 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 09:37:52 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4[38016]: 2024-09-21T09:37:52.417+0000 7efe9a645640 -1 osd.4 0 waiting for initial osdmap 2024-09-21T09:37:52.669 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 09:37:52 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4[38016]: 2024-09-21T09:37:52.421+0000 7efe93478640 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:37:53.594 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:53 smithi129 ceph-mon[32729]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T09:37:53.594 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:53 smithi129 ceph-mon[32729]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T09:37:53.594 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:53 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:37:53.594 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:53 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:37:53.594 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:53 smithi129 ceph-mon[32729]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:53 smithi107 ceph-mon[29382]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T09:37:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:53 smithi107 ceph-mon[29382]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T09:37:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:53 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:37:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:53 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:37:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:53 smithi107 ceph-mon[29382]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:53 smithi107 ceph-mon[41185]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T09:37:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:53 smithi107 ceph-mon[41185]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T09:37:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:53 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:37:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:53 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:37:53.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:53 smithi107 ceph-mon[41185]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:37:54.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:54 smithi129 ceph-mon[32729]: purged_snaps scrub starts 2024-09-21T09:37:54.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:54 smithi129 ceph-mon[32729]: purged_snaps scrub ok 2024-09-21T09:37:54.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:54 smithi129 ceph-mon[32729]: osd.4 [v2:172.21.15.129:6800/4101249335,v1:172.21.15.129:6801/4101249335] boot 2024-09-21T09:37:54.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:54 smithi129 ceph-mon[32729]: osdmap e30: 5 total, 5 up, 5 in 2024-09-21T09:37:54.665 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:54 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:37:54.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:54 smithi107 ceph-mon[29382]: purged_snaps scrub starts 2024-09-21T09:37:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:54 smithi107 ceph-mon[29382]: purged_snaps scrub ok 2024-09-21T09:37:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:54 smithi107 ceph-mon[29382]: osd.4 [v2:172.21.15.129:6800/4101249335,v1:172.21.15.129:6801/4101249335] boot 2024-09-21T09:37:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:54 smithi107 ceph-mon[29382]: osdmap e30: 5 total, 5 up, 5 in 2024-09-21T09:37:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:54 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:37:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:54 smithi107 ceph-mon[41185]: purged_snaps scrub starts 2024-09-21T09:37:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:54 smithi107 ceph-mon[41185]: purged_snaps scrub ok 2024-09-21T09:37:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:54 smithi107 ceph-mon[41185]: osd.4 [v2:172.21.15.129:6800/4101249335,v1:172.21.15.129:6801/4101249335] boot 2024-09-21T09:37:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:54 smithi107 ceph-mon[41185]: osdmap e30: 5 total, 5 up, 5 in 2024-09-21T09:37:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:54 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:37:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:55 smithi129 ceph-mon[32729]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T09:37:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:55 smithi129 ceph-mon[32729]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:37:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:55 smithi129 ceph-mon[32729]: Detected new or changed devices on smithi129 2024-09-21T09:37:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:55 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:55 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:55 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:55 smithi129 ceph-mon[32729]: Adjusting osd_memory_target on smithi129 to 17088M 2024-09-21T09:37:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:55 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:55 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:55.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:55 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:37:55.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:55 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:55.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:55 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:37:55.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:55 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:55.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:55 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:37:55.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:55 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[29382]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T09:37:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[29382]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:37:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[29382]: Detected new or changed devices on smithi129 2024-09-21T09:37:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[29382]: Adjusting osd_memory_target on smithi129 to 17088M 2024-09-21T09:37:55.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:55.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:55.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:37:55.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:55.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:37:55.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:55.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:37:55.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:55.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[41185]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T09:37:55.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[41185]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:37:55.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[41185]: Detected new or changed devices on smithi129 2024-09-21T09:37:55.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:55.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:55.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:37:55.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[41185]: Adjusting osd_memory_target on smithi129 to 17088M 2024-09-21T09:37:55.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:55.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:55.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:37:55.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:55.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:37:55.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:55.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:37:55.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:55 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:37:55.961 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:37:55.980 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi129:vg_nvme/lv_3 2024-09-21T09:37:56.221 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:37:56.565 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:56 smithi129 ceph-mon[32729]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T09:37:56.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:56 smithi107 ceph-mon[29382]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T09:37:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:56 smithi107 ceph-mon[41185]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T09:37:57.546 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:57 smithi129 ceph-mon[32729]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:37:57.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:57 smithi107 ceph-mon[29382]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:37:57.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:57 smithi107 ceph-mon[41185]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:37:58.624 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:58 smithi129 ceph-mon[32729]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:37:58.625 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:58 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:37:58.625 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:58 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:37:58.625 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:58 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:58.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:58 smithi107 ceph-mon[29382]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:37:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:58 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:37:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:58 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:37:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:58 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:58 smithi107 ceph-mon[41185]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:37:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:58 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:37:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:58 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:37:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:58 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:37:59.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:37:59 smithi129 ceph-mon[32729]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:37:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:37:59 smithi107 ceph-mon[29382]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:37:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:37:59 smithi107 ceph-mon[41185]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:38:00.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:00 smithi129 ceph-mon[32729]: from='client.? 172.21.15.129:0/246564429' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9da25cd-1680-4d97-a9d1-98dded10766f"}]: dispatch 2024-09-21T09:38:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:00 smithi129 ceph-mon[32729]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9da25cd-1680-4d97-a9d1-98dded10766f"}]: dispatch 2024-09-21T09:38:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:00 smithi129 ceph-mon[32729]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9da25cd-1680-4d97-a9d1-98dded10766f"}]': finished 2024-09-21T09:38:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:00 smithi129 ceph-mon[32729]: osdmap e33: 6 total, 5 up, 6 in 2024-09-21T09:38:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:00 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:38:00.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:00 smithi107 ceph-mon[29382]: from='client.? 172.21.15.129:0/246564429' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9da25cd-1680-4d97-a9d1-98dded10766f"}]: dispatch 2024-09-21T09:38:00.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:00 smithi107 ceph-mon[29382]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9da25cd-1680-4d97-a9d1-98dded10766f"}]: dispatch 2024-09-21T09:38:00.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:00 smithi107 ceph-mon[29382]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9da25cd-1680-4d97-a9d1-98dded10766f"}]': finished 2024-09-21T09:38:00.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:00 smithi107 ceph-mon[29382]: osdmap e33: 6 total, 5 up, 6 in 2024-09-21T09:38:00.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:00 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:38:00.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:00 smithi107 ceph-mon[41185]: from='client.? 172.21.15.129:0/246564429' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9da25cd-1680-4d97-a9d1-98dded10766f"}]: dispatch 2024-09-21T09:38:00.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:00 smithi107 ceph-mon[41185]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f9da25cd-1680-4d97-a9d1-98dded10766f"}]: dispatch 2024-09-21T09:38:00.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:00 smithi107 ceph-mon[41185]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f9da25cd-1680-4d97-a9d1-98dded10766f"}]': finished 2024-09-21T09:38:00.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:00 smithi107 ceph-mon[41185]: osdmap e33: 6 total, 5 up, 6 in 2024-09-21T09:38:00.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:00 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:38:01.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:01 smithi107 ceph-mon[29382]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T09:38:01.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:01 smithi107 ceph-mon[29382]: from='client.? 172.21.15.129:0/2375101655' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:38:01.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:01 smithi107 ceph-mon[41185]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T09:38:01.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:01 smithi107 ceph-mon[41185]: from='client.? 172.21.15.129:0/2375101655' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:38:01.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:01 smithi129 ceph-mon[32729]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T09:38:01.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:01 smithi129 ceph-mon[32729]: from='client.? 172.21.15.129:0/2375101655' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:38:03.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:03 smithi129 ceph-mon[32729]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T09:38:04.108 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:03 smithi107 ceph-mon[41185]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T09:38:04.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:03 smithi107 ceph-mon[29382]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T09:38:05.997 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:05 smithi129 ceph-mon[32729]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-21T09:38:06.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:05 smithi107 ceph-mon[29382]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-21T09:38:06.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:05 smithi107 ceph-mon[41185]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-21T09:38:07.095 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:06 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-21T09:38:07.095 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:06 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:07.095 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:06 smithi129 ceph-mon[32729]: Deploying daemon osd.5 on smithi129 2024-09-21T09:38:07.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:06 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-21T09:38:07.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:06 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:07.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:06 smithi107 ceph-mon[29382]: Deploying daemon osd.5 on smithi129 2024-09-21T09:38:07.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:06 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-21T09:38:07.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:06 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:07.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:06 smithi107 ceph-mon[41185]: Deploying daemon osd.5 on smithi129 2024-09-21T09:38:07.815 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:07 smithi129 ceph-mon[32729]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:38:08.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:07 smithi107 ceph-mon[29382]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:38:08.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:07 smithi107 ceph-mon[41185]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:38:10.000 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:09 smithi129 ceph-mon[32729]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:38:10.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:09 smithi107 ceph-mon[29382]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:38:10.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:09 smithi107 ceph-mon[41185]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:38:10.995 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:10 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:38:10.995 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:10 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:10.995 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:10 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:11.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:10 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:38:11.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:10 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:11.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:10 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:11.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:10 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:38:11.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:10 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:11.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:10 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:11.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:11 smithi129 ceph-mon[32729]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:38:12.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:11 smithi107 ceph-mon[29382]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:38:12.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:11 smithi107 ceph-mon[41185]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:38:13.645 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 5 on host 'smithi129' 2024-09-21T09:38:13.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:13 smithi107 ceph-mon[29382]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:38:13.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:13.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:13 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:13.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:13 smithi107 ceph-mon[29382]: from='osd.5 [v2:172.21.15.129:6808/2733195233,v1:172.21.15.129:6809/2733195233]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:38:13.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:13 smithi107 ceph-mon[29382]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:38:13.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:13 smithi107 ceph-mon[41185]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:38:13.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:13 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:13.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:13 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:13.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:13 smithi107 ceph-mon[41185]: from='osd.5 [v2:172.21.15.129:6808/2733195233,v1:172.21.15.129:6809/2733195233]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:38:13.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:13 smithi107 ceph-mon[41185]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:38:13.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:13 smithi129 ceph-mon[32729]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:38:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:13 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:13 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:13 smithi129 ceph-mon[32729]: from='osd.5 [v2:172.21.15.129:6808/2733195233,v1:172.21.15.129:6809/2733195233]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:38:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:13 smithi129 ceph-mon[32729]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:38:14.190 DEBUG:teuthology.orchestra.run.smithi129:osd.5> sudo journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.5.service 2024-09-21T09:38:14.193 INFO:tasks.cephadm:Deploying osd.6 on smithi129 with /dev/vg_nvme/lv_2... 2024-09-21T09:38:14.193 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T09:38:14.494 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:38:15.545 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:15 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:15.545 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:15 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:15.545 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:15 smithi129 ceph-mon[32729]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T09:38:15.545 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:15 smithi129 ceph-mon[32729]: osdmap e34: 6 total, 5 up, 6 in 2024-09-21T09:38:15.546 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:15 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:38:15.546 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:15 smithi129 ceph-mon[32729]: from='osd.5 [v2:172.21.15.129:6808/2733195233,v1:172.21.15.129:6809/2733195233]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:38:15.546 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:15 smithi129 ceph-mon[32729]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:38:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:15 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:15 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:15 smithi107 ceph-mon[29382]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T09:38:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:15 smithi107 ceph-mon[29382]: osdmap e34: 6 total, 5 up, 6 in 2024-09-21T09:38:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:15 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:38:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:15 smithi107 ceph-mon[29382]: from='osd.5 [v2:172.21.15.129:6808/2733195233,v1:172.21.15.129:6809/2733195233]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:38:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:15 smithi107 ceph-mon[29382]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:38:15.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:15 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:15.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:15 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:15.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:15 smithi107 ceph-mon[41185]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T09:38:15.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:15 smithi107 ceph-mon[41185]: osdmap e34: 6 total, 5 up, 6 in 2024-09-21T09:38:15.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:15 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:38:15.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:15 smithi107 ceph-mon[41185]: from='osd.5 [v2:172.21.15.129:6808/2733195233,v1:172.21.15.129:6809/2733195233]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:38:15.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:15 smithi107 ceph-mon[41185]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:38:15.882 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 09:38:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T09:38:15.663+0000 7f17ee2bc640 -1 osd.5 0 waiting for initial osdmap 2024-09-21T09:38:15.883 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 09:38:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T09:38:15.667+0000 7f17e90f3640 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:38:16.500 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:16 smithi129 ceph-mon[32729]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:38:16.500 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:16 smithi129 ceph-mon[32729]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T09:38:16.501 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:16 smithi129 ceph-mon[32729]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T09:38:16.501 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:16 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:38:16.501 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:16 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:38:16.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:16 smithi107 ceph-mon[29382]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:38:16.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:16 smithi107 ceph-mon[29382]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T09:38:16.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:16 smithi107 ceph-mon[29382]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T09:38:16.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:16 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:38:16.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:16 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:38:16.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:16 smithi107 ceph-mon[41185]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:38:16.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:16 smithi107 ceph-mon[41185]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T09:38:16.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:16 smithi107 ceph-mon[41185]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T09:38:16.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:16 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:38:16.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:16 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:38:17.799 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:17 smithi129 ceph-mon[32729]: purged_snaps scrub starts 2024-09-21T09:38:17.799 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:17 smithi129 ceph-mon[32729]: purged_snaps scrub ok 2024-09-21T09:38:17.799 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:17 smithi129 ceph-mon[32729]: osd.5 [v2:172.21.15.129:6808/2733195233,v1:172.21.15.129:6809/2733195233] boot 2024-09-21T09:38:17.799 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:17 smithi129 ceph-mon[32729]: osdmap e36: 6 total, 6 up, 6 in 2024-09-21T09:38:17.799 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:17 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:38:17.799 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:17 smithi129 ceph-mon[32729]: pgmap v126: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:18.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:17 smithi107 ceph-mon[29382]: purged_snaps scrub starts 2024-09-21T09:38:18.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:17 smithi107 ceph-mon[29382]: purged_snaps scrub ok 2024-09-21T09:38:18.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:17 smithi107 ceph-mon[29382]: osd.5 [v2:172.21.15.129:6808/2733195233,v1:172.21.15.129:6809/2733195233] boot 2024-09-21T09:38:18.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:17 smithi107 ceph-mon[29382]: osdmap e36: 6 total, 6 up, 6 in 2024-09-21T09:38:18.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:17 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:38:18.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:17 smithi107 ceph-mon[29382]: pgmap v126: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:18.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:17 smithi107 ceph-mon[41185]: purged_snaps scrub starts 2024-09-21T09:38:18.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:17 smithi107 ceph-mon[41185]: purged_snaps scrub ok 2024-09-21T09:38:18.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:17 smithi107 ceph-mon[41185]: osd.5 [v2:172.21.15.129:6808/2733195233,v1:172.21.15.129:6809/2733195233] boot 2024-09-21T09:38:18.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:17 smithi107 ceph-mon[41185]: osdmap e36: 6 total, 6 up, 6 in 2024-09-21T09:38:18.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:17 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:38:18.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:17 smithi107 ceph-mon[41185]: pgmap v126: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:18 smithi129 ceph-mon[32729]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T09:38:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:18 smithi129 ceph-mon[32729]: osdmap e37: 6 total, 6 up, 6 in 2024-09-21T09:38:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:18 smithi129 ceph-mon[32729]: Detected new or changed devices on smithi129 2024-09-21T09:38:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:18 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:18 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:18.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:18 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:38:18.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:18 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:38:18.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:18 smithi129 ceph-mon[32729]: Adjusting osd_memory_target on smithi129 to 8544M 2024-09-21T09:38:18.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:18 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:18.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:18 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:18.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:18 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:38:18.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:18 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:18.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:18 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:38:18.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:18 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:18.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:18 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:38:18.923 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:18 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:19.011 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[29382]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T09:38:19.011 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[29382]: osdmap e37: 6 total, 6 up, 6 in 2024-09-21T09:38:19.011 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[29382]: Detected new or changed devices on smithi129 2024-09-21T09:38:19.011 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:19.011 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:19.011 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:38:19.012 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:38:19.012 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[29382]: Adjusting osd_memory_target on smithi129 to 8544M 2024-09-21T09:38:19.012 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:19.012 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:19.012 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:38:19.012 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:19.012 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:38:19.012 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:19.012 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:38:19.012 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:19.012 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[41185]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T09:38:19.012 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[41185]: osdmap e37: 6 total, 6 up, 6 in 2024-09-21T09:38:19.013 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[41185]: Detected new or changed devices on smithi129 2024-09-21T09:38:19.013 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:19.013 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:19.013 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:38:19.013 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:38:19.013 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[41185]: Adjusting osd_memory_target on smithi129 to 8544M 2024-09-21T09:38:19.013 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:19.013 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:19.013 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:38:19.013 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:19.013 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:38:19.013 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:19.013 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:38:19.014 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:18 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:19.412 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:38:19.434 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi129:vg_nvme/lv_2 2024-09-21T09:38:19.675 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:38:20.018 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:19 smithi129 ceph-mon[32729]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T09:38:20.104 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:19 smithi129 ceph-mon[32729]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:20.104 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:19 smithi107 ceph-mon[29382]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T09:38:20.105 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:19 smithi107 ceph-mon[29382]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:20.105 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:19 smithi107 ceph-mon[41185]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T09:38:20.105 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:19 smithi107 ceph-mon[41185]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:21.985 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:21 smithi129 ceph-mon[32729]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:21.985 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:21 smithi129 ceph-mon[32729]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:38:21.985 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:21 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:38:21.985 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:21 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:38:21.985 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:21 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:22.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:21 smithi107 ceph-mon[29382]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:22.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:21 smithi107 ceph-mon[29382]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:38:22.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:21 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:38:22.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:21 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:38:22.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:21 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:22.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:21 smithi107 ceph-mon[41185]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:22.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:21 smithi107 ceph-mon[41185]: from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:38:22.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:21 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:38:22.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:21 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:38:22.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:21 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:22.949 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:22 smithi129 ceph-mon[32729]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T09:38:22.949 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:22 smithi129 ceph-mon[32729]: Cluster is now healthy 2024-09-21T09:38:23.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:22 smithi107 ceph-mon[29382]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T09:38:23.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:22 smithi107 ceph-mon[29382]: Cluster is now healthy 2024-09-21T09:38:23.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:22 smithi107 ceph-mon[41185]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T09:38:23.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:22 smithi107 ceph-mon[41185]: Cluster is now healthy 2024-09-21T09:38:23.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:23 smithi129 ceph-mon[32729]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T09:38:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:23 smithi129 ceph-mon[32729]: from='client.? 172.21.15.129:0/3505054366' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6d2bd99-0178-409c-a548-cbc30b8b76fb"}]: dispatch 2024-09-21T09:38:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:23 smithi129 ceph-mon[32729]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6d2bd99-0178-409c-a548-cbc30b8b76fb"}]: dispatch 2024-09-21T09:38:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:23 smithi129 ceph-mon[32729]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6d2bd99-0178-409c-a548-cbc30b8b76fb"}]': finished 2024-09-21T09:38:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:23 smithi129 ceph-mon[32729]: osdmap e39: 7 total, 6 up, 7 in 2024-09-21T09:38:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:23 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:38:24.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:23 smithi107 ceph-mon[29382]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T09:38:24.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:23 smithi107 ceph-mon[29382]: from='client.? 172.21.15.129:0/3505054366' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6d2bd99-0178-409c-a548-cbc30b8b76fb"}]: dispatch 2024-09-21T09:38:24.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:23 smithi107 ceph-mon[29382]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6d2bd99-0178-409c-a548-cbc30b8b76fb"}]: dispatch 2024-09-21T09:38:24.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:23 smithi107 ceph-mon[29382]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6d2bd99-0178-409c-a548-cbc30b8b76fb"}]': finished 2024-09-21T09:38:24.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:23 smithi107 ceph-mon[29382]: osdmap e39: 7 total, 6 up, 7 in 2024-09-21T09:38:24.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:23 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:38:24.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:23 smithi107 ceph-mon[41185]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T09:38:24.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:23 smithi107 ceph-mon[41185]: from='client.? 172.21.15.129:0/3505054366' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6d2bd99-0178-409c-a548-cbc30b8b76fb"}]: dispatch 2024-09-21T09:38:24.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:23 smithi107 ceph-mon[41185]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d6d2bd99-0178-409c-a548-cbc30b8b76fb"}]: dispatch 2024-09-21T09:38:24.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:23 smithi107 ceph-mon[41185]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d6d2bd99-0178-409c-a548-cbc30b8b76fb"}]': finished 2024-09-21T09:38:24.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:23 smithi107 ceph-mon[41185]: osdmap e39: 7 total, 6 up, 7 in 2024-09-21T09:38:24.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:23 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:38:25.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:24 smithi107 ceph-mon[29382]: from='client.? 172.21.15.129:0/3125804664' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:38:25.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:24 smithi107 ceph-mon[41185]: from='client.? 172.21.15.129:0/3125804664' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:38:25.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:24 smithi129 ceph-mon[32729]: from='client.? 172.21.15.129:0/3125804664' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:38:26.064 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:25 smithi129 ceph-mon[32729]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-21T09:38:26.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:25 smithi107 ceph-mon[29382]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-21T09:38:26.108 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:25 smithi107 ceph-mon[41185]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-21T09:38:28.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:27 smithi107 ceph-mon[29382]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T09:38:28.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:27 smithi107 ceph-mon[41185]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T09:38:28.135 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:27 smithi129 ceph-mon[32729]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T09:38:29.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:29 smithi129 ceph-mon[32729]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:38:29.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:29 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-21T09:38:29.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:29 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:29.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:29 smithi129 ceph-mon[32729]: Deploying daemon osd.6 on smithi129 2024-09-21T09:38:30.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:29 smithi107 ceph-mon[29382]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:38:30.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:29 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-21T09:38:30.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:29 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:30.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:29 smithi107 ceph-mon[29382]: Deploying daemon osd.6 on smithi129 2024-09-21T09:38:30.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:29 smithi107 ceph-mon[41185]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:38:30.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:29 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-21T09:38:30.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:29 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:30.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:29 smithi107 ceph-mon[41185]: Deploying daemon osd.6 on smithi129 2024-09-21T09:38:31.986 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:31 smithi129 ceph-mon[32729]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:38:32.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:31 smithi107 ceph-mon[29382]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:38:32.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:31 smithi107 ceph-mon[41185]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:38:34.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:33 smithi107 ceph-mon[29382]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:34.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:33 smithi107 ceph-mon[41185]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:34.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:33 smithi129 ceph-mon[32729]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:34.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:34 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:38:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:34 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:34 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:34 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:38:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:34 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:34 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:35.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:34 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:38:35.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:34 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:35.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:34 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:36.129 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:35 smithi129 ceph-mon[32729]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:36.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:35 smithi107 ceph-mon[29382]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:36.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:35 smithi107 ceph-mon[41185]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:37.120 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 6 on host 'smithi129' 2024-09-21T09:38:37.735 DEBUG:teuthology.orchestra.run.smithi129:osd.6> sudo journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.6.service 2024-09-21T09:38:37.737 INFO:tasks.cephadm:Deploying osd.7 on smithi129 with /dev/vg_nvme/lv_1... 2024-09-21T09:38:37.737 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T09:38:38.026 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:38:38.381 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:38 smithi129 ceph-mon[32729]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:38.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:38 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:38.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:38 smithi129 ceph-mon[32729]: from='osd.6 [v2:172.21.15.129:6816/3400496608,v1:172.21.15.129:6817/3400496608]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:38:38.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:38 smithi129 ceph-mon[32729]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:38:38.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:38 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:38.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:38 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:38.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:38 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:38.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:38 smithi107 ceph-mon[29382]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:38.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:38 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:38.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:38 smithi107 ceph-mon[29382]: from='osd.6 [v2:172.21.15.129:6816/3400496608,v1:172.21.15.129:6817/3400496608]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:38:38.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:38 smithi107 ceph-mon[29382]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:38:38.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:38 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:38.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:38 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:38.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:38 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:38 smithi107 ceph-mon[41185]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:38 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:38 smithi107 ceph-mon[41185]: from='osd.6 [v2:172.21.15.129:6816/3400496608,v1:172.21.15.129:6817/3400496608]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:38:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:38 smithi107 ceph-mon[41185]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:38:38.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:38 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:38.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:38 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:38.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:38 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:39.288 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:39 smithi129 ceph-mon[32729]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T09:38:39.289 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:39 smithi129 ceph-mon[32729]: osdmap e40: 7 total, 6 up, 7 in 2024-09-21T09:38:39.289 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:39 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:38:39.289 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:39 smithi129 ceph-mon[32729]: from='osd.6 [v2:172.21.15.129:6816/3400496608,v1:172.21.15.129:6817/3400496608]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:38:39.289 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:39 smithi129 ceph-mon[32729]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:38:39.289 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 09:38:39 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T09:38:39.130+0000 7f9f2650a640 -1 osd.6 0 waiting for initial osdmap 2024-09-21T09:38:39.289 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 09:38:39 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T09:38:39.138+0000 7f9f21341640 -1 osd.6 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:38:39.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:39 smithi107 ceph-mon[29382]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T09:38:39.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:39 smithi107 ceph-mon[29382]: osdmap e40: 7 total, 6 up, 7 in 2024-09-21T09:38:39.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:39 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:38:39.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:39 smithi107 ceph-mon[29382]: from='osd.6 [v2:172.21.15.129:6816/3400496608,v1:172.21.15.129:6817/3400496608]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:38:39.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:39 smithi107 ceph-mon[29382]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:38:39.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:39 smithi107 ceph-mon[41185]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T09:38:39.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:39 smithi107 ceph-mon[41185]: osdmap e40: 7 total, 6 up, 7 in 2024-09-21T09:38:39.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:39 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:38:39.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:39 smithi107 ceph-mon[41185]: from='osd.6 [v2:172.21.15.129:6816/3400496608,v1:172.21.15.129:6817/3400496608]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:38:39.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:39 smithi107 ceph-mon[41185]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:38:40.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:40 smithi129 ceph-mon[32729]: purged_snaps scrub starts 2024-09-21T09:38:40.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:40 smithi129 ceph-mon[32729]: purged_snaps scrub ok 2024-09-21T09:38:40.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:40 smithi129 ceph-mon[32729]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:40.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:40 smithi129 ceph-mon[32729]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T09:38:40.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:40 smithi129 ceph-mon[32729]: osdmap e41: 7 total, 6 up, 7 in 2024-09-21T09:38:40.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:40 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:38:40.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:40 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:38:40.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:40 smithi129 ceph-mon[32729]: osd.6 [v2:172.21.15.129:6816/3400496608,v1:172.21.15.129:6817/3400496608] boot 2024-09-21T09:38:40.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:40 smithi129 ceph-mon[32729]: osdmap e42: 7 total, 7 up, 7 in 2024-09-21T09:38:40.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:40 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:38:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:40 smithi107 ceph-mon[29382]: purged_snaps scrub starts 2024-09-21T09:38:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:40 smithi107 ceph-mon[29382]: purged_snaps scrub ok 2024-09-21T09:38:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:40 smithi107 ceph-mon[29382]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:40 smithi107 ceph-mon[29382]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T09:38:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:40 smithi107 ceph-mon[29382]: osdmap e41: 7 total, 6 up, 7 in 2024-09-21T09:38:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:40 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:38:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:40 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:38:40.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:40 smithi107 ceph-mon[29382]: osd.6 [v2:172.21.15.129:6816/3400496608,v1:172.21.15.129:6817/3400496608] boot 2024-09-21T09:38:40.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:40 smithi107 ceph-mon[29382]: osdmap e42: 7 total, 7 up, 7 in 2024-09-21T09:38:40.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:40 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:38:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:40 smithi107 ceph-mon[41185]: purged_snaps scrub starts 2024-09-21T09:38:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:40 smithi107 ceph-mon[41185]: purged_snaps scrub ok 2024-09-21T09:38:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:40 smithi107 ceph-mon[41185]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:38:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:40 smithi107 ceph-mon[41185]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T09:38:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:40 smithi107 ceph-mon[41185]: osdmap e41: 7 total, 6 up, 7 in 2024-09-21T09:38:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:40 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:38:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:40 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:38:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:40 smithi107 ceph-mon[41185]: osd.6 [v2:172.21.15.129:6816/3400496608,v1:172.21.15.129:6817/3400496608] boot 2024-09-21T09:38:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:40 smithi107 ceph-mon[41185]: osdmap e42: 7 total, 7 up, 7 in 2024-09-21T09:38:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:40 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:38:40.962 INFO:teuthology.orchestra.run.smithi129.stdout: 2024-09-21T09:38:40.983 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi129:vg_nvme/lv_1 2024-09-21T09:38:41.509 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:38:42.199 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:42 smithi129 ceph-mon[32729]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:42.199 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:42 smithi129 ceph-mon[32729]: osdmap e43: 7 total, 7 up, 7 in 2024-09-21T09:38:42.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:42 smithi107 ceph-mon[29382]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:42 smithi107 ceph-mon[29382]: osdmap e43: 7 total, 7 up, 7 in 2024-09-21T09:38:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:42 smithi107 ceph-mon[41185]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:42 smithi107 ceph-mon[41185]: osdmap e43: 7 total, 7 up, 7 in 2024-09-21T09:38:43.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:43 smithi129 ceph-mon[32729]: osdmap e44: 7 total, 7 up, 7 in 2024-09-21T09:38:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:43 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:43 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:43 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:38:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:43 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:38:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:43 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:38:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:43 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:43 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:43 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:38:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:43 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:43 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:38:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:43 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:43.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:43 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:38:43.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:43 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:43.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[29382]: osdmap e44: 7 total, 7 up, 7 in 2024-09-21T09:38:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:38:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:38:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:38:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:38:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:38:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:43.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:38:43.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:43.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[41185]: osdmap e44: 7 total, 7 up, 7 in 2024-09-21T09:38:43.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:43.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:43.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:38:43.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:38:43.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:38:43.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:43.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:43.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:38:43.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:43.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:38:43.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:43.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:38:43.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:43 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:44.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:44 smithi129 ceph-mon[32729]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:44.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:44 smithi129 ceph-mon[32729]: Detected new or changed devices on smithi129 2024-09-21T09:38:44.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:44 smithi129 ceph-mon[32729]: Adjusting osd_memory_target on smithi129 to 5696M 2024-09-21T09:38:44.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:44 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:38:44.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:44 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:38:44.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:44 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:44.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:44 smithi107 ceph-mon[29382]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:44 smithi107 ceph-mon[29382]: Detected new or changed devices on smithi129 2024-09-21T09:38:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:44 smithi107 ceph-mon[29382]: Adjusting osd_memory_target on smithi129 to 5696M 2024-09-21T09:38:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:44 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:38:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:44 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:38:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:44 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:44.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:44 smithi107 ceph-mon[41185]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:44.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:44 smithi107 ceph-mon[41185]: Detected new or changed devices on smithi129 2024-09-21T09:38:44.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:44 smithi107 ceph-mon[41185]: Adjusting osd_memory_target on smithi129 to 5696M 2024-09-21T09:38:44.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:44 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:38:44.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:44 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:38:44.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:44 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:45.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:45 smithi129 ceph-mon[32729]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:38:45.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:45 smithi107 ceph-mon[29382]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:38:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:45 smithi107 ceph-mon[41185]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi129:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:38:46.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:46 smithi129 ceph-mon[32729]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:46.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:46 smithi129 ceph-mon[32729]: from='client.? 172.21.15.129:0/119005189' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c0a9723-0328-4134-b5d4-cf307839018a"}]: dispatch 2024-09-21T09:38:46.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:46 smithi129 ceph-mon[32729]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c0a9723-0328-4134-b5d4-cf307839018a"}]: dispatch 2024-09-21T09:38:46.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:46 smithi129 ceph-mon[32729]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c0a9723-0328-4134-b5d4-cf307839018a"}]': finished 2024-09-21T09:38:46.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:46 smithi129 ceph-mon[32729]: osdmap e45: 8 total, 7 up, 8 in 2024-09-21T09:38:46.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:46 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:38:46.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:46 smithi107 ceph-mon[29382]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:46.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:46 smithi107 ceph-mon[29382]: from='client.? 172.21.15.129:0/119005189' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c0a9723-0328-4134-b5d4-cf307839018a"}]: dispatch 2024-09-21T09:38:46.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:46 smithi107 ceph-mon[29382]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c0a9723-0328-4134-b5d4-cf307839018a"}]: dispatch 2024-09-21T09:38:46.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:46 smithi107 ceph-mon[29382]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c0a9723-0328-4134-b5d4-cf307839018a"}]': finished 2024-09-21T09:38:46.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:46 smithi107 ceph-mon[29382]: osdmap e45: 8 total, 7 up, 8 in 2024-09-21T09:38:46.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:46 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:38:46.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:46 smithi107 ceph-mon[41185]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:46.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:46 smithi107 ceph-mon[41185]: from='client.? 172.21.15.129:0/119005189' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c0a9723-0328-4134-b5d4-cf307839018a"}]: dispatch 2024-09-21T09:38:46.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:46 smithi107 ceph-mon[41185]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c0a9723-0328-4134-b5d4-cf307839018a"}]: dispatch 2024-09-21T09:38:46.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:46 smithi107 ceph-mon[41185]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c0a9723-0328-4134-b5d4-cf307839018a"}]': finished 2024-09-21T09:38:46.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:46 smithi107 ceph-mon[41185]: osdmap e45: 8 total, 7 up, 8 in 2024-09-21T09:38:46.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:46 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:38:47.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:47 smithi129 ceph-mon[32729]: from='client.? 172.21.15.129:0/2594433587' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:38:47.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:47 smithi107 ceph-mon[29382]: from='client.? 172.21.15.129:0/2594433587' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:38:47.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:47 smithi107 ceph-mon[41185]: from='client.? 172.21.15.129:0/2594433587' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:38:48.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:48 smithi129 ceph-mon[32729]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:48.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:48 smithi107 ceph-mon[29382]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:48.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:48 smithi107 ceph-mon[41185]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:50.411 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:50 smithi107 ceph-mon[29382]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:50.411 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:50 smithi107 ceph-mon[41185]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:50.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:50 smithi129 ceph-mon[32729]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:52.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:52 smithi129 ceph-mon[32729]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:52 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-21T09:38:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:52 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:52.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:52 smithi107 ceph-mon[29382]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:52.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:52 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-21T09:38:52.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:52 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:52.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:52 smithi107 ceph-mon[41185]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:52.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:52 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-21T09:38:52.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:52 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:38:53.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:53 smithi129 ceph-mon[32729]: Deploying daemon osd.7 on smithi129 2024-09-21T09:38:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:53 smithi107 ceph-mon[29382]: Deploying daemon osd.7 on smithi129 2024-09-21T09:38:53.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:53 smithi107 ceph-mon[41185]: Deploying daemon osd.7 on smithi129 2024-09-21T09:38:54.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:54 smithi129 ceph-mon[32729]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:54.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:54 smithi107 ceph-mon[29382]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:54.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:54 smithi107 ceph-mon[41185]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:56.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:56 smithi129 ceph-mon[32729]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:56.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:56 smithi107 ceph-mon[29382]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:56 smithi107 ceph-mon[41185]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:57.432 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:57 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:38:57.433 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:57 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:57.433 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:57 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:57.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:57 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:38:57.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:57 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:57.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:57 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:57.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:57 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:38:57.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:57 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:57.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:57 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:38:58.454 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:38:58 smithi129 ceph-mon[32729]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:58.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:38:58 smithi107 ceph-mon[29382]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:38:58.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:38:58 smithi107 ceph-mon[41185]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:39:00.257 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:00 smithi129 ceph-mon[32729]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:39:00.257 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:00 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:00.257 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:00 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:00.257 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:00 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:39:00.257 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:00 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:39:00.257 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:00 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:00.257 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:00 smithi129 ceph-mon[32729]: from='osd.7 [v2:172.21.15.129:6824/101668996,v1:172.21.15.129:6825/101668996]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:39:00.257 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:00 smithi129 ceph-mon[32729]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:39:00.293 INFO:teuthology.orchestra.run.smithi129.stdout:Created osd(s) 7 on host 'smithi129' 2024-09-21T09:39:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:00 smithi107 ceph-mon[29382]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:39:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:00 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:00 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:00 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:39:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:00 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:39:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:00 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:00 smithi107 ceph-mon[29382]: from='osd.7 [v2:172.21.15.129:6824/101668996,v1:172.21.15.129:6825/101668996]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:39:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:00 smithi107 ceph-mon[29382]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:39:00.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:00 smithi107 ceph-mon[41185]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:39:00.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:00 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:00.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:00 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:00.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:00 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:39:00.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:00 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:39:00.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:00 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:00.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:00 smithi107 ceph-mon[41185]: from='osd.7 [v2:172.21.15.129:6824/101668996,v1:172.21.15.129:6825/101668996]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:39:00.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:00 smithi107 ceph-mon[41185]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:39:00.849 DEBUG:teuthology.orchestra.run.smithi129:osd.7> sudo journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.7.service 2024-09-21T09:39:00.852 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-21T09:39:00.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd stat -f json 2024-09-21T09:39:01.106 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:39:01.346 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:01 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:39:01.346 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:01 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:01.346 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:01 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:01.346 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:01 smithi129 ceph-mon[32729]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T09:39:01.346 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:01 smithi129 ceph-mon[32729]: osdmap e46: 8 total, 7 up, 8 in 2024-09-21T09:39:01.346 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:01 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:39:01.346 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:01 smithi129 ceph-mon[32729]: from='osd.7 [v2:172.21.15.129:6824/101668996,v1:172.21.15.129:6825/101668996]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:39:01.346 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:01 smithi129 ceph-mon[32729]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:39:01.470 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:01 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:39:01.471 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:01 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:01.471 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:01 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:01.471 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:01 smithi107 ceph-mon[29382]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T09:39:01.471 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:01 smithi107 ceph-mon[29382]: osdmap e46: 8 total, 7 up, 8 in 2024-09-21T09:39:01.471 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:01 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:39:01.471 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:01 smithi107 ceph-mon[29382]: from='osd.7 [v2:172.21.15.129:6824/101668996,v1:172.21.15.129:6825/101668996]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:39:01.471 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:01 smithi107 ceph-mon[29382]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:39:01.471 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:01 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:39:01.471 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:01 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:01.472 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:01 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:01.472 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:01 smithi107 ceph-mon[41185]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T09:39:01.472 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:01 smithi107 ceph-mon[41185]: osdmap e46: 8 total, 7 up, 8 in 2024-09-21T09:39:01.472 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:01 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:39:01.472 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:01 smithi107 ceph-mon[41185]: from='osd.7 [v2:172.21.15.129:6824/101668996,v1:172.21.15.129:6825/101668996]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:39:01.472 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:01 smithi107 ceph-mon[41185]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]: dispatch 2024-09-21T09:39:02.011 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 09:39:01 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[56850]: 2024-09-21T09:39:01.742+0000 7fcad6766640 -1 osd.7 0 waiting for initial osdmap 2024-09-21T09:39:02.011 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 09:39:01 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[56850]: 2024-09-21T09:39:01.749+0000 7fcad2da0640 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:39:02.569 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:02 smithi107 ceph-mon[29382]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:39:02.569 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:02 smithi107 ceph-mon[29382]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T09:39:02.569 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:02 smithi107 ceph-mon[29382]: osdmap e47: 8 total, 7 up, 8 in 2024-09-21T09:39:02.569 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:02 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:39:02.569 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:02 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:39:02.570 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:02 smithi107 ceph-mon[41185]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:39:02.570 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:02 smithi107 ceph-mon[41185]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T09:39:02.570 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:02 smithi107 ceph-mon[41185]: osdmap e47: 8 total, 7 up, 8 in 2024-09-21T09:39:02.570 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:02 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:39:02.570 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:02 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:39:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:02 smithi129 ceph-mon[32729]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:39:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:02 smithi129 ceph-mon[32729]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]}]': finished 2024-09-21T09:39:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:02 smithi129 ceph-mon[32729]: osdmap e47: 8 total, 7 up, 8 in 2024-09-21T09:39:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:02 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:39:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:02 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:39:02.943 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:39:03.571 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":48,"num_osds":8,"num_up_osds":8,"osd_up_since":1726911542,"num_in_osds":8,"osd_in_since":1726911526,"num_remapped_pgs":1} 2024-09-21T09:39:03.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:39:03.812 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:39:03.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[41185]: purged_snaps scrub starts 2024-09-21T09:39:03.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[41185]: purged_snaps scrub ok 2024-09-21T09:39:03.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[41185]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:39:03.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[41185]: osd.7 [v2:172.21.15.129:6824/101668996,v1:172.21.15.129:6825/101668996] boot 2024-09-21T09:39:03.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[41185]: osdmap e48: 8 total, 8 up, 8 in 2024-09-21T09:39:03.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:39:03.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3716605551' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T09:39:03.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:03.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:03.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:03.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:03.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:03.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:03.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:03.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:39:03.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:39:03.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:03.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[41185]: osdmap e49: 8 total, 8 up, 8 in 2024-09-21T09:39:03.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[29382]: purged_snaps scrub starts 2024-09-21T09:39:03.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[29382]: purged_snaps scrub ok 2024-09-21T09:39:03.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[29382]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:39:03.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[29382]: osd.7 [v2:172.21.15.129:6824/101668996,v1:172.21.15.129:6825/101668996] boot 2024-09-21T09:39:03.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[29382]: osdmap e48: 8 total, 8 up, 8 in 2024-09-21T09:39:03.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:39:03.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3716605551' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T09:39:03.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:03.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:03.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:03.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:03.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:03.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:03.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:03.864 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:39:03.864 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:39:03.864 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:03.864 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:03 smithi107 ceph-mon[29382]: osdmap e49: 8 total, 8 up, 8 in 2024-09-21T09:39:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:03 smithi129 ceph-mon[32729]: purged_snaps scrub starts 2024-09-21T09:39:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:03 smithi129 ceph-mon[32729]: purged_snaps scrub ok 2024-09-21T09:39:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:03 smithi129 ceph-mon[32729]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:39:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:03 smithi129 ceph-mon[32729]: osd.7 [v2:172.21.15.129:6824/101668996,v1:172.21.15.129:6825/101668996] boot 2024-09-21T09:39:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:03 smithi129 ceph-mon[32729]: osdmap e48: 8 total, 8 up, 8 in 2024-09-21T09:39:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:03 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:39:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:03 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3716605551' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T09:39:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:03 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:03 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:03 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:03 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:03 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:04.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:03 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:04.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:03 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:04.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:03 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:39:04.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:03 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:39:04.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:03 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:04.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:03 smithi129 ceph-mon[32729]: osdmap e49: 8 total, 8 up, 8 in 2024-09-21T09:39:04.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:04 smithi107 ceph-mon[29382]: Detected new or changed devices on smithi129 2024-09-21T09:39:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:04 smithi107 ceph-mon[29382]: Adjusting osd_memory_target on smithi129 to 4272M 2024-09-21T09:39:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:04 smithi107 ceph-mon[29382]: osdmap e50: 8 total, 8 up, 8 in 2024-09-21T09:39:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:04 smithi107 ceph-mon[41185]: Detected new or changed devices on smithi129 2024-09-21T09:39:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:04 smithi107 ceph-mon[41185]: Adjusting osd_memory_target on smithi129 to 4272M 2024-09-21T09:39:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:04 smithi107 ceph-mon[41185]: osdmap e50: 8 total, 8 up, 8 in 2024-09-21T09:39:05.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:04 smithi129 ceph-mon[32729]: Detected new or changed devices on smithi129 2024-09-21T09:39:05.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:04 smithi129 ceph-mon[32729]: Adjusting osd_memory_target on smithi129 to 4272M 2024-09-21T09:39:05.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:04 smithi129 ceph-mon[32729]: osdmap e50: 8 total, 8 up, 8 in 2024-09-21T09:39:05.541 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:39:05.541 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":50,"fsid":"86c49514-77fc-11ef-baf6-efdc52797490","created":"2024-09-21T09:34:03.635940+0000","modified":"2024-09-21T09:39:04.433428+0000","last_up_change":"2024-09-21T09:39:02.736647+0000","last_in_change":"2024-09-21T09:38:46.049422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:37:08.948207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"635c1eb0-29b1-4fdd-bd47-881ef05e49bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":174831156},{"type":"v1","addr":"172.21.15.107:6803","nonce":174831156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":174831156},{"type":"v1","addr":"172.21.15.107:6805","nonce":174831156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":174831156},{"type":"v1","addr":"172.21.15.107:6809","nonce":174831156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":174831156},{"type":"v1","addr":"172.21.15.107:6807","nonce":174831156}]},"public_addr":"172.21.15.107:6803/174831156","cluster_addr":"172.21.15.107:6805/174831156","heartbeat_back_addr":"172.21.15.107:6809/174831156","heartbeat_front_addr":"172.21.15.107:6807/174831156","state":["exists","up"]},{"osd":1,"uuid":"a39b6a18-939d-4999-89b2-1126c1b2d1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1227448003},{"type":"v1","addr":"172.21.15.107:6811","nonce":1227448003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1227448003},{"type":"v1","addr":"172.21.15.107:6813","nonce":1227448003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1227448003},{"type":"v1","addr":"172.21.15.107:6817","nonce":1227448003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1227448003},{"type":"v1","addr":"172.21.15.107:6815","nonce":1227448003}]},"public_addr":"172.21.15.107:6811/1227448003","cluster_addr":"172.21.15.107:6813/1227448003","heartbeat_back_addr":"172.21.15.107:6817/1227448003","heartbeat_front_addr":"172.21.15.107:6815/1227448003","state":["exists","up"]},{"osd":2,"uuid":"b01c5f1b-77e4-462f-80d0-fd1d3afb786b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3613749103},{"type":"v1","addr":"172.21.15.107:6819","nonce":3613749103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3613749103},{"type":"v1","addr":"172.21.15.107:6821","nonce":3613749103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3613749103},{"type":"v1","addr":"172.21.15.107:6825","nonce":3613749103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3613749103},{"type":"v1","addr":"172.21.15.107:6823","nonce":3613749103}]},"public_addr":"172.21.15.107:6819/3613749103","cluster_addr":"172.21.15.107:6821/3613749103","heartbeat_back_addr":"172.21.15.107:6825/3613749103","heartbeat_front_addr":"172.21.15.107:6823/3613749103","state":["exists","up"]},{"osd":3,"uuid":"5135d269-b6c4-4a9d-bcf9-ffe19bc5c9ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3206618601},{"type":"v1","addr":"172.21.15.107:6827","nonce":3206618601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":3206618601},{"type":"v1","addr":"172.21.15.107:6829","nonce":3206618601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3206618601},{"type":"v1","addr":"172.21.15.107:6833","nonce":3206618601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3206618601},{"type":"v1","addr":"172.21.15.107:6831","nonce":3206618601}]},"public_addr":"172.21.15.107:6827/3206618601","cluster_addr":"172.21.15.107:6829/3206618601","heartbeat_back_addr":"172.21.15.107:6833/3206618601","heartbeat_front_addr":"172.21.15.107:6831/3206618601","state":["exists","up"]},{"osd":4,"uuid":"23242972-e341-4773-809f-299aabeacd56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6801","nonce":4101249335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6803","nonce":4101249335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6807","nonce":4101249335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6805","nonce":4101249335}]},"public_addr":"172.21.15.129:6801/4101249335","cluster_addr":"172.21.15.129:6803/4101249335","heartbeat_back_addr":"172.21.15.129:6807/4101249335","heartbeat_front_addr":"172.21.15.129:6805/4101249335","state":["exists","up"]},{"osd":5,"uuid":"f9da25cd-1680-4d97-a9d1-98dded10766f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6809","nonce":2733195233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6811","nonce":2733195233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6815","nonce":2733195233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6813","nonce":2733195233}]},"public_addr":"172.21.15.129:6809/2733195233","cluster_addr":"172.21.15.129:6811/2733195233","heartbeat_back_addr":"172.21.15.129:6815/2733195233","heartbeat_front_addr":"172.21.15.129:6813/2733195233","state":["exists","up"]},{"osd":6,"uuid":"d6d2bd99-0178-409c-a548-cbc30b8b76fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6817","nonce":3400496608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6819","nonce":3400496608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6823","nonce":3400496608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6821","nonce":3400496608}]},"public_addr":"172.21.15.129:6817/3400496608","cluster_addr":"172.21.15.129:6819/3400496608","heartbeat_back_addr":"172.21.15.129:6823/3400496608","heartbeat_front_addr":"172.21.15.129:6821/3400496608","state":["exists","up"]},{"osd":7,"uuid":"2c0a9723-0328-4134-b5d4-cf307839018a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6825","nonce":101668996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6827","nonce":101668996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6831","nonce":101668996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6829","nonce":101668996}]},"public_addr":"172.21.15.129:6825/101668996","cluster_addr":"172.21.15.129:6827/101668996","heartbeat_back_addr":"172.21.15.129:6831/101668996","heartbeat_front_addr":"172.21.15.129:6829/101668996","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:36:18.300841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:36:42.334925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:06.141467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:30.344630+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:51.661657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:38:14.654068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:38:38.089961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:39:01.119878+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/3424783814":"2024-09-22T09:34:56.643317+0000","172.21.15.107:6801/4208383830":"2024-09-22T09:34:56.643317+0000","172.21.15.107:6800/4208383830":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/837896416":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/1942117255":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/2492859118":"2024-09-22T09:34:34.697433+0000","172.21.15.107:6801/1686228827":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/782384657":"2024-09-22T09:34:56.643317+0000","172.21.15.107:6800/1686228827":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/637981220":"2024-09-22T09:34:34.697433+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:39:05.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:05 smithi107 ceph-mon[29382]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:05.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:05 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1487667432' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:39:05.796 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:05 smithi107 ceph-mon[41185]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:05.796 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:05 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1487667432' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:39:06.070 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:05 smithi129 ceph-mon[32729]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:06.070 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:05 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1487667432' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:39:06.178 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T09:37:08.948207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-09-21T09:39:06.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-21T09:39:06.418 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:39:07.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:07 smithi107 ceph-mon[29382]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:07.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:07 smithi107 ceph-mon[41185]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:08.167 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-09-21T09:39:08.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:07 smithi129 ceph-mon[32729]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:08.791 INFO:tasks.cephadm:Adding prometheus.a on smithi129 2024-09-21T09:39:08.791 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph orch apply prometheus '1;smithi129=a' 2024-09-21T09:39:09.039 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:39:09.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:08 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/629971186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-21T09:39:09.108 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:08 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/629971186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-21T09:39:09.159 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:08 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/629971186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-21T09:39:09.845 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:09 smithi129 ceph-mon[32729]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:10.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:09 smithi107 ceph-mon[29382]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:10.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:09 smithi107 ceph-mon[41185]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:10.702 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled prometheus update... 2024-09-21T09:39:11.331 DEBUG:teuthology.orchestra.run.smithi129:prometheus.a> sudo journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@prometheus.a.service 2024-09-21T09:39:11.333 INFO:tasks.cephadm:Adding node-exporter.a on smithi107 2024-09-21T09:39:11.334 INFO:tasks.cephadm:Adding node-exporter.b on smithi129 2024-09-21T09:39:11.334 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph orch apply node-exporter '2;smithi107=a;smithi129=b' 2024-09-21T09:39:11.617 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:39:12.039 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:11 smithi129 ceph-mon[32729]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:12.039 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:11 smithi129 ceph-mon[32729]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:39:12.039 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:11 smithi129 ceph-mon[32729]: Saving service prometheus spec with placement smithi129=a;count:1 2024-09-21T09:39:12.039 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:11 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:12.039 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:11 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:39:12.039 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:11 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:39:12.039 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:11 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:39:12.040 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:11 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:12.040 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:11 smithi129 ceph-mon[32729]: Deploying daemon prometheus.a on smithi129 2024-09-21T09:39:12.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:11 smithi107 ceph-mon[29382]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:12.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:11 smithi107 ceph-mon[29382]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:39:12.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:11 smithi107 ceph-mon[29382]: Saving service prometheus spec with placement smithi129=a;count:1 2024-09-21T09:39:12.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:11 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:12.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:11 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:39:12.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:11 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:39:12.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:11 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:39:12.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:11 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:12.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:11 smithi107 ceph-mon[29382]: Deploying daemon prometheus.a on smithi129 2024-09-21T09:39:12.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:11 smithi107 ceph-mon[41185]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:12.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:11 smithi107 ceph-mon[41185]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:39:12.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:11 smithi107 ceph-mon[41185]: Saving service prometheus spec with placement smithi129=a;count:1 2024-09-21T09:39:12.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:11 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:12.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:11 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:39:12.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:11 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:39:12.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:11 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:39:12.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:11 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:12.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:11 smithi107 ceph-mon[41185]: Deploying daemon prometheus.a on smithi129 2024-09-21T09:39:13.958 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:13 smithi129 ceph-mon[32729]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:14.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:13 smithi107 ceph-mon[29382]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:14.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:13 smithi107 ceph-mon[41185]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:14.663 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled node-exporter update... 2024-09-21T09:39:15.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:15 smithi129 ceph-mon[32729]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi107=a;smithi129=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:39:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:15 smithi129 ceph-mon[32729]: Saving service node-exporter spec with placement smithi107=a;smithi129=b;count:2 2024-09-21T09:39:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:15 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:15 smithi129 ceph-mon[32729]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:16.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:15 smithi107 ceph-mon[29382]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi107=a;smithi129=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:39:16.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:15 smithi107 ceph-mon[29382]: Saving service node-exporter spec with placement smithi107=a;smithi129=b;count:2 2024-09-21T09:39:16.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:15 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:16.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:15 smithi107 ceph-mon[29382]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:16.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:15 smithi107 ceph-mon[41185]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi107=a;smithi129=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:39:16.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:15 smithi107 ceph-mon[41185]: Saving service node-exporter spec with placement smithi107=a;smithi129=b;count:2 2024-09-21T09:39:16.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:15 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:16.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:15 smithi107 ceph-mon[41185]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:16.746 DEBUG:teuthology.orchestra.run.smithi107:node-exporter.a> sudo journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@node-exporter.a.service 2024-09-21T09:39:16.749 DEBUG:teuthology.orchestra.run.smithi129:node-exporter.b> sudo journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@node-exporter.b.service 2024-09-21T09:39:16.754 INFO:tasks.cephadm:Adding alertmanager.a on smithi107 2024-09-21T09:39:16.754 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph orch apply alertmanager '1;smithi107=a' 2024-09-21T09:39:17.032 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:39:18.015 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:17 smithi129 ceph-mon[32729]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:18.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:17 smithi107 ceph-mon[29382]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:18.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:17 smithi107 ceph-mon[41185]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:20.038 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:19 smithi129 ceph-mon[32729]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:20.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:19 smithi107 ceph-mon[29382]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:20.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:19 smithi107 ceph-mon[41185]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:20.413 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled alertmanager update... 2024-09-21T09:39:21.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:21 smithi129 ceph-mon[32729]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi107=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:39:21.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:21 smithi129 ceph-mon[32729]: Saving service alertmanager spec with placement smithi107=a;count:1 2024-09-21T09:39:21.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:21 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:21.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:21 smithi129 ceph-mon[32729]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:21.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:21 smithi107 ceph-mon[29382]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi107=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:39:21.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:21 smithi107 ceph-mon[29382]: Saving service alertmanager spec with placement smithi107=a;count:1 2024-09-21T09:39:21.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:21 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:21.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:21 smithi107 ceph-mon[29382]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:21 smithi107 ceph-mon[41185]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi107=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:39:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:21 smithi107 ceph-mon[41185]: Saving service alertmanager spec with placement smithi107=a;count:1 2024-09-21T09:39:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:21 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:21 smithi107 ceph-mon[41185]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:22.320 DEBUG:teuthology.orchestra.run.smithi107:alertmanager.a> sudo journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@alertmanager.a.service 2024-09-21T09:39:22.323 INFO:tasks.cephadm:Adding grafana.a on smithi129 2024-09-21T09:39:22.323 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph orch apply grafana '1;smithi129=a' 2024-09-21T09:39:22.907 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:39:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:23 smithi129 ceph-mon[32729]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:24.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:23 smithi107 ceph-mon[29382]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:24.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:23 smithi107 ceph-mon[41185]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:24.555 INFO:teuthology.orchestra.run.smithi129.stdout:Scheduled grafana update... 2024-09-21T09:39:25.263 DEBUG:teuthology.orchestra.run.smithi129:grafana.a> sudo journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@grafana.a.service 2024-09-21T09:39:25.266 INFO:tasks.cephadm:Setting up client nodes... 2024-09-21T09:39:25.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T09:39:25.514 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:39:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[61890]: ts=2024-09-21T09:39:25.471Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T09:39:25.514 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:39:25 smithi129 bash[61862]: 183f60fc314a1b323fd22573c03f93458ae3cc0cc8a4b48e79dfe2cfb6c5cc55 2024-09-21T09:39:25.517 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:39:25.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:25 smithi107 ceph-mon[41185]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:39:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:25 smithi107 ceph-mon[41185]: Saving service grafana spec with placement smithi129=a;count:1 2024-09-21T09:39:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:25 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:25 smithi107 ceph-mon[41185]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:25 smithi107 ceph-mon[29382]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:39:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:25 smithi107 ceph-mon[29382]: Saving service grafana spec with placement smithi129=a;count:1 2024-09-21T09:39:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:25 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:25 smithi107 ceph-mon[29382]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:25.918 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:39:25 smithi129 systemd[1]: Started Ceph prometheus.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:39:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:25 smithi129 ceph-mon[32729]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi129=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:39:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:25 smithi129 ceph-mon[32729]: Saving service grafana spec with placement smithi129=a;count:1 2024-09-21T09:39:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:25 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:25 smithi129 ceph-mon[32729]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:26.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:26 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:26.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:26 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:26.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:26 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:26.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:26 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-21T09:39:26.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:26 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:26.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:26 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:26.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:26 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:26.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:26 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-21T09:39:26.853 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:26 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:26.853 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:26 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:26.853 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:26 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' 2024-09-21T09:39:26.853 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:26 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-21T09:39:26.854 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ignoring --setuser ceph since I am not root 2024-09-21T09:39:26.854 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ignoring --setgroup ceph since I am not root 2024-09-21T09:39:26.854 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:26 smithi129 ceph-mgr[34801]: -- 172.21.15.129:0/2324362873 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5580cc3f4340 con 0x5580cc401000 2024-09-21T09:39:26.854 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:26.731+0000 7fc9d8a6fec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:39:26.854 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:26.785+0000 7fc9d8a6fec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:39:26.854 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:26.850+0000 7fc9d8a6fec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:39:26.918 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ignoring --setuser ceph since I am not root 2024-09-21T09:39:26.919 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ignoring --setgroup ceph since I am not root 2024-09-21T09:39:26.919 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:26 smithi107 ceph-mgr[29656]: -- 172.21.15.107:0/471243240 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559c524d6340 con 0x559c524e3000 2024-09-21T09:39:26.919 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:26.729+0000 7fd770a5cec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:39:26.919 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:26.786+0000 7fd770a5cec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:39:26.919 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:26.855+0000 7fd770a5cec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:39:27.111 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:26.974+0000 7fc9d8a6fec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:39:27.243 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:26.979+0000 7fd770a5cec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:39:27.243 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:27 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:27.122+0000 7fd770a5cec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:39:27.365 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:27 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:27.109+0000 7fc9d8a6fec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:39:27.478 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2024-09-21T09:39:27.478 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBPlO5mrIE0HBAAxLo9q3YE784SBujZfcOrMQ== 2024-09-21T09:39:27.591 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:27 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:27.384+0000 7fd770a5cec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:39:27.592 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:27 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:27.444+0000 7fd770a5cec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:39:27.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:27 smithi129 ceph-mon[32729]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T09:39:27.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:27 smithi129 ceph-mon[32729]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T09:39:27.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:27 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1361470048' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:39:27.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:27 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1361470048' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:39:27.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:27 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:27.362+0000 7fc9d8a6fec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:39:27.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:27 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:27.422+0000 7fc9d8a6fec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:39:27.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:27 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:27.582+0000 7fc9d8a6fec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:39:27.858 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:27 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:27.619+0000 7fd770a5cec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:39:27.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:27 smithi107 ceph-mon[41185]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T09:39:27.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:27 smithi107 ceph-mon[41185]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T09:39:27.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:27 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1361470048' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:39:27.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:27 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1361470048' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:39:27.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:27 smithi107 ceph-mon[29382]: from='mgr.14150 172.21.15.107:0/2894414677' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T09:39:27.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:27 smithi107 ceph-mon[29382]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T09:39:27.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:27 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1361470048' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:39:27.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:27 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1361470048' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:39:28.088 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T09:39:28.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-21T09:39:28.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T09:39:28.126 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T09:39:28.379 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:39:28.733 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:28 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:28.448+0000 7fc9d8a6fec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:39:28.733 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:28 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:28.514+0000 7fc9d8a6fec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:39:28.858 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:28 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:28.470+0000 7fd770a5cec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:39:28.858 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:28 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:28.534+0000 7fd770a5cec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:39:29.472 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:29 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:29.238+0000 7fc9d8a6fec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:39:29.472 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:29 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:29.352+0000 7fc9d8a6fec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:39:29.522 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:29 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:29.232+0000 7fd770a5cec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:39:29.522 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:29 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:29.341+0000 7fd770a5cec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:39:29.522 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:29 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:29.461+0000 7fd770a5cec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:39:29.742 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:29 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:29.475+0000 7fc9d8a6fec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:39:29.742 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:29 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:29.540+0000 7fc9d8a6fec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:39:29.742 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:29 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:29.671+0000 7fc9d8a6fec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:39:29.742 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:29 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:29.740+0000 7fc9d8a6fec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:39:29.858 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:29 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:29.520+0000 7fd770a5cec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:39:29.858 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:29 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:29.645+0000 7fd770a5cec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:39:29.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:29 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:29.710+0000 7fd770a5cec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:39:29.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:29 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:29.767+0000 7fd770a5cec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:39:30.012 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:29 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:29.804+0000 7fc9d8a6fec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:39:30.012 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:29 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:29.943+0000 7fc9d8a6fec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:39:30.012 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:30.010+0000 7fc9d8a6fec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:39:30.164 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:29 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:29.900+0000 7fd770a5cec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:39:30.164 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:29 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:29.963+0000 7fd770a5cec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:39:30.164 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:30 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:30.027+0000 7fd770a5cec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:39:30.164 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:30 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:30.106+0000 7fd770a5cec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:39:30.164 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:30 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:30.162+0000 7fd770a5cec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:39:30.212 INFO:teuthology.orchestra.run.smithi129.stdout:[client.1] 2024-09-21T09:39:30.212 INFO:teuthology.orchestra.run.smithi129.stdout: key = AQBSlO5mGVhiDBAAQGjvmsxz7Ah4xMEf97ra8g== 2024-09-21T09:39:30.283 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:30 smithi129 ceph-mon[32729]: from='client.? 172.21.15.129:0/2319810274' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:39:30.283 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:30 smithi129 ceph-mon[32729]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:39:30.283 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:30 smithi129 ceph-mon[32729]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:39:30.283 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:30.077+0000 7fc9d8a6fec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:39:30.283 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:30.161+0000 7fc9d8a6fec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:39:30.283 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:30.221+0000 7fc9d8a6fec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:39:30.283 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:30.281+0000 7fc9d8a6fec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:39:30.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:30 smithi107 ceph-mon[29382]: from='client.? 172.21.15.129:0/2319810274' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:39:30.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:30 smithi107 ceph-mon[29382]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:39:30.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:30 smithi107 ceph-mon[29382]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:39:30.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:30 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:30.219+0000 7fd770a5cec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:39:30.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:30 smithi107 ceph-mon[41185]: from='client.? 172.21.15.129:0/2319810274' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:39:30.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:30 smithi107 ceph-mon[41185]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:39:30.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:30 smithi107 ceph-mon[41185]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:39:30.771 DEBUG:teuthology.orchestra.run.smithi129:> set -ex 2024-09-21T09:39:30.771 DEBUG:teuthology.orchestra.run.smithi129:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-21T09:39:30.771 DEBUG:teuthology.orchestra.run.smithi129:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-21T09:39:30.809 INFO:tasks.cephadm:Creating RBD pool 2024-09-21T09:39:30.809 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool create rbd 8 2024-09-21T09:39:31.079 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:39:31.108 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:30 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:30.972+0000 7fd770a5cec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:39:31.108 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:39:31] ENGINE Bus STARTING 2024-09-21T09:39:31.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: CherryPy Checker: 2024-09-21T09:39:31.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: The Application mounted at '' has an empty config. 2024-09-21T09:39:31.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:31.315 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:31.047+0000 7fc9d8a6fec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:39:31.315 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: [21/Sep/2024:09:39:31] ENGINE Bus STARTING 2024-09-21T09:39:31.315 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: CherryPy Checker: 2024-09-21T09:39:31.315 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: The Application mounted at '' has an empty config. 2024-09-21T09:39:31.315 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: 2024-09-21T09:39:31.315 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: [21/Sep/2024:09:39:31] ENGINE Serving on http://:::9283 2024-09-21T09:39:31.315 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: [21/Sep/2024:09:39:31] ENGINE Bus STARTED 2024-09-21T09:39:31.316 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: Active manager daemon y restarted 2024-09-21T09:39:31.316 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: Activating manager daemon y 2024-09-21T09:39:31.316 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: osdmap e51: 8 total, 8 up, 8 in 2024-09-21T09:39:31.316 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: mgrmap e17: y(active, starting, since 0.0124271s), standbys: x 2024-09-21T09:39:31.316 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:39:31.316 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:39:31.316 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:39:31.316 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:39:31.316 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:39:31.316 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:39:31.316 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:39:31.316 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:39:31.316 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:39:31.317 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:39:31.317 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:39:31.317 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:39:31.317 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:39:31.317 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:39:31.317 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:39:31.317 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:39:31.317 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: Manager daemon y is now available 2024-09-21T09:39:31.317 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:39:31.317 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:31.317 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:39:31.317 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: Standby manager daemon x restarted 2024-09-21T09:39:31.317 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: Standby manager daemon x started 2024-09-21T09:39:31.317 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/2137358010' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:39:31.318 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/2137358010' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:39:31.318 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:39:31.318 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/2137358010' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:39:31.318 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/2137358010' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:39:31.318 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:39:31.484 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:39:31] ENGINE Serving on http://:::9283 2024-09-21T09:39:31.485 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:39:31] ENGINE Bus STARTED 2024-09-21T09:39:31.485 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: Active manager daemon y restarted 2024-09-21T09:39:31.485 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: Activating manager daemon y 2024-09-21T09:39:31.485 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: osdmap e51: 8 total, 8 up, 8 in 2024-09-21T09:39:31.485 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: mgrmap e17: y(active, starting, since 0.0124271s), standbys: x 2024-09-21T09:39:31.485 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:39:31.485 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:39:31.485 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:39:31.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:39:31.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:39:31.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:39:31.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:39:31.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:39:31.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:39:31.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:39:31.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:39:31.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:39:31.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:39:31.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:39:31.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:39:31.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:39:31.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: Manager daemon y is now available 2024-09-21T09:39:31.487 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:39:31.487 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:31.487 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:39:31.487 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: Standby manager daemon x restarted 2024-09-21T09:39:31.487 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: Standby manager daemon x started 2024-09-21T09:39:31.487 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/2137358010' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:39:31.487 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/2137358010' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:39:31.487 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:39:31.487 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: Active manager daemon y restarted 2024-09-21T09:39:31.487 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: Activating manager daemon y 2024-09-21T09:39:31.487 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: osdmap e51: 8 total, 8 up, 8 in 2024-09-21T09:39:31.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: mgrmap e17: y(active, starting, since 0.0124271s), standbys: x 2024-09-21T09:39:31.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:39:31.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:39:31.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:39:31.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:39:31.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:39:31.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:39:31.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:39:31.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:39:31.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:39:31.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:39:31.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:39:31.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:39:31.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:39:31.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:39:31.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:39:31.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:39:31.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: Manager daemon y is now available 2024-09-21T09:39:31.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:39:31.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:31.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:39:31.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: Standby manager daemon x restarted 2024-09-21T09:39:31.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: Standby manager daemon x started 2024-09-21T09:39:31.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.129:0/2137358010' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:39:31.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.129:0/2137358010' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:39:31.489 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:39:31.490 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/2137358010' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:39:31.490 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/2137358010' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:39:31.490 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:39:31.490 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.129:0/2137358010' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:39:31.490 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.129:0/2137358010' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:39:31.490 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:39:33.144 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:32.884+0000 7fd71381f640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:39:32] ENGINE Error in HTTPServer.serve 2024-09-21T09:39:33.144 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: Traceback (most recent call last): 2024-09-21T09:39:33.144 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:39:33.144 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: self._connections.run(self.expiration_interval) 2024-09-21T09:39:33.144 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:39:33.144 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: self._run(expiration_interval) 2024-09-21T09:39:33.144 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:39:33.144 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:39:33.144 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:39:33.144 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:39:33.145 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:39:33.145 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: s = self.context.wrap_socket( 2024-09-21T09:39:33.145 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:39:33.145 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: return self.sslsocket_class._create( 2024-09-21T09:39:33.145 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:39:33.145 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: self.do_handshake() 2024-09-21T09:39:33.145 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:39:33.145 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: self._sslobj.do_handshake() 2024-09-21T09:39:33.145 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:39:33.145 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:39:33.235 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:33 smithi129 ceph-mon[32729]: mgrmap e18: y(active, since 1.01467s), standbys: x 2024-09-21T09:39:33.235 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:33 smithi129 ceph-mon[32729]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:33.235 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:33 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:33.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:33 smithi107 ceph-mon[41185]: mgrmap e18: y(active, since 1.01467s), standbys: x 2024-09-21T09:39:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:33 smithi107 ceph-mon[41185]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:33 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:33.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:33 smithi107 ceph-mon[29382]: mgrmap e18: y(active, since 1.01467s), standbys: x 2024-09-21T09:39:33.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:33 smithi107 ceph-mon[29382]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:33.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:33 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:33.900 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'rbd' created 2024-09-21T09:39:34.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:34 smithi129 ceph-mon[32729]: [21/Sep/2024:09:39:32] ENGINE Bus STARTING 2024-09-21T09:39:34.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:34 smithi129 ceph-mon[32729]: [21/Sep/2024:09:39:32] ENGINE Serving on https://172.21.15.107:7150 2024-09-21T09:39:34.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:34 smithi129 ceph-mon[32729]: [21/Sep/2024:09:39:32] ENGINE Bus STARTED 2024-09-21T09:39:34.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:34 smithi129 ceph-mon[32729]: [21/Sep/2024:09:39:32] ENGINE Error in HTTPServer.serve 2024-09-21T09:39:34.419 INFO:journalctl@ceph.mon.b.smithi129.stdout: Traceback (most recent call last): 2024-09-21T09:39:34.419 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:39:34.419 INFO:journalctl@ceph.mon.b.smithi129.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:39:34.419 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:39:34.419 INFO:journalctl@ceph.mon.b.smithi129.stdout: self._run(expiration_interval) 2024-09-21T09:39:34.419 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:39:34.419 INFO:journalctl@ceph.mon.b.smithi129.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:39:34.420 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:39:34.420 INFO:journalctl@ceph.mon.b.smithi129.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:39:34.420 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:39:34.420 INFO:journalctl@ceph.mon.b.smithi129.stdout: s = self.context.wrap_socket( 2024-09-21T09:39:34.420 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:39:34.420 INFO:journalctl@ceph.mon.b.smithi129.stdout: return self.sslsocket_class._create( 2024-09-21T09:39:34.420 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:39:34.420 INFO:journalctl@ceph.mon.b.smithi129.stdout: self.do_handshake() 2024-09-21T09:39:34.420 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:39:34.420 INFO:journalctl@ceph.mon.b.smithi129.stdout: self._sslobj.do_handshake() 2024-09-21T09:39:34.420 INFO:journalctl@ceph.mon.b.smithi129.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:39:34.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:34 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3805617876' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:39:34.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:34 smithi129 ceph-mon[32729]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:34.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:34 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:34.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:34 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:34.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:34 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:34.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:34 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:34.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:34 smithi129 ceph-mon[32729]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-21T09:39:34.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:34 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3805617876' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-21T09:39:34.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:34 smithi129 ceph-mon[32729]: osdmap e52: 8 total, 8 up, 8 in 2024-09-21T09:39:34.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[41185]: [21/Sep/2024:09:39:32] ENGINE Bus STARTING 2024-09-21T09:39:34.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[41185]: [21/Sep/2024:09:39:32] ENGINE Serving on https://172.21.15.107:7150 2024-09-21T09:39:34.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[41185]: [21/Sep/2024:09:39:32] ENGINE Bus STARTED 2024-09-21T09:39:34.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[41185]: [21/Sep/2024:09:39:32] ENGINE Error in HTTPServer.serve 2024-09-21T09:39:34.525 INFO:journalctl@ceph.mon.c.smithi107.stdout: Traceback (most recent call last): 2024-09-21T09:39:34.525 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:39:34.525 INFO:journalctl@ceph.mon.c.smithi107.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:39:34.525 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:39:34.525 INFO:journalctl@ceph.mon.c.smithi107.stdout: self._run(expiration_interval) 2024-09-21T09:39:34.525 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:39:34.525 INFO:journalctl@ceph.mon.c.smithi107.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:39:34.525 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:39:34.525 INFO:journalctl@ceph.mon.c.smithi107.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:39:34.525 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:39:34.525 INFO:journalctl@ceph.mon.c.smithi107.stdout: s = self.context.wrap_socket( 2024-09-21T09:39:34.525 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:39:34.525 INFO:journalctl@ceph.mon.c.smithi107.stdout: return self.sslsocket_class._create( 2024-09-21T09:39:34.525 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:39:34.525 INFO:journalctl@ceph.mon.c.smithi107.stdout: self.do_handshake() 2024-09-21T09:39:34.526 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:39:34.526 INFO:journalctl@ceph.mon.c.smithi107.stdout: self._sslobj.do_handshake() 2024-09-21T09:39:34.526 INFO:journalctl@ceph.mon.c.smithi107.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:39:34.526 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3805617876' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:39:34.526 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[41185]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:34.526 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:34.526 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:34.526 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:34.526 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:34.526 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[41185]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-21T09:39:34.526 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3805617876' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-21T09:39:34.526 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[41185]: osdmap e52: 8 total, 8 up, 8 in 2024-09-21T09:39:34.527 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[29382]: [21/Sep/2024:09:39:32] ENGINE Bus STARTING 2024-09-21T09:39:34.527 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[29382]: [21/Sep/2024:09:39:32] ENGINE Serving on https://172.21.15.107:7150 2024-09-21T09:39:34.527 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[29382]: [21/Sep/2024:09:39:32] ENGINE Bus STARTED 2024-09-21T09:39:34.527 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[29382]: [21/Sep/2024:09:39:32] ENGINE Error in HTTPServer.serve 2024-09-21T09:39:34.527 INFO:journalctl@ceph.mon.a.smithi107.stdout: Traceback (most recent call last): 2024-09-21T09:39:34.527 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:39:34.527 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:39:34.527 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:39:34.527 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._run(expiration_interval) 2024-09-21T09:39:34.527 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:39:34.527 INFO:journalctl@ceph.mon.a.smithi107.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:39:34.527 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:39:34.527 INFO:journalctl@ceph.mon.a.smithi107.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:39:34.528 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:39:34.528 INFO:journalctl@ceph.mon.a.smithi107.stdout: s = self.context.wrap_socket( 2024-09-21T09:39:34.528 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:39:34.528 INFO:journalctl@ceph.mon.a.smithi107.stdout: return self.sslsocket_class._create( 2024-09-21T09:39:34.528 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:39:34.528 INFO:journalctl@ceph.mon.a.smithi107.stdout: self.do_handshake() 2024-09-21T09:39:34.528 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:39:34.528 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._sslobj.do_handshake() 2024-09-21T09:39:34.528 INFO:journalctl@ceph.mon.a.smithi107.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:39:34.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3805617876' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:39:34.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[29382]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:34.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:34.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:34.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:34.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:34.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[29382]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-21T09:39:34.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3805617876' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-21T09:39:34.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:34 smithi107 ceph-mon[29382]: osdmap e52: 8 total, 8 up, 8 in 2024-09-21T09:39:34.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- rbd pool init rbd 2024-09-21T09:39:34.829 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:39:35.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:35.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:35.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:35.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:35.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:35.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:35.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[29382]: Adjusting osd_memory_target on smithi129 to 4016M 2024-09-21T09:39:35.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:35.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:35.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:35.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:39:35.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:39:35.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[29382]: Updating smithi107:/etc/ceph/ceph.conf 2024-09-21T09:39:35.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[29382]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T09:39:35.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[29382]: osdmap e53: 8 total, 8 up, 8 in 2024-09-21T09:39:35.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:35.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:35.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:35.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:35.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:35.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:35.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[41185]: Adjusting osd_memory_target on smithi129 to 4016M 2024-09-21T09:39:35.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:35.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:35.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:35.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:39:35.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:39:35.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[41185]: Updating smithi107:/etc/ceph/ceph.conf 2024-09-21T09:39:35.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[41185]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T09:39:35.288 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:35 smithi107 ceph-mon[41185]: osdmap e53: 8 total, 8 up, 8 in 2024-09-21T09:39:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:35 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:35 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:35 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:35 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:35 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:35 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:39:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:35 smithi129 ceph-mon[32729]: Adjusting osd_memory_target on smithi129 to 4016M 2024-09-21T09:39:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:35 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:35 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:35 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:35 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:39:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:35 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:39:35.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:35 smithi129 ceph-mon[32729]: Updating smithi107:/etc/ceph/ceph.conf 2024-09-21T09:39:35.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:35 smithi129 ceph-mon[32729]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T09:39:35.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:35 smithi129 ceph-mon[32729]: osdmap e53: 8 total, 8 up, 8 in 2024-09-21T09:39:36.667 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[29382]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:39:36.667 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[29382]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:39:36.667 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[29382]: pgmap v7: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:36.667 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[29382]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:39:36.667 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[29382]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:39:36.667 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[29382]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:39:36.667 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[29382]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:39:36.667 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:36.668 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:36.668 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:36.668 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:36.668 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:36.668 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[29382]: Deploying daemon alertmanager.a on smithi107 2024-09-21T09:39:36.668 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[29382]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:39:36.668 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[29382]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T09:39:36.668 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[29382]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-21T09:39:36.668 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[41185]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:39:36.668 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[41185]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:39:36.668 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[41185]: pgmap v7: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:36.668 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[41185]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:39:36.669 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[41185]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:39:36.669 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[41185]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:39:36.669 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[41185]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:39:36.669 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:36.669 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:36.669 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:36.669 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:36.669 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:36.669 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[41185]: Deploying daemon alertmanager.a on smithi107 2024-09-21T09:39:36.669 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[41185]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:39:36.669 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[41185]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T09:39:36.669 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:36 smithi107 ceph-mon[41185]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-21T09:39:36.836 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:36 smithi129 ceph-mon[32729]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:39:36.871 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:36 smithi129 ceph-mon[32729]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:39:36.871 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:36 smithi129 ceph-mon[32729]: pgmap v7: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:36.871 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:36 smithi129 ceph-mon[32729]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:39:36.872 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:36 smithi129 ceph-mon[32729]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:39:36.872 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:36 smithi129 ceph-mon[32729]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:39:36.872 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:36 smithi129 ceph-mon[32729]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:39:36.872 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:36 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:36.872 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:36 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:36.872 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:36 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:36.872 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:36 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:36.872 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:36 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:36.872 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:36 smithi129 ceph-mon[32729]: Deploying daemon alertmanager.a on smithi107 2024-09-21T09:39:36.872 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:36 smithi129 ceph-mon[32729]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:39:36.872 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:36 smithi129 ceph-mon[32729]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T09:39:36.872 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:36 smithi129 ceph-mon[32729]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-21T09:39:37.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:37 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1147346000' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:39:37.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:37 smithi129 ceph-mon[32729]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:39:37.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:37 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:39:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:39:37.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:37 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1147346000' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:39:37.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:37 smithi107 ceph-mon[41185]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:39:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:37 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1147346000' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:39:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:37 smithi107 ceph-mon[29382]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:39:38.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:38 smithi107 ceph-mon[29382]: pgmap v9: 9 pgs: 5 unknown, 4 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:38 smithi107 ceph-mon[29382]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-21T09:39:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:38 smithi107 ceph-mon[29382]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T09:39:38.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:38 smithi107 ceph-mon[41185]: pgmap v9: 9 pgs: 5 unknown, 4 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:38.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:38 smithi107 ceph-mon[41185]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-21T09:39:38.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:38 smithi107 ceph-mon[41185]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T09:39:38.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:38 smithi129 ceph-mon[32729]: pgmap v9: 9 pgs: 5 unknown, 4 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:38.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:38 smithi129 ceph-mon[32729]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-21T09:39:38.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:38 smithi129 ceph-mon[32729]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T09:39:39.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:39 smithi107 ceph-mon[29382]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T09:39:39.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:39 smithi107 ceph-mon[41185]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T09:39:39.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:39 smithi129 ceph-mon[32729]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T09:39:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:40 smithi107 ceph-mon[29382]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-21T09:39:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:40 smithi107 ceph-mon[29382]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:39:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:40 smithi107 ceph-mon[29382]: Cluster is now healthy 2024-09-21T09:39:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:40 smithi107 ceph-mon[29382]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T09:39:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:40 smithi107 ceph-mon[41185]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-21T09:39:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:40 smithi107 ceph-mon[41185]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:39:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:40 smithi107 ceph-mon[41185]: Cluster is now healthy 2024-09-21T09:39:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:40 smithi107 ceph-mon[41185]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T09:39:40.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:40 smithi129 ceph-mon[32729]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 43 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-21T09:39:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:40 smithi129 ceph-mon[32729]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:39:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:40 smithi129 ceph-mon[32729]: Cluster is now healthy 2024-09-21T09:39:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:40 smithi129 ceph-mon[32729]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T09:39:40.929 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph config log 1 --format=json 2024-09-21T09:39:41.750 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:39:42.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:42 smithi107 ceph-mon[29382]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T09:39:42.674 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:42 smithi107 ceph-mon[41185]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T09:39:42.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:42 smithi129 ceph-mon[32729]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 14 op/s 2024-09-21T09:39:44.359 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:39:44 smithi107 systemd[1]: Starting Ceph alertmanager.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:39:44.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:44 smithi107 ceph-mon[29382]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 341 B/s wr, 12 op/s 2024-09-21T09:39:44.805 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:44 smithi107 ceph-mon[41185]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 341 B/s wr, 12 op/s 2024-09-21T09:39:44.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:44 smithi129 ceph-mon[32729]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 341 B/s wr, 12 op/s 2024-09-21T09:39:45.108 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:39:44 smithi107 podman[71616]: 2024-09-21 09:39:44.832654998 +0000 UTC m=+0.448240936 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-21T09:39:45.108 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:39:44 smithi107 podman[71616]: 2024-09-21 09:39:44.946715337 +0000 UTC m=+0.562301315 volume create 221e7c012bf87428c3a85780f9fb74cbcffb7672777df3647f9b6a052f693752 2024-09-21T09:39:45.109 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:39:45 smithi107 podman[71616]: 2024-09-21 09:39:45.04751935 +0000 UTC m=+0.663105276 container create fa175a563d3cb8fe5e4b48ff9b711fc564dfa68ad84ad361ae271ebb1e8739ab (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:39:45.608 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:39:45] ENGINE Bus STOPPING 2024-09-21T09:39:45.609 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:39:45 smithi107 podman[71616]: 2024-09-21 09:39:45.35030082 +0000 UTC m=+0.965886743 container init fa175a563d3cb8fe5e4b48ff9b711fc564dfa68ad84ad361ae271ebb1e8739ab (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:39:45.609 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:39:45 smithi107 podman[71616]: 2024-09-21 09:39:45.355259303 +0000 UTC m=+0.970845226 container start fa175a563d3cb8fe5e4b48ff9b711fc564dfa68ad84ad361ae271ebb1e8739ab (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:39:45.609 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:39:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[71659]: ts=2024-09-21T09:39:45.367Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-21T09:39:45.609 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:39:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[71659]: ts=2024-09-21T09:39:45.367Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-21T09:39:45.609 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:39:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[71659]: ts=2024-09-21T09:39:45.368Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.107 port=9094 2024-09-21T09:39:45.609 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:39:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[71659]: ts=2024-09-21T09:39:45.369Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-21T09:39:45.609 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:39:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[71659]: ts=2024-09-21T09:39:45.392Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T09:39:45.609 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:39:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[71659]: ts=2024-09-21T09:39:45.392Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T09:39:45.609 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:39:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[71659]: ts=2024-09-21T09:39:45.394Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-21T09:39:45.609 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:39:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[71659]: ts=2024-09-21T09:39:45.394Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-21T09:39:45.610 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:39:45 smithi107 bash[71616]: fa175a563d3cb8fe5e4b48ff9b711fc564dfa68ad84ad361ae271ebb1e8739ab 2024-09-21T09:39:45.610 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:39:45 smithi107 systemd[1]: Started Ceph alertmanager.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:39:45.906 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:39:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:39:45.906 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:39:45] ENGINE Bus STOPPED 2024-09-21T09:39:45.906 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:39:45] ENGINE Bus STARTING 2024-09-21T09:39:45.906 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:39:45] ENGINE Serving on http://:::9283 2024-09-21T09:39:45.906 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:39:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:39:45] ENGINE Bus STARTED 2024-09-21T09:39:46.154 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:39:46.802 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[29382]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 272 B/s wr, 0 op/s 2024-09-21T09:39:46.802 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.802 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.802 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.802 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.802 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.802 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.802 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:39:46.803 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[29382]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:39:46.803 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.803 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[29382]: Deploying daemon grafana.a on smithi129 2024-09-21T09:39:46.803 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.803 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:39:46.803 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1467171576' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-21T09:39:46.803 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[41185]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 272 B/s wr, 0 op/s 2024-09-21T09:39:46.803 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.803 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.803 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.803 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.803 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.803 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.804 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:39:46.804 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[41185]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:39:46.804 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.804 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[41185]: Deploying daemon grafana.a on smithi129 2024-09-21T09:39:46.804 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.804 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:39:46.804 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:46 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1467171576' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-21T09:39:46.827 INFO:teuthology.orchestra.run.smithi107.stdout:[{"version":19,"timestamp":"2024-09-21T09:39:45.570012+0000","name":"","changes":[{"name":"mgr/mgr/dashboard/GRAFANA_API_SSL_VERIFY","new_value":"false"}]}] 2024-09-21T09:39:46.827 INFO:tasks.ceph_manager:config epoch is 19 2024-09-21T09:39:46.827 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T09:39:46.827 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T09:39:46.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T09:39:46.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:46 smithi129 ceph-mon[32729]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 272 B/s wr, 0 op/s 2024-09-21T09:39:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:46 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:46 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:46 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:46 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:46 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:46 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:46 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:39:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:46 smithi129 ceph-mon[32729]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:39:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:46 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:46 smithi129 ceph-mon[32729]: Deploying daemon grafana.a on smithi129 2024-09-21T09:39:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:46 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:46 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:39:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:46 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1467171576' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-21T09:39:47.072 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:39:47.447 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:39:47 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[71659]: ts=2024-09-21T09:39:47.369Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000316789s 2024-09-21T09:39:47.668 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:47 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:39:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:39:48.838 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:48 smithi107 ceph-mon[29382]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 240 B/s wr, 0 op/s 2024-09-21T09:39:48.839 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:48 smithi107 ceph-mon[29382]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T09:39:48.839 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:48 smithi107 ceph-mon[41185]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 240 B/s wr, 0 op/s 2024-09-21T09:39:48.839 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:48 smithi107 ceph-mon[41185]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T09:39:48.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:48 smithi129 ceph-mon[32729]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 240 B/s wr, 0 op/s 2024-09-21T09:39:48.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:48 smithi129 ceph-mon[32729]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T09:39:49.004 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:39:49.760 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"flags":0,"active_gid":24344,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4055614859},{"type":"v1","addr":"172.21.15.107:6801","nonce":4055614859}]},"active_addr":"172.21.15.107:6801/4055614859","active_change":"2024-09-21T09:39:30.975631+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24352,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.107:8443/","prometheus":"http://172.21.15.107:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":51,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":3025710247}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":1526680749}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":51270168}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":2701254891}]}]} 2024-09-21T09:39:49.765 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T09:39:49.765 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T09:39:49.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:39:49.788 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:49 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3859806472' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:39:49.788 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:49 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3859806472' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:39:49.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:49 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3859806472' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:39:50.005 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:39:50.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:50 smithi107 ceph-mon[29382]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 215 B/s wr, 0 op/s 2024-09-21T09:39:50.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:50 smithi107 ceph-mon[41185]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 215 B/s wr, 0 op/s 2024-09-21T09:39:50.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:50 smithi129 ceph-mon[32729]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 215 B/s wr, 0 op/s 2024-09-21T09:39:51.882 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:39:51.882 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":58,"fsid":"86c49514-77fc-11ef-baf6-efdc52797490","created":"2024-09-21T09:34:03.635940+0000","modified":"2024-09-21T09:39:48.439468+0000","last_up_change":"2024-09-21T09:39:02.736647+0000","last_in_change":"2024-09-21T09:38:46.049422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:37:08.948207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:39:33.152023+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"635c1eb0-29b1-4fdd-bd47-881ef05e49bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":174831156},{"type":"v1","addr":"172.21.15.107:6803","nonce":174831156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":174831156},{"type":"v1","addr":"172.21.15.107:6805","nonce":174831156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":174831156},{"type":"v1","addr":"172.21.15.107:6809","nonce":174831156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":174831156},{"type":"v1","addr":"172.21.15.107:6807","nonce":174831156}]},"public_addr":"172.21.15.107:6803/174831156","cluster_addr":"172.21.15.107:6805/174831156","heartbeat_back_addr":"172.21.15.107:6809/174831156","heartbeat_front_addr":"172.21.15.107:6807/174831156","state":["exists","up"]},{"osd":1,"uuid":"a39b6a18-939d-4999-89b2-1126c1b2d1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1227448003},{"type":"v1","addr":"172.21.15.107:6811","nonce":1227448003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1227448003},{"type":"v1","addr":"172.21.15.107:6813","nonce":1227448003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1227448003},{"type":"v1","addr":"172.21.15.107:6817","nonce":1227448003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1227448003},{"type":"v1","addr":"172.21.15.107:6815","nonce":1227448003}]},"public_addr":"172.21.15.107:6811/1227448003","cluster_addr":"172.21.15.107:6813/1227448003","heartbeat_back_addr":"172.21.15.107:6817/1227448003","heartbeat_front_addr":"172.21.15.107:6815/1227448003","state":["exists","up"]},{"osd":2,"uuid":"b01c5f1b-77e4-462f-80d0-fd1d3afb786b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3613749103},{"type":"v1","addr":"172.21.15.107:6819","nonce":3613749103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3613749103},{"type":"v1","addr":"172.21.15.107:6821","nonce":3613749103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3613749103},{"type":"v1","addr":"172.21.15.107:6825","nonce":3613749103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3613749103},{"type":"v1","addr":"172.21.15.107:6823","nonce":3613749103}]},"public_addr":"172.21.15.107:6819/3613749103","cluster_addr":"172.21.15.107:6821/3613749103","heartbeat_back_addr":"172.21.15.107:6825/3613749103","heartbeat_front_addr":"172.21.15.107:6823/3613749103","state":["exists","up"]},{"osd":3,"uuid":"5135d269-b6c4-4a9d-bcf9-ffe19bc5c9ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3206618601},{"type":"v1","addr":"172.21.15.107:6827","nonce":3206618601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":3206618601},{"type":"v1","addr":"172.21.15.107:6829","nonce":3206618601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3206618601},{"type":"v1","addr":"172.21.15.107:6833","nonce":3206618601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3206618601},{"type":"v1","addr":"172.21.15.107:6831","nonce":3206618601}]},"public_addr":"172.21.15.107:6827/3206618601","cluster_addr":"172.21.15.107:6829/3206618601","heartbeat_back_addr":"172.21.15.107:6833/3206618601","heartbeat_front_addr":"172.21.15.107:6831/3206618601","state":["exists","up"]},{"osd":4,"uuid":"23242972-e341-4773-809f-299aabeacd56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6801","nonce":4101249335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6803","nonce":4101249335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6807","nonce":4101249335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6805","nonce":4101249335}]},"public_addr":"172.21.15.129:6801/4101249335","cluster_addr":"172.21.15.129:6803/4101249335","heartbeat_back_addr":"172.21.15.129:6807/4101249335","heartbeat_front_addr":"172.21.15.129:6805/4101249335","state":["exists","up"]},{"osd":5,"uuid":"f9da25cd-1680-4d97-a9d1-98dded10766f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6809","nonce":2733195233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6811","nonce":2733195233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6815","nonce":2733195233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6813","nonce":2733195233}]},"public_addr":"172.21.15.129:6809/2733195233","cluster_addr":"172.21.15.129:6811/2733195233","heartbeat_back_addr":"172.21.15.129:6815/2733195233","heartbeat_front_addr":"172.21.15.129:6813/2733195233","state":["exists","up"]},{"osd":6,"uuid":"d6d2bd99-0178-409c-a548-cbc30b8b76fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6817","nonce":3400496608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6819","nonce":3400496608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6823","nonce":3400496608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6821","nonce":3400496608}]},"public_addr":"172.21.15.129:6817/3400496608","cluster_addr":"172.21.15.129:6819/3400496608","heartbeat_back_addr":"172.21.15.129:6823/3400496608","heartbeat_front_addr":"172.21.15.129:6821/3400496608","state":["exists","up"]},{"osd":7,"uuid":"2c0a9723-0328-4134-b5d4-cf307839018a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6825","nonce":101668996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6827","nonce":101668996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6831","nonce":101668996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6829","nonce":101668996}]},"public_addr":"172.21.15.129:6825/101668996","cluster_addr":"172.21.15.129:6827/101668996","heartbeat_back_addr":"172.21.15.129:6831/101668996","heartbeat_front_addr":"172.21.15.129:6829/101668996","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:36:18.300841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:36:42.334925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:06.141467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:30.344630+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:51.661657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:38:14.654068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:38:38.089961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:39:01.119878+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/218684017":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/637981220":"2024-09-22T09:34:34.697433+0000","172.21.15.107:6800/1686228827":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/2156530655":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/782384657":"2024-09-22T09:34:56.643317+0000","172.21.15.107:6801/1686228827":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/2492859118":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/1942117255":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/837896416":"2024-09-22T09:34:34.697433+0000","172.21.15.107:6800/2716631102":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/4103644615":"2024-09-22T09:39:30.975484+0000","172.21.15.107:6800/4208383830":"2024-09-22T09:34:56.643317+0000","172.21.15.107:6801/4208383830":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/3424783814":"2024-09-22T09:34:56.643317+0000","172.21.15.107:6801/2716631102":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/2526044436":"2024-09-22T09:39:30.975484+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:39:52.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:52 smithi107 ceph-mon[29382]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-09-21T09:39:52.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:52 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:52.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:52 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2607363154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:39:52.176 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:52 smithi107 ceph-mon[41185]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-09-21T09:39:52.176 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:52 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:52.176 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:52 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/2607363154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:39:52.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:52 smithi129 ceph-mon[32729]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-09-21T09:39:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:52 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:52 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2607363154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:39:52.503 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T09:39:52.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:39:52.793 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:39:54.305 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:54 smithi107 ceph-mon[29382]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:54.305 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:54 smithi107 ceph-mon[41185]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:54.307 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:54 smithi129 ceph-mon[32729]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:54.657 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:39:54.657 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":58,"fsid":"86c49514-77fc-11ef-baf6-efdc52797490","created":"2024-09-21T09:34:03.635940+0000","modified":"2024-09-21T09:39:48.439468+0000","last_up_change":"2024-09-21T09:39:02.736647+0000","last_in_change":"2024-09-21T09:38:46.049422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:37:08.948207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:39:33.152023+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"635c1eb0-29b1-4fdd-bd47-881ef05e49bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":174831156},{"type":"v1","addr":"172.21.15.107:6803","nonce":174831156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":174831156},{"type":"v1","addr":"172.21.15.107:6805","nonce":174831156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":174831156},{"type":"v1","addr":"172.21.15.107:6809","nonce":174831156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":174831156},{"type":"v1","addr":"172.21.15.107:6807","nonce":174831156}]},"public_addr":"172.21.15.107:6803/174831156","cluster_addr":"172.21.15.107:6805/174831156","heartbeat_back_addr":"172.21.15.107:6809/174831156","heartbeat_front_addr":"172.21.15.107:6807/174831156","state":["exists","up"]},{"osd":1,"uuid":"a39b6a18-939d-4999-89b2-1126c1b2d1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1227448003},{"type":"v1","addr":"172.21.15.107:6811","nonce":1227448003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1227448003},{"type":"v1","addr":"172.21.15.107:6813","nonce":1227448003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1227448003},{"type":"v1","addr":"172.21.15.107:6817","nonce":1227448003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1227448003},{"type":"v1","addr":"172.21.15.107:6815","nonce":1227448003}]},"public_addr":"172.21.15.107:6811/1227448003","cluster_addr":"172.21.15.107:6813/1227448003","heartbeat_back_addr":"172.21.15.107:6817/1227448003","heartbeat_front_addr":"172.21.15.107:6815/1227448003","state":["exists","up"]},{"osd":2,"uuid":"b01c5f1b-77e4-462f-80d0-fd1d3afb786b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3613749103},{"type":"v1","addr":"172.21.15.107:6819","nonce":3613749103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3613749103},{"type":"v1","addr":"172.21.15.107:6821","nonce":3613749103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3613749103},{"type":"v1","addr":"172.21.15.107:6825","nonce":3613749103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3613749103},{"type":"v1","addr":"172.21.15.107:6823","nonce":3613749103}]},"public_addr":"172.21.15.107:6819/3613749103","cluster_addr":"172.21.15.107:6821/3613749103","heartbeat_back_addr":"172.21.15.107:6825/3613749103","heartbeat_front_addr":"172.21.15.107:6823/3613749103","state":["exists","up"]},{"osd":3,"uuid":"5135d269-b6c4-4a9d-bcf9-ffe19bc5c9ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3206618601},{"type":"v1","addr":"172.21.15.107:6827","nonce":3206618601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":3206618601},{"type":"v1","addr":"172.21.15.107:6829","nonce":3206618601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3206618601},{"type":"v1","addr":"172.21.15.107:6833","nonce":3206618601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3206618601},{"type":"v1","addr":"172.21.15.107:6831","nonce":3206618601}]},"public_addr":"172.21.15.107:6827/3206618601","cluster_addr":"172.21.15.107:6829/3206618601","heartbeat_back_addr":"172.21.15.107:6833/3206618601","heartbeat_front_addr":"172.21.15.107:6831/3206618601","state":["exists","up"]},{"osd":4,"uuid":"23242972-e341-4773-809f-299aabeacd56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6801","nonce":4101249335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6803","nonce":4101249335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6807","nonce":4101249335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6805","nonce":4101249335}]},"public_addr":"172.21.15.129:6801/4101249335","cluster_addr":"172.21.15.129:6803/4101249335","heartbeat_back_addr":"172.21.15.129:6807/4101249335","heartbeat_front_addr":"172.21.15.129:6805/4101249335","state":["exists","up"]},{"osd":5,"uuid":"f9da25cd-1680-4d97-a9d1-98dded10766f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6809","nonce":2733195233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6811","nonce":2733195233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6815","nonce":2733195233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6813","nonce":2733195233}]},"public_addr":"172.21.15.129:6809/2733195233","cluster_addr":"172.21.15.129:6811/2733195233","heartbeat_back_addr":"172.21.15.129:6815/2733195233","heartbeat_front_addr":"172.21.15.129:6813/2733195233","state":["exists","up"]},{"osd":6,"uuid":"d6d2bd99-0178-409c-a548-cbc30b8b76fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6817","nonce":3400496608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6819","nonce":3400496608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6823","nonce":3400496608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6821","nonce":3400496608}]},"public_addr":"172.21.15.129:6817/3400496608","cluster_addr":"172.21.15.129:6819/3400496608","heartbeat_back_addr":"172.21.15.129:6823/3400496608","heartbeat_front_addr":"172.21.15.129:6821/3400496608","state":["exists","up"]},{"osd":7,"uuid":"2c0a9723-0328-4134-b5d4-cf307839018a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6825","nonce":101668996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6827","nonce":101668996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6831","nonce":101668996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6829","nonce":101668996}]},"public_addr":"172.21.15.129:6825/101668996","cluster_addr":"172.21.15.129:6827/101668996","heartbeat_back_addr":"172.21.15.129:6831/101668996","heartbeat_front_addr":"172.21.15.129:6829/101668996","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:36:18.300841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:36:42.334925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:06.141467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:30.344630+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:51.661657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:38:14.654068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:38:38.089961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:39:01.119878+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/218684017":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/637981220":"2024-09-22T09:34:34.697433+0000","172.21.15.107:6800/1686228827":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/2156530655":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/782384657":"2024-09-22T09:34:56.643317+0000","172.21.15.107:6801/1686228827":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/2492859118":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/1942117255":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/837896416":"2024-09-22T09:34:34.697433+0000","172.21.15.107:6800/2716631102":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/4103644615":"2024-09-22T09:39:30.975484+0000","172.21.15.107:6800/4208383830":"2024-09-22T09:34:56.643317+0000","172.21.15.107:6801/4208383830":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/3424783814":"2024-09-22T09:34:56.643317+0000","172.21.15.107:6801/2716631102":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/2526044436":"2024-09-22T09:39:30.975484+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:39:55.328 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:55 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/4114636134' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:39:55.328 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:55 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/4114636134' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:39:55.352 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T09:39:55.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T09:39:55.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T09:39:55.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T09:39:55.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T09:39:55.355 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T09:39:55.355 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T09:39:55.355 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T09:39:55.405 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:55 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/4114636134' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:39:55.610 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:39:55 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[71659]: ts=2024-09-21T09:39:55.371Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001641972s 2024-09-21T09:39:55.880 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:39:55.882 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:39:55.889 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:39:55.894 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:39:55.895 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:39:55.903 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:39:55.903 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:39:55.936 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:39:56.053 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:55 smithi129 podman[64011]: 2024-09-21 09:39:55.98939201 +0000 UTC m=+0.119925346 container create 70638332abb2421839ee027825ee70c2ea7a523a7e53dbcd5de3c017139084c9 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a, vcs-type=git, architecture=x86_64, distribution-scope=public, maintainer=Paul Cuzner , io.buildah.version=1.32.0, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, io.openshift.tags=base rhel8, version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-09-21T09:39:56.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:56 smithi107 ceph-mon[29382]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:56.291 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:56 smithi107 ceph-mon[41185]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:56.303 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-mon[32729]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:56.303 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 podman[64011]: 2024-09-21 09:39:56.208311701 +0000 UTC m=+0.338845037 container init 70638332abb2421839ee027825ee70c2ea7a523a7e53dbcd5de3c017139084c9 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, name=ubi8, build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, description=Ceph Grafana Container, release=239.1651231664, version=8.5, distribution-scope=public, io.buildah.version=1.32.0, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-09-21T09:39:56.303 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 podman[64011]: 2024-09-21 09:39:56.212932176 +0000 UTC m=+0.343465512 container start 70638332abb2421839ee027825ee70c2ea7a523a7e53dbcd5de3c017139084c9 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a, com.redhat.component=ubi8-container, release=239.1651231664, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, build-date=2022-04-29T11:48:11.055897, vendor=Red Hat, Inc., version=8.5, architecture=x86_64, io.openshift.expose-services=, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-09-21T09:39:56.304 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 bash[64011]: 70638332abb2421839ee027825ee70c2ea7a523a7e53dbcd5de3c017139084c9 2024-09-21T09:39:56.304 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 systemd[1]: Started Ceph grafana.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:39:56.555 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=settings t=2024-09-21T09:39:56.367194905Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-21T09:39:56.555 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=settings t=2024-09-21T09:39:56.367383164Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-21T09:39:56.555 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=settings t=2024-09-21T09:39:56.367411686Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-21T09:39:56.555 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=settings t=2024-09-21T09:39:56.367423743Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-21T09:39:56.555 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=settings t=2024-09-21T09:39:56.367433043Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-21T09:39:56.555 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=settings t=2024-09-21T09:39:56.367442457Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-21T09:39:56.555 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=settings t=2024-09-21T09:39:56.367450652Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-21T09:39:56.555 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=settings t=2024-09-21T09:39:56.367459983Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-21T09:39:56.555 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=settings t=2024-09-21T09:39:56.367468411Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-21T09:39:56.556 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=settings t=2024-09-21T09:39:56.367477112Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-21T09:39:56.556 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=settings t=2024-09-21T09:39:56.367485811Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-21T09:39:56.556 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=settings t=2024-09-21T09:39:56.367493736Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-21T09:39:56.556 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=settings t=2024-09-21T09:39:56.367502115Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-21T09:39:56.556 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=settings t=2024-09-21T09:39:56.367510205Z level=info msg="App mode production" 2024-09-21T09:39:56.556 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=sqlstore t=2024-09-21T09:39:56.367572483Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-21T09:39:56.556 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=sqlstore t=2024-09-21T09:39:56.367596948Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-21T09:39:56.556 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.368841322Z level=info msg="Starting DB migrations" 2024-09-21T09:39:56.556 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.368923436Z level=info msg="Executing migration" id="create migration_log table" 2024-09-21T09:39:56.556 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.369459651Z level=info msg="Executing migration" id="create user table" 2024-09-21T09:39:56.557 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.369913659Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-21T09:39:56.557 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.370360754Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-21T09:39:56.557 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.370796114Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-21T09:39:56.557 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.371350469Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-21T09:39:56.557 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.371739057Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-21T09:39:56.557 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.372404017Z level=info msg="Executing migration" id="create user table v2" 2024-09-21T09:39:56.557 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.37281431Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-21T09:39:56.557 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.373165873Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-21T09:39:56.557 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.373569688Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-21T09:39:56.558 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.373903777Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-21T09:39:56.558 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.374220525Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-21T09:39:56.558 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.374632925Z level=info msg="Executing migration" id="Update user table charset" 2024-09-21T09:39:56.558 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.374905908Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-21T09:39:56.558 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.37528478Z level=info msg="Executing migration" id="Add missing user data" 2024-09-21T09:39:56.558 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.375638893Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-21T09:39:56.558 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.376046606Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-21T09:39:56.558 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.376430249Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-21T09:39:56.558 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.376871334Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-21T09:39:56.558 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.378424454Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-21T09:39:56.558 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.378850476Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-21T09:39:56.558 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.379278435Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-21T09:39:56.558 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.379671748Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-21T09:39:56.558 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.380059539Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-21T09:39:56.559 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.380632889Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-21T09:39:56.559 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.380924653Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-21T09:39:56.559 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.381265656Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-21T09:39:56.559 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.381653147Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-21T09:39:56.559 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.382023601Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-21T09:39:56.559 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.382369594Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-21T09:39:56.559 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.383009193Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-21T09:39:56.559 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.383383113Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-21T09:39:56.559 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.383780935Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-21T09:39:56.559 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.38412823Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-21T09:39:56.559 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.384472369Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-21T09:39:56.559 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.384861902Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-21T09:39:56.559 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.385154908Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-21T09:39:56.560 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.385475992Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-21T09:39:56.560 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.38580363Z level=info msg="Executing migration" id="create star table" 2024-09-21T09:39:56.560 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.386143357Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-21T09:39:56.560 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.386535748Z level=info msg="Executing migration" id="create org table v1" 2024-09-21T09:39:56.560 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.386911132Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-21T09:39:56.560 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.387292203Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-21T09:39:56.560 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.387658045Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-21T09:39:56.560 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.388038335Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-21T09:39:56.560 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.38845448Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-21T09:39:56.560 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.388827309Z level=info msg="Executing migration" id="Update org table charset" 2024-09-21T09:39:56.560 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.389079727Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-21T09:39:56.560 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.389342189Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-21T09:39:56.560 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.389653113Z level=info msg="Executing migration" id="create dashboard table" 2024-09-21T09:39:56.560 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.390008576Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-21T09:39:56.560 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.390960925Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-21T09:39:56.561 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.39169446Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-21T09:39:56.561 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.392063061Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-21T09:39:56.561 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.39295898Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-21T09:39:56.561 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.39377849Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-21T09:39:56.561 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.394648129Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-21T09:39:56.561 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.395037863Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-21T09:39:56.561 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.39602813Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-21T09:39:56.562 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.396718515Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-21T09:39:56.562 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.397557169Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-21T09:39:56.562 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.398395378Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-21T09:39:56.562 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.399311019Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-21T09:39:56.562 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.400067496Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-21T09:39:56.562 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.400973618Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-21T09:39:56.562 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.401469251Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-21T09:39:56.562 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.40246673Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-21T09:39:56.563 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.405156461Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-21T09:39:56.563 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.40808396Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-21T09:39:56.563 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.408667617Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-21T09:39:56.563 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.411601058Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-21T09:39:56.563 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.414305128Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-21T09:39:56.563 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.417051185Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-21T09:39:56.563 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.417580194Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-21T09:39:56.563 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.420552215Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-21T09:39:56.563 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.423230696Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-21T09:39:56.563 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.425910033Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-21T09:39:56.563 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.426492631Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-21T09:39:56.563 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.429238359Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-21T09:39:56.563 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.432030025Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-21T09:39:56.563 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.434680882Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-21T09:39:56.564 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.435077088Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-21T09:39:56.564 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.438007896Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-21T09:39:56.564 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.440706944Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-21T09:39:56.564 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.443445234Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-21T09:39:56.564 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.443865531Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-21T09:39:56.564 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.446467924Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-21T09:39:56.564 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.446960746Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-21T09:39:56.564 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.448961862Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-21T09:39:56.564 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.449391585Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-21T09:39:56.564 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.451760265Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-21T09:39:56.564 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.4520828Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-21T09:39:56.564 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.454552245Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-21T09:39:56.564 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.457296081Z level=info msg="Executing migration" id="create data_source table" 2024-09-21T09:39:56.564 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.45982453Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-21T09:39:56.565 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.46024671Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-21T09:39:56.565 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.463031713Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-21T09:39:56.565 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.465464218Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-21T09:39:56.565 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.467981528Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-21T09:39:56.565 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.469041727Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-21T09:39:56.565 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.471822817Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-21T09:39:56.565 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.474151705Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-21T09:39:56.565 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.474962841Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-21T09:39:56.565 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.475341781Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-21T09:39:56.565 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.476216846Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-21T09:39:56.565 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.476998887Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-21T09:39:56.565 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.477805844Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-21T09:39:56.565 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.478115437Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-21T09:39:56.565 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.478896544Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-21T09:39:56.566 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.479856904Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-21T09:39:56.566 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.480551245Z level=info msg="Executing migration" id="Add uid column" 2024-09-21T09:39:56.566 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.481074515Z level=info msg="Executing migration" id="Update uid value" 2024-09-21T09:39:56.566 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.481973977Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-21T09:39:56.566 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.482518145Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-21T09:39:56.566 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.482931283Z level=info msg="Executing migration" id="create api_key table" 2024-09-21T09:39:56.566 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.48370951Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-21T09:39:56.566 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.484143023Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-21T09:39:56.566 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.48490312Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-21T09:39:56.566 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.485890933Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-21T09:39:56.566 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.486539626Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-21T09:39:56.566 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.487522939Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-21T09:39:56.566 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.48818884Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-21T09:39:56.566 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.489333131Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-21T09:39:56.566 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.490183138Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-21T09:39:56.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.491210209Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-21T09:39:56.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.492046798Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-21T09:39:56.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.492807498Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-21T09:39:56.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.493668205Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-21T09:39:56.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.494528697Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-21T09:39:56.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.49548941Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-21T09:39:56.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.496267367Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-21T09:39:56.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.497090225Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-21T09:39:56.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.497904648Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-21T09:39:56.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.49875036Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-21T09:39:56.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.499603027Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-21T09:39:56.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.500490555Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-21T09:39:56.567 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.501369057Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-21T09:39:56.568 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.504055399Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-21T09:39:56.568 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.506809051Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-21T09:39:56.568 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.509569343Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-21T09:39:56.568 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.512365197Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-21T09:39:56.568 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.51512315Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-21T09:39:56.568 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.517927503Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-21T09:39:56.568 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.520653222Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-21T09:39:56.568 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.523488345Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-21T09:39:56.568 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.526198311Z level=info msg="Executing migration" id="create quota table v1" 2024-09-21T09:39:56.568 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.528853803Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-21T09:39:56.568 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.531595925Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-21T09:39:56.568 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.534386687Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-21T09:39:56.568 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.536946095Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-21T09:39:56.568 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.539569609Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-21T09:39:56.569 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.542455905Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-21T09:39:56.569 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.545160527Z level=info msg="Executing migration" id="create session table" 2024-09-21T09:39:56.569 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.54783944Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-21T09:39:56.569 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.550511037Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-21T09:39:56.569 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.55327518Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-21T09:39:56.569 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.556025389Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-21T09:39:56.569 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.558650727Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-21T09:39:56.807 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.561225572Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-21T09:39:56.808 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.564005455Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-21T09:39:56.808 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.566652963Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-21T09:39:56.808 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.569334089Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-21T09:39:56.808 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.571989528Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-21T09:39:56.808 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.574649793Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-21T09:39:56.808 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.577316799Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-21T09:39:56.808 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.579967544Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-21T09:39:56.808 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.582560519Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-21T09:39:56.808 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.585092515Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-21T09:39:56.808 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.58773342Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-21T09:39:56.808 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.590465407Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-21T09:39:56.808 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.593329747Z level=info msg="Executing migration" id="create alert table v1" 2024-09-21T09:39:56.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.595949215Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-21T09:39:56.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.596640806Z level=info msg="Executing migration" id="add index alert state" 2024-09-21T09:39:56.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.597553796Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-21T09:39:56.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.598331183Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-21T09:39:56.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.599121255Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-21T09:39:56.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.599966289Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-21T09:39:56.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.600787848Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-21T09:39:56.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.602233385Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-21T09:39:56.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.603074756Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-21T09:39:56.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.603555394Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-21T09:39:56.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.603903296Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-21T09:39:56.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.604261413Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-21T09:39:56.809 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.604682859Z level=info msg="Executing migration" id="Add column is_default" 2024-09-21T09:39:56.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.605334991Z level=info msg="Executing migration" id="Add column frequency" 2024-09-21T09:39:56.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.606005375Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-21T09:39:56.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.60668981Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-21T09:39:56.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.607362335Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-21T09:39:56.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.607832413Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-21T09:39:56.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.608142938Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-21T09:39:56.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.60843852Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-21T09:39:56.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.60887629Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-21T09:39:56.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.609315707Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-21T09:39:56.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.609736303Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-21T09:39:56.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.61014654Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-21T09:39:56.810 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.610604077Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-21T09:39:56.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.611280673Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-21T09:39:56.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.611936867Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-21T09:39:56.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.612272542Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-21T09:39:56.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.612737846Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-21T09:39:56.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.613162205Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-21T09:39:56.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.613836848Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-21T09:39:56.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.614171027Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-21T09:39:56.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.614623146Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-21T09:39:56.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.615080712Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-21T09:39:56.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.615386882Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-21T09:39:56.812 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.615891231Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-21T09:39:56.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.616346221Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-21T09:39:56.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.616794112Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-21T09:39:56.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.617230077Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-21T09:39:56.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.618500125Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-21T09:39:56.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.621123819Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-21T09:39:56.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.623666898Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-21T09:39:56.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.624399098Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-21T09:39:56.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.627101435Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-21T09:39:56.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.6295921Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-21T09:39:56.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.632268429Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-21T09:39:56.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.634978799Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-21T09:39:56.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.637380233Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-21T09:39:56.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.64023818Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-21T09:39:56.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.640863662Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-21T09:39:56.813 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.641736653Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-21T09:39:56.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.642508162Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-21T09:39:56.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.643471013Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-21T09:39:56.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.644245563Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-21T09:39:56.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.645063139Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-21T09:39:56.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.645852516Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-21T09:39:56.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.646702276Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-21T09:39:56.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.647534045Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-21T09:39:56.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.648315244Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-21T09:39:56.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.649160996Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-21T09:39:56.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.649948291Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-21T09:39:56.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.650828286Z level=info msg="Executing migration" id="Move region to single row" 2024-09-21T09:39:56.814 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.651738719Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-21T09:39:56.815 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.652625573Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-21T09:39:56.815 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.653536884Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-21T09:39:56.815 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.654195522Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-21T09:39:56.815 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.655068533Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-21T09:39:56.815 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.655930067Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-21T09:39:56.815 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.656780826Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-21T09:39:56.815 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.657729147Z level=info msg="Executing migration" id="create test_data table" 2024-09-21T09:39:56.815 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.658484362Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-21T09:39:56.815 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.658887009Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-21T09:39:56.815 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.659853033Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-21T09:39:56.815 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.660725628Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-21T09:39:56.815 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.661440911Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-21T09:39:56.815 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.662361854Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-21T09:39:56.815 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.663132338Z level=info msg="Executing migration" id="create team table" 2024-09-21T09:39:56.816 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.664128371Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-21T09:39:56.816 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.664849152Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-21T09:39:56.816 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.665711084Z level=info msg="Executing migration" id="create team member table" 2024-09-21T09:39:56.816 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.666541866Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-21T09:39:56.816 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.667561839Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-21T09:39:56.816 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.66826611Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-21T09:39:56.816 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.669149796Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-21T09:39:56.816 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.669998082Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-21T09:39:56.816 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.672921506Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-21T09:39:56.816 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.675690876Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-21T09:39:56.816 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.678420318Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-21T09:39:56.816 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.68120045Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-21T09:39:56.816 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.684005255Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-21T09:39:56.816 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.684632762Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-21T09:39:56.817 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.687509364Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-21T09:39:56.817 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.690233542Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-21T09:39:56.817 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.693145168Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-21T09:39:56.817 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.696093634Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-21T09:39:56.817 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.698658021Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-21T09:39:56.817 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.699145598Z level=info msg="Executing migration" id="create tag table" 2024-09-21T09:39:56.817 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.701973355Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-21T09:39:56.817 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.70481858Z level=info msg="Executing migration" id="create login attempt table" 2024-09-21T09:39:56.817 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.707704184Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-21T09:39:56.817 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.7081663Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-21T09:39:56.817 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.710790053Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-21T09:39:56.817 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.713599133Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-21T09:39:56.817 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.71633154Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-21T09:39:56.817 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.716820314Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-21T09:39:56.817 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.719518499Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-21T09:39:56.818 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.722140364Z level=info msg="Executing migration" id="create user auth table" 2024-09-21T09:39:56.818 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.724771736Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-21T09:39:56.818 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.725353328Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-21T09:39:56.818 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.72827874Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-21T09:39:56.818 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.730773303Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-21T09:39:56.818 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.733529488Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-21T09:39:56.818 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.734285109Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-21T09:39:56.818 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.736970712Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-21T09:39:56.818 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.739714362Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-21T09:39:56.818 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.742324325Z level=info msg="Executing migration" id="create server_lock table" 2024-09-21T09:39:56.818 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.742811495Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-21T09:39:56.818 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.7455991Z level=info msg="Executing migration" id="create user auth token table" 2024-09-21T09:39:56.818 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.748127855Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-21T09:39:56.818 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.750822806Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-21T09:39:56.819 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.751424064Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-21T09:39:56.819 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.752397253Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-21T09:39:56.819 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.753360416Z level=info msg="Executing migration" id="create cache_data table" 2024-09-21T09:39:56.819 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.754150234Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-21T09:39:56.819 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.754596727Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-21T09:39:56.819 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.755455489Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-21T09:39:56.819 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.756290593Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-21T09:39:56.819 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.756596759Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-21T09:39:56.819 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.756894321Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-21T09:39:56.819 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.757331995Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-21T09:39:56.819 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.758094851Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-21T09:39:56.819 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.758505476Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-21T09:39:56.819 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.758795546Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-21T09:39:56.819 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.759165924Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-21T09:39:56.820 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.75956595Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-21T09:39:56.820 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.759975406Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-21T09:39:56.820 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.760378418Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-21T09:39:56.820 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.761148762Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-21T09:39:56.820 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.761596151Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-21T09:39:56.820 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.761891681Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-21T09:39:56.820 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.762266239Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-21T09:39:56.820 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.762706672Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-21T09:39:56.820 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.763129781Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-21T09:39:56.820 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.764012525Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-21T09:39:56.820 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.764462695Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-21T09:39:56.820 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.764872793Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-21T09:39:56.820 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.765691562Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-21T09:39:56.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.766121364Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-21T09:39:56.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.766905166Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-21T09:39:56.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.767687751Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-21T09:39:56.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.7685894Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-21T09:39:56.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.772175433Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-21T09:39:56.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.775531808Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-21T09:39:56.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.776433914Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-21T09:39:56.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.777047143Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-21T09:39:56.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.777882238Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-21T09:39:56.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.778299331Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-21T09:39:56.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.779082523Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-21T09:39:56.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.779935606Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-21T09:39:56.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.780824186Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-21T09:39:56.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.781109793Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-21T09:39:56.821 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.781965471Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-21T09:39:56.822 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.782806514Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-21T09:39:56.822 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.783654444Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-21T09:39:56.822 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.784057751Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-21T09:39:56.822 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.785021339Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-21T09:39:56.822 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.786079279Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-21T09:39:56.822 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.786915468Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-21T09:39:56.822 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.787766738Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-21T09:39:56.822 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.788571469Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-21T09:39:56.822 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.789424034Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-21T09:39:56.822 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.790313944Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-21T09:39:56.822 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.791294904Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-21T09:39:56.822 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.792092796Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-21T09:39:56.822 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.79280987Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-21T09:39:56.823 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.793619874Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-21T09:39:56.823 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.796587366Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-21T09:39:56.823 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.79755659Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-21T09:39:56.823 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.80037798Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-21T09:39:56.823 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.803143159Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-21T09:39:56.823 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.805919246Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-21T09:39:56.823 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.806446614Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-21T09:39:56.823 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.809162337Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-21T09:39:57.081 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.811971742Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-21T09:39:57.082 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.81468827Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-21T09:39:57.082 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.815626755Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-21T09:39:57.082 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.818352071Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-21T09:39:57.082 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.821111424Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-21T09:39:57.082 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.82378773Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-21T09:39:57.082 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.824241705Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-21T09:39:57.082 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.827171648Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-21T09:39:57.082 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.8297649Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-21T09:39:57.082 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.832577129Z level=info msg="Executing migration" id="create alert_image table" 2024-09-21T09:39:57.082 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.833055364Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-21T09:39:57.083 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.835734607Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-21T09:39:57.083 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.838385529Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-21T09:39:57.083 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.840954529Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-21T09:39:57.083 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.84146014Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-21T09:39:57.083 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.841534807Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-21T09:39:57.083 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.844255637Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-21T09:39:57.083 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.846937511Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-21T09:39:57.083 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.849451525Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-21T09:39:57.083 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.849565283Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-21T09:39:57.083 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.849679277Z level=info msg="alerts found to migrate" alerts=0 2024-09-21T09:39:57.083 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.850062051Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-21T09:39:57.084 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.852623486Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-21T09:39:57.084 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.855170822Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-21T09:39:57.084 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.857982975Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-21T09:39:57.084 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.858452646Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-21T09:39:57.084 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.860847802Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-21T09:39:57.084 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.863579209Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-21T09:39:57.084 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.865996681Z level=info msg="Executing migration" id="create data_keys table" 2024-09-21T09:39:57.084 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.866451399Z level=info msg="Executing migration" id="create secrets table" 2024-09-21T09:39:57.084 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.869057872Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-21T09:39:57.084 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.873432652Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-21T09:39:57.084 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.874312981Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-21T09:39:57.084 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.874647117Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-21T09:39:57.084 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.878885472Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-21T09:39:57.084 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.883203324Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-21T09:39:57.085 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.884123673Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-21T09:39:57.085 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.884766053Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-21T09:39:57.085 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.885708903Z level=info msg="Executing migration" id="create permission table" 2024-09-21T09:39:57.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.886411198Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-21T09:39:57.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.887245018Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-21T09:39:57.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.887716739Z level=info msg="Executing migration" id="create role table" 2024-09-21T09:39:57.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.888542624Z level=info msg="Executing migration" id="add column display_name" 2024-09-21T09:39:57.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.889462879Z level=info msg="Executing migration" id="add column group_name" 2024-09-21T09:39:57.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.890384024Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-21T09:39:57.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.890833647Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-21T09:39:57.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.891601914Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-21T09:39:57.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.892422991Z level=info msg="Executing migration" id="create team role table" 2024-09-21T09:39:57.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.893229185Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-21T09:39:57.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.893680391Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-21T09:39:57.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.894466858Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-21T09:39:57.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.895295522Z level=info msg="Executing migration" id="create user role table" 2024-09-21T09:39:57.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.896108377Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-21T09:39:57.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.896569968Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-21T09:39:57.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.897413529Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-21T09:39:57.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.898349279Z level=info msg="Executing migration" id="create builtin role table" 2024-09-21T09:39:57.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.899091948Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-21T09:39:57.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.89994397Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-21T09:39:57.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.900925904Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-21T09:39:57.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.901993043Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-21T09:39:57.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.902967925Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-21T09:39:57.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.903698795Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-21T09:39:57.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.904563746Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-21T09:39:57.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.905437959Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-21T09:39:57.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.906367869Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-21T09:39:57.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.90712398Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-21T09:39:57.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.90987256Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-21T09:39:57.088 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.91252457Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-21T09:39:57.088 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.91526189Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-21T09:39:57.088 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.918216902Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-21T09:39:57.088 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.920921013Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-21T09:39:57.088 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.923737471Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-21T09:39:57.088 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.926372622Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-21T09:39:57.088 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.92918439Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-21T09:39:57.088 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.93190326Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-21T09:39:57.088 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.934622947Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-21T09:39:57.088 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.93732649Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-21T09:39:57.088 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.940058149Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-21T09:39:57.088 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.942544653Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-21T09:39:57.088 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.945327423Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-21T09:39:57.089 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.9479957Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-21T09:39:57.089 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.950887184Z level=info msg="Executing migration" id="add correlation config column" 2024-09-21T09:39:57.089 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.953468704Z level=info msg="Executing migration" id="create entity_events table" 2024-09-21T09:39:57.089 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.956142396Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-21T09:39:57.089 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.958758614Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-21T09:39:57.089 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.958856824Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-21T09:39:57.089 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.961454856Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T09:39:57.089 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.96154252Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T09:39:57.089 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.964172871Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-21T09:39:57.089 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.966678183Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-21T09:39:57.089 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.969539791Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-21T09:39:57.089 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.972090418Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T09:39:57.089 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.974702741Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-21T09:39:57.089 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.977461144Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-21T09:39:57.090 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.980018386Z level=info msg="Executing migration" id="Drop public config table" 2024-09-21T09:39:57.090 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.980453134Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-21T09:39:57.090 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.980877532Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-21T09:39:57.090 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.981451664Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-21T09:39:57.090 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.98190616Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-21T09:39:57.090 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.982357113Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-21T09:39:57.090 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.985346132Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-21T09:39:57.090 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.986357743Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-21T09:39:57.090 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.987384299Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-21T09:39:57.090 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.987705742Z level=info msg="Executing migration" id="add share column" 2024-09-21T09:39:57.090 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.988705241Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-21T09:39:57.090 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.989125575Z level=info msg="Executing migration" id="create file table" 2024-09-21T09:39:57.090 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.989561475Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-21T09:39:57.090 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.989986386Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-21T09:39:57.090 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.990492164Z level=info msg="Executing migration" id="create file_meta table" 2024-09-21T09:39:57.091 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.991496606Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-21T09:39:57.091 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.992182933Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-21T09:39:57.091 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.992986661Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-21T09:39:57.091 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.993611191Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-21T09:39:57.091 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.993930118Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-21T09:39:57.091 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.994808686Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-21T09:39:57.091 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.99589785Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-21T09:39:57.091 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.996217026Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-21T09:39:57.091 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.997013503Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-21T09:39:57.091 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.997886922Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-21T09:39:57.091 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.998691577Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-21T09:39:57.091 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:56.999633586Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-21T09:39:57.091 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:57.000826403Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-21T09:39:57.091 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:57.001893256Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-21T09:39:57.092 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:57.002779533Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-21T09:39:57.092 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:57.012670752Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-21T09:39:57.092 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:57.013595865Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-21T09:39:57.092 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:57.014416953Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-21T09:39:57.092 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:57.017924613Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-21T09:39:57.092 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:57.018821877Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-21T09:39:57.092 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:57.019755432Z level=info msg="Executing migration" id="create folder table" 2024-09-21T09:39:57.092 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:57.020504226Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-21T09:39:57.092 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:57.021548481Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-21T09:39:57.092 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:57.022209559Z level=info msg="Executing migration" id="Update folder title length" 2024-09-21T09:39:57.092 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:57.023105416Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-21T09:39:57.092 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=migrator t=2024-09-21T09:39:57.023730358Z level=info msg="migrations completed" performed=485 skipped=0 duration=654.83429ms 2024-09-21T09:39:57.092 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=sqlstore t=2024-09-21T09:39:57.024107505Z level=info msg="Created default organization" 2024-09-21T09:39:57.093 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=plugin.loader t=2024-09-21T09:39:57.080091489Z level=info msg="Plugin registered" pluginID=input 2024-09-21T09:39:57.366 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:39:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:39:57.367 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=plugin.loader t=2024-09-21T09:39:57.107382246Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-21T09:39:57.367 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=plugin.loader t=2024-09-21T09:39:57.107402853Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-21T09:39:57.367 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=secrets t=2024-09-21T09:39:57.107446471Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-21T09:39:57.367 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=query_data t=2024-09-21T09:39:57.109062475Z level=info msg="Query Service initialization" 2024-09-21T09:39:57.367 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=live.push_http t=2024-09-21T09:39:57.111354017Z level=info msg="Live Push Gateway initialization" 2024-09-21T09:39:57.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-mon[32729]: Deploying daemon node-exporter.a on smithi107 2024-09-21T09:39:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:57 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:57 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:57 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:57 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:57 smithi107 ceph-mon[29382]: Deploying daemon node-exporter.a on smithi107 2024-09-21T09:39:57.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:57 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:57.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:57 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:57.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:57 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:57.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:57 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:39:57.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:57 smithi107 ceph-mon[41185]: Deploying daemon node-exporter.a on smithi107 2024-09-21T09:39:58.169 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=infra.usagestats.collector t=2024-09-21T09:39:57.820475458Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-21T09:39:58.169 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=provisioning.datasources t=2024-09-21T09:39:57.820789478Z level=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-09-21T09:39:58.169 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=provisioning.datasources t=2024-09-21T09:39:57.853432802Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-21T09:39:58.169 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=provisioning.datasources t=2024-09-21T09:39:57.885259323Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-21T09:39:58.169 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=provisioning.alerting t=2024-09-21T09:39:57.906808912Z level=info msg="starting to provision alerting" 2024-09-21T09:39:58.169 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=provisioning.alerting t=2024-09-21T09:39:57.90682803Z level=info msg="finished to provision alerting" 2024-09-21T09:39:58.170 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=ngalert.state.manager t=2024-09-21T09:39:57.906932868Z level=info msg="Warming state cache for startup" 2024-09-21T09:39:58.170 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=grafanaStorageLogger t=2024-09-21T09:39:57.907270736Z level=info msg="storage starting" 2024-09-21T09:39:58.170 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=http.server t=2024-09-21T09:39:57.908868808Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-21T09:39:58.170 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=sqlstore.transactions t=2024-09-21T09:39:57.91723133Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T09:39:58.170 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=ngalert.state.manager t=2024-09-21T09:39:57.917295142Z level=info msg="State cache has been initialized" states=0 duration=10.358491ms 2024-09-21T09:39:58.170 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=ticker t=2024-09-21T09:39:57.917455502Z level=info msg=starting first_tick=2024-09-21T09:40:00Z 2024-09-21T09:39:58.170 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=ngalert.multiorg.alertmanager t=2024-09-21T09:39:57.917438409Z level=info msg="starting MultiOrg Alertmanager" 2024-09-21T09:39:58.170 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=sqlstore.transactions t=2024-09-21T09:39:57.962866002Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T09:39:58.170 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=sqlstore.transactions t=2024-09-21T09:39:57.974192617Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-21T09:39:58.170 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=sqlstore.transactions t=2024-09-21T09:39:57.985537452Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T09:39:58.170 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:39:58 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[64026]: logger=sqlstore.transactions t=2024-09-21T09:39:58.02787455Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T09:39:58.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:39:58 smithi107 ceph-mon[29382]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:58.531 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:39:58 smithi107 ceph-mon[41185]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:39:58.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:39:58 smithi129 ceph-mon[32729]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:00.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:00 smithi107 ceph-mon[29382]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:00 smithi107 ceph-mon[29382]: overall HEALTH_OK 2024-09-21T09:40:00.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:00 smithi107 ceph-mon[41185]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:00.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:00 smithi107 ceph-mon[41185]: overall HEALTH_OK 2024-09-21T09:40:00.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:00 smithi129 ceph-mon[32729]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:00.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:00 smithi129 ceph-mon[32729]: overall HEALTH_OK 2024-09-21T09:40:01.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:01 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:40:01.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:01 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:01.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:01 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:40:01.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:01 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:01.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:01 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:40:01.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:01 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:02.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:02 smithi129 ceph-mon[32729]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:02.724 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:02 smithi107 ceph-mon[29382]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:02.725 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:02 smithi107 ceph-mon[41185]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:04.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:04 smithi107 ceph-mon[29382]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:04.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:04 smithi107 ceph-mon[41185]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:04.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:04 smithi129 ceph-mon[32729]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:06.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:06 smithi129 ceph-mon[32729]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:06.713 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:06 smithi107 ceph-mon[29382]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:06.714 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:06 smithi107 ceph-mon[41185]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:07.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:40:07 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:40:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:40:07.830 INFO:teuthology.orchestra.run.smithi107.stdout:55834574892 2024-09-21T09:40:07.830 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T09:40:08.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:08 smithi107 ceph-mon[29382]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:08.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:08 smithi107 ceph-mon[41185]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:08.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:08 smithi129 ceph-mon[32729]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:08.905 INFO:teuthology.orchestra.run.smithi107.stdout:77309411366 2024-09-21T09:40:08.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T09:40:09.580 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:09 smithi107 bash[74116]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-21T09:40:09.602 INFO:teuthology.orchestra.run.smithi107.stdout:34359738416 2024-09-21T09:40:09.603 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T09:40:09.858 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:09 smithi107 bash[74116]: Getting image source signatures 2024-09-21T09:40:09.858 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:09 smithi107 bash[74116]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-21T09:40:09.858 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:09 smithi107 bash[74116]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-21T09:40:09.859 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:09 smithi107 bash[74116]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-21T09:40:10.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:10 smithi107 ceph-mon[29382]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:10.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:10 smithi107 ceph-mon[41185]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:10.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:10 smithi129 ceph-mon[32729]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:11.858 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:11 smithi107 bash[74116]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-21T09:40:12.358 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:11 smithi107 bash[74116]: Writing manifest to image destination 2024-09-21T09:40:12.568 INFO:teuthology.orchestra.run.smithi107.stdout:128849018908 2024-09-21T09:40:12.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T09:40:12.667 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:12 smithi107 ceph-mon[29382]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:12.668 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:12 smithi107 ceph-mon[41185]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:12.668 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:12 smithi107 podman[74116]: 2024-09-21 09:40:12.417059207 +0000 UTC m=+3.309410069 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-21T09:40:12.668 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:12 smithi107 podman[74116]: 2024-09-21 09:40:12.666422625 +0000 UTC m=+3.558773475 container create bbf92f500b2cb4457383b94e39ef31e0a898876384ad8f7997a5414928992207 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:40:12.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:12 smithi129 ceph-mon[32729]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:13.976 INFO:teuthology.orchestra.run.smithi107.stdout:180388626452 2024-09-21T09:40:13.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T09:40:13.999 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 podman[74116]: 2024-09-21 09:40:13.742706146 +0000 UTC m=+4.635056994 container init bbf92f500b2cb4457383b94e39ef31e0a898876384ad8f7997a5414928992207 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:40:14.000 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 podman[74116]: 2024-09-21 09:40:13.749177791 +0000 UTC m=+4.641528647 container start bbf92f500b2cb4457383b94e39ef31e0a898876384ad8f7997a5414928992207 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:40:14.000 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.753Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-21T09:40:14.000 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.753Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-21T09:40:14.000 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.753Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T09:40:14.000 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.753Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T09:40:14.000 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T09:40:14.000 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T09:40:14.000 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T09:40:14.000 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T09:40:14.000 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T09:40:14.000 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T09:40:14.000 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T09:40:14.000 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T09:40:14.001 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T09:40:14.001 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T09:40:14.001 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T09:40:14.001 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T09:40:14.001 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T09:40:14.001 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T09:40:14.001 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T09:40:14.002 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T09:40:14.002 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T09:40:14.002 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T09:40:14.002 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T09:40:14.002 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T09:40:14.002 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T09:40:14.002 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T09:40:14.002 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T09:40:14.002 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T09:40:14.002 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T09:40:14.002 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T09:40:14.002 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T09:40:14.002 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T09:40:14.003 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T09:40:14.003 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T09:40:14.003 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T09:40:14.003 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T09:40:14.003 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T09:40:14.003 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T09:40:14.003 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T09:40:14.003 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T09:40:14.003 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T09:40:14.003 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T09:40:14.003 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T09:40:14.003 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T09:40:14.003 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T09:40:14.003 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T09:40:14.004 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T09:40:14.004 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T09:40:14.004 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T09:40:14.004 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T09:40:14.004 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.754Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T09:40:14.004 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.755Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-21T09:40:14.004 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[74411]: ts=2024-09-21T09:40:13.755Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T09:40:14.004 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 bash[74116]: bbf92f500b2cb4457383b94e39ef31e0a898876384ad8f7997a5414928992207 2024-09-21T09:40:14.004 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:40:13 smithi107 systemd[1]: Started Ceph node-exporter.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:40:14.297 INFO:teuthology.orchestra.run.smithi107.stdout:154618822679 2024-09-21T09:40:14.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T09:40:14.327 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:40:14.569 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:14 smithi129 ceph-mon[32729]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:14.569 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:14 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:14.569 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:14 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:14.569 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:14 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:14.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:14 smithi107 ceph-mon[29382]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:14.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:14 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:14.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:14 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:14 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:14 smithi107 ceph-mon[41185]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:14 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:14 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:14 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:14.919 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:14 smithi129 bash[64306]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-21T09:40:14.943 INFO:teuthology.orchestra.run.smithi107.stdout:107374182434 2024-09-21T09:40:14.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T09:40:15.382 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:40:15.385 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:40:15.404 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:40:15.418 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:15 smithi129 bash[64306]: Getting image source signatures 2024-09-21T09:40:15.418 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:15 smithi129 bash[64306]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-21T09:40:15.418 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:15 smithi129 bash[64306]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-21T09:40:15.418 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:15 smithi129 bash[64306]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-21T09:40:15.420 INFO:teuthology.orchestra.run.smithi107.stdout:206158430223 2024-09-21T09:40:15.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T09:40:15.634 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:15 smithi107 ceph-mon[29382]: Deploying daemon node-exporter.b on smithi129 2024-09-21T09:40:15.634 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:15 smithi107 ceph-mon[41185]: Deploying daemon node-exporter.b on smithi129 2024-09-21T09:40:15.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:15 smithi129 ceph-mon[32729]: Deploying daemon node-exporter.b on smithi129 2024-09-21T09:40:16.722 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:16 smithi129 ceph-mon[32729]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:16.722 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:16 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:40:16.722 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:16 smithi129 bash[64306]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-21T09:40:16.834 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:40:16.835 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:40:16.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:16 smithi107 ceph-mon[29382]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:16.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:16 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:40:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:16 smithi107 ceph-mon[41185]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:16 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:40:17.086 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:16 smithi129 bash[64306]: Writing manifest to image destination 2024-09-21T09:40:17.411 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:40:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:40:17.411 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 podman[64306]: 2024-09-21 09:40:17.018323838 +0000 UTC m=+2.356279622 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-21T09:40:17.411 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 podman[64306]: 2024-09-21 09:40:17.100336162 +0000 UTC m=+2.438291988 container create a71c52ae1a694cf0a1f9f4115b77e100f07afcfd6db6ab148b0f9b9557b91537 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:40:17.412 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 podman[64306]: 2024-09-21 09:40:17.302454234 +0000 UTC m=+2.640410021 container init a71c52ae1a694cf0a1f9f4115b77e100f07afcfd6db6ab148b0f9b9557b91537 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:40:17.412 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 podman[64306]: 2024-09-21 09:40:17.307558317 +0000 UTC m=+2.645514102 container start a71c52ae1a694cf0a1f9f4115b77e100f07afcfd6db6ab148b0f9b9557b91537 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:40:17.412 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.310Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-21T09:40:17.412 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.310Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-21T09:40:17.412 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.311Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T09:40:17.412 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.311Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T09:40:17.412 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.311Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T09:40:17.412 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.311Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T09:40:17.412 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T09:40:17.412 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T09:40:17.412 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T09:40:17.412 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T09:40:17.412 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T09:40:17.413 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T09:40:17.413 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T09:40:17.413 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T09:40:17.413 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T09:40:17.413 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T09:40:17.413 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T09:40:17.413 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T09:40:17.413 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T09:40:17.414 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T09:40:17.414 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T09:40:17.414 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T09:40:17.414 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T09:40:17.414 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T09:40:17.414 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T09:40:17.414 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T09:40:17.414 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T09:40:17.414 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T09:40:17.414 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T09:40:17.414 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T09:40:17.414 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T09:40:17.414 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T09:40:17.414 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T09:40:17.415 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T09:40:17.415 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T09:40:17.415 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T09:40:17.415 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T09:40:17.415 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T09:40:17.415 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T09:40:17.415 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T09:40:17.415 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T09:40:17.415 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T09:40:17.415 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T09:40:17.415 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T09:40:17.415 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T09:40:17.415 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T09:40:17.415 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T09:40:17.416 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T09:40:17.416 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T09:40:17.416 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T09:40:17.416 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T09:40:17.416 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-21T09:40:17.416 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[64379]: ts=2024-09-21T09:40:17.312Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T09:40:17.416 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 bash[64306]: a71c52ae1a694cf0a1f9f4115b77e100f07afcfd6db6ab148b0f9b9557b91537 2024-09-21T09:40:17.668 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:40:17 smithi129 systemd[1]: Started Ceph node-exporter.b for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:40:17.765 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:40:18.745 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:18 smithi107 ceph-mon[29382]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:18.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:18 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:18.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:18 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:18.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:18 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:18.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:18 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:18.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:18 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:18.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:18 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:40:18.746 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:18 smithi107 ceph-mon[41185]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:18.746 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:18 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:18.746 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:18 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:18.746 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:18 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:18.747 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:18 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:18.747 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:18 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:18.747 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:18 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:40:18.762 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:18 smithi129 ceph-mon[32729]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:18.762 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:18 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:18.763 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:18 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:18.763 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:18 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:18.763 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:18 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:18.763 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:18 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:18.763 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:18 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:40:19.362 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:40:19.946 INFO:teuthology.orchestra.run.smithi107.stdout:55834574894 2024-09-21T09:40:20.735 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:20 smithi107 ceph-mon[29382]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:20.735 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:20 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/906213474' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:40:20.735 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:20 smithi107 ceph-mon[41185]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:20.735 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:20 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/906213474' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:40:20.768 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:20 smithi129 ceph-mon[32729]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:20.768 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:20 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/906213474' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:40:22.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:22 smithi107 ceph-mon[29382]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:22.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:22 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:22.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:22 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:22.182 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:22 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:22.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:22 smithi107 ceph-mon[41185]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:22.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:22 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:22.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:22 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:22.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:22 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:22.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:22 smithi129 ceph-mon[32729]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:22.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:22 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:22.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:22 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:22.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:22 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:22.488 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574894 for osd.1 2024-09-21T09:40:22.489 DEBUG:teuthology.parallel:result is None 2024-09-21T09:40:24.274 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:24 smithi107 ceph-mon[29382]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:24.275 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:24 smithi107 ceph-mon[41185]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:24.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:24 smithi129 ceph-mon[32729]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:25.340 INFO:teuthology.orchestra.run.smithi107.stdout:77309411370 2024-09-21T09:40:25.539 INFO:teuthology.orchestra.run.smithi107.stdout:34359738420 2024-09-21T09:40:26.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:26 smithi107 ceph-mon[29382]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:26.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:26 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1663902040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:40:26.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:26 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3257940244' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:40:26.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:26 smithi107 ceph-mon[41185]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:26.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:26 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1663902040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:40:26.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:26 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3257940244' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:40:26.137 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738416 got 34359738420 for osd.0 2024-09-21T09:40:26.137 DEBUG:teuthology.parallel:result is None 2024-09-21T09:40:26.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:26 smithi129 ceph-mon[32729]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:26.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:26 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1663902040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:40:26.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:26 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3257940244' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:40:26.518 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411366 got 77309411370 for osd.2 2024-09-21T09:40:26.519 DEBUG:teuthology.parallel:result is None 2024-09-21T09:40:27.097 INFO:teuthology.orchestra.run.smithi107.stdout:206158430227 2024-09-21T09:40:27.668 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:40:27 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:40:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:40:28.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:28 smithi107 ceph-mon[29382]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:28.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:28 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3159860628' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:40:28.291 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:28 smithi107 ceph-mon[41185]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:28.291 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:28 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3159860628' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:40:28.398 INFO:teuthology.orchestra.run.smithi107.stdout:107374182438 2024-09-21T09:40:28.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:28 smithi129 ceph-mon[32729]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:28.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:28 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3159860628' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:40:28.467 INFO:teuthology.orchestra.run.smithi107.stdout:128849018913 2024-09-21T09:40:29.158 INFO:teuthology.orchestra.run.smithi107.stdout:180388626455 2024-09-21T09:40:29.169 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:29 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3808699154' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:40:29.169 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:29 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/855012395' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:40:29.169 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:29 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3808699154' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:40:29.169 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:29 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/855012395' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:40:29.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:29 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3808699154' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:40:29.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:29 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/855012395' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:40:29.598 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018908 got 128849018913 for osd.4 2024-09-21T09:40:29.598 DEBUG:teuthology.parallel:result is None 2024-09-21T09:40:30.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:30 smithi107 ceph-mon[29382]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:30.177 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:30 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3002464523' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:40:30.177 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:30 smithi107 ceph-mon[41185]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:30.177 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:30 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3002464523' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:40:30.178 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626452 got 180388626455 for osd.6 2024-09-21T09:40:30.178 DEBUG:teuthology.parallel:result is None 2024-09-21T09:40:30.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:30 smithi129 ceph-mon[32729]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:30.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:30 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3002464523' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:40:30.821 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182434 got 107374182438 for osd.3 2024-09-21T09:40:30.821 DEBUG:teuthology.parallel:result is None 2024-09-21T09:40:31.329 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:40:31.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:40:31.351 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430223 got 206158430227 for osd.7 2024-09-21T09:40:31.352 DEBUG:teuthology.parallel:result is None 2024-09-21T09:40:31.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:40:31.688 INFO:teuthology.orchestra.run.smithi107.stdout:154618822684 2024-09-21T09:40:32.259 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:32 smithi107 ceph-mon[29382]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:32.259 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:32 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2219751785' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:40:32.259 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:32 smithi107 ceph-mon[41185]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:32.259 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:32 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/2219751785' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:40:32.285 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822679 got 154618822684 for osd.5 2024-09-21T09:40:32.285 DEBUG:teuthology.parallel:result is None 2024-09-21T09:40:32.285 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T09:40:32.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T09:40:32.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:32 smithi129 ceph-mon[32729]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:32.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:32 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2219751785' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:40:32.767 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:40:34.145 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:34 smithi107 ceph-mon[29382]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:34.145 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:34 smithi107 ceph-mon[41185]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:34.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:34 smithi129 ceph-mon[32729]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:34.491 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:40:34.491 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T09:40:35.302 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:35 smithi107 ceph-mon[29382]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:40:35.302 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:35 smithi107 ceph-mon[41185]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:40:35.331 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-09-21T09:40:33.001421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":115,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385100,"kb_used_data":6252,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747576116,"statfs":{"total":767960285184,"available":765517942784,"internally_reserved":0,"allocated":6402048,"data_stored":3716368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001785"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-21T09:39:48.881934+0000","last_change":"2024-09-21T09:39:39.496046+0000","last_active":"2024-09-21T09:39:48.881934+0000","last_peered":"2024-09-21T09:39:48.881934+0000","last_clean":"2024-09-21T09:39:48.881934+0000","last_became_active":"2024-09-21T09:39:34.903061+0000","last_became_peered":"2024-09-21T09:39:34.903061+0000","last_unstale":"2024-09-21T09:39:48.881934+0000","last_undegraded":"2024-09-21T09:39:48.881934+0000","last_fullsized":"2024-09-21T09:39:48.881934+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:41:37.281272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-21T09:39:48.443501+0000","last_change":"2024-09-21T09:39:39.495635+0000","last_active":"2024-09-21T09:39:48.443501+0000","last_peered":"2024-09-21T09:39:48.443501+0000","last_clean":"2024-09-21T09:39:48.443501+0000","last_became_active":"2024-09-21T09:39:35.595263+0000","last_became_peered":"2024-09-21T09:39:35.595263+0000","last_unstale":"2024-09-21T09:39:48.443501+0000","last_undegraded":"2024-09-21T09:39:48.443501+0000","last_fullsized":"2024-09-21T09:39:48.443501+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:29.108133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00019257899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-21T09:39:48.996591+0000","last_change":"2024-09-21T09:39:39.583831+0000","last_active":"2024-09-21T09:39:48.996591+0000","last_peered":"2024-09-21T09:39:48.996591+0000","last_clean":"2024-09-21T09:39:48.996591+0000","last_became_active":"2024-09-21T09:39:35.595483+0000","last_became_peered":"2024-09-21T09:39:35.595483+0000","last_unstale":"2024-09-21T09:39:48.996591+0000","last_undegraded":"2024-09-21T09:39:48.996591+0000","last_fullsized":"2024-09-21T09:39:48.996591+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:06.734095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-21T09:39:48.443681+0000","last_change":"2024-09-21T09:39:39.495639+0000","last_active":"2024-09-21T09:39:48.443681+0000","last_peered":"2024-09-21T09:39:48.443681+0000","last_clean":"2024-09-21T09:39:48.443681+0000","last_became_active":"2024-09-21T09:39:34.987584+0000","last_became_peered":"2024-09-21T09:39:34.987584+0000","last_unstale":"2024-09-21T09:39:48.443681+0000","last_undegraded":"2024-09-21T09:39:48.443681+0000","last_fullsized":"2024-09-21T09:39:48.443681+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:31.185487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":21,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-21T09:39:49.234179+0000","last_change":"2024-09-21T09:39:39.495072+0000","last_active":"2024-09-21T09:39:49.234179+0000","last_peered":"2024-09-21T09:39:49.234179+0000","last_clean":"2024-09-21T09:39:49.234179+0000","last_became_active":"2024-09-21T09:39:34.903742+0000","last_became_peered":"2024-09-21T09:39:34.903742+0000","last_unstale":"2024-09-21T09:39:49.234179+0000","last_undegraded":"2024-09-21T09:39:49.234179+0000","last_fullsized":"2024-09-21T09:39:49.234179+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:32.765230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091939799999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-21T09:39:49.160732+0000","last_change":"2024-09-21T09:39:39.485018+0000","last_active":"2024-09-21T09:39:49.160732+0000","last_peered":"2024-09-21T09:39:49.160732+0000","last_clean":"2024-09-21T09:39:49.160732+0000","last_became_active":"2024-09-21T09:39:35.184521+0000","last_became_peered":"2024-09-21T09:39:35.184521+0000","last_unstale":"2024-09-21T09:39:49.160732+0000","last_undegraded":"2024-09-21T09:39:49.160732+0000","last_fullsized":"2024-09-21T09:39:49.160732+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:29.520758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-21T09:39:48.996457+0000","last_change":"2024-09-21T09:39:39.583589+0000","last_active":"2024-09-21T09:39:48.996457+0000","last_peered":"2024-09-21T09:39:48.996457+0000","last_clean":"2024-09-21T09:39:48.996457+0000","last_became_active":"2024-09-21T09:39:34.987518+0000","last_became_peered":"2024-09-21T09:39:34.987518+0000","last_unstale":"2024-09-21T09:39:48.996457+0000","last_undegraded":"2024-09-21T09:39:48.996457+0000","last_fullsized":"2024-09-21T09:39:48.996457+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:39.087119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-21T09:39:49.234223+0000","last_change":"2024-09-21T09:39:39.494546+0000","last_active":"2024-09-21T09:39:49.234223+0000","last_peered":"2024-09-21T09:39:49.234223+0000","last_clean":"2024-09-21T09:39:49.234223+0000","last_became_active":"2024-09-21T09:39:34.903724+0000","last_became_peered":"2024-09-21T09:39:34.903724+0000","last_unstale":"2024-09-21T09:39:49.234223+0000","last_undegraded":"2024-09-21T09:39:49.234223+0000","last_fullsized":"2024-09-21T09:39:49.234223+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:38:35.213577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039581700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"51'79","reported_seq":49,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-21T09:39:39.583250+0000","last_change":"2024-09-21T09:39:04.444589+0000","last_active":"2024-09-21T09:39:39.583250+0000","last_peered":"2024-09-21T09:39:39.583250+0000","last_clean":"2024-09-21T09:39:39.583250+0000","last_became_active":"2024-09-21T09:39:04.439052+0000","last_became_peered":"2024-09-21T09:39:04.439052+0000","last_unstale":"2024-09-21T09:39:39.583250+0000","last_undegraded":"2024-09-21T09:39:39.583250+0000","last_fullsized":"2024-09-21T09:39:39.583250+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:43.843355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430228,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298500,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446652,"statfs":{"total":95995035648,"available":95689371648,"internally_reserved":0,"allocated":1171456,"data_stored":833637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.46200000000000002}]},{"osd":1,"last update":"Sat Sep 21 09:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.68899999999999995}]},{"osd":2,"last update":"Sat Sep 21 09:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.48999999999999999}]},{"osd":3,"last update":"Sat Sep 21 09:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.52100000000000002}]},{"osd":4,"last update":"Sat Sep 21 09:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.47499999999999998}]},{"osd":5,"last update":"Sat Sep 21 09:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.50600000000000001}]},{"osd":6,"last update":"Sat Sep 21 09:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.63}]}]},{"osd":6,"up_from":42,"seq":180388626456,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298504,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446648,"statfs":{"total":95995035648,"available":95689367552,"internally_reserved":0,"allocated":1175552,"data_stored":833656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.54000000000000004}]},{"osd":1,"last update":"Sat Sep 21 09:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.47999999999999998}]},{"osd":2,"last update":"Sat Sep 21 09:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.38},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.442}]},{"osd":3,"last update":"Sat Sep 21 09:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.60999999999999999}]},{"osd":4,"last update":"Sat Sep 21 09:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.65000000000000002}]},{"osd":5,"last update":"Sat Sep 21 09:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.625},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.66500000000000004}]},{"osd":7,"last update":"Sat Sep 21 09:40:02 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.69199999999999995}]}]},{"osd":5,"up_from":36,"seq":154618822685,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297916,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447236,"statfs":{"total":95995035648,"available":95689969664,"internally_reserved":0,"allocated":573440,"data_stored":242658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:40:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.51800000000000002,"15min":0.52500000000000002},"min":{"1min":0.376,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68300000000000005,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.435},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.50600000000000001,"15min":0.504},"min":{"1min":0.34899999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.41599999999999998}]},{"osd":1,"last update":"Sat Sep 21 09:40:18 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.34300000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.63900000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.36399999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.45800000000000002}]},{"osd":2,"last update":"Sat Sep 21 09:40:18 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52800000000000002,"15min":0.53200000000000003},"min":{"1min":0.35199999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.66500000000000004,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52700000000000002,"15min":0.52300000000000002},"min":{"1min":0.33900000000000002,"5min":0.312,"15min":0.312},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.502}]},{"osd":3,"last update":"Sat Sep 21 09:40:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54400000000000004,"15min":0.54500000000000004},"min":{"1min":0.38300000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.39400000000000002}]},{"osd":4,"last update":"Sat Sep 21 09:40:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.38600000000000001,"15min":0.375},"min":{"1min":0.28999999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.40400000000000003,"15min":0.39600000000000002},"min":{"1min":0.28799999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.60499999999999998}]},{"osd":6,"last update":"Sat Sep 21 09:39:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.54900000000000004}]},{"osd":7,"last update":"Sat Sep 21 09:40:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.627}]}]},{"osd":4,"up_from":30,"seq":128849018914,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297920,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447232,"statfs":{"total":95995035648,"available":95689965568,"internally_reserved":0,"allocated":577536,"data_stored":243269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:39:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.53900000000000003,"15min":0.54700000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.61199999999999999,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.623},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.55100000000000005,"15min":0.55700000000000005},"min":{"1min":0.39300000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68000000000000005,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.61599999999999999}]},{"osd":1,"last update":"Sat Sep 21 09:39:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.51900000000000002,"15min":0.52900000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.64100000000000001,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.437},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.499,"15min":0.503},"min":{"1min":0.35199999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.64900000000000002,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.48599999999999999}]},{"osd":2,"last update":"Sat Sep 21 09:39:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.318,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.625,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.53400000000000003,"15min":0.54200000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.56100000000000005}]},{"osd":3,"last update":"Sat Sep 21 09:39:57 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.54000000000000004,"15min":0.54700000000000004},"min":{"1min":0.38300000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73999999999999999,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.54000000000000004,"15min":0.54700000000000004},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.39600000000000002}]},{"osd":5,"last update":"Sat Sep 21 09:40:18 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.38800000000000001,"15min":0.38200000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.378},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.43099999999999999,"15min":0.42299999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.53300000000000003}]},{"osd":6,"last update":"Sat Sep 21 09:39:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.44600000000000001}]},{"osd":7,"last update":"Sat Sep 21 09:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.45800000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182439,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297920,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447232,"statfs":{"total":95995035648,"available":95689965568,"internally_reserved":0,"allocated":577536,"data_stored":243269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.38800000000000001,"15min":0.39200000000000002},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.214},"max":{"1min":0.53300000000000003,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.379,"5min":0.40699999999999997,"15min":0.41199999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.47899999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.48399999999999999}]},{"osd":1,"last update":"Sat Sep 21 09:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.39800000000000002,"15min":0.40100000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.54700000000000004,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.44,"15min":0.443},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.40000000000000002}]},{"osd":2,"last update":"Sat Sep 21 09:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.46500000000000002,"15min":0.46899999999999997},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.436},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.436,"15min":0.44},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.45200000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:39:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55600000000000005,"15min":0.55500000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.76300000000000001}]},{"osd":5,"last update":"Sat Sep 21 09:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.58399999999999996,"15min":0.58999999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70499999999999996,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58299999999999996,"15min":0.58899999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.71799999999999997,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.71099999999999997}]},{"osd":6,"last update":"Sat Sep 21 09:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.63100000000000001}]},{"osd":7,"last update":"Sat Sep 21 09:40:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.751},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.66100000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297920,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447232,"statfs":{"total":95995035648,"available":95689965568,"internally_reserved":0,"allocated":577536,"data_stored":243269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34399999999999997,"15min":0.34100000000000003},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.48799999999999999,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.36399999999999999,"15min":0.36299999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.44700000000000001,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.25700000000000001}]},{"osd":1,"last update":"Sat Sep 21 09:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.35899999999999999,"15min":0.36099999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.38400000000000001,"15min":0.38500000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.495,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.34499999999999997}]},{"osd":3,"last update":"Sat Sep 21 09:39:33 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.30199999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.56599999999999995,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.41999999999999998,"15min":0.42599999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:39:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55900000000000005,"15min":0.56299999999999994},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.78900000000000003,"5min":0.878,"15min":0.878},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56100000000000005,"15min":0.56699999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68000000000000005,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.55900000000000005}]},{"osd":5,"last update":"Sat Sep 21 09:40:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58299999999999996,"15min":0.58699999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76100000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.74099999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.752}]},{"osd":6,"last update":"Sat Sep 21 09:39:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.54200000000000004}]},{"osd":7,"last update":"Sat Sep 21 09:40:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.74199999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297920,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447232,"statfs":{"total":95995035648,"available":95689965568,"internally_reserved":0,"allocated":577536,"data_stored":242973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:39:45 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.32800000000000001,"15min":0.312},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.53700000000000003,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.33100000000000002,"15min":0.30499999999999999},"min":{"1min":0.255,"5min":0.184,"15min":0.184},"max":{"1min":0.51000000000000001,"5min":0.627,"15min":0.627},"last":0.25600000000000001}]},{"osd":2,"last update":"Sat Sep 21 09:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.32600000000000001},"min":{"1min":0.22800000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.497,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.378,"5min":0.37,"15min":0.35599999999999998},"min":{"1min":0.28599999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.46100000000000002,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.35099999999999998}]},{"osd":3,"last update":"Sat Sep 21 09:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.42499999999999999,"15min":0.42999999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.59299999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.436,"15min":0.439},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.55700000000000005,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.31900000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:39:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.41799999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.69599999999999995,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56899999999999995,"15min":0.57199999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.65300000000000002}]},{"osd":5,"last update":"Sat Sep 21 09:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54500000000000004,"15min":0.54400000000000004},"min":{"1min":0.39300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.39400000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.75800000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.58599999999999997}]},{"osd":6,"last update":"Sat Sep 21 09:39:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.55900000000000005}]},{"osd":7,"last update":"Sat Sep 21 09:40:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.60499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298500,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446652,"statfs":{"total":95995035648,"available":95689371648,"internally_reserved":0,"allocated":1171456,"data_stored":833637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 09:39:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.33100000000000002,"15min":0.308},"min":{"1min":0.26400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.51300000000000001,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.33300000000000002,"15min":0.317},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.50900000000000001,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.36499999999999999}]},{"osd":2,"last update":"Sat Sep 21 09:40:13 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.37,"15min":0.372},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.47599999999999998,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.38700000000000001,"15min":0.38500000000000001},"min":{"1min":0.27700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.58399999999999996,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.46800000000000003}]},{"osd":3,"last update":"Sat Sep 21 09:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.42899999999999999,"15min":0.43099999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.60099999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40999999999999998,"15min":0.41099999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.56399999999999995,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.441}]},{"osd":4,"last update":"Sat Sep 21 09:39:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.53000000000000003,"15min":0.53200000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.71499999999999997,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68400000000000005,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.498}]},{"osd":5,"last update":"Sat Sep 21 09:40:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.54800000000000004,"15min":0.55300000000000005},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.56100000000000005,"15min":0.56699999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.59099999999999997}]},{"osd":6,"last update":"Sat Sep 21 09:39:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.54000000000000004}]},{"osd":7,"last update":"Sat Sep 21 09:40:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.626}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T09:40:35.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T09:40:35.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:35 smithi129 ceph-mon[32729]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:40:35.767 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:40:36.223 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:36 smithi107 ceph-mon[29382]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:36.223 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:36 smithi107 ceph-mon[41185]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:36.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:36 smithi129 ceph-mon[32729]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:37.598 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:40:37 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:40:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:40:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:37 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:37 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:37 smithi129 ceph-mon[32729]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:37 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:40:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:37 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:40:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:37 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:37 smithi129 ceph-mon[32729]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:40:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:37 smithi129 ceph-mon[32729]: Reconfiguring daemon alertmanager.a on smithi107 2024-09-21T09:40:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:37 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:37 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:37 smithi107 ceph-mon[29382]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:37 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:40:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:37 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:40:38.043 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:37 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:38.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:37 smithi107 ceph-mon[29382]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:40:38.044 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:37 smithi107 ceph-mon[29382]: Reconfiguring daemon alertmanager.a on smithi107 2024-09-21T09:40:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:37 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:37 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:37 smithi107 ceph-mon[41185]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:37 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:40:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:37 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:40:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:37 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:37 smithi107 ceph-mon[41185]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:40:38.044 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:37 smithi107 ceph-mon[41185]: Reconfiguring daemon alertmanager.a on smithi107 2024-09-21T09:40:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:40:39.215 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T09:40:40.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:40 smithi107 ceph-mon[29382]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:40.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:40 smithi107 ceph-mon[29382]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:40:40.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:40 smithi107 ceph-mon[41185]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:40.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:40 smithi107 ceph-mon[41185]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:40:40.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:40 smithi129 ceph-mon[32729]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:40.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:40 smithi129 ceph-mon[32729]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:40:41.183 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-09-21T09:40:39.002342+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":115,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385100,"kb_used_data":6252,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747576116,"statfs":{"total":767960285184,"available":765517942784,"internally_reserved":0,"allocated":6402048,"data_stored":3716368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001708"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-21T09:39:48.881934+0000","last_change":"2024-09-21T09:39:39.496046+0000","last_active":"2024-09-21T09:39:48.881934+0000","last_peered":"2024-09-21T09:39:48.881934+0000","last_clean":"2024-09-21T09:39:48.881934+0000","last_became_active":"2024-09-21T09:39:34.903061+0000","last_became_peered":"2024-09-21T09:39:34.903061+0000","last_unstale":"2024-09-21T09:39:48.881934+0000","last_undegraded":"2024-09-21T09:39:48.881934+0000","last_fullsized":"2024-09-21T09:39:48.881934+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:41:37.281272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-21T09:39:48.443501+0000","last_change":"2024-09-21T09:39:39.495635+0000","last_active":"2024-09-21T09:39:48.443501+0000","last_peered":"2024-09-21T09:39:48.443501+0000","last_clean":"2024-09-21T09:39:48.443501+0000","last_became_active":"2024-09-21T09:39:35.595263+0000","last_became_peered":"2024-09-21T09:39:35.595263+0000","last_unstale":"2024-09-21T09:39:48.443501+0000","last_undegraded":"2024-09-21T09:39:48.443501+0000","last_fullsized":"2024-09-21T09:39:48.443501+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:29.108133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00019257899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-21T09:39:48.996591+0000","last_change":"2024-09-21T09:39:39.583831+0000","last_active":"2024-09-21T09:39:48.996591+0000","last_peered":"2024-09-21T09:39:48.996591+0000","last_clean":"2024-09-21T09:39:48.996591+0000","last_became_active":"2024-09-21T09:39:35.595483+0000","last_became_peered":"2024-09-21T09:39:35.595483+0000","last_unstale":"2024-09-21T09:39:48.996591+0000","last_undegraded":"2024-09-21T09:39:48.996591+0000","last_fullsized":"2024-09-21T09:39:48.996591+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:06.734095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-21T09:39:48.443681+0000","last_change":"2024-09-21T09:39:39.495639+0000","last_active":"2024-09-21T09:39:48.443681+0000","last_peered":"2024-09-21T09:39:48.443681+0000","last_clean":"2024-09-21T09:39:48.443681+0000","last_became_active":"2024-09-21T09:39:34.987584+0000","last_became_peered":"2024-09-21T09:39:34.987584+0000","last_unstale":"2024-09-21T09:39:48.443681+0000","last_undegraded":"2024-09-21T09:39:48.443681+0000","last_fullsized":"2024-09-21T09:39:48.443681+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:31.185487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":21,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-21T09:39:49.234179+0000","last_change":"2024-09-21T09:39:39.495072+0000","last_active":"2024-09-21T09:39:49.234179+0000","last_peered":"2024-09-21T09:39:49.234179+0000","last_clean":"2024-09-21T09:39:49.234179+0000","last_became_active":"2024-09-21T09:39:34.903742+0000","last_became_peered":"2024-09-21T09:39:34.903742+0000","last_unstale":"2024-09-21T09:39:49.234179+0000","last_undegraded":"2024-09-21T09:39:49.234179+0000","last_fullsized":"2024-09-21T09:39:49.234179+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:32.765230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091939799999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-21T09:39:49.160732+0000","last_change":"2024-09-21T09:39:39.485018+0000","last_active":"2024-09-21T09:39:49.160732+0000","last_peered":"2024-09-21T09:39:49.160732+0000","last_clean":"2024-09-21T09:39:49.160732+0000","last_became_active":"2024-09-21T09:39:35.184521+0000","last_became_peered":"2024-09-21T09:39:35.184521+0000","last_unstale":"2024-09-21T09:39:49.160732+0000","last_undegraded":"2024-09-21T09:39:49.160732+0000","last_fullsized":"2024-09-21T09:39:49.160732+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:29.520758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-21T09:39:48.996457+0000","last_change":"2024-09-21T09:39:39.583589+0000","last_active":"2024-09-21T09:39:48.996457+0000","last_peered":"2024-09-21T09:39:48.996457+0000","last_clean":"2024-09-21T09:39:48.996457+0000","last_became_active":"2024-09-21T09:39:34.987518+0000","last_became_peered":"2024-09-21T09:39:34.987518+0000","last_unstale":"2024-09-21T09:39:48.996457+0000","last_undegraded":"2024-09-21T09:39:48.996457+0000","last_fullsized":"2024-09-21T09:39:48.996457+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:39.087119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-21T09:39:49.234223+0000","last_change":"2024-09-21T09:39:39.494546+0000","last_active":"2024-09-21T09:39:49.234223+0000","last_peered":"2024-09-21T09:39:49.234223+0000","last_clean":"2024-09-21T09:39:49.234223+0000","last_became_active":"2024-09-21T09:39:34.903724+0000","last_became_peered":"2024-09-21T09:39:34.903724+0000","last_unstale":"2024-09-21T09:39:49.234223+0000","last_undegraded":"2024-09-21T09:39:49.234223+0000","last_fullsized":"2024-09-21T09:39:49.234223+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:38:35.213577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039581700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"51'79","reported_seq":49,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-21T09:39:39.583250+0000","last_change":"2024-09-21T09:39:04.444589+0000","last_active":"2024-09-21T09:39:39.583250+0000","last_peered":"2024-09-21T09:39:39.583250+0000","last_clean":"2024-09-21T09:39:39.583250+0000","last_became_active":"2024-09-21T09:39:04.439052+0000","last_became_peered":"2024-09-21T09:39:04.439052+0000","last_unstale":"2024-09-21T09:39:39.583250+0000","last_undegraded":"2024-09-21T09:39:39.583250+0000","last_fullsized":"2024-09-21T09:39:39.583250+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:43.843355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430229,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298500,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446652,"statfs":{"total":95995035648,"available":95689371648,"internally_reserved":0,"allocated":1171456,"data_stored":833637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.55400000000000005}]},{"osd":1,"last update":"Sat Sep 21 09:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.433},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.52900000000000003}]},{"osd":2,"last update":"Sat Sep 21 09:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.54300000000000004}]},{"osd":3,"last update":"Sat Sep 21 09:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.49299999999999999}]},{"osd":4,"last update":"Sat Sep 21 09:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.61699999999999999}]},{"osd":5,"last update":"Sat Sep 21 09:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.46800000000000003}]},{"osd":6,"last update":"Sat Sep 21 09:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.56499999999999995}]}]},{"osd":6,"up_from":42,"seq":180388626457,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298504,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446648,"statfs":{"total":95995035648,"available":95689367552,"internally_reserved":0,"allocated":1175552,"data_stored":833656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.56799999999999995}]},{"osd":1,"last update":"Sat Sep 21 09:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.63200000000000001}]},{"osd":2,"last update":"Sat Sep 21 09:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.65300000000000002}]},{"osd":3,"last update":"Sat Sep 21 09:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.68100000000000005}]},{"osd":4,"last update":"Sat Sep 21 09:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.38300000000000001}]},{"osd":5,"last update":"Sat Sep 21 09:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.40100000000000002}]},{"osd":7,"last update":"Sat Sep 21 09:40:02 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.501}]}]},{"osd":5,"up_from":36,"seq":154618822686,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297916,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447236,"statfs":{"total":95995035648,"available":95689969664,"internally_reserved":0,"allocated":573440,"data_stored":242658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:40:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.51800000000000002,"15min":0.52500000000000002},"min":{"1min":0.376,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.68300000000000005,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.435},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.50600000000000001,"15min":0.504},"min":{"1min":0.34899999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.41599999999999998}]},{"osd":1,"last update":"Sat Sep 21 09:40:18 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.34300000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.63900000000000001,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.36399999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.45800000000000002}]},{"osd":2,"last update":"Sat Sep 21 09:40:18 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52800000000000002,"15min":0.53200000000000003},"min":{"1min":0.35199999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.66500000000000004,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52700000000000002,"15min":0.52300000000000002},"min":{"1min":0.33900000000000002,"5min":0.312,"15min":0.312},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.502}]},{"osd":3,"last update":"Sat Sep 21 09:40:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54400000000000004,"15min":0.54500000000000004},"min":{"1min":0.38300000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.39400000000000002}]},{"osd":4,"last update":"Sat Sep 21 09:40:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.38600000000000001,"15min":0.375},"min":{"1min":0.28999999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.40400000000000003,"15min":0.39600000000000002},"min":{"1min":0.28799999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.60499999999999998}]},{"osd":6,"last update":"Sat Sep 21 09:39:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.54900000000000004}]},{"osd":7,"last update":"Sat Sep 21 09:40:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.627}]}]},{"osd":4,"up_from":30,"seq":128849018915,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297920,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447232,"statfs":{"total":95995035648,"available":95689965568,"internally_reserved":0,"allocated":577536,"data_stored":243269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:39:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.53900000000000003,"15min":0.54700000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.61199999999999999,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.55100000000000005,"15min":0.55700000000000005},"min":{"1min":0.39300000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.68000000000000005,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.623}]},{"osd":1,"last update":"Sat Sep 21 09:39:57 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.51900000000000002,"15min":0.52900000000000003},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.64100000000000001,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.495},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.499,"15min":0.503},"min":{"1min":0.35199999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.64900000000000002,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.60199999999999998}]},{"osd":2,"last update":"Sat Sep 21 09:39:57 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.52800000000000002,"15min":0.53700000000000003},"min":{"1min":0.318,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.625,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.505},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.53400000000000003,"15min":0.54200000000000004},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.58999999999999997}]},{"osd":3,"last update":"Sat Sep 21 09:39:57 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.54000000000000004,"15min":0.54700000000000004},"min":{"1min":0.38300000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73999999999999999,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.54000000000000004,"15min":0.54700000000000004},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.57399999999999995}]},{"osd":5,"last update":"Sat Sep 21 09:40:18 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.38800000000000001,"15min":0.38200000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.313},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.43099999999999999,"15min":0.42299999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.47699999999999998}]},{"osd":6,"last update":"Sat Sep 21 09:39:42 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.42099999999999999}]},{"osd":7,"last update":"Sat Sep 21 09:40:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.46100000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182440,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297920,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447232,"statfs":{"total":95995035648,"available":95689965568,"internally_reserved":0,"allocated":577536,"data_stored":243269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.38800000000000001,"15min":0.39200000000000002},"min":{"1min":0.23699999999999999,"5min":0.214,"15min":0.214},"max":{"1min":0.53300000000000003,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.377},{"interface":"front","average":{"1min":0.379,"5min":0.40699999999999997,"15min":0.41199999999999998},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.47899999999999998,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.32300000000000001}]},{"osd":1,"last update":"Sat Sep 21 09:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.39800000000000002,"15min":0.40100000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.54700000000000004,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.44,"15min":0.443},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.44}]},{"osd":2,"last update":"Sat Sep 21 09:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.46500000000000002,"15min":0.46899999999999997},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.436,"15min":0.44},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.40000000000000002}]},{"osd":4,"last update":"Sat Sep 21 09:39:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.55600000000000005,"15min":0.55500000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.60399999999999998}]},{"osd":5,"last update":"Sat Sep 21 09:40:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.58399999999999996,"15min":0.58999999999999997},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70499999999999996,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58299999999999996,"15min":0.58899999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.71799999999999997,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.56999999999999995}]},{"osd":6,"last update":"Sat Sep 21 09:39:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.48199999999999998}]},{"osd":7,"last update":"Sat Sep 21 09:40:05 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.55900000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297920,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447232,"statfs":{"total":95995035648,"available":95689965568,"internally_reserved":0,"allocated":577536,"data_stored":243269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34399999999999997,"15min":0.34100000000000003},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.48799999999999999,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.36399999999999999,"15min":0.36299999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.44700000000000001,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.46400000000000002}]},{"osd":1,"last update":"Sat Sep 21 09:40:17 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.35899999999999999,"15min":0.36099999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.38400000000000001,"15min":0.38500000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.495,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.40200000000000002}]},{"osd":3,"last update":"Sat Sep 21 09:39:33 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.30199999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.56599999999999995,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.41999999999999998,"15min":0.42599999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.5}]},{"osd":4,"last update":"Sat Sep 21 09:39:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55900000000000005,"15min":0.56299999999999994},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.78900000000000003,"5min":0.878,"15min":0.878},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.56100000000000005,"15min":0.56699999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68000000000000005,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.76700000000000002}]},{"osd":5,"last update":"Sat Sep 21 09:40:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58299999999999996,"15min":0.58699999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76100000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.74099999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.70599999999999996}]},{"osd":6,"last update":"Sat Sep 21 09:39:45 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.67100000000000004}]},{"osd":7,"last update":"Sat Sep 21 09:40:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.69799999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297920,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447232,"statfs":{"total":95995035648,"available":95689965568,"internally_reserved":0,"allocated":577536,"data_stored":242973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:39:45 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.32800000000000001,"15min":0.312},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.53700000000000003,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.33100000000000002,"15min":0.30499999999999999},"min":{"1min":0.255,"5min":0.184,"15min":0.184},"max":{"1min":0.51000000000000001,"5min":0.627,"15min":0.627},"last":0.32100000000000001}]},{"osd":2,"last update":"Sat Sep 21 09:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.32600000000000001},"min":{"1min":0.22800000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.497,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.378,"5min":0.37,"15min":0.35599999999999998},"min":{"1min":0.28599999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.46100000000000002,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.41499999999999998}]},{"osd":3,"last update":"Sat Sep 21 09:39:36 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.42499999999999999,"15min":0.42999999999999999},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.59299999999999997,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.436,"15min":0.439},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.55700000000000005,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.39800000000000002}]},{"osd":4,"last update":"Sat Sep 21 09:39:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.41799999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.69599999999999995,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.437},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56899999999999995,"15min":0.57199999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.54900000000000004}]},{"osd":5,"last update":"Sat Sep 21 09:40:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54500000000000004,"15min":0.54400000000000004},"min":{"1min":0.39300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.39400000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.75800000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.46999999999999997}]},{"osd":6,"last update":"Sat Sep 21 09:39:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.56200000000000006}]},{"osd":7,"last update":"Sat Sep 21 09:40:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.48099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298500,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446652,"statfs":{"total":95995035648,"available":95689371648,"internally_reserved":0,"allocated":1171456,"data_stored":833637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 09:39:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.33100000000000002,"15min":0.308},"min":{"1min":0.26400000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.51300000000000001,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.33300000000000002,"15min":0.317},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.50900000000000001,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.30099999999999999}]},{"osd":2,"last update":"Sat Sep 21 09:40:13 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.37,"15min":0.372},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.47599999999999998,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.246},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.38700000000000001,"15min":0.38500000000000001},"min":{"1min":0.27700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.58399999999999996,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.45800000000000002}]},{"osd":3,"last update":"Sat Sep 21 09:39:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.42899999999999999,"15min":0.43099999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.60099999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.497},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.40999999999999998,"15min":0.41099999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.56399999999999995,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.33700000000000002}]},{"osd":4,"last update":"Sat Sep 21 09:39:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.53000000000000003,"15min":0.53200000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.71499999999999997,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.502,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68400000000000005,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.47099999999999997}]},{"osd":5,"last update":"Sat Sep 21 09:40:21 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.54800000000000004,"15min":0.55300000000000005},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.56100000000000005,"15min":0.56699999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.49199999999999999}]},{"osd":6,"last update":"Sat Sep 21 09:39:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.68200000000000005}]},{"osd":7,"last update":"Sat Sep 21 09:40:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.50700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T09:40:41.185 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T09:40:41.185 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T09:40:41.185 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T09:40:41.186 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph health --format=json 2024-09-21T09:40:41.774 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:40:41.858 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:41 smithi107 systemd[1]: Stopping Ceph alertmanager.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:40:42.150 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:42 smithi107 ceph-mon[29382]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:42.150 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:42 smithi107 ceph-mon[41185]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:42.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:42 smithi129 ceph-mon[32729]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:42.608 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:42 smithi107 podman[77269]: 2024-09-21 09:40:42.257808907 +0000 UTC m=+0.544815558 container died f3a3b9a8a7507d37fdd7b4da046b36df854187b27975ad77f0cbf9cefa7fcab1 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=focused_zhukovsky, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True) 2024-09-21T09:40:43.603 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:40:43.603 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T09:40:43.603 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:43 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[71659]: ts=2024-09-21T09:40:43.536Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T09:40:43.603 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:43 smithi107 podman[77377]: 2024-09-21 09:40:43.557248972 +0000 UTC m=+1.054610336 container died fa175a563d3cb8fe5e4b48ff9b711fc564dfa68ad84ad361ae271ebb1e8739ab (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:40:44.329 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:44 smithi107 ceph-mon[29382]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:44.329 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:44 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/4167115085' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:40:44.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:44 smithi107 ceph-mon[41185]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:44.330 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:44 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/4167115085' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:40:44.330 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:44 smithi107 podman[77377]: 2024-09-21 09:40:44.097522036 +0000 UTC m=+1.594883401 container cleanup fa175a563d3cb8fe5e4b48ff9b711fc564dfa68ad84ad361ae271ebb1e8739ab (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:40:44.330 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:44 smithi107 bash[77377]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T09:40:44.330 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:44 smithi107 podman[77466]: 2024-09-21 09:40:44.235831035 +0000 UTC m=+0.677230432 container remove fa175a563d3cb8fe5e4b48ff9b711fc564dfa68ad84ad361ae271ebb1e8739ab (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:40:44.330 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:44 smithi107 podman[77466]: 2024-09-21 09:40:44.328277772 +0000 UTC m=+0.769677169 volume remove 221e7c012bf87428c3a85780f9fb74cbcffb7672777df3647f9b6a052f693752 2024-09-21T09:40:44.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:44 smithi129 ceph-mon[32729]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:44.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:44 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/4167115085' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:40:44.756 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T09:40:44.757 INFO:tasks.cephadm:Setup complete, yielding 2024-09-21T09:40:44.757 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-21T09:40:44.769 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi107.front.sepia.ceph.com 2024-09-21T09:40:44.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- bash -c 'ceph fs volume create foo' 2024-09-21T09:40:45.042 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:40:45.108 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:44 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T09:40:45.108 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:44 smithi107 systemd[1]: Stopped Ceph alertmanager.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:40:45.109 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:44 smithi107 systemd[1]: Starting Ceph alertmanager.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:40:45.505 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:45 smithi107 podman[77642]: 2024-09-21 09:40:45.050496971 +0000 UTC m=+0.018630157 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-21T09:40:45.505 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:45 smithi107 podman[77642]: 2024-09-21 09:40:45.184366234 +0000 UTC m=+0.152499450 volume create af5e4e3d1e3802948cad151277808d8488b468fb2652eaaeb3e027274a836e32 2024-09-21T09:40:45.506 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:45 smithi107 podman[77642]: 2024-09-21 09:40:45.293142776 +0000 UTC m=+0.261275992 container create 432a2da9f5f4dcda926c9f7b58b22c58f96f56ffac91892a9e35757173f0e72c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:40:45.856 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:45 smithi107 podman[77642]: 2024-09-21 09:40:45.721873179 +0000 UTC m=+0.690006358 container init 432a2da9f5f4dcda926c9f7b58b22c58f96f56ffac91892a9e35757173f0e72c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:40:45.856 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:45 smithi107 podman[77642]: 2024-09-21 09:40:45.726753858 +0000 UTC m=+0.694887028 container start 432a2da9f5f4dcda926c9f7b58b22c58f96f56ffac91892a9e35757173f0e72c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:40:45.857 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[77670]: ts=2024-09-21T09:40:45.738Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-21T09:40:45.857 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[77670]: ts=2024-09-21T09:40:45.738Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-21T09:40:45.857 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[77670]: ts=2024-09-21T09:40:45.740Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.107 port=9094 2024-09-21T09:40:45.857 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[77670]: ts=2024-09-21T09:40:45.741Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-21T09:40:45.857 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[77670]: ts=2024-09-21T09:40:45.764Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T09:40:45.857 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[77670]: ts=2024-09-21T09:40:45.764Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T09:40:45.857 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[77670]: ts=2024-09-21T09:40:45.767Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-21T09:40:45.857 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[77670]: ts=2024-09-21T09:40:45.767Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-21T09:40:45.857 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:45 smithi107 bash[77642]: 432a2da9f5f4dcda926c9f7b58b22c58f96f56ffac91892a9e35757173f0e72c 2024-09-21T09:40:46.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:46 smithi107 ceph-mon[29382]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:46.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:46 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:46.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:46 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:46.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:46 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:40:46.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:46 smithi107 ceph-mon[41185]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:46.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:46 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:46.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:46 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:46.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:46 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:40:46.110 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:45 smithi107 systemd[1]: Started Ceph alertmanager.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:40:46.369 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:46 smithi129 ceph-mon[32729]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:46.369 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:46 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:46.370 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:46 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:46.370 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:46 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:40:47.201 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:47 smithi107 ceph-mon[29382]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:40:47.201 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:47 smithi107 ceph-mon[29382]: Reconfiguring daemon prometheus.a on smithi129 2024-09-21T09:40:47.201 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:47 smithi107 ceph-mon[41185]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:40:47.201 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:47 smithi107 ceph-mon[41185]: Reconfiguring daemon prometheus.a on smithi129 2024-09-21T09:40:47.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:47 smithi129 ceph-mon[32729]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:40:47.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:47 smithi129 ceph-mon[32729]: Reconfiguring daemon prometheus.a on smithi129 2024-09-21T09:40:47.419 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:40:47 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:40:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:40:48.069 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:47 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[77670]: ts=2024-09-21T09:40:47.742Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000968665s 2024-09-21T09:40:48.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:48 smithi107 ceph-mon[29382]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:48.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:48 smithi107 ceph-mon[29382]: from='client.24533 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:40:48.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:48 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-21T09:40:48.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:48 smithi107 ceph-mon[41185]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:48.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:48 smithi107 ceph-mon[41185]: from='client.24533 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:40:48.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:48 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-21T09:40:48.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:48 smithi129 ceph-mon[32729]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:48.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:48 smithi129 ceph-mon[32729]: from='client.24533 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:40:48.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:48 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-21T09:40:49.083 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:48 smithi129 systemd[1]: Stopping Ceph prometheus.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:40:49.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:49 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-21T09:40:49.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:49 smithi107 ceph-mon[29382]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T09:40:49.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:49 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-21T09:40:49.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:49 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a[29360]: 2024-09-21T09:40:49.090+0000 7f31b796b640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T09:40:49.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:49 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-21T09:40:49.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:49 smithi107 ceph-mon[41185]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T09:40:49.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:49 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-21T09:40:49.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:49 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-21T09:40:49.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:49 smithi129 ceph-mon[32729]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T09:40:49.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:49 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-21T09:40:49.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[61890]: ts=2024-09-21T09:40:49.280Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T09:40:49.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[61890]: ts=2024-09-21T09:40:49.281Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-21T09:40:49.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[61890]: ts=2024-09-21T09:40:49.281Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-21T09:40:49.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[61890]: ts=2024-09-21T09:40:49.281Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T09:40:49.420 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[61890]: ts=2024-09-21T09:40:49.281Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-21T09:40:49.420 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[61890]: ts=2024-09-21T09:40:49.281Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-21T09:40:49.420 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[61890]: ts=2024-09-21T09:40:49.281Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T09:40:49.420 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[61890]: ts=2024-09-21T09:40:49.281Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-21T09:40:49.420 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[61890]: ts=2024-09-21T09:40:49.281Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-21T09:40:49.420 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[61890]: ts=2024-09-21T09:40:49.281Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T09:40:49.420 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[61890]: ts=2024-09-21T09:40:49.281Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-21T09:40:49.420 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[61890]: ts=2024-09-21T09:40:49.281Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-21T09:40:49.420 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:49 smithi129 podman[65259]: 2024-09-21 09:40:49.291047689 +0000 UTC m=+0.135075944 container died 183f60fc314a1b323fd22573c03f93458ae3cc0cc8a4b48e79dfe2cfb6c5cc55 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:40:49.814 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:49 smithi129 podman[65259]: 2024-09-21 09:40:49.620379656 +0000 UTC m=+0.464407900 container cleanup 183f60fc314a1b323fd22573c03f93458ae3cc0cc8a4b48e79dfe2cfb6c5cc55 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:40:49.814 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:49 smithi129 bash[65259]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T09:40:49.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-09-21T09:40:50.083 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:49 smithi129 podman[65276]: 2024-09-21 09:40:49.813311566 +0000 UTC m=+0.524067645 container remove 183f60fc314a1b323fd22573c03f93458ae3cc0cc8a4b48e79dfe2cfb6c5cc55 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:40:50.083 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T09:40:50.083 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 systemd[1]: Stopped Ceph prometheus.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:40:50.083 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 1.103s CPU time. 2024-09-21T09:40:50.084 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 systemd[1]: Starting Ceph prometheus.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:40:50.118 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:40:50.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[29382]: pgmap v50: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:50.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[29382]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:40:50.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-21T09:40:50.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[29382]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T09:40:50.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-21T09:40:50.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[29382]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T09:40:50.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[29382]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T09:40:50.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[29382]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T09:40:50.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-21T09:40:50.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[29382]: fsmap foo:0 2024-09-21T09:40:50.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[29382]: Saving service mds.foo spec with placement count:2 2024-09-21T09:40:50.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:50.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[41185]: pgmap v50: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:50.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[41185]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:40:50.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-21T09:40:50.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[41185]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T09:40:50.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-21T09:40:50.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[41185]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T09:40:50.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[41185]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T09:40:50.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[41185]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T09:40:50.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-21T09:40:50.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[41185]: fsmap foo:0 2024-09-21T09:40:50.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[41185]: Saving service mds.foo spec with placement count:2 2024-09-21T09:40:50.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:50.418 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 podman[65375]: 2024-09-21 09:40:50.13373379 +0000 UTC m=+0.016533739 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-21T09:40:50.418 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 podman[65375]: 2024-09-21 09:40:50.323073067 +0000 UTC m=+0.205873054 container create aa11510fd42834986b97cb1bc5145db901cf509160d2adf5463522e50e000a8f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:40:50.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-mon[32729]: pgmap v50: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:50.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-mon[32729]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:40:50.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-21T09:40:50.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-mon[32729]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T09:40:50.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-21T09:40:50.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-mon[32729]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T09:40:50.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-mon[32729]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T09:40:50.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-mon[32729]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T09:40:50.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-21T09:40:50.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-mon[32729]: fsmap foo:0 2024-09-21T09:40:50.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-mon[32729]: Saving service mds.foo spec with placement count:2 2024-09-21T09:40:50.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:50.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 podman[65375]: 2024-09-21 09:40:50.550080646 +0000 UTC m=+0.432880596 container init aa11510fd42834986b97cb1bc5145db901cf509160d2adf5463522e50e000a8f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:40:50.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 podman[65375]: 2024-09-21 09:40:50.555404134 +0000 UTC m=+0.438204077 container start aa11510fd42834986b97cb1bc5145db901cf509160d2adf5463522e50e000a8f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:40:50.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.578Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-21T09:40:50.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.578Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-21T09:40:50.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.578Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi129 (none))" 2024-09-21T09:40:50.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.578Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T09:40:50.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.578Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T09:40:50.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.580Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T09:40:50.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.580Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-21T09:40:50.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.581Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T09:40:50.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.581Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T09:40:50.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.584Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T09:40:50.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.584Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.884µs 2024-09-21T09:40:50.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.584Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T09:40:50.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.584Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-21T09:40:50.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.584Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-21T09:40:50.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.584Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=30.05µs wal_replay_duration=564.071µs wbl_replay_duration=107ns total_replay_duration=612.781µs 2024-09-21T09:40:50.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.585Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T09:40:50.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.585Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-21T09:40:50.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.585Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T09:40:50.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.600Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.183191ms db_storage=706ns remote_storage=979ns web_handler=694ns query_engine=574ns scrape=372.428µs scrape_sd=26.645µs notify=25.742µs notify_sd=13.883µs rules=14.512506ms tracing=5.978µs 2024-09-21T09:40:50.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.600Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-21T09:40:50.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:40:50.600Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T09:40:50.921 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 bash[65375]: aa11510fd42834986b97cb1bc5145db901cf509160d2adf5463522e50e000a8f 2024-09-21T09:40:50.921 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:40:50 smithi129 systemd[1]: Started Ceph prometheus.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:40:50.959 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:40:50] ENGINE Bus STOPPING 2024-09-21T09:40:50.959 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:40:50] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:40:50.959 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:40:50] ENGINE Bus STOPPED 2024-09-21T09:40:50.959 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:40:50] ENGINE Bus STARTING 2024-09-21T09:40:50.959 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:40:50] ENGINE Serving on http://:::9283 2024-09-21T09:40:50.959 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:40:50] ENGINE Bus STARTED 2024-09-21T09:40:51.303 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:40:50 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:40:50] ENGINE Bus STOPPING 2024-09-21T09:40:51.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[29382]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T09:40:51.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:51.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:51.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:40:51.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi107.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:40:51.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:51.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:40:51.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:40:51.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:51.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:40:51.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:40:51.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:51.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:40:51.304 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[41185]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T09:40:51.304 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:51.304 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:51.304 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:40:51.304 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi107.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:40:51.304 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:51.305 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:40:51.305 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:40:51.305 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:51.305 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:40:51.305 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:40:51.305 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:51.305 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:40:51.362 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:51 smithi129 ceph-mon[32729]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T09:40:51.362 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:51 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:51.362 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:51 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:51.362 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:51 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:40:51.362 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:51 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi107.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:40:51.362 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:51 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:51.362 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:51 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:40:51.362 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:51 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:40:51.362 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:51 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:51.362 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:51 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:40:51.362 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:51 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:40:51.363 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:51 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:51.363 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:51 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:40:51.567 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:40:51] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:40:51.567 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:40:51] ENGINE Bus STOPPED 2024-09-21T09:40:51.567 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:40:51] ENGINE Bus STARTING 2024-09-21T09:40:51.567 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:40:51] ENGINE Serving on http://:::9283 2024-09-21T09:40:51.567 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:40:51] ENGINE Bus STARTED 2024-09-21T09:40:51.858 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:40:51] ENGINE Bus STOPPING 2024-09-21T09:40:51.858 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:40:51] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:40:51.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:40:51] ENGINE Bus STOPPED 2024-09-21T09:40:51.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:40:51] ENGINE Bus STARTING 2024-09-21T09:40:51.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:40:51] ENGINE Serving on http://:::9283 2024-09-21T09:40:51.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:40:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:40:51] ENGINE Bus STARTED 2024-09-21T09:40:52.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:52 smithi107 ceph-mon[29382]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:40:52.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:52 smithi107 ceph-mon[29382]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi107.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:40:52.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:52 smithi107 ceph-mon[29382]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:40:52.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:52 smithi107 ceph-mon[29382]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:40:52.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:52 smithi107 ceph-mon[29382]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:40:52.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:52 smithi107 ceph-mon[29382]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:40:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:52 smithi107 ceph-mon[29382]: pgmap v54: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:52 smithi107 ceph-mon[29382]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T09:40:52.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:52 smithi107 ceph-mon[29382]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:40:52.292 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:52 smithi107 ceph-mon[41185]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:40:52.292 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:52 smithi107 ceph-mon[41185]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi107.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:40:52.292 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:52 smithi107 ceph-mon[41185]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:40:52.292 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:52 smithi107 ceph-mon[41185]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:40:52.292 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:52 smithi107 ceph-mon[41185]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:40:52.292 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:52 smithi107 ceph-mon[41185]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:40:52.292 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:52 smithi107 ceph-mon[41185]: pgmap v54: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:52.292 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:52 smithi107 ceph-mon[41185]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T09:40:52.292 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:52 smithi107 ceph-mon[41185]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:40:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:52 smithi129 ceph-mon[32729]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:40:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:52 smithi129 ceph-mon[32729]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi107.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:40:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:52 smithi129 ceph-mon[32729]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:40:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:52 smithi129 ceph-mon[32729]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi129.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:40:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:52 smithi129 ceph-mon[32729]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:40:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:52 smithi129 ceph-mon[32729]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi129.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:40:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:52 smithi129 ceph-mon[32729]: pgmap v54: 73 pgs: 64 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:52 smithi129 ceph-mon[32729]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T09:40:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:52 smithi129 ceph-mon[32729]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:40:52.797 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-09-21T09:40:53.284 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:40:54.320 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:54 smithi107 ceph-mon[29382]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:54.320 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:54 smithi107 ceph-mon[41185]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:54.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:54 smithi129 ceph-mon[32729]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:55.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:55 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:55.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:55 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:55.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:55 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:55.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:55 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:55 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:55 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:55.859 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:40:55 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[77670]: ts=2024-09-21T09:40:55.744Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003610569s 2024-09-21T09:40:56.594 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:56 smithi107 ceph-mon[29382]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:56.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:56 smithi107 ceph-mon[41185]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:56.617 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-09-21T09:40:56.627 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T09:40:56.627 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T09:40:56.627 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T09:40:56.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:56 smithi129 ceph-mon[32729]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:57.110 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:40:57.668 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:40:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:40:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:40:58.172 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:58 smithi107 ceph-mon[41185]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:58.172 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:58 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:58.172 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:58 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:58.172 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:58 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:40:58.172 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:58 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:40:58.172 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:58 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:58.172 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:58 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi107.sedied", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:40:58.172 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:58 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi107.sedied", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:40:58.172 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:58 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:40:58.172 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:40:58 smithi107 ceph-mon[41185]: Deploying daemon mds.foo.smithi107.sedied on smithi107 2024-09-21T09:40:58.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:58 smithi107 ceph-mon[29382]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:58.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:58 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:58.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:58 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:58.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:58 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:40:58.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:58 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:40:58.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:58 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:58.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:58 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi107.sedied", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:40:58.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:58 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi107.sedied", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:40:58.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:58 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:40:58.173 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:40:58 smithi107 ceph-mon[29382]: Deploying daemon mds.foo.smithi107.sedied on smithi107 2024-09-21T09:40:58.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:58 smithi129 ceph-mon[32729]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:40:58.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:58 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:58.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:58 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:58.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:58 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:40:58.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:58 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:40:58.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:58 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:40:58.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:58 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi107.sedied", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:40:58.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:58 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi107.sedied", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:40:58.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:58 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:40:58.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:40:58 smithi129 ceph-mon[32729]: Deploying daemon mds.foo.smithi107.sedied on smithi107 2024-09-21T09:40:59.608 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:40:59 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:40:59] "GET /metrics HTTP/1.1" 200 203824 "" "Prometheus/2.43.0" 2024-09-21T09:41:00.124 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:41:00.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:00 smithi107 ceph-mon[29382]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:00.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:00 smithi107 ceph-mon[41185]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:00.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:00 smithi129 ceph-mon[32729]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:00.746 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":20,"flags":0,"active_gid":24344,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":4055614859},{"type":"v1","addr":"172.21.15.107:6801","nonce":4055614859}]},"active_addr":"172.21.15.107:6801/4055614859","active_change":"2024-09-21T09:39:30.975631+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24352,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.107:8443/","prometheus":"http://172.21.15.107:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":51,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":3025710247}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":1526680749}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":51270168}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":2701254891}]}]} 2024-09-21T09:41:00.749 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T09:41:00.749 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T09:41:00.749 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:41:01.295 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:01.358 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:01 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/315983619' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:41:01.358 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:01 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:01.358 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:01 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:41:01.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:01 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/315983619' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:41:01.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:01 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:01.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:01 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:41:01.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:01 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/315983619' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:41:01.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:01 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:01.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:01 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:41:02.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[29382]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:02.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:02.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:02.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:02.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi129.zdzmqr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:41:02.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi129.zdzmqr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:41:02.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:41:02.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[29382]: Deploying daemon mds.foo.smithi129.zdzmqr on smithi129 2024-09-21T09:41:02.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[29382]: mds.? [v2:172.21.15.107:6834/2570718287,v1:172.21.15.107:6835/2570718287] up:boot 2024-09-21T09:41:02.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[29382]: daemon mds.foo.smithi107.sedied assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-21T09:41:02.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[29382]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T09:41:02.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[29382]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T09:41:02.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[29382]: Cluster is now healthy 2024-09-21T09:41:02.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[29382]: fsmap foo:0 1 up:standby 2024-09-21T09:41:02.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi107.sedied"}]: dispatch 2024-09-21T09:41:02.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[29382]: fsmap foo:1 {0=foo.smithi107.sedied=up:creating} 2024-09-21T09:41:02.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[29382]: daemon mds.foo.smithi107.sedied is now active in filesystem foo as rank 0 2024-09-21T09:41:02.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[41185]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:02.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:02.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:02.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:02.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi129.zdzmqr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:41:02.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi129.zdzmqr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:41:02.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:41:02.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[41185]: Deploying daemon mds.foo.smithi129.zdzmqr on smithi129 2024-09-21T09:41:02.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[41185]: mds.? [v2:172.21.15.107:6834/2570718287,v1:172.21.15.107:6835/2570718287] up:boot 2024-09-21T09:41:02.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[41185]: daemon mds.foo.smithi107.sedied assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-21T09:41:02.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[41185]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T09:41:02.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[41185]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T09:41:02.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[41185]: Cluster is now healthy 2024-09-21T09:41:02.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[41185]: fsmap foo:0 1 up:standby 2024-09-21T09:41:02.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi107.sedied"}]: dispatch 2024-09-21T09:41:02.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[41185]: fsmap foo:1 {0=foo.smithi107.sedied=up:creating} 2024-09-21T09:41:02.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:02 smithi107 ceph-mon[41185]: daemon mds.foo.smithi107.sedied is now active in filesystem foo as rank 0 2024-09-21T09:41:02.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:02 smithi129 ceph-mon[32729]: pgmap v60: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:02 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:02 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:02 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:02 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi129.zdzmqr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:41:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:02 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi129.zdzmqr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:41:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:02 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:41:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:02 smithi129 ceph-mon[32729]: Deploying daemon mds.foo.smithi129.zdzmqr on smithi129 2024-09-21T09:41:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:02 smithi129 ceph-mon[32729]: mds.? [v2:172.21.15.107:6834/2570718287,v1:172.21.15.107:6835/2570718287] up:boot 2024-09-21T09:41:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:02 smithi129 ceph-mon[32729]: daemon mds.foo.smithi107.sedied assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-21T09:41:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:02 smithi129 ceph-mon[32729]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T09:41:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:02 smithi129 ceph-mon[32729]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T09:41:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:02 smithi129 ceph-mon[32729]: Cluster is now healthy 2024-09-21T09:41:02.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:02 smithi129 ceph-mon[32729]: fsmap foo:0 1 up:standby 2024-09-21T09:41:02.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:02 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi107.sedied"}]: dispatch 2024-09-21T09:41:02.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:02 smithi129 ceph-mon[32729]: fsmap foo:1 {0=foo.smithi107.sedied=up:creating} 2024-09-21T09:41:02.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:02 smithi129 ceph-mon[32729]: daemon mds.foo.smithi107.sedied is now active in filesystem foo as rank 0 2024-09-21T09:41:03.225 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:41:03.225 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":63,"fsid":"86c49514-77fc-11ef-baf6-efdc52797490","created":"2024-09-21T09:34:03.635940+0000","modified":"2024-09-21T09:40:51.099411+0000","last_up_change":"2024-09-21T09:39:02.736647+0000","last_in_change":"2024-09-21T09:38:46.049422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:37:08.948207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:39:33.152023+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:40:47.823974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:40:48.313916+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}}}],"osds":[{"osd":0,"uuid":"635c1eb0-29b1-4fdd-bd47-881ef05e49bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":174831156},{"type":"v1","addr":"172.21.15.107:6803","nonce":174831156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":174831156},{"type":"v1","addr":"172.21.15.107:6805","nonce":174831156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":174831156},{"type":"v1","addr":"172.21.15.107:6809","nonce":174831156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":174831156},{"type":"v1","addr":"172.21.15.107:6807","nonce":174831156}]},"public_addr":"172.21.15.107:6803/174831156","cluster_addr":"172.21.15.107:6805/174831156","heartbeat_back_addr":"172.21.15.107:6809/174831156","heartbeat_front_addr":"172.21.15.107:6807/174831156","state":["exists","up"]},{"osd":1,"uuid":"a39b6a18-939d-4999-89b2-1126c1b2d1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1227448003},{"type":"v1","addr":"172.21.15.107:6811","nonce":1227448003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1227448003},{"type":"v1","addr":"172.21.15.107:6813","nonce":1227448003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1227448003},{"type":"v1","addr":"172.21.15.107:6817","nonce":1227448003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1227448003},{"type":"v1","addr":"172.21.15.107:6815","nonce":1227448003}]},"public_addr":"172.21.15.107:6811/1227448003","cluster_addr":"172.21.15.107:6813/1227448003","heartbeat_back_addr":"172.21.15.107:6817/1227448003","heartbeat_front_addr":"172.21.15.107:6815/1227448003","state":["exists","up"]},{"osd":2,"uuid":"b01c5f1b-77e4-462f-80d0-fd1d3afb786b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3613749103},{"type":"v1","addr":"172.21.15.107:6819","nonce":3613749103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3613749103},{"type":"v1","addr":"172.21.15.107:6821","nonce":3613749103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3613749103},{"type":"v1","addr":"172.21.15.107:6825","nonce":3613749103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3613749103},{"type":"v1","addr":"172.21.15.107:6823","nonce":3613749103}]},"public_addr":"172.21.15.107:6819/3613749103","cluster_addr":"172.21.15.107:6821/3613749103","heartbeat_back_addr":"172.21.15.107:6825/3613749103","heartbeat_front_addr":"172.21.15.107:6823/3613749103","state":["exists","up"]},{"osd":3,"uuid":"5135d269-b6c4-4a9d-bcf9-ffe19bc5c9ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3206618601},{"type":"v1","addr":"172.21.15.107:6827","nonce":3206618601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":3206618601},{"type":"v1","addr":"172.21.15.107:6829","nonce":3206618601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3206618601},{"type":"v1","addr":"172.21.15.107:6833","nonce":3206618601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3206618601},{"type":"v1","addr":"172.21.15.107:6831","nonce":3206618601}]},"public_addr":"172.21.15.107:6827/3206618601","cluster_addr":"172.21.15.107:6829/3206618601","heartbeat_back_addr":"172.21.15.107:6833/3206618601","heartbeat_front_addr":"172.21.15.107:6831/3206618601","state":["exists","up"]},{"osd":4,"uuid":"23242972-e341-4773-809f-299aabeacd56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6801","nonce":4101249335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6803","nonce":4101249335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6807","nonce":4101249335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6805","nonce":4101249335}]},"public_addr":"172.21.15.129:6801/4101249335","cluster_addr":"172.21.15.129:6803/4101249335","heartbeat_back_addr":"172.21.15.129:6807/4101249335","heartbeat_front_addr":"172.21.15.129:6805/4101249335","state":["exists","up"]},{"osd":5,"uuid":"f9da25cd-1680-4d97-a9d1-98dded10766f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6809","nonce":2733195233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6811","nonce":2733195233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6815","nonce":2733195233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6813","nonce":2733195233}]},"public_addr":"172.21.15.129:6809/2733195233","cluster_addr":"172.21.15.129:6811/2733195233","heartbeat_back_addr":"172.21.15.129:6815/2733195233","heartbeat_front_addr":"172.21.15.129:6813/2733195233","state":["exists","up"]},{"osd":6,"uuid":"d6d2bd99-0178-409c-a548-cbc30b8b76fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6817","nonce":3400496608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6819","nonce":3400496608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6823","nonce":3400496608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6821","nonce":3400496608}]},"public_addr":"172.21.15.129:6817/3400496608","cluster_addr":"172.21.15.129:6819/3400496608","heartbeat_back_addr":"172.21.15.129:6823/3400496608","heartbeat_front_addr":"172.21.15.129:6821/3400496608","state":["exists","up"]},{"osd":7,"uuid":"2c0a9723-0328-4134-b5d4-cf307839018a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6825","nonce":101668996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6827","nonce":101668996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6831","nonce":101668996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6829","nonce":101668996}]},"public_addr":"172.21.15.129:6825/101668996","cluster_addr":"172.21.15.129:6827/101668996","heartbeat_back_addr":"172.21.15.129:6831/101668996","heartbeat_front_addr":"172.21.15.129:6829/101668996","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:36:18.300841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:36:42.334925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:06.141467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:30.344630+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:51.661657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:38:14.654068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:38:38.089961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:39:01.119878+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/218684017":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/637981220":"2024-09-22T09:34:34.697433+0000","172.21.15.107:6800/1686228827":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/2156530655":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/782384657":"2024-09-22T09:34:56.643317+0000","172.21.15.107:6801/1686228827":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/2492859118":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/1942117255":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/837896416":"2024-09-22T09:34:34.697433+0000","172.21.15.107:6800/2716631102":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/4103644615":"2024-09-22T09:39:30.975484+0000","172.21.15.107:6800/4208383830":"2024-09-22T09:34:56.643317+0000","172.21.15.107:6801/4208383830":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/3424783814":"2024-09-22T09:34:56.643317+0000","172.21.15.107:6801/2716631102":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/2526044436":"2024-09-22T09:39:30.975484+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:41:03.853 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T09:41:03.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:41:04.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:04 smithi107 ceph-mon[29382]: pgmap v61: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:04.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:04 smithi107 ceph-mon[29382]: mds.? [v2:172.21.15.107:6834/2570718287,v1:172.21.15.107:6835/2570718287] up:active 2024-09-21T09:41:04.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:04 smithi107 ceph-mon[29382]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 2024-09-21T09:41:04.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:04 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3395778313' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:41:04.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:04 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:04.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:04 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:04.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:04 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:04.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:04 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:04.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:04 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:04.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:04 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:41:04.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:04 smithi107 ceph-mon[41185]: pgmap v61: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:04.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:04 smithi107 ceph-mon[41185]: mds.? [v2:172.21.15.107:6834/2570718287,v1:172.21.15.107:6835/2570718287] up:active 2024-09-21T09:41:04.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:04 smithi107 ceph-mon[41185]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 2024-09-21T09:41:04.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:04 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3395778313' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:41:04.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:04 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:04.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:04 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:04.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:04 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:04.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:04 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:04.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:04 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:04.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:04 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:41:04.112 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:04.398 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:04 smithi129 ceph-mon[32729]: pgmap v61: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:04.398 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:04 smithi129 ceph-mon[32729]: mds.? [v2:172.21.15.107:6834/2570718287,v1:172.21.15.107:6835/2570718287] up:active 2024-09-21T09:41:04.398 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:04 smithi129 ceph-mon[32729]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 2024-09-21T09:41:04.399 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:04 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3395778313' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:41:04.399 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:04 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:04.399 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:04 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:04.399 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:04 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:04.399 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:04 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:04.399 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:04 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:04.399 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:04 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:41:05.358 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:05 smithi107 ceph-mon[41185]: mds.? [v2:172.21.15.129:6832/194235284,v1:172.21.15.129:6833/194235284] up:boot 2024-09-21T09:41:05.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:05 smithi107 ceph-mon[41185]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:41:05.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:05 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi129.zdzmqr"}]: dispatch 2024-09-21T09:41:05.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:05 smithi107 ceph-mon[41185]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:41:05.362 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:05 smithi107 ceph-mon[29382]: mds.? [v2:172.21.15.129:6832/194235284,v1:172.21.15.129:6833/194235284] up:boot 2024-09-21T09:41:05.362 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:05 smithi107 ceph-mon[29382]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:41:05.362 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:05 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi129.zdzmqr"}]: dispatch 2024-09-21T09:41:05.362 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:05 smithi107 ceph-mon[29382]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:41:05.628 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:05 smithi129 ceph-mon[32729]: mds.? [v2:172.21.15.129:6832/194235284,v1:172.21.15.129:6833/194235284] up:boot 2024-09-21T09:41:05.628 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:05 smithi129 ceph-mon[32729]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:41:05.628 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:05 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi129.zdzmqr"}]: dispatch 2024-09-21T09:41:05.628 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:05 smithi129 ceph-mon[32729]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:41:06.041 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:41:06.041 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":63,"fsid":"86c49514-77fc-11ef-baf6-efdc52797490","created":"2024-09-21T09:34:03.635940+0000","modified":"2024-09-21T09:40:51.099411+0000","last_up_change":"2024-09-21T09:39:02.736647+0000","last_in_change":"2024-09-21T09:38:46.049422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:37:08.948207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:39:33.152023+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:40:47.823974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:40:48.313916+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}}}],"osds":[{"osd":0,"uuid":"635c1eb0-29b1-4fdd-bd47-881ef05e49bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":174831156},{"type":"v1","addr":"172.21.15.107:6803","nonce":174831156}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":174831156},{"type":"v1","addr":"172.21.15.107:6805","nonce":174831156}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":174831156},{"type":"v1","addr":"172.21.15.107:6809","nonce":174831156}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":174831156},{"type":"v1","addr":"172.21.15.107:6807","nonce":174831156}]},"public_addr":"172.21.15.107:6803/174831156","cluster_addr":"172.21.15.107:6805/174831156","heartbeat_back_addr":"172.21.15.107:6809/174831156","heartbeat_front_addr":"172.21.15.107:6807/174831156","state":["exists","up"]},{"osd":1,"uuid":"a39b6a18-939d-4999-89b2-1126c1b2d1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1227448003},{"type":"v1","addr":"172.21.15.107:6811","nonce":1227448003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1227448003},{"type":"v1","addr":"172.21.15.107:6813","nonce":1227448003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":1227448003},{"type":"v1","addr":"172.21.15.107:6817","nonce":1227448003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1227448003},{"type":"v1","addr":"172.21.15.107:6815","nonce":1227448003}]},"public_addr":"172.21.15.107:6811/1227448003","cluster_addr":"172.21.15.107:6813/1227448003","heartbeat_back_addr":"172.21.15.107:6817/1227448003","heartbeat_front_addr":"172.21.15.107:6815/1227448003","state":["exists","up"]},{"osd":2,"uuid":"b01c5f1b-77e4-462f-80d0-fd1d3afb786b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":3613749103},{"type":"v1","addr":"172.21.15.107:6819","nonce":3613749103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":3613749103},{"type":"v1","addr":"172.21.15.107:6821","nonce":3613749103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":3613749103},{"type":"v1","addr":"172.21.15.107:6825","nonce":3613749103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":3613749103},{"type":"v1","addr":"172.21.15.107:6823","nonce":3613749103}]},"public_addr":"172.21.15.107:6819/3613749103","cluster_addr":"172.21.15.107:6821/3613749103","heartbeat_back_addr":"172.21.15.107:6825/3613749103","heartbeat_front_addr":"172.21.15.107:6823/3613749103","state":["exists","up"]},{"osd":3,"uuid":"5135d269-b6c4-4a9d-bcf9-ffe19bc5c9ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":3206618601},{"type":"v1","addr":"172.21.15.107:6827","nonce":3206618601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":3206618601},{"type":"v1","addr":"172.21.15.107:6829","nonce":3206618601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":3206618601},{"type":"v1","addr":"172.21.15.107:6833","nonce":3206618601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":3206618601},{"type":"v1","addr":"172.21.15.107:6831","nonce":3206618601}]},"public_addr":"172.21.15.107:6827/3206618601","cluster_addr":"172.21.15.107:6829/3206618601","heartbeat_back_addr":"172.21.15.107:6833/3206618601","heartbeat_front_addr":"172.21.15.107:6831/3206618601","state":["exists","up"]},{"osd":4,"uuid":"23242972-e341-4773-809f-299aabeacd56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6801","nonce":4101249335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6803","nonce":4101249335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6807","nonce":4101249335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6805","nonce":4101249335}]},"public_addr":"172.21.15.129:6801/4101249335","cluster_addr":"172.21.15.129:6803/4101249335","heartbeat_back_addr":"172.21.15.129:6807/4101249335","heartbeat_front_addr":"172.21.15.129:6805/4101249335","state":["exists","up"]},{"osd":5,"uuid":"f9da25cd-1680-4d97-a9d1-98dded10766f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6809","nonce":2733195233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6811","nonce":2733195233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6815","nonce":2733195233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6813","nonce":2733195233}]},"public_addr":"172.21.15.129:6809/2733195233","cluster_addr":"172.21.15.129:6811/2733195233","heartbeat_back_addr":"172.21.15.129:6815/2733195233","heartbeat_front_addr":"172.21.15.129:6813/2733195233","state":["exists","up"]},{"osd":6,"uuid":"d6d2bd99-0178-409c-a548-cbc30b8b76fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6817","nonce":3400496608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6819","nonce":3400496608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6823","nonce":3400496608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6821","nonce":3400496608}]},"public_addr":"172.21.15.129:6817/3400496608","cluster_addr":"172.21.15.129:6819/3400496608","heartbeat_back_addr":"172.21.15.129:6823/3400496608","heartbeat_front_addr":"172.21.15.129:6821/3400496608","state":["exists","up"]},{"osd":7,"uuid":"2c0a9723-0328-4134-b5d4-cf307839018a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6825","nonce":101668996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6827","nonce":101668996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6831","nonce":101668996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6829","nonce":101668996}]},"public_addr":"172.21.15.129:6825/101668996","cluster_addr":"172.21.15.129:6827/101668996","heartbeat_back_addr":"172.21.15.129:6831/101668996","heartbeat_front_addr":"172.21.15.129:6829/101668996","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:36:18.300841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:36:42.334925+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:06.141467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:30.344630+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:51.661657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:38:14.654068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:38:38.089961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:39:01.119878+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/218684017":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/637981220":"2024-09-22T09:34:34.697433+0000","172.21.15.107:6800/1686228827":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/2156530655":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/782384657":"2024-09-22T09:34:56.643317+0000","172.21.15.107:6801/1686228827":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/2492859118":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/1942117255":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/837896416":"2024-09-22T09:34:34.697433+0000","172.21.15.107:6800/2716631102":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/4103644615":"2024-09-22T09:39:30.975484+0000","172.21.15.107:6800/4208383830":"2024-09-22T09:34:56.643317+0000","172.21.15.107:6801/4208383830":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/3424783814":"2024-09-22T09:34:56.643317+0000","172.21.15.107:6801/2716631102":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/2526044436":"2024-09-22T09:39:30.975484+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:41:06.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:06 smithi129 ceph-mon[32729]: pgmap v62: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:06.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:06 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1785561269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:41:06.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:06 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:06.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:06 smithi107 ceph-mon[41185]: pgmap v62: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:06.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:06 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1785561269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:41:06.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:06 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:06.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:06 smithi107 ceph-mon[29382]: pgmap v62: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:06.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:06 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1785561269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:41:06.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:06 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:07.098 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T09:41:07.098 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T09:41:07.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T09:41:07.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T09:41:07.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T09:41:07.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T09:41:07.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T09:41:07.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T09:41:07.548 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:41:07 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:41:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:41:07.566 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:07.681 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:07.682 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:07.683 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:07.684 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:07.685 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:07.688 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:07.693 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:08.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:08 smithi107 ceph-mon[29382]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:41:08.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:08 smithi107 ceph-mon[29382]: mds.? [v2:172.21.15.107:6834/2570718287,v1:172.21.15.107:6835/2570718287] up:active 2024-09-21T09:41:08.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:08 smithi107 ceph-mon[29382]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:41:08.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:08 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:08.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:08 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:08.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:08 smithi107 ceph-mon[41185]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:41:08.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:08 smithi107 ceph-mon[41185]: mds.? [v2:172.21.15.107:6834/2570718287,v1:172.21.15.107:6835/2570718287] up:active 2024-09-21T09:41:08.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:08 smithi107 ceph-mon[41185]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:41:08.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:08 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:08.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:08 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:08.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:08 smithi129 ceph-mon[32729]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:41:08.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:08 smithi129 ceph-mon[32729]: mds.? [v2:172.21.15.107:6834/2570718287,v1:172.21.15.107:6835/2570718287] up:active 2024-09-21T09:41:08.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:08 smithi129 ceph-mon[32729]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:41:08.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:08 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:08.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:08 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:09.334 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:41:09 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:41:09] "GET /metrics HTTP/1.1" 200 208816 "" "Prometheus/2.43.0" 2024-09-21T09:41:09.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:09 smithi107 ceph-mon[41185]: mds.? [v2:172.21.15.129:6832/194235284,v1:172.21.15.129:6833/194235284] up:standby 2024-09-21T09:41:09.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:09 smithi107 ceph-mon[41185]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:41:09.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:09 smithi107 ceph-mon[29382]: mds.? [v2:172.21.15.129:6832/194235284,v1:172.21.15.129:6833/194235284] up:standby 2024-09-21T09:41:09.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:09 smithi107 ceph-mon[29382]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:41:09.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:09 smithi129 ceph-mon[32729]: mds.? [v2:172.21.15.129:6832/194235284,v1:172.21.15.129:6833/194235284] up:standby 2024-09-21T09:41:09.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:09 smithi129 ceph-mon[32729]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:41:10.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:10 smithi107 ceph-mon[29382]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:41:10.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:10 smithi107 ceph-mon[41185]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:41:10.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:10 smithi129 ceph-mon[32729]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:41:12.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:12 smithi107 ceph-mon[29382]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:41:12.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:12 smithi107 ceph-mon[41185]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:41:12.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:12 smithi129 ceph-mon[32729]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:41:14.360 INFO:teuthology.orchestra.run.smithi107.stdout:77309411381 2024-09-21T09:41:14.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T09:41:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:14 smithi107 ceph-mon[29382]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:41:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:14 smithi107 ceph-mon[41185]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:41:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:14 smithi129 ceph-mon[32729]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:41:16.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:16 smithi107 ceph-mon[29382]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:41:16.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:16 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:16.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:16 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:41:16.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:16 smithi107 ceph-mon[41185]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:41:16.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:16 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:16.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:16 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:41:16.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:16 smithi129 ceph-mon[32729]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:41:16.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:16 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:16.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:16 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:41:17.668 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:41:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:41:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:41:18.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:18 smithi107 ceph-mon[29382]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:41:18.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:18 smithi107 ceph-mon[41185]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:41:18.663 INFO:teuthology.orchestra.run.smithi107.stdout:55834574906 2024-09-21T09:41:18.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T09:41:18.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:18 smithi129 ceph-mon[32729]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:41:19.608 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:41:19 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:41:19] "GET /metrics HTTP/1.1" 200 239787 "" "Prometheus/2.43.0" 2024-09-21T09:41:20.535 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:20 smithi107 ceph-mon[29382]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:20.535 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:20 smithi107 ceph-mon[41185]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:20.559 INFO:teuthology.orchestra.run.smithi107.stdout:206158430238 2024-09-21T09:41:20.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T09:41:20.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:20 smithi129 ceph-mon[32729]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:20.953 INFO:teuthology.orchestra.run.smithi107.stdout:154618822695 2024-09-21T09:41:20.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T09:41:21.453 INFO:teuthology.orchestra.run.smithi107.stdout:34359738431 2024-09-21T09:41:21.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T09:41:22.214 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:22.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:22 smithi107 ceph-mon[29382]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:22.478 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:22 smithi107 ceph-mon[41185]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:22.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:22 smithi129 ceph-mon[32729]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:23.311 INFO:teuthology.orchestra.run.smithi107.stdout:180388626466 2024-09-21T09:41:23.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T09:41:24.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:24 smithi107 ceph-mon[29382]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:24.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:24 smithi107 ceph-mon[41185]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:24.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:24 smithi129 ceph-mon[32729]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:24.995 INFO:teuthology.orchestra.run.smithi107.stdout:128849018925 2024-09-21T09:41:24.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T09:41:26.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:26 smithi107 ceph-mon[29382]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:26.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:26 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:26.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:26 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:26.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:26 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:41:26.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:26 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:41:26.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:26 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:26.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:26 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:41:26.281 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:26 smithi107 ceph-mon[41185]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:26.281 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:26 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:26.281 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:26 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:26.281 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:26 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:41:26.281 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:26 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:41:26.281 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:26 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:26.281 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:26 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:41:26.301 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:26.302 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:26.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:26 smithi129 ceph-mon[32729]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:26.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:26 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:26.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:26 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:26.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:26 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:41:26.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:26 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:41:26.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:26 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:26.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:26 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:41:26.756 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:26.756 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:27.656 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:41:27 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:41:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:41:28.105 INFO:teuthology.orchestra.run.smithi107.stdout:77309411384 2024-09-21T09:41:28.310 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:28 smithi129 ceph-mon[32729]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:28.358 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:28 smithi107 ceph-mon[41185]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:28.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:28 smithi107 ceph-mon[29382]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:28.766 INFO:teuthology.orchestra.run.smithi107.stdout:107374182450 2024-09-21T09:41:28.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T09:41:29.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:29 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1095581801' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:41:29.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:29 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:29.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:29 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:29.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:29 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1095581801' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:41:29.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:29 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:29.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:29 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:29.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:29 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1095581801' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:41:29.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:29 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:29.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:29 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:29.551 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:41:29 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:41:29] "GET /metrics HTTP/1.1" 200 239787 "" "Prometheus/2.43.0" 2024-09-21T09:41:29.552 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411381 got 77309411384 for osd.2 2024-09-21T09:41:29.552 DEBUG:teuthology.parallel:result is None 2024-09-21T09:41:30.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:30 smithi107 ceph-mon[29382]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:30 smithi107 ceph-mon[41185]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:30.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:30 smithi129 ceph-mon[32729]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:31.543 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:31.544 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:31.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-21T09:41:31.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-21T09:41:31.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-09-21T09:41:31.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:41:31.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-21T09:41:31.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-21T09:41:31.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-09-21T09:41:31.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:41:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-09-21T09:41:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-21T09:41:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-09-21T09:41:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:41:32.412 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:32.596 INFO:teuthology.orchestra.run.smithi107.stdout:206158430240 2024-09-21T09:41:32.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:32 smithi107 ceph-mon[29382]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:32.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:32 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-21T09:41:32.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:32 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-21T09:41:32.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:32 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-09-21T09:41:32.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:32 smithi107 ceph-mon[29382]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T09:41:32.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:32 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1573537948' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:41:32.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:32 smithi107 ceph-mon[41185]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:32.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:32 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-21T09:41:32.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:32 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-21T09:41:32.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:32 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-09-21T09:41:32.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:32 smithi107 ceph-mon[41185]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T09:41:32.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:32 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1573537948' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:41:32.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:32 smithi129 ceph-mon[32729]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:32 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-09-21T09:41:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:32 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-21T09:41:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:32 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-09-21T09:41:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:32 smithi129 ceph-mon[32729]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T09:41:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:32 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1573537948' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:41:33.490 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430238 got 206158430240 for osd.7 2024-09-21T09:41:33.491 DEBUG:teuthology.parallel:result is None 2024-09-21T09:41:33.723 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:33 smithi107 ceph-mon[29382]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T09:41:33.723 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:33 smithi107 ceph-mon[41185]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T09:41:33.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:33 smithi129 ceph-mon[32729]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T09:41:34.731 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:34 smithi107 ceph-mon[29382]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:34.731 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:34 smithi107 ceph-mon[41185]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:34.893 INFO:teuthology.orchestra.run.smithi107.stdout:55834574910 2024-09-21T09:41:34.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:34 smithi129 ceph-mon[32729]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:35.857 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:35 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/411596828' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:41:35.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:35 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/411596828' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:41:35.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:35 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/411596828' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:41:36.599 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574906 got 55834574910 for osd.1 2024-09-21T09:41:36.599 DEBUG:teuthology.parallel:result is None 2024-09-21T09:41:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:36 smithi107 ceph-mon[29382]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:36.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:36 smithi107 ceph-mon[41185]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:36.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:36 smithi129 ceph-mon[32729]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:37.419 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:41:37 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:41:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:41:37.776 INFO:teuthology.orchestra.run.smithi107.stdout:34359738436 2024-09-21T09:41:38.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:38 smithi129 ceph-mon[32729]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:38 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/220344893' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:41:38.927 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:38 smithi107 ceph-mon[29382]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:38.927 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:38 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/220344893' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:41:38.927 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:38 smithi107 ceph-mon[41185]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:38.927 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:38 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/220344893' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:41:38.970 INFO:teuthology.orchestra.run.smithi107.stdout:180388626470 2024-09-21T09:41:39.585 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:41:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:41:39] "GET /metrics HTTP/1.1" 200 239768 "" "Prometheus/2.43.0" 2024-09-21T09:41:39.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:39 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1481614181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:41:39.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:39 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1481614181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:41:39.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:39 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1481614181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:41:40.531 INFO:teuthology.orchestra.run.smithi107.stdout:128849018928 2024-09-21T09:41:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:40 smithi107 ceph-mon[29382]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:40 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3519714495' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:41:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:40 smithi107 ceph-mon[41185]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:40 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3519714495' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:41:40.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:40 smithi129 ceph-mon[32729]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:40.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:40 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3519714495' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:41:41.340 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626466 got 180388626470 for osd.6 2024-09-21T09:41:41.340 DEBUG:teuthology.parallel:result is None 2024-09-21T09:41:41.656 INFO:teuthology.orchestra.run.smithi107.stdout:154618822699 2024-09-21T09:41:41.886 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:41 smithi107 ceph-mon[29382]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:41.886 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:41 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1979917524' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:41:41.886 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:41 smithi107 ceph-mon[41185]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:41.886 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:41 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1979917524' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:41:41.909 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738431 got 34359738436 for osd.0 2024-09-21T09:41:41.909 DEBUG:teuthology.parallel:result is None 2024-09-21T09:41:41.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:41 smithi129 ceph-mon[32729]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:41.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:41 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1979917524' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:41:42.342 INFO:teuthology.orchestra.run.smithi107.stdout:107374182453 2024-09-21T09:41:42.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:42 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1485952032' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:41:43.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:42 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1485952032' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:41:43.047 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:42 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1485952032' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:41:43.072 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182450 got 107374182453 for osd.3 2024-09-21T09:41:43.072 DEBUG:teuthology.parallel:result is None 2024-09-21T09:41:43.826 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:43 smithi107 ceph-mon[41185]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:43.827 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:43 smithi107 ceph-mon[29382]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:43.854 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822695 got 154618822699 for osd.5 2024-09-21T09:41:43.854 DEBUG:teuthology.parallel:result is None 2024-09-21T09:41:44.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:43 smithi129 ceph-mon[32729]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:44.429 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018925 got 128849018928 for osd.4 2024-09-21T09:41:44.430 DEBUG:teuthology.parallel:result is None 2024-09-21T09:41:44.430 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T09:41:44.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T09:41:45.247 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:46.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:46 smithi107 ceph-mon[29382]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:46.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:46 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:41:46.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:46 smithi107 ceph-mon[41185]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:46.326 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:46 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:41:46.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:46 smithi129 ceph-mon[32729]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:46.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:46 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:41:47.561 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:41:47 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:41:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:41:47.768 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:41:47.769 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T09:41:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:48 smithi129 ceph-mon[32729]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:48 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:48 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:48 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:41:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:48 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:41:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:48 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:48.504 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:48 smithi107 ceph-mon[29382]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:48.504 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:48 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:48.504 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:48 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:48.504 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:48 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:41:48.504 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:48 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:41:48.504 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:48 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:48.504 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:48 smithi107 ceph-mon[41185]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:48.505 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:48 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:48.505 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:48 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:48.505 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:48 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:41:48.505 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:48 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:41:48.505 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:48 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:48.531 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-09-21T09:41:47.016059+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592725,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":160,"num_write_kb":1643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121,"ondisk_log_size":121,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385872,"kb_used_data":7020,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575344,"statfs":{"total":767960285184,"available":765517152256,"internally_reserved":0,"allocated":7188480,"data_stored":4126734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002909"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299480+0000","last_change":"2024-09-21T09:40:50.114134+0000","last_active":"2024-09-21T09:41:32.299480+0000","last_peered":"2024-09-21T09:41:32.299480+0000","last_clean":"2024-09-21T09:41:32.299480+0000","last_became_active":"2024-09-21T09:40:50.113655+0000","last_became_peered":"2024-09-21T09:40:50.113655+0000","last_unstale":"2024-09-21T09:41:32.299480+0000","last_undegraded":"2024-09-21T09:41:32.299480+0000","last_fullsized":"2024-09-21T09:41:32.299480+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:22:23.755499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299725+0000","last_change":"2024-09-21T09:40:49.104723+0000","last_active":"2024-09-21T09:41:32.299725+0000","last_peered":"2024-09-21T09:41:32.299725+0000","last_clean":"2024-09-21T09:41:32.299725+0000","last_became_active":"2024-09-21T09:40:49.104457+0000","last_became_peered":"2024-09-21T09:40:49.104457+0000","last_unstale":"2024-09-21T09:41:32.299725+0000","last_undegraded":"2024-09-21T09:41:32.299725+0000","last_fullsized":"2024-09-21T09:41:32.299725+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:16:54.926805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.636222+0000","last_change":"2024-09-21T09:40:49.397748+0000","last_active":"2024-09-21T09:41:31.636222+0000","last_peered":"2024-09-21T09:41:31.636222+0000","last_clean":"2024-09-21T09:41:31.636222+0000","last_became_active":"2024-09-21T09:40:49.397402+0000","last_became_peered":"2024-09-21T09:40:49.397402+0000","last_unstale":"2024-09-21T09:41:31.636222+0000","last_undegraded":"2024-09-21T09:41:31.636222+0000","last_fullsized":"2024-09-21T09:41:31.636222+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:10:55.764418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637455+0000","last_change":"2024-09-21T09:40:49.097777+0000","last_active":"2024-09-21T09:41:31.637455+0000","last_peered":"2024-09-21T09:41:31.637455+0000","last_clean":"2024-09-21T09:41:31.637455+0000","last_became_active":"2024-09-21T09:40:49.097404+0000","last_became_peered":"2024-09-21T09:40:49.097404+0000","last_unstale":"2024-09-21T09:41:31.637455+0000","last_undegraded":"2024-09-21T09:41:31.637455+0000","last_fullsized":"2024-09-21T09:41:31.637455+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:11:49.098768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637600+0000","last_change":"2024-09-21T09:40:50.109420+0000","last_active":"2024-09-21T09:41:31.637600+0000","last_peered":"2024-09-21T09:41:31.637600+0000","last_clean":"2024-09-21T09:41:31.637600+0000","last_became_active":"2024-09-21T09:40:50.109137+0000","last_became_peered":"2024-09-21T09:40:50.109137+0000","last_unstale":"2024-09-21T09:41:31.637600+0000","last_undegraded":"2024-09-21T09:41:31.637600+0000","last_fullsized":"2024-09-21T09:41:31.637600+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:16.565490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299629+0000","last_change":"2024-09-21T09:40:49.101911+0000","last_active":"2024-09-21T09:41:32.299629+0000","last_peered":"2024-09-21T09:41:32.299629+0000","last_clean":"2024-09-21T09:41:32.299629+0000","last_became_active":"2024-09-21T09:40:49.101629+0000","last_became_peered":"2024-09-21T09:40:49.101629+0000","last_unstale":"2024-09-21T09:41:32.299629+0000","last_undegraded":"2024-09-21T09:41:32.299629+0000","last_fullsized":"2024-09-21T09:41:32.299629+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:07:19.215551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299613+0000","last_change":"2024-09-21T09:40:50.113148+0000","last_active":"2024-09-21T09:41:32.299613+0000","last_peered":"2024-09-21T09:41:32.299613+0000","last_clean":"2024-09-21T09:41:32.299613+0000","last_became_active":"2024-09-21T09:40:50.112896+0000","last_became_peered":"2024-09-21T09:40:50.112896+0000","last_unstale":"2024-09-21T09:41:32.299613+0000","last_undegraded":"2024-09-21T09:41:32.299613+0000","last_fullsized":"2024-09-21T09:41:32.299613+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:53.011179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":22,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635726+0000","last_change":"2024-09-21T09:40:49.099918+0000","last_active":"2024-09-21T09:41:31.635726+0000","last_peered":"2024-09-21T09:41:31.635726+0000","last_clean":"2024-09-21T09:41:31.635726+0000","last_became_active":"2024-09-21T09:40:49.099642+0000","last_became_peered":"2024-09-21T09:40:49.099642+0000","last_unstale":"2024-09-21T09:41:31.635726+0000","last_undegraded":"2024-09-21T09:41:31.635726+0000","last_fullsized":"2024-09-21T09:41:31.635726+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:40:55.840709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.300025+0000","last_change":"2024-09-21T09:40:49.103815+0000","last_active":"2024-09-21T09:41:32.300025+0000","last_peered":"2024-09-21T09:41:32.300025+0000","last_clean":"2024-09-21T09:41:32.300025+0000","last_became_active":"2024-09-21T09:40:49.101820+0000","last_became_peered":"2024-09-21T09:40:49.101820+0000","last_unstale":"2024-09-21T09:41:32.300025+0000","last_undegraded":"2024-09-21T09:41:32.300025+0000","last_fullsized":"2024-09-21T09:41:32.300025+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:09:32.403362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635364+0000","last_change":"2024-09-21T09:40:49.397673+0000","last_active":"2024-09-21T09:41:31.635364+0000","last_peered":"2024-09-21T09:41:31.635364+0000","last_clean":"2024-09-21T09:41:31.635364+0000","last_became_active":"2024-09-21T09:40:49.397365+0000","last_became_peered":"2024-09-21T09:40:49.397365+0000","last_unstale":"2024-09-21T09:41:31.635364+0000","last_undegraded":"2024-09-21T09:41:31.635364+0000","last_fullsized":"2024-09-21T09:41:31.635364+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:48:55.191816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.300022+0000","last_change":"2024-09-21T09:40:49.101165+0000","last_active":"2024-09-21T09:41:32.300022+0000","last_peered":"2024-09-21T09:41:32.300022+0000","last_clean":"2024-09-21T09:41:32.300022+0000","last_became_active":"2024-09-21T09:40:49.100866+0000","last_became_peered":"2024-09-21T09:40:49.100866+0000","last_unstale":"2024-09-21T09:41:32.300022+0000","last_undegraded":"2024-09-21T09:41:32.300022+0000","last_fullsized":"2024-09-21T09:41:32.300022+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:00.626279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.633013+0000","last_change":"2024-09-21T09:40:49.358400+0000","last_active":"2024-09-21T09:41:31.633013+0000","last_peered":"2024-09-21T09:41:31.633013+0000","last_clean":"2024-09-21T09:41:31.633013+0000","last_became_active":"2024-09-21T09:40:49.358128+0000","last_became_peered":"2024-09-21T09:40:49.358128+0000","last_unstale":"2024-09-21T09:41:31.633013+0000","last_undegraded":"2024-09-21T09:41:31.633013+0000","last_fullsized":"2024-09-21T09:41:31.633013+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:40.278954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299930+0000","last_change":"2024-09-21T09:40:49.096812+0000","last_active":"2024-09-21T09:41:32.299930+0000","last_peered":"2024-09-21T09:41:32.299930+0000","last_clean":"2024-09-21T09:41:32.299930+0000","last_became_active":"2024-09-21T09:40:49.096549+0000","last_became_peered":"2024-09-21T09:40:49.096549+0000","last_unstale":"2024-09-21T09:41:32.299930+0000","last_undegraded":"2024-09-21T09:41:32.299930+0000","last_fullsized":"2024-09-21T09:41:32.299930+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:24.512887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635683+0000","last_change":"2024-09-21T09:40:50.110183+0000","last_active":"2024-09-21T09:41:31.635683+0000","last_peered":"2024-09-21T09:41:31.635683+0000","last_clean":"2024-09-21T09:41:31.635683+0000","last_became_active":"2024-09-21T09:40:50.109905+0000","last_became_peered":"2024-09-21T09:40:50.109905+0000","last_unstale":"2024-09-21T09:41:31.635683+0000","last_undegraded":"2024-09-21T09:41:31.635683+0000","last_fullsized":"2024-09-21T09:41:31.635683+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:27.951067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635899+0000","last_change":"2024-09-21T09:40:49.094180+0000","last_active":"2024-09-21T09:41:31.635899+0000","last_peered":"2024-09-21T09:41:31.635899+0000","last_clean":"2024-09-21T09:41:31.635899+0000","last_became_active":"2024-09-21T09:40:49.093764+0000","last_became_peered":"2024-09-21T09:40:49.093764+0000","last_unstale":"2024-09-21T09:41:31.635899+0000","last_undegraded":"2024-09-21T09:41:31.635899+0000","last_fullsized":"2024-09-21T09:41:31.635899+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:07:14.691262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637355+0000","last_change":"2024-09-21T09:40:50.115176+0000","last_active":"2024-09-21T09:41:31.637355+0000","last_peered":"2024-09-21T09:41:31.637355+0000","last_clean":"2024-09-21T09:41:31.637355+0000","last_became_active":"2024-09-21T09:40:50.114898+0000","last_became_peered":"2024-09-21T09:40:50.114898+0000","last_unstale":"2024-09-21T09:41:31.637355+0000","last_undegraded":"2024-09-21T09:41:31.637355+0000","last_fullsized":"2024-09-21T09:41:31.637355+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:36:41.487791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299449+0000","last_change":"2024-09-21T09:40:49.095578+0000","last_active":"2024-09-21T09:41:32.299449+0000","last_peered":"2024-09-21T09:41:32.299449+0000","last_clean":"2024-09-21T09:41:32.299449+0000","last_became_active":"2024-09-21T09:40:49.095255+0000","last_became_peered":"2024-09-21T09:40:49.095255+0000","last_unstale":"2024-09-21T09:41:32.299449+0000","last_undegraded":"2024-09-21T09:41:32.299449+0000","last_fullsized":"2024-09-21T09:41:32.299449+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:07:03.615721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.633204+0000","last_change":"2024-09-21T09:40:49.358568+0000","last_active":"2024-09-21T09:41:31.633204+0000","last_peered":"2024-09-21T09:41:31.633204+0000","last_clean":"2024-09-21T09:41:31.633204+0000","last_became_active":"2024-09-21T09:40:49.358032+0000","last_became_peered":"2024-09-21T09:40:49.358032+0000","last_unstale":"2024-09-21T09:41:31.633204+0000","last_undegraded":"2024-09-21T09:41:31.633204+0000","last_fullsized":"2024-09-21T09:41:31.633204+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:05:58.568478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635804+0000","last_change":"2024-09-21T09:40:49.098236+0000","last_active":"2024-09-21T09:41:31.635804+0000","last_peered":"2024-09-21T09:41:31.635804+0000","last_clean":"2024-09-21T09:41:31.635804+0000","last_became_active":"2024-09-21T09:40:49.097957+0000","last_became_peered":"2024-09-21T09:40:49.097957+0000","last_unstale":"2024-09-21T09:41:31.635804+0000","last_undegraded":"2024-09-21T09:41:31.635804+0000","last_fullsized":"2024-09-21T09:41:31.635804+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:18.193464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:41:33.337603+0000","last_change":"2024-09-21T09:41:33.337603+0000","last_active":"2024-09-21T09:41:33.337603+0000","last_peered":"2024-09-21T09:41:33.337603+0000","last_clean":"2024-09-21T09:41:33.337603+0000","last_became_active":"2024-09-21T09:41:33.337279+0000","last_became_peered":"2024-09-21T09:41:33.337279+0000","last_unstale":"2024-09-21T09:41:33.337603+0000","last_undegraded":"2024-09-21T09:41:33.337603+0000","last_fullsized":"2024-09-21T09:41:33.337603+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:20:41.290806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.633134+0000","last_change":"2024-09-21T09:40:49.099331+0000","last_active":"2024-09-21T09:41:31.633134+0000","last_peered":"2024-09-21T09:41:31.633134+0000","last_clean":"2024-09-21T09:41:31.633134+0000","last_became_active":"2024-09-21T09:40:49.099092+0000","last_became_peered":"2024-09-21T09:40:49.099092+0000","last_unstale":"2024-09-21T09:41:31.633134+0000","last_undegraded":"2024-09-21T09:41:31.633134+0000","last_fullsized":"2024-09-21T09:41:31.633134+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:08:46.341069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.633542+0000","last_change":"2024-09-21T09:40:49.343111+0000","last_active":"2024-09-21T09:41:31.633542+0000","last_peered":"2024-09-21T09:41:31.633542+0000","last_clean":"2024-09-21T09:41:31.633542+0000","last_became_active":"2024-09-21T09:40:49.342758+0000","last_became_peered":"2024-09-21T09:40:49.342758+0000","last_unstale":"2024-09-21T09:41:31.633542+0000","last_undegraded":"2024-09-21T09:41:31.633542+0000","last_fullsized":"2024-09-21T09:41:31.633542+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:11:18.775157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299347+0000","last_change":"2024-09-21T09:40:49.099674+0000","last_active":"2024-09-21T09:41:32.299347+0000","last_peered":"2024-09-21T09:41:32.299347+0000","last_clean":"2024-09-21T09:41:32.299347+0000","last_became_active":"2024-09-21T09:40:49.099393+0000","last_became_peered":"2024-09-21T09:40:49.099393+0000","last_unstale":"2024-09-21T09:41:32.299347+0000","last_undegraded":"2024-09-21T09:41:32.299347+0000","last_fullsized":"2024-09-21T09:41:32.299347+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:05:22.862304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635305+0000","last_change":"2024-09-21T09:40:50.115454+0000","last_active":"2024-09-21T09:41:31.635305+0000","last_peered":"2024-09-21T09:41:31.635305+0000","last_clean":"2024-09-21T09:41:31.635305+0000","last_became_active":"2024-09-21T09:40:50.115194+0000","last_became_peered":"2024-09-21T09:40:50.115194+0000","last_unstale":"2024-09-21T09:41:31.635305+0000","last_undegraded":"2024-09-21T09:41:31.635305+0000","last_fullsized":"2024-09-21T09:41:31.635305+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:00:07.015169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"51'79","reported_seq":56,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299630+0000","last_change":"2024-09-21T09:39:04.444589+0000","last_active":"2024-09-21T09:41:32.299630+0000","last_peered":"2024-09-21T09:41:32.299630+0000","last_clean":"2024-09-21T09:41:32.299630+0000","last_became_active":"2024-09-21T09:39:04.439052+0000","last_became_peered":"2024-09-21T09:39:04.439052+0000","last_unstale":"2024-09-21T09:41:32.299630+0000","last_undegraded":"2024-09-21T09:41:32.299630+0000","last_fullsized":"2024-09-21T09:41:32.299630+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:43.843355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":27,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635545+0000","last_change":"2024-09-21T09:39:39.494546+0000","last_active":"2024-09-21T09:41:31.635545+0000","last_peered":"2024-09-21T09:41:31.635545+0000","last_clean":"2024-09-21T09:41:31.635545+0000","last_became_active":"2024-09-21T09:39:34.903724+0000","last_became_peered":"2024-09-21T09:39:34.903724+0000","last_unstale":"2024-09-21T09:41:31.635545+0000","last_undegraded":"2024-09-21T09:41:31.635545+0000","last_fullsized":"2024-09-21T09:41:31.635545+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:38:35.213577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039581700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299407+0000","last_change":"2024-09-21T09:40:49.096746+0000","last_active":"2024-09-21T09:41:32.299407+0000","last_peered":"2024-09-21T09:41:32.299407+0000","last_clean":"2024-09-21T09:41:32.299407+0000","last_became_active":"2024-09-21T09:40:49.096323+0000","last_became_peered":"2024-09-21T09:40:49.096323+0000","last_unstale":"2024-09-21T09:41:32.299407+0000","last_undegraded":"2024-09-21T09:41:32.299407+0000","last_fullsized":"2024-09-21T09:41:32.299407+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:09:55.752922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635967+0000","last_change":"2024-09-21T09:40:50.115273+0000","last_active":"2024-09-21T09:41:31.635967+0000","last_peered":"2024-09-21T09:41:31.635967+0000","last_clean":"2024-09-21T09:41:31.635967+0000","last_became_active":"2024-09-21T09:40:50.114998+0000","last_became_peered":"2024-09-21T09:40:50.114998+0000","last_unstale":"2024-09-21T09:41:31.635967+0000","last_undegraded":"2024-09-21T09:41:31.635967+0000","last_fullsized":"2024-09-21T09:41:31.635967+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:42:39.208914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299548+0000","last_change":"2024-09-21T09:39:39.583589+0000","last_active":"2024-09-21T09:41:32.299548+0000","last_peered":"2024-09-21T09:41:32.299548+0000","last_clean":"2024-09-21T09:41:32.299548+0000","last_became_active":"2024-09-21T09:39:34.987518+0000","last_became_peered":"2024-09-21T09:39:34.987518+0000","last_unstale":"2024-09-21T09:41:32.299548+0000","last_undegraded":"2024-09-21T09:41:32.299548+0000","last_fullsized":"2024-09-21T09:41:32.299548+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:39.087119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637324+0000","last_change":"2024-09-21T09:40:49.098617+0000","last_active":"2024-09-21T09:41:31.637324+0000","last_peered":"2024-09-21T09:41:31.637324+0000","last_clean":"2024-09-21T09:41:31.637324+0000","last_became_active":"2024-09-21T09:40:49.097400+0000","last_became_peered":"2024-09-21T09:40:49.097400+0000","last_unstale":"2024-09-21T09:41:31.637324+0000","last_undegraded":"2024-09-21T09:41:31.637324+0000","last_fullsized":"2024-09-21T09:41:31.637324+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:13.494052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637561+0000","last_change":"2024-09-21T09:40:50.111425+0000","last_active":"2024-09-21T09:41:31.637561+0000","last_peered":"2024-09-21T09:41:31.637561+0000","last_clean":"2024-09-21T09:41:31.637561+0000","last_became_active":"2024-09-21T09:40:50.111164+0000","last_became_peered":"2024-09-21T09:40:50.111164+0000","last_unstale":"2024-09-21T09:41:31.637561+0000","last_undegraded":"2024-09-21T09:41:31.637561+0000","last_fullsized":"2024-09-21T09:41:31.637561+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:23:07.850398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.631424+0000","last_change":"2024-09-21T09:39:39.485018+0000","last_active":"2024-09-21T09:41:31.631424+0000","last_peered":"2024-09-21T09:41:31.631424+0000","last_clean":"2024-09-21T09:41:31.631424+0000","last_became_active":"2024-09-21T09:39:35.184521+0000","last_became_peered":"2024-09-21T09:39:35.184521+0000","last_unstale":"2024-09-21T09:41:31.631424+0000","last_undegraded":"2024-09-21T09:41:31.631424+0000","last_fullsized":"2024-09-21T09:41:31.631424+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:29.520758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635220+0000","last_change":"2024-09-21T09:40:49.101998+0000","last_active":"2024-09-21T09:41:31.635220+0000","last_peered":"2024-09-21T09:41:31.635220+0000","last_clean":"2024-09-21T09:41:31.635220+0000","last_became_active":"2024-09-21T09:40:49.101616+0000","last_became_peered":"2024-09-21T09:40:49.101616+0000","last_unstale":"2024-09-21T09:41:31.635220+0000","last_undegraded":"2024-09-21T09:41:31.635220+0000","last_fullsized":"2024-09-21T09:41:31.635220+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:39:07.393584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:41:33.337588+0000","last_change":"2024-09-21T09:41:33.337588+0000","last_active":"2024-09-21T09:41:33.337588+0000","last_peered":"2024-09-21T09:41:33.337588+0000","last_clean":"2024-09-21T09:41:33.337588+0000","last_became_active":"2024-09-21T09:41:33.337235+0000","last_became_peered":"2024-09-21T09:41:33.337235+0000","last_unstale":"2024-09-21T09:41:33.337588+0000","last_undegraded":"2024-09-21T09:41:33.337588+0000","last_fullsized":"2024-09-21T09:41:33.337588+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:11.889336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":28,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635592+0000","last_change":"2024-09-21T09:39:39.495072+0000","last_active":"2024-09-21T09:41:31.635592+0000","last_peered":"2024-09-21T09:41:31.635592+0000","last_clean":"2024-09-21T09:41:31.635592+0000","last_became_active":"2024-09-21T09:39:34.903742+0000","last_became_peered":"2024-09-21T09:39:34.903742+0000","last_unstale":"2024-09-21T09:41:31.635592+0000","last_undegraded":"2024-09-21T09:41:31.635592+0000","last_fullsized":"2024-09-21T09:41:31.635592+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:32.765230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091939799999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.633458+0000","last_change":"2024-09-21T09:40:49.098504+0000","last_active":"2024-09-21T09:41:31.633458+0000","last_peered":"2024-09-21T09:41:31.633458+0000","last_clean":"2024-09-21T09:41:31.633458+0000","last_became_active":"2024-09-21T09:40:49.098227+0000","last_became_peered":"2024-09-21T09:40:49.098227+0000","last_unstale":"2024-09-21T09:41:31.633458+0000","last_undegraded":"2024-09-21T09:41:31.633458+0000","last_fullsized":"2024-09-21T09:41:31.633458+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:02:24.430552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635233+0000","last_change":"2024-09-21T09:40:50.112909+0000","last_active":"2024-09-21T09:41:31.635233+0000","last_peered":"2024-09-21T09:41:31.635233+0000","last_clean":"2024-09-21T09:41:31.635233+0000","last_became_active":"2024-09-21T09:40:50.112632+0000","last_became_peered":"2024-09-21T09:40:50.112632+0000","last_unstale":"2024-09-21T09:41:31.635233+0000","last_undegraded":"2024-09-21T09:41:31.635233+0000","last_fullsized":"2024-09-21T09:41:31.635233+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:03.052630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.636450+0000","last_change":"2024-09-21T09:40:49.097438+0000","last_active":"2024-09-21T09:41:31.636450+0000","last_peered":"2024-09-21T09:41:31.636450+0000","last_clean":"2024-09-21T09:41:31.636450+0000","last_became_active":"2024-09-21T09:40:49.097092+0000","last_became_peered":"2024-09-21T09:40:49.097092+0000","last_unstale":"2024-09-21T09:41:31.636450+0000","last_undegraded":"2024-09-21T09:41:31.636450+0000","last_fullsized":"2024-09-21T09:41:31.636450+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:09:32.660792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635392+0000","last_change":"2024-09-21T09:39:39.495639+0000","last_active":"2024-09-21T09:41:31.635392+0000","last_peered":"2024-09-21T09:41:31.635392+0000","last_clean":"2024-09-21T09:41:31.635392+0000","last_became_active":"2024-09-21T09:39:34.987584+0000","last_became_peered":"2024-09-21T09:39:34.987584+0000","last_unstale":"2024-09-21T09:41:31.635392+0000","last_undegraded":"2024-09-21T09:41:31.635392+0000","last_fullsized":"2024-09-21T09:41:31.635392+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:31.185487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635315+0000","last_change":"2024-09-21T09:40:50.501447+0000","last_active":"2024-09-21T09:41:31.635315+0000","last_peered":"2024-09-21T09:41:31.635315+0000","last_clean":"2024-09-21T09:41:31.635315+0000","last_became_active":"2024-09-21T09:40:50.501154+0000","last_became_peered":"2024-09-21T09:40:50.501154+0000","last_unstale":"2024-09-21T09:41:31.635315+0000","last_undegraded":"2024-09-21T09:41:31.635315+0000","last_fullsized":"2024-09-21T09:41:31.635315+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:48.620260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635336+0000","last_change":"2024-09-21T09:40:49.102857+0000","last_active":"2024-09-21T09:41:31.635336+0000","last_peered":"2024-09-21T09:41:31.635336+0000","last_clean":"2024-09-21T09:41:31.635336+0000","last_became_active":"2024-09-21T09:40:49.102241+0000","last_became_peered":"2024-09-21T09:40:49.102241+0000","last_unstale":"2024-09-21T09:41:31.635336+0000","last_undegraded":"2024-09-21T09:41:31.635336+0000","last_fullsized":"2024-09-21T09:41:31.635336+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:26:57.404532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299983+0000","last_change":"2024-09-21T09:39:39.583831+0000","last_active":"2024-09-21T09:41:32.299983+0000","last_peered":"2024-09-21T09:41:32.299983+0000","last_clean":"2024-09-21T09:41:32.299983+0000","last_became_active":"2024-09-21T09:39:35.595483+0000","last_became_peered":"2024-09-21T09:39:35.595483+0000","last_unstale":"2024-09-21T09:41:32.299983+0000","last_undegraded":"2024-09-21T09:41:32.299983+0000","last_fullsized":"2024-09-21T09:41:32.299983+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:06.734095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637485+0000","last_change":"2024-09-21T09:40:50.501453+0000","last_active":"2024-09-21T09:41:31.637485+0000","last_peered":"2024-09-21T09:41:31.637485+0000","last_clean":"2024-09-21T09:41:31.637485+0000","last_became_active":"2024-09-21T09:40:50.501154+0000","last_became_peered":"2024-09-21T09:40:50.501154+0000","last_unstale":"2024-09-21T09:41:31.637485+0000","last_undegraded":"2024-09-21T09:41:31.637485+0000","last_fullsized":"2024-09-21T09:41:31.637485+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:47:16.531897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299649+0000","last_change":"2024-09-21T09:40:49.099022+0000","last_active":"2024-09-21T09:41:32.299649+0000","last_peered":"2024-09-21T09:41:32.299649+0000","last_clean":"2024-09-21T09:41:32.299649+0000","last_became_active":"2024-09-21T09:40:49.098513+0000","last_became_peered":"2024-09-21T09:40:49.098513+0000","last_unstale":"2024-09-21T09:41:32.299649+0000","last_undegraded":"2024-09-21T09:41:32.299649+0000","last_fullsized":"2024-09-21T09:41:32.299649+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:17:18.023236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635343+0000","last_change":"2024-09-21T09:39:39.495635+0000","last_active":"2024-09-21T09:41:31.635343+0000","last_peered":"2024-09-21T09:41:31.635343+0000","last_clean":"2024-09-21T09:41:31.635343+0000","last_became_active":"2024-09-21T09:39:35.595263+0000","last_became_peered":"2024-09-21T09:39:35.595263+0000","last_unstale":"2024-09-21T09:41:31.635343+0000","last_undegraded":"2024-09-21T09:41:31.635343+0000","last_fullsized":"2024-09-21T09:41:31.635343+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:29.108133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00019257899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299248+0000","last_change":"2024-09-21T09:40:50.109547+0000","last_active":"2024-09-21T09:41:32.299248+0000","last_peered":"2024-09-21T09:41:32.299248+0000","last_clean":"2024-09-21T09:41:32.299248+0000","last_became_active":"2024-09-21T09:40:50.109234+0000","last_became_peered":"2024-09-21T09:40:50.109234+0000","last_unstale":"2024-09-21T09:41:32.299248+0000","last_undegraded":"2024-09-21T09:41:32.299248+0000","last_fullsized":"2024-09-21T09:41:32.299248+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:14:52.206266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637628+0000","last_change":"2024-09-21T09:40:49.102365+0000","last_active":"2024-09-21T09:41:31.637628+0000","last_peered":"2024-09-21T09:41:31.637628+0000","last_clean":"2024-09-21T09:41:31.637628+0000","last_became_active":"2024-09-21T09:40:49.102080+0000","last_became_peered":"2024-09-21T09:40:49.102080+0000","last_unstale":"2024-09-21T09:41:31.637628+0000","last_undegraded":"2024-09-21T09:41:31.637628+0000","last_fullsized":"2024-09-21T09:41:31.637628+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:58.182812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":26,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635974+0000","last_change":"2024-09-21T09:39:39.496046+0000","last_active":"2024-09-21T09:41:31.635974+0000","last_peered":"2024-09-21T09:41:31.635974+0000","last_clean":"2024-09-21T09:41:31.635974+0000","last_became_active":"2024-09-21T09:39:34.903061+0000","last_became_peered":"2024-09-21T09:39:34.903061+0000","last_unstale":"2024-09-21T09:41:31.635974+0000","last_undegraded":"2024-09-21T09:41:31.635974+0000","last_fullsized":"2024-09-21T09:41:31.635974+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:41:37.281272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.633065+0000","last_change":"2024-09-21T09:40:49.190763+0000","last_active":"2024-09-21T09:41:31.633065+0000","last_peered":"2024-09-21T09:41:31.633065+0000","last_clean":"2024-09-21T09:41:31.633065+0000","last_became_active":"2024-09-21T09:40:49.190463+0000","last_became_peered":"2024-09-21T09:40:49.190463+0000","last_unstale":"2024-09-21T09:41:31.633065+0000","last_undegraded":"2024-09-21T09:41:31.633065+0000","last_fullsized":"2024-09-21T09:41:31.633065+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:15.669404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":23,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:41:33.258402+0000","last_change":"2024-09-21T09:41:33.258402+0000","last_active":"2024-09-21T09:41:33.258402+0000","last_peered":"2024-09-21T09:41:33.258402+0000","last_clean":"2024-09-21T09:41:33.258402+0000","last_became_active":"2024-09-21T09:41:33.258069+0000","last_became_peered":"2024-09-21T09:41:33.258069+0000","last_unstale":"2024-09-21T09:41:33.258402+0000","last_undegraded":"2024-09-21T09:41:33.258402+0000","last_fullsized":"2024-09-21T09:41:33.258402+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:40:12.150462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"63'6","reported_seq":20,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.633457+0000","last_change":"2024-09-21T09:40:49.096338+0000","last_active":"2024-09-21T09:41:31.633457+0000","last_peered":"2024-09-21T09:41:31.633457+0000","last_clean":"2024-09-21T09:41:31.633457+0000","last_became_active":"2024-09-21T09:40:49.096078+0000","last_became_peered":"2024-09-21T09:40:49.096078+0000","last_unstale":"2024-09-21T09:41:31.633457+0000","last_undegraded":"2024-09-21T09:41:31.633457+0000","last_fullsized":"2024-09-21T09:41:31.633457+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:56.500555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1552,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635049+0000","last_change":"2024-09-21T09:40:50.112032+0000","last_active":"2024-09-21T09:41:31.635049+0000","last_peered":"2024-09-21T09:41:31.635049+0000","last_clean":"2024-09-21T09:41:31.635049+0000","last_became_active":"2024-09-21T09:40:50.111707+0000","last_became_peered":"2024-09-21T09:40:50.111707+0000","last_unstale":"2024-09-21T09:41:31.635049+0000","last_undegraded":"2024-09-21T09:41:31.635049+0000","last_fullsized":"2024-09-21T09:41:31.635049+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:38:21.339598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":17,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299829+0000","last_change":"2024-09-21T09:40:49.104195+0000","last_active":"2024-09-21T09:41:32.299829+0000","last_peered":"2024-09-21T09:41:32.299829+0000","last_clean":"2024-09-21T09:41:32.299829+0000","last_became_active":"2024-09-21T09:40:49.103940+0000","last_became_peered":"2024-09-21T09:40:49.103940+0000","last_unstale":"2024-09-21T09:41:32.299829+0000","last_undegraded":"2024-09-21T09:41:32.299829+0000","last_fullsized":"2024-09-21T09:41:32.299829+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:33:18.339902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635520+0000","last_change":"2024-09-21T09:40:49.097293+0000","last_active":"2024-09-21T09:41:31.635520+0000","last_peered":"2024-09-21T09:41:31.635520+0000","last_clean":"2024-09-21T09:41:31.635520+0000","last_became_active":"2024-09-21T09:40:49.097030+0000","last_became_peered":"2024-09-21T09:40:49.097030+0000","last_unstale":"2024-09-21T09:41:31.635520+0000","last_undegraded":"2024-09-21T09:41:31.635520+0000","last_fullsized":"2024-09-21T09:41:31.635520+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:38.667234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299409+0000","last_change":"2024-09-21T09:40:50.110750+0000","last_active":"2024-09-21T09:41:32.299409+0000","last_peered":"2024-09-21T09:41:32.299409+0000","last_clean":"2024-09-21T09:41:32.299409+0000","last_became_active":"2024-09-21T09:40:50.110512+0000","last_became_peered":"2024-09-21T09:40:50.110512+0000","last_unstale":"2024-09-21T09:41:32.299409+0000","last_undegraded":"2024-09-21T09:41:32.299409+0000","last_fullsized":"2024-09-21T09:41:32.299409+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:45:44.520662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637872+0000","last_change":"2024-09-21T09:40:49.099387+0000","last_active":"2024-09-21T09:41:31.637872+0000","last_peered":"2024-09-21T09:41:31.637872+0000","last_clean":"2024-09-21T09:41:31.637872+0000","last_became_active":"2024-09-21T09:40:49.099129+0000","last_became_peered":"2024-09-21T09:40:49.099129+0000","last_unstale":"2024-09-21T09:41:31.637872+0000","last_undegraded":"2024-09-21T09:41:31.637872+0000","last_fullsized":"2024-09-21T09:41:31.637872+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:55:59.770545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299252+0000","last_change":"2024-09-21T09:40:50.113290+0000","last_active":"2024-09-21T09:41:32.299252+0000","last_peered":"2024-09-21T09:41:32.299252+0000","last_clean":"2024-09-21T09:41:32.299252+0000","last_became_active":"2024-09-21T09:40:50.113029+0000","last_became_peered":"2024-09-21T09:40:50.113029+0000","last_unstale":"2024-09-21T09:41:32.299252+0000","last_undegraded":"2024-09-21T09:41:32.299252+0000","last_fullsized":"2024-09-21T09:41:32.299252+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:59:41.871996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.636157+0000","last_change":"2024-09-21T09:40:50.501575+0000","last_active":"2024-09-21T09:41:31.636157+0000","last_peered":"2024-09-21T09:41:31.636157+0000","last_clean":"2024-09-21T09:41:31.636157+0000","last_became_active":"2024-09-21T09:40:50.501238+0000","last_became_peered":"2024-09-21T09:40:50.501238+0000","last_unstale":"2024-09-21T09:41:31.636157+0000","last_undegraded":"2024-09-21T09:41:31.636157+0000","last_fullsized":"2024-09-21T09:41:31.636157+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:37.454625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299338+0000","last_change":"2024-09-21T09:40:49.091638+0000","last_active":"2024-09-21T09:41:32.299338+0000","last_peered":"2024-09-21T09:41:32.299338+0000","last_clean":"2024-09-21T09:41:32.299338+0000","last_became_active":"2024-09-21T09:40:49.091305+0000","last_became_peered":"2024-09-21T09:40:49.091305+0000","last_unstale":"2024-09-21T09:41:32.299338+0000","last_undegraded":"2024-09-21T09:41:32.299338+0000","last_fullsized":"2024-09-21T09:41:32.299338+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:09.684078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637487+0000","last_change":"2024-09-21T09:40:50.113606+0000","last_active":"2024-09-21T09:41:31.637487+0000","last_peered":"2024-09-21T09:41:31.637487+0000","last_clean":"2024-09-21T09:41:31.637487+0000","last_became_active":"2024-09-21T09:40:50.113236+0000","last_became_peered":"2024-09-21T09:40:50.113236+0000","last_unstale":"2024-09-21T09:41:31.637487+0000","last_undegraded":"2024-09-21T09:41:31.637487+0000","last_fullsized":"2024-09-21T09:41:31.637487+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:33:52.693638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635038+0000","last_change":"2024-09-21T09:40:49.095954+0000","last_active":"2024-09-21T09:41:31.635038+0000","last_peered":"2024-09-21T09:41:31.635038+0000","last_clean":"2024-09-21T09:41:31.635038+0000","last_became_active":"2024-09-21T09:40:49.095641+0000","last_became_peered":"2024-09-21T09:40:49.095641+0000","last_unstale":"2024-09-21T09:41:31.635038+0000","last_undegraded":"2024-09-21T09:41:31.635038+0000","last_fullsized":"2024-09-21T09:41:31.635038+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:39:21.262097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299538+0000","last_change":"2024-09-21T09:40:50.116266+0000","last_active":"2024-09-21T09:41:32.299538+0000","last_peered":"2024-09-21T09:41:32.299538+0000","last_clean":"2024-09-21T09:41:32.299538+0000","last_became_active":"2024-09-21T09:40:50.115989+0000","last_became_peered":"2024-09-21T09:40:50.115989+0000","last_unstale":"2024-09-21T09:41:32.299538+0000","last_undegraded":"2024-09-21T09:41:32.299538+0000","last_fullsized":"2024-09-21T09:41:32.299538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:36.909000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.633304+0000","last_change":"2024-09-21T09:40:49.099618+0000","last_active":"2024-09-21T09:41:31.633304+0000","last_peered":"2024-09-21T09:41:31.633304+0000","last_clean":"2024-09-21T09:41:31.633304+0000","last_became_active":"2024-09-21T09:40:49.099133+0000","last_became_peered":"2024-09-21T09:40:49.099133+0000","last_unstale":"2024-09-21T09:41:31.633304+0000","last_undegraded":"2024-09-21T09:41:31.633304+0000","last_fullsized":"2024-09-21T09:41:31.633304+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:06:43.237147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.631561+0000","last_change":"2024-09-21T09:40:50.113409+0000","last_active":"2024-09-21T09:41:31.631561+0000","last_peered":"2024-09-21T09:41:31.631561+0000","last_clean":"2024-09-21T09:41:31.631561+0000","last_became_active":"2024-09-21T09:40:50.113150+0000","last_became_peered":"2024-09-21T09:40:50.113150+0000","last_unstale":"2024-09-21T09:41:31.631561+0000","last_undegraded":"2024-09-21T09:41:31.631561+0000","last_fullsized":"2024-09-21T09:41:31.631561+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:56.266738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637552+0000","last_change":"2024-09-21T09:40:49.100726+0000","last_active":"2024-09-21T09:41:31.637552+0000","last_peered":"2024-09-21T09:41:31.637552+0000","last_clean":"2024-09-21T09:41:31.637552+0000","last_became_active":"2024-09-21T09:40:49.100332+0000","last_became_peered":"2024-09-21T09:40:49.100332+0000","last_unstale":"2024-09-21T09:41:31.637552+0000","last_undegraded":"2024-09-21T09:41:31.637552+0000","last_fullsized":"2024-09-21T09:41:31.637552+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:12:54.280752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.633222+0000","last_change":"2024-09-21T09:40:49.358329+0000","last_active":"2024-09-21T09:41:31.633222+0000","last_peered":"2024-09-21T09:41:31.633222+0000","last_clean":"2024-09-21T09:41:31.633222+0000","last_became_active":"2024-09-21T09:40:49.358049+0000","last_became_peered":"2024-09-21T09:40:49.358049+0000","last_unstale":"2024-09-21T09:41:31.633222+0000","last_undegraded":"2024-09-21T09:41:31.633222+0000","last_fullsized":"2024-09-21T09:41:31.633222+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:55.412979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635800+0000","last_change":"2024-09-21T09:40:49.096426+0000","last_active":"2024-09-21T09:41:31.635800+0000","last_peered":"2024-09-21T09:41:31.635800+0000","last_clean":"2024-09-21T09:41:31.635800+0000","last_became_active":"2024-09-21T09:40:49.096167+0000","last_became_peered":"2024-09-21T09:40:49.096167+0000","last_unstale":"2024-09-21T09:41:31.635800+0000","last_undegraded":"2024-09-21T09:41:31.635800+0000","last_fullsized":"2024-09-21T09:41:31.635800+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:15:47.077072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.633089+0000","last_change":"2024-09-21T09:40:49.357760+0000","last_active":"2024-09-21T09:41:31.633089+0000","last_peered":"2024-09-21T09:41:31.633089+0000","last_clean":"2024-09-21T09:41:31.633089+0000","last_became_active":"2024-09-21T09:40:49.357530+0000","last_became_peered":"2024-09-21T09:40:49.357530+0000","last_unstale":"2024-09-21T09:41:31.633089+0000","last_undegraded":"2024-09-21T09:41:31.633089+0000","last_fullsized":"2024-09-21T09:41:31.633089+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:25.545361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.636329+0000","last_change":"2024-09-21T09:40:49.094916+0000","last_active":"2024-09-21T09:41:31.636329+0000","last_peered":"2024-09-21T09:41:31.636329+0000","last_clean":"2024-09-21T09:41:31.636329+0000","last_became_active":"2024-09-21T09:40:49.094264+0000","last_became_peered":"2024-09-21T09:40:49.094264+0000","last_unstale":"2024-09-21T09:41:31.636329+0000","last_undegraded":"2024-09-21T09:41:31.636329+0000","last_fullsized":"2024-09-21T09:41:31.636329+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:53.663226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299431+0000","last_change":"2024-09-21T09:40:50.109643+0000","last_active":"2024-09-21T09:41:32.299431+0000","last_peered":"2024-09-21T09:41:32.299431+0000","last_clean":"2024-09-21T09:41:32.299431+0000","last_became_active":"2024-09-21T09:40:50.109409+0000","last_became_peered":"2024-09-21T09:40:50.109409+0000","last_unstale":"2024-09-21T09:41:32.299431+0000","last_undegraded":"2024-09-21T09:41:32.299431+0000","last_fullsized":"2024-09-21T09:41:32.299431+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:26:30.062747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299613+0000","last_change":"2024-09-21T09:40:49.100579+0000","last_active":"2024-09-21T09:41:32.299613+0000","last_peered":"2024-09-21T09:41:32.299613+0000","last_clean":"2024-09-21T09:41:32.299613+0000","last_became_active":"2024-09-21T09:40:49.100281+0000","last_became_peered":"2024-09-21T09:40:49.100281+0000","last_unstale":"2024-09-21T09:41:32.299613+0000","last_undegraded":"2024-09-21T09:41:32.299613+0000","last_fullsized":"2024-09-21T09:41:32.299613+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:38:59.424535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.632891+0000","last_change":"2024-09-21T09:40:49.343171+0000","last_active":"2024-09-21T09:41:31.632891+0000","last_peered":"2024-09-21T09:41:31.632891+0000","last_clean":"2024-09-21T09:41:31.632891+0000","last_became_active":"2024-09-21T09:40:49.342795+0000","last_became_peered":"2024-09-21T09:40:49.342795+0000","last_unstale":"2024-09-21T09:41:31.632891+0000","last_undegraded":"2024-09-21T09:41:31.632891+0000","last_fullsized":"2024-09-21T09:41:31.632891+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:02.522569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":18,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637876+0000","last_change":"2024-09-21T09:40:49.097888+0000","last_active":"2024-09-21T09:41:31.637876+0000","last_peered":"2024-09-21T09:41:31.637876+0000","last_clean":"2024-09-21T09:41:31.637876+0000","last_became_active":"2024-09-21T09:40:49.097597+0000","last_became_peered":"2024-09-21T09:40:49.097597+0000","last_unstale":"2024-09-21T09:41:31.637876+0000","last_undegraded":"2024-09-21T09:41:31.637876+0000","last_fullsized":"2024-09-21T09:41:31.637876+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:29.096456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2338,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430244,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298616,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446536,"statfs":{"total":95995035648,"available":95689252864,"internally_reserved":0,"allocated":1286144,"data_stored":885824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:41:03 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.68899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.48099999999999998,"15min":0.47199999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.56100000000000005}]},{"osd":1,"last update":"Sat Sep 21 09:41:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.499,"15min":0.496},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.38600000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.53300000000000003}]},{"osd":2,"last update":"Sat Sep 21 09:41:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64200000000000002,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.50900000000000001,"15min":0.50600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.48099999999999998}]},{"osd":3,"last update":"Sat Sep 21 09:41:03 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.48199999999999998,"15min":0.47899999999999998},"min":{"1min":0.376,"5min":0.371,"15min":0.371},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.442},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.53600000000000003,"15min":0.53400000000000003},"min":{"1min":0.39900000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.50600000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:41:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.442,"15min":0.436},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.64900000000000002,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.434,"15min":0.43099999999999999},"min":{"1min":0.28499999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.52100000000000002}]},{"osd":5,"last update":"Sat Sep 21 09:41:03 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.46100000000000002,"15min":0.45900000000000002},"min":{"1min":0.33800000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.374},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.48699999999999999,"15min":0.48199999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71799999999999997,"5min":0.748,"15min":0.748},"last":0.46400000000000002}]},{"osd":6,"last update":"Sat Sep 21 09:41:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.496,"15min":0.49099999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.499,"15min":0.495},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67200000000000004,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.39200000000000002}]}]},{"osd":6,"up_from":42,"seq":180388626472,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298608,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446544,"statfs":{"total":95995035648,"available":95689261056,"internally_reserved":0,"allocated":1277952,"data_stored":885697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.46999999999999997,"15min":0.46600000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50600000000000001,"15min":0.504},"min":{"1min":0.35599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.40200000000000002}]},{"osd":1,"last update":"Sat Sep 21 09:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.50700000000000001,"15min":0.501},"min":{"1min":0.32200000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.40899999999999997}]},{"osd":2,"last update":"Sat Sep 21 09:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.50600000000000001,"15min":0.5},"min":{"1min":0.38,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73099999999999998,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.504},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.51400000000000001,"15min":0.51000000000000001},"min":{"1min":0.35599999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65600000000000003,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.38600000000000001}]},{"osd":3,"last update":"Sat Sep 21 09:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52600000000000002,"15min":0.52400000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.67600000000000005,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.52500000000000002,"15min":0.51900000000000002},"min":{"1min":0.38500000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.56599999999999995}]},{"osd":4,"last update":"Sat Sep 21 09:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.40899999999999997,"15min":0.40200000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.44500000000000001,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.44500000000000001}]},{"osd":5,"last update":"Sat Sep 21 09:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.442,"15min":0.42899999999999999},"min":{"1min":0.40699999999999997,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.46200000000000002,"15min":0.45200000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.52200000000000002}]},{"osd":7,"last update":"Sat Sep 21 09:41:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.505,"15min":0.504},"min":{"1min":0.35099999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71599999999999997,"5min":0.752,"15min":0.752},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.57499999999999996,"15min":0.57199999999999995},"min":{"1min":0.48199999999999998,"5min":0.432,"15min":0.432},"max":{"1min":0.79300000000000004,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.54600000000000004}]}]},{"osd":5,"up_from":36,"seq":154618822701,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298008,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447144,"statfs":{"total":95995035648,"available":95689875456,"internally_reserved":0,"allocated":667648,"data_stored":293717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:41:23 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51100000000000001,"15min":0.52300000000000002},"min":{"1min":0.35799999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64500000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.495,"5min":0.504,"15min":0.503},"min":{"1min":0.34100000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68100000000000005,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.57399999999999995}]},{"osd":1,"last update":"Sat Sep 21 09:41:23 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.51400000000000001,"15min":0.52200000000000002},"min":{"1min":0.35199999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.66200000000000003,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.625},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.52600000000000002,"15min":0.52500000000000002},"min":{"1min":0.39500000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69799999999999995,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.48599999999999999}]},{"osd":2,"last update":"Sat Sep 21 09:41:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51900000000000002,"15min":0.52900000000000003},"min":{"1min":0.34200000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.622,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53200000000000003,"15min":0.52500000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77600000000000002,"5min":1.054,"15min":1.054},"last":0.66500000000000004}]},{"osd":3,"last update":"Sat Sep 21 09:41:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.375,"5min":0.311,"15min":0.311},"max":{"1min":0.69599999999999995,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.53300000000000003,"15min":0.54100000000000004},"min":{"1min":0.316,"5min":0.309,"15min":0.309},"max":{"1min":0.72399999999999998,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.51500000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:41:23 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.40200000000000002,"15min":0.38},"min":{"1min":0.24199999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.71099999999999997,"5min":1.004,"15min":1.004},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.41299999999999998,"15min":0.39900000000000002},"min":{"1min":0.25700000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.74099999999999999,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.36699999999999999}]},{"osd":6,"last update":"Sat Sep 21 09:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.27300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67100000000000004,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.48399999999999999,"15min":0.49299999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67600000000000005,"5min":1.036,"15min":1.036},"last":0.435}]},{"osd":7,"last update":"Sat Sep 21 09:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.52100000000000002,"15min":0.52700000000000002},"min":{"1min":0.32300000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76100000000000001,"5min":1.107,"15min":1.107},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.54600000000000004,"15min":0.55400000000000005},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.745,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.54300000000000004}]}]},{"osd":4,"up_from":30,"seq":128849018930,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298024,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447128,"statfs":{"total":95995035648,"available":95689859072,"internally_reserved":0,"allocated":688128,"data_stored":295386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:40:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53600000000000003,"15min":0.54600000000000004},"min":{"1min":0.34599999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68600000000000005,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54700000000000004,"15min":0.55600000000000005},"min":{"1min":0.33800000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.69799999999999995}]},{"osd":1,"last update":"Sat Sep 21 09:40:58 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.50900000000000001,"15min":0.52500000000000002},"min":{"1min":0.29299999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73699999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.503,"15min":0.504},"min":{"1min":0.32300000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.55500000000000005}]},{"osd":2,"last update":"Sat Sep 21 09:40:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52500000000000002,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52800000000000002,"15min":0.54000000000000004},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.74299999999999999,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.63600000000000001}]},{"osd":3,"last update":"Sat Sep 21 09:40:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53400000000000003,"15min":0.54500000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.67400000000000004,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.53300000000000003,"15min":0.54500000000000004},"min":{"1min":0.32500000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.72499999999999998}]},{"osd":5,"last update":"Sat Sep 21 09:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.38700000000000001},"min":{"1min":0.29999999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.68600000000000005,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.44800000000000001,"15min":0.42899999999999999},"min":{"1min":0.312,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.75800000000000001,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.57299999999999995}]},{"osd":6,"last update":"Sat Sep 21 09:40:44 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.40600000000000003,"15min":0.40000000000000002},"min":{"1min":0.27000000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.623,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.42299999999999999,"15min":0.41899999999999998},"min":{"1min":0.255,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.71599999999999997,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.61599999999999999}]},{"osd":7,"last update":"Sat Sep 21 09:41:03 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47999999999999998,"15min":0.48199999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.74299999999999999,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.65000000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298008,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447144,"statfs":{"total":95995035648,"available":95689875456,"internally_reserved":0,"allocated":667648,"data_stored":293778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.35799999999999998,"15min":0.38200000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.48499999999999999,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.376,"15min":0.40200000000000002},"min":{"1min":0.254,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.45200000000000001,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.45800000000000002}]},{"osd":1,"last update":"Sat Sep 21 09:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.37,"15min":0.39200000000000002},"min":{"1min":0.251,"5min":0.22,"15min":0.22},"max":{"1min":0.46100000000000002,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.40799999999999997,"15min":0.432},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.54300000000000004,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.54300000000000004}]},{"osd":2,"last update":"Sat Sep 21 09:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.43099999999999999,"15min":0.45700000000000002},"min":{"1min":0.27200000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.57199999999999995,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.40600000000000003,"15min":0.42999999999999999},"min":{"1min":0.26400000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.49099999999999999,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.496}]},{"osd":4,"last update":"Sat Sep 21 09:40:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.56699999999999995,"15min":0.57299999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55700000000000005,"15min":0.55500000000000005},"min":{"1min":0.42099999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.57599999999999996}]},{"osd":5,"last update":"Sat Sep 21 09:41:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.57499999999999996,"15min":0.58699999999999997},"min":{"1min":0.379,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.69899999999999995,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57499999999999996,"15min":0.58699999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.59899999999999998}]},{"osd":6,"last update":"Sat Sep 21 09:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58399999999999996,"15min":0.59399999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.74299999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.59999999999999998,"15min":0.61899999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.69399999999999995}]},{"osd":7,"last update":"Sat Sep 21 09:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59099999999999997,"15min":0.59399999999999997},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.66600000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298012,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447140,"statfs":{"total":95995035648,"available":95689871360,"internally_reserved":0,"allocated":671744,"data_stored":294292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.34300000000000003,"15min":0.34100000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.315,"5min":0.35499999999999998,"15min":0.35999999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.59699999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.26600000000000001}]},{"osd":1,"last update":"Sat Sep 21 09:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.35299999999999998,"15min":0.35899999999999999},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38300000000000001,"15min":0.38500000000000001},"min":{"1min":0.28299999999999997,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.35199999999999998}]},{"osd":3,"last update":"Sat Sep 21 09:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.41299999999999998,"15min":0.43099999999999999},"min":{"1min":0.29099999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.39100000000000001,"15min":0.41599999999999998},"min":{"1min":0.26700000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.32100000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:40:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55300000000000005,"15min":0.56100000000000005},"min":{"1min":0.38600000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83599999999999997,"5min":0.878,"15min":0.878},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56100000000000005,"15min":0.56699999999999995},"min":{"1min":0.40400000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.49199999999999999}]},{"osd":5,"last update":"Sat Sep 21 09:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58099999999999996,"15min":0.58599999999999997},"min":{"1min":0.39400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56000000000000005,"15min":0.56599999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.58799999999999997}]},{"osd":6,"last update":"Sat Sep 21 09:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.376,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.5},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58599999999999997,"15min":0.58999999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73499999999999999,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53500000000000003}]},{"osd":7,"last update":"Sat Sep 21 09:41:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.68000000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74199999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.52300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298004,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447148,"statfs":{"total":95995035648,"available":95689879552,"internally_reserved":0,"allocated":663552,"data_stored":293392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:40:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33700000000000002,"15min":0.315},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34100000000000003,"15min":0.309},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.56699999999999995,"5min":0.627,"15min":0.627},"last":0.26800000000000002}]},{"osd":2,"last update":"Sat Sep 21 09:41:22 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34799999999999998,"15min":0.32800000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.502,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.253},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.377,"15min":0.35899999999999999},"min":{"1min":0.28100000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.28699999999999998}]},{"osd":3,"last update":"Sat Sep 21 09:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.40500000000000003,"15min":0.42299999999999999},"min":{"1min":0.26600000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.56000000000000005,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.41699999999999998,"15min":0.433},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.52200000000000002,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.32200000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:41:02 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.53300000000000003,"15min":0.54200000000000004},"min":{"1min":0.38100000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.68600000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.39200000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.79400000000000004,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.56699999999999995}]},{"osd":5,"last update":"Sat Sep 21 09:41:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.54900000000000004,"15min":0.54500000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76000000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.61499999999999999}]},{"osd":6,"last update":"Sat Sep 21 09:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.58099999999999996,"15min":0.58599999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58199999999999996,"15min":0.57399999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.58599999999999997}]},{"osd":7,"last update":"Sat Sep 21 09:41:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56699999999999995,"15min":0.57199999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.81100000000000005,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56299999999999994,"15min":0.56799999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.747,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.625}]}]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298592,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446560,"statfs":{"total":95995035648,"available":95689277440,"internally_reserved":0,"allocated":1265664,"data_stored":884648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 09:40:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.34200000000000003,"15min":0.312},"min":{"1min":0.20599999999999999,"5min":0.187,"15min":0.187},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.378},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.34200000000000003,"15min":0.32000000000000001},"min":{"1min":0.219,"5min":0.188,"15min":0.188},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.29499999999999998}]},{"osd":2,"last update":"Sat Sep 21 09:41:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36799999999999999,"15min":0.372},"min":{"1min":0.221,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.312},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38800000000000001,"15min":0.38600000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.40999999999999998}]},{"osd":3,"last update":"Sat Sep 21 09:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.41399999999999998,"15min":0.42599999999999999},"min":{"1min":0.30399999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.60199999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39200000000000002,"15min":0.40500000000000003},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.54400000000000004,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.36099999999999999}]},{"osd":4,"last update":"Sat Sep 21 09:41:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53100000000000003,"15min":0.53300000000000003},"min":{"1min":0.38300000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.67700000000000005,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.39100000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.83999999999999997,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.57299999999999995}]},{"osd":5,"last update":"Sat Sep 21 09:41:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.42699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.55300000000000005,"15min":0.56499999999999995},"min":{"1min":0.36199999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70199999999999996,"5min":0.87,"15min":0.87},"last":0.54200000000000004}]},{"osd":6,"last update":"Sat Sep 21 09:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55000000000000004,"15min":0.55400000000000005},"min":{"1min":0.373,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71499999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.623}]},{"osd":7,"last update":"Sat Sep 21 09:41:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54900000000000004,"15min":0.54400000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55700000000000005,"15min":0.55200000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.69899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T09:41:48.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T09:41:48.781 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:49.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:49 smithi107 ceph-mon[41185]: from='client.24677 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:41:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:49 smithi107 ceph-mon[29382]: from='client.24677 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:41:49.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:41:49 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:41:49] "GET /metrics HTTP/1.1" 200 239815 "" "Prometheus/2.43.0" 2024-09-21T09:41:49.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:49 smithi129 ceph-mon[32729]: from='client.24677 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:41:50.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:50 smithi107 ceph-mon[29382]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:50.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:50 smithi107 ceph-mon[41185]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:50.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:50 smithi129 ceph-mon[32729]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:50.675 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:41:50.676 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T09:41:51.301 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-09-21T09:41:49.016548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592725,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":160,"num_write_kb":1643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121,"ondisk_log_size":121,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385872,"kb_used_data":7020,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575344,"statfs":{"total":767960285184,"available":765517152256,"internally_reserved":0,"allocated":7188480,"data_stored":4126734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003014"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299480+0000","last_change":"2024-09-21T09:40:50.114134+0000","last_active":"2024-09-21T09:41:32.299480+0000","last_peered":"2024-09-21T09:41:32.299480+0000","last_clean":"2024-09-21T09:41:32.299480+0000","last_became_active":"2024-09-21T09:40:50.113655+0000","last_became_peered":"2024-09-21T09:40:50.113655+0000","last_unstale":"2024-09-21T09:41:32.299480+0000","last_undegraded":"2024-09-21T09:41:32.299480+0000","last_fullsized":"2024-09-21T09:41:32.299480+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:22:23.755499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299725+0000","last_change":"2024-09-21T09:40:49.104723+0000","last_active":"2024-09-21T09:41:32.299725+0000","last_peered":"2024-09-21T09:41:32.299725+0000","last_clean":"2024-09-21T09:41:32.299725+0000","last_became_active":"2024-09-21T09:40:49.104457+0000","last_became_peered":"2024-09-21T09:40:49.104457+0000","last_unstale":"2024-09-21T09:41:32.299725+0000","last_undegraded":"2024-09-21T09:41:32.299725+0000","last_fullsized":"2024-09-21T09:41:32.299725+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:16:54.926805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.636222+0000","last_change":"2024-09-21T09:40:49.397748+0000","last_active":"2024-09-21T09:41:31.636222+0000","last_peered":"2024-09-21T09:41:31.636222+0000","last_clean":"2024-09-21T09:41:31.636222+0000","last_became_active":"2024-09-21T09:40:49.397402+0000","last_became_peered":"2024-09-21T09:40:49.397402+0000","last_unstale":"2024-09-21T09:41:31.636222+0000","last_undegraded":"2024-09-21T09:41:31.636222+0000","last_fullsized":"2024-09-21T09:41:31.636222+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:10:55.764418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637455+0000","last_change":"2024-09-21T09:40:49.097777+0000","last_active":"2024-09-21T09:41:31.637455+0000","last_peered":"2024-09-21T09:41:31.637455+0000","last_clean":"2024-09-21T09:41:31.637455+0000","last_became_active":"2024-09-21T09:40:49.097404+0000","last_became_peered":"2024-09-21T09:40:49.097404+0000","last_unstale":"2024-09-21T09:41:31.637455+0000","last_undegraded":"2024-09-21T09:41:31.637455+0000","last_fullsized":"2024-09-21T09:41:31.637455+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:11:49.098768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637600+0000","last_change":"2024-09-21T09:40:50.109420+0000","last_active":"2024-09-21T09:41:31.637600+0000","last_peered":"2024-09-21T09:41:31.637600+0000","last_clean":"2024-09-21T09:41:31.637600+0000","last_became_active":"2024-09-21T09:40:50.109137+0000","last_became_peered":"2024-09-21T09:40:50.109137+0000","last_unstale":"2024-09-21T09:41:31.637600+0000","last_undegraded":"2024-09-21T09:41:31.637600+0000","last_fullsized":"2024-09-21T09:41:31.637600+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:16.565490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299629+0000","last_change":"2024-09-21T09:40:49.101911+0000","last_active":"2024-09-21T09:41:32.299629+0000","last_peered":"2024-09-21T09:41:32.299629+0000","last_clean":"2024-09-21T09:41:32.299629+0000","last_became_active":"2024-09-21T09:40:49.101629+0000","last_became_peered":"2024-09-21T09:40:49.101629+0000","last_unstale":"2024-09-21T09:41:32.299629+0000","last_undegraded":"2024-09-21T09:41:32.299629+0000","last_fullsized":"2024-09-21T09:41:32.299629+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:07:19.215551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299613+0000","last_change":"2024-09-21T09:40:50.113148+0000","last_active":"2024-09-21T09:41:32.299613+0000","last_peered":"2024-09-21T09:41:32.299613+0000","last_clean":"2024-09-21T09:41:32.299613+0000","last_became_active":"2024-09-21T09:40:50.112896+0000","last_became_peered":"2024-09-21T09:40:50.112896+0000","last_unstale":"2024-09-21T09:41:32.299613+0000","last_undegraded":"2024-09-21T09:41:32.299613+0000","last_fullsized":"2024-09-21T09:41:32.299613+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:53.011179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":22,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635726+0000","last_change":"2024-09-21T09:40:49.099918+0000","last_active":"2024-09-21T09:41:31.635726+0000","last_peered":"2024-09-21T09:41:31.635726+0000","last_clean":"2024-09-21T09:41:31.635726+0000","last_became_active":"2024-09-21T09:40:49.099642+0000","last_became_peered":"2024-09-21T09:40:49.099642+0000","last_unstale":"2024-09-21T09:41:31.635726+0000","last_undegraded":"2024-09-21T09:41:31.635726+0000","last_fullsized":"2024-09-21T09:41:31.635726+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:40:55.840709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.300025+0000","last_change":"2024-09-21T09:40:49.103815+0000","last_active":"2024-09-21T09:41:32.300025+0000","last_peered":"2024-09-21T09:41:32.300025+0000","last_clean":"2024-09-21T09:41:32.300025+0000","last_became_active":"2024-09-21T09:40:49.101820+0000","last_became_peered":"2024-09-21T09:40:49.101820+0000","last_unstale":"2024-09-21T09:41:32.300025+0000","last_undegraded":"2024-09-21T09:41:32.300025+0000","last_fullsized":"2024-09-21T09:41:32.300025+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:09:32.403362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635364+0000","last_change":"2024-09-21T09:40:49.397673+0000","last_active":"2024-09-21T09:41:31.635364+0000","last_peered":"2024-09-21T09:41:31.635364+0000","last_clean":"2024-09-21T09:41:31.635364+0000","last_became_active":"2024-09-21T09:40:49.397365+0000","last_became_peered":"2024-09-21T09:40:49.397365+0000","last_unstale":"2024-09-21T09:41:31.635364+0000","last_undegraded":"2024-09-21T09:41:31.635364+0000","last_fullsized":"2024-09-21T09:41:31.635364+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:48:55.191816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.300022+0000","last_change":"2024-09-21T09:40:49.101165+0000","last_active":"2024-09-21T09:41:32.300022+0000","last_peered":"2024-09-21T09:41:32.300022+0000","last_clean":"2024-09-21T09:41:32.300022+0000","last_became_active":"2024-09-21T09:40:49.100866+0000","last_became_peered":"2024-09-21T09:40:49.100866+0000","last_unstale":"2024-09-21T09:41:32.300022+0000","last_undegraded":"2024-09-21T09:41:32.300022+0000","last_fullsized":"2024-09-21T09:41:32.300022+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:00.626279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.633013+0000","last_change":"2024-09-21T09:40:49.358400+0000","last_active":"2024-09-21T09:41:31.633013+0000","last_peered":"2024-09-21T09:41:31.633013+0000","last_clean":"2024-09-21T09:41:31.633013+0000","last_became_active":"2024-09-21T09:40:49.358128+0000","last_became_peered":"2024-09-21T09:40:49.358128+0000","last_unstale":"2024-09-21T09:41:31.633013+0000","last_undegraded":"2024-09-21T09:41:31.633013+0000","last_fullsized":"2024-09-21T09:41:31.633013+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:40.278954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299930+0000","last_change":"2024-09-21T09:40:49.096812+0000","last_active":"2024-09-21T09:41:32.299930+0000","last_peered":"2024-09-21T09:41:32.299930+0000","last_clean":"2024-09-21T09:41:32.299930+0000","last_became_active":"2024-09-21T09:40:49.096549+0000","last_became_peered":"2024-09-21T09:40:49.096549+0000","last_unstale":"2024-09-21T09:41:32.299930+0000","last_undegraded":"2024-09-21T09:41:32.299930+0000","last_fullsized":"2024-09-21T09:41:32.299930+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:24.512887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635683+0000","last_change":"2024-09-21T09:40:50.110183+0000","last_active":"2024-09-21T09:41:31.635683+0000","last_peered":"2024-09-21T09:41:31.635683+0000","last_clean":"2024-09-21T09:41:31.635683+0000","last_became_active":"2024-09-21T09:40:50.109905+0000","last_became_peered":"2024-09-21T09:40:50.109905+0000","last_unstale":"2024-09-21T09:41:31.635683+0000","last_undegraded":"2024-09-21T09:41:31.635683+0000","last_fullsized":"2024-09-21T09:41:31.635683+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:27.951067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635899+0000","last_change":"2024-09-21T09:40:49.094180+0000","last_active":"2024-09-21T09:41:31.635899+0000","last_peered":"2024-09-21T09:41:31.635899+0000","last_clean":"2024-09-21T09:41:31.635899+0000","last_became_active":"2024-09-21T09:40:49.093764+0000","last_became_peered":"2024-09-21T09:40:49.093764+0000","last_unstale":"2024-09-21T09:41:31.635899+0000","last_undegraded":"2024-09-21T09:41:31.635899+0000","last_fullsized":"2024-09-21T09:41:31.635899+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:07:14.691262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637355+0000","last_change":"2024-09-21T09:40:50.115176+0000","last_active":"2024-09-21T09:41:31.637355+0000","last_peered":"2024-09-21T09:41:31.637355+0000","last_clean":"2024-09-21T09:41:31.637355+0000","last_became_active":"2024-09-21T09:40:50.114898+0000","last_became_peered":"2024-09-21T09:40:50.114898+0000","last_unstale":"2024-09-21T09:41:31.637355+0000","last_undegraded":"2024-09-21T09:41:31.637355+0000","last_fullsized":"2024-09-21T09:41:31.637355+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:36:41.487791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299449+0000","last_change":"2024-09-21T09:40:49.095578+0000","last_active":"2024-09-21T09:41:32.299449+0000","last_peered":"2024-09-21T09:41:32.299449+0000","last_clean":"2024-09-21T09:41:32.299449+0000","last_became_active":"2024-09-21T09:40:49.095255+0000","last_became_peered":"2024-09-21T09:40:49.095255+0000","last_unstale":"2024-09-21T09:41:32.299449+0000","last_undegraded":"2024-09-21T09:41:32.299449+0000","last_fullsized":"2024-09-21T09:41:32.299449+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:07:03.615721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.633204+0000","last_change":"2024-09-21T09:40:49.358568+0000","last_active":"2024-09-21T09:41:31.633204+0000","last_peered":"2024-09-21T09:41:31.633204+0000","last_clean":"2024-09-21T09:41:31.633204+0000","last_became_active":"2024-09-21T09:40:49.358032+0000","last_became_peered":"2024-09-21T09:40:49.358032+0000","last_unstale":"2024-09-21T09:41:31.633204+0000","last_undegraded":"2024-09-21T09:41:31.633204+0000","last_fullsized":"2024-09-21T09:41:31.633204+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:05:58.568478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635804+0000","last_change":"2024-09-21T09:40:49.098236+0000","last_active":"2024-09-21T09:41:31.635804+0000","last_peered":"2024-09-21T09:41:31.635804+0000","last_clean":"2024-09-21T09:41:31.635804+0000","last_became_active":"2024-09-21T09:40:49.097957+0000","last_became_peered":"2024-09-21T09:40:49.097957+0000","last_unstale":"2024-09-21T09:41:31.635804+0000","last_undegraded":"2024-09-21T09:41:31.635804+0000","last_fullsized":"2024-09-21T09:41:31.635804+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:18.193464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:41:33.337603+0000","last_change":"2024-09-21T09:41:33.337603+0000","last_active":"2024-09-21T09:41:33.337603+0000","last_peered":"2024-09-21T09:41:33.337603+0000","last_clean":"2024-09-21T09:41:33.337603+0000","last_became_active":"2024-09-21T09:41:33.337279+0000","last_became_peered":"2024-09-21T09:41:33.337279+0000","last_unstale":"2024-09-21T09:41:33.337603+0000","last_undegraded":"2024-09-21T09:41:33.337603+0000","last_fullsized":"2024-09-21T09:41:33.337603+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:20:41.290806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.633134+0000","last_change":"2024-09-21T09:40:49.099331+0000","last_active":"2024-09-21T09:41:31.633134+0000","last_peered":"2024-09-21T09:41:31.633134+0000","last_clean":"2024-09-21T09:41:31.633134+0000","last_became_active":"2024-09-21T09:40:49.099092+0000","last_became_peered":"2024-09-21T09:40:49.099092+0000","last_unstale":"2024-09-21T09:41:31.633134+0000","last_undegraded":"2024-09-21T09:41:31.633134+0000","last_fullsized":"2024-09-21T09:41:31.633134+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:08:46.341069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.633542+0000","last_change":"2024-09-21T09:40:49.343111+0000","last_active":"2024-09-21T09:41:31.633542+0000","last_peered":"2024-09-21T09:41:31.633542+0000","last_clean":"2024-09-21T09:41:31.633542+0000","last_became_active":"2024-09-21T09:40:49.342758+0000","last_became_peered":"2024-09-21T09:40:49.342758+0000","last_unstale":"2024-09-21T09:41:31.633542+0000","last_undegraded":"2024-09-21T09:41:31.633542+0000","last_fullsized":"2024-09-21T09:41:31.633542+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:11:18.775157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299347+0000","last_change":"2024-09-21T09:40:49.099674+0000","last_active":"2024-09-21T09:41:32.299347+0000","last_peered":"2024-09-21T09:41:32.299347+0000","last_clean":"2024-09-21T09:41:32.299347+0000","last_became_active":"2024-09-21T09:40:49.099393+0000","last_became_peered":"2024-09-21T09:40:49.099393+0000","last_unstale":"2024-09-21T09:41:32.299347+0000","last_undegraded":"2024-09-21T09:41:32.299347+0000","last_fullsized":"2024-09-21T09:41:32.299347+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:05:22.862304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635305+0000","last_change":"2024-09-21T09:40:50.115454+0000","last_active":"2024-09-21T09:41:31.635305+0000","last_peered":"2024-09-21T09:41:31.635305+0000","last_clean":"2024-09-21T09:41:31.635305+0000","last_became_active":"2024-09-21T09:40:50.115194+0000","last_became_peered":"2024-09-21T09:40:50.115194+0000","last_unstale":"2024-09-21T09:41:31.635305+0000","last_undegraded":"2024-09-21T09:41:31.635305+0000","last_fullsized":"2024-09-21T09:41:31.635305+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:00:07.015169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"51'79","reported_seq":56,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299630+0000","last_change":"2024-09-21T09:39:04.444589+0000","last_active":"2024-09-21T09:41:32.299630+0000","last_peered":"2024-09-21T09:41:32.299630+0000","last_clean":"2024-09-21T09:41:32.299630+0000","last_became_active":"2024-09-21T09:39:04.439052+0000","last_became_peered":"2024-09-21T09:39:04.439052+0000","last_unstale":"2024-09-21T09:41:32.299630+0000","last_undegraded":"2024-09-21T09:41:32.299630+0000","last_fullsized":"2024-09-21T09:41:32.299630+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:43.843355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":27,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635545+0000","last_change":"2024-09-21T09:39:39.494546+0000","last_active":"2024-09-21T09:41:31.635545+0000","last_peered":"2024-09-21T09:41:31.635545+0000","last_clean":"2024-09-21T09:41:31.635545+0000","last_became_active":"2024-09-21T09:39:34.903724+0000","last_became_peered":"2024-09-21T09:39:34.903724+0000","last_unstale":"2024-09-21T09:41:31.635545+0000","last_undegraded":"2024-09-21T09:41:31.635545+0000","last_fullsized":"2024-09-21T09:41:31.635545+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:38:35.213577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039581700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299407+0000","last_change":"2024-09-21T09:40:49.096746+0000","last_active":"2024-09-21T09:41:32.299407+0000","last_peered":"2024-09-21T09:41:32.299407+0000","last_clean":"2024-09-21T09:41:32.299407+0000","last_became_active":"2024-09-21T09:40:49.096323+0000","last_became_peered":"2024-09-21T09:40:49.096323+0000","last_unstale":"2024-09-21T09:41:32.299407+0000","last_undegraded":"2024-09-21T09:41:32.299407+0000","last_fullsized":"2024-09-21T09:41:32.299407+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:09:55.752922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635967+0000","last_change":"2024-09-21T09:40:50.115273+0000","last_active":"2024-09-21T09:41:31.635967+0000","last_peered":"2024-09-21T09:41:31.635967+0000","last_clean":"2024-09-21T09:41:31.635967+0000","last_became_active":"2024-09-21T09:40:50.114998+0000","last_became_peered":"2024-09-21T09:40:50.114998+0000","last_unstale":"2024-09-21T09:41:31.635967+0000","last_undegraded":"2024-09-21T09:41:31.635967+0000","last_fullsized":"2024-09-21T09:41:31.635967+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:42:39.208914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299548+0000","last_change":"2024-09-21T09:39:39.583589+0000","last_active":"2024-09-21T09:41:32.299548+0000","last_peered":"2024-09-21T09:41:32.299548+0000","last_clean":"2024-09-21T09:41:32.299548+0000","last_became_active":"2024-09-21T09:39:34.987518+0000","last_became_peered":"2024-09-21T09:39:34.987518+0000","last_unstale":"2024-09-21T09:41:32.299548+0000","last_undegraded":"2024-09-21T09:41:32.299548+0000","last_fullsized":"2024-09-21T09:41:32.299548+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:39.087119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637324+0000","last_change":"2024-09-21T09:40:49.098617+0000","last_active":"2024-09-21T09:41:31.637324+0000","last_peered":"2024-09-21T09:41:31.637324+0000","last_clean":"2024-09-21T09:41:31.637324+0000","last_became_active":"2024-09-21T09:40:49.097400+0000","last_became_peered":"2024-09-21T09:40:49.097400+0000","last_unstale":"2024-09-21T09:41:31.637324+0000","last_undegraded":"2024-09-21T09:41:31.637324+0000","last_fullsized":"2024-09-21T09:41:31.637324+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:13.494052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637561+0000","last_change":"2024-09-21T09:40:50.111425+0000","last_active":"2024-09-21T09:41:31.637561+0000","last_peered":"2024-09-21T09:41:31.637561+0000","last_clean":"2024-09-21T09:41:31.637561+0000","last_became_active":"2024-09-21T09:40:50.111164+0000","last_became_peered":"2024-09-21T09:40:50.111164+0000","last_unstale":"2024-09-21T09:41:31.637561+0000","last_undegraded":"2024-09-21T09:41:31.637561+0000","last_fullsized":"2024-09-21T09:41:31.637561+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:23:07.850398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.631424+0000","last_change":"2024-09-21T09:39:39.485018+0000","last_active":"2024-09-21T09:41:31.631424+0000","last_peered":"2024-09-21T09:41:31.631424+0000","last_clean":"2024-09-21T09:41:31.631424+0000","last_became_active":"2024-09-21T09:39:35.184521+0000","last_became_peered":"2024-09-21T09:39:35.184521+0000","last_unstale":"2024-09-21T09:41:31.631424+0000","last_undegraded":"2024-09-21T09:41:31.631424+0000","last_fullsized":"2024-09-21T09:41:31.631424+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:29.520758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635220+0000","last_change":"2024-09-21T09:40:49.101998+0000","last_active":"2024-09-21T09:41:31.635220+0000","last_peered":"2024-09-21T09:41:31.635220+0000","last_clean":"2024-09-21T09:41:31.635220+0000","last_became_active":"2024-09-21T09:40:49.101616+0000","last_became_peered":"2024-09-21T09:40:49.101616+0000","last_unstale":"2024-09-21T09:41:31.635220+0000","last_undegraded":"2024-09-21T09:41:31.635220+0000","last_fullsized":"2024-09-21T09:41:31.635220+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:39:07.393584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:41:33.337588+0000","last_change":"2024-09-21T09:41:33.337588+0000","last_active":"2024-09-21T09:41:33.337588+0000","last_peered":"2024-09-21T09:41:33.337588+0000","last_clean":"2024-09-21T09:41:33.337588+0000","last_became_active":"2024-09-21T09:41:33.337235+0000","last_became_peered":"2024-09-21T09:41:33.337235+0000","last_unstale":"2024-09-21T09:41:33.337588+0000","last_undegraded":"2024-09-21T09:41:33.337588+0000","last_fullsized":"2024-09-21T09:41:33.337588+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:11.889336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":28,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635592+0000","last_change":"2024-09-21T09:39:39.495072+0000","last_active":"2024-09-21T09:41:31.635592+0000","last_peered":"2024-09-21T09:41:31.635592+0000","last_clean":"2024-09-21T09:41:31.635592+0000","last_became_active":"2024-09-21T09:39:34.903742+0000","last_became_peered":"2024-09-21T09:39:34.903742+0000","last_unstale":"2024-09-21T09:41:31.635592+0000","last_undegraded":"2024-09-21T09:41:31.635592+0000","last_fullsized":"2024-09-21T09:41:31.635592+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:32.765230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091939799999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.633458+0000","last_change":"2024-09-21T09:40:49.098504+0000","last_active":"2024-09-21T09:41:31.633458+0000","last_peered":"2024-09-21T09:41:31.633458+0000","last_clean":"2024-09-21T09:41:31.633458+0000","last_became_active":"2024-09-21T09:40:49.098227+0000","last_became_peered":"2024-09-21T09:40:49.098227+0000","last_unstale":"2024-09-21T09:41:31.633458+0000","last_undegraded":"2024-09-21T09:41:31.633458+0000","last_fullsized":"2024-09-21T09:41:31.633458+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:02:24.430552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635233+0000","last_change":"2024-09-21T09:40:50.112909+0000","last_active":"2024-09-21T09:41:31.635233+0000","last_peered":"2024-09-21T09:41:31.635233+0000","last_clean":"2024-09-21T09:41:31.635233+0000","last_became_active":"2024-09-21T09:40:50.112632+0000","last_became_peered":"2024-09-21T09:40:50.112632+0000","last_unstale":"2024-09-21T09:41:31.635233+0000","last_undegraded":"2024-09-21T09:41:31.635233+0000","last_fullsized":"2024-09-21T09:41:31.635233+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:03.052630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.636450+0000","last_change":"2024-09-21T09:40:49.097438+0000","last_active":"2024-09-21T09:41:31.636450+0000","last_peered":"2024-09-21T09:41:31.636450+0000","last_clean":"2024-09-21T09:41:31.636450+0000","last_became_active":"2024-09-21T09:40:49.097092+0000","last_became_peered":"2024-09-21T09:40:49.097092+0000","last_unstale":"2024-09-21T09:41:31.636450+0000","last_undegraded":"2024-09-21T09:41:31.636450+0000","last_fullsized":"2024-09-21T09:41:31.636450+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:09:32.660792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635392+0000","last_change":"2024-09-21T09:39:39.495639+0000","last_active":"2024-09-21T09:41:31.635392+0000","last_peered":"2024-09-21T09:41:31.635392+0000","last_clean":"2024-09-21T09:41:31.635392+0000","last_became_active":"2024-09-21T09:39:34.987584+0000","last_became_peered":"2024-09-21T09:39:34.987584+0000","last_unstale":"2024-09-21T09:41:31.635392+0000","last_undegraded":"2024-09-21T09:41:31.635392+0000","last_fullsized":"2024-09-21T09:41:31.635392+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:31.185487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635315+0000","last_change":"2024-09-21T09:40:50.501447+0000","last_active":"2024-09-21T09:41:31.635315+0000","last_peered":"2024-09-21T09:41:31.635315+0000","last_clean":"2024-09-21T09:41:31.635315+0000","last_became_active":"2024-09-21T09:40:50.501154+0000","last_became_peered":"2024-09-21T09:40:50.501154+0000","last_unstale":"2024-09-21T09:41:31.635315+0000","last_undegraded":"2024-09-21T09:41:31.635315+0000","last_fullsized":"2024-09-21T09:41:31.635315+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:48.620260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635336+0000","last_change":"2024-09-21T09:40:49.102857+0000","last_active":"2024-09-21T09:41:31.635336+0000","last_peered":"2024-09-21T09:41:31.635336+0000","last_clean":"2024-09-21T09:41:31.635336+0000","last_became_active":"2024-09-21T09:40:49.102241+0000","last_became_peered":"2024-09-21T09:40:49.102241+0000","last_unstale":"2024-09-21T09:41:31.635336+0000","last_undegraded":"2024-09-21T09:41:31.635336+0000","last_fullsized":"2024-09-21T09:41:31.635336+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:26:57.404532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299983+0000","last_change":"2024-09-21T09:39:39.583831+0000","last_active":"2024-09-21T09:41:32.299983+0000","last_peered":"2024-09-21T09:41:32.299983+0000","last_clean":"2024-09-21T09:41:32.299983+0000","last_became_active":"2024-09-21T09:39:35.595483+0000","last_became_peered":"2024-09-21T09:39:35.595483+0000","last_unstale":"2024-09-21T09:41:32.299983+0000","last_undegraded":"2024-09-21T09:41:32.299983+0000","last_fullsized":"2024-09-21T09:41:32.299983+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:06.734095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637485+0000","last_change":"2024-09-21T09:40:50.501453+0000","last_active":"2024-09-21T09:41:31.637485+0000","last_peered":"2024-09-21T09:41:31.637485+0000","last_clean":"2024-09-21T09:41:31.637485+0000","last_became_active":"2024-09-21T09:40:50.501154+0000","last_became_peered":"2024-09-21T09:40:50.501154+0000","last_unstale":"2024-09-21T09:41:31.637485+0000","last_undegraded":"2024-09-21T09:41:31.637485+0000","last_fullsized":"2024-09-21T09:41:31.637485+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:47:16.531897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299649+0000","last_change":"2024-09-21T09:40:49.099022+0000","last_active":"2024-09-21T09:41:32.299649+0000","last_peered":"2024-09-21T09:41:32.299649+0000","last_clean":"2024-09-21T09:41:32.299649+0000","last_became_active":"2024-09-21T09:40:49.098513+0000","last_became_peered":"2024-09-21T09:40:49.098513+0000","last_unstale":"2024-09-21T09:41:32.299649+0000","last_undegraded":"2024-09-21T09:41:32.299649+0000","last_fullsized":"2024-09-21T09:41:32.299649+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:17:18.023236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635343+0000","last_change":"2024-09-21T09:39:39.495635+0000","last_active":"2024-09-21T09:41:31.635343+0000","last_peered":"2024-09-21T09:41:31.635343+0000","last_clean":"2024-09-21T09:41:31.635343+0000","last_became_active":"2024-09-21T09:39:35.595263+0000","last_became_peered":"2024-09-21T09:39:35.595263+0000","last_unstale":"2024-09-21T09:41:31.635343+0000","last_undegraded":"2024-09-21T09:41:31.635343+0000","last_fullsized":"2024-09-21T09:41:31.635343+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:29.108133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00019257899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299248+0000","last_change":"2024-09-21T09:40:50.109547+0000","last_active":"2024-09-21T09:41:32.299248+0000","last_peered":"2024-09-21T09:41:32.299248+0000","last_clean":"2024-09-21T09:41:32.299248+0000","last_became_active":"2024-09-21T09:40:50.109234+0000","last_became_peered":"2024-09-21T09:40:50.109234+0000","last_unstale":"2024-09-21T09:41:32.299248+0000","last_undegraded":"2024-09-21T09:41:32.299248+0000","last_fullsized":"2024-09-21T09:41:32.299248+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:14:52.206266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637628+0000","last_change":"2024-09-21T09:40:49.102365+0000","last_active":"2024-09-21T09:41:31.637628+0000","last_peered":"2024-09-21T09:41:31.637628+0000","last_clean":"2024-09-21T09:41:31.637628+0000","last_became_active":"2024-09-21T09:40:49.102080+0000","last_became_peered":"2024-09-21T09:40:49.102080+0000","last_unstale":"2024-09-21T09:41:31.637628+0000","last_undegraded":"2024-09-21T09:41:31.637628+0000","last_fullsized":"2024-09-21T09:41:31.637628+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:58.182812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":26,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635974+0000","last_change":"2024-09-21T09:39:39.496046+0000","last_active":"2024-09-21T09:41:31.635974+0000","last_peered":"2024-09-21T09:41:31.635974+0000","last_clean":"2024-09-21T09:41:31.635974+0000","last_became_active":"2024-09-21T09:39:34.903061+0000","last_became_peered":"2024-09-21T09:39:34.903061+0000","last_unstale":"2024-09-21T09:41:31.635974+0000","last_undegraded":"2024-09-21T09:41:31.635974+0000","last_fullsized":"2024-09-21T09:41:31.635974+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:41:37.281272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.633065+0000","last_change":"2024-09-21T09:40:49.190763+0000","last_active":"2024-09-21T09:41:31.633065+0000","last_peered":"2024-09-21T09:41:31.633065+0000","last_clean":"2024-09-21T09:41:31.633065+0000","last_became_active":"2024-09-21T09:40:49.190463+0000","last_became_peered":"2024-09-21T09:40:49.190463+0000","last_unstale":"2024-09-21T09:41:31.633065+0000","last_undegraded":"2024-09-21T09:41:31.633065+0000","last_fullsized":"2024-09-21T09:41:31.633065+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:15.669404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":23,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:41:33.258402+0000","last_change":"2024-09-21T09:41:33.258402+0000","last_active":"2024-09-21T09:41:33.258402+0000","last_peered":"2024-09-21T09:41:33.258402+0000","last_clean":"2024-09-21T09:41:33.258402+0000","last_became_active":"2024-09-21T09:41:33.258069+0000","last_became_peered":"2024-09-21T09:41:33.258069+0000","last_unstale":"2024-09-21T09:41:33.258402+0000","last_undegraded":"2024-09-21T09:41:33.258402+0000","last_fullsized":"2024-09-21T09:41:33.258402+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:40:12.150462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"63'6","reported_seq":20,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.633457+0000","last_change":"2024-09-21T09:40:49.096338+0000","last_active":"2024-09-21T09:41:31.633457+0000","last_peered":"2024-09-21T09:41:31.633457+0000","last_clean":"2024-09-21T09:41:31.633457+0000","last_became_active":"2024-09-21T09:40:49.096078+0000","last_became_peered":"2024-09-21T09:40:49.096078+0000","last_unstale":"2024-09-21T09:41:31.633457+0000","last_undegraded":"2024-09-21T09:41:31.633457+0000","last_fullsized":"2024-09-21T09:41:31.633457+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:56.500555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1552,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635049+0000","last_change":"2024-09-21T09:40:50.112032+0000","last_active":"2024-09-21T09:41:31.635049+0000","last_peered":"2024-09-21T09:41:31.635049+0000","last_clean":"2024-09-21T09:41:31.635049+0000","last_became_active":"2024-09-21T09:40:50.111707+0000","last_became_peered":"2024-09-21T09:40:50.111707+0000","last_unstale":"2024-09-21T09:41:31.635049+0000","last_undegraded":"2024-09-21T09:41:31.635049+0000","last_fullsized":"2024-09-21T09:41:31.635049+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:38:21.339598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":17,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299829+0000","last_change":"2024-09-21T09:40:49.104195+0000","last_active":"2024-09-21T09:41:32.299829+0000","last_peered":"2024-09-21T09:41:32.299829+0000","last_clean":"2024-09-21T09:41:32.299829+0000","last_became_active":"2024-09-21T09:40:49.103940+0000","last_became_peered":"2024-09-21T09:40:49.103940+0000","last_unstale":"2024-09-21T09:41:32.299829+0000","last_undegraded":"2024-09-21T09:41:32.299829+0000","last_fullsized":"2024-09-21T09:41:32.299829+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:33:18.339902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635520+0000","last_change":"2024-09-21T09:40:49.097293+0000","last_active":"2024-09-21T09:41:31.635520+0000","last_peered":"2024-09-21T09:41:31.635520+0000","last_clean":"2024-09-21T09:41:31.635520+0000","last_became_active":"2024-09-21T09:40:49.097030+0000","last_became_peered":"2024-09-21T09:40:49.097030+0000","last_unstale":"2024-09-21T09:41:31.635520+0000","last_undegraded":"2024-09-21T09:41:31.635520+0000","last_fullsized":"2024-09-21T09:41:31.635520+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:38.667234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299409+0000","last_change":"2024-09-21T09:40:50.110750+0000","last_active":"2024-09-21T09:41:32.299409+0000","last_peered":"2024-09-21T09:41:32.299409+0000","last_clean":"2024-09-21T09:41:32.299409+0000","last_became_active":"2024-09-21T09:40:50.110512+0000","last_became_peered":"2024-09-21T09:40:50.110512+0000","last_unstale":"2024-09-21T09:41:32.299409+0000","last_undegraded":"2024-09-21T09:41:32.299409+0000","last_fullsized":"2024-09-21T09:41:32.299409+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:45:44.520662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637872+0000","last_change":"2024-09-21T09:40:49.099387+0000","last_active":"2024-09-21T09:41:31.637872+0000","last_peered":"2024-09-21T09:41:31.637872+0000","last_clean":"2024-09-21T09:41:31.637872+0000","last_became_active":"2024-09-21T09:40:49.099129+0000","last_became_peered":"2024-09-21T09:40:49.099129+0000","last_unstale":"2024-09-21T09:41:31.637872+0000","last_undegraded":"2024-09-21T09:41:31.637872+0000","last_fullsized":"2024-09-21T09:41:31.637872+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:55:59.770545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299252+0000","last_change":"2024-09-21T09:40:50.113290+0000","last_active":"2024-09-21T09:41:32.299252+0000","last_peered":"2024-09-21T09:41:32.299252+0000","last_clean":"2024-09-21T09:41:32.299252+0000","last_became_active":"2024-09-21T09:40:50.113029+0000","last_became_peered":"2024-09-21T09:40:50.113029+0000","last_unstale":"2024-09-21T09:41:32.299252+0000","last_undegraded":"2024-09-21T09:41:32.299252+0000","last_fullsized":"2024-09-21T09:41:32.299252+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:59:41.871996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.636157+0000","last_change":"2024-09-21T09:40:50.501575+0000","last_active":"2024-09-21T09:41:31.636157+0000","last_peered":"2024-09-21T09:41:31.636157+0000","last_clean":"2024-09-21T09:41:31.636157+0000","last_became_active":"2024-09-21T09:40:50.501238+0000","last_became_peered":"2024-09-21T09:40:50.501238+0000","last_unstale":"2024-09-21T09:41:31.636157+0000","last_undegraded":"2024-09-21T09:41:31.636157+0000","last_fullsized":"2024-09-21T09:41:31.636157+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:37.454625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299338+0000","last_change":"2024-09-21T09:40:49.091638+0000","last_active":"2024-09-21T09:41:32.299338+0000","last_peered":"2024-09-21T09:41:32.299338+0000","last_clean":"2024-09-21T09:41:32.299338+0000","last_became_active":"2024-09-21T09:40:49.091305+0000","last_became_peered":"2024-09-21T09:40:49.091305+0000","last_unstale":"2024-09-21T09:41:32.299338+0000","last_undegraded":"2024-09-21T09:41:32.299338+0000","last_fullsized":"2024-09-21T09:41:32.299338+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:09.684078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637487+0000","last_change":"2024-09-21T09:40:50.113606+0000","last_active":"2024-09-21T09:41:31.637487+0000","last_peered":"2024-09-21T09:41:31.637487+0000","last_clean":"2024-09-21T09:41:31.637487+0000","last_became_active":"2024-09-21T09:40:50.113236+0000","last_became_peered":"2024-09-21T09:40:50.113236+0000","last_unstale":"2024-09-21T09:41:31.637487+0000","last_undegraded":"2024-09-21T09:41:31.637487+0000","last_fullsized":"2024-09-21T09:41:31.637487+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:33:52.693638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635038+0000","last_change":"2024-09-21T09:40:49.095954+0000","last_active":"2024-09-21T09:41:31.635038+0000","last_peered":"2024-09-21T09:41:31.635038+0000","last_clean":"2024-09-21T09:41:31.635038+0000","last_became_active":"2024-09-21T09:40:49.095641+0000","last_became_peered":"2024-09-21T09:40:49.095641+0000","last_unstale":"2024-09-21T09:41:31.635038+0000","last_undegraded":"2024-09-21T09:41:31.635038+0000","last_fullsized":"2024-09-21T09:41:31.635038+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:39:21.262097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299538+0000","last_change":"2024-09-21T09:40:50.116266+0000","last_active":"2024-09-21T09:41:32.299538+0000","last_peered":"2024-09-21T09:41:32.299538+0000","last_clean":"2024-09-21T09:41:32.299538+0000","last_became_active":"2024-09-21T09:40:50.115989+0000","last_became_peered":"2024-09-21T09:40:50.115989+0000","last_unstale":"2024-09-21T09:41:32.299538+0000","last_undegraded":"2024-09-21T09:41:32.299538+0000","last_fullsized":"2024-09-21T09:41:32.299538+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:36.909000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.633304+0000","last_change":"2024-09-21T09:40:49.099618+0000","last_active":"2024-09-21T09:41:31.633304+0000","last_peered":"2024-09-21T09:41:31.633304+0000","last_clean":"2024-09-21T09:41:31.633304+0000","last_became_active":"2024-09-21T09:40:49.099133+0000","last_became_peered":"2024-09-21T09:40:49.099133+0000","last_unstale":"2024-09-21T09:41:31.633304+0000","last_undegraded":"2024-09-21T09:41:31.633304+0000","last_fullsized":"2024-09-21T09:41:31.633304+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:06:43.237147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.631561+0000","last_change":"2024-09-21T09:40:50.113409+0000","last_active":"2024-09-21T09:41:31.631561+0000","last_peered":"2024-09-21T09:41:31.631561+0000","last_clean":"2024-09-21T09:41:31.631561+0000","last_became_active":"2024-09-21T09:40:50.113150+0000","last_became_peered":"2024-09-21T09:40:50.113150+0000","last_unstale":"2024-09-21T09:41:31.631561+0000","last_undegraded":"2024-09-21T09:41:31.631561+0000","last_fullsized":"2024-09-21T09:41:31.631561+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:56.266738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637552+0000","last_change":"2024-09-21T09:40:49.100726+0000","last_active":"2024-09-21T09:41:31.637552+0000","last_peered":"2024-09-21T09:41:31.637552+0000","last_clean":"2024-09-21T09:41:31.637552+0000","last_became_active":"2024-09-21T09:40:49.100332+0000","last_became_peered":"2024-09-21T09:40:49.100332+0000","last_unstale":"2024-09-21T09:41:31.637552+0000","last_undegraded":"2024-09-21T09:41:31.637552+0000","last_fullsized":"2024-09-21T09:41:31.637552+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:12:54.280752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.633222+0000","last_change":"2024-09-21T09:40:49.358329+0000","last_active":"2024-09-21T09:41:31.633222+0000","last_peered":"2024-09-21T09:41:31.633222+0000","last_clean":"2024-09-21T09:41:31.633222+0000","last_became_active":"2024-09-21T09:40:49.358049+0000","last_became_peered":"2024-09-21T09:40:49.358049+0000","last_unstale":"2024-09-21T09:41:31.633222+0000","last_undegraded":"2024-09-21T09:41:31.633222+0000","last_fullsized":"2024-09-21T09:41:31.633222+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:55.412979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.635800+0000","last_change":"2024-09-21T09:40:49.096426+0000","last_active":"2024-09-21T09:41:31.635800+0000","last_peered":"2024-09-21T09:41:31.635800+0000","last_clean":"2024-09-21T09:41:31.635800+0000","last_became_active":"2024-09-21T09:40:49.096167+0000","last_became_peered":"2024-09-21T09:40:49.096167+0000","last_unstale":"2024-09-21T09:41:31.635800+0000","last_undegraded":"2024-09-21T09:41:31.635800+0000","last_fullsized":"2024-09-21T09:41:31.635800+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:15:47.077072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.633089+0000","last_change":"2024-09-21T09:40:49.357760+0000","last_active":"2024-09-21T09:41:31.633089+0000","last_peered":"2024-09-21T09:41:31.633089+0000","last_clean":"2024-09-21T09:41:31.633089+0000","last_became_active":"2024-09-21T09:40:49.357530+0000","last_became_peered":"2024-09-21T09:40:49.357530+0000","last_unstale":"2024-09-21T09:41:31.633089+0000","last_undegraded":"2024-09-21T09:41:31.633089+0000","last_fullsized":"2024-09-21T09:41:31.633089+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:25.545361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":16,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.636329+0000","last_change":"2024-09-21T09:40:49.094916+0000","last_active":"2024-09-21T09:41:31.636329+0000","last_peered":"2024-09-21T09:41:31.636329+0000","last_clean":"2024-09-21T09:41:31.636329+0000","last_became_active":"2024-09-21T09:40:49.094264+0000","last_became_peered":"2024-09-21T09:40:49.094264+0000","last_unstale":"2024-09-21T09:41:31.636329+0000","last_undegraded":"2024-09-21T09:41:31.636329+0000","last_fullsized":"2024-09-21T09:41:31.636329+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:53.663226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299431+0000","last_change":"2024-09-21T09:40:50.109643+0000","last_active":"2024-09-21T09:41:32.299431+0000","last_peered":"2024-09-21T09:41:32.299431+0000","last_clean":"2024-09-21T09:41:32.299431+0000","last_became_active":"2024-09-21T09:40:50.109409+0000","last_became_peered":"2024-09-21T09:40:50.109409+0000","last_unstale":"2024-09-21T09:41:32.299431+0000","last_undegraded":"2024-09-21T09:41:32.299431+0000","last_fullsized":"2024-09-21T09:41:32.299431+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:26:30.062747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":15,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:32.299613+0000","last_change":"2024-09-21T09:40:49.100579+0000","last_active":"2024-09-21T09:41:32.299613+0000","last_peered":"2024-09-21T09:41:32.299613+0000","last_clean":"2024-09-21T09:41:32.299613+0000","last_became_active":"2024-09-21T09:40:49.100281+0000","last_became_peered":"2024-09-21T09:40:49.100281+0000","last_unstale":"2024-09-21T09:41:32.299613+0000","last_undegraded":"2024-09-21T09:41:32.299613+0000","last_fullsized":"2024-09-21T09:41:32.299613+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:38:59.424535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.632891+0000","last_change":"2024-09-21T09:40:49.343171+0000","last_active":"2024-09-21T09:41:31.632891+0000","last_peered":"2024-09-21T09:41:31.632891+0000","last_clean":"2024-09-21T09:41:31.632891+0000","last_became_active":"2024-09-21T09:40:49.342795+0000","last_became_peered":"2024-09-21T09:40:49.342795+0000","last_unstale":"2024-09-21T09:41:31.632891+0000","last_undegraded":"2024-09-21T09:41:31.632891+0000","last_fullsized":"2024-09-21T09:41:31.632891+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:02.522569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":18,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-21T09:41:31.637876+0000","last_change":"2024-09-21T09:40:49.097888+0000","last_active":"2024-09-21T09:41:31.637876+0000","last_peered":"2024-09-21T09:41:31.637876+0000","last_clean":"2024-09-21T09:41:31.637876+0000","last_became_active":"2024-09-21T09:40:49.097597+0000","last_became_peered":"2024-09-21T09:40:49.097597+0000","last_unstale":"2024-09-21T09:41:31.637876+0000","last_undegraded":"2024-09-21T09:41:31.637876+0000","last_fullsized":"2024-09-21T09:41:31.637876+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:29.096456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2338,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430244,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298616,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446536,"statfs":{"total":95995035648,"available":95689252864,"internally_reserved":0,"allocated":1286144,"data_stored":885824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:41:03 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.48499999999999999,"15min":0.48299999999999998},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.68899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.48099999999999998,"15min":0.47199999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.56100000000000005}]},{"osd":1,"last update":"Sat Sep 21 09:41:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.499,"15min":0.496},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.38600000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.53300000000000003}]},{"osd":2,"last update":"Sat Sep 21 09:41:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64200000000000002,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.50900000000000001,"15min":0.50600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.48099999999999998}]},{"osd":3,"last update":"Sat Sep 21 09:41:03 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.48199999999999998,"15min":0.47899999999999998},"min":{"1min":0.376,"5min":0.371,"15min":0.371},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.442},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.53600000000000003,"15min":0.53400000000000003},"min":{"1min":0.39900000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.50600000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:41:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.442,"15min":0.436},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.64900000000000002,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.434,"15min":0.43099999999999999},"min":{"1min":0.28499999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.52100000000000002}]},{"osd":5,"last update":"Sat Sep 21 09:41:03 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.46100000000000002,"15min":0.45900000000000002},"min":{"1min":0.33800000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.374},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.48699999999999999,"15min":0.48199999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71799999999999997,"5min":0.748,"15min":0.748},"last":0.46400000000000002}]},{"osd":6,"last update":"Sat Sep 21 09:41:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.496,"15min":0.49099999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.499,"15min":0.495},"min":{"1min":0.35499999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.67200000000000004,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.39200000000000002}]}]},{"osd":6,"up_from":42,"seq":180388626472,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298608,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446544,"statfs":{"total":95995035648,"available":95689261056,"internally_reserved":0,"allocated":1277952,"data_stored":885697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.46999999999999997,"15min":0.46600000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.50600000000000001,"15min":0.504},"min":{"1min":0.35599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.40200000000000002}]},{"osd":1,"last update":"Sat Sep 21 09:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.50700000000000001,"15min":0.501},"min":{"1min":0.32200000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.40899999999999997}]},{"osd":2,"last update":"Sat Sep 21 09:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.50600000000000001,"15min":0.5},"min":{"1min":0.38,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.73099999999999998,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.504},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.51400000000000001,"15min":0.51000000000000001},"min":{"1min":0.35599999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65600000000000003,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.38600000000000001}]},{"osd":3,"last update":"Sat Sep 21 09:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.52600000000000002,"15min":0.52400000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.67600000000000005,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.52500000000000002,"15min":0.51900000000000002},"min":{"1min":0.38500000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.56599999999999995}]},{"osd":4,"last update":"Sat Sep 21 09:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.40899999999999997,"15min":0.40200000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.44500000000000001,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.44500000000000001}]},{"osd":5,"last update":"Sat Sep 21 09:40:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.442,"15min":0.42899999999999999},"min":{"1min":0.40699999999999997,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.46200000000000002,"15min":0.45200000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.52200000000000002}]},{"osd":7,"last update":"Sat Sep 21 09:41:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.505,"15min":0.504},"min":{"1min":0.35099999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.71599999999999997,"5min":0.752,"15min":0.752},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.57499999999999996,"15min":0.57199999999999995},"min":{"1min":0.48199999999999998,"5min":0.432,"15min":0.432},"max":{"1min":0.79300000000000004,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.54600000000000004}]}]},{"osd":5,"up_from":36,"seq":154618822701,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298008,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447144,"statfs":{"total":95995035648,"available":95689875456,"internally_reserved":0,"allocated":667648,"data_stored":293717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:41:23 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51100000000000001,"15min":0.52300000000000002},"min":{"1min":0.35799999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64500000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.495,"5min":0.504,"15min":0.503},"min":{"1min":0.34100000000000003,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68100000000000005,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.57399999999999995}]},{"osd":1,"last update":"Sat Sep 21 09:41:23 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.51400000000000001,"15min":0.52200000000000002},"min":{"1min":0.35199999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.66200000000000003,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.625},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.52600000000000002,"15min":0.52500000000000002},"min":{"1min":0.39500000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69799999999999995,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.48599999999999999}]},{"osd":2,"last update":"Sat Sep 21 09:41:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51900000000000002,"15min":0.52900000000000003},"min":{"1min":0.34200000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.622,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53200000000000003,"15min":0.52500000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77600000000000002,"5min":1.054,"15min":1.054},"last":0.66500000000000004}]},{"osd":3,"last update":"Sat Sep 21 09:41:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.375,"5min":0.311,"15min":0.311},"max":{"1min":0.69599999999999995,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.53300000000000003,"15min":0.54100000000000004},"min":{"1min":0.316,"5min":0.309,"15min":0.309},"max":{"1min":0.72399999999999998,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.51500000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:41:23 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.40200000000000002,"15min":0.38},"min":{"1min":0.24199999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.71099999999999997,"5min":1.004,"15min":1.004},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.41299999999999998,"15min":0.39900000000000002},"min":{"1min":0.25700000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.74099999999999999,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.36699999999999999}]},{"osd":6,"last update":"Sat Sep 21 09:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.27300000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67100000000000004,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.48399999999999999,"15min":0.49299999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67600000000000005,"5min":1.036,"15min":1.036},"last":0.435}]},{"osd":7,"last update":"Sat Sep 21 09:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.52100000000000002,"15min":0.52700000000000002},"min":{"1min":0.32300000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76100000000000001,"5min":1.107,"15min":1.107},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.54600000000000004,"15min":0.55400000000000005},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.745,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.54300000000000004}]}]},{"osd":4,"up_from":30,"seq":128849018930,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298024,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447128,"statfs":{"total":95995035648,"available":95689859072,"internally_reserved":0,"allocated":688128,"data_stored":295386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:40:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53600000000000003,"15min":0.54600000000000004},"min":{"1min":0.34599999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68600000000000005,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54700000000000004,"15min":0.55600000000000005},"min":{"1min":0.33800000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.69799999999999995}]},{"osd":1,"last update":"Sat Sep 21 09:40:58 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.50900000000000001,"15min":0.52500000000000002},"min":{"1min":0.29299999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73699999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.503,"15min":0.504},"min":{"1min":0.32300000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.55500000000000005}]},{"osd":2,"last update":"Sat Sep 21 09:40:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52500000000000002,"15min":0.53600000000000003},"min":{"1min":0.311,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52800000000000002,"15min":0.54000000000000004},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.74299999999999999,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.63600000000000001}]},{"osd":3,"last update":"Sat Sep 21 09:40:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53400000000000003,"15min":0.54500000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.67400000000000004,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.53300000000000003,"15min":0.54500000000000004},"min":{"1min":0.32500000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.72499999999999998}]},{"osd":5,"last update":"Sat Sep 21 09:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.38700000000000001},"min":{"1min":0.29999999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.68600000000000005,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.44800000000000001,"15min":0.42899999999999999},"min":{"1min":0.312,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.75800000000000001,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.57299999999999995}]},{"osd":6,"last update":"Sat Sep 21 09:40:44 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.40600000000000003,"15min":0.40000000000000002},"min":{"1min":0.27000000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.623,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.42299999999999999,"15min":0.41899999999999998},"min":{"1min":0.255,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.71599999999999997,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.61599999999999999}]},{"osd":7,"last update":"Sat Sep 21 09:41:03 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47999999999999998,"15min":0.48199999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.74299999999999999,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.65000000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298008,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447144,"statfs":{"total":95995035648,"available":95689875456,"internally_reserved":0,"allocated":667648,"data_stored":293778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.35799999999999998,"15min":0.38200000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.48499999999999999,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.376,"15min":0.40200000000000002},"min":{"1min":0.254,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.45200000000000001,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.45800000000000002}]},{"osd":1,"last update":"Sat Sep 21 09:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.37,"15min":0.39200000000000002},"min":{"1min":0.251,"5min":0.22,"15min":0.22},"max":{"1min":0.46100000000000002,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.40799999999999997,"15min":0.432},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.54300000000000004,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.54300000000000004}]},{"osd":2,"last update":"Sat Sep 21 09:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.43099999999999999,"15min":0.45700000000000002},"min":{"1min":0.27200000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.57199999999999995,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.40600000000000003,"15min":0.42999999999999999},"min":{"1min":0.26400000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.49099999999999999,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.496}]},{"osd":4,"last update":"Sat Sep 21 09:40:57 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.56699999999999995,"15min":0.57299999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55700000000000005,"15min":0.55500000000000005},"min":{"1min":0.42099999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.57599999999999996}]},{"osd":5,"last update":"Sat Sep 21 09:41:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.57499999999999996,"15min":0.58699999999999997},"min":{"1min":0.379,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.69899999999999995,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.57499999999999996,"15min":0.58699999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.59899999999999998}]},{"osd":6,"last update":"Sat Sep 21 09:41:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58399999999999996,"15min":0.59399999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.74299999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.59999999999999998,"15min":0.61899999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.69399999999999995}]},{"osd":7,"last update":"Sat Sep 21 09:41:09 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.59099999999999997,"15min":0.59399999999999997},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.66600000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298012,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447140,"statfs":{"total":95995035648,"available":95689871360,"internally_reserved":0,"allocated":671744,"data_stored":294292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.34300000000000003,"15min":0.34100000000000003},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.622,"5min":0.622,"15min":0.622},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.315,"5min":0.35499999999999998,"15min":0.35999999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.59699999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.26600000000000001}]},{"osd":1,"last update":"Sat Sep 21 09:41:18 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.35299999999999998,"15min":0.35899999999999999},"min":{"1min":0.218,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38300000000000001,"15min":0.38500000000000001},"min":{"1min":0.28299999999999997,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.35199999999999998}]},{"osd":3,"last update":"Sat Sep 21 09:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.41299999999999998,"15min":0.43099999999999999},"min":{"1min":0.29099999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.39100000000000001,"15min":0.41599999999999998},"min":{"1min":0.26700000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.32100000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:40:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.55300000000000005,"15min":0.56100000000000005},"min":{"1min":0.38600000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83599999999999997,"5min":0.878,"15min":0.878},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56100000000000005,"15min":0.56699999999999995},"min":{"1min":0.40400000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.49199999999999999}]},{"osd":5,"last update":"Sat Sep 21 09:41:25 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.58099999999999996,"15min":0.58599999999999997},"min":{"1min":0.39400000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56000000000000005,"15min":0.56599999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.58799999999999997}]},{"osd":6,"last update":"Sat Sep 21 09:40:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.376,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.5},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58599999999999997,"15min":0.58999999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.73499999999999999,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53500000000000003}]},{"osd":7,"last update":"Sat Sep 21 09:41:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.68000000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74199999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.52300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298004,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447148,"statfs":{"total":95995035648,"available":95689879552,"internally_reserved":0,"allocated":663552,"data_stored":293392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:40:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.33700000000000002,"15min":0.315},"min":{"1min":0.224,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.34100000000000003,"15min":0.309},"min":{"1min":0.25600000000000001,"5min":0.184,"15min":0.184},"max":{"1min":0.56699999999999995,"5min":0.627,"15min":0.627},"last":0.26800000000000002}]},{"osd":2,"last update":"Sat Sep 21 09:41:22 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.34799999999999998,"15min":0.32800000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.502,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.253},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.377,"15min":0.35899999999999999},"min":{"1min":0.28100000000000003,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.28699999999999998}]},{"osd":3,"last update":"Sat Sep 21 09:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.40500000000000003,"15min":0.42299999999999999},"min":{"1min":0.26600000000000001,"5min":0.245,"15min":0.245},"max":{"1min":0.56000000000000005,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.41699999999999998,"15min":0.433},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.52200000000000002,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.32200000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:41:02 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.53300000000000003,"15min":0.54200000000000004},"min":{"1min":0.38100000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.68600000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.39200000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.79400000000000004,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.56699999999999995}]},{"osd":5,"last update":"Sat Sep 21 09:41:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.54900000000000004,"15min":0.54500000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54900000000000004,"15min":0.55500000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76000000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.61499999999999999}]},{"osd":6,"last update":"Sat Sep 21 09:41:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.58099999999999996,"15min":0.58599999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.58199999999999996,"15min":0.57399999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.58599999999999997}]},{"osd":7,"last update":"Sat Sep 21 09:41:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56699999999999995,"15min":0.57199999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.81100000000000005,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56299999999999994,"15min":0.56799999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.747,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.625}]}]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298592,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446560,"statfs":{"total":95995035648,"available":95689277440,"internally_reserved":0,"allocated":1265664,"data_stored":884648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 09:40:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.34200000000000003,"15min":0.312},"min":{"1min":0.20599999999999999,"5min":0.187,"15min":0.187},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.378},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.34200000000000003,"15min":0.32000000000000001},"min":{"1min":0.219,"5min":0.188,"15min":0.188},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.29499999999999998}]},{"osd":2,"last update":"Sat Sep 21 09:41:14 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.36799999999999999,"15min":0.372},"min":{"1min":0.221,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.312},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.38800000000000001,"15min":0.38600000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.40999999999999998}]},{"osd":3,"last update":"Sat Sep 21 09:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.41399999999999998,"15min":0.42599999999999999},"min":{"1min":0.30399999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.60199999999999998,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39200000000000002,"15min":0.40500000000000003},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.54400000000000004,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.36099999999999999}]},{"osd":4,"last update":"Sat Sep 21 09:41:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53100000000000003,"15min":0.53300000000000003},"min":{"1min":0.38300000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.67700000000000005,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.39100000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.83999999999999997,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.57299999999999995}]},{"osd":5,"last update":"Sat Sep 21 09:41:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.42699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.55300000000000005,"15min":0.56499999999999995},"min":{"1min":0.36199999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.70199999999999996,"5min":0.87,"15min":0.87},"last":0.54200000000000004}]},{"osd":6,"last update":"Sat Sep 21 09:40:48 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55000000000000004,"15min":0.55400000000000005},"min":{"1min":0.373,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71499999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.623}]},{"osd":7,"last update":"Sat Sep 21 09:41:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54900000000000004,"15min":0.54400000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55700000000000005,"15min":0.55200000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.69899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T09:41:51.307 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T09:41:51.307 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T09:41:51.307 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T09:41:51.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph health --format=json 2024-09-21T09:41:51.551 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:52.659 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:52 smithi107 ceph-mon[29382]: from='client.24683 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:41:52.660 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:52 smithi107 ceph-mon[29382]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:52.660 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:52 smithi107 ceph-mon[41185]: from='client.24683 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:41:52.660 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:52 smithi107 ceph-mon[41185]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:52.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:52 smithi129 ceph-mon[32729]: from='client.24683 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:41:52.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:52 smithi129 ceph-mon[32729]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:53.290 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:41:53.291 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T09:41:53.591 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:53 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/38185660' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:41:53.591 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:53 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/38185660' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:41:53.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:53 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/38185660' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:41:54.006 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T09:41:54.007 INFO:teuthology.run_tasks:Running task print... 2024-09-21T09:41:54.016 INFO:teuthology.task.print:**** upgrading first half of cluster, with stress **** 2024-09-21T09:41:54.016 INFO:teuthology.run_tasks:Running task parallel... 2024-09-21T09:41:54.044 INFO:teuthology.task.parallel:starting parallel... 2024-09-21T09:41:54.044 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-21T09:41:54.044 INFO:teuthology.task.sequential:In sequential, running task full_sequential... 2024-09-21T09:41:54.045 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T09:41:54.046 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T09:41:54.047 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-21T09:41:54.047 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_0 16 2024-09-21T09:41:54.047 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-21T09:41:54.047 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-09-21T09:41:54.048 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi107.front.sepia.ceph.com 2024-09-21T09:41:54.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph config set mgr mgr/cephadm/daemon_cache_timeout 60' 2024-09-21T09:41:54.323 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:54.323 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:54.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:54 smithi129 ceph-mon[32729]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:54.695 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:54 smithi107 ceph-mon[29382]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:54.695 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:54 smithi107 ceph-mon[41185]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:56.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:56 smithi129 ceph-mon[32729]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:56.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:56 smithi107 ceph-mon[29382]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:56.792 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:56 smithi107 ceph-mon[41185]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:57.502 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_0' created 2024-09-21T09:41:57.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:57 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/736166658' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-21T09:41:57.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:57 smithi129 ceph-mon[32729]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-21T09:41:57.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:41:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:41:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:41:57.725 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:57 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/736166658' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-21T09:41:57.726 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:57 smithi107 ceph-mon[29382]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-21T09:41:57.726 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:57 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/736166658' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-21T09:41:57.726 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:57 smithi107 ceph-mon[41185]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-21T09:41:57.726 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:41:57 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:41:57] ENGINE Bus STOPPING 2024-09-21T09:41:58.026 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:41:57 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:41:57] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:41:58.026 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:41:57 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:41:57] ENGINE Bus STOPPED 2024-09-21T09:41:58.026 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:41:57 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:41:57] ENGINE Bus STARTING 2024-09-21T09:41:58.026 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:41:57 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:41:57] ENGINE Serving on http://:::9283 2024-09-21T09:41:58.027 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:41:57 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:41:57] ENGINE Bus STARTED 2024-09-21T09:41:58.452 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph config set global log_to_journald false --force' 2024-09-21T09:41:58.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:58 smithi107 ceph-mon[29382]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:58.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:58 smithi107 ceph-mon[29382]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T09:41:58.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:58 smithi107 ceph-mon[29382]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T09:41:58.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:58 smithi107 ceph-mon[29382]: from='client.? ' entity='client.admin' 2024-09-21T09:41:58.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:58 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:41:58.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:58 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:41:58.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:58 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:41:58.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:58 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:58.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:41:58 smithi107 ceph-mon[29382]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T09:41:58.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:58 smithi107 ceph-mon[41185]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:58.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:58 smithi107 ceph-mon[41185]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T09:41:58.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:58 smithi107 ceph-mon[41185]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T09:41:58.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:58 smithi107 ceph-mon[41185]: from='client.? ' entity='client.admin' 2024-09-21T09:41:58.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:58 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:41:58.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:58 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:41:58.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:58 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:41:58.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:58 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:58.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:41:58 smithi107 ceph-mon[41185]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T09:41:58.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-21T09:41:58.888 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:58 smithi129 ceph-mon[32729]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:41:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:58 smithi129 ceph-mon[32729]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T09:41:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:58 smithi129 ceph-mon[32729]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T09:41:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:58 smithi129 ceph-mon[32729]: from='client.? ' entity='client.admin' 2024-09-21T09:41:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:58 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:41:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:58 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:41:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:58 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:41:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:58 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:41:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:41:58 smithi129 ceph-mon[32729]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T09:41:59.584 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:41:59.608 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:41:59 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:41:59] "GET /metrics HTTP/1.1" 200 239815 "" "Prometheus/2.43.0" 2024-09-21T09:42:00.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:00 smithi107 ceph-mon[29382]: pgmap v93: 89 pgs: 16 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:42:00.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:00 smithi107 ceph-mon[29382]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T09:42:00.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:00 smithi107 ceph-mon[29382]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:42:00.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:00 smithi107 ceph-mon[41185]: pgmap v93: 89 pgs: 16 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:42:00.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:00 smithi107 ceph-mon[41185]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T09:42:00.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:00 smithi107 ceph-mon[41185]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:42:00.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:00 smithi129 ceph-mon[32729]: pgmap v93: 89 pgs: 16 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:42:00.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:00 smithi129 ceph-mon[32729]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T09:42:00.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:00 smithi129 ceph-mon[32729]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:42:02.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:02 smithi107 ceph-mon[29382]: pgmap v95: 89 pgs: 16 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:42:02.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:02 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:42:02.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:02 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:42:02.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:02 smithi107 ceph-mon[41185]: pgmap v95: 89 pgs: 16 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:42:02.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:02 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:42:02.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:02 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:42:02.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:02 smithi129 ceph-mon[32729]: pgmap v95: 89 pgs: 16 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:42:02.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:02 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:42:02.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:02 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:42:02.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-09-21T09:42:03.053 INFO:teuthology.orchestra.run.smithi107.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-21T09:42:03.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:03 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3830615222' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:42:03.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:03 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3830615222' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:42:03.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:03 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3830615222' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:42:03.698 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:42:04.249 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:04 smithi107 ceph-mon[29382]: pgmap v96: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:42:04.250 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:04 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3830615222' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:42:04.250 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:04 smithi107 ceph-mon[29382]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T09:42:04.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:04 smithi107 ceph-mon[41185]: pgmap v96: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:42:04.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:04 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3830615222' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:42:04.250 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:04 smithi107 ceph-mon[41185]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T09:42:04.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:04 smithi129 ceph-mon[32729]: pgmap v96: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:42:04.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:04 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3830615222' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:42:04.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:04 smithi129 ceph-mon[32729]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T09:42:04.690 DEBUG:teuthology.orchestra.run.smithi107:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-21T09:42:04.692 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T09:42:04.692 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T09:42:04.755 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:hints = 1 2024-09-21T09:42:04.755 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T09:42:04.756 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object prefix: benchmark_data_smithi107_85726 2024-09-21T09:42:04.756 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T09:42:04.756 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 0 0 0 0 0 0 - 0 2024-09-21T09:42:05.273 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:05 smithi107 ceph-mon[29382]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:42:05.274 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:05 smithi107 ceph-mon[29382]: Cluster is now healthy 2024-09-21T09:42:05.274 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:05 smithi107 ceph-mon[41185]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:42:05.274 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:05 smithi107 ceph-mon[41185]: Cluster is now healthy 2024-09-21T09:42:05.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:05 smithi129 ceph-mon[32729]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:42:05.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:05 smithi129 ceph-mon[32729]: Cluster is now healthy 2024-09-21T09:42:05.756 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1 16 4428 4412 275.73 275.75 0.00445964 0.00361482 2024-09-21T09:42:06.141 INFO:teuthology.orchestra.run.smithi107.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:42:06.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:06 smithi107 ceph-mon[29382]: pgmap v98: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:42:06.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:06 smithi107 ceph-mon[41185]: pgmap v98: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:42:06.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:06 smithi129 ceph-mon[32729]: pgmap v98: 89 pgs: 89 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:42:06.756 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2 16 8219 8203 256.323 236.938 0.00327049 0.00389464 2024-09-21T09:42:06.932 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch ps' 2024-09-21T09:42:07.405 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:42:07.418 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:42:07 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:42:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:42:07.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:07 smithi129 ceph-mon[32729]: from='client.24589 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:07.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:07 smithi129 ceph-mon[32729]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:42:07.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:07 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:42:07.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:07 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:42:07.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:07 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:42:07.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:07 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:42:07.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:07 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:42:07.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:07 smithi129 ceph-mon[32729]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:42:07.545 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:07 smithi107 ceph-mon[41185]: from='client.24589 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:07.545 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:07 smithi107 ceph-mon[41185]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:42:07.545 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:07 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:42:07.545 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:07 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:42:07.545 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:07 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:42:07.545 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:07 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:42:07.545 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:07 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:42:07.545 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:07 smithi107 ceph-mon[41185]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:42:07.545 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:07 smithi107 ceph-mon[29382]: from='client.24589 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:07.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:07 smithi107 ceph-mon[29382]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:42:07.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:07 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:42:07.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:07 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:42:07.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:07 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:42:07.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:07 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:42:07.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:07 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:42:07.546 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:07 smithi107 ceph-mon[29382]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:42:07.756 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3 16 11475 11459 238.671 203.5 0.00582327 0.0041809 2024-09-21T09:42:08.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:08 smithi129 ceph-mon[32729]: pgmap v99: 89 pgs: 89 active+clean; 569 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T09:42:08.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:08 smithi107 ceph-mon[29382]: pgmap v99: 89 pgs: 89 active+clean; 569 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T09:42:08.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:08 smithi107 ceph-mon[41185]: pgmap v99: 89 pgs: 89 active+clean; 569 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T09:42:08.756 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 4 16 14443 14427 225.376 185.5 0.00292287 0.00442914 2024-09-21T09:42:09.608 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:42:09 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:42:09] "GET /metrics HTTP/1.1" 200 242163 "" "Prometheus/2.43.0" 2024-09-21T09:42:09.757 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 5 16 16918 16902 211.237 154.688 0.00590563 0.00472767 2024-09-21T09:42:10.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:10 smithi107 ceph-mon[29382]: pgmap v100: 89 pgs: 89 active+clean; 569 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 60 MiB/s wr, 955 op/s 2024-09-21T09:42:10.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:10 smithi107 ceph-mon[41185]: pgmap v100: 89 pgs: 89 active+clean; 569 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 60 MiB/s wr, 955 op/s 2024-09-21T09:42:10.757 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 6 16 19927 19911 207.372 188.062 0.00427181 0.00481636 2024-09-21T09:42:10.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:10 smithi129 ceph-mon[32729]: pgmap v100: 89 pgs: 89 active+clean; 569 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 60 MiB/s wr, 955 op/s 2024-09-21T09:42:11.757 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 7 16 22645 22629 202.001 169.875 0.0070622 0.00494603 2024-09-21T09:42:11.763 INFO:teuthology.orchestra.run.smithi107.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T09:42:11.763 INFO:teuthology.orchestra.run.smithi107.stdout:alertmanager.a smithi107 *:9093,9094 running (86s) 24s ago 2m 19.8M - 0.25.0 c8568f914cd2 432a2da9f5f4 2024-09-21T09:42:11.763 INFO:teuthology.orchestra.run.smithi107.stdout:grafana.a smithi129 *:3000 running (2m) 43s ago 2m 85.3M - 9.4.7 954c08fa6188 70638332abb2 2024-09-21T09:42:11.763 INFO:teuthology.orchestra.run.smithi107.stdout:mds.foo.smithi107.sedied smithi107 running (70s) 24s ago 70s 16.4M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 83d74619fe7a 2024-09-21T09:42:11.763 INFO:teuthology.orchestra.run.smithi107.stdout:mds.foo.smithi129.zdzmqr smithi129 running (68s) 43s ago 68s 14.3M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 2133ce1b60f9 2024-09-21T09:42:11.763 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.x smithi129 *:8443 running (6m) 43s ago 6m 399M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 1f321f7d8eff 2024-09-21T09:42:11.763 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.y smithi107 *:9283,8765 running (8m) 24s ago 8m 444M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 5ab1b97eb209 2024-09-21T09:42:11.763 INFO:teuthology.orchestra.run.smithi107.stdout:mon.a smithi107 running (8m) 24s ago 8m 47.2M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 7135d1946aea 2024-09-21T09:42:11.763 INFO:teuthology.orchestra.run.smithi107.stdout:mon.b smithi129 running (6m) 43s ago 6m 39.9M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 589a4d4f8d6b 2024-09-21T09:42:11.763 INFO:teuthology.orchestra.run.smithi107.stdout:mon.c smithi107 running (6m) 24s ago 6m 38.6M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 4d41a86eb0a8 2024-09-21T09:42:11.763 INFO:teuthology.orchestra.run.smithi107.stdout:node-exporter.a smithi107 *:9100 running (119s) 24s ago 117s 15.3M - 1.5.0 0da6a335fe13 bbf92f500b2c 2024-09-21T09:42:11.763 INFO:teuthology.orchestra.run.smithi107.stdout:node-exporter.b smithi129 *:9100 running (114s) 43s ago 114s 8006k - 1.5.0 0da6a335fe13 a71c52ae1a69 2024-09-21T09:42:11.764 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 smithi107 running (5m) 24s ago 5m 42.9M 4015M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 96047e77ae78 2024-09-21T09:42:11.764 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1 smithi107 running (5m) 24s ago 5m 45.4M 4015M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 508718820c25 2024-09-21T09:42:11.764 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2 smithi107 running (5m) 24s ago 5m 42.2M 4015M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be ae11dfe6a988 2024-09-21T09:42:11.764 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3 smithi107 running (4m) 24s ago 4m 41.5M 4015M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be cf6bac271e88 2024-09-21T09:42:11.764 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4 smithi129 running (4m) 43s ago 4m 41.1M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 7c6ca82e1ac1 2024-09-21T09:42:11.764 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5 smithi129 running (4m) 43s ago 4m 39.5M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 9ae131a2ef13 2024-09-21T09:42:11.764 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6 smithi129 running (3m) 43s ago 3m 39.7M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be a5ccfa3e24c2 2024-09-21T09:42:11.764 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7 smithi129 running (3m) 43s ago 3m 40.8M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be a32c9bd56124 2024-09-21T09:42:11.764 INFO:teuthology.orchestra.run.smithi107.stdout:prometheus.a smithi129 *:9095 running (81s) 43s ago 2m 38.2M - 2.43.0 a07b618ecd1d aa11510fd428 2024-09-21T09:42:12.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'echo wait for minority of mons to upgrade' 2024-09-21T09:42:12.758 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 8 16 25274 25258 197.283 164.312 0.00500173 0.00506211 2024-09-21T09:42:12.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:12 smithi107 ceph-mon[29382]: pgmap v101: 89 pgs: 89 active+clean; 569 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 57 MiB/s wr, 909 op/s 2024-09-21T09:42:12.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:12 smithi107 ceph-mon[41185]: pgmap v101: 89 pgs: 89 active+clean; 569 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 57 MiB/s wr, 909 op/s 2024-09-21T09:42:12.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:12 smithi129 ceph-mon[32729]: pgmap v101: 89 pgs: 89 active+clean; 569 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 57 MiB/s wr, 909 op/s 2024-09-21T09:42:12.978 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:42:13.758 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 9 16 27409 27393 190.189 133.438 0.00814837 0.00525279 2024-09-21T09:42:13.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:13 smithi107 ceph-mon[41185]: from='client.14832 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:13.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:13 smithi107 ceph-mon[29382]: from='client.14832 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:13.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:13 smithi129 ceph-mon[32729]: from='client.14832 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:14.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:14 smithi107 ceph-mon[29382]: pgmap v102: 89 pgs: 89 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 146 MiB/s wr, 2.33k op/s 2024-09-21T09:42:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:14 smithi107 ceph-mon[41185]: pgmap v102: 89 pgs: 89 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 146 MiB/s wr, 2.33k op/s 2024-09-21T09:42:14.758 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 10 16 29820 29804 186.237 150.688 0.00673594 0.00536488 2024-09-21T09:42:14.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:14 smithi129 ceph-mon[32729]: pgmap v102: 89 pgs: 89 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 146 MiB/s wr, 2.33k op/s 2024-09-21T09:42:15.758 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 11 16 31678 31662 179.863 116.125 0.00276526 0.00555476 2024-09-21T09:42:16.758 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 12 16 34038 34022 177.165 147.5 0.00554448 0.00564027 2024-09-21T09:42:16.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:16 smithi107 ceph-mon[41185]: pgmap v103: 89 pgs: 89 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-21T09:42:16.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:16 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:42:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:16 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:42:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:16 smithi107 ceph-mon[29382]: pgmap v103: 89 pgs: 89 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-21T09:42:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:16 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:42:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:16 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:42:16.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:16 smithi129 ceph-mon[32729]: pgmap v103: 89 pgs: 89 active+clean; 1.4 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-21T09:42:16.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:16 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:42:16.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:16 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:42:17.538 INFO:teuthology.orchestra.run.smithi107.stdout:wait for minority of mons to upgrade 2024-09-21T09:42:17.569 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:42:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:42:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:42:17.759 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 13 16 36264 36248 174.232 139.125 0.0074495 0.00573422 2024-09-21T09:42:18.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'while ! ceph mon versions | grep $sha1 && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-21T09:42:18.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:18 smithi107 ceph-mon[29382]: pgmap v104: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 180 MiB/s wr, 2.88k op/s 2024-09-21T09:42:18.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:18 smithi107 ceph-mon[41185]: pgmap v104: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 180 MiB/s wr, 2.88k op/s 2024-09-21T09:42:18.759 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 14 16 38400 38384 171.322 133.5 0.0072358 0.00583181 2024-09-21T09:42:18.857 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:42:18.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:18 smithi129 ceph-mon[32729]: pgmap v104: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 180 MiB/s wr, 2.88k op/s 2024-09-21T09:42:19.585 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:42:19 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:42:19] "GET /metrics HTTP/1.1" 200 243079 "" "Prometheus/2.43.0" 2024-09-21T09:42:19.759 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 15 16 40517 40501 168.721 132.312 0.00583909 0.00592277 2024-09-21T09:42:20.759 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 16 16 42409 42393 165.566 118.25 0.0118804 0.00603544 2024-09-21T09:42:20.823 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:20 smithi107 ceph-mon[29382]: pgmap v105: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 133 MiB/s wr, 2.12k op/s 2024-09-21T09:42:20.823 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:20 smithi107 ceph-mon[41185]: pgmap v105: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 133 MiB/s wr, 2.12k op/s 2024-09-21T09:42:20.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:20 smithi129 ceph-mon[32729]: pgmap v105: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 133 MiB/s wr, 2.12k op/s 2024-09-21T09:42:21.759 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 17 16 44333 44317 162.9 120.25 0.00509004 0.00613309 2024-09-21T09:42:22.326 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:42:22.759 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 18 16 45915 45899 159.343 98.875 0.00244094 0.00626929 2024-09-21T09:42:22.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:22 smithi107 ceph-mon[29382]: pgmap v106: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 133 MiB/s wr, 2.12k op/s 2024-09-21T09:42:22.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:22 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1681243185' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:22.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:22 smithi107 ceph-mon[41185]: pgmap v106: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 133 MiB/s wr, 2.12k op/s 2024-09-21T09:42:22.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:22 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1681243185' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:22.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:22 smithi129 ceph-mon[32729]: pgmap v106: 89 pgs: 89 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 133 MiB/s wr, 2.12k op/s 2024-09-21T09:42:22.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:22 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1681243185' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:23.759 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 19 16 47573 47557 156.41 103.625 0.0166942 0.00638839 2024-09-21T09:42:23.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:23 smithi107 ceph-mon[29382]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:23 smithi107 ceph-mon[41185]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:23.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:23 smithi129 ceph-mon[32729]: from='client.14835 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:24.526 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:24 smithi107 ceph-mon[29382]: from='client.24749 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:24.759 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T09:42:24.758682+0000 min lat: 0.0012353 max lat: 0.113387 avg lat: 0.0064716 2024-09-21T09:42:24.759 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T09:42:24.759 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 20 16 49434 49418 154.405 116.312 0.00580998 0.0064716 2024-09-21T09:42:24.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:24 smithi107 ceph-mon[29382]: pgmap v107: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 186 MiB/s wr, 2.97k op/s 2024-09-21T09:42:24.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:24 smithi107 ceph-mon[41185]: from='client.24749 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:24.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:24 smithi107 ceph-mon[41185]: pgmap v107: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 186 MiB/s wr, 2.97k op/s 2024-09-21T09:42:24.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:24 smithi129 ceph-mon[32729]: from='client.24749 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:24.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:24 smithi129 ceph-mon[32729]: pgmap v107: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 186 MiB/s wr, 2.97k op/s 2024-09-21T09:42:25.760 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 21 16 51153 51137 152.169 107.438 0.0084739 0.00656704 2024-09-21T09:42:25.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:25 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3615612463' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:25 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3615612463' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:25.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:25 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3615612463' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:25.968 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:42:26.759 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 22 16 52870 52854 150.129 107.312 0.00760641 0.00665667 2024-09-21T09:42:26.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:26 smithi107 ceph-mon[29382]: pgmap v108: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-21T09:42:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:26 smithi107 ceph-mon[41185]: pgmap v108: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-21T09:42:26.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:26 smithi129 ceph-mon[32729]: pgmap v108: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-21T09:42:27.557 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:42:27 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:42:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:42:27.760 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 23 16 54600 54584 148.303 108.125 0.0102855 0.00673854 2024-09-21T09:42:27.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:27 smithi107 ceph-mon[29382]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:27.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:27 smithi107 ceph-mon[29382]: from='client.24758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:27.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:27 smithi107 ceph-mon[41185]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:27.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:27 smithi107 ceph-mon[41185]: from='client.24758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:27.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:27 smithi129 ceph-mon[32729]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:27 smithi129 ceph-mon[32729]: from='client.24758 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:28.760 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 24 16 56345 56329 146.663 109.062 0.00710611 0.00681411 2024-09-21T09:42:28.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:28 smithi107 ceph-mon[29382]: pgmap v109: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 156 MiB/s wr, 2.50k op/s 2024-09-21T09:42:28.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:28 smithi107 ceph-mon[41185]: pgmap v109: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 156 MiB/s wr, 2.50k op/s 2024-09-21T09:42:28.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:28 smithi129 ceph-mon[32729]: pgmap v109: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 156 MiB/s wr, 2.50k op/s 2024-09-21T09:42:29.572 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:42:29.573 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:42:29 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:42:29] "GET /metrics HTTP/1.1" 200 243079 "" "Prometheus/2.43.0" 2024-09-21T09:42:29.761 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 25 16 57804 57788 144.442 91.1875 0.0182775 0.00691814 2024-09-21T09:42:29.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:29 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2880153423' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:29.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:29 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/2880153423' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:29.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:29 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2880153423' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:30.761 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 26 16 59410 59394 142.747 100.375 0.00541594 0.00700112 2024-09-21T09:42:30.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:30 smithi107 ceph-mon[29382]: pgmap v110: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-21T09:42:30.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:30 smithi107 ceph-mon[29382]: from='client.24767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:30 smithi107 ceph-mon[41185]: pgmap v110: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-21T09:42:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:30 smithi107 ceph-mon[41185]: from='client.24767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:30.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:30 smithi129 ceph-mon[32729]: pgmap v110: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-21T09:42:30.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:30 smithi129 ceph-mon[32729]: from='client.24767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:31.761 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 27 16 61012 60996 141.168 100.125 0.00492083 0.00707907 2024-09-21T09:42:31.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:31 smithi107 ceph-mon[29382]: from='client.24619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:31.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:42:31.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:31 smithi107 ceph-mon[41185]: from='client.24619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:31.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:42:31.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:31 smithi129 ceph-mon[32729]: from='client.24619 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:42:32.761 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 28 16 62464 62448 139.365 90.75 0.0174169 0.00717077 2024-09-21T09:42:32.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:32 smithi107 ceph-mon[41185]: pgmap v111: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-21T09:42:32.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:32 smithi107 ceph-mon[29382]: pgmap v111: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-21T09:42:32.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:32 smithi129 ceph-mon[32729]: pgmap v111: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-21T09:42:33.210 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:42:33.761 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 29 16 63794 63778 137.426 83.125 0.0101712 0.00727248 2024-09-21T09:42:33.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:33 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3002218416' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:33.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:33 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3002218416' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:33.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:33 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3002218416' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:34.762 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 30 16 65310 65294 136.002 94.75 0.0102288 0.00734807 2024-09-21T09:42:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:34 smithi129 ceph-mon[32729]: pgmap v112: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-09-21T09:42:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:34 smithi129 ceph-mon[32729]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:35.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:34 smithi107 ceph-mon[29382]: pgmap v112: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-09-21T09:42:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:34 smithi107 ceph-mon[29382]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:35.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:34 smithi107 ceph-mon[41185]: pgmap v112: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 140 MiB/s wr, 2.23k op/s 2024-09-21T09:42:35.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:34 smithi107 ceph-mon[41185]: from='client.14874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:35.762 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 31 16 66853 66837 134.725 96.4375 0.00490886 0.00741803 2024-09-21T09:42:35.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:35 smithi129 ceph-mon[32729]: from='client.24634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:36.111 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:35 smithi107 ceph-mon[29382]: from='client.24634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:36.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:35 smithi107 ceph-mon[41185]: from='client.24634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:36.763 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 32 16 68154 68138 133.052 81.3125 0.00685583 0.00751065 2024-09-21T09:42:36.823 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:42:36.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:36 smithi129 ceph-mon[32729]: pgmap v113: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-21T09:42:36.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:36 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1304262568' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:36 smithi107 ceph-mon[29382]: pgmap v113: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-21T09:42:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:36 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1304262568' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:37.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:36 smithi107 ceph-mon[41185]: pgmap v113: 89 pgs: 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-21T09:42:37.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:36 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1304262568' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:37.668 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:42:37 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:42:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:42:37.764 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 33 16 69468 69452 131.506 82.125 0.00801235 0.00759872 2024-09-21T09:42:38.765 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 34 16 70778 70762 130.043 81.875 0.00606331 0.00768452 2024-09-21T09:42:38.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:38 smithi129 ceph-mon[32729]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:38 smithi129 ceph-mon[32729]: pgmap v114: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-21T09:42:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:38 smithi129 ceph-mon[32729]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:39.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:38 smithi107 ceph-mon[29382]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:39.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:38 smithi107 ceph-mon[29382]: pgmap v114: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-21T09:42:39.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:38 smithi107 ceph-mon[29382]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:39.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:38 smithi107 ceph-mon[41185]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:39.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:38 smithi107 ceph-mon[41185]: pgmap v114: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-21T09:42:39.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:38 smithi107 ceph-mon[41185]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:39.608 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:42:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:42:39] "GET /metrics HTTP/1.1" 200 243265 "" "Prometheus/2.43.0" 2024-09-21T09:42:39.765 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 35 16 72116 72100 128.717 83.625 0.00586971 0.00776399 2024-09-21T09:42:40.447 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:42:40.765 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 36 16 73418 73402 127.402 81.375 0.0139396 0.00784444 2024-09-21T09:42:40.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:40 smithi107 ceph-mon[29382]: pgmap v115: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T09:42:40.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:40 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1487570286' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:40 smithi107 ceph-mon[41185]: pgmap v115: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T09:42:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:40 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1487570286' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:40.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:40 smithi129 ceph-mon[32729]: pgmap v115: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T09:42:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:40 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1487570286' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:41.765 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 37 16 74872 74856 126.414 90.875 0.0107711 0.00790584 2024-09-21T09:42:41.896 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:41 smithi107 ceph-mon[29382]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:41.896 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:41 smithi107 ceph-mon[41185]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:41.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:41 smithi129 ceph-mon[32729]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:42.765 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 38 16 76389 76373 125.583 94.8125 0.00461102 0.007958 2024-09-21T09:42:42.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:42 smithi107 ceph-mon[29382]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:42.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:42 smithi107 ceph-mon[29382]: pgmap v116: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T09:42:42.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:42 smithi107 ceph-mon[41185]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:42.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:42 smithi107 ceph-mon[41185]: pgmap v116: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T09:42:42.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:42 smithi129 ceph-mon[32729]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:42.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:42 smithi129 ceph-mon[32729]: pgmap v116: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T09:42:43.765 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 39 16 77725 77709 124.504 83.5 0.012001 0.00802723 2024-09-21T09:42:43.965 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:42:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:44 smithi107 ceph-mon[29382]: pgmap v117: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-09-21T09:42:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:44 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:42:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:44 smithi107 ceph-mon[29382]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (unknown) 2024-09-21T09:42:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:44 smithi107 ceph-mon[29382]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T09:42:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:44 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:42:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:44 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:42:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:44 smithi107 ceph-mon[29382]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-21T09:42:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:44 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/4277899506' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:44.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:44 smithi107 ceph-mon[41185]: pgmap v117: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-09-21T09:42:44.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:44 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:42:44.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:44 smithi107 ceph-mon[41185]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (unknown) 2024-09-21T09:42:44.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:44 smithi107 ceph-mon[41185]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T09:42:44.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:44 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:42:44.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:44 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:42:44.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:44 smithi107 ceph-mon[41185]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-21T09:42:44.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:44 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/4277899506' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:44 smithi129 ceph-mon[32729]: pgmap v117: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-09-21T09:42:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:44 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:42:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:44 smithi129 ceph-mon[32729]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (unknown) 2024-09-21T09:42:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:44 smithi129 ceph-mon[32729]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T09:42:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:44 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:42:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:44 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:42:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:44 smithi129 ceph-mon[32729]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-21T09:42:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:44 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/4277899506' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:44.765 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T09:42:44.764796+0000 min lat: 0.0012353 max lat: 0.113387 avg lat: 0.00809721 2024-09-21T09:42:44.765 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T09:42:44.765 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 40 16 79023 79007 123.419 81.125 0.0116051 0.00809721 2024-09-21T09:42:45.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:45 smithi107 ceph-mon[29382]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 on smithi129 2024-09-21T09:42:45.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:45 smithi107 ceph-mon[29382]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:45.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:45 smithi107 ceph-mon[29382]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:45 smithi107 ceph-mon[41185]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 on smithi129 2024-09-21T09:42:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:45 smithi107 ceph-mon[41185]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:45 smithi107 ceph-mon[41185]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:45.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:45 smithi129 ceph-mon[32729]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 on smithi129 2024-09-21T09:42:45.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:45 smithi129 ceph-mon[32729]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:45 smithi129 ceph-mon[32729]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:45.765 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 41 16 80295 80279 122.348 79.5 0.00839011 0.00816753 2024-09-21T09:42:46.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:46 smithi107 ceph-mon[29382]: pgmap v118: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-21T09:42:46.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:46 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:42:46.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:46 smithi107 ceph-mon[41185]: pgmap v118: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-21T09:42:46.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:46 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:42:46.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:46 smithi129 ceph-mon[32729]: pgmap v118: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-21T09:42:46.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:46 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:42:46.765 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 42 16 81536 81520 121.281 77.5625 0.0111102 0.00824077 2024-09-21T09:42:47.378 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:42:47.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:47 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3098205534' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:47.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:47 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3098205534' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:47.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:47 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3098205534' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:47.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:42:47 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:42:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:42:47.765 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 43 16 82765 82749 120.247 76.8125 0.00743356 0.00831162 2024-09-21T09:42:48.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:48 smithi107 ceph-mon[29382]: pgmap v119: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-21T09:42:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:48 smithi107 ceph-mon[29382]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:48.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:48 smithi107 ceph-mon[41185]: pgmap v119: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-21T09:42:48.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:48 smithi107 ceph-mon[41185]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:48.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:48 smithi129 ceph-mon[32729]: pgmap v119: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-21T09:42:48.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:48 smithi129 ceph-mon[32729]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:48.766 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 44 16 83983 83967 119.245 76.125 0.0157408 0.00838156 2024-09-21T09:42:49.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:49 smithi107 ceph-mon[29382]: from='client.24676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:49.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:49 smithi107 ceph-mon[41185]: from='client.24676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:49.610 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:42:49 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:42:49] "GET /metrics HTTP/1.1" 200 243332 "" "Prometheus/2.43.0" 2024-09-21T09:42:49.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:49 smithi129 ceph-mon[32729]: from='client.24676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:49.766 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 45 16 85197 85181 118.281 75.875 0.0232694 0.0084496 2024-09-21T09:42:50.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:50 smithi107 ceph-mon[29382]: pgmap v120: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-21T09:42:50.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:50 smithi107 ceph-mon[41185]: pgmap v120: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-21T09:42:50.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:50 smithi129 ceph-mon[32729]: pgmap v120: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-21T09:42:50.758 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:42:50.766 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 46 16 86324 86308 117.241 70.4375 0.00800231 0.008523 2024-09-21T09:42:51.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:51 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2700070505' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:51.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:51 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/2700070505' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:51.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:51 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2700070505' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:51.766 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 47 16 87511 87495 116.325 74.1875 0.00636409 0.00859145 2024-09-21T09:42:52.513 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:52 smithi107 ceph-mon[29382]: from='client.24685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:52.513 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:52 smithi107 ceph-mon[29382]: pgmap v121: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T09:42:52.513 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:52 smithi107 ceph-mon[29382]: from='client.24866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:52.513 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:52 smithi107 ceph-mon[41185]: from='client.24685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:52.513 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:52 smithi107 ceph-mon[41185]: pgmap v121: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T09:42:52.513 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:52 smithi107 ceph-mon[41185]: from='client.24866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:52.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:52 smithi129 ceph-mon[32729]: from='client.24685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:52.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:52 smithi129 ceph-mon[32729]: pgmap v121: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T09:42:52.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:52 smithi129 ceph-mon[32729]: from='client.24866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:52.766 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 48 16 88670 88654 115.41 72.4375 0.0132486 0.0086593 2024-09-21T09:42:53.766 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 49 16 89788 89772 114.481 69.875 0.00875831 0.0087306 2024-09-21T09:42:54.227 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:42:54.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:54 smithi107 ceph-mon[29382]: pgmap v122: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-09-21T09:42:54.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:54 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/772737375' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:54.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:54 smithi107 ceph-mon[41185]: pgmap v122: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-09-21T09:42:54.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:54 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/772737375' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:54.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:54 smithi129 ceph-mon[32729]: pgmap v122: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-09-21T09:42:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:54 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/772737375' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:54.766 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 50 16 90911 90895 113.595 70.1875 0.0162334 0.00879871 2024-09-21T09:42:55.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:55 smithi107 ceph-mon[29382]: from='client.24697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:55.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:55 smithi107 ceph-mon[29382]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:55.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:55 smithi107 ceph-mon[41185]: from='client.24697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:55.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:55 smithi107 ceph-mon[41185]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:55.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:55 smithi129 ceph-mon[32729]: from='client.24697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:55.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:55 smithi129 ceph-mon[32729]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:55.766 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 51 16 92016 92000 112.722 69.0625 0.0041354 0.00886677 2024-09-21T09:42:56.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:56 smithi107 ceph-mon[29382]: pgmap v123: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.05k op/s 2024-09-21T09:42:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:56 smithi107 ceph-mon[41185]: pgmap v123: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.05k op/s 2024-09-21T09:42:56.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:56 smithi129 ceph-mon[32729]: pgmap v123: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.05k op/s 2024-09-21T09:42:56.766 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 52 16 93141 93125 111.906 70.3125 0.0101897 0.00893154 2024-09-21T09:42:57.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:57 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1735180574' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:57.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:57 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1735180574' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:57.649 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:42:57.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:57 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1735180574' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:42:57.668 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:42:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:42:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:42:57.766 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 53 16 94272 94256 111.129 70.6875 0.0198544 0.00899429 2024-09-21T09:42:58.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:58 smithi107 ceph-mon[29382]: pgmap v124: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T09:42:58.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:58 smithi107 ceph-mon[29382]: from='client.24896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:58.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:58 smithi107 ceph-mon[41185]: pgmap v124: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T09:42:58.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:58 smithi107 ceph-mon[41185]: from='client.24896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:58.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:58 smithi129 ceph-mon[32729]: pgmap v124: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T09:42:58.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:58 smithi129 ceph-mon[32729]: from='client.24896 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:58.766 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 54 16 95391 95375 110.366 69.9375 0.00828709 0.00905642 2024-09-21T09:42:59.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:42:59 smithi107 ceph-mon[29382]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:59.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:42:59 smithi107 ceph-mon[41185]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:59.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:42:59 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:42:59] "GET /metrics HTTP/1.1" 200 243332 "" "Prometheus/2.43.0" 2024-09-21T09:42:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:42:59 smithi129 ceph-mon[32729]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:42:59.766 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 55 16 96484 96468 109.601 68.3125 0.0115403 0.00911983 2024-09-21T09:43:00.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:00 smithi107 ceph-mon[29382]: pgmap v125: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-21T09:43:00.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:00 smithi107 ceph-mon[41185]: pgmap v125: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-21T09:43:00.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:00 smithi129 ceph-mon[32729]: pgmap v125: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-21T09:43:00.767 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 56 16 97572 97556 108.858 68 0.0137595 0.00918117 2024-09-21T09:43:01.059 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:43:01.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:01 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3037489149' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:01.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:01 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:43:01.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:01 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3037489149' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:01.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:01 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:43:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:01 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3037489149' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:01 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:43:01.767 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 57 16 98670 98654 108.152 68.625 0.00773416 0.00924114 2024-09-21T09:43:02.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:02 smithi107 ceph-mon[29382]: from='client.24911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:02.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:02 smithi107 ceph-mon[29382]: pgmap v126: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 60 MiB/s wr, 964 op/s 2024-09-21T09:43:02.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:02 smithi107 ceph-mon[29382]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:02.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:02 smithi107 ceph-mon[41185]: from='client.24911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:02.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:02 smithi107 ceph-mon[41185]: pgmap v126: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 60 MiB/s wr, 964 op/s 2024-09-21T09:43:02.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:02 smithi107 ceph-mon[41185]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:02.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:02 smithi129 ceph-mon[32729]: from='client.24911 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:02.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:02 smithi129 ceph-mon[32729]: pgmap v126: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 60 MiB/s wr, 964 op/s 2024-09-21T09:43:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:02 smithi129 ceph-mon[32729]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:02.767 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 58 16 99745 99729 107.446 67.1875 0.0127973 0.00930223 2024-09-21T09:43:03.767 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 59 16 100823 100807 106.767 67.375 0.00840512 0.00936111 2024-09-21T09:43:04.445 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:43:04.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:04 smithi129 ceph-mon[32729]: pgmap v127: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-21T09:43:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:04 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/487634845' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:04.767 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T09:43:04.766582+0000 min lat: 0.0012353 max lat: 0.113387 avg lat: 0.00942376 2024-09-21T09:43:04.767 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T09:43:04.767 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 60 16 101854 101838 106.061 64.4375 0.0216558 0.00942376 2024-09-21T09:43:04.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:04 smithi107 ceph-mon[29382]: pgmap v127: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-21T09:43:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:04 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/487634845' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:04 smithi107 ceph-mon[41185]: pgmap v127: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-21T09:43:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:04 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/487634845' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:05.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:05 smithi129 ceph-mon[32729]: from='client.24929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:05.767 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 61 16 102927 102911 105.422 67.0625 0.0143121 0.009481 2024-09-21T09:43:05.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:05 smithi107 ceph-mon[29382]: from='client.24929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:05.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:05 smithi107 ceph-mon[41185]: from='client.24929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:06.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:06 smithi129 ceph-mon[32729]: from='client.24935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:06.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:06 smithi129 ceph-mon[32729]: pgmap v128: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 58 MiB/s wr, 930 op/s 2024-09-21T09:43:06.767 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 62 16 103923 103907 104.726 62.25 0.0281871 0.00954425 2024-09-21T09:43:06.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:06 smithi107 ceph-mon[29382]: from='client.24935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:06.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:06 smithi107 ceph-mon[29382]: pgmap v128: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 58 MiB/s wr, 930 op/s 2024-09-21T09:43:06.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:06 smithi107 ceph-mon[41185]: from='client.24935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:06.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:06 smithi107 ceph-mon[41185]: pgmap v128: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 58 MiB/s wr, 930 op/s 2024-09-21T09:43:07.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:07 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1831547924' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:07.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:07 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:43:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:43:07.767 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 63 16 104881 104865 104.014 59.875 0.0181135 0.00960934 2024-09-21T09:43:07.775 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:43:07.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:07 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1831547924' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:07.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:07 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1831547924' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:08.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:08 smithi129 ceph-mon[32729]: pgmap v129: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T09:43:08.767 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 64 16 105856 105840 103.341 60.9375 0.00779205 0.00967167 2024-09-21T09:43:08.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:08 smithi107 ceph-mon[29382]: pgmap v129: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T09:43:08.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:08 smithi107 ceph-mon[41185]: pgmap v129: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T09:43:09.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:09 smithi107 ceph-mon[29382]: from='client.24947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:09.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:09 smithi107 ceph-mon[29382]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:09.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:09 smithi107 ceph-mon[41185]: from='client.24947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:09.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:09 smithi107 ceph-mon[41185]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:09.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:09 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:43:09] "GET /metrics HTTP/1.1" 200 243397 "" "Prometheus/2.43.0" 2024-09-21T09:43:09.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:09 smithi129 ceph-mon[32729]: from='client.24947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:09.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:09 smithi129 ceph-mon[32729]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:09.768 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 65 16 106824 106808 102.682 60.5 0.0301363 0.00973378 2024-09-21T09:43:10.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:10 smithi129 ceph-mon[32729]: pgmap v130: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-21T09:43:10.767 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 66 16 107753 107737 102.005 58.0625 0.0076179 0.00979833 2024-09-21T09:43:10.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:10 smithi107 ceph-mon[29382]: pgmap v130: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-21T09:43:10.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:10 smithi107 ceph-mon[41185]: pgmap v130: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-21T09:43:11.081 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:43:11.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:11 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1949338981' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:11.768 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 67 16 108671 108655 101.339 57.375 0.0135613 0.0098625 2024-09-21T09:43:11.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:11 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1949338981' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:11.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:11 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1949338981' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:12 smithi129 ceph-mon[32729]: pgmap v131: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-21T09:43:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:12 smithi129 ceph-mon[32729]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:12 smithi129 ceph-mon[32729]: from='client.24968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:12.768 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 68 16 109631 109615 100.731 60 0.0127478 0.0099223 2024-09-21T09:43:12.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:12 smithi107 ceph-mon[29382]: pgmap v131: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-21T09:43:12.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:12 smithi107 ceph-mon[29382]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:12.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:12 smithi107 ceph-mon[29382]: from='client.24968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:12.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:12 smithi107 ceph-mon[41185]: pgmap v131: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-21T09:43:12.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:12 smithi107 ceph-mon[41185]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:12.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:12 smithi107 ceph-mon[41185]: from='client.24968 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:13.768 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 69 16 110574 110558 100.126 58.9375 0.00656965 0.00998207 2024-09-21T09:43:14.398 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:43:14.768 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 70 16 111494 111478 99.5167 57.5 0.0139545 0.0100432 2024-09-21T09:43:14.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:14 smithi107 ceph-mon[29382]: pgmap v132: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-21T09:43:14.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:14 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3750837194' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:14 smithi107 ceph-mon[41185]: pgmap v132: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-21T09:43:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:14 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3750837194' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:14.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:14 smithi129 ceph-mon[32729]: pgmap v132: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-21T09:43:14.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:14 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3750837194' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:15.768 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 71 16 112465 112449 98.9698 60.6875 0.00611531 0.0100998 2024-09-21T09:43:15.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:15 smithi107 ceph-mon[29382]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:15.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:15 smithi107 ceph-mon[41185]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:15.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:15 smithi129 ceph-mon[32729]: from='client.15036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:16.768 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 72 16 113391 113375 98.399 57.875 0.0151145 0.0101581 2024-09-21T09:43:16.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:16 smithi107 ceph-mon[29382]: from='client.24983 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:16 smithi107 ceph-mon[29382]: pgmap v133: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 52 MiB/s wr, 830 op/s 2024-09-21T09:43:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:16 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:43:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:16 smithi107 ceph-mon[41185]: from='client.24983 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:16 smithi107 ceph-mon[41185]: pgmap v133: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 52 MiB/s wr, 830 op/s 2024-09-21T09:43:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:16 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:43:16.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:16 smithi129 ceph-mon[32729]: from='client.24983 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:16.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:16 smithi129 ceph-mon[32729]: pgmap v133: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 52 MiB/s wr, 830 op/s 2024-09-21T09:43:16.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:16 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:43:17.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:17 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2678534866' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:17.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:43:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:43:17.747 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:43:17.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:17 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2678534866' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:17.747 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:17 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/2678534866' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:17.768 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 73 16 114340 114324 97.8636 59.3125 0.00555651 0.0102135 2024-09-21T09:43:18.768 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 74 16 115414 115398 97.4481 67.125 0.0092591 0.0102575 2024-09-21T09:43:18.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:18 smithi107 ceph-mon[29382]: pgmap v134: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 56 MiB/s wr, 903 op/s 2024-09-21T09:43:18.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:18 smithi107 ceph-mon[41185]: pgmap v134: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 56 MiB/s wr, 903 op/s 2024-09-21T09:43:18.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:18 smithi129 ceph-mon[32729]: pgmap v134: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 56 MiB/s wr, 903 op/s 2024-09-21T09:43:19.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:19 smithi107 ceph-mon[29382]: from='client.24995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:19.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:19 smithi107 ceph-mon[29382]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:19.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:19 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:43:19] "GET /metrics HTTP/1.1" 200 243432 "" "Prometheus/2.43.0" 2024-09-21T09:43:19.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:19 smithi107 ceph-mon[41185]: from='client.24995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:19.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:19 smithi107 ceph-mon[41185]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:19.768 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 75 16 116476 116460 97.0338 66.375 0.00825764 0.010301 2024-09-21T09:43:19.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:19 smithi129 ceph-mon[32729]: from='client.24995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:19 smithi129 ceph-mon[32729]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:20.768 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 76 16 117474 117458 96.5777 62.375 0.0187972 0.0103495 2024-09-21T09:43:20.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:20 smithi107 ceph-mon[29382]: pgmap v135: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T09:43:20.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:20 smithi107 ceph-mon[41185]: pgmap v135: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T09:43:20.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:20 smithi129 ceph-mon[32729]: pgmap v135: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T09:43:21.216 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:43:21.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:21 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/979415002' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:21.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:21 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/979415002' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:21.769 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 77 16 118514 118498 96.1676 65 0.0157965 0.0103935 2024-09-21T09:43:21.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:21 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/979415002' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:22.769 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 78 16 119551 119535 95.7655 64.8125 0.0160914 0.0104376 2024-09-21T09:43:22.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:22 smithi107 ceph-mon[29382]: pgmap v136: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 49 MiB/s wr, 789 op/s 2024-09-21T09:43:22.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:22 smithi107 ceph-mon[29382]: from='client.24787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:22.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:22 smithi107 ceph-mon[29382]: from='client.25013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:22.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:22 smithi107 ceph-mon[41185]: pgmap v136: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 49 MiB/s wr, 789 op/s 2024-09-21T09:43:22.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:22 smithi107 ceph-mon[41185]: from='client.24787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:22.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:22 smithi107 ceph-mon[41185]: from='client.25013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:22.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:22 smithi129 ceph-mon[32729]: pgmap v136: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 49 MiB/s wr, 789 op/s 2024-09-21T09:43:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:22 smithi129 ceph-mon[32729]: from='client.24787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:22 smithi129 ceph-mon[32729]: from='client.25013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:23.769 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 79 16 120588 120572 95.3737 64.8125 0.0131614 0.0104802 2024-09-21T09:43:24.674 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:43:24.769 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T09:43:24.768292+0000 min lat: 0.0012353 max lat: 0.113387 avg lat: 0.0105274 2024-09-21T09:43:24.769 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T09:43:24.769 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 80 16 121570 121554 94.9487 61.375 0.011363 0.0105274 2024-09-21T09:43:24.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:24 smithi107 ceph-mon[29382]: pgmap v137: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-21T09:43:24.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:24 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/444257230' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:24.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:24 smithi107 ceph-mon[41185]: pgmap v137: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-21T09:43:24.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:24 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/444257230' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:24.885 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:24 smithi129 ceph-mon[32729]: pgmap v137: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-21T09:43:24.885 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:24 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/444257230' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:25.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:25 smithi129 ceph-mon[32729]: from='client.24799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:25.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:25 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:25.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:25 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:43:25.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:25 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:43:25.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:25 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:43:25.769 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 81 16 122558 122542 94.5388 61.75 0.0249776 0.0105728 2024-09-21T09:43:25.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:25 smithi107 ceph-mon[29382]: from='client.24799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:25 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:25 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:43:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:25 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:43:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:25 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:43:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:25 smithi107 ceph-mon[41185]: from='client.24799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:25 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:25 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:43:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:25 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:43:25.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:25 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:43:26.769 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 82 16 123585 123569 94.1686 64.1875 0.0143503 0.0106146 2024-09-21T09:43:26.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:26 smithi107 ceph-mon[29382]: pgmap v138: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 51 MiB/s wr, 819 op/s 2024-09-21T09:43:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:26 smithi107 ceph-mon[29382]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:26 smithi107 ceph-mon[29382]: Upgrade: Updating mgr.x 2024-09-21T09:43:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:26 smithi107 ceph-mon[29382]: Deploying daemon mgr.x on smithi129 2024-09-21T09:43:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:26 smithi107 ceph-mon[41185]: pgmap v138: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 51 MiB/s wr, 819 op/s 2024-09-21T09:43:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:26 smithi107 ceph-mon[41185]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:26 smithi107 ceph-mon[41185]: Upgrade: Updating mgr.x 2024-09-21T09:43:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:26 smithi107 ceph-mon[41185]: Deploying daemon mgr.x on smithi129 2024-09-21T09:43:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:26 smithi129 ceph-mon[32729]: pgmap v138: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 51 MiB/s wr, 819 op/s 2024-09-21T09:43:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:26 smithi129 ceph-mon[32729]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:26 smithi129 ceph-mon[32729]: Upgrade: Updating mgr.x 2024-09-21T09:43:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:26 smithi129 ceph-mon[32729]: Deploying daemon mgr.x on smithi129 2024-09-21T09:43:27.668 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:27 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[34778]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:43:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:43:27.769 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 83 16 124576 124560 93.7802 61.9375 0.0164042 0.0106586 2024-09-21T09:43:28.135 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:43:28.419 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:28 smithi129 systemd[1]: Stopping Ceph mgr.x for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:43:28.769 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 84 16 125519 125503 93.3654 58.9375 0.0172645 0.010706 2024-09-21T09:43:28.781 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:28 smithi129 podman[68233]: 2024-09-21 09:43:28.50809485 +0000 UTC m=+0.181009905 container died 1f321f7d8eff1220945e51d5357db00e290120c2ad9d9b1a13007900297080f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x, RELEASE=quincy-fbd1a6a, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-21T09:43:28.782 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:28 smithi129 ceph-mon[32729]: pgmap v139: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 56 MiB/s wr, 899 op/s 2024-09-21T09:43:28.782 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:28 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1124248028' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:28.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:28 smithi107 ceph-mon[29382]: pgmap v139: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 56 MiB/s wr, 899 op/s 2024-09-21T09:43:28.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:28 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1124248028' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:28.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:28 smithi107 ceph-mon[41185]: pgmap v139: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 56 MiB/s wr, 899 op/s 2024-09-21T09:43:28.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:28 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1124248028' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:29.168 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:28 smithi129 podman[68233]: 2024-09-21 09:43:28.967473483 +0000 UTC m=+0.640388532 container cleanup 1f321f7d8eff1220945e51d5357db00e290120c2ad9d9b1a13007900297080f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-21T09:43:29.169 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:28 smithi129 bash[68233]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x 2024-09-21T09:43:29.491 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:29 smithi129 podman[68249]: 2024-09-21 09:43:29.230225392 +0000 UTC m=+0.714772037 container remove 1f321f7d8eff1220945e51d5357db00e290120c2ad9d9b1a13007900297080f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T09:43:29.491 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:29 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-21T09:43:29.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:29 smithi107 ceph-mon[29382]: from='client.24811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:29.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:29 smithi107 ceph-mon[29382]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:29.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:29 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:43:29] "GET /metrics HTTP/1.1" 200 243432 "" "Prometheus/2.43.0" 2024-09-21T09:43:29.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:29 smithi107 ceph-mon[41185]: from='client.24811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:29.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:29 smithi107 ceph-mon[41185]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:29.769 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 85 16 126462 126446 92.9603 58.9375 0.0255564 0.0107519 2024-09-21T09:43:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:29 smithi129 ceph-mon[32729]: from='client.24811 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:29 smithi129 ceph-mon[32729]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:29.919 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:29 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@mgr.x.service: Failed with result 'exit-code'. 2024-09-21T09:43:29.919 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:29 smithi129 systemd[1]: Stopped Ceph mgr.x for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:43:29.920 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:29 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@mgr.x.service: Consumed 10.455s CPU time. 2024-09-21T09:43:30.326 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:30 smithi129 systemd[1]: Starting Ceph mgr.x for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:43:30.653 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:30 smithi129 podman[68368]: 2024-09-21 09:43:30.230546015 +0000 UTC m=+0.030271720 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:43:30.653 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:30 smithi129 podman[68368]: 2024-09-21 09:43:30.356608853 +0000 UTC m=+0.156334550 container create dc5e5c81266c860986986aeac1803c80675ecea2fece0ee41a835ab0d1399c2c (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x, org.label-schema.build-date=20240731, ceph=True, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T09:43:30.653 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:30 smithi129 ceph-mon[32729]: pgmap v140: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T09:43:30.769 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 86 16 127391 127375 92.5545 58.0625 0.0127124 0.0108 2024-09-21T09:43:30.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:30 smithi107 ceph-mon[29382]: pgmap v140: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T09:43:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:30 smithi107 ceph-mon[41185]: pgmap v140: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T09:43:30.918 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:30 smithi129 podman[68368]: 2024-09-21 09:43:30.6518253 +0000 UTC m=+0.451551008 container init dc5e5c81266c860986986aeac1803c80675ecea2fece0ee41a835ab0d1399c2c (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-21T09:43:30.919 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:30 smithi129 podman[68368]: 2024-09-21 09:43:30.667952398 +0000 UTC m=+0.467678095 container start dc5e5c81266c860986986aeac1803c80675ecea2fece0ee41a835ab0d1399c2c (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-21T09:43:30.919 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:30 smithi129 ceph-mgr[68404]: -- 172.21.15.129:0/1730068757 <== mon.2 v2:172.21.15.107:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a97abab0e0 con 0x55a97ab89000 2024-09-21T09:43:30.919 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:30 smithi129 bash[68368]: dc5e5c81266c860986986aeac1803c80675ecea2fece0ee41a835ab0d1399c2c 2024-09-21T09:43:30.919 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:30 smithi129 systemd[1]: Started Ceph mgr.x for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:43:31.536 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:43:31.771 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 87 16 128173 128157 92.0504 48.875 0.021546 0.0108591 2024-09-21T09:43:32.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:32.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:32.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:43:32.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:31 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:43:32.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:31 smithi107 ceph-mon[29382]: pgmap v141: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 53 MiB/s wr, 851 op/s 2024-09-21T09:43:32.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:31 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2063551530' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:32.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:31 smithi107 ceph-mon[29382]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:32.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:32.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:32.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:43:32.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:31 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:43:32.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:31 smithi107 ceph-mon[41185]: pgmap v141: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 53 MiB/s wr, 851 op/s 2024-09-21T09:43:32.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:31 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/2063551530' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:32.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:31 smithi107 ceph-mon[41185]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:32.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:32.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:32.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:43:32.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:31 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:43:32.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:31 smithi129 ceph-mon[32729]: pgmap v141: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 53 MiB/s wr, 851 op/s 2024-09-21T09:43:32.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:31 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2063551530' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:32.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:31 smithi129 ceph-mon[32729]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:32.771 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 88 16 129015 128999 91.6024 52.625 0.00853394 0.0109118 2024-09-21T09:43:33.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:32 smithi129 ceph-mon[32729]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:33.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:32 smithi107 ceph-mon[29382]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:33.202 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:32 smithi107 ceph-mon[41185]: from='client.25061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:33.771 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 89 16 129840 129824 91.1525 51.5625 0.0319083 0.0109658 2024-09-21T09:43:34.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:33 smithi129 ceph-mon[32729]: pgmap v142: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T09:43:34.171 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:33 smithi107 ceph-mon[29382]: pgmap v142: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T09:43:34.171 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:33 smithi107 ceph-mon[41185]: pgmap v142: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T09:43:34.772 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total time run: 90.0126 2024-09-21T09:43:34.772 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total writes made: 130611 2024-09-21T09:43:34.772 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Write size: 65536 2024-09-21T09:43:34.772 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object size: 65536 2024-09-21T09:43:34.772 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Bandwidth (MB/sec): 90.6894 2024-09-21T09:43:34.772 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Bandwidth: 42.2969 2024-09-21T09:43:34.772 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max bandwidth (MB/sec): 275.75 2024-09-21T09:43:34.772 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min bandwidth (MB/sec): 48.875 2024-09-21T09:43:34.772 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average IOPS: 1451 2024-09-21T09:43:34.772 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev IOPS: 676.751 2024-09-21T09:43:34.772 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max IOPS: 4412 2024-09-21T09:43:34.772 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min IOPS: 782 2024-09-21T09:43:34.773 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average Latency(s): 0.0110225 2024-09-21T09:43:34.773 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Latency(s): 0.00771863 2024-09-21T09:43:34.773 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max latency(s): 0.113387 2024-09-21T09:43:34.773 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min latency(s): 0.0012353 2024-09-21T09:43:34.779 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T09:43:34.985 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:43:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:35 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:35 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1677935390' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:35 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:35.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:35 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:35.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:35 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1677935390' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:35.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:35 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:35.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:35 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:35.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:35 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1677935390' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:35.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:35 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:36.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:36 smithi129 ceph-mon[32729]: from='client.25073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:36.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:36 smithi129 ceph-mon[32729]: pgmap v143: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 50 MiB/s wr, 802 op/s 2024-09-21T09:43:36.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:36 smithi129 ceph-mon[32729]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:36.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:36 smithi107 ceph-mon[29382]: from='client.25073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:36.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:36 smithi107 ceph-mon[29382]: pgmap v143: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 50 MiB/s wr, 802 op/s 2024-09-21T09:43:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:36 smithi107 ceph-mon[29382]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:36.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:36 smithi107 ceph-mon[41185]: from='client.25073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:36.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:36 smithi107 ceph-mon[41185]: pgmap v143: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 50 MiB/s wr, 802 op/s 2024-09-21T09:43:36.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:36 smithi107 ceph-mon[41185]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:38.200 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:43:38.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:38 smithi129 ceph-mon[32729]: pgmap v144: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 52 MiB/s wr, 866 op/s 2024-09-21T09:43:38.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:38 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:38.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:38 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:38.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:38 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/4161967856' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:38.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:38 smithi107 ceph-mon[29382]: pgmap v144: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 52 MiB/s wr, 866 op/s 2024-09-21T09:43:38.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:38 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:38.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:38 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:38.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:38 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/4161967856' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:38 smithi107 ceph-mon[41185]: pgmap v144: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 52 MiB/s wr, 866 op/s 2024-09-21T09:43:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:38 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:38 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:38 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/4161967856' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:39.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:39 smithi129 ceph-mon[32729]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:39.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:39 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:39.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:39 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:39.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:39 smithi129 ceph-mon[32729]: from='client.25097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:39.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:39 smithi107 ceph-mon[29382]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:39.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:39 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:39.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:39 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:39.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:39 smithi107 ceph-mon[29382]: from='client.25097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:39.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:39 smithi107 ceph-mon[41185]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:39.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:39 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:39.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:39 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:39.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:39 smithi107 ceph-mon[41185]: from='client.25097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:39.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:43:39] "GET /metrics HTTP/1.1" 200 243432 "" "Prometheus/2.43.0" 2024-09-21T09:43:39.919 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:39 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: [21/Sep/2024:09:43:39] ENGINE Bus STARTING 2024-09-21T09:43:39.919 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:39 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: CherryPy Checker: 2024-09-21T09:43:39.919 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:39 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: The Application mounted at '' has an empty config. 2024-09-21T09:43:39.919 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:39 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: 2024-09-21T09:43:39.919 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:39 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: [21/Sep/2024:09:43:39] ENGINE Serving on http://:::9283 2024-09-21T09:43:39.919 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:39 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: [21/Sep/2024:09:43:39] ENGINE Bus STARTED 2024-09-21T09:43:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:40 smithi107 ceph-mon[29382]: pgmap v145: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 57 MiB/s wr, 1.07k op/s 2024-09-21T09:43:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:40 smithi107 ceph-mon[29382]: Standby manager daemon x restarted 2024-09-21T09:43:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:40 smithi107 ceph-mon[29382]: Standby manager daemon x started 2024-09-21T09:43:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:40 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:43:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:40 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:43:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:40 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:43:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:40 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:43:40.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:40 smithi107 ceph-mon[41185]: pgmap v145: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 57 MiB/s wr, 1.07k op/s 2024-09-21T09:43:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:40 smithi107 ceph-mon[41185]: Standby manager daemon x restarted 2024-09-21T09:43:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:40 smithi107 ceph-mon[41185]: Standby manager daemon x started 2024-09-21T09:43:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:40 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:43:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:40 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:43:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:40 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:43:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:40 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:43:40.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:40 smithi129 ceph-mon[32729]: pgmap v145: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 57 MiB/s wr, 1.07k op/s 2024-09-21T09:43:40.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:40 smithi129 ceph-mon[32729]: Standby manager daemon x restarted 2024-09-21T09:43:40.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:40 smithi129 ceph-mon[32729]: Standby manager daemon x started 2024-09-21T09:43:40.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:40 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:43:40.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:40 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:43:40.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:40 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:43:40.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:40 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:43:41.525 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:43:41.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:41 smithi107 ceph-mon[29382]: mgrmap e21: y(active, since 4m), standbys: x 2024-09-21T09:43:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:41 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/518527425' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:41.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:41 smithi107 ceph-mon[41185]: mgrmap e21: y(active, since 4m), standbys: x 2024-09-21T09:43:41.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:41 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/518527425' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:41 smithi129 ceph-mon[32729]: mgrmap e21: y(active, since 4m), standbys: x 2024-09-21T09:43:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:41 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/518527425' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:42.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:42 smithi107 ceph-mon[29382]: pgmap v146: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 35 MiB/s wr, 727 op/s 2024-09-21T09:43:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:42 smithi107 ceph-mon[29382]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:42 smithi107 ceph-mon[41185]: pgmap v146: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 35 MiB/s wr, 727 op/s 2024-09-21T09:43:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:42 smithi107 ceph-mon[41185]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:42.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:42 smithi129 ceph-mon[32729]: pgmap v146: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 35 MiB/s wr, 727 op/s 2024-09-21T09:43:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:42 smithi129 ceph-mon[32729]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:43.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:43 smithi107 ceph-mon[29382]: from='client.24865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:43.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:43 smithi107 ceph-mon[41185]: from='client.24865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:43 smithi129 ceph-mon[32729]: from='client.24865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:44.136 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: [21/Sep/2024:09:43:44] ENGINE Bus STOPPING 2024-09-21T09:43:44.419 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: [21/Sep/2024:09:43:44] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:43:44.419 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: [21/Sep/2024:09:43:44] ENGINE Bus STOPPED 2024-09-21T09:43:44.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: pgmap v147: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 35 MiB/s wr, 1.07k op/s 2024-09-21T09:43:44.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:44.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:44.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:43:44.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:43:44.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:44.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:43:44.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T09:43:44.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: Activating manager daemon x 2024-09-21T09:43:44.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T09:43:44.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-21T09:43:44.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: mgrmap e22: x(active, starting, since 0.0166905s) 2024-09-21T09:43:44.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:43:44.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:43:44.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:43:44.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi129.zdzmqr"}]: dispatch 2024-09-21T09:43:44.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi107.sedied"}]: dispatch 2024-09-21T09:43:44.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:43:44.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:43:44.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:43:44.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:43:44.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:43:44.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:43:44.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:43:44.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:43:44.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:43:44.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:43:44.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:43:44.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:44 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:43:44.438 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: pgmap v147: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 35 MiB/s wr, 1.07k op/s 2024-09-21T09:43:44.438 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:44.438 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:44.438 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:43:44.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:43:44.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:44.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:43:44.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T09:43:44.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: Activating manager daemon x 2024-09-21T09:43:44.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T09:43:44.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-21T09:43:44.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: mgrmap e22: x(active, starting, since 0.0166905s) 2024-09-21T09:43:44.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:43:44.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:43:44.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:43:44.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi129.zdzmqr"}]: dispatch 2024-09-21T09:43:44.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi107.sedied"}]: dispatch 2024-09-21T09:43:44.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:43:44.439 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:43:44.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:43:44.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:43:44.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:43:44.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:43:44.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:43:44.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:43:44.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:43:44.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:43:44.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:43:44.440 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:43:44.440 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ignoring --setuser ceph since I am not root 2024-09-21T09:43:44.441 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ignoring --setgroup ceph since I am not root 2024-09-21T09:43:44.441 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:44.343+0000 7f6cbf94cec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:43:44.441 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:44.435+0000 7f6cbf94cec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:43:44.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: pgmap v147: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 35 MiB/s wr, 1.07k op/s 2024-09-21T09:43:44.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:44.441 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:44.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:43:44.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:43:44.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' 2024-09-21T09:43:44.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:43:44.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T09:43:44.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: Activating manager daemon x 2024-09-21T09:43:44.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T09:43:44.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24344 172.21.15.107:0/2941112402' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-21T09:43:44.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: mgrmap e22: x(active, starting, since 0.0166905s) 2024-09-21T09:43:44.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:43:44.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:43:44.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:43:44.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi129.zdzmqr"}]: dispatch 2024-09-21T09:43:44.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi107.sedied"}]: dispatch 2024-09-21T09:43:44.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:43:44.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:43:44.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:43:44.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:43:44.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:43:44.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:43:44.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:43:44.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:43:44.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:43:44.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:43:44.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:43:44.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:43:44.747 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:44.546+0000 7f6cbf94cec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:43:45.108 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:44.745+0000 7f6cbf94cec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:43:45.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:44 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:44.963+0000 7f6cbf94cec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:43:45.393 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date 2024-09-21T09:43:45.393 INFO:teuthology.orchestra.run.smithi107.stderr:Module 'orchestrator' is not enabled/loaded (required by command 'orch upgrade status'): use `ceph mgr module enable orchestrator` to enable it 2024-09-21T09:43:45.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:45 smithi107 ceph-mon[41185]: Manager daemon x is now available 2024-09-21T09:43:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:45 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3362191925' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:45 smithi107 ceph-mon[29382]: Manager daemon x is now available 2024-09-21T09:43:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:45 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3362191925' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:45.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:45.338+0000 7f6cbf94cec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:43:45.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:45.426+0000 7f6cbf94cec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:43:45.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:45 smithi129 ceph-mon[32729]: Manager daemon x is now available 2024-09-21T09:43:45.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:45 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3362191925' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:43:45.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:45 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: 2024-09-21T09:43:45.385+0000 7fbe790f2640 -1 mgr.server reply reply (95) Operation not supported Warning: due to ceph-mgr restart, some PG states may not be up to date 2024-09-21T09:43:45.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:45 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: Module 'orchestrator' is not enabled/loaded (required by command 'orch upgrade status'): use `ceph mgr module enable orchestrator` to enable it 2024-09-21T09:43:46.106 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:45.678+0000 7f6cbf94cec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:43:46.163 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:45 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: [21/Sep/2024:09:43:45] ENGINE Bus STARTING 2024-09-21T09:43:46.163 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: [21/Sep/2024:09:43:46] ENGINE Serving on http://:::9283 2024-09-21T09:43:46.163 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: [21/Sep/2024:09:43:46] ENGINE Bus STARTED 2024-09-21T09:43:46.163 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: 2024-09-21T09:43:46.004+0000 7fbe568ad640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:43:46.163 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: 2024-09-21T09:43:46.004+0000 7fbe568ad640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:43:46.163 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: 2024-09-21T09:43:46.004+0000 7fbe568ad640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:43:46.163 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: 2024-09-21T09:43:46.004+0000 7fbe568ad640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:43:46.163 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: 2024-09-21T09:43:46.004+0000 7fbe568ad640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:43:46.164 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: 2024-09-21T09:43:46.014+0000 7fbe4f09e640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:43:46.164 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: 2024-09-21T09:43:46.014+0000 7fbe4f09e640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:43:46.164 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: 2024-09-21T09:43:46.014+0000 7fbe4f09e640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:43:46.164 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: 2024-09-21T09:43:46.014+0000 7fbe4f09e640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:43:46.164 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: 2024-09-21T09:43:46.014+0000 7fbe4f09e640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:43:46.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-mon[32729]: mgrmap e23: x(active, since 1.3063s) 2024-09-21T09:43:46.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-mon[32729]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:46.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-mon[32729]: pgmap v3: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail 2024-09-21T09:43:46.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:46.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:46.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:46.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:46.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:46.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:43:46.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:43:46.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:43:46.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:43:46.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:46 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:43:46.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[41185]: mgrmap e23: x(active, since 1.3063s) 2024-09-21T09:43:46.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[41185]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:46.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[41185]: pgmap v3: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail 2024-09-21T09:43:46.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:46.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:46.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:46.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:46.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:46.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:43:46.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:43:46.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:43:46.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:43:46.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:43:46.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[29382]: mgrmap e23: x(active, since 1.3063s) 2024-09-21T09:43:46.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[29382]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:46.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[29382]: pgmap v3: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail 2024-09-21T09:43:46.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:46.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:46.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:46.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:46.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:46.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:43:46.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:43:46.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:43:46.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:43:46.832 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:43:46.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch ps' 2024-09-21T09:43:47.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:46 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:46.905+0000 7f6cbf94cec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:43:47.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:47 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:47.000+0000 7f6cbf94cec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:43:47.225 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:43:47.668 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:47 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:43:47] "GET /metrics HTTP/1.1" 200 36134 "" "Prometheus/2.43.0" 2024-09-21T09:43:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:47 smithi129 ceph-mon[32729]: Checking for cert/key for grafana.a 2024-09-21T09:43:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:47 smithi129 ceph-mon[32729]: Migrating grafana.a cert to cert store 2024-09-21T09:43:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:47 smithi129 ceph-mon[32729]: Migrating grafana.a key to cert store 2024-09-21T09:43:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:47 smithi129 ceph-mon[32729]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:43:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:47 smithi129 ceph-mon[32729]: pgmap v4: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail 2024-09-21T09:43:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:47 smithi129 ceph-mon[32729]: Deploying cephadm binary to smithi107 2024-09-21T09:43:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:47 smithi129 ceph-mon[32729]: Deploying cephadm binary to smithi129 2024-09-21T09:43:47.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:47 smithi107 ceph-mon[29382]: Checking for cert/key for grafana.a 2024-09-21T09:43:47.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:47 smithi107 ceph-mon[29382]: Migrating grafana.a cert to cert store 2024-09-21T09:43:47.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:47 smithi107 ceph-mon[29382]: Migrating grafana.a key to cert store 2024-09-21T09:43:47.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:47 smithi107 ceph-mon[29382]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:43:47.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:47 smithi107 ceph-mon[29382]: pgmap v4: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail 2024-09-21T09:43:47.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:47 smithi107 ceph-mon[29382]: Deploying cephadm binary to smithi107 2024-09-21T09:43:47.749 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:47 smithi107 ceph-mon[29382]: Deploying cephadm binary to smithi129 2024-09-21T09:43:47.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:47 smithi107 ceph-mon[41185]: Checking for cert/key for grafana.a 2024-09-21T09:43:47.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:47 smithi107 ceph-mon[41185]: Migrating grafana.a cert to cert store 2024-09-21T09:43:47.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:47 smithi107 ceph-mon[41185]: Migrating grafana.a key to cert store 2024-09-21T09:43:47.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:47 smithi107 ceph-mon[41185]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:43:47.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:47 smithi107 ceph-mon[41185]: pgmap v4: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail 2024-09-21T09:43:47.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:47 smithi107 ceph-mon[41185]: Deploying cephadm binary to smithi107 2024-09-21T09:43:47.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:47 smithi107 ceph-mon[41185]: Deploying cephadm binary to smithi129 2024-09-21T09:43:48.359 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:48 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:48.038+0000 7f6cbf94cec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:43:48.359 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:48 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:48.211+0000 7f6cbf94cec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:43:48.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:48 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: 2024-09-21T09:43:48.245+0000 7fbe2f562640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:43:48] ENGINE Error in HTTPServer.serve 2024-09-21T09:43:48.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:48 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: Traceback (most recent call last): 2024-09-21T09:43:48.670 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:48 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:43:48.670 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:48 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: self._connections.run(self.expiration_interval) 2024-09-21T09:43:48.670 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:48 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:43:48.670 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:48 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: self._run(expiration_interval) 2024-09-21T09:43:48.670 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:48 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:43:48.670 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:48 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:43:48.670 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:48 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:43:48.670 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:48 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:43:48.670 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:48 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:43:48.670 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:48 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: s = self.context.wrap_socket( 2024-09-21T09:43:48.671 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:48 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:43:48.671 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:48 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: return self.sslsocket_class._create( 2024-09-21T09:43:48.671 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:48 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:43:48.671 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:48 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: self.do_handshake() 2024-09-21T09:43:48.671 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:48 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:43:48.671 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:48 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: self._sslobj.do_handshake() 2024-09-21T09:43:48.671 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:48 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:43:48.671 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:48 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: 2024-09-21T09:43:48.718 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:48 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:48.427+0000 7f6cbf94cec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:43:48.718 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:48 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:48.526+0000 7f6cbf94cec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:43:48.718 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:48 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:48.715+0000 7f6cbf94cec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:43:49.013 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:48 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:48.821+0000 7f6cbf94cec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:43:49.013 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:48 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:48.921+0000 7f6cbf94cec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:43:49.332 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:49 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:49.133+0000 7f6cbf94cec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:43:49.332 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:49 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:49.223+0000 7f6cbf94cec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:43:49.332 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:49 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:49.327+0000 7f6cbf94cec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:43:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:49 smithi107 ceph-mon[29382]: [21/Sep/2024:09:43:47] ENGINE Bus STARTING 2024-09-21T09:43:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:49 smithi107 ceph-mon[29382]: [21/Sep/2024:09:43:48] ENGINE Serving on http://172.21.15.129:8765 2024-09-21T09:43:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:49 smithi107 ceph-mon[29382]: pgmap v5: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail 2024-09-21T09:43:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:49 smithi107 ceph-mon[29382]: [21/Sep/2024:09:43:48] ENGINE Serving on https://172.21.15.129:7150 2024-09-21T09:43:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:49 smithi107 ceph-mon[29382]: [21/Sep/2024:09:43:48] ENGINE Bus STARTED 2024-09-21T09:43:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:49 smithi107 ceph-mon[29382]: [21/Sep/2024:09:43:48] ENGINE Error in HTTPServer.serve 2024-09-21T09:43:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout: Traceback (most recent call last): 2024-09-21T09:43:49.610 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:43:49.610 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:43:49.610 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:43:49.610 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._run(expiration_interval) 2024-09-21T09:43:49.610 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:43:49.610 INFO:journalctl@ceph.mon.a.smithi107.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:43:49.610 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:43:49.610 INFO:journalctl@ceph.mon.a.smithi107.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:43:49.610 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:43:49.610 INFO:journalctl@ceph.mon.a.smithi107.stdout: s = self.context.wrap_socket( 2024-09-21T09:43:49.610 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:43:49.611 INFO:journalctl@ceph.mon.a.smithi107.stdout: return self.sslsocket_class._create( 2024-09-21T09:43:49.611 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:43:49.611 INFO:journalctl@ceph.mon.a.smithi107.stdout: self.do_handshake() 2024-09-21T09:43:49.611 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:43:49.611 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._sslobj.do_handshake() 2024-09-21T09:43:49.611 INFO:journalctl@ceph.mon.a.smithi107.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:43:49.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:49 smithi107 ceph-mon[29382]: mgrmap e24: x(active, since 4s) 2024-09-21T09:43:49.611 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:49 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:49.463+0000 7f6cbf94cec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:43:49.612 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:49 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:49.557+0000 7f6cbf94cec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:43:49.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:49 smithi107 ceph-mon[41185]: [21/Sep/2024:09:43:47] ENGINE Bus STARTING 2024-09-21T09:43:49.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:49 smithi107 ceph-mon[41185]: [21/Sep/2024:09:43:48] ENGINE Serving on http://172.21.15.129:8765 2024-09-21T09:43:49.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:49 smithi107 ceph-mon[41185]: pgmap v5: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail 2024-09-21T09:43:49.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:49 smithi107 ceph-mon[41185]: [21/Sep/2024:09:43:48] ENGINE Serving on https://172.21.15.129:7150 2024-09-21T09:43:49.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:49 smithi107 ceph-mon[41185]: [21/Sep/2024:09:43:48] ENGINE Bus STARTED 2024-09-21T09:43:49.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:49 smithi107 ceph-mon[41185]: [21/Sep/2024:09:43:48] ENGINE Error in HTTPServer.serve 2024-09-21T09:43:49.612 INFO:journalctl@ceph.mon.c.smithi107.stdout: Traceback (most recent call last): 2024-09-21T09:43:49.612 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:43:49.612 INFO:journalctl@ceph.mon.c.smithi107.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:43:49.612 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:43:49.613 INFO:journalctl@ceph.mon.c.smithi107.stdout: self._run(expiration_interval) 2024-09-21T09:43:49.613 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:43:49.613 INFO:journalctl@ceph.mon.c.smithi107.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:43:49.613 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:43:49.613 INFO:journalctl@ceph.mon.c.smithi107.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:43:49.613 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:43:49.613 INFO:journalctl@ceph.mon.c.smithi107.stdout: s = self.context.wrap_socket( 2024-09-21T09:43:49.613 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:43:49.613 INFO:journalctl@ceph.mon.c.smithi107.stdout: return self.sslsocket_class._create( 2024-09-21T09:43:49.613 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:43:49.613 INFO:journalctl@ceph.mon.c.smithi107.stdout: self.do_handshake() 2024-09-21T09:43:49.613 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:43:49.613 INFO:journalctl@ceph.mon.c.smithi107.stdout: self._sslobj.do_handshake() 2024-09-21T09:43:49.613 INFO:journalctl@ceph.mon.c.smithi107.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:43:49.613 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:49 smithi107 ceph-mon[41185]: mgrmap e24: x(active, since 4s) 2024-09-21T09:43:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:49 smithi129 ceph-mon[32729]: [21/Sep/2024:09:43:47] ENGINE Bus STARTING 2024-09-21T09:43:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:49 smithi129 ceph-mon[32729]: [21/Sep/2024:09:43:48] ENGINE Serving on http://172.21.15.129:8765 2024-09-21T09:43:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:49 smithi129 ceph-mon[32729]: pgmap v5: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail 2024-09-21T09:43:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:49 smithi129 ceph-mon[32729]: [21/Sep/2024:09:43:48] ENGINE Serving on https://172.21.15.129:7150 2024-09-21T09:43:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:49 smithi129 ceph-mon[32729]: [21/Sep/2024:09:43:48] ENGINE Bus STARTED 2024-09-21T09:43:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:49 smithi129 ceph-mon[32729]: [21/Sep/2024:09:43:48] ENGINE Error in HTTPServer.serve 2024-09-21T09:43:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout: Traceback (most recent call last): 2024-09-21T09:43:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:43:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:43:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:43:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout: self._run(expiration_interval) 2024-09-21T09:43:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:43:49.670 INFO:journalctl@ceph.mon.b.smithi129.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:43:49.670 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:43:49.670 INFO:journalctl@ceph.mon.b.smithi129.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:43:49.670 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:43:49.670 INFO:journalctl@ceph.mon.b.smithi129.stdout: s = self.context.wrap_socket( 2024-09-21T09:43:49.670 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:43:49.670 INFO:journalctl@ceph.mon.b.smithi129.stdout: return self.sslsocket_class._create( 2024-09-21T09:43:49.670 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:43:49.670 INFO:journalctl@ceph.mon.b.smithi129.stdout: self.do_handshake() 2024-09-21T09:43:49.670 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:43:49.670 INFO:journalctl@ceph.mon.b.smithi129.stdout: self._sslobj.do_handshake() 2024-09-21T09:43:49.670 INFO:journalctl@ceph.mon.b.smithi129.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:43:49.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:49 smithi129 ceph-mon[32729]: mgrmap e24: x(active, since 4s) 2024-09-21T09:43:49.884 INFO:teuthology.orchestra.run.smithi107.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T09:43:49.885 INFO:teuthology.orchestra.run.smithi107.stdout:alertmanager.a smithi107 *:9093,9094 running (3m) 15s ago 4m 25.5M - 0.25.0 c8568f914cd2 432a2da9f5f4 2024-09-21T09:43:49.885 INFO:teuthology.orchestra.run.smithi107.stdout:grafana.a smithi129 *:3000 running (3m) 12s ago 3m 92.1M - 9.4.7 954c08fa6188 70638332abb2 2024-09-21T09:43:49.885 INFO:teuthology.orchestra.run.smithi107.stdout:mds.foo.smithi107.sedied smithi107 running (2m) 15s ago 2m 17.1M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 83d74619fe7a 2024-09-21T09:43:49.885 INFO:teuthology.orchestra.run.smithi107.stdout:mds.foo.smithi129.zdzmqr smithi129 running (2m) 12s ago 2m 14.4M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 2133ce1b60f9 2024-09-21T09:43:49.885 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.x smithi129 *:8443,9283 running (19s) 12s ago 7m 64.8M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 dc5e5c81266c 2024-09-21T09:43:49.885 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.y smithi107 *:9283,8765 running (9m) 15s ago 9m 452M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 5ab1b97eb209 2024-09-21T09:43:49.885 INFO:teuthology.orchestra.run.smithi107.stdout:mon.a smithi107 running (9m) 15s ago 9m 52.3M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 7135d1946aea 2024-09-21T09:43:49.885 INFO:teuthology.orchestra.run.smithi107.stdout:mon.b smithi129 running (8m) 12s ago 8m 43.4M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 589a4d4f8d6b 2024-09-21T09:43:49.885 INFO:teuthology.orchestra.run.smithi107.stdout:mon.c smithi107 running (8m) 15s ago 8m 41.6M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 4d41a86eb0a8 2024-09-21T09:43:49.885 INFO:teuthology.orchestra.run.smithi107.stdout:node-exporter.a smithi107 *:9100 running (3m) 15s ago 3m 16.0M - 1.5.0 0da6a335fe13 bbf92f500b2c 2024-09-21T09:43:49.885 INFO:teuthology.orchestra.run.smithi107.stdout:node-exporter.b smithi129 *:9100 running (3m) 12s ago 3m 16.2M - 1.5.0 0da6a335fe13 a71c52ae1a69 2024-09-21T09:43:49.885 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 smithi107 running (7m) 15s ago 7m 343M 4015M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 96047e77ae78 2024-09-21T09:43:49.886 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1 smithi107 running (7m) 15s ago 7m 230M 4015M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 508718820c25 2024-09-21T09:43:49.886 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2 smithi107 running (6m) 15s ago 6m 283M 4015M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be ae11dfe6a988 2024-09-21T09:43:49.886 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3 smithi107 running (6m) 15s ago 6m 155M 4015M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be cf6bac271e88 2024-09-21T09:43:49.886 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4 smithi129 running (6m) 12s ago 6m 340M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 7c6ca82e1ac1 2024-09-21T09:43:49.886 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5 smithi129 running (5m) 12s ago 5m 372M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 9ae131a2ef13 2024-09-21T09:43:49.886 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6 smithi129 running (5m) 12s ago 5m 191M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be a5ccfa3e24c2 2024-09-21T09:43:49.886 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7 smithi129 running (4m) 12s ago 4m 233M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be a32c9bd56124 2024-09-21T09:43:49.886 INFO:teuthology.orchestra.run.smithi107.stdout:prometheus.a smithi129 *:9095 running (2m) 12s ago 4m 55.9M - 2.43.0 a07b618ecd1d aa11510fd428 2024-09-21T09:43:49.910 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:49 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:49.651+0000 7f6cbf94cec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:43:51.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:50 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:50.819+0000 7f6cbf94cec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:43:51.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:50 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:43:50] ENGINE Bus STARTING 2024-09-21T09:43:51.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:50 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: CherryPy Checker: 2024-09-21T09:43:51.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:50 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: The Application mounted at '' has an empty config. 2024-09-21T09:43:51.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:50 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: 2024-09-21T09:43:51.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:50 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:43:50] ENGINE Serving on http://:::9283 2024-09-21T09:43:51.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:50 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: [21/Sep/2024:09:43:50] ENGINE Bus STARTED 2024-09-21T09:43:51.732 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:51 smithi129 ceph-mon[32729]: from='client.24913 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:51.732 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:51 smithi129 ceph-mon[32729]: pgmap v6: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail 2024-09-21T09:43:51.732 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:51 smithi129 ceph-mon[32729]: mgrmap e25: x(active, since 6s) 2024-09-21T09:43:51.732 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:51 smithi129 ceph-mon[32729]: Standby manager daemon y started 2024-09-21T09:43:51.732 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:51 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.107:0/3266442088' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:43:51.732 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:51 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.107:0/3266442088' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:43:51.732 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:51 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.107:0/3266442088' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:43:51.732 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:51 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.107:0/3266442088' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:43:51.737 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:51 smithi107 ceph-mon[29382]: from='client.24913 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:51.737 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:51 smithi107 ceph-mon[29382]: pgmap v6: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail 2024-09-21T09:43:51.738 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:51 smithi107 ceph-mon[29382]: mgrmap e25: x(active, since 6s) 2024-09-21T09:43:51.738 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:51 smithi107 ceph-mon[29382]: Standby manager daemon y started 2024-09-21T09:43:51.738 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:51 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.107:0/3266442088' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:43:51.738 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:51 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.107:0/3266442088' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:43:51.738 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:51 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.107:0/3266442088' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:43:51.738 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:51 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.107:0/3266442088' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:43:51.738 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:51 smithi107 ceph-mon[41185]: from='client.24913 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:51.738 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:51 smithi107 ceph-mon[41185]: pgmap v6: 89 pgs: 89 active+clean; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail 2024-09-21T09:43:51.738 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:51 smithi107 ceph-mon[41185]: mgrmap e25: x(active, since 6s) 2024-09-21T09:43:51.738 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:51 smithi107 ceph-mon[41185]: Standby manager daemon y started 2024-09-21T09:43:51.738 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:51 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.107:0/3266442088' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:43:51.738 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:51 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.107:0/3266442088' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:43:51.739 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:51 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.107:0/3266442088' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:43:51.739 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:51 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.107:0/3266442088' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:43:52.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade pause' 2024-09-21T09:43:52.746 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:43:52.826 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:52 smithi107 ceph-mon[41185]: mgrmap e26: x(active, since 7s), standbys: y 2024-09-21T09:43:52.826 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:52 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:43:52.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:52 smithi107 ceph-mon[29382]: mgrmap e26: x(active, since 7s), standbys: y 2024-09-21T09:43:52.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:52 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:43:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:52 smithi129 ceph-mon[32729]: mgrmap e26: x(active, since 7s), standbys: y 2024-09-21T09:43:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:52 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:43:53.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:53 smithi129 ceph-mon[32729]: pgmap v7: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 26 KiB/s rd, 458 B/s wr, 594 op/s 2024-09-21T09:43:53.754 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:53 smithi107 ceph-mon[29382]: pgmap v7: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 26 KiB/s rd, 458 B/s wr, 594 op/s 2024-09-21T09:43:53.754 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:53 smithi107 ceph-mon[41185]: pgmap v7: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 26 KiB/s rd, 458 B/s wr, 594 op/s 2024-09-21T09:43:55.522 INFO:teuthology.orchestra.run.smithi107.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:43:55.650 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:55 smithi129 ceph-mon[32729]: pgmap v8: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 20 KiB/s rd, 353 B/s wr, 458 op/s 2024-09-21T09:43:55.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:55 smithi107 ceph-mon[29382]: pgmap v8: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 20 KiB/s rd, 353 B/s wr, 458 op/s 2024-09-21T09:43:55.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:55 smithi107 ceph-mon[41185]: pgmap v8: 89 pgs: 89 active+clean; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 20 KiB/s rd, 353 B/s wr, 458 op/s 2024-09-21T09:43:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:56 smithi107 ceph-mon[29382]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:56 smithi107 ceph-mon[29382]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:43:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:56 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:56 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:56 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:56 smithi107 ceph-mon[41185]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:56 smithi107 ceph-mon[41185]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:43:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:56 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:56 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:56.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:56 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:56 smithi129 ceph-mon[32729]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:43:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:56 smithi129 ceph-mon[32729]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:43:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:56 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:56 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:56 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:56.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'sleep 60' 2024-09-21T09:43:57.527 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:57 smithi129 ceph-mon[32729]: pgmap v9: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 16 KiB/s rd, 287 B/s wr, 781 op/s 2024-09-21T09:43:57.527 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:57 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:57.528 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:57 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:57.528 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:43:57 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:43:57] "GET /metrics HTTP/1.1" 200 36134 "" "Prometheus/2.43.0" 2024-09-21T09:43:57.760 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:43:57.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:57 smithi107 ceph-mon[29382]: pgmap v9: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 16 KiB/s rd, 287 B/s wr, 781 op/s 2024-09-21T09:43:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:57 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:57 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:57.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:57 smithi107 ceph-mon[41185]: pgmap v9: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 16 KiB/s rd, 287 B/s wr, 781 op/s 2024-09-21T09:43:57.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:57 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:57.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:57 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:59.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:58 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:59.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:58 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:59.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:43:58 smithi107 ceph-mon[29382]: pgmap v10: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 696 op/s 2024-09-21T09:43:59.042 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:58 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:59.042 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:58 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:59.042 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:43:58 smithi107 ceph-mon[41185]: pgmap v10: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 696 op/s 2024-09-21T09:43:59.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:58 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:59.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:58 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:43:59.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:43:58 smithi129 ceph-mon[32729]: pgmap v10: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 696 op/s 2024-09-21T09:43:59.358 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:43:59 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:43:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:44:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:00 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:00 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:00 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:00 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:00 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:00 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:00 smithi107 ceph-mon[29382]: Adjusting osd_memory_target on smithi107 to 2479M 2024-09-21T09:44:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:00 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:00.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:00 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:00.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:00 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:00.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:00 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:00.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:00 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:00.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:00 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:00.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:00 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:00.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:00 smithi107 ceph-mon[41185]: Adjusting osd_memory_target on smithi107 to 2479M 2024-09-21T09:44:00.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:00 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:00 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:00 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:00.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:00 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:00.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:00 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:00.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:00 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:00.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:00 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:00.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:00 smithi129 ceph-mon[32729]: Adjusting osd_memory_target on smithi107 to 2479M 2024-09-21T09:44:00.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:00 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:01.458 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:01 smithi129 ceph-mon[32729]: pgmap v11: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 696 op/s 2024-09-21T09:44:01.458 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:01 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:44:01.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:01 smithi107 ceph-mon[41185]: pgmap v11: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 696 op/s 2024-09-21T09:44:01.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:01 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:44:01.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:01 smithi107 ceph-mon[29382]: pgmap v11: 89 pgs: 89 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 696 op/s 2024-09-21T09:44:01.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:01 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:44:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[29382]: pgmap v12: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 1.07k op/s 2024-09-21T09:44:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[29382]: Adjusting osd_memory_target on smithi129 to 2480M 2024-09-21T09:44:03.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:03.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:44:03.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:44:03.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[29382]: Updating smithi107:/etc/ceph/ceph.conf 2024-09-21T09:44:03.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[29382]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T09:44:03.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[29382]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:44:03.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[29382]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:44:03.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[29382]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:44:03.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:03.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:03.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[41185]: pgmap v12: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 1.07k op/s 2024-09-21T09:44:03.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:03.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:03.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:03.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:03.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:03.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:03.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[41185]: Adjusting osd_memory_target on smithi129 to 2480M 2024-09-21T09:44:03.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:03.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:44:03.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:44:03.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[41185]: Updating smithi107:/etc/ceph/ceph.conf 2024-09-21T09:44:03.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[41185]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T09:44:03.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[41185]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:44:03.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[41185]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:44:03.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[41185]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:44:03.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:03.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:03 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:03 smithi129 ceph-mon[32729]: pgmap v12: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 1.07k op/s 2024-09-21T09:44:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:03 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:03 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:03 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:03 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:03 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:03 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:44:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:03 smithi129 ceph-mon[32729]: Adjusting osd_memory_target on smithi129 to 2480M 2024-09-21T09:44:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:03 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:03 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:44:03.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:03 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:44:03.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:03 smithi129 ceph-mon[32729]: Updating smithi107:/etc/ceph/ceph.conf 2024-09-21T09:44:03.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:03 smithi129 ceph-mon[32729]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T09:44:03.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:03 smithi129 ceph-mon[32729]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:44:03.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:03 smithi129 ceph-mon[32729]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:44:03.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:03 smithi129 ceph-mon[32729]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:44:03.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:03 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:03.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:03 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[29382]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:44:04.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[29382]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:44:04.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[29382]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:44:04.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[29382]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:44:04.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[29382]: Deploying daemon alertmanager.a on smithi107 2024-09-21T09:44:04.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[41185]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:44:04.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[41185]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:44:04.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[41185]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:44:04.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[41185]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:44:04.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:04 smithi107 ceph-mon[41185]: Deploying daemon alertmanager.a on smithi107 2024-09-21T09:44:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:04 smithi129 ceph-mon[32729]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:44:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:04 smithi129 ceph-mon[32729]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:44:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:04 smithi129 ceph-mon[32729]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:44:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:04 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:04 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:04 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:04 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:04 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:04 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:04 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:04.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:04 smithi129 ceph-mon[32729]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:44:04.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:04 smithi129 ceph-mon[32729]: Deploying daemon alertmanager.a on smithi107 2024-09-21T09:44:05.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:05 smithi107 ceph-mon[29382]: pgmap v13: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 737 op/s 2024-09-21T09:44:05.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:05 smithi107 ceph-mon[41185]: pgmap v13: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 737 op/s 2024-09-21T09:44:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:05 smithi129 ceph-mon[32729]: pgmap v13: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 737 op/s 2024-09-21T09:44:07.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:07 smithi129 ceph-mon[32729]: pgmap v14: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-21T09:44:07.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:44:07 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:44:07] "GET /metrics HTTP/1.1" 200 36417 "" "Prometheus/2.43.0" 2024-09-21T09:44:07.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:07 smithi107 ceph-mon[29382]: pgmap v14: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-21T09:44:07.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:07 smithi107 ceph-mon[41185]: pgmap v14: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-21T09:44:09.359 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:44:09 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:44:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:44:09.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:09 smithi129 ceph-mon[32729]: pgmap v15: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 747 op/s 2024-09-21T09:44:09.695 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:09 smithi107 ceph-mon[29382]: pgmap v15: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 747 op/s 2024-09-21T09:44:09.696 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:09 smithi107 ceph-mon[41185]: pgmap v15: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 747 op/s 2024-09-21T09:44:11.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:11 smithi107 ceph-mon[29382]: pgmap v16: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 747 op/s 2024-09-21T09:44:11.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:11 smithi107 ceph-mon[41185]: pgmap v16: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 747 op/s 2024-09-21T09:44:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:11 smithi129 ceph-mon[32729]: pgmap v16: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 747 op/s 2024-09-21T09:44:13.516 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:13 smithi107 systemd[1]: Stopping Ceph alertmanager.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:44:13.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:13 smithi107 ceph-mon[29382]: pgmap v17: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-21T09:44:13.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:13 smithi107 ceph-mon[41185]: pgmap v17: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-21T09:44:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:13 smithi129 ceph-mon[32729]: pgmap v17: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s 2024-09-21T09:44:14.804 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:14 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[77670]: ts=2024-09-21T09:44:14.436Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T09:44:14.804 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:14 smithi107 podman[90997]: 2024-09-21 09:44:14.48106325 +0000 UTC m=+0.992959108 container died 432a2da9f5f4dcda926c9f7b58b22c58f96f56ffac91892a9e35757173f0e72c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:44:15.108 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:14 smithi107 podman[90997]: 2024-09-21 09:44:14.892152388 +0000 UTC m=+1.404048236 container cleanup 432a2da9f5f4dcda926c9f7b58b22c58f96f56ffac91892a9e35757173f0e72c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:44:15.108 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:14 smithi107 bash[90997]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T09:44:15.520 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:15 smithi107 podman[91012]: 2024-09-21 09:44:15.227910024 +0000 UTC m=+0.729147779 container remove 432a2da9f5f4dcda926c9f7b58b22c58f96f56ffac91892a9e35757173f0e72c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:44:15.520 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:15 smithi107 podman[91012]: 2024-09-21 09:44:15.320738278 +0000 UTC m=+0.821976031 volume remove af5e4e3d1e3802948cad151277808d8488b468fb2652eaaeb3e027274a836e32 2024-09-21T09:44:15.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:15 smithi107 ceph-mon[29382]: pgmap v18: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 749 op/s 2024-09-21T09:44:15.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:15 smithi107 ceph-mon[41185]: pgmap v18: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 749 op/s 2024-09-21T09:44:15.859 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:15 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T09:44:15.859 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:15 smithi107 systemd[1]: Stopped Ceph alertmanager.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:44:15.859 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:15 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@alertmanager.a.service: Consumed 1.395s CPU time. 2024-09-21T09:44:15.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:15 smithi129 ceph-mon[32729]: pgmap v18: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 749 op/s 2024-09-21T09:44:16.243 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:15 smithi107 systemd[1]: Starting Ceph alertmanager.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:44:16.530 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:16 smithi107 podman[91133]: 2024-09-21 09:44:16.145713711 +0000 UTC m=+0.038329490 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-21T09:44:16.530 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:16 smithi107 podman[91133]: 2024-09-21 09:44:16.268839568 +0000 UTC m=+0.161455327 volume create b5fffda4bb334deca3d69d6b2dba52b4fce6f6cbaeb0564ca9a4209071007a44 2024-09-21T09:44:16.530 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:16 smithi107 podman[91133]: 2024-09-21 09:44:16.381076997 +0000 UTC m=+0.273692750 container create 6fa4ebf1f69dc6a718c10a6f65067b5566505bfac921cba11285301c58e901c7 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:44:16.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:16 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:44:16.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:16 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:44:16.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:16 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:44:17.109 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:16 smithi107 podman[91133]: 2024-09-21 09:44:16.793528942 +0000 UTC m=+0.686144698 container init 6fa4ebf1f69dc6a718c10a6f65067b5566505bfac921cba11285301c58e901c7 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:44:17.109 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:16 smithi107 podman[91133]: 2024-09-21 09:44:16.801155996 +0000 UTC m=+0.693771757 container start 6fa4ebf1f69dc6a718c10a6f65067b5566505bfac921cba11285301c58e901c7 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:44:17.109 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:16 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[91145]: ts=2024-09-21T09:44:16.817Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-21T09:44:17.109 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:16 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[91145]: ts=2024-09-21T09:44:16.818Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-21T09:44:17.109 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:16 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[91145]: ts=2024-09-21T09:44:16.824Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.107 port=9094 2024-09-21T09:44:17.109 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:16 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[91145]: ts=2024-09-21T09:44:16.829Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-21T09:44:17.109 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:16 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[91145]: ts=2024-09-21T09:44:16.855Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T09:44:17.109 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:16 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[91145]: ts=2024-09-21T09:44:16.856Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T09:44:17.109 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:16 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[91145]: ts=2024-09-21T09:44:16.860Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-21T09:44:17.110 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:16 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[91145]: ts=2024-09-21T09:44:16.860Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-21T09:44:17.110 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:16 smithi107 bash[91133]: 6fa4ebf1f69dc6a718c10a6f65067b5566505bfac921cba11285301c58e901c7 2024-09-21T09:44:17.110 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:16 smithi107 systemd[1]: Started Ceph alertmanager.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:44:17.535 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:44:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:44:17] "GET /metrics HTTP/1.1" 200 36413 "" "Prometheus/2.43.0" 2024-09-21T09:44:17.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:17 smithi107 ceph-mon[29382]: pgmap v19: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-09-21T09:44:17.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:17 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:17.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:17 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:17.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:17 smithi107 ceph-mon[41185]: pgmap v19: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-09-21T09:44:17.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:17 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:17.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:17 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:17.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:17 smithi129 ceph-mon[32729]: pgmap v19: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-09-21T09:44:17.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:17 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:17.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:17 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:18.832 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:18 smithi107 ceph-mon[29382]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-21T09:44:18.832 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:18 smithi107 ceph-mon[29382]: Deploying daemon node-exporter.a on smithi107 2024-09-21T09:44:18.832 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:18 smithi107 ceph-mon[41185]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-21T09:44:18.833 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:18 smithi107 ceph-mon[41185]: Deploying daemon node-exporter.a on smithi107 2024-09-21T09:44:18.833 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:18 smithi107 systemd[1]: Stopping Ceph node-exporter.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:44:18.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:18 smithi129 ceph-mon[32729]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-21T09:44:18.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:18 smithi129 ceph-mon[32729]: Deploying daemon node-exporter.a on smithi107 2024-09-21T09:44:19.108 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:18 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[91145]: ts=2024-09-21T09:44:18.830Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000497955s 2024-09-21T09:44:19.546 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:44:19 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:44:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:44:19.547 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:19 smithi107 podman[91291]: 2024-09-21 09:44:19.364332161 +0000 UTC m=+0.613383911 container died bbf92f500b2cb4457383b94e39ef31e0a898876384ad8f7997a5414928992207 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:44:19.819 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:19 smithi129 ceph-mon[32729]: pgmap v20: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-09-21T09:44:19.856 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:19 smithi107 ceph-mon[29382]: pgmap v20: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-09-21T09:44:19.856 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:19 smithi107 ceph-mon[41185]: pgmap v20: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-09-21T09:44:20.609 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:20 smithi107 podman[91291]: 2024-09-21 09:44:20.109414059 +0000 UTC m=+1.358465806 container cleanup bbf92f500b2cb4457383b94e39ef31e0a898876384ad8f7997a5414928992207 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:44:20.609 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:20 smithi107 bash[91291]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T09:44:20.609 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:20 smithi107 podman[91307]: 2024-09-21 09:44:20.320182746 +0000 UTC m=+0.953615987 container remove bbf92f500b2cb4457383b94e39ef31e0a898876384ad8f7997a5414928992207 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:44:20.609 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:20 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T09:44:21.004 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:20 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-21T09:44:21.004 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:20 smithi107 systemd[1]: Stopped Ceph node-exporter.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:44:21.004 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:20 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@node-exporter.a.service: Consumed 2.544s CPU time. 2024-09-21T09:44:21.358 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:21 smithi107 systemd[1]: Starting Ceph node-exporter.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:44:21.705 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:21 smithi107 ceph-mon[41185]: pgmap v21: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-09-21T09:44:21.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:21 smithi107 ceph-mon[29382]: pgmap v21: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-09-21T09:44:21.705 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:21 smithi107 bash[91428]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-21T09:44:21.705 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:21 smithi107 bash[91428]: Getting image source signatures 2024-09-21T09:44:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:21 smithi129 ceph-mon[32729]: pgmap v21: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 756 op/s 2024-09-21T09:44:22.017 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:21 smithi107 bash[91428]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-21T09:44:22.017 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:21 smithi107 bash[91428]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-21T09:44:22.017 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:21 smithi107 bash[91428]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-21T09:44:23.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:23 smithi107 ceph-mon[29382]: pgmap v22: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-09-21T09:44:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:23 smithi107 ceph-mon[41185]: pgmap v22: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-09-21T09:44:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:23 smithi129 ceph-mon[32729]: pgmap v22: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-09-21T09:44:24.358 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:24 smithi107 bash[91428]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-21T09:44:24.732 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:24 smithi107 bash[91428]: Writing manifest to image destination 2024-09-21T09:44:24.983 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:24 smithi107 podman[91428]: 2024-09-21 09:44:24.714624972 +0000 UTC m=+3.522312420 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-21T09:44:24.984 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:24 smithi107 podman[91428]: 2024-09-21 09:44:24.837420513 +0000 UTC m=+3.645107949 container create 2444eabb7c7e00cc120684747c3c88e9900cb82db45b8588bc340dc25f7a957e (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:44:25.563 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 podman[91428]: 2024-09-21 09:44:25.124525543 +0000 UTC m=+3.932212983 container init 2444eabb7c7e00cc120684747c3c88e9900cb82db45b8588bc340dc25f7a957e (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:44:25.563 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 podman[91428]: 2024-09-21 09:44:25.136144587 +0000 UTC m=+3.943832026 container start 2444eabb7c7e00cc120684747c3c88e9900cb82db45b8588bc340dc25f7a957e (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:44:25.563 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.139Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-21T09:44:25.563 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.139Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-21T09:44:25.563 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.140Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T09:44:25.563 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.140Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T09:44:25.563 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.141Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T09:44:25.564 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.141Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T09:44:25.564 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.141Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T09:44:25.564 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.141Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T09:44:25.564 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.141Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T09:44:25.564 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.141Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T09:44:25.564 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.141Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T09:44:25.564 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.141Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T09:44:25.564 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.141Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T09:44:25.564 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.141Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T09:44:25.564 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.141Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T09:44:25.564 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.141Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T09:44:25.564 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.141Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T09:44:25.565 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.141Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T09:44:25.565 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.141Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T09:44:25.565 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.141Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T09:44:25.565 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.141Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T09:44:25.565 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.141Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T09:44:25.565 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.141Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T09:44:25.565 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.141Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T09:44:25.565 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T09:44:25.565 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T09:44:25.565 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T09:44:25.565 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T09:44:25.565 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T09:44:25.565 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T09:44:25.566 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T09:44:25.566 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T09:44:25.566 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T09:44:25.566 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T09:44:25.566 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T09:44:25.566 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T09:44:25.566 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T09:44:25.566 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T09:44:25.566 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T09:44:25.566 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T09:44:25.566 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T09:44:25.567 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T09:44:25.567 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T09:44:25.567 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T09:44:25.567 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T09:44:25.567 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T09:44:25.567 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T09:44:25.567 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T09:44:25.567 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T09:44:25.567 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T09:44:25.567 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.142Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T09:44:25.567 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.143Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-21T09:44:25.567 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a[91506]: ts=2024-09-21T09:44:25.143Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T09:44:25.567 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 bash[91428]: 2444eabb7c7e00cc120684747c3c88e9900cb82db45b8588bc340dc25f7a957e 2024-09-21T09:44:25.568 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 09:44:25 smithi107 systemd[1]: Started Ceph node-exporter.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:44:25.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-mon[29382]: pgmap v23: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 760 op/s 2024-09-21T09:44:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-mon[41185]: pgmap v23: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 760 op/s 2024-09-21T09:44:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:25 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:25 smithi129 ceph-mon[32729]: pgmap v23: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 760 op/s 2024-09-21T09:44:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:25 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:25 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:26.835 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:26 smithi107 ceph-mon[29382]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:44:26.835 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:26 smithi107 ceph-mon[29382]: Deploying daemon prometheus.a on smithi129 2024-09-21T09:44:26.835 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:26 smithi107 ceph-mon[41185]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:44:26.835 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:26 smithi107 ceph-mon[41185]: Deploying daemon prometheus.a on smithi129 2024-09-21T09:44:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:26 smithi129 ceph-mon[32729]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:44:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:26 smithi129 ceph-mon[32729]: Deploying daemon prometheus.a on smithi129 2024-09-21T09:44:27.108 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 09:44:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[91145]: ts=2024-09-21T09:44:26.833Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003901344s 2024-09-21T09:44:27.571 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:44:27 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:44:27] "GET /metrics HTTP/1.1" 200 36413 "" "Prometheus/2.43.0" 2024-09-21T09:44:27.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:27 smithi107 ceph-mon[29382]: pgmap v24: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-09-21T09:44:27.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:27 smithi107 ceph-mon[41185]: pgmap v24: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-09-21T09:44:27.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:27 smithi129 ceph-mon[32729]: pgmap v24: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-09-21T09:44:29.576 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:44:29 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:44:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:44:29.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:29 smithi107 ceph-mon[29382]: pgmap v25: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 746 op/s 2024-09-21T09:44:29.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:29 smithi107 ceph-mon[41185]: pgmap v25: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 746 op/s 2024-09-21T09:44:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:29 smithi129 ceph-mon[32729]: pgmap v25: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 746 op/s 2024-09-21T09:44:31.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:31 smithi107 ceph-mon[29382]: pgmap v26: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 746 op/s 2024-09-21T09:44:31.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:31 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:44:31.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:31 smithi107 ceph-mon[41185]: pgmap v26: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 746 op/s 2024-09-21T09:44:31.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:31 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:44:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:31 smithi129 ceph-mon[32729]: pgmap v26: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 746 op/s 2024-09-21T09:44:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:31 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:44:33.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:33 smithi107 ceph-mon[29382]: pgmap v27: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-21T09:44:33.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:33 smithi107 ceph-mon[41185]: pgmap v27: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-21T09:44:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:33 smithi129 ceph-mon[32729]: pgmap v27: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s 2024-09-21T09:44:35.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:35 smithi107 ceph-mon[29382]: pgmap v28: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 727 op/s 2024-09-21T09:44:35.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:35 smithi107 ceph-mon[41185]: pgmap v28: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 727 op/s 2024-09-21T09:44:35.868 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:35 smithi129 ceph-mon[32729]: pgmap v28: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 727 op/s 2024-09-21T09:44:37.442 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:44:37 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:44:37] "GET /metrics HTTP/1.1" 200 36411 "" "Prometheus/2.43.0" 2024-09-21T09:44:37.855 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:37 smithi129 ceph-mon[32729]: pgmap v29: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T09:44:37.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:37 smithi107 ceph-mon[41185]: pgmap v29: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T09:44:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:37 smithi107 ceph-mon[29382]: pgmap v29: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T09:44:39.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:38 smithi129 systemd[1]: Stopping Ceph prometheus.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:44:39.604 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:44:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[29633]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:44:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:44:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:39 smithi129 ceph-mon[32729]: pgmap v30: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 709 op/s 2024-09-21T09:44:39.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:39 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:44:39.370Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T09:44:39.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:39 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:44:39.371Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-21T09:44:39.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:39 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:44:39.371Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-21T09:44:39.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:39 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:44:39.371Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T09:44:39.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:39 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:44:39.371Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-21T09:44:39.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:39 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:44:39.371Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-21T09:44:39.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:39 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:44:39.371Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T09:44:39.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:39 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:44:39.371Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-21T09:44:39.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:39 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:44:39.371Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-21T09:44:39.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:39 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:44:39.371Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T09:44:39.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:39 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:44:39.372Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-21T09:44:39.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:39 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[65390]: ts=2024-09-21T09:44:39.372Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-21T09:44:39.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:39 smithi129 podman[76686]: 2024-09-21 09:44:39.390331407 +0000 UTC m=+0.175194847 container died aa11510fd42834986b97cb1bc5145db901cf509160d2adf5463522e50e000a8f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:44:39.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:39 smithi107 ceph-mon[29382]: pgmap v30: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 709 op/s 2024-09-21T09:44:39.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:39 smithi107 ceph-mon[41185]: pgmap v30: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 709 op/s 2024-09-21T09:44:40.038 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:39 smithi129 podman[76686]: 2024-09-21 09:44:39.76567757 +0000 UTC m=+0.550541006 container cleanup aa11510fd42834986b97cb1bc5145db901cf509160d2adf5463522e50e000a8f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:44:40.038 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:39 smithi129 bash[76686]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T09:44:40.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:40 smithi129 podman[76705]: 2024-09-21 09:44:40.036609307 +0000 UTC m=+0.642154849 container remove aa11510fd42834986b97cb1bc5145db901cf509160d2adf5463522e50e000a8f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:44:40.796 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:40 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T09:44:40.797 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:40 smithi129 systemd[1]: Stopped Ceph prometheus.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:44:40.797 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:40 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 2.183s CPU time. 2024-09-21T09:44:41.137 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:40 smithi129 systemd[1]: Starting Ceph prometheus.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:44:41.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 podman[76825]: 2024-09-21 09:44:41.044901324 +0000 UTC m=+0.034093399 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-21T09:44:41.749 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-mon[32729]: pgmap v31: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 709 op/s 2024-09-21T09:44:41.749 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 podman[76825]: 2024-09-21 09:44:41.469363532 +0000 UTC m=+0.458555594 container create 9fccc20685963d81891b9c16e7d81c29d08e574d44046bdbf86bd1bba19f7c38 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:44:42.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:41 smithi107 ceph-mon[29382]: pgmap v31: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 709 op/s 2024-09-21T09:44:42.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:41 smithi107 ceph-mon[41185]: pgmap v31: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 709 op/s 2024-09-21T09:44:42.172 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 podman[76825]: 2024-09-21 09:44:41.747564159 +0000 UTC m=+0.736756234 container init 9fccc20685963d81891b9c16e7d81c29d08e574d44046bdbf86bd1bba19f7c38 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:44:42.173 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 podman[76825]: 2024-09-21 09:44:41.757537614 +0000 UTC m=+0.746729678 container start 9fccc20685963d81891b9c16e7d81c29d08e574d44046bdbf86bd1bba19f7c38 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:44:42.173 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.799Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-21T09:44:42.173 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.799Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-21T09:44:42.173 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.799Z caller=main.go:623 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi129 (none))" 2024-09-21T09:44:42.173 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.800Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T09:44:42.173 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.800Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T09:44:42.173 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.802Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T09:44:42.173 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.803Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-21T09:44:42.173 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.804Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T09:44:42.173 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.804Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T09:44:42.173 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.807Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T09:44:42.173 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.808Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.715µs 2024-09-21T09:44:42.173 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.808Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T09:44:42.174 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.810Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-09-21T09:44:42.174 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 bash[76825]: 9fccc20685963d81891b9c16e7d81c29d08e574d44046bdbf86bd1bba19f7c38 2024-09-21T09:44:42.174 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 systemd[1]: Started Ceph prometheus.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:44:42.174 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.864Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-09-21T09:44:42.174 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.865Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-09-21T09:44:42.174 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.865Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=39.798µs wal_replay_duration=57.606ms wbl_replay_duration=233ns total_replay_duration=57.719546ms 2024-09-21T09:44:42.174 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.870Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T09:44:42.174 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.871Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-21T09:44:42.174 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.871Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T09:44:42.174 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.896Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=25.224988ms db_storage=1.062µs remote_storage=1.489µs web_handler=785ns query_engine=826ns scrape=621.618µs scrape_sd=179.218µs notify=22.523µs notify_sd=1.124451ms rules=22.712532ms tracing=8.9µs 2024-09-21T09:44:42.174 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.896Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-21T09:44:42.174 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:41.896Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T09:44:43.312 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:42 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:43.312 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:42 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:43.312 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:42 smithi129 ceph-mon[32729]: Reconfiguring grafana.a (dependencies changed)... 2024-09-21T09:44:43.312 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:42 smithi129 ceph-mon[32729]: Reconfiguring daemon grafana.a on smithi129 2024-09-21T09:44:43.313 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:42 smithi129 ceph-mon[32729]: pgmap v32: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T09:44:43.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:42 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:43.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:42 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:43.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:42 smithi107 ceph-mon[29382]: Reconfiguring grafana.a (dependencies changed)... 2024-09-21T09:44:43.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:42 smithi107 ceph-mon[29382]: Reconfiguring daemon grafana.a on smithi129 2024-09-21T09:44:43.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:42 smithi107 ceph-mon[29382]: pgmap v32: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T09:44:43.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:42 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:43.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:42 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:43.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:42 smithi107 ceph-mon[41185]: Reconfiguring grafana.a (dependencies changed)... 2024-09-21T09:44:43.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:42 smithi107 ceph-mon[41185]: Reconfiguring daemon grafana.a on smithi129 2024-09-21T09:44:43.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:42 smithi107 ceph-mon[41185]: pgmap v32: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T09:44:45.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:45 smithi107 ceph-mon[29382]: pgmap v33: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 704 op/s 2024-09-21T09:44:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:45 smithi107 ceph-mon[41185]: pgmap v33: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 704 op/s 2024-09-21T09:44:45.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:45 smithi129 ceph-mon[32729]: pgmap v33: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 704 op/s 2024-09-21T09:44:46.168 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:45 smithi129 systemd[1]: Stopping Ceph grafana.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:44:46.466 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:46 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:44:46.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:46 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:44:46.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:46 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:44:46.919 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:46 smithi129 podman[77059]: 2024-09-21 09:44:46.464905897 +0000 UTC m=+0.365788618 container stop 70638332abb2421839ee027825ee70c2ea7a523a7e53dbcd5de3c017139084c9 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a, io.buildah.version=1.32.0, name=ubi8, io.openshift.tags=base rhel8, release=239.1651231664, com.redhat.component=ubi8-container, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vcs-type=git, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., description=Ceph Grafana Container, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-09-21T09:44:46.919 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:46 smithi129 podman[77059]: 2024-09-21 09:44:46.469785289 +0000 UTC m=+0.370667996 container died 70638332abb2421839ee027825ee70c2ea7a523a7e53dbcd5de3c017139084c9 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , architecture=x86_64, version=8.5, distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, build-date=2022-04-29T11:48:11.055897, name=ubi8, vendor=Red Hat, Inc., vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container) 2024-09-21T09:44:47.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:47 smithi107 ceph-mon[29382]: pgmap v34: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T09:44:47.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:47 smithi107 ceph-mon[41185]: pgmap v34: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T09:44:47.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:47 smithi129 ceph-mon[32729]: pgmap v34: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T09:44:48.169 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:48 smithi129 podman[77059]: 2024-09-21 09:44:48.013973837 +0000 UTC m=+1.914856540 container cleanup 70638332abb2421839ee027825ee70c2ea7a523a7e53dbcd5de3c017139084c9 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a, name=ubi8, vendor=Red Hat, Inc., description=Ceph Grafana Container, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.openshift.expose-services=, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, version=8.5, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, build-date=2022-04-29T11:48:11.055897) 2024-09-21T09:44:48.169 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:48 smithi129 bash[77059]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a 2024-09-21T09:44:48.598 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:48 smithi129 podman[77074]: 2024-09-21 09:44:48.259480819 +0000 UTC m=+1.775710059 container remove 70638332abb2421839ee027825ee70c2ea7a523a7e53dbcd5de3c017139084c9 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a, version=8.5, description=Ceph Grafana Container, distribution-scope=public, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, maintainer=Paul Cuzner , architecture=x86_64) 2024-09-21T09:44:48.598 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:48 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T09:44:48.919 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:48 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@grafana.a.service: Failed with result 'exit-code'. 2024-09-21T09:44:48.919 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:48 smithi129 systemd[1]: Stopped Ceph grafana.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:44:48.919 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:48 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@grafana.a.service: Consumed 6.125s CPU time. 2024-09-21T09:44:48.919 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:48 smithi129 systemd[1]: Starting Ceph grafana.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:44:49.419 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:44:49] "GET /metrics HTTP/1.1" 200 36412 "" "Prometheus/2.51.0" 2024-09-21T09:44:49.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-mon[32729]: pgmap v35: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 712 op/s 2024-09-21T09:44:49.419 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 podman[77154]: 2024-09-21 09:44:48.959511888 +0000 UTC m=+0.038016629 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-21T09:44:49.420 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 podman[77154]: 2024-09-21 09:44:49.093863676 +0000 UTC m=+0.172368418 container create 72cc8ff672f5005bafc273cf66b2b7719280a74d029925698dac30d555e4c837 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, version=8.5, maintainer=Paul Cuzner , description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=239.1651231664, name=ubi8, build-date=2022-04-29T11:48:11.055897, distribution-scope=public, vendor=Red Hat, Inc., vcs-type=git, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.buildah.version=1.32.0) 2024-09-21T09:44:49.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:49 smithi107 ceph-mon[29382]: pgmap v35: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 712 op/s 2024-09-21T09:44:49.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:49 smithi107 ceph-mon[41185]: pgmap v35: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 712 op/s 2024-09-21T09:44:49.754 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 podman[77154]: 2024-09-21 09:44:49.422748133 +0000 UTC m=+0.501252881 container init 72cc8ff672f5005bafc273cf66b2b7719280a74d029925698dac30d555e4c837 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, maintainer=Paul Cuzner , vcs-type=git, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, name=ubi8, io.openshift.expose-services=, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-09-21T09:44:49.754 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 podman[77154]: 2024-09-21 09:44:49.43545523 +0000 UTC m=+0.513959966 container start 72cc8ff672f5005bafc273cf66b2b7719280a74d029925698dac30d555e4c837 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a, maintainer=Paul Cuzner , vcs-type=git, name=ubi8, io.openshift.expose-services=, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=239.1651231664, io.buildah.version=1.32.0, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, io.openshift.tags=base rhel8, version=8.5, build-date=2022-04-29T11:48:11.055897, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-09-21T09:44:49.754 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 bash[77154]: 72cc8ff672f5005bafc273cf66b2b7719280a74d029925698dac30d555e4c837 2024-09-21T09:44:49.754 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 systemd[1]: Started Ceph grafana.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:44:50.085 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=settings t=2024-09-21T09:44:49.751830878Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-21T09:44:50.085 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=settings t=2024-09-21T09:44:49.752083217Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-21T09:44:50.085 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=settings t=2024-09-21T09:44:49.752108538Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-21T09:44:50.085 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=settings t=2024-09-21T09:44:49.752119302Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-21T09:44:50.085 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=settings t=2024-09-21T09:44:49.752128547Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-21T09:44:50.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=settings t=2024-09-21T09:44:49.752137601Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-21T09:44:50.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=settings t=2024-09-21T09:44:49.752146409Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-21T09:44:50.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=settings t=2024-09-21T09:44:49.752155429Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-21T09:44:50.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=settings t=2024-09-21T09:44:49.752164595Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-21T09:44:50.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=settings t=2024-09-21T09:44:49.752174378Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-21T09:44:50.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=settings t=2024-09-21T09:44:49.752182987Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-21T09:44:50.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=settings t=2024-09-21T09:44:49.752191374Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-21T09:44:50.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=settings t=2024-09-21T09:44:49.752200205Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-21T09:44:50.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=settings t=2024-09-21T09:44:49.752208837Z level=info msg="App mode production" 2024-09-21T09:44:50.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=sqlstore t=2024-09-21T09:44:49.752280958Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-21T09:44:50.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=sqlstore t=2024-09-21T09:44:49.752305656Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-21T09:44:50.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=migrator t=2024-09-21T09:44:49.77664269Z level=info msg="Starting DB migrations" 2024-09-21T09:44:50.086 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=migrator t=2024-09-21T09:44:49.784542899Z level=info msg="migrations completed" performed=0 skipped=481 duration=683.766µs 2024-09-21T09:44:50.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=sqlstore t=2024-09-21T09:44:49.78528679Z level=info msg="Created default organization" 2024-09-21T09:44:50.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=plugin.loader t=2024-09-21T09:44:49.877070164Z level=info msg="Plugin registered" pluginID=input 2024-09-21T09:44:50.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=plugin.loader t=2024-09-21T09:44:49.922886676Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-21T09:44:50.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=plugin.loader t=2024-09-21T09:44:49.924019395Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-21T09:44:50.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=secrets t=2024-09-21T09:44:49.924256622Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-21T09:44:50.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=query_data t=2024-09-21T09:44:49.927580629Z level=info msg="Query Service initialization" 2024-09-21T09:44:50.087 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=live.push_http t=2024-09-21T09:44:49.932352228Z level=info msg="Live Push Gateway initialization" 2024-09-21T09:44:50.669 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:50 smithi129 systemd[1]: Stopping Ceph node-exporter.b for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:44:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:50 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:50 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:50 smithi129 ceph-mon[32729]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-21T09:44:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:50 smithi129 ceph-mon[32729]: Deploying daemon node-exporter.b on smithi129 2024-09-21T09:44:50.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:50 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:50.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:50 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:50.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:50 smithi107 ceph-mon[29382]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-21T09:44:50.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:50 smithi107 ceph-mon[29382]: Deploying daemon node-exporter.b on smithi129 2024-09-21T09:44:50.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:50 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:50.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:50 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:50.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:50 smithi107 ceph-mon[41185]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-21T09:44:50.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:50 smithi107 ceph-mon[41185]: Deploying daemon node-exporter.b on smithi129 2024-09-21T09:44:51.617 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=infra.usagestats.collector t=2024-09-21T09:44:51.218673026Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-21T09:44:51.617 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=provisioning.datasources t=2024-09-21T09:44:51.307826195Z level=info msg="deleted correlations based on configuration" ds_name=Dashboard1 2024-09-21T09:44:51.617 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=provisioning.datasources t=2024-09-21T09:44:51.307875799Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-09-21T09:44:51.617 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=provisioning.datasources t=2024-09-21T09:44:51.318353007Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-21T09:44:51.618 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=provisioning.alerting t=2024-09-21T09:44:51.419461185Z level=info msg="starting to provision alerting" 2024-09-21T09:44:51.618 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=provisioning.alerting t=2024-09-21T09:44:51.419501877Z level=info msg="finished to provision alerting" 2024-09-21T09:44:51.618 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=ngalert.state.manager t=2024-09-21T09:44:51.421264631Z level=info msg="Warming state cache for startup" 2024-09-21T09:44:51.618 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=http.server t=2024-09-21T09:44:51.422105032Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-21T09:44:51.618 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=grafanaStorageLogger t=2024-09-21T09:44:51.422878955Z level=info msg="storage starting" 2024-09-21T09:44:51.618 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=ngalert.state.manager t=2024-09-21T09:44:51.432029908Z level=info msg="State cache has been initialized" states=0 duration=10.762145ms 2024-09-21T09:44:51.618 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=ngalert.multiorg.alertmanager t=2024-09-21T09:44:51.432327858Z level=info msg="starting MultiOrg Alertmanager" 2024-09-21T09:44:51.618 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:44:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=ticker t=2024-09-21T09:44:51.432344387Z level=info msg=starting first_tick=2024-09-21T09:45:00Z 2024-09-21T09:44:51.919 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:51 smithi129 podman[77305]: 2024-09-21 09:44:51.814819611 +0000 UTC m=+1.122098858 container died a71c52ae1a694cf0a1f9f4115b77e100f07afcfd6db6ab148b0f9b9557b91537 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:44:51.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:51 smithi129 ceph-mon[32729]: pgmap v36: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 712 op/s 2024-09-21T09:44:52.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:51 smithi107 ceph-mon[29382]: pgmap v36: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 712 op/s 2024-09-21T09:44:52.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:51 smithi107 ceph-mon[41185]: pgmap v36: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 712 op/s 2024-09-21T09:44:53.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:53 smithi107 ceph-mon[29382]: pgmap v37: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T09:44:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:53 smithi107 ceph-mon[41185]: pgmap v37: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T09:44:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:53 smithi129 ceph-mon[32729]: pgmap v37: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T09:44:54.419 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:54 smithi129 podman[77305]: 2024-09-21 09:44:54.329307131 +0000 UTC m=+3.636586374 container cleanup a71c52ae1a694cf0a1f9f4115b77e100f07afcfd6db6ab148b0f9b9557b91537 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:44:54.419 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:54 smithi129 bash[77305]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b 2024-09-21T09:44:54.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:54 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:54.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:44:54.893 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:54 smithi129 podman[77321]: 2024-09-21 09:44:54.63991571 +0000 UTC m=+2.813114905 container remove a71c52ae1a694cf0a1f9f4115b77e100f07afcfd6db6ab148b0f9b9557b91537 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:44:54.894 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:54 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-21T09:44:55.169 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:54 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-21T09:44:55.169 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:54 smithi129 systemd[1]: Stopped Ceph node-exporter.b for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:44:55.169 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:54 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@node-exporter.b.service: Consumed 2.622s CPU time. 2024-09-21T09:44:55.598 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:55 smithi129 systemd[1]: Starting Ceph node-exporter.b for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:44:55.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:55 smithi129 ceph-mon[32729]: pgmap v38: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 702 op/s 2024-09-21T09:44:55.919 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:55 smithi129 bash[77440]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-21T09:44:55.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:55 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:55.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:44:55.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:55 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:55.742Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:44:56.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:55 smithi107 ceph-mon[29382]: pgmap v38: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 702 op/s 2024-09-21T09:44:56.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:55 smithi107 ceph-mon[41185]: pgmap v38: 89 pgs: 89 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 702 op/s 2024-09-21T09:44:56.419 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:55 smithi129 bash[77440]: Getting image source signatures 2024-09-21T09:44:56.419 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:55 smithi129 bash[77440]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-21T09:44:56.419 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:55 smithi129 bash[77440]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-21T09:44:56.419 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:55 smithi129 bash[77440]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-21T09:44:57.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:44:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:44:56.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:44:57.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:57 smithi129 ceph-mon[32729]: pgmap v39: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-21T09:44:58.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:57 smithi107 ceph-mon[29382]: pgmap v39: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-21T09:44:58.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:57 smithi107 ceph-mon[41185]: pgmap v39: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-21T09:44:58.668 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:58 smithi129 bash[77440]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-21T09:44:58.669 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:58 smithi129 bash[77440]: Writing manifest to image destination 2024-09-21T09:44:59.041 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:58 smithi129 podman[77440]: 2024-09-21 09:44:58.66433927 +0000 UTC m=+3.256204210 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-21T09:44:59.041 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:58 smithi129 podman[77440]: 2024-09-21 09:44:58.770534235 +0000 UTC m=+3.362399165 container create ce5db8669c3d7532df39b68d9012c444911b23f5cd55f97a28b1f65338e6f5bf (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:44:59.421 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:44:59] "GET /metrics HTTP/1.1" 200 36412 "" "Prometheus/2.51.0" 2024-09-21T09:44:59.421 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 podman[77440]: 2024-09-21 09:44:59.040299239 +0000 UTC m=+3.632164185 container init ce5db8669c3d7532df39b68d9012c444911b23f5cd55f97a28b1f65338e6f5bf (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:44:59.421 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 podman[77440]: 2024-09-21 09:44:59.052188702 +0000 UTC m=+3.644053637 container start ce5db8669c3d7532df39b68d9012c444911b23f5cd55f97a28b1f65338e6f5bf (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:44:59.421 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.055Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-21T09:44:59.421 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.055Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-21T09:44:59.422 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.056Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T09:44:59.422 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.056Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T09:44:59.422 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.056Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T09:44:59.422 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.056Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T09:44:59.422 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T09:44:59.422 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T09:44:59.422 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T09:44:59.422 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T09:44:59.422 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T09:44:59.422 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T09:44:59.422 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T09:44:59.422 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T09:44:59.422 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T09:44:59.423 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T09:44:59.423 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T09:44:59.423 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T09:44:59.423 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T09:44:59.423 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T09:44:59.423 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T09:44:59.423 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T09:44:59.423 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T09:44:59.423 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T09:44:59.423 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T09:44:59.423 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T09:44:59.423 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T09:44:59.423 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T09:44:59.423 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T09:44:59.424 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T09:44:59.424 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T09:44:59.424 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T09:44:59.424 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T09:44:59.424 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T09:44:59.424 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T09:44:59.424 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T09:44:59.424 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T09:44:59.424 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T09:44:59.424 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T09:44:59.424 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T09:44:59.424 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T09:44:59.424 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T09:44:59.424 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T09:44:59.424 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T09:44:59.425 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T09:44:59.425 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T09:44:59.425 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T09:44:59.425 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T09:44:59.425 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T09:44:59.425 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T09:44:59.425 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.057Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T09:44:59.425 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.058Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-21T09:44:59.425 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b[77505]: ts=2024-09-21T09:44:59.058Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T09:44:59.425 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 bash[77440]: ce5db8669c3d7532df39b68d9012c444911b23f5cd55f97a28b1f65338e6f5bf 2024-09-21T09:44:59.425 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 09:44:59 smithi129 systemd[1]: Started Ceph node-exporter.b for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:44:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:59 smithi107 ceph-mon[29382]: pgmap v40: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 690 op/s 2024-09-21T09:44:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:59 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:59 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:59 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:44:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:59 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:44:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:59 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:44:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:44:59 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:44:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:59 smithi107 ceph-mon[41185]: pgmap v40: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 690 op/s 2024-09-21T09:44:59.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:59 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:59.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:59 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:59.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:59 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:44:59.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:59 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:44:59.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:59 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:44:59.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:44:59 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:44:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-mon[32729]: pgmap v40: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 690 op/s 2024-09-21T09:44:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:44:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:44:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:44:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:44:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:44:59 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:45:00.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade resume' 2024-09-21T09:45:00.550 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:00.824 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:45:00.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:00 smithi129 ceph-mon[32729]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:45:00.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:00 smithi129 ceph-mon[32729]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:45:00.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:00 smithi129 ceph-mon[32729]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:45:00.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:00 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:00.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:00 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:00.953 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:00 smithi107 ceph-mon[29382]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:45:00.953 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:00 smithi107 ceph-mon[29382]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:45:00.953 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:00 smithi107 ceph-mon[29382]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:45:00.953 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:00 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:00.953 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:00 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:00.954 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:00 smithi107 ceph-mon[41185]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:45:00.954 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:00 smithi107 ceph-mon[41185]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:45:00.954 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:00 smithi107 ceph-mon[41185]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:45:00.954 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:00 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:00.954 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:00 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:01.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:01 smithi129 ceph-mon[32729]: pgmap v41: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 690 op/s 2024-09-21T09:45:01.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:01 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:45:02.102 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:01 smithi107 ceph-mon[29382]: pgmap v41: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 690 op/s 2024-09-21T09:45:02.102 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:01 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:45:02.103 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:01 smithi107 ceph-mon[41185]: pgmap v41: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 690 op/s 2024-09-21T09:45:02.103 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:01 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:45:03.098 INFO:teuthology.orchestra.run.smithi107.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:45:03.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:03 smithi107 ceph-mon[29382]: pgmap v42: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T09:45:03.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:03 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:03.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:03 smithi107 ceph-mon[41185]: pgmap v42: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T09:45:03.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:03 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:03.877 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:03 smithi129 ceph-mon[32729]: pgmap v42: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T09:45:03.878 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:03 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:04.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'echo wait for majority of mons to upgrade' 2024-09-21T09:45:04.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:04 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:04.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:04.862 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:45:04.890 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:04 smithi129 ceph-mon[32729]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:04.890 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:04 smithi129 ceph-mon[32729]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:45:05.002 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:04 smithi107 ceph-mon[29382]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:05.002 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:04 smithi107 ceph-mon[29382]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:45:05.003 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:04 smithi107 ceph-mon[41185]: from='client.25145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:05.003 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:04 smithi107 ceph-mon[41185]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:45:05.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:05 smithi107 ceph-mon[29382]: pgmap v43: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 713 op/s 2024-09-21T09:45:05.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:05 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:05.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:05 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:05.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:05 smithi107 ceph-mon[41185]: pgmap v43: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 713 op/s 2024-09-21T09:45:05.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:05 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:05.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:05 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:05 smithi129 ceph-mon[32729]: pgmap v43: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 713 op/s 2024-09-21T09:45:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:05 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:05 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:05.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:05 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:05.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:05 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:05.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:07.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:06 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:06.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:07.770 INFO:teuthology.orchestra.run.smithi107.stdout:wait for majority of mons to upgrade 2024-09-21T09:45:07.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:07 smithi129 ceph-mon[32729]: pgmap v44: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T09:45:08.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:07 smithi107 ceph-mon[29382]: pgmap v44: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T09:45:08.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:07 smithi107 ceph-mon[41185]: pgmap v44: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T09:45:08.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'while ! ceph mon versions | grep $sha1 | egrep '"'"': [23]'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-21T09:45:09.419 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:09 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:45:09] "GET /metrics HTTP/1.1" 200 36410 "" "Prometheus/2.51.0" 2024-09-21T09:45:09.683 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:45:09.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:09 smithi107 ceph-mon[29382]: pgmap v45: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-21T09:45:09.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:09 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:09.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:09 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:09.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:09 smithi107 ceph-mon[41185]: pgmap v45: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-21T09:45:09.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:09 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:09.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:09 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:10.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:09 smithi129 ceph-mon[32729]: pgmap v45: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-21T09:45:10.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:09 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:10.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:09 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:10.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:11.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:10 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:45:11.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:10 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:45:11.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:10 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:11.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:10 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:45:11.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:10 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:45:11.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:10 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:45:11.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:10 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:11.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:10 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:45:11.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:10 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:45:11.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:10 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:45:11.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:10 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:11.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:10 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:45:11.828 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:11 smithi107 ceph-mon[29382]: pgmap v46: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-21T09:45:11.828 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:11 smithi107 ceph-mon[29382]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T09:45:11.828 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:11 smithi107 ceph-mon[29382]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T09:45:11.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:11 smithi107 ceph-mon[41185]: pgmap v46: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-21T09:45:11.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:11 smithi107 ceph-mon[41185]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T09:45:11.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:11 smithi107 ceph-mon[41185]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T09:45:12.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:11 smithi129 ceph-mon[32729]: pgmap v46: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 717 op/s 2024-09-21T09:45:12.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:11 smithi129 ceph-mon[32729]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T09:45:12.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:11 smithi129 ceph-mon[32729]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T09:45:12.880 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:45:13.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:12 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2458214511' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:13.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:12 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/2458214511' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:13.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:12 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2458214511' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:14.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:13 smithi107 ceph-mon[29382]: pgmap v47: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T09:45:14.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:13 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:14.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:13 smithi107 ceph-mon[41185]: pgmap v47: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T09:45:14.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:13 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:14.148 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:13 smithi129 ceph-mon[32729]: pgmap v47: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T09:45:14.148 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:13 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:14.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:14 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:14 smithi107 ceph-mon[29382]: from='client.25151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:14 smithi107 ceph-mon[29382]: from='client.25154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:14 smithi107 ceph-mon[29382]: Upgrade: Updating mgr.y 2024-09-21T09:45:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:14 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:45:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:14 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:45:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:14 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:45:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:14 smithi107 ceph-mon[29382]: Deploying daemon mgr.y on smithi107 2024-09-21T09:45:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:14 smithi107 ceph-mon[41185]: from='client.25151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:14.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:14 smithi107 ceph-mon[41185]: from='client.25154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:14.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:14 smithi107 ceph-mon[41185]: Upgrade: Updating mgr.y 2024-09-21T09:45:14.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:14 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:45:14.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:14 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:45:14.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:14 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:45:14.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:14 smithi107 ceph-mon[41185]: Deploying daemon mgr.y on smithi107 2024-09-21T09:45:15.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:14 smithi129 ceph-mon[32729]: from='client.25151 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:15.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:14 smithi129 ceph-mon[32729]: from='client.25154 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:15.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:14 smithi129 ceph-mon[32729]: Upgrade: Updating mgr.y 2024-09-21T09:45:15.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:14 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:45:15.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:14 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:45:15.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:14 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:45:15.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:14 smithi129 ceph-mon[32729]: Deploying daemon mgr.y on smithi107 2024-09-21T09:45:16.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:15 smithi107 ceph-mon[29382]: pgmap v48: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 710 op/s 2024-09-21T09:45:16.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:15 smithi107 systemd[1]: Stopping Ceph mgr.y for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:45:16.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:15 smithi107 ceph-mon[41185]: pgmap v48: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 710 op/s 2024-09-21T09:45:16.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:15 smithi129 ceph-mon[32729]: pgmap v48: 89 pgs: 89 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 710 op/s 2024-09-21T09:45:16.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:15.742Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:16.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:15.743Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:16.554 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:16 smithi107 podman[92961]: 2024-09-21 09:45:16.233287386 +0000 UTC m=+0.229345703 container died 5ab1b97eb209e0bad7b9ccee96ed698fbe53313670bf52acb80a62e23bef52d2 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=quincy-fbd1a6a, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-21T09:45:16.554 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:45:16.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:16 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:45:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:16 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1345378690' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:16 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:45:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:16 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1345378690' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:16.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:16 smithi107 podman[92961]: 2024-09-21 09:45:16.781915557 +0000 UTC m=+0.777973850 container cleanup 5ab1b97eb209e0bad7b9ccee96ed698fbe53313670bf52acb80a62e23bef52d2 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=quincy-fbd1a6a, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-21T09:45:16.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:16 smithi107 bash[92961]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y 2024-09-21T09:45:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:16 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:45:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:16 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1345378690' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:17.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:16 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:16.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:17.359 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:17 smithi107 podman[93000]: 2024-09-21 09:45:17.034815578 +0000 UTC m=+0.789706264 container remove 5ab1b97eb209e0bad7b9ccee96ed698fbe53313670bf52acb80a62e23bef52d2 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T09:45:17.359 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:17 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-21T09:45:17.724 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:17 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@mgr.y.service: Failed with result 'exit-code'. 2024-09-21T09:45:17.724 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:17 smithi107 systemd[1]: Stopped Ceph mgr.y for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:45:17.724 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:17 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@mgr.y.service: Consumed 42.838s CPU time. 2024-09-21T09:45:18.059 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:17 smithi107 ceph-mon[29382]: pgmap v49: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T09:45:18.059 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:17 smithi107 ceph-mon[29382]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:18.059 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:17 smithi107 ceph-mon[41185]: pgmap v49: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T09:45:18.059 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:17 smithi107 ceph-mon[41185]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:18.060 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:17 smithi107 systemd[1]: Starting Ceph mgr.y for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:45:18.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:17 smithi129 ceph-mon[32729]: pgmap v49: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T09:45:18.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:17 smithi129 ceph-mon[32729]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:18.359 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:18 smithi107 podman[93148]: 2024-09-21 09:45:17.962651852 +0000 UTC m=+0.035687548 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:45:19.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:18 smithi107 ceph-mon[29382]: from='client.25166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:19.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:18 smithi107 ceph-mon[29382]: mgrmap e27: x(active, since 93s), standbys: y 2024-09-21T09:45:19.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:18 smithi107 ceph-mon[41185]: from='client.25166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:19.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:18 smithi107 ceph-mon[41185]: mgrmap e27: x(active, since 93s), standbys: y 2024-09-21T09:45:19.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:18 smithi107 podman[93148]: 2024-09-21 09:45:18.978464364 +0000 UTC m=+1.051500072 container create 849409c95398c5ab98493bea5e35bdfdbe0cda89ae5fb02dd29fb6a58b1146bd (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-21T09:45:19.113 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:18 smithi129 ceph-mon[32729]: from='client.25166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:19.113 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:18 smithi129 ceph-mon[32729]: mgrmap e27: x(active, since 93s), standbys: y 2024-09-21T09:45:19.418 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:19 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:45:19] "GET /metrics HTTP/1.1" 200 36411 "" "Prometheus/2.51.0" 2024-09-21T09:45:19.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:19 smithi129 ceph-mon[32729]: pgmap v50: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 733 op/s 2024-09-21T09:45:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:19 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2430124633' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:20.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:19 smithi107 ceph-mon[29382]: pgmap v50: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 733 op/s 2024-09-21T09:45:20.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:19 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2430124633' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:20.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:19 smithi107 ceph-mon[41185]: pgmap v50: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 733 op/s 2024-09-21T09:45:20.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:19 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/2430124633' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:20.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:19 smithi107 podman[93148]: 2024-09-21 09:45:19.757698477 +0000 UTC m=+1.830734203 container init 849409c95398c5ab98493bea5e35bdfdbe0cda89ae5fb02dd29fb6a58b1146bd (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T09:45:20.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:19 smithi107 podman[93148]: 2024-09-21 09:45:19.771473655 +0000 UTC m=+1.844509360 container start 849409c95398c5ab98493bea5e35bdfdbe0cda89ae5fb02dd29fb6a58b1146bd (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-09-21T09:45:20.110 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:19 smithi107 bash[93148]: 849409c95398c5ab98493bea5e35bdfdbe0cda89ae5fb02dd29fb6a58b1146bd 2024-09-21T09:45:20.110 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:19 smithi107 systemd[1]: Started Ceph mgr.y for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:45:20.210 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:45:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:21.287 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:20 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:21.287 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:20 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:21.287 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:20 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:45:21.287 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:20 smithi129 ceph-mon[32729]: pgmap v51: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 733 op/s 2024-09-21T09:45:21.287 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:20 smithi129 ceph-mon[32729]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:21.287 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:20 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:21.287 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:20 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:21.287 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:20 smithi129 ceph-mon[32729]: from='client.25190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:21.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:20 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:21.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:20 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:21.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:20 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:45:21.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:20 smithi107 ceph-mon[29382]: pgmap v51: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 733 op/s 2024-09-21T09:45:21.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:20 smithi107 ceph-mon[29382]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:21.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:20 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:21.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:20 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:21.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:20 smithi107 ceph-mon[29382]: from='client.25190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:21.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:20 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:21.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:20 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:21.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:20 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:45:21.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:20 smithi107 ceph-mon[41185]: pgmap v51: 89 pgs: 89 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 733 op/s 2024-09-21T09:45:21.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:20 smithi107 ceph-mon[41185]: from='client.25178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:21.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:20 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:21.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:20 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:21.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:20 smithi107 ceph-mon[41185]: from='client.25190 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:23.466 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:23 smithi129 ceph-mon[32729]: pgmap v52: 89 pgs: 89 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 1.08k op/s 2024-09-21T09:45:23.554 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:23 smithi107 ceph-mon[29382]: pgmap v52: 89 pgs: 89 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 1.08k op/s 2024-09-21T09:45:23.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:23 smithi107 ceph-mon[41185]: pgmap v52: 89 pgs: 89 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 1.08k op/s 2024-09-21T09:45:23.971 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:45:24.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:24 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2704453713' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:24.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:24 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/2704453713' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:24.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:24 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2704453713' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:24.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:24 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:24.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:25.537 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:25 smithi129 ceph-mon[32729]: from='client.25199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:25.537 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:25 smithi129 ceph-mon[32729]: pgmap v53: 89 pgs: 89 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 716 op/s 2024-09-21T09:45:25.537 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:25 smithi129 ceph-mon[32729]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:25.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:25 smithi107 ceph-mon[29382]: from='client.25199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:25.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:25 smithi107 ceph-mon[29382]: pgmap v53: 89 pgs: 89 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 716 op/s 2024-09-21T09:45:25.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:25 smithi107 ceph-mon[29382]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:25.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:25 smithi107 ceph-mon[41185]: from='client.25199 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:25.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:25 smithi107 ceph-mon[41185]: pgmap v53: 89 pgs: 89 active+clean; 2.4 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 716 op/s 2024-09-21T09:45:25.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:25 smithi107 ceph-mon[41185]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:25.789 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:25.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:25.789 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:25.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:26 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:26 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:26 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:26 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:26 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:26 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:26 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:26 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:26 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:26 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:26 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:26 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:26 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:26 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:26.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:26 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:26.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:26 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:26.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:26 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:26.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:26 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:27.391 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:26.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:27.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:27 smithi129 ceph-mon[32729]: pgmap v54: 89 pgs: 89 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 1.02k op/s 2024-09-21T09:45:27.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:27 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/607402023' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:27.747 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:45:27.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:27 smithi107 ceph-mon[29382]: pgmap v54: 89 pgs: 89 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 1.02k op/s 2024-09-21T09:45:27.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:27 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/607402023' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:27.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:27 smithi107 ceph-mon[41185]: pgmap v54: 89 pgs: 89 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 1.02k op/s 2024-09-21T09:45:27.750 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:27 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/607402023' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:28.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:28 smithi129 ceph-mon[32729]: from='client.24979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:28.705 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:28 smithi107 ceph-mon[41185]: from='client.24979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:28.705 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:28 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:45:28] ENGINE Bus STARTING 2024-09-21T09:45:28.705 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:28 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: CherryPy Checker: 2024-09-21T09:45:28.705 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:28 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: The Application mounted at '' has an empty config. 2024-09-21T09:45:28.705 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:28 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:45:28.705 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:28 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:45:28] ENGINE Serving on http://:::9283 2024-09-21T09:45:28.705 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:28 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:45:28] ENGINE Bus STARTED 2024-09-21T09:45:28.706 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:28 smithi107 ceph-mon[29382]: from='client.24979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:29.410 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:29 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:45:29] "GET /metrics HTTP/1.1" 200 36411 "" "Prometheus/2.51.0" 2024-09-21T09:45:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:29 smithi107 ceph-mon[29382]: pgmap v55: 89 pgs: 89 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 645 op/s 2024-09-21T09:45:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:29 smithi107 ceph-mon[29382]: from='client.25223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:29 smithi107 ceph-mon[29382]: Standby manager daemon y restarted 2024-09-21T09:45:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:29 smithi107 ceph-mon[29382]: Standby manager daemon y started 2024-09-21T09:45:29.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:29 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:45:29.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:29 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:45:29.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:29 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:45:29.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:29 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:45:29.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:29 smithi107 ceph-mon[41185]: pgmap v55: 89 pgs: 89 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 645 op/s 2024-09-21T09:45:29.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:29 smithi107 ceph-mon[41185]: from='client.25223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:29.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:29 smithi107 ceph-mon[41185]: Standby manager daemon y restarted 2024-09-21T09:45:29.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:29 smithi107 ceph-mon[41185]: Standby manager daemon y started 2024-09-21T09:45:29.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:29 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:45:29.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:29 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:45:29.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:29 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:45:29.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:29 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:45:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:29 smithi129 ceph-mon[32729]: pgmap v55: 89 pgs: 89 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 645 op/s 2024-09-21T09:45:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:29 smithi129 ceph-mon[32729]: from='client.25223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:29 smithi129 ceph-mon[32729]: Standby manager daemon y restarted 2024-09-21T09:45:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:29 smithi129 ceph-mon[32729]: Standby manager daemon y started 2024-09-21T09:45:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:29 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:45:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:29 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:45:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:29 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:45:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:29 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:45:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:30 smithi129 ceph-mon[32729]: mgrmap e28: x(active, since 105s), standbys: y 2024-09-21T09:45:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:30.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:30 smithi107 ceph-mon[29382]: mgrmap e28: x(active, since 105s), standbys: y 2024-09-21T09:45:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:30 smithi107 ceph-mon[41185]: mgrmap e28: x(active, since 105s), standbys: y 2024-09-21T09:45:31.387 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:45:31.652 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:31 smithi107 ceph-mon[29382]: pgmap v56: 89 pgs: 89 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 645 op/s 2024-09-21T09:45:31.652 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:31 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:45:31.652 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:31 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2524830175' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:31.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:31 smithi107 ceph-mon[41185]: pgmap v56: 89 pgs: 89 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 645 op/s 2024-09-21T09:45:31.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:31 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:45:31.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:31 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/2524830175' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:31.700 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:31 smithi129 ceph-mon[32729]: pgmap v56: 89 pgs: 89 active+clean; 2.2 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 255 B/s wr, 645 op/s 2024-09-21T09:45:31.700 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:31 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:45:31.700 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:31 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2524830175' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:31.700 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:31 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: 2024-09-21T09:45:31.626+0000 7fbee2626640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-21T09:45:31.700 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:31 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: ignoring --setuser ceph since I am not root 2024-09-21T09:45:32.028 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:45:31] ENGINE Bus STOPPING 2024-09-21T09:45:32.168 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:31 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: ignoring --setgroup ceph since I am not root 2024-09-21T09:45:32.169 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:31 smithi129 ceph-mgr[68404]: -- 172.21.15.129:0/37765022 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b120a890e0 con 0x55b120a66800 2024-09-21T09:45:32.359 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:45:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:45:32.359 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:45:32] ENGINE Bus STOPPED 2024-09-21T09:45:32.697 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:32.697 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:32.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:45:32.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:45:32.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:32.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:45:32.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T09:45:32.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: Activating manager daemon y 2024-09-21T09:45:32.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T09:45:32.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-21T09:45:32.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: mgrmap e29: y(active, starting, since 0.022063s) 2024-09-21T09:45:32.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:45:32.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:45:32.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:45:32.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi129.zdzmqr"}]: dispatch 2024-09-21T09:45:32.698 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi107.sedied"}]: dispatch 2024-09-21T09:45:32.699 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:45:32.699 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:45:32.699 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:45:32.699 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:45:32.699 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:45:32.699 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:45:32.699 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:45:32.699 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:45:32.699 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:45:32.699 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:45:32.699 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:45:32.699 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:45:32.699 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[29382]: Manager daemon y is now available 2024-09-21T09:45:32.700 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:45:32] ENGINE Bus STARTING 2024-09-21T09:45:32.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:32.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:32.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:45:32.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:45:32.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:32.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:45:32.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T09:45:32.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: Activating manager daemon y 2024-09-21T09:45:32.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T09:45:32.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-21T09:45:32.701 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: mgrmap e29: y(active, starting, since 0.022063s) 2024-09-21T09:45:32.701 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:45:32.701 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:45:32.701 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:45:32.701 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi129.zdzmqr"}]: dispatch 2024-09-21T09:45:32.701 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi107.sedied"}]: dispatch 2024-09-21T09:45:32.701 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:45:32.701 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:45:32.701 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:45:32.701 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:45:32.701 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:45:32.701 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:45:32.701 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:45:32.701 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:45:32.701 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:45:32.701 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:45:32.702 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:45:32.702 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:45:32.702 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-mon[41185]: Manager daemon y is now available 2024-09-21T09:45:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:45:32.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:45:32.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' 2024-09-21T09:45:32.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:45:32.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T09:45:32.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: Activating manager daemon y 2024-09-21T09:45:32.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T09:45:32.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.24826 172.21.15.129:0/4020783302' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-21T09:45:32.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: mgrmap e29: y(active, starting, since 0.022063s) 2024-09-21T09:45:32.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:45:32.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:45:32.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:45:32.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi129.zdzmqr"}]: dispatch 2024-09-21T09:45:32.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi107.sedied"}]: dispatch 2024-09-21T09:45:32.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:45:32.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:45:32.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:45:32.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:45:32.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:45:32.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:45:32.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:45:32.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:45:32.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:45:32.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:45:32.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:45:32.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:45:32.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:32 smithi129 ceph-mon[32729]: Manager daemon y is now available 2024-09-21T09:45:32.963 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:45:32.869+0000 7f5194b5b640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:45:32.963 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:45:32.869+0000 7f5194b5b640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:45:32.963 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:45:32.869+0000 7f518c850640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:45:32.963 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:45:32.869+0000 7f518c850640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:45:32.963 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:45:32.869+0000 7f518c850640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:45:32.963 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:45:32.869+0000 7f518c850640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:45:32.963 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:45:32.869+0000 7f518c850640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:45:32.963 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:45:32.870+0000 7f5194b5b640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:45:32.964 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:45:32.870+0000 7f5194b5b640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:45:32.964 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:45:32.870+0000 7f5194b5b640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:45:33.237 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:45:32] ENGINE Serving on http://:::9283 2024-09-21T09:45:33.237 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:45:32] ENGINE Bus STARTED 2024-09-21T09:45:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:33 smithi129 ceph-mon[32729]: mgrmap e30: y(active, since 1.01928s) 2024-09-21T09:45:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:33 smithi129 ceph-mon[32729]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:33 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:45:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:33 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:45:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:33 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:45:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:33 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:45:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:33 smithi129 ceph-mon[32729]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:45:33.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:33 smithi129 ceph-mon[32729]: pgmap v3: 89 pgs: 89 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T09:45:33.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:33 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:45:34.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:33 smithi107 ceph-mon[29382]: mgrmap e30: y(active, since 1.01928s) 2024-09-21T09:45:34.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:33 smithi107 ceph-mon[29382]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:34.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:33 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:45:34.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:33 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:45:34.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:33 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:45:34.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:33 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:45:34.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:33 smithi107 ceph-mon[29382]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:45:34.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:33 smithi107 ceph-mon[29382]: pgmap v3: 89 pgs: 89 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T09:45:34.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:33 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:45:34.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:33 smithi107 ceph-mon[41185]: mgrmap e30: y(active, since 1.01928s) 2024-09-21T09:45:34.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:33 smithi107 ceph-mon[41185]: from='client.25003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:34.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:33 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:45:34.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:33 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:45:34.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:33 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:45:34.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:33 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:45:34.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:33 smithi107 ceph-mon[41185]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:45:34.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:33 smithi107 ceph-mon[41185]: pgmap v3: 89 pgs: 89 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T09:45:34.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:33 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:45:34.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:34 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:34.150Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:34.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-mon[29382]: pgmap v4: 89 pgs: 89 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T09:45:34.860 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:45:34.483+0000 7f51738df640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:45:34] ENGINE Error in HTTPServer.serve 2024-09-21T09:45:34.860 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: Traceback (most recent call last): 2024-09-21T09:45:34.860 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:45:34.860 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: self._connections.run(self.expiration_interval) 2024-09-21T09:45:34.860 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:45:34.860 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: self._run(expiration_interval) 2024-09-21T09:45:34.860 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:45:34.861 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:45:34.861 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:45:34.861 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:45:34.861 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:45:34.861 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: s = self.context.wrap_socket( 2024-09-21T09:45:34.861 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:45:34.861 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: return self.sslsocket_class._create( 2024-09-21T09:45:34.861 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:45:34.861 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: self.do_handshake() 2024-09-21T09:45:34.861 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:45:34.861 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: self._sslobj.do_handshake() 2024-09-21T09:45:34.861 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:45:34.861 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:45:34.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:34 smithi107 ceph-mon[41185]: pgmap v4: 89 pgs: 89 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T09:45:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:34 smithi129 ceph-mon[32729]: pgmap v4: 89 pgs: 89 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T09:45:35.773 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:45:35.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:35 smithi129 ceph-mon[32729]: [21/Sep/2024:09:45:34] ENGINE Bus STARTING 2024-09-21T09:45:35.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:35 smithi129 ceph-mon[32729]: [21/Sep/2024:09:45:34] ENGINE Serving on http://172.21.15.107:8765 2024-09-21T09:45:35.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:35 smithi129 ceph-mon[32729]: [21/Sep/2024:09:45:34] ENGINE Serving on https://172.21.15.107:7150 2024-09-21T09:45:35.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:35 smithi129 ceph-mon[32729]: [21/Sep/2024:09:45:34] ENGINE Bus STARTED 2024-09-21T09:45:35.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:35 smithi129 ceph-mon[32729]: [21/Sep/2024:09:45:34] ENGINE Error in HTTPServer.serve 2024-09-21T09:45:35.919 INFO:journalctl@ceph.mon.b.smithi129.stdout: Traceback (most recent call last): 2024-09-21T09:45:35.919 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:45:35.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:45:35.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:45:35.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: self._run(expiration_interval) 2024-09-21T09:45:35.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:45:35.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:45:35.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:45:35.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:45:35.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:45:35.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: s = self.context.wrap_socket( 2024-09-21T09:45:35.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:45:35.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: return self.sslsocket_class._create( 2024-09-21T09:45:35.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:45:35.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: self.do_handshake() 2024-09-21T09:45:35.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:45:35.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: self._sslobj.do_handshake() 2024-09-21T09:45:35.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:45:35.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:35 smithi129 ceph-mon[32729]: mgrmap e31: y(active, since 3s) 2024-09-21T09:45:35.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:35 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2374701091' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:35.921 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:35 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:35.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:35.921 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:35 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:35.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:36.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:35 smithi107 ceph-mon[29382]: [21/Sep/2024:09:45:34] ENGINE Bus STARTING 2024-09-21T09:45:36.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:35 smithi107 ceph-mon[29382]: [21/Sep/2024:09:45:34] ENGINE Serving on http://172.21.15.107:8765 2024-09-21T09:45:36.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:35 smithi107 ceph-mon[29382]: [21/Sep/2024:09:45:34] ENGINE Serving on https://172.21.15.107:7150 2024-09-21T09:45:36.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:35 smithi107 ceph-mon[29382]: [21/Sep/2024:09:45:34] ENGINE Bus STARTED 2024-09-21T09:45:36.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:35 smithi107 ceph-mon[29382]: [21/Sep/2024:09:45:34] ENGINE Error in HTTPServer.serve 2024-09-21T09:45:36.109 INFO:journalctl@ceph.mon.a.smithi107.stdout: Traceback (most recent call last): 2024-09-21T09:45:36.109 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:45:36.109 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:45:36.109 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:45:36.109 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._run(expiration_interval) 2024-09-21T09:45:36.109 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:45:36.109 INFO:journalctl@ceph.mon.a.smithi107.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:45:36.110 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:45:36.110 INFO:journalctl@ceph.mon.a.smithi107.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:45:36.110 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:45:36.110 INFO:journalctl@ceph.mon.a.smithi107.stdout: s = self.context.wrap_socket( 2024-09-21T09:45:36.110 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:45:36.110 INFO:journalctl@ceph.mon.a.smithi107.stdout: return self.sslsocket_class._create( 2024-09-21T09:45:36.110 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:45:36.110 INFO:journalctl@ceph.mon.a.smithi107.stdout: self.do_handshake() 2024-09-21T09:45:36.110 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:45:36.110 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._sslobj.do_handshake() 2024-09-21T09:45:36.110 INFO:journalctl@ceph.mon.a.smithi107.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:45:36.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:35 smithi107 ceph-mon[29382]: mgrmap e31: y(active, since 3s) 2024-09-21T09:45:36.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:35 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2374701091' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:36.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:35 smithi107 ceph-mon[41185]: [21/Sep/2024:09:45:34] ENGINE Bus STARTING 2024-09-21T09:45:36.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:35 smithi107 ceph-mon[41185]: [21/Sep/2024:09:45:34] ENGINE Serving on http://172.21.15.107:8765 2024-09-21T09:45:36.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:35 smithi107 ceph-mon[41185]: [21/Sep/2024:09:45:34] ENGINE Serving on https://172.21.15.107:7150 2024-09-21T09:45:36.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:35 smithi107 ceph-mon[41185]: [21/Sep/2024:09:45:34] ENGINE Bus STARTED 2024-09-21T09:45:36.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:35 smithi107 ceph-mon[41185]: [21/Sep/2024:09:45:34] ENGINE Error in HTTPServer.serve 2024-09-21T09:45:36.111 INFO:journalctl@ceph.mon.c.smithi107.stdout: Traceback (most recent call last): 2024-09-21T09:45:36.111 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:45:36.111 INFO:journalctl@ceph.mon.c.smithi107.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:45:36.111 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:45:36.111 INFO:journalctl@ceph.mon.c.smithi107.stdout: self._run(expiration_interval) 2024-09-21T09:45:36.111 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:45:36.111 INFO:journalctl@ceph.mon.c.smithi107.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:45:36.111 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:45:36.111 INFO:journalctl@ceph.mon.c.smithi107.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:45:36.112 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:45:36.112 INFO:journalctl@ceph.mon.c.smithi107.stdout: s = self.context.wrap_socket( 2024-09-21T09:45:36.112 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:45:36.112 INFO:journalctl@ceph.mon.c.smithi107.stdout: return self.sslsocket_class._create( 2024-09-21T09:45:36.112 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:45:36.112 INFO:journalctl@ceph.mon.c.smithi107.stdout: self.do_handshake() 2024-09-21T09:45:36.112 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:45:36.112 INFO:journalctl@ceph.mon.c.smithi107.stdout: self._sslobj.do_handshake() 2024-09-21T09:45:36.112 INFO:journalctl@ceph.mon.c.smithi107.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:45:36.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:35 smithi107 ceph-mon[41185]: mgrmap e31: y(active, since 3s) 2024-09-21T09:45:36.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:35 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/2374701091' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:36 smithi129 ceph-mon[32729]: pgmap v5: 89 pgs: 89 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T09:45:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:36 smithi129 ceph-mon[32729]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:36 smithi107 ceph-mon[29382]: pgmap v5: 89 pgs: 89 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T09:45:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:36 smithi107 ceph-mon[29382]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:37.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:36 smithi107 ceph-mon[41185]: pgmap v5: 89 pgs: 89 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T09:45:37.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:36 smithi107 ceph-mon[41185]: from='client.25268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:37.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:36 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:36.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:37.703 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:37 smithi129 ceph-mon[32729]: from='client.25274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:37.703 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:37 smithi129 ceph-mon[32729]: mgrmap e32: y(active, since 5s) 2024-09-21T09:45:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:37 smithi107 ceph-mon[29382]: from='client.25274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:37 smithi107 ceph-mon[29382]: mgrmap e32: y(active, since 5s) 2024-09-21T09:45:37.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:37 smithi107 ceph-mon[41185]: from='client.25274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:37.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:37 smithi107 ceph-mon[41185]: mgrmap e32: y(active, since 5s) 2024-09-21T09:45:39.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:38 smithi107 ceph-mon[29382]: pgmap v6: 89 pgs: 89 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T09:45:39.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:38 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:39.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:38 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:39.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:38 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:39.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:38 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:39.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:38 smithi107 ceph-mon[41185]: pgmap v6: 89 pgs: 89 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T09:45:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:38 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:38 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:38 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:39.093 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:38 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:39.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:38 smithi129 ceph-mon[32729]: pgmap v6: 89 pgs: 89 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail 2024-09-21T09:45:39.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:38 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:39.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:38 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:39.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:38 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:39.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:38 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:39.225 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:45:40.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:39 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/3185819862' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:40.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:39 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3185819862' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:40.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:39 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3185819862' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:40.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: [21/Sep/2024:09:45:40] ENGINE Bus STARTING 2024-09-21T09:45:40.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: CherryPy Checker: 2024-09-21T09:45:40.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: The Application mounted at '' has an empty config. 2024-09-21T09:45:40.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: 2024-09-21T09:45:40.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: [21/Sep/2024:09:45:40] ENGINE Serving on http://:::9283 2024-09-21T09:45:40.669 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: [21/Sep/2024:09:45:40] ENGINE Bus STARTED 2024-09-21T09:45:40.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:40 smithi107 ceph-mon[29382]: from='client.25286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:40 smithi107 ceph-mon[29382]: pgmap v7: 89 pgs: 89 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 25 KiB/s rd, 442 B/s wr, 519 op/s 2024-09-21T09:45:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:40 smithi107 ceph-mon[29382]: from='client.25292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:40 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.129:0/2372543960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:45:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:40 smithi107 ceph-mon[29382]: Standby manager daemon x started 2024-09-21T09:45:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:40 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.129:0/2372543960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:45:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:40 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.129:0/2372543960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:45:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:40 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.129:0/2372543960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:45:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:40 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:40 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:41.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:40 smithi107 ceph-mon[41185]: from='client.25286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:41.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:40 smithi107 ceph-mon[41185]: pgmap v7: 89 pgs: 89 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 25 KiB/s rd, 442 B/s wr, 519 op/s 2024-09-21T09:45:41.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:40 smithi107 ceph-mon[41185]: from='client.25292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:41.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:40 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/2372543960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:45:41.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:40 smithi107 ceph-mon[41185]: Standby manager daemon x started 2024-09-21T09:45:41.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:40 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/2372543960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:45:41.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:40 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/2372543960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:45:41.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:40 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/2372543960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:45:41.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:40 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:41.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:40 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:40 smithi129 ceph-mon[32729]: from='client.25286 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:40 smithi129 ceph-mon[32729]: pgmap v7: 89 pgs: 89 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 25 KiB/s rd, 442 B/s wr, 519 op/s 2024-09-21T09:45:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:40 smithi129 ceph-mon[32729]: from='client.25292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:40 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/2372543960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:45:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:40 smithi129 ceph-mon[32729]: Standby manager daemon x started 2024-09-21T09:45:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:40 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/2372543960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:45:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:40 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/2372543960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:45:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:40 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/2372543960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:45:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:40 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:40 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:42.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:41 smithi107 ceph-mon[41185]: mgrmap e33: y(active, since 9s), standbys: x 2024-09-21T09:45:42.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:41 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:45:42.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:41 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:42.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:41 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:42.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:41 smithi107 ceph-mon[29382]: mgrmap e33: y(active, since 9s), standbys: x 2024-09-21T09:45:42.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:41 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:45:42.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:41 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:42.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:41 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:42.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:41 smithi129 ceph-mon[32729]: mgrmap e33: y(active, since 9s), standbys: x 2024-09-21T09:45:42.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:41 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:45:42.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:41 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:42.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:41 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:42.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:41.877Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.129:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.129:8765: connect: connection refused" 2024-09-21T09:45:42.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:41.877Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.129:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.129:8765: connect: connection refused" 2024-09-21T09:45:42.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:41.883Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.129:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.129:8765: connect: connection refused" 2024-09-21T09:45:42.170 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:41.883Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.129:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.129:8765: connect: connection refused" 2024-09-21T09:45:42.170 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:41.883Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=smb msg="Unable to refresh target groups" err="Get \"http://172.21.15.129:8765/sd/prometheus/sd-config?service=smb\": dial tcp 172.21.15.129:8765: connect: connection refused" 2024-09-21T09:45:42.170 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:41.883Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.129:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.129:8765: connect: connection refused" 2024-09-21T09:45:42.170 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:41 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:41.886Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.129:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.129:8765: connect: connection refused" 2024-09-21T09:45:42.843 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:45:43.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:42 smithi107 ceph-mon[29382]: pgmap v8: 89 pgs: 89 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 20 KiB/s rd, 343 B/s wr, 403 op/s 2024-09-21T09:45:43.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:42 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1144843119' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:43.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:42 smithi107 ceph-mon[41185]: pgmap v8: 89 pgs: 89 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 20 KiB/s rd, 343 B/s wr, 403 op/s 2024-09-21T09:45:43.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:42 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1144843119' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:43.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:42 smithi129 ceph-mon[32729]: pgmap v8: 89 pgs: 89 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 20 KiB/s rd, 343 B/s wr, 403 op/s 2024-09-21T09:45:43.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:42 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1144843119' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:43 smithi107 ceph-mon[29382]: from='client.25304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:43 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:43 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:43 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:45:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:43 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:45:43.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:43 smithi107 ceph-mon[41185]: from='client.25304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:43.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:43 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:43.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:43 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:43.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:43 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:45:43.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:43 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:45:44.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:43 smithi129 ceph-mon[32729]: from='client.25304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:44.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:43 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:44.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:43 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:44.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:43 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:45:44.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:43 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:45:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:44 smithi107 ceph-mon[29382]: from='client.25310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:44 smithi107 ceph-mon[29382]: pgmap v9: 89 pgs: 89 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 280 B/s wr, 665 op/s 2024-09-21T09:45:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:44 smithi107 ceph-mon[29382]: Updating smithi107:/etc/ceph/ceph.conf 2024-09-21T09:45:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:44 smithi107 ceph-mon[29382]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T09:45:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:44 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:44 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:44 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:44 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:45.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:44 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:45.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:44 smithi107 ceph-mon[41185]: from='client.25310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:45.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:44 smithi107 ceph-mon[41185]: pgmap v9: 89 pgs: 89 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 280 B/s wr, 665 op/s 2024-09-21T09:45:45.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:44 smithi107 ceph-mon[41185]: Updating smithi107:/etc/ceph/ceph.conf 2024-09-21T09:45:45.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:44 smithi107 ceph-mon[41185]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T09:45:45.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:44 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:45.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:44 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:45.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:44 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:45.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:44 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:45.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:44 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:45.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:44 smithi129 ceph-mon[32729]: from='client.25310 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:45.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:44 smithi129 ceph-mon[32729]: pgmap v9: 89 pgs: 89 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 16 KiB/s rd, 280 B/s wr, 665 op/s 2024-09-21T09:45:45.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:44 smithi129 ceph-mon[32729]: Updating smithi107:/etc/ceph/ceph.conf 2024-09-21T09:45:45.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:44 smithi129 ceph-mon[32729]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T09:45:45.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:44 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:45.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:44 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:45.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:44 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:45.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:44 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:45.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:44 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:46.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:45 smithi107 ceph-mon[41185]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:45:46.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:45 smithi107 ceph-mon[41185]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:45:46.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:45 smithi107 ceph-mon[41185]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:45:46.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:45 smithi107 ceph-mon[41185]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:45:46.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:45 smithi107 ceph-mon[41185]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:45:46.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:45 smithi107 ceph-mon[41185]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:45:46.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:45 smithi107 ceph-mon[41185]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:45:46.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:45 smithi107 ceph-mon[41185]: Reconfiguring daemon prometheus.a on smithi129 2024-09-21T09:45:46.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:45 smithi107 ceph-mon[41185]: pgmap v10: 89 pgs: 89 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 606 op/s 2024-09-21T09:45:46.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:45 smithi107 ceph-mon[29382]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:45:46.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:45 smithi107 ceph-mon[29382]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:45:46.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:45 smithi107 ceph-mon[29382]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:45:46.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:45 smithi107 ceph-mon[29382]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:45:46.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:45 smithi107 ceph-mon[29382]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:45:46.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:45 smithi107 ceph-mon[29382]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:45:46.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:45 smithi107 ceph-mon[29382]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:45:46.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:45 smithi107 ceph-mon[29382]: Reconfiguring daemon prometheus.a on smithi129 2024-09-21T09:45:46.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:45 smithi107 ceph-mon[29382]: pgmap v10: 89 pgs: 89 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 606 op/s 2024-09-21T09:45:46.150 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:45 smithi129 ceph-mon[32729]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:45:46.150 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:45 smithi129 ceph-mon[32729]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:45:46.150 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:45 smithi129 ceph-mon[32729]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:45:46.150 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:45 smithi129 ceph-mon[32729]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:45:46.150 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:45 smithi129 ceph-mon[32729]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:45:46.150 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:45 smithi129 ceph-mon[32729]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:45:46.151 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:45 smithi129 ceph-mon[32729]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:45:46.151 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:45 smithi129 ceph-mon[32729]: Reconfiguring daemon prometheus.a on smithi129 2024-09-21T09:45:46.151 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:45 smithi129 ceph-mon[32729]: pgmap v10: 89 pgs: 89 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 606 op/s 2024-09-21T09:45:46.415 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:45:47.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:46 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/4086714171' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:47.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:46 smithi107 ceph-mon[29382]: from='client.25319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:47.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:46 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/4086714171' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:47.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:46 smithi107 ceph-mon[41185]: from='client.25319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:47.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:46 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/4086714171' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:47.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:46 smithi129 ceph-mon[32729]: from='client.25319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:47.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:46 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:46.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:48.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:47 smithi107 ceph-mon[29382]: from='client.25325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:48.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:47 smithi107 ceph-mon[29382]: pgmap v11: 89 pgs: 89 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 606 op/s 2024-09-21T09:45:48.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:47 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:45:48.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:47 smithi107 ceph-mon[41185]: from='client.25325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:48.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:47 smithi107 ceph-mon[41185]: pgmap v11: 89 pgs: 89 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 606 op/s 2024-09-21T09:45:48.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:47 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:45:48.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:47 smithi129 ceph-mon[32729]: from='client.25325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:48.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:47 smithi129 ceph-mon[32729]: pgmap v11: 89 pgs: 89 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 606 op/s 2024-09-21T09:45:48.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:47 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:45:49.115 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:48 smithi129 systemd[1]: Stopping Ceph prometheus.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:45:49.418 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[68380]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:45:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.51.0" 2024-09-21T09:45:49.755 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:49 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2954847389' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:49.755 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:49.429Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T09:45:49.755 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:49.429Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-21T09:45:49.755 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:49.429Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-21T09:45:49.755 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:49.429Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T09:45:49.755 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:49.429Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T09:45:49.755 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:49.429Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-21T09:45:49.755 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:49.429Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-21T09:45:49.756 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:49.429Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-21T09:45:49.756 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:49.433Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-21T09:45:49.756 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:49.433Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T09:45:49.756 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:49.433Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-21T09:45:49.756 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:49 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[76836]: ts=2024-09-21T09:45:49.433Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-21T09:45:49.756 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:49 smithi129 podman[80047]: 2024-09-21 09:45:49.441623007 +0000 UTC m=+0.162103684 container died 9fccc20685963d81891b9c16e7d81c29d08e574d44046bdbf86bd1bba19f7c38 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:45:49.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:49 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2954847389' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:49.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:49 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/2954847389' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:49.968 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:45:50.168 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:49 smithi129 podman[80047]: 2024-09-21 09:45:49.935112451 +0000 UTC m=+0.655593134 container cleanup 9fccc20685963d81891b9c16e7d81c29d08e574d44046bdbf86bd1bba19f7c38 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:45:50.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:49 smithi129 bash[80047]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T09:45:50.526 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:50 smithi129 podman[80062]: 2024-09-21 09:45:50.262447502 +0000 UTC m=+0.806628371 container remove 9fccc20685963d81891b9c16e7d81c29d08e574d44046bdbf86bd1bba19f7c38 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:45:50.527 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:50 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T09:45:50.527 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:50 smithi129 systemd[1]: Stopped Ceph prometheus.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:45:50.527 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:50 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 1.650s CPU time. 2024-09-21T09:45:50.527 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:50 smithi129 systemd[1]: Starting Ceph prometheus.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:45:50.804 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:50 smithi129 ceph-mon[32729]: pgmap v12: 89 pgs: 89 active+clean; 1.1 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 966 op/s 2024-09-21T09:45:50.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:50 smithi107 ceph-mon[29382]: pgmap v12: 89 pgs: 89 active+clean; 1.1 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 966 op/s 2024-09-21T09:45:50.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:50 smithi107 ceph-mon[41185]: pgmap v12: 89 pgs: 89 active+clean; 1.1 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 966 op/s 2024-09-21T09:45:51.098 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:50 smithi129 podman[80146]: 2024-09-21 09:45:50.714127361 +0000 UTC m=+0.036833739 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-21T09:45:51.098 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 podman[80146]: 2024-09-21 09:45:51.097063291 +0000 UTC m=+0.419769668 container create 24d10324469fb7098ed95c535553a37e4f1530cd9bb6cbabb0221add82cc91e8 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:45:51.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-mon[32729]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:51.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-mon[32729]: from='client.25334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:51.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 podman[80146]: 2024-09-21 09:45:51.383271688 +0000 UTC m=+0.705978080 container init 24d10324469fb7098ed95c535553a37e4f1530cd9bb6cbabb0221add82cc91e8 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:45:51.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 podman[80146]: 2024-09-21 09:45:51.391316271 +0000 UTC m=+0.714022654 container start 24d10324469fb7098ed95c535553a37e4f1530cd9bb6cbabb0221add82cc91e8 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:45:51.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.423Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-21T09:45:51.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.423Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-21T09:45:51.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.423Z caller=main.go:623 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi129 (none))" 2024-09-21T09:45:51.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.423Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T09:45:51.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.423Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T09:45:51.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.429Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T09:45:51.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.429Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-21T09:45:51.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.431Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T09:45:51.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.431Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T09:45:51.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.441Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T09:45:51.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.441Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.958µs 2024-09-21T09:45:51.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.441Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T09:45:51.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.442Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-09-21T09:45:51.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.471Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-09-21T09:45:51.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.480Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-09-21T09:45:51.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.480Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-09-21T09:45:51.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.480Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=39.267µs wal_replay_duration=39.275101ms wbl_replay_duration=123ns total_replay_duration=39.34383ms 2024-09-21T09:45:51.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.486Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T09:45:51.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.486Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-21T09:45:51.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.486Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T09:45:51.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 bash[80146]: 24d10324469fb7098ed95c535553a37e4f1530cd9bb6cbabb0221add82cc91e8 2024-09-21T09:45:51.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 systemd[1]: Started Ceph prometheus.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:45:51.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.529Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=42.137084ms db_storage=1.217µs remote_storage=1.524µs web_handler=475ns query_engine=840ns scrape=322.964µs scrape_sd=181.318µs notify=17.351µs notify_sd=12.488µs rules=40.801391ms tracing=8.232µs 2024-09-21T09:45:51.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.529Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-21T09:45:51.672 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:51.529Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T09:45:51.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:51 smithi107 ceph-mon[29382]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:51.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:51 smithi107 ceph-mon[29382]: from='client.25334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:51.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:51 smithi107 ceph-mon[41185]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:51.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:51 smithi107 ceph-mon[41185]: from='client.25334 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:52.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:52 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:52.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:52 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:52.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:52 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:45:52.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:52 smithi107 ceph-mon[29382]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:45:52.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:52 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:45:52.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:52 smithi107 ceph-mon[29382]: pgmap v13: 89 pgs: 89 active+clean; 1.1 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 665 op/s 2024-09-21T09:45:52.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:52 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:52.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:52 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:52.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:52 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:45:52.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:52 smithi107 ceph-mon[41185]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:45:52.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:52 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:45:52.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:52 smithi107 ceph-mon[41185]: pgmap v13: 89 pgs: 89 active+clean; 1.1 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 665 op/s 2024-09-21T09:45:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:52 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:52 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:45:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:52 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:45:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:52 smithi129 ceph-mon[32729]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:45:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:52 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:45:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:52 smithi129 ceph-mon[32729]: pgmap v13: 89 pgs: 89 active+clean; 1.1 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 665 op/s 2024-09-21T09:45:53.649 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:45:53.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:53 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2959822240' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:53 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:45:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:53 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/2959822240' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:53 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:45:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:53 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2959822240' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:53 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:45:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:54 smithi129 ceph-mon[32729]: Upgrade: Updating mgr.x 2024-09-21T09:45:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:54 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:45:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:54 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:45:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:54 smithi129 ceph-mon[32729]: Deploying daemon mgr.x on smithi129 2024-09-21T09:45:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:54 smithi129 ceph-mon[32729]: from='client.15423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:54 smithi129 ceph-mon[32729]: pgmap v14: 89 pgs: 89 active+clean; 874 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-21T09:45:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:54 smithi107 ceph-mon[29382]: Upgrade: Updating mgr.x 2024-09-21T09:45:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:54 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:45:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:54 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:45:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:54 smithi107 ceph-mon[29382]: Deploying daemon mgr.x on smithi129 2024-09-21T09:45:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:54 smithi107 ceph-mon[29382]: from='client.15423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:54 smithi107 ceph-mon[29382]: pgmap v14: 89 pgs: 89 active+clean; 874 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-21T09:45:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:54 smithi107 ceph-mon[41185]: Upgrade: Updating mgr.x 2024-09-21T09:45:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:54 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:45:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:54 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:45:54.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:54 smithi107 ceph-mon[41185]: Deploying daemon mgr.x on smithi129 2024-09-21T09:45:54.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:54 smithi107 ceph-mon[41185]: from='client.15423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:54.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:54 smithi107 ceph-mon[41185]: pgmap v14: 89 pgs: 89 active+clean; 874 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-21T09:45:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:55 smithi107 ceph-mon[29382]: from='client.25349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:55.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:55 smithi107 ceph-mon[41185]: from='client.25349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:55.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:55 smithi129 ceph-mon[32729]: from='client.25349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:55.919 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:55 smithi129 systemd[1]: Stopping Ceph mgr.x for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:45:56.419 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:56 smithi129 podman[80473]: 2024-09-21 09:45:56.124642812 +0000 UTC m=+0.197535252 container stop dc5e5c81266c860986986aeac1803c80675ecea2fece0ee41a835ab0d1399c2c (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=main-62da54b, maintainer=Guillaume Abrioux ) 2024-09-21T09:45:56.419 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:56 smithi129 podman[80473]: 2024-09-21 09:45:56.129717236 +0000 UTC m=+0.202609686 container died dc5e5c81266c860986986aeac1803c80675ecea2fece0ee41a835ab0d1399c2c (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-62da54b, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T09:45:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:56 smithi107 ceph-mon[29382]: pgmap v15: 89 pgs: 89 active+clean; 874 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 716 op/s 2024-09-21T09:45:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:56 smithi107 ceph-mon[41185]: pgmap v15: 89 pgs: 89 active+clean; 874 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 716 op/s 2024-09-21T09:45:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:56 smithi129 ceph-mon[32729]: pgmap v15: 89 pgs: 89 active+clean; 874 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 716 op/s 2024-09-21T09:45:57.281 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:45:57.419 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:57 smithi129 podman[80473]: 2024-09-21 09:45:57.172796079 +0000 UTC m=+1.245688517 container cleanup dc5e5c81266c860986986aeac1803c80675ecea2fece0ee41a835ab0d1399c2c (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True) 2024-09-21T09:45:57.419 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:57 smithi129 bash[80473]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x 2024-09-21T09:45:57.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:45:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:45:56.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:45:57.806 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:57 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2535443916' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:57.806 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:57 smithi129 podman[80490]: 2024-09-21 09:45:57.421566596 +0000 UTC m=+1.285960507 container remove dc5e5c81266c860986986aeac1803c80675ecea2fece0ee41a835ab0d1399c2c (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, maintainer=Guillaume Abrioux ) 2024-09-21T09:45:57.806 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:57 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-21T09:45:57.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:57 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2535443916' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:57.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:57 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/2535443916' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:45:58.147 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:57 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@mgr.x.service: Failed with result 'exit-code'. 2024-09-21T09:45:58.148 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:57 smithi129 systemd[1]: Stopped Ceph mgr.x for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:45:58.148 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:57 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@mgr.x.service: Consumed 29.649s CPU time. 2024-09-21T09:45:58.148 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:58 smithi129 systemd[1]: Starting Ceph mgr.x for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:45:58.862 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:58 smithi129 ceph-mon[32729]: from='client.25126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:58.862 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:58 smithi129 ceph-mon[32729]: pgmap v16: 89 pgs: 89 active+clean; 874 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 716 op/s 2024-09-21T09:45:58.862 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:45:58 smithi129 ceph-mon[32729]: from='client.25132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:58.863 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:58 smithi129 podman[80608]: 2024-09-21 09:45:58.405729038 +0000 UTC m=+0.057467234 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:45:58.863 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:58 smithi129 podman[80608]: 2024-09-21 09:45:58.724774285 +0000 UTC m=+0.376512479 container create 59e80c9df333c233436a37c607a3d0cdf0fd857c162c517f6b7a64f32fed6d8a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-62da54b) 2024-09-21T09:45:59.108 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:58 smithi107 ceph-mon[41185]: from='client.25126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:59.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:58 smithi107 ceph-mon[41185]: pgmap v16: 89 pgs: 89 active+clean; 874 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 716 op/s 2024-09-21T09:45:59.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:45:58 smithi107 ceph-mon[41185]: from='client.25132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:59.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:58 smithi107 ceph-mon[29382]: from='client.25126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:59.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:58 smithi107 ceph-mon[29382]: pgmap v16: 89 pgs: 89 active+clean; 874 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 716 op/s 2024-09-21T09:45:59.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:45:58 smithi107 ceph-mon[29382]: from='client.25132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:45:59.119 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:59 smithi129 podman[80608]: 2024-09-21 09:45:59.018062589 +0000 UTC m=+0.669800782 container init 59e80c9df333c233436a37c607a3d0cdf0fd857c162c517f6b7a64f32fed6d8a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-21T09:45:59.119 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:59 smithi129 podman[80608]: 2024-09-21 09:45:59.031992748 +0000 UTC m=+0.683730936 container start 59e80c9df333c233436a37c607a3d0cdf0fd857c162c517f6b7a64f32fed6d8a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-21T09:45:59.419 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:59 smithi129 bash[80608]: 59e80c9df333c233436a37c607a3d0cdf0fd857c162c517f6b7a64f32fed6d8a 2024-09-21T09:45:59.419 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:45:59 smithi129 systemd[1]: Started Ceph mgr.x for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:46:00.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:00 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:00 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:00 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:00 smithi107 ceph-mon[29382]: pgmap v17: 89 pgs: 89 active+clean; 614 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T09:46:00.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:00 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:00.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:00 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:00.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:00 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:00.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:00 smithi107 ceph-mon[41185]: pgmap v17: 89 pgs: 89 active+clean; 614 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T09:46:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:00 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:00 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:00 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:00 smithi129 ceph-mon[32729]: pgmap v17: 89 pgs: 89 active+clean; 614 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T09:46:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:00.776 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:46:01.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:01 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1735951280' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:01.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:01 smithi129 ceph-mon[32729]: from='client.25373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:01.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:01 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1735951280' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:01.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:01 smithi107 ceph-mon[29382]: from='client.25373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:01.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:01 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1735951280' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:01.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:01 smithi107 ceph-mon[41185]: from='client.25373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:02.108 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:46:01] "GET /metrics HTTP/1.1" 200 36407 "" "Prometheus/2.51.0" 2024-09-21T09:46:02.531 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:02 smithi129 ceph-mon[32729]: from='client.15465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:02.531 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:02 smithi129 ceph-mon[32729]: pgmap v18: 89 pgs: 89 active+clean; 614 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 704 op/s 2024-09-21T09:46:02.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:02 smithi107 ceph-mon[29382]: from='client.15465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:02.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:02 smithi107 ceph-mon[29382]: pgmap v18: 89 pgs: 89 active+clean; 614 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 704 op/s 2024-09-21T09:46:02.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:02 smithi107 ceph-mon[41185]: from='client.15465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:02.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:02 smithi107 ceph-mon[41185]: pgmap v18: 89 pgs: 89 active+clean; 614 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 704 op/s 2024-09-21T09:46:03.528 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:03 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:46:03.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:03 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:46:03.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:03 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:46:04.213 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:46:04.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:04 smithi129 ceph-mon[32729]: pgmap v19: 89 pgs: 89 active+clean; 373 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-21T09:46:04.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:04 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2753362791' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:04.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:04 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:04.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:04.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:04 smithi107 ceph-mon[29382]: pgmap v19: 89 pgs: 89 active+clean; 373 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-21T09:46:04.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:04 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2753362791' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:04.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:04 smithi107 ceph-mon[41185]: pgmap v19: 89 pgs: 89 active+clean; 373 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-21T09:46:04.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:04 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/2753362791' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:05.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:05 smithi129 ceph-mon[32729]: from='client.25382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:05.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:05 smithi129 ceph-mon[32729]: from='client.15477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:05.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:05 smithi107 ceph-mon[29382]: from='client.25382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:05.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:05 smithi107 ceph-mon[29382]: from='client.15477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:05.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:05 smithi107 ceph-mon[41185]: from='client.25382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:05.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:05 smithi107 ceph-mon[41185]: from='client.15477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:06.171 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:05 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:06.171 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:05 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:05.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:06.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:06 smithi129 ceph-mon[32729]: pgmap v20: 89 pgs: 89 active+clean; 373 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 667 op/s 2024-09-21T09:46:06.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:06 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:06.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:06 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:07.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:06 smithi107 ceph-mon[29382]: pgmap v20: 89 pgs: 89 active+clean; 373 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 667 op/s 2024-09-21T09:46:07.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:06 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:07.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:06 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:07.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:06 smithi107 ceph-mon[41185]: pgmap v20: 89 pgs: 89 active+clean; 373 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 667 op/s 2024-09-21T09:46:07.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:06 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:07.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:06 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:07.349 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:06 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:06.950Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:07.612 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:46:07.887 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:07 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:07.887 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:07 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:07.887 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:07 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1578230222' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:08.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:07 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:08.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:07 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:08.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:07 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1578230222' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:08.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:07 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:08.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:07 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:08.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:07 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1578230222' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:08.169 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:46:07 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[80620]: [21/Sep/2024:09:46:07] ENGINE Bus STARTING 2024-09-21T09:46:08.169 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:46:07 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[80620]: CherryPy Checker: 2024-09-21T09:46:08.169 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:46:07 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[80620]: The Application mounted at '' has an empty config. 2024-09-21T09:46:08.169 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:46:07 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[80620]: 2024-09-21T09:46:08.169 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:46:08 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[80620]: [21/Sep/2024:09:46:08] ENGINE Serving on http://:::9283 2024-09-21T09:46:08.169 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:46:08 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[80620]: [21/Sep/2024:09:46:08] ENGINE Bus STARTED 2024-09-21T09:46:09.002 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:08 smithi129 ceph-mon[32729]: from='client.25400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:09.002 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:08 smithi129 ceph-mon[32729]: pgmap v21: 89 pgs: 89 active+clean; 373 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 667 op/s 2024-09-21T09:46:09.002 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:08 smithi129 ceph-mon[32729]: Standby manager daemon x restarted 2024-09-21T09:46:09.002 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:08 smithi129 ceph-mon[32729]: Standby manager daemon x started 2024-09-21T09:46:09.002 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:08 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/3128492084' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:46:09.002 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:08 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/3128492084' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:46:09.002 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:08 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/3128492084' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:46:09.003 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:08 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/3128492084' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:46:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:08 smithi107 ceph-mon[29382]: from='client.25400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:08 smithi107 ceph-mon[29382]: pgmap v21: 89 pgs: 89 active+clean; 373 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 667 op/s 2024-09-21T09:46:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:08 smithi107 ceph-mon[29382]: Standby manager daemon x restarted 2024-09-21T09:46:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:08 smithi107 ceph-mon[29382]: Standby manager daemon x started 2024-09-21T09:46:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:08 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.129:0/3128492084' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:46:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:08 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.129:0/3128492084' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:46:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:08 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.129:0/3128492084' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:46:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:08 smithi107 ceph-mon[29382]: from='mgr.? 172.21.15.129:0/3128492084' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:46:09.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:08 smithi107 ceph-mon[41185]: from='client.25400 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:09.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:08 smithi107 ceph-mon[41185]: pgmap v21: 89 pgs: 89 active+clean; 373 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 667 op/s 2024-09-21T09:46:09.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:08 smithi107 ceph-mon[41185]: Standby manager daemon x restarted 2024-09-21T09:46:09.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:08 smithi107 ceph-mon[41185]: Standby manager daemon x started 2024-09-21T09:46:09.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:08 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/3128492084' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:46:09.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:08 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/3128492084' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:46:09.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:08 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/3128492084' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:46:09.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:08 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/3128492084' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:46:10.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:09 smithi107 ceph-mon[29382]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:10.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:09 smithi107 ceph-mon[29382]: mgrmap e34: y(active, since 37s), standbys: x 2024-09-21T09:46:10.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:09 smithi107 ceph-mon[41185]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:10.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:09 smithi107 ceph-mon[41185]: mgrmap e34: y(active, since 37s), standbys: x 2024-09-21T09:46:10.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:09 smithi129 ceph-mon[32729]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:10.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:09 smithi129 ceph-mon[32729]: mgrmap e34: y(active, since 37s), standbys: x 2024-09-21T09:46:10.558 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:10.648 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Removed 130611 objects 2024-09-21T09:46:10.648 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Clean up completed and total clean up time :155.869 2024-09-21T09:46:10.900 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:10 smithi129 ceph-mon[32729]: pgmap v22: 89 pgs: 89 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 972 op/s 2024-09-21T09:46:10.901 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:10 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/875072841' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:10.915 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:46:11.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:10 smithi107 ceph-mon[29382]: pgmap v22: 89 pgs: 89 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 972 op/s 2024-09-21T09:46:11.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:10 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/875072841' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:11.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:10 smithi107 ceph-mon[41185]: pgmap v22: 89 pgs: 89 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 972 op/s 2024-09-21T09:46:11.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:10 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/875072841' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:12.017 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:46:11] "GET /metrics HTTP/1.1" 200 36403 "" "Prometheus/2.51.0" 2024-09-21T09:46:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[29382]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[29382]: from='client.25421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:46:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:46:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:12.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:12.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:46:12.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[29382]: Upgrade: Setting container_image for all mgr 2024-09-21T09:46:12.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:12.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-21T09:46:12.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-21T09:46:12.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-21T09:46:12.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-21T09:46:12.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T09:46:12.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-21T09:46:12.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[29382]: Upgrade: It appears safe to stop mon.a 2024-09-21T09:46:12.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[29382]: pgmap v23: 89 pgs: 89 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 626 op/s 2024-09-21T09:46:12.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[41185]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:12.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[41185]: from='client.25421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:12.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:12.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:12.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:46:12.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:46:12.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:12.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:12.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:46:12.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[41185]: Upgrade: Setting container_image for all mgr 2024-09-21T09:46:12.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:12.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-21T09:46:12.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-21T09:46:12.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-21T09:46:12.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-21T09:46:12.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T09:46:12.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-21T09:46:12.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[41185]: Upgrade: It appears safe to stop mon.a 2024-09-21T09:46:12.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:12 smithi107 ceph-mon[41185]: pgmap v23: 89 pgs: 89 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 626 op/s 2024-09-21T09:46:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:12 smithi129 ceph-mon[32729]: from='client.15501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:12 smithi129 ceph-mon[32729]: from='client.25421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:12 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:12 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:12 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:46:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:12 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:46:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:12 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:12.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:12 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:12.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:12 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:46:12.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:12 smithi129 ceph-mon[32729]: Upgrade: Setting container_image for all mgr 2024-09-21T09:46:12.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:12 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:12.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:12 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-21T09:46:12.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:12 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-21T09:46:12.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:12 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-21T09:46:12.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:12 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-21T09:46:12.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:12 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T09:46:12.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:12 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-21T09:46:12.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:12 smithi129 ceph-mon[32729]: Upgrade: It appears safe to stop mon.a 2024-09-21T09:46:12.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:12 smithi129 ceph-mon[32729]: pgmap v23: 89 pgs: 89 active+clean; 144 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 626 op/s 2024-09-21T09:46:13.836 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:46:13.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:13 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/2345200648' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:13.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:13 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/2345200648' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:13 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2345200648' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:14.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:14 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:14.849 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-21T09:46:14.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-21T09:46:14.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:14 smithi107 ceph-mon[29382]: pgmap v24: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 817 op/s 2024-09-21T09:46:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:14 smithi107 ceph-mon[29382]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:14 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:14 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:46:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:14 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:46:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:14 smithi107 ceph-mon[29382]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:46:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:14 smithi107 ceph-mon[41185]: pgmap v24: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 817 op/s 2024-09-21T09:46:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:14 smithi107 ceph-mon[41185]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:14 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:14 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:46:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:14 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:46:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:14 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:46:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:14 smithi129 ceph-mon[32729]: pgmap v24: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 817 op/s 2024-09-21T09:46:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:14 smithi129 ceph-mon[32729]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:14 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' 2024-09-21T09:46:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:14 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:46:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:14 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:46:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:14 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:46:15.525 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:46:15.730 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:15 smithi107 ceph-mon[29382]: Upgrade: Updating mon.a 2024-09-21T09:46:15.730 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:15 smithi107 ceph-mon[29382]: Deploying daemon mon.a on smithi107 2024-09-21T09:46:15.730 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:15 smithi107 ceph-mon[29382]: from='client.15525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:15.731 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:15 smithi107 systemd[1]: Stopping Ceph mon.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:46:15.731 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:15 smithi107 ceph-mon[41185]: Upgrade: Updating mon.a 2024-09-21T09:46:15.731 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:15 smithi107 ceph-mon[41185]: Deploying daemon mon.a on smithi107 2024-09-21T09:46:15.731 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:15 smithi107 ceph-mon[41185]: from='client.15525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:15 smithi129 ceph-mon[32729]: Upgrade: Updating mon.a 2024-09-21T09:46:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:15 smithi129 ceph-mon[32729]: Deploying daemon mon.a on smithi107 2024-09-21T09:46:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:15 smithi129 ceph-mon[32729]: from='client.15525 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:15.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:15.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:15.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:15.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:16.773 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:46:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:16 smithi107 ceph-mon[29382]: pgmap v25: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 496 op/s 2024-09-21T09:46:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:16 smithi107 ceph-mon[29382]: from='client.? 172.21.15.107:0/1112122763' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:16 smithi107 ceph-mon[41185]: pgmap v25: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 496 op/s 2024-09-21T09:46:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:16 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1112122763' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:16.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:16 smithi129 ceph-mon[32729]: pgmap v25: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 496 op/s 2024-09-21T09:46:16.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:16 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1112122763' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:17.328 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:17 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a[29360]: 2024-09-21T09:46:17.220+0000 7f31bd176640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T09:46:17.329 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:17 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a[29360]: 2024-09-21T09:46:17.220+0000 7f31bd176640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-21T09:46:17.329 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:17 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:46:17] ENGINE Bus STOPPING 2024-09-21T09:46:17.329 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:17 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:46:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:46:17.329 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:17 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:46:17] ENGINE Bus STOPPED 2024-09-21T09:46:17.329 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:17 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:46:17] ENGINE Bus STARTING 2024-09-21T09:46:17.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:16 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:16.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:17.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:17 smithi107 podman[101955]: 2024-09-21 09:46:17.411921596 +0000 UTC m=+0.316976022 container died 7135d1946aead717a053d91222f0f9fd6d96e6a9ec78be14381becd8db0beac9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a) 2024-09-21T09:46:17.610 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:17 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:46:17] ENGINE Serving on http://:::9283 2024-09-21T09:46:17.610 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:17 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:46:17] ENGINE Bus STARTED 2024-09-21T09:46:17.982 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:17 smithi107 podman[101955]: 2024-09-21 09:46:17.779964207 +0000 UTC m=+0.685018621 container cleanup 7135d1946aead717a053d91222f0f9fd6d96e6a9ec78be14381becd8db0beac9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-21T09:46:17.982 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:17 smithi107 bash[101955]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a 2024-09-21T09:46:18.588 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:18 smithi107 podman[102001]: 2024-09-21 09:46:18.232916582 +0000 UTC m=+0.839165611 container remove 7135d1946aead717a053d91222f0f9fd6d96e6a9ec78be14381becd8db0beac9 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-21T09:46:18.588 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:18 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-21T09:46:18.588 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:18 smithi107 systemd[1]: Stopped Ceph mon.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:46:18.588 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:18 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.a.service: Consumed 13.573s CPU time. 2024-09-21T09:46:18.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:18 smithi107 systemd[1]: Starting Ceph mon.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:46:18.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:18 smithi107 podman[102183]: 2024-09-21 09:46:18.696278259 +0000 UTC m=+0.015617477 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:46:18.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:18 smithi107 podman[102183]: 2024-09-21 09:46:18.800676933 +0000 UTC m=+0.120016132 container create 34ae11d9ef1bef09d439cd3adc6611bf21acd49bced5be16110e17da397cdd3b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-09-21T09:46:19.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 podman[102183]: 2024-09-21 09:46:19.028499842 +0000 UTC m=+0.347839055 container init 34ae11d9ef1bef09d439cd3adc6611bf21acd49bced5be16110e17da397cdd3b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-21T09:46:19.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 podman[102183]: 2024-09-21 09:46:19.033453124 +0000 UTC m=+0.352792337 container start 34ae11d9ef1bef09d439cd3adc6611bf21acd49bced5be16110e17da397cdd3b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, RELEASE=main-62da54b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-21T09:46:19.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T09:46:19.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev), process ceph-mon, pid 7 2024-09-21T09:46:19.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: pidfile_write: ignore empty --pid-file 2024-09-21T09:46:19.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: load: jerasure load: lrc 2024-09-21T09:46:19.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: RocksDB version: 7.9.2 2024-09-21T09:46:19.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Git sha 0 2024-09-21T09:46:19.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Compile date 2024-09-20 20:32:28 2024-09-21T09:46:19.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: DB SUMMARY 2024-09-21T09:46:19.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: DB Session ID: AM5E5KXDNBKJIS5YCXX7 2024-09-21T09:46:19.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: CURRENT file: CURRENT 2024-09-21T09:46:19.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T09:46:19.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1179 Bytes 2024-09-21T09:46:19.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000033.sst 2024-09-21T09:46:19.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000031.log size: 1051656 ; 2024-09-21T09:46:19.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.error_if_exists: 0 2024-09-21T09:46:19.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.create_if_missing: 0 2024-09-21T09:46:19.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.paranoid_checks: 1 2024-09-21T09:46:19.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T09:46:19.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T09:46:19.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T09:46:19.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.env: 0x55dc56755120 2024-09-21T09:46:19.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T09:46:19.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.info_log: 0x55dc57e9d700 2024-09-21T09:46:19.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T09:46:19.362 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.statistics: (nil) 2024-09-21T09:46:19.362 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.use_fsync: 0 2024-09-21T09:46:19.362 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_log_file_size: 0 2024-09-21T09:46:19.362 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T09:46:19.363 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T09:46:19.363 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T09:46:19.363 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T09:46:19.363 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.allow_fallocate: 1 2024-09-21T09:46:19.363 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T09:46:19.363 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T09:46:19.363 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.use_direct_reads: 0 2024-09-21T09:46:19.363 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T09:46:19.363 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T09:46:19.363 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.db_log_dir: 2024-09-21T09:46:19.363 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.wal_dir: 2024-09-21T09:46:19.363 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T09:46:19.363 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T09:46:19.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T09:46:19.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T09:46:19.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T09:46:19.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T09:46:19.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T09:46:19.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T09:46:19.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.write_buffer_manager: 0x55dc57ea01e0 2024-09-21T09:46:19.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T09:46:19.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T09:46:19.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T09:46:19.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T09:46:19.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T09:46:19.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T09:46:19.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T09:46:19.364 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T09:46:19.365 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.unordered_write: 0 2024-09-21T09:46:19.365 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T09:46:19.365 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T09:46:19.365 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T09:46:19.365 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T09:46:19.365 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.row_cache: None 2024-09-21T09:46:19.365 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.wal_filter: None 2024-09-21T09:46:19.365 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T09:46:19.365 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T09:46:19.365 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.two_write_queues: 0 2024-09-21T09:46:19.365 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T09:46:19.365 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.wal_compression: 0 2024-09-21T09:46:19.365 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.atomic_flush: 0 2024-09-21T09:46:19.365 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T09:46:19.366 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T09:46:19.366 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T09:46:19.366 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.log_readahead_size: 0 2024-09-21T09:46:19.366 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T09:46:19.366 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T09:46:19.366 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T09:46:19.366 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T09:46:19.366 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T09:46:19.366 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T09:46:19.366 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T09:46:19.366 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_background_jobs: 2 2024-09-21T09:46:19.366 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_background_compactions: -1 2024-09-21T09:46:19.366 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_subcompactions: 1 2024-09-21T09:46:19.366 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T09:46:19.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T09:46:19.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T09:46:19.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T09:46:19.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T09:46:19.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T09:46:19.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T09:46:19.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T09:46:19.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_open_files: -1 2024-09-21T09:46:19.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T09:46:19.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T09:46:19.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T09:46:19.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T09:46:19.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_background_flushes: -1 2024-09-21T09:46:19.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Compression algorithms supported: 2024-09-21T09:46:19.368 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: kZSTD supported: 0 2024-09-21T09:46:19.368 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: kXpressCompression supported: 0 2024-09-21T09:46:19.368 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: kBZip2Compression supported: 0 2024-09-21T09:46:19.368 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T09:46:19.368 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: kLZ4Compression supported: 1 2024-09-21T09:46:19.368 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: kZlibCompression supported: 1 2024-09-21T09:46:19.368 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T09:46:19.368 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: kSnappyCompression supported: 1 2024-09-21T09:46:19.368 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T09:46:19.368 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T09:46:19.368 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-09-21T09:46:19.368 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T09:46:19.368 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T09:46:19.368 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.merge_operator: 2024-09-21T09:46:19.369 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compaction_filter: None 2024-09-21T09:46:19.369 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T09:46:19.369 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T09:46:19.369 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T09:46:19.369 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T09:46:19.369 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55dc57e9d320) 2024-09-21T09:46:19.369 INFO:journalctl@ceph.mon.a.smithi107.stdout: cache_index_and_filter_blocks: 1 2024-09-21T09:46:19.369 INFO:journalctl@ceph.mon.a.smithi107.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T09:46:19.369 INFO:journalctl@ceph.mon.a.smithi107.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T09:46:19.369 INFO:journalctl@ceph.mon.a.smithi107.stdout: pin_top_level_index_and_filter: 1 2024-09-21T09:46:19.369 INFO:journalctl@ceph.mon.a.smithi107.stdout: index_type: 0 2024-09-21T09:46:19.369 INFO:journalctl@ceph.mon.a.smithi107.stdout: data_block_index_type: 0 2024-09-21T09:46:19.369 INFO:journalctl@ceph.mon.a.smithi107.stdout: index_shortening: 1 2024-09-21T09:46:19.369 INFO:journalctl@ceph.mon.a.smithi107.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T09:46:19.370 INFO:journalctl@ceph.mon.a.smithi107.stdout: checksum: 4 2024-09-21T09:46:19.370 INFO:journalctl@ceph.mon.a.smithi107.stdout: no_block_cache: 0 2024-09-21T09:46:19.370 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache: 0x55dc57e91610 2024-09-21T09:46:19.370 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache_name: BinnedLRUCache 2024-09-21T09:46:19.370 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache_options: 2024-09-21T09:46:19.370 INFO:journalctl@ceph.mon.a.smithi107.stdout: capacity : 536870912 2024-09-21T09:46:19.370 INFO:journalctl@ceph.mon.a.smithi107.stdout: num_shard_bits : 4 2024-09-21T09:46:19.370 INFO:journalctl@ceph.mon.a.smithi107.stdout: strict_capacity_limit : 0 2024-09-21T09:46:19.370 INFO:journalctl@ceph.mon.a.smithi107.stdout: high_pri_pool_ratio: 0.000 2024-09-21T09:46:19.370 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache_compressed: (nil) 2024-09-21T09:46:19.370 INFO:journalctl@ceph.mon.a.smithi107.stdout: persistent_cache: (nil) 2024-09-21T09:46:19.370 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_size: 4096 2024-09-21T09:46:19.370 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_size_deviation: 10 2024-09-21T09:46:19.370 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_restart_interval: 16 2024-09-21T09:46:19.371 INFO:journalctl@ceph.mon.a.smithi107.stdout: index_block_restart_interval: 1 2024-09-21T09:46:19.371 INFO:journalctl@ceph.mon.a.smithi107.stdout: metadata_block_size: 4096 2024-09-21T09:46:19.371 INFO:journalctl@ceph.mon.a.smithi107.stdout: partition_filters: 0 2024-09-21T09:46:19.371 INFO:journalctl@ceph.mon.a.smithi107.stdout: use_delta_encoding: 1 2024-09-21T09:46:19.371 INFO:journalctl@ceph.mon.a.smithi107.stdout: filter_policy: bloomfilter 2024-09-21T09:46:19.371 INFO:journalctl@ceph.mon.a.smithi107.stdout: whole_key_filtering: 1 2024-09-21T09:46:19.371 INFO:journalctl@ceph.mon.a.smithi107.stdout: verify_compression: 0 2024-09-21T09:46:19.371 INFO:journalctl@ceph.mon.a.smithi107.stdout: read_amp_bytes_per_bit: 0 2024-09-21T09:46:19.371 INFO:journalctl@ceph.mon.a.smithi107.stdout: format_version: 5 2024-09-21T09:46:19.371 INFO:journalctl@ceph.mon.a.smithi107.stdout: enable_index_compression: 1 2024-09-21T09:46:19.371 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_align: 0 2024-09-21T09:46:19.371 INFO:journalctl@ceph.mon.a.smithi107.stdout: max_auto_readahead_size: 262144 2024-09-21T09:46:19.371 INFO:journalctl@ceph.mon.a.smithi107.stdout: prepopulate_block_cache: 0 2024-09-21T09:46:19.371 INFO:journalctl@ceph.mon.a.smithi107.stdout: initial_auto_readahead_size: 8192 2024-09-21T09:46:19.372 INFO:journalctl@ceph.mon.a.smithi107.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T09:46:19.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T09:46:19.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T09:46:19.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compression: NoCompression 2024-09-21T09:46:19.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T09:46:19.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T09:46:19.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T09:46:19.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.num_levels: 7 2024-09-21T09:46:19.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T09:46:19.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T09:46:19.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T09:46:19.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T09:46:19.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T09:46:19.372 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T09:46:19.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T09:46:19.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:46:19.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T09:46:19.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T09:46:19.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T09:46:19.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T09:46:19.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T09:46:19.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T09:46:19.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T09:46:19.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T09:46:19.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:46:19.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T09:46:19.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T09:46:19.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T09:46:19.373 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T09:46:19.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T09:46:19.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T09:46:19.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T09:46:19.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T09:46:19.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T09:46:19.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T09:46:19.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T09:46:19.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T09:46:19.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T09:46:19.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T09:46:19.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T09:46:19.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T09:46:19.374 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T09:46:19.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T09:46:19.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T09:46:19.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T09:46:19.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T09:46:19.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T09:46:19.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T09:46:19.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T09:46:19.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T09:46:19.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T09:46:19.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T09:46:19.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T09:46:19.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T09:46:19.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T09:46:19.375 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T09:46:19.376 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T09:46:19.376 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T09:46:19.376 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T09:46:19.376 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T09:46:19.376 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T09:46:19.376 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T09:46:19.376 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.inplace_update_support: 0 2024-09-21T09:46:19.376 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T09:46:19.376 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T09:46:19.376 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T09:46:19.376 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T09:46:19.376 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.bloom_locality: 0 2024-09-21T09:46:19.376 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.max_successive_merges: 0 2024-09-21T09:46:19.376 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T09:46:19.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T09:46:19.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T09:46:19.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T09:46:19.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.ttl: 2592000 2024-09-21T09:46:19.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T09:46:19.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T09:46:19.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T09:46:19.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.enable_blob_files: false 2024-09-21T09:46:19.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.min_blob_size: 0 2024-09-21T09:46:19.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T09:46:19.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T09:46:19.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T09:46:19.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T09:46:19.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T09:46:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T09:46:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T09:46:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T09:46:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 33.sst 2024-09-21T09:46:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T09:46:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 35, last_sequence is 12139, log_number is 31,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T09:46:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 31 2024-09-21T09:46:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 34211761-7902-49b4-b259-2c8224153f0f 2024-09-21T09:46:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726911979060739, "job": 1, "event": "recovery_started", "wal_files": [31]} 2024-09-21T09:46:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #31 mode 2 2024-09-21T09:46:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726911979064695, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 36, "file_size": 953111, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12140, "largest_seqno": 12789, "table_properties": {"data_size": 949344, "index_size": 1895, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 837, "raw_key_size": 7278, "raw_average_key_size": 23, "raw_value_size": 942755, "raw_average_value_size": 3050, "num_data_blocks": 85, "num_entries": 309, "num_filter_entries": 309, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726911979, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "34211761-7902-49b4-b259-2c8224153f0f", "db_session_id": "AM5E5KXDNBKJIS5YCXX7", "orig_file_number": 36, "seqno_to_time_mapping": "N/A"}} 2024-09-21T09:46:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726911979064805, "job": 1, "event": "recovery_finished"} 2024-09-21T09:46:19.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: [db/version_set.cc:5047] Creating manifest 38 2024-09-21T09:46:19.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T09:46:19.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000031.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T09:46:19.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55dc57ec0e00 2024-09-21T09:46:19.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: DB pointer 0x55dc57fc2000 2024-09-21T09:46:19.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T09:46:19.379 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T09:46:19.379 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** DB Stats ** 2024-09-21T09:46:19.379 INFO:journalctl@ceph.mon.a.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:46:19.379 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T09:46:19.379 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:46:19.379 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:46:19.379 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T09:46:19.379 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:46:19.379 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:46:19.379 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-09-21T09:46:19.380 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** Compaction Stats [default] ** 2024-09-21T09:46:19.380 INFO:journalctl@ceph.mon.a.smithi107.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T09:46:19.380 INFO:journalctl@ceph.mon.a.smithi107.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T09:46:19.380 INFO:journalctl@ceph.mon.a.smithi107.stdout: L0 1/0 930.77 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 407.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-21T09:46:19.380 INFO:journalctl@ceph.mon.a.smithi107.stdout: L6 1/0 7.39 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-21T09:46:19.380 INFO:journalctl@ceph.mon.a.smithi107.stdout: Sum 2/0 8.30 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 407.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-21T09:46:19.380 INFO:journalctl@ceph.mon.a.smithi107.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 407.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-21T09:46:19.380 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-09-21T09:46:19.380 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** Compaction Stats [default] ** 2024-09-21T09:46:19.380 INFO:journalctl@ceph.mon.a.smithi107.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T09:46:19.380 INFO:journalctl@ceph.mon.a.smithi107.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:46:19.380 INFO:journalctl@ceph.mon.a.smithi107.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 407.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-21T09:46:19.380 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-09-21T09:46:19.380 INFO:journalctl@ceph.mon.a.smithi107.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T09:46:19.381 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-09-21T09:46:19.381 INFO:journalctl@ceph.mon.a.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:46:19.381 INFO:journalctl@ceph.mon.a.smithi107.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-09-21T09:46:19.381 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:46:19.381 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:46:19.381 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:46:19.381 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:46:19.381 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative compaction: 0.00 GB write, 143.80 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:46:19.381 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval compaction: 0.00 GB write, 143.80 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:46:19.381 INFO:journalctl@ceph.mon.a.smithi107.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:46:19.381 INFO:journalctl@ceph.mon.a.smithi107.stdout: Block cache BinnedLRUCache@0x55dc57e91610#7 capacity: 512.00 MB usage: 2.98 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-09-21T09:46:19.381 INFO:journalctl@ceph.mon.a.smithi107.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.91 KB,0.000172853%) IndexBlock(1,2.08 KB,0.000396371%) Misc(1,0.00 KB,0%) 2024-09-21T09:46:19.381 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-09-21T09:46:19.381 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:46:19.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: starting mon.a rank 0 at public addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] at bind addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:46:19.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: mon.a@-1(???) e3 preinit fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:46:19.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: mon.a@-1(???).mds e9 new map 2024-09-21T09:46:19.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: mon.a@-1(???).mds e9 print_map 2024-09-21T09:46:19.382 INFO:journalctl@ceph.mon.a.smithi107.stdout: e9 2024-09-21T09:46:19.382 INFO:journalctl@ceph.mon.a.smithi107.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-21T09:46:19.382 INFO:journalctl@ceph.mon.a.smithi107.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T09:46:19.382 INFO:journalctl@ceph.mon.a.smithi107.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:46:19.382 INFO:journalctl@ceph.mon.a.smithi107.stdout: legacy client fscid: 1 2024-09-21T09:46:19.382 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-09-21T09:46:19.382 INFO:journalctl@ceph.mon.a.smithi107.stdout: Filesystem 'foo' (1) 2024-09-21T09:46:19.382 INFO:journalctl@ceph.mon.a.smithi107.stdout: fs_name foo 2024-09-21T09:46:19.382 INFO:journalctl@ceph.mon.a.smithi107.stdout: epoch 8 2024-09-21T09:46:19.382 INFO:journalctl@ceph.mon.a.smithi107.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-21T09:46:19.383 INFO:journalctl@ceph.mon.a.smithi107.stdout: created 2024-09-21T09:40:49.090560+0000 2024-09-21T09:46:19.383 INFO:journalctl@ceph.mon.a.smithi107.stdout: modified 2024-09-21T09:41:07.064557+0000 2024-09-21T09:46:19.383 INFO:journalctl@ceph.mon.a.smithi107.stdout: tableserver 0 2024-09-21T09:46:19.383 INFO:journalctl@ceph.mon.a.smithi107.stdout: root 0 2024-09-21T09:46:19.383 INFO:journalctl@ceph.mon.a.smithi107.stdout: session_timeout 60 2024-09-21T09:46:19.383 INFO:journalctl@ceph.mon.a.smithi107.stdout: session_autoclose 300 2024-09-21T09:46:19.383 INFO:journalctl@ceph.mon.a.smithi107.stdout: max_file_size 1099511627776 2024-09-21T09:46:19.383 INFO:journalctl@ceph.mon.a.smithi107.stdout: max_xattr_size 65536 2024-09-21T09:46:19.383 INFO:journalctl@ceph.mon.a.smithi107.stdout: required_client_features {} 2024-09-21T09:46:19.383 INFO:journalctl@ceph.mon.a.smithi107.stdout: last_failure 0 2024-09-21T09:46:19.383 INFO:journalctl@ceph.mon.a.smithi107.stdout: last_failure_osd_epoch 0 2024-09-21T09:46:19.383 INFO:journalctl@ceph.mon.a.smithi107.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:46:19.383 INFO:journalctl@ceph.mon.a.smithi107.stdout: max_mds 1 2024-09-21T09:46:19.383 INFO:journalctl@ceph.mon.a.smithi107.stdout: in 0 2024-09-21T09:46:19.384 INFO:journalctl@ceph.mon.a.smithi107.stdout: up {0=24557} 2024-09-21T09:46:19.384 INFO:journalctl@ceph.mon.a.smithi107.stdout: failed 2024-09-21T09:46:19.384 INFO:journalctl@ceph.mon.a.smithi107.stdout: damaged 2024-09-21T09:46:19.384 INFO:journalctl@ceph.mon.a.smithi107.stdout: stopped 2024-09-21T09:46:19.384 INFO:journalctl@ceph.mon.a.smithi107.stdout: data_pools [4] 2024-09-21T09:46:19.384 INFO:journalctl@ceph.mon.a.smithi107.stdout: metadata_pool 3 2024-09-21T09:46:19.384 INFO:journalctl@ceph.mon.a.smithi107.stdout: inline_data disabled 2024-09-21T09:46:19.384 INFO:journalctl@ceph.mon.a.smithi107.stdout: balancer 2024-09-21T09:46:19.384 INFO:journalctl@ceph.mon.a.smithi107.stdout: bal_rank_mask -1 2024-09-21T09:46:19.384 INFO:journalctl@ceph.mon.a.smithi107.stdout: standby_count_wanted 1 2024-09-21T09:46:19.384 INFO:journalctl@ceph.mon.a.smithi107.stdout: qdb_cluster leader: 0 members: 2024-09-21T09:46:19.384 INFO:journalctl@ceph.mon.a.smithi107.stdout: [mds.foo.smithi107.sedied{0:24557} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.107:6834/2570718287,v1:172.21.15.107:6835/2570718287] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T09:46:19.384 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-09-21T09:46:19.384 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-09-21T09:46:19.384 INFO:journalctl@ceph.mon.a.smithi107.stdout: Standby daemons: 2024-09-21T09:46:19.385 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-09-21T09:46:19.385 INFO:journalctl@ceph.mon.a.smithi107.stdout: [mds.foo.smithi129.zdzmqr{-1:24526} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.129:6832/194235284,v1:172.21.15.129:6833/194235284] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T09:46:19.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: mon.a@-1(???).osd e71 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-21T09:46:19.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: mon.a@-1(???).osd e71 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T09:46:19.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: mon.a@-1(???).osd e71 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T09:46:19.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: mon.a@-1(???).osd e71 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T09:46:19.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 ceph-mon[102221]: mon.a@-1(???).paxosservice(auth 1..19) refresh upgraded, format 0 -> 3 2024-09-21T09:46:19.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 bash[102183]: 34ae11d9ef1bef09d439cd3adc6611bf21acd49bced5be16110e17da397cdd3b 2024-09-21T09:46:19.385 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:19 smithi107 systemd[1]: Started Ceph mon.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:46:19.688 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:46:20.532 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:20.557 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_0' removed 2024-09-21T09:46:20.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ignoring --setuser ceph since I am not root 2024-09-21T09:46:20.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ignoring --setgroup ceph since I am not root 2024-09-21T09:46:20.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:46:20.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:46:20.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:46:20.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:46:20.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:46:20.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:46:20.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: pgmap v26: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 496 op/s 2024-09-21T09:46:20.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1805767457' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-21T09:46:20.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1368457815' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:20.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: pgmap v27: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 496 op/s 2024-09-21T09:46:20.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: from='client.25475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:20.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: mon.a calling monitor election 2024-09-21T09:46:20.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:46:20.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: from='client.? ' entity='' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T09:46:20.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: monmap epoch 3 2024-09-21T09:46:20.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:46:20.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: last_changed 2024-09-21T09:35:39.850868+0000 2024-09-21T09:46:20.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: created 2024-09-21T09:34:00.747992+0000 2024-09-21T09:46:20.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: min_mon_release 17 (quincy) 2024-09-21T09:46:20.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: election_strategy: 1 2024-09-21T09:46:20.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-09-21T09:46:20.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: 1: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.b 2024-09-21T09:46:20.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: 2: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.c 2024-09-21T09:46:20.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:46:20.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T09:46:20.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: mgrmap e34: y(active, since 48s), standbys: x 2024-09-21T09:46:20.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: overall HEALTH_OK 2024-09-21T09:46:20.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: from='mgr.25184 ' entity='' 2024-09-21T09:46:20.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[41185]: mgrmap e35: y(active, since 48s), standbys: x 2024-09-21T09:46:20.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:46:20.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:46:20.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:46:20.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:46:20.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:46:20.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:46:20.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: pgmap v26: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 496 op/s 2024-09-21T09:46:20.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1805767457' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-21T09:46:20.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1368457815' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:20.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: pgmap v27: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 496 op/s 2024-09-21T09:46:20.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: from='client.25475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:20.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: mon.a calling monitor election 2024-09-21T09:46:20.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:46:20.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: from='client.? ' entity='' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T09:46:20.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: monmap epoch 3 2024-09-21T09:46:20.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:46:20.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: last_changed 2024-09-21T09:35:39.850868+0000 2024-09-21T09:46:20.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: created 2024-09-21T09:34:00.747992+0000 2024-09-21T09:46:20.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: min_mon_release 17 (quincy) 2024-09-21T09:46:20.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: election_strategy: 1 2024-09-21T09:46:20.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-09-21T09:46:20.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: 1: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.b 2024-09-21T09:46:20.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: 2: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.c 2024-09-21T09:46:20.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:46:20.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T09:46:20.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: mgrmap e34: y(active, since 48s), standbys: x 2024-09-21T09:46:20.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: overall HEALTH_OK 2024-09-21T09:46:20.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: from='mgr.25184 ' entity='' 2024-09-21T09:46:20.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:20 smithi107 ceph-mon[102221]: mgrmap e35: y(active, since 48s), standbys: x 2024-09-21T09:46:20.919 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[80620]: ignoring --setuser ceph since I am not root 2024-09-21T09:46:20.919 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[80620]: ignoring --setgroup ceph since I am not root 2024-09-21T09:46:20.919 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mgr[80643]: -- 172.21.15.129:0/1661944282 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b27cbc30e0 con 0x55b27cba0800 2024-09-21T09:46:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:46:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:46:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:46:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:46:20.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:46:20.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: from='mgr.25184 172.21.15.107:0/903666586' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:46:20.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: pgmap v26: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 496 op/s 2024-09-21T09:46:20.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1805767457' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-21T09:46:20.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1368457815' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:20.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: pgmap v27: 89 pgs: 89 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 496 op/s 2024-09-21T09:46:20.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: from='client.25475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:20.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: mon.a calling monitor election 2024-09-21T09:46:20.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:46:20.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: from='client.? ' entity='' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T09:46:20.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: monmap epoch 3 2024-09-21T09:46:20.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:46:20.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: last_changed 2024-09-21T09:35:39.850868+0000 2024-09-21T09:46:20.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: created 2024-09-21T09:34:00.747992+0000 2024-09-21T09:46:20.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: min_mon_release 17 (quincy) 2024-09-21T09:46:20.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: election_strategy: 1 2024-09-21T09:46:20.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-09-21T09:46:20.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: 1: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.b 2024-09-21T09:46:20.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: 2: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.c 2024-09-21T09:46:20.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:46:20.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T09:46:20.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: mgrmap e34: y(active, since 48s), standbys: x 2024-09-21T09:46:20.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: overall HEALTH_OK 2024-09-21T09:46:20.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: from='mgr.25184 ' entity='' 2024-09-21T09:46:20.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:20 smithi129 ceph-mon[32729]: mgrmap e35: y(active, since 48s), standbys: x 2024-09-21T09:46:21.310 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T09:46:21.310 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T09:46:21.311 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-21T09:46:21.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_1 16 2024-09-21T09:46:21.605 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:46:21.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:21 smithi107 ceph-mon[102221]: from='client.? ' entity='' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T09:46:21.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:21 smithi107 ceph-mon[102221]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T09:46:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:21 smithi107 ceph-mon[41185]: from='client.? ' entity='' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T09:46:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:21 smithi107 ceph-mon[41185]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T09:46:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:21 smithi129 ceph-mon[32729]: from='client.? ' entity='' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T09:46:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:21 smithi129 ceph-mon[32729]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T09:46:22.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:22 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2826843660' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:23.048 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:22 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/2826843660' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:23.048 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:22 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2826843660' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:23.671 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_1' created 2024-09-21T09:46:23.984 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:23 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T09:46:23.984 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:23 smithi107 ceph-mon[41185]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T09:46:24.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:23 smithi129 ceph-mon[32729]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T09:46:24.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-21T09:46:24.636 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:46:25.031 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:24 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T09:46:25.031 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:24 smithi107 ceph-mon[102221]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T09:46:25.031 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:24 smithi107 ceph-mon[41185]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T09:46:25.032 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:24 smithi107 ceph-mon[41185]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T09:46:25.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:24 smithi129 ceph-mon[32729]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T09:46:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:24 smithi129 ceph-mon[32729]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T09:46:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:25 smithi107 ceph-mon[102221]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T09:46:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:25 smithi107 ceph-mon[102221]: Standby manager daemon x restarted 2024-09-21T09:46:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:25 smithi107 ceph-mon[102221]: Standby manager daemon x started 2024-09-21T09:46:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:25 smithi107 ceph-mon[102221]: from='mgr.? 172.21.15.129:0/198274267' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:46:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:25 smithi107 ceph-mon[102221]: from='mgr.? 172.21.15.129:0/198274267' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:46:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:25 smithi107 ceph-mon[102221]: from='mgr.? 172.21.15.129:0/198274267' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:46:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:25 smithi107 ceph-mon[102221]: from='mgr.? 172.21.15.129:0/198274267' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:46:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:25 smithi107 ceph-mon[41185]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T09:46:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:25 smithi107 ceph-mon[41185]: Standby manager daemon x restarted 2024-09-21T09:46:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:25 smithi107 ceph-mon[41185]: Standby manager daemon x started 2024-09-21T09:46:25.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:25 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/198274267' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:46:25.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:25 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/198274267' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:46:25.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:25 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/198274267' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:46:25.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:25 smithi107 ceph-mon[41185]: from='mgr.? 172.21.15.129:0/198274267' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:46:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:25 smithi129 ceph-mon[32729]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T09:46:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:25 smithi129 ceph-mon[32729]: Standby manager daemon x restarted 2024-09-21T09:46:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:25 smithi129 ceph-mon[32729]: Standby manager daemon x started 2024-09-21T09:46:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:25 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/198274267' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:46:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:25 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/198274267' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:46:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:25 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/198274267' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:46:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:25 smithi129 ceph-mon[32729]: from='mgr.? 172.21.15.129:0/198274267' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:46:25.920 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:46:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[80620]: [21/Sep/2024:09:46:25] ENGINE Bus STARTING 2024-09-21T09:46:25.920 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:46:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[80620]: CherryPy Checker: 2024-09-21T09:46:25.920 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:46:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[80620]: The Application mounted at '' has an empty config. 2024-09-21T09:46:25.920 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:46:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[80620]: 2024-09-21T09:46:25.920 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:46:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[80620]: [21/Sep/2024:09:46:25] ENGINE Serving on http://:::9283 2024-09-21T09:46:25.920 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 09:46:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x[80620]: [21/Sep/2024:09:46:25] ENGINE Bus STARTED 2024-09-21T09:46:26.492 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:46:26.178+0000 7efd836e5640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:46:26.492 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:46:26.178+0000 7efd836e5640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:46:26.492 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:46:26.178+0000 7efd836e5640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:46:26.492 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:46:26.178+0000 7efd836e5640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:46:26.492 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:46:26.178+0000 7efd836e5640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:46:26.492 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:46:26.180+0000 7efd84ee8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:46:26.492 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:46:26.180+0000 7efd84ee8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:46:26.492 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:46:26.180+0000 7efd84ee8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:46:26.492 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:46:26.180+0000 7efd84ee8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:46:26.492 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:46:26.180+0000 7efd84ee8640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:46:26.814 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T09:46:26.815 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: mgrmap e36: y(active, since 54s), standbys: x 2024-09-21T09:46:26.815 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: Active manager daemon y restarted 2024-09-21T09:46:26.815 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: Activating manager daemon y 2024-09-21T09:46:26.815 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T09:46:26.815 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: mgrmap e37: y(active, starting, since 0.00965992s), standbys: x 2024-09-21T09:46:26.815 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:46:26.815 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:46:26.815 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:46:26.815 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi129.zdzmqr"}]: dispatch 2024-09-21T09:46:26.815 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi107.sedied"}]: dispatch 2024-09-21T09:46:26.815 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:46:26.815 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:46:26.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:46:26.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:46:26.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:46:26.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:46:26.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:46:26.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:46:26.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:46:26.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:46:26.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:46:26.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:46:26.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:46:26.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: Manager daemon y is now available 2024-09-21T09:46:26.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:26.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:46:26.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:46:26.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:46:26.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T09:46:26.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:46:26.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T09:46:26.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1865227950' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:46:26.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T09:46:26.817 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:46:26.811+0000 7efdb5a17640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-21T09:46:26.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T09:46:26.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: mgrmap e36: y(active, since 54s), standbys: x 2024-09-21T09:46:26.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: Active manager daemon y restarted 2024-09-21T09:46:26.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: Activating manager daemon y 2024-09-21T09:46:26.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T09:46:26.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: mgrmap e37: y(active, starting, since 0.00965992s), standbys: x 2024-09-21T09:46:26.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:46:26.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:46:26.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:46:26.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi129.zdzmqr"}]: dispatch 2024-09-21T09:46:26.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi107.sedied"}]: dispatch 2024-09-21T09:46:26.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:46:26.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:46:26.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:46:26.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:46:26.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:46:26.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:46:26.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:46:26.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:46:26.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:46:26.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:46:26.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:46:26.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:46:26.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:46:26.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: Manager daemon y is now available 2024-09-21T09:46:26.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:26.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:46:26.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:46:26.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:46:26.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T09:46:26.820 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:46:26.820 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='mgr.25219 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T09:46:26.820 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1865227950' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:46:26.820 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-mon[41185]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T09:46:26.825 INFO:teuthology.orchestra.run.smithi107.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-21T09:46:26.837 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:46:26.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T09:46:26.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: mgrmap e36: y(active, since 54s), standbys: x 2024-09-21T09:46:26.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: Active manager daemon y restarted 2024-09-21T09:46:26.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: Activating manager daemon y 2024-09-21T09:46:26.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T09:46:26.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: mgrmap e37: y(active, starting, since 0.00965992s), standbys: x 2024-09-21T09:46:26.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:46:26.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:46:26.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:46:26.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi129.zdzmqr"}]: dispatch 2024-09-21T09:46:26.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi107.sedied"}]: dispatch 2024-09-21T09:46:26.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:46:26.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:46:26.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:46:26.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:46:26.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:46:26.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:46:26.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:46:26.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:46:26.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:46:26.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:46:26.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:46:26.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:46:26.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:46:26.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: Manager daemon y is now available 2024-09-21T09:46:26.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:26.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:46:26.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T09:46:26.953 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:46:26.954 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T09:46:26.954 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:46:26.954 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T09:46:26.954 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1865227950' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:46:26.954 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-mon[32729]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T09:46:27.110 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:46:26] ENGINE Bus STARTING 2024-09-21T09:46:27.110 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: CherryPy Checker: 2024-09-21T09:46:27.110 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: The Application mounted at '' has an empty config. 2024-09-21T09:46:27.110 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:46:27.110 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:46:26] ENGINE Serving on http://:::9283 2024-09-21T09:46:27.110 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:46:26] ENGINE Bus STARTED 2024-09-21T09:46:27.110 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:46:26.950+0000 7efd694f7640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:46:26] ENGINE Error in HTTPServer.serve 2024-09-21T09:46:27.110 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: Traceback (most recent call last): 2024-09-21T09:46:27.110 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:46:27.110 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: self._connections.run(self.expiration_interval) 2024-09-21T09:46:27.110 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:46:27.110 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: self._run(expiration_interval) 2024-09-21T09:46:27.111 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:46:27.111 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:46:27.111 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:46:27.111 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:46:27.111 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:46:27.111 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: s = self.context.wrap_socket( 2024-09-21T09:46:27.111 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:46:27.111 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: return self.sslsocket_class._create( 2024-09-21T09:46:27.111 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:46:27.111 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: self.do_handshake() 2024-09-21T09:46:27.111 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:46:27.111 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: self._sslobj.do_handshake() 2024-09-21T09:46:27.111 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:46:27.111 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:46:27.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:26.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:27.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:27 smithi107 ceph-mon[102221]: mgrmap e38: y(active, since 1.0135s), standbys: x 2024-09-21T09:46:27.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:27 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:46:27.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:27 smithi107 ceph-mon[102221]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T09:46:27.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:27 smithi107 ceph-mon[102221]: [21/Sep/2024:09:46:26] ENGINE Bus STARTING 2024-09-21T09:46:27.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:27 smithi107 ceph-mon[102221]: [21/Sep/2024:09:46:26] ENGINE Serving on https://172.21.15.107:7150 2024-09-21T09:46:27.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:27 smithi107 ceph-mon[102221]: [21/Sep/2024:09:46:26] ENGINE Error in HTTPServer.serve 2024-09-21T09:46:27.859 INFO:journalctl@ceph.mon.a.smithi107.stdout: Traceback (most recent call last): 2024-09-21T09:46:27.859 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:46:27.859 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:46:27.859 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:46:27.860 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._run(expiration_interval) 2024-09-21T09:46:27.860 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:46:27.860 INFO:journalctl@ceph.mon.a.smithi107.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:46:27.860 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:46:27.860 INFO:journalctl@ceph.mon.a.smithi107.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:46:27.860 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:46:27.860 INFO:journalctl@ceph.mon.a.smithi107.stdout: s = self.context.wrap_socket( 2024-09-21T09:46:27.860 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:46:27.860 INFO:journalctl@ceph.mon.a.smithi107.stdout: return self.sslsocket_class._create( 2024-09-21T09:46:27.860 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:46:27.860 INFO:journalctl@ceph.mon.a.smithi107.stdout: self.do_handshake() 2024-09-21T09:46:27.860 INFO:journalctl@ceph.mon.a.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:46:27.860 INFO:journalctl@ceph.mon.a.smithi107.stdout: self._sslobj.do_handshake() 2024-09-21T09:46:27.860 INFO:journalctl@ceph.mon.a.smithi107.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:46:27.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:27 smithi107 ceph-mon[102221]: [21/Sep/2024:09:46:27] ENGINE Serving on http://172.21.15.107:8765 2024-09-21T09:46:27.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:27 smithi107 ceph-mon[102221]: [21/Sep/2024:09:46:27] ENGINE Bus STARTED 2024-09-21T09:46:27.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:27 smithi107 ceph-mon[102221]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:27.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:27 smithi107 ceph-mon[41185]: mgrmap e38: y(active, since 1.0135s), standbys: x 2024-09-21T09:46:27.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:27 smithi107 ceph-mon[41185]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:46:27.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:27 smithi107 ceph-mon[41185]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T09:46:27.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:27 smithi107 ceph-mon[41185]: [21/Sep/2024:09:46:26] ENGINE Bus STARTING 2024-09-21T09:46:27.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:27 smithi107 ceph-mon[41185]: [21/Sep/2024:09:46:26] ENGINE Serving on https://172.21.15.107:7150 2024-09-21T09:46:27.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:27 smithi107 ceph-mon[41185]: [21/Sep/2024:09:46:26] ENGINE Error in HTTPServer.serve 2024-09-21T09:46:27.861 INFO:journalctl@ceph.mon.c.smithi107.stdout: Traceback (most recent call last): 2024-09-21T09:46:27.861 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:46:27.861 INFO:journalctl@ceph.mon.c.smithi107.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:46:27.862 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:46:27.862 INFO:journalctl@ceph.mon.c.smithi107.stdout: self._run(expiration_interval) 2024-09-21T09:46:27.862 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:46:27.862 INFO:journalctl@ceph.mon.c.smithi107.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:46:27.862 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:46:27.862 INFO:journalctl@ceph.mon.c.smithi107.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:46:27.862 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:46:27.862 INFO:journalctl@ceph.mon.c.smithi107.stdout: s = self.context.wrap_socket( 2024-09-21T09:46:27.862 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:46:27.862 INFO:journalctl@ceph.mon.c.smithi107.stdout: return self.sslsocket_class._create( 2024-09-21T09:46:27.862 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:46:27.862 INFO:journalctl@ceph.mon.c.smithi107.stdout: self.do_handshake() 2024-09-21T09:46:27.862 INFO:journalctl@ceph.mon.c.smithi107.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:46:27.862 INFO:journalctl@ceph.mon.c.smithi107.stdout: self._sslobj.do_handshake() 2024-09-21T09:46:27.863 INFO:journalctl@ceph.mon.c.smithi107.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:46:27.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:27 smithi107 ceph-mon[41185]: [21/Sep/2024:09:46:27] ENGINE Serving on http://172.21.15.107:8765 2024-09-21T09:46:27.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:27 smithi107 ceph-mon[41185]: [21/Sep/2024:09:46:27] ENGINE Bus STARTED 2024-09-21T09:46:27.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:27 smithi107 ceph-mon[41185]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:27 smithi129 ceph-mon[32729]: mgrmap e38: y(active, since 1.0135s), standbys: x 2024-09-21T09:46:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:27 smithi129 ceph-mon[32729]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:46:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:27 smithi129 ceph-mon[32729]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T09:46:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:27 smithi129 ceph-mon[32729]: [21/Sep/2024:09:46:26] ENGINE Bus STARTING 2024-09-21T09:46:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:27 smithi129 ceph-mon[32729]: [21/Sep/2024:09:46:26] ENGINE Serving on https://172.21.15.107:7150 2024-09-21T09:46:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:27 smithi129 ceph-mon[32729]: [21/Sep/2024:09:46:26] ENGINE Error in HTTPServer.serve 2024-09-21T09:46:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout: Traceback (most recent call last): 2024-09-21T09:46:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:46:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:46:27.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:46:27.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: self._run(expiration_interval) 2024-09-21T09:46:27.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:46:27.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:46:27.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:46:27.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:46:27.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:46:27.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: s = self.context.wrap_socket( 2024-09-21T09:46:27.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:46:27.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: return self.sslsocket_class._create( 2024-09-21T09:46:27.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:46:27.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: self.do_handshake() 2024-09-21T09:46:27.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:46:27.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: self._sslobj.do_handshake() 2024-09-21T09:46:27.920 INFO:journalctl@ceph.mon.b.smithi129.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:46:27.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:27 smithi129 ceph-mon[32729]: [21/Sep/2024:09:46:27] ENGINE Serving on http://172.21.15.107:8765 2024-09-21T09:46:27.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:27 smithi129 ceph-mon[32729]: [21/Sep/2024:09:46:27] ENGINE Bus STARTED 2024-09-21T09:46:27.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:27 smithi129 ceph-mon[32729]: from='client.34164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:28.630 DEBUG:teuthology.orchestra.run.smithi107:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-21T09:46:28.632 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T09:46:28.632 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T09:46:28.678 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:hints = 1 2024-09-21T09:46:28.678 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T09:46:28.678 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object prefix: benchmark_data_smithi107_103326 2024-09-21T09:46:28.678 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T09:46:28.678 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 0 0 0 0 0 0 - 0 2024-09-21T09:46:29.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:28 smithi107 ceph-mon[41185]: pgmap v5: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s wr, 37 op/s 2024-09-21T09:46:29.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:28 smithi107 ceph-mon[41185]: mgrmap e39: y(active, since 2s), standbys: x 2024-09-21T09:46:29.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:28 smithi107 ceph-mon[102221]: pgmap v5: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s wr, 37 op/s 2024-09-21T09:46:29.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:28 smithi107 ceph-mon[102221]: mgrmap e39: y(active, since 2s), standbys: x 2024-09-21T09:46:29.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:28 smithi129 ceph-mon[32729]: pgmap v5: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s wr, 37 op/s 2024-09-21T09:46:29.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:28 smithi129 ceph-mon[32729]: mgrmap e39: y(active, since 2s), standbys: x 2024-09-21T09:46:29.678 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1 16 3806 3790 236.862 236.875 0.00491823 0.00421035 2024-09-21T09:46:30.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:29 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2537227182' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T09:46:30.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:29 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/2537227182' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T09:46:30.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:29 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2537227182' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T09:46:30.381 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:46:30.647 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:30.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:30.680 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2 16 6946 6930 216.546 196.25 0.00812722 0.00460991 2024-09-21T09:46:30.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:30 smithi129 ceph-mon[32729]: pgmap v6: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s wr, 12 op/s 2024-09-21T09:46:30.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:30 smithi129 ceph-mon[32729]: mgrmap e40: y(active, since 4s), standbys: x 2024-09-21T09:46:30.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:30 smithi129 ceph-mon[32729]: from='client.25288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:31.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:30 smithi107 ceph-mon[102221]: pgmap v6: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s wr, 12 op/s 2024-09-21T09:46:31.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:30 smithi107 ceph-mon[102221]: mgrmap e40: y(active, since 4s), standbys: x 2024-09-21T09:46:31.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:30 smithi107 ceph-mon[102221]: from='client.25288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:31.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:30 smithi107 ceph-mon[41185]: pgmap v6: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s wr, 12 op/s 2024-09-21T09:46:31.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:30 smithi107 ceph-mon[41185]: mgrmap e40: y(active, since 4s), standbys: x 2024-09-21T09:46:31.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:30 smithi107 ceph-mon[41185]: from='client.25288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:31.678 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3 16 10293 10277 214.087 209.188 0.00280275 0.00466523 2024-09-21T09:46:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:46:31] "GET /metrics HTTP/1.1" 200 33658 "" "Prometheus/2.51.0" 2024-09-21T09:46:32.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:32 smithi107 ceph-mon[102221]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:32.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:32 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:32.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:32 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:32.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:32 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:32.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:32 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:32.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:32 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:32.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:32 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:32.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:32 smithi107 ceph-mon[41185]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:32.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:32 smithi107 ceph-mon[41185]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:32.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:32 smithi107 ceph-mon[41185]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:32.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:32 smithi107 ceph-mon[41185]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:32.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:32 smithi107 ceph-mon[41185]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:32.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:32 smithi107 ceph-mon[41185]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:32.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:32 smithi107 ceph-mon[41185]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:32.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:32 smithi129 ceph-mon[32729]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:32.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:32 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:32.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:32 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:32.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:32 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:32.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:32 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:32.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:32 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:32.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:32 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:32.679 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 4 16 13414 13398 209.327 195.062 0.0052569 0.00477041 2024-09-21T09:46:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:33 smithi129 ceph-mon[32729]: pgmap v7: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s wr, 7 op/s 2024-09-21T09:46:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:33 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:33 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:33.678 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 5 16 16380 16364 204.533 185.375 0.00353375 0.00488193 2024-09-21T09:46:33.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:33 smithi107 ceph-mon[102221]: pgmap v7: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s wr, 7 op/s 2024-09-21T09:46:33.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:33 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:33.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:33 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:33.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:33 smithi107 ceph-mon[41185]: pgmap v7: 89 pgs: 89 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s wr, 7 op/s 2024-09-21T09:46:33.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:33 smithi107 ceph-mon[41185]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:33.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:33 smithi107 ceph-mon[41185]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:34.251 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:34 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:34.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:34.256 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:46:34.608 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:34 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/3843712359' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:34.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:34 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3843712359' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:34.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:34 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/3843712359' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:34.679 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 6 16 19185 19169 199.661 175.312 0.0107011 0.00500164 2024-09-21T09:46:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:35 smithi129 ceph-mon[32729]: pgmap v8: 89 pgs: 89 active+clean; 669 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 25 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-21T09:46:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:35 smithi129 ceph-mon[32729]: from='client.25583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:35.679 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 7 16 22101 22085 197.172 182.25 0.00544519 0.00506661 2024-09-21T09:46:35.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:35 smithi107 ceph-mon[102221]: pgmap v8: 89 pgs: 89 active+clean; 669 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 25 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-21T09:46:35.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:35 smithi107 ceph-mon[102221]: from='client.25583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:35.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:35 smithi107 ceph-mon[41185]: pgmap v8: 89 pgs: 89 active+clean; 669 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 25 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-09-21T09:46:35.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:35 smithi107 ceph-mon[41185]: from='client.25583 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:36.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:35 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:35.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:36.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:35 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:35.743Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:36.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:36 smithi129 ceph-mon[32729]: from='client.25589 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:36.679 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 8 16 24783 24767 193.477 167.625 0.00519773 0.00516217 2024-09-21T09:46:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:36 smithi107 ceph-mon[102221]: from='client.25589 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:36.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:36 smithi107 ceph-mon[41185]: from='client.25589 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:37.329 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:36 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:36.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:37.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:37 smithi129 ceph-mon[32729]: pgmap v9: 89 pgs: 89 active+clean; 669 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 20 KiB/s rd, 74 MiB/s wr, 1.20k op/s 2024-09-21T09:46:37.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:37 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:37.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:37 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:37.679 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 9 16 27144 27128 188.374 147.562 0.00797821 0.00530229 2024-09-21T09:46:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:37 smithi107 ceph-mon[102221]: pgmap v9: 89 pgs: 89 active+clean; 669 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 20 KiB/s rd, 74 MiB/s wr, 1.20k op/s 2024-09-21T09:46:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:37 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:37 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:37.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:37 smithi107 ceph-mon[41185]: pgmap v9: 89 pgs: 89 active+clean; 669 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 20 KiB/s rd, 74 MiB/s wr, 1.20k op/s 2024-09-21T09:46:37.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:37 smithi107 ceph-mon[41185]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:37.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:37 smithi107 ceph-mon[41185]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:38.141 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:46:38.679 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 10 16 29069 29053 181.566 120.312 0.0109387 0.00550145 2024-09-21T09:46:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:46:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:46:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:38 smithi107 ceph-mon[102221]: Updating smithi107:/etc/ceph/ceph.conf 2024-09-21T09:46:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:38 smithi107 ceph-mon[102221]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T09:46:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:38 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/400631625' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:38 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:38 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:38.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:38 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:46:38.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:38 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:46:38.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:38 smithi107 ceph-mon[41185]: Updating smithi107:/etc/ceph/ceph.conf 2024-09-21T09:46:38.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:38 smithi107 ceph-mon[41185]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T09:46:38.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:38 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/400631625' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:38.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:38 smithi107 ceph-mon[41185]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:38.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:38 smithi107 ceph-mon[41185]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:38 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:46:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:38 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:46:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:38 smithi129 ceph-mon[32729]: Updating smithi107:/etc/ceph/ceph.conf 2024-09-21T09:46:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:38 smithi129 ceph-mon[32729]: Updating smithi129:/etc/ceph/ceph.conf 2024-09-21T09:46:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:38 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/400631625' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:38 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:38 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:39 smithi129 ceph-mon[32729]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:46:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:39 smithi129 ceph-mon[32729]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:46:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:39 smithi129 ceph-mon[32729]: pgmap v10: 89 pgs: 89 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 16 KiB/s rd, 143 MiB/s wr, 2.30k op/s 2024-09-21T09:46:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:39 smithi129 ceph-mon[32729]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:46:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:39 smithi129 ceph-mon[32729]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:46:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:39 smithi129 ceph-mon[32729]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:46:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:39 smithi129 ceph-mon[32729]: from='client.25318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:39.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:39 smithi129 ceph-mon[32729]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:46:39.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:39 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:39.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:39 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:39.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:39 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:39.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:39 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:39.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:39 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:46:39.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:39 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T09:46:39.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:39 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-21T09:46:39.679 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 11 16 31158 31142 176.928 130.562 0.038994 0.00563292 2024-09-21T09:46:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[102221]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:46:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[102221]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:46:39.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[102221]: pgmap v10: 89 pgs: 89 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 16 KiB/s rd, 143 MiB/s wr, 2.30k op/s 2024-09-21T09:46:39.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[102221]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:46:39.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[102221]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:46:39.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[102221]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:46:39.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[102221]: from='client.25318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:39.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[102221]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:46:39.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:39.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:39.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:39.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:39.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:46:39.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T09:46:39.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-21T09:46:39.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[41185]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:46:39.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[41185]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:46:39.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[41185]: pgmap v10: 89 pgs: 89 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 16 KiB/s rd, 143 MiB/s wr, 2.30k op/s 2024-09-21T09:46:39.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[41185]: Updating smithi129:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:46:39.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[41185]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:46:39.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[41185]: Updating smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:46:39.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[41185]: from='client.25318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:39.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[41185]: Updating smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:46:39.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[41185]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:39.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[41185]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:39.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[41185]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:39.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:46:39.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:46:39.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T09:46:39.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:39 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-21T09:46:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:40 smithi107 ceph-mon[102221]: Upgrade: It appears safe to stop mon.c 2024-09-21T09:46:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:40 smithi107 ceph-mon[102221]: from='client.25321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:40.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:40 smithi107 ceph-mon[41185]: Upgrade: It appears safe to stop mon.c 2024-09-21T09:46:40.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:40 smithi107 ceph-mon[41185]: from='client.25321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:40.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:40 smithi129 ceph-mon[32729]: Upgrade: It appears safe to stop mon.c 2024-09-21T09:46:40.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:40 smithi129 ceph-mon[32729]: from='client.25321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:40.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:40.679 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 12 16 33220 33204 172.923 128.875 0.00168776 0.00575781 2024-09-21T09:46:41.680 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 13 16 34921 34905 167.798 106.312 0.0104875 0.00595456 2024-09-21T09:46:41.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:41 smithi107 ceph-mon[102221]: pgmap v11: 89 pgs: 89 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-09-21T09:46:41.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:46:41.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:41 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/134294044' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:41.768 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:41 smithi107 ceph-mon[41185]: pgmap v11: 89 pgs: 89 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-09-21T09:46:41.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:41 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:46:41.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:41 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/134294044' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:41.803 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:46:41.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:41 smithi129 ceph-mon[32729]: pgmap v11: 89 pgs: 89 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-09-21T09:46:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:41 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:46:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:41 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/134294044' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:42.086 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:46:41] "GET /metrics HTTP/1.1" 200 36404 "" "Prometheus/2.51.0" 2024-09-21T09:46:42.679 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 14 16 37034 37018 165.245 132.062 0.00507714 0.00604669 2024-09-21T09:46:43.680 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 15 16 39174 39158 163.144 133.75 0.00887515 0.00612451 2024-09-21T09:46:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:43 smithi107 ceph-mon[102221]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:43 smithi107 ceph-mon[102221]: pgmap v12: 89 pgs: 89 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-09-21T09:46:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:43 smithi107 ceph-mon[102221]: from='client.25616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:43.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:43 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:43.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:43 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:46:43.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:43 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:46:43.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:43 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:46:43.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:43 smithi107 ceph-mon[41185]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:43.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:43 smithi107 ceph-mon[41185]: pgmap v12: 89 pgs: 89 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-09-21T09:46:43.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:43 smithi107 ceph-mon[41185]: from='client.25616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:43.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:43 smithi107 ceph-mon[41185]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:43.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:43 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:46:43.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:43 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:46:43.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:43 smithi107 ceph-mon[41185]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:46:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:43 smithi129 ceph-mon[32729]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:43 smithi129 ceph-mon[32729]: pgmap v12: 89 pgs: 89 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 15 KiB/s rd, 131 MiB/s wr, 2.10k op/s 2024-09-21T09:46:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:43 smithi129 ceph-mon[32729]: from='client.25616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:43 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:46:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:43 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:46:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:43 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:46:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:43 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:46:44.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:44 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:44.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:44 smithi107 ceph-mon[102221]: Upgrade: Updating mon.c 2024-09-21T09:46:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:44 smithi107 ceph-mon[102221]: Deploying daemon mon.c on smithi107 2024-09-21T09:46:44.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:44 smithi107 ceph-mon[41185]: Upgrade: Updating mon.c 2024-09-21T09:46:44.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:44 smithi107 ceph-mon[41185]: Deploying daemon mon.c on smithi107 2024-09-21T09:46:44.680 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 16 16 41164 41148 160.72 124.375 0.00549414 0.00621712 2024-09-21T09:46:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:44 smithi129 ceph-mon[32729]: Upgrade: Updating mon.c 2024-09-21T09:46:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:44 smithi129 ceph-mon[32729]: Deploying daemon mon.c on smithi107 2024-09-21T09:46:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:45 smithi107 ceph-mon[102221]: pgmap v13: 89 pgs: 89 active+clean; 2.1 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 15 KiB/s rd, 183 MiB/s wr, 2.94k op/s 2024-09-21T09:46:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:45 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1650341466' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:45 smithi107 ceph-mon[41185]: pgmap v13: 89 pgs: 89 active+clean; 2.1 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 15 KiB/s rd, 183 MiB/s wr, 2.94k op/s 2024-09-21T09:46:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:45 smithi107 ceph-mon[41185]: from='client.? 172.21.15.107:0/1650341466' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:45.628 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:46:45.680 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 17 16 42920 42904 157.722 109.75 0.00613443 0.00633391 2024-09-21T09:46:45.742 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:45 smithi129 ceph-mon[32729]: pgmap v13: 89 pgs: 89 active+clean; 2.1 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 15 KiB/s rd, 183 MiB/s wr, 2.94k op/s 2024-09-21T09:46:45.742 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:45 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/1650341466' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:46:46.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:45 smithi107 systemd[1]: Stopping Ceph mon.c for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:46:46.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:45 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:46.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:45 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:45.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:46.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:46 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-c[41162]: 2024-09-21T09:46:46.114+0000 7ffb62a69640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T09:46:46.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:46 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-c[41162]: 2024-09-21T09:46:46.114+0000 7ffb62a69640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-21T09:46:46.385 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:46 smithi107 podman[104811]: 2024-09-21 09:46:46.243643115 +0000 UTC m=+0.291487119 container died 4d41a86eb0a87952e40ca59b886a0acb256292d85a3655133e60b4a0b8fca615 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-c, RELEASE=quincy-fbd1a6a, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-21T09:46:46.385 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:46 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:46:46] ENGINE Bus STOPPING 2024-09-21T09:46:46.385 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:46 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:46:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:46:46.386 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:46 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:46:46] ENGINE Bus STOPPED 2024-09-21T09:46:46.386 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:46 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:46:46] ENGINE Bus STARTING 2024-09-21T09:46:46.680 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 18 16 44786 44770 155.438 116.625 0.00350626 0.00642809 2024-09-21T09:46:46.813 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:46 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:46:46] ENGINE Serving on http://:::9283 2024-09-21T09:46:46.813 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:46 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:46:46] ENGINE Bus STARTED 2024-09-21T09:46:47.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:46 smithi107 podman[104811]: 2024-09-21 09:46:46.811731435 +0000 UTC m=+0.859575434 container cleanup 4d41a86eb0a87952e40ca59b886a0acb256292d85a3655133e60b4a0b8fca615 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-21T09:46:47.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:46 smithi107 bash[104811]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-c 2024-09-21T09:46:47.411 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:47 smithi107 podman[104829]: 2024-09-21 09:46:47.155793944 +0000 UTC m=+0.910663960 container remove 4d41a86eb0a87952e40ca59b886a0acb256292d85a3655133e60b4a0b8fca615 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-09-21T09:46:47.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:46 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:46.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:47.680 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 19 16 46648 46632 153.381 116.375 0.00592432 0.00651431 2024-09-21T09:46:47.741 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:47 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.c.service: Deactivated successfully. 2024-09-21T09:46:47.741 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:47 smithi107 systemd[1]: Stopped Ceph mon.c for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:46:47.742 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:47 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.c.service: Consumed 8.358s CPU time. 2024-09-21T09:46:48.050 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:47 smithi107 systemd[1]: Starting Ceph mon.c for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:46:48.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 podman[104959]: 2024-09-21 09:46:47.957720548 +0000 UTC m=+0.037435517 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:46:48.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 podman[104959]: 2024-09-21 09:46:48.074493527 +0000 UTC m=+0.154208478 container create 04f7b77f15006980802ffeb5bdcc25491298cdb8dcae3a9f8a0733805698feae (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-c, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-21T09:46:48.680 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T09:46:48.679350+0000 min lat: 0.00113027 max lat: 0.127732 avg lat: 0.0066122 2024-09-21T09:46:48.680 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T09:46:48.680 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 20 16 48374 48358 151.106 107.875 0.00690402 0.0066122 2024-09-21T09:46:48.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 podman[104959]: 2024-09-21 09:46:48.37164184 +0000 UTC m=+0.451356805 container init 04f7b77f15006980802ffeb5bdcc25491298cdb8dcae3a9f8a0733805698feae (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-c, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-21T09:46:48.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 podman[104959]: 2024-09-21 09:46:48.385491814 +0000 UTC m=+0.465206769 container start 04f7b77f15006980802ffeb5bdcc25491298cdb8dcae3a9f8a0733805698feae (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-c, RELEASE=main-62da54b, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-21T09:46:48.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T09:46:48.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev), process ceph-mon, pid 7 2024-09-21T09:46:48.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: pidfile_write: ignore empty --pid-file 2024-09-21T09:46:48.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: load: jerasure load: lrc 2024-09-21T09:46:48.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: RocksDB version: 7.9.2 2024-09-21T09:46:48.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Git sha 0 2024-09-21T09:46:48.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Compile date 2024-09-20 20:32:28 2024-09-21T09:46:48.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: DB SUMMARY 2024-09-21T09:46:48.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: DB Session ID: D2JP1C96KVZPSZWN1D73 2024-09-21T09:46:48.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: CURRENT file: CURRENT 2024-09-21T09:46:48.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T09:46:48.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1103 Bytes 2024-09-21T09:46:48.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-09-21T09:46:48.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 4930497 ; 2024-09-21T09:46:48.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.error_if_exists: 0 2024-09-21T09:46:48.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.create_if_missing: 0 2024-09-21T09:46:48.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.paranoid_checks: 1 2024-09-21T09:46:48.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T09:46:48.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T09:46:48.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T09:46:48.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.env: 0x55d73c586120 2024-09-21T09:46:48.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T09:46:48.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.info_log: 0x55d73dc7d700 2024-09-21T09:46:48.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T09:46:48.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.statistics: (nil) 2024-09-21T09:46:48.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.use_fsync: 0 2024-09-21T09:46:48.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_log_file_size: 0 2024-09-21T09:46:48.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T09:46:48.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T09:46:48.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T09:46:48.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T09:46:48.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.allow_fallocate: 1 2024-09-21T09:46:48.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T09:46:48.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T09:46:48.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.use_direct_reads: 0 2024-09-21T09:46:48.865 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T09:46:48.866 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T09:46:48.866 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.db_log_dir: 2024-09-21T09:46:48.866 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.wal_dir: 2024-09-21T09:46:48.866 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T09:46:48.866 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T09:46:48.866 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T09:46:48.867 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T09:46:48.867 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T09:46:48.867 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T09:46:48.867 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T09:46:48.867 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T09:46:48.867 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.write_buffer_manager: 0x55d73dc801e0 2024-09-21T09:46:48.867 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T09:46:48.868 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T09:46:48.868 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T09:46:48.868 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T09:46:48.868 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T09:46:48.868 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T09:46:48.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T09:46:48.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T09:46:48.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.unordered_write: 0 2024-09-21T09:46:48.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T09:46:48.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T09:46:48.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T09:46:48.869 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T09:46:48.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.row_cache: None 2024-09-21T09:46:48.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.wal_filter: None 2024-09-21T09:46:48.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T09:46:48.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T09:46:48.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.two_write_queues: 0 2024-09-21T09:46:48.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T09:46:48.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.wal_compression: 0 2024-09-21T09:46:48.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.atomic_flush: 0 2024-09-21T09:46:48.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T09:46:48.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T09:46:48.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T09:46:48.870 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.log_readahead_size: 0 2024-09-21T09:46:48.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T09:46:48.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T09:46:48.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T09:46:48.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T09:46:48.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T09:46:48.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T09:46:48.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T09:46:48.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_background_jobs: 2 2024-09-21T09:46:48.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_background_compactions: -1 2024-09-21T09:46:48.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_subcompactions: 1 2024-09-21T09:46:48.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T09:46:48.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T09:46:48.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T09:46:48.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T09:46:48.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T09:46:48.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T09:46:48.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T09:46:48.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T09:46:48.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_open_files: -1 2024-09-21T09:46:48.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T09:46:48.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T09:46:48.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T09:46:48.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T09:46:48.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_background_flushes: -1 2024-09-21T09:46:48.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Compression algorithms supported: 2024-09-21T09:46:48.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: kZSTD supported: 0 2024-09-21T09:46:48.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: kXpressCompression supported: 0 2024-09-21T09:46:48.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: kBZip2Compression supported: 0 2024-09-21T09:46:48.872 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T09:46:48.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: kLZ4Compression supported: 1 2024-09-21T09:46:48.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: kZlibCompression supported: 1 2024-09-21T09:46:48.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T09:46:48.873 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: kSnappyCompression supported: 1 2024-09-21T09:46:48.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T09:46:48.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T09:46:48.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-09-21T09:46:48.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T09:46:48.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T09:46:48.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.merge_operator: 2024-09-21T09:46:48.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compaction_filter: None 2024-09-21T09:46:48.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T09:46:48.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T09:46:48.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T09:46:48.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T09:46:48.874 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d73dc7d320) 2024-09-21T09:46:48.874 INFO:journalctl@ceph.mon.c.smithi107.stdout: cache_index_and_filter_blocks: 1 2024-09-21T09:46:48.874 INFO:journalctl@ceph.mon.c.smithi107.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T09:46:48.875 INFO:journalctl@ceph.mon.c.smithi107.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T09:46:48.875 INFO:journalctl@ceph.mon.c.smithi107.stdout: pin_top_level_index_and_filter: 1 2024-09-21T09:46:48.875 INFO:journalctl@ceph.mon.c.smithi107.stdout: index_type: 0 2024-09-21T09:46:48.875 INFO:journalctl@ceph.mon.c.smithi107.stdout: data_block_index_type: 0 2024-09-21T09:46:48.875 INFO:journalctl@ceph.mon.c.smithi107.stdout: index_shortening: 1 2024-09-21T09:46:48.875 INFO:journalctl@ceph.mon.c.smithi107.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T09:46:48.875 INFO:journalctl@ceph.mon.c.smithi107.stdout: checksum: 4 2024-09-21T09:46:48.875 INFO:journalctl@ceph.mon.c.smithi107.stdout: no_block_cache: 0 2024-09-21T09:46:48.875 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_cache: 0x55d73dc71610 2024-09-21T09:46:48.875 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_cache_name: BinnedLRUCache 2024-09-21T09:46:48.875 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_cache_options: 2024-09-21T09:46:48.875 INFO:journalctl@ceph.mon.c.smithi107.stdout: capacity : 536870912 2024-09-21T09:46:48.875 INFO:journalctl@ceph.mon.c.smithi107.stdout: num_shard_bits : 4 2024-09-21T09:46:48.875 INFO:journalctl@ceph.mon.c.smithi107.stdout: strict_capacity_limit : 0 2024-09-21T09:46:48.876 INFO:journalctl@ceph.mon.c.smithi107.stdout: high_pri_pool_ratio: 0.000 2024-09-21T09:46:48.876 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_cache_compressed: (nil) 2024-09-21T09:46:48.876 INFO:journalctl@ceph.mon.c.smithi107.stdout: persistent_cache: (nil) 2024-09-21T09:46:48.876 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_size: 4096 2024-09-21T09:46:48.876 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_size_deviation: 10 2024-09-21T09:46:48.876 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_restart_interval: 16 2024-09-21T09:46:48.876 INFO:journalctl@ceph.mon.c.smithi107.stdout: index_block_restart_interval: 1 2024-09-21T09:46:48.876 INFO:journalctl@ceph.mon.c.smithi107.stdout: metadata_block_size: 4096 2024-09-21T09:46:48.876 INFO:journalctl@ceph.mon.c.smithi107.stdout: partition_filters: 0 2024-09-21T09:46:48.876 INFO:journalctl@ceph.mon.c.smithi107.stdout: use_delta_encoding: 1 2024-09-21T09:46:48.876 INFO:journalctl@ceph.mon.c.smithi107.stdout: filter_policy: bloomfilter 2024-09-21T09:46:48.876 INFO:journalctl@ceph.mon.c.smithi107.stdout: whole_key_filtering: 1 2024-09-21T09:46:48.876 INFO:journalctl@ceph.mon.c.smithi107.stdout: verify_compression: 0 2024-09-21T09:46:48.876 INFO:journalctl@ceph.mon.c.smithi107.stdout: read_amp_bytes_per_bit: 0 2024-09-21T09:46:48.877 INFO:journalctl@ceph.mon.c.smithi107.stdout: format_version: 5 2024-09-21T09:46:48.877 INFO:journalctl@ceph.mon.c.smithi107.stdout: enable_index_compression: 1 2024-09-21T09:46:48.877 INFO:journalctl@ceph.mon.c.smithi107.stdout: block_align: 0 2024-09-21T09:46:48.877 INFO:journalctl@ceph.mon.c.smithi107.stdout: max_auto_readahead_size: 262144 2024-09-21T09:46:48.877 INFO:journalctl@ceph.mon.c.smithi107.stdout: prepopulate_block_cache: 0 2024-09-21T09:46:48.877 INFO:journalctl@ceph.mon.c.smithi107.stdout: initial_auto_readahead_size: 8192 2024-09-21T09:46:48.877 INFO:journalctl@ceph.mon.c.smithi107.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T09:46:48.877 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T09:46:48.877 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T09:46:48.877 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compression: NoCompression 2024-09-21T09:46:48.877 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T09:46:48.877 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T09:46:48.877 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T09:46:48.877 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.num_levels: 7 2024-09-21T09:46:48.877 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T09:46:48.878 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T09:46:48.878 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T09:46:48.878 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T09:46:48.878 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T09:46:48.878 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T09:46:48.878 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T09:46:48.878 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:46:48.878 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T09:46:48.878 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T09:46:48.878 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T09:46:48.878 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T09:46:48.878 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T09:46:48.878 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T09:46:48.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T09:46:48.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T09:46:48.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:46:48.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T09:46:48.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T09:46:48.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T09:46:48.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T09:46:48.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T09:46:48.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T09:46:48.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T09:46:48.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T09:46:48.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T09:46:48.880 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T09:46:48.880 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T09:46:48.880 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T09:46:48.880 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T09:46:48.880 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T09:46:48.880 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T09:46:48.880 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T09:46:48.880 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T09:46:48.880 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T09:46:48.880 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T09:46:48.881 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T09:46:48.881 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T09:46:48.881 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T09:46:48.881 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T09:46:48.881 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T09:46:48.881 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T09:46:48.881 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T09:46:48.881 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T09:46:48.881 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T09:46:48.881 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T09:46:48.881 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T09:46:48.881 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T09:46:48.881 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T09:46:48.881 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T09:46:48.882 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T09:46:48.882 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T09:46:48.882 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T09:46:48.882 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T09:46:48.882 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.inplace_update_support: 0 2024-09-21T09:46:48.882 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T09:46:48.882 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T09:46:48.882 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T09:46:48.882 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T09:46:48.882 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.bloom_locality: 0 2024-09-21T09:46:48.882 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.max_successive_merges: 0 2024-09-21T09:46:48.882 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T09:46:48.882 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T09:46:48.882 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T09:46:48.883 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T09:46:48.883 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.ttl: 2592000 2024-09-21T09:46:48.883 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T09:46:48.883 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T09:46:48.883 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T09:46:48.883 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.enable_blob_files: false 2024-09-21T09:46:48.883 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.min_blob_size: 0 2024-09-21T09:46:48.883 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T09:46:48.883 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T09:46:48.883 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T09:46:48.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T09:46:48.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T09:46:48.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T09:46:48.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T09:46:48.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T09:46:48.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-09-21T09:46:48.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T09:46:48.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 12736, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T09:46:48.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-21T09:46:48.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a739d123-6959-410f-bf81-740a833f3d5b 2024-09-21T09:46:48.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912008428718, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-21T09:46:48.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-21T09:46:48.884 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912008450638, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 3079960, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12741, "largest_seqno": 14116, "table_properties": {"data_size": 3073600, "index_size": 3779, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1541, "raw_key_size": 14045, "raw_average_key_size": 23, "raw_value_size": 3060965, "raw_average_value_size": 5101, "num_data_blocks": 171, "num_entries": 600, "num_filter_entries": 600, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726912008, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a739d123-6959-410f-bf81-740a833f3d5b", "db_session_id": "D2JP1C96KVZPSZWN1D73", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-21T09:46:48.885 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912008450793, "job": 1, "event": "recovery_finished"} 2024-09-21T09:46:48.885 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-21T09:46:48.885 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T09:46:48.885 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T09:46:48.885 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d73dca0e00 2024-09-21T09:46:48.885 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: DB pointer 0x55d73dda2000 2024-09-21T09:46:48.885 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: starting mon.c rank 2 at public addrs [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] at bind addrs [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:46:48.885 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T09:46:48.885 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T09:46:48.885 INFO:journalctl@ceph.mon.c.smithi107.stdout: ** DB Stats ** 2024-09-21T09:46:48.885 INFO:journalctl@ceph.mon.c.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:46:48.885 INFO:journalctl@ceph.mon.c.smithi107.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T09:46:48.885 INFO:journalctl@ceph.mon.c.smithi107.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:46:48.886 INFO:journalctl@ceph.mon.c.smithi107.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:46:48.886 INFO:journalctl@ceph.mon.c.smithi107.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T09:46:48.886 INFO:journalctl@ceph.mon.c.smithi107.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:46:48.886 INFO:journalctl@ceph.mon.c.smithi107.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:46:48.886 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-09-21T09:46:48.886 INFO:journalctl@ceph.mon.c.smithi107.stdout: ** Compaction Stats [default] ** 2024-09-21T09:46:48.886 INFO:journalctl@ceph.mon.c.smithi107.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T09:46:48.886 INFO:journalctl@ceph.mon.c.smithi107.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T09:46:48.886 INFO:journalctl@ceph.mon.c.smithi107.stdout: L0 1/0 2.94 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 358.0 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-21T09:46:48.886 INFO:journalctl@ceph.mon.c.smithi107.stdout: L6 1/0 7.39 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-21T09:46:48.886 INFO:journalctl@ceph.mon.c.smithi107.stdout: Sum 2/0 10.33 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 358.0 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-21T09:46:48.886 INFO:journalctl@ceph.mon.c.smithi107.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 358.0 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-21T09:46:48.886 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-09-21T09:46:48.886 INFO:journalctl@ceph.mon.c.smithi107.stdout: ** Compaction Stats [default] ** 2024-09-21T09:46:48.887 INFO:journalctl@ceph.mon.c.smithi107.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T09:46:48.887 INFO:journalctl@ceph.mon.c.smithi107.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:46:48.887 INFO:journalctl@ceph.mon.c.smithi107.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 358.0 0.01 0.00 1 0.008 0 0 0.0 0.0 2024-09-21T09:46:48.887 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-09-21T09:46:48.887 INFO:journalctl@ceph.mon.c.smithi107.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T09:46:48.887 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-09-21T09:46:48.887 INFO:journalctl@ceph.mon.c.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:46:48.887 INFO:journalctl@ceph.mon.c.smithi107.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-09-21T09:46:48.887 INFO:journalctl@ceph.mon.c.smithi107.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:46:48.887 INFO:journalctl@ceph.mon.c.smithi107.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:46:48.887 INFO:journalctl@ceph.mon.c.smithi107.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:46:48.887 INFO:journalctl@ceph.mon.c.smithi107.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:46:48.887 INFO:journalctl@ceph.mon.c.smithi107.stdout: Cumulative compaction: 0.00 GB write, 90.87 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:46:48.887 INFO:journalctl@ceph.mon.c.smithi107.stdout: Interval compaction: 0.00 GB write, 90.87 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:46:48.888 INFO:journalctl@ceph.mon.c.smithi107.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:46:48.888 INFO:journalctl@ceph.mon.c.smithi107.stdout: Block cache BinnedLRUCache@0x55d73dc71610#7 capacity: 512.00 MB usage: 61.78 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.6e-05 secs_since: 0 2024-09-21T09:46:48.888 INFO:journalctl@ceph.mon.c.smithi107.stdout: Block cache entry stats(count,size,portion): DataBlock(4,30.81 KB,0.00587702%) FilterBlock(2,10.81 KB,0.00206232%) IndexBlock(2,20.16 KB,0.0038445%) Misc(1,0.00 KB,0%) 2024-09-21T09:46:48.888 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-09-21T09:46:48.888 INFO:journalctl@ceph.mon.c.smithi107.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:46:48.888 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: mon.c@-1(???) e3 preinit fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:46:48.888 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: mon.c@-1(???).mds e9 new map 2024-09-21T09:46:48.888 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: mon.c@-1(???).mds e9 print_map 2024-09-21T09:46:48.888 INFO:journalctl@ceph.mon.c.smithi107.stdout: e9 2024-09-21T09:46:48.888 INFO:journalctl@ceph.mon.c.smithi107.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-21T09:46:48.888 INFO:journalctl@ceph.mon.c.smithi107.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T09:46:48.888 INFO:journalctl@ceph.mon.c.smithi107.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:46:48.888 INFO:journalctl@ceph.mon.c.smithi107.stdout: legacy client fscid: 1 2024-09-21T09:46:48.888 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-09-21T09:46:48.889 INFO:journalctl@ceph.mon.c.smithi107.stdout: Filesystem 'foo' (1) 2024-09-21T09:46:48.889 INFO:journalctl@ceph.mon.c.smithi107.stdout: fs_name foo 2024-09-21T09:46:48.889 INFO:journalctl@ceph.mon.c.smithi107.stdout: epoch 8 2024-09-21T09:46:48.889 INFO:journalctl@ceph.mon.c.smithi107.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-21T09:46:48.889 INFO:journalctl@ceph.mon.c.smithi107.stdout: created 2024-09-21T09:40:49.090560+0000 2024-09-21T09:46:48.889 INFO:journalctl@ceph.mon.c.smithi107.stdout: modified 2024-09-21T09:41:07.064557+0000 2024-09-21T09:46:48.889 INFO:journalctl@ceph.mon.c.smithi107.stdout: tableserver 0 2024-09-21T09:46:48.889 INFO:journalctl@ceph.mon.c.smithi107.stdout: root 0 2024-09-21T09:46:48.889 INFO:journalctl@ceph.mon.c.smithi107.stdout: session_timeout 60 2024-09-21T09:46:48.889 INFO:journalctl@ceph.mon.c.smithi107.stdout: session_autoclose 300 2024-09-21T09:46:48.889 INFO:journalctl@ceph.mon.c.smithi107.stdout: max_file_size 1099511627776 2024-09-21T09:46:48.889 INFO:journalctl@ceph.mon.c.smithi107.stdout: max_xattr_size 65536 2024-09-21T09:46:48.889 INFO:journalctl@ceph.mon.c.smithi107.stdout: required_client_features {} 2024-09-21T09:46:48.890 INFO:journalctl@ceph.mon.c.smithi107.stdout: last_failure 0 2024-09-21T09:46:48.890 INFO:journalctl@ceph.mon.c.smithi107.stdout: last_failure_osd_epoch 0 2024-09-21T09:46:48.890 INFO:journalctl@ceph.mon.c.smithi107.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:46:48.890 INFO:journalctl@ceph.mon.c.smithi107.stdout: max_mds 1 2024-09-21T09:46:48.890 INFO:journalctl@ceph.mon.c.smithi107.stdout: in 0 2024-09-21T09:46:48.890 INFO:journalctl@ceph.mon.c.smithi107.stdout: up {0=24557} 2024-09-21T09:46:48.890 INFO:journalctl@ceph.mon.c.smithi107.stdout: failed 2024-09-21T09:46:48.890 INFO:journalctl@ceph.mon.c.smithi107.stdout: damaged 2024-09-21T09:46:48.890 INFO:journalctl@ceph.mon.c.smithi107.stdout: stopped 2024-09-21T09:46:48.890 INFO:journalctl@ceph.mon.c.smithi107.stdout: data_pools [4] 2024-09-21T09:46:48.891 INFO:journalctl@ceph.mon.c.smithi107.stdout: metadata_pool 3 2024-09-21T09:46:48.891 INFO:journalctl@ceph.mon.c.smithi107.stdout: inline_data disabled 2024-09-21T09:46:48.891 INFO:journalctl@ceph.mon.c.smithi107.stdout: balancer 2024-09-21T09:46:48.891 INFO:journalctl@ceph.mon.c.smithi107.stdout: bal_rank_mask -1 2024-09-21T09:46:48.891 INFO:journalctl@ceph.mon.c.smithi107.stdout: standby_count_wanted 1 2024-09-21T09:46:48.891 INFO:journalctl@ceph.mon.c.smithi107.stdout: qdb_cluster leader: 0 members: 2024-09-21T09:46:48.891 INFO:journalctl@ceph.mon.c.smithi107.stdout: [mds.foo.smithi107.sedied{0:24557} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.107:6834/2570718287,v1:172.21.15.107:6835/2570718287] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T09:46:48.891 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-09-21T09:46:48.891 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-09-21T09:46:48.891 INFO:journalctl@ceph.mon.c.smithi107.stdout: Standby daemons: 2024-09-21T09:46:48.891 INFO:journalctl@ceph.mon.c.smithi107.stdout: 2024-09-21T09:46:48.891 INFO:journalctl@ceph.mon.c.smithi107.stdout: [mds.foo.smithi129.zdzmqr{-1:24526} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.129:6832/194235284,v1:172.21.15.129:6833/194235284] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T09:46:48.891 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: mon.c@-1(???).osd e77 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-21T09:46:48.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: mon.c@-1(???).osd e77 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T09:46:48.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: mon.c@-1(???).osd e77 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T09:46:48.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: mon.c@-1(???).osd e77 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T09:46:48.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: mon.c@-1(???).paxosservice(auth 1..20) refresh upgraded, format 0 -> 3 2024-09-21T09:46:48.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: mon.c@-1(???).mgr e0 loading version 40 2024-09-21T09:46:48.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: mon.c@-1(???).mgr e40 active server: [v2:172.21.15.107:6800/1060510503,v1:172.21.15.107:6801/1060510503](25219) 2024-09-21T09:46:48.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 ceph-mon[105022]: mon.c@-1(???).mgr e40 mkfs or daemon transitioned to available, loading commands 2024-09-21T09:46:48.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 bash[104959]: 04f7b77f15006980802ffeb5bdcc25491298cdb8dcae3a9f8a0733805698feae 2024-09-21T09:46:48.892 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:48 smithi107 systemd[1]: Started Ceph mon.c for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:46:49.680 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 21 16 50260 50244 149.523 117.875 0.00407148 0.00668307 2024-09-21T09:46:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:50.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:50.680 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 22 16 52212 52196 148.271 122 0.00299216 0.00673892 2024-09-21T09:46:51.680 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 23 16 54118 54102 147.003 119.125 0.00616264 0.00679777 2024-09-21T09:46:52.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:46:51] "GET /metrics HTTP/1.1" 200 36404 "" "Prometheus/2.51.0" 2024-09-21T09:46:52.680 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 24 16 55972 55956 145.706 115.875 0.00701572 0.00685843 2024-09-21T09:46:53.680 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 25 16 57646 57630 144.062 104.625 0.012253 0.00693681 2024-09-21T09:46:54.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:54 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:54.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:54.681 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 26 16 58964 58948 141.688 82.375 0.00600402 0.0070512 2024-09-21T09:46:55.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[102221]: mon.c calling monitor election 2024-09-21T09:46:55.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[102221]: mon.b calling monitor election 2024-09-21T09:46:55.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[102221]: mon.a calling monitor election 2024-09-21T09:46:55.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[102221]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:46:55.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[102221]: monmap epoch 3 2024-09-21T09:46:55.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[102221]: fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:46:55.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[102221]: last_changed 2024-09-21T09:35:39.850868+0000 2024-09-21T09:46:55.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[102221]: created 2024-09-21T09:34:00.747992+0000 2024-09-21T09:46:55.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[102221]: min_mon_release 17 (quincy) 2024-09-21T09:46:55.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[102221]: election_strategy: 1 2024-09-21T09:46:55.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[102221]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-09-21T09:46:55.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[102221]: 1: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.b 2024-09-21T09:46:55.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[102221]: 2: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.c 2024-09-21T09:46:55.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[102221]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:46:55.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[102221]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T09:46:55.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[102221]: mgrmap e40: y(active, since 28s), standbys: x 2024-09-21T09:46:55.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[102221]: overall HEALTH_OK 2024-09-21T09:46:55.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[105022]: mon.c calling monitor election 2024-09-21T09:46:55.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[105022]: mon.b calling monitor election 2024-09-21T09:46:55.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[105022]: mon.a calling monitor election 2024-09-21T09:46:55.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[105022]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:46:55.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[105022]: monmap epoch 3 2024-09-21T09:46:55.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[105022]: fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:46:55.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[105022]: last_changed 2024-09-21T09:35:39.850868+0000 2024-09-21T09:46:55.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[105022]: created 2024-09-21T09:34:00.747992+0000 2024-09-21T09:46:55.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[105022]: min_mon_release 17 (quincy) 2024-09-21T09:46:55.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[105022]: election_strategy: 1 2024-09-21T09:46:55.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[105022]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-09-21T09:46:55.113 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[105022]: 1: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.b 2024-09-21T09:46:55.113 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[105022]: 2: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.c 2024-09-21T09:46:55.113 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[105022]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:46:55.113 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[105022]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T09:46:55.113 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[105022]: mgrmap e40: y(active, since 28s), standbys: x 2024-09-21T09:46:55.113 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:54 smithi107 ceph-mon[105022]: overall HEALTH_OK 2024-09-21T09:46:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:54 smithi129 ceph-mon[32729]: mon.c calling monitor election 2024-09-21T09:46:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:54 smithi129 ceph-mon[32729]: mon.b calling monitor election 2024-09-21T09:46:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:54 smithi129 ceph-mon[32729]: mon.a calling monitor election 2024-09-21T09:46:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:54 smithi129 ceph-mon[32729]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:46:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:54 smithi129 ceph-mon[32729]: monmap epoch 3 2024-09-21T09:46:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:54 smithi129 ceph-mon[32729]: fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:46:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:54 smithi129 ceph-mon[32729]: last_changed 2024-09-21T09:35:39.850868+0000 2024-09-21T09:46:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:54 smithi129 ceph-mon[32729]: created 2024-09-21T09:34:00.747992+0000 2024-09-21T09:46:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:54 smithi129 ceph-mon[32729]: min_mon_release 17 (quincy) 2024-09-21T09:46:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:54 smithi129 ceph-mon[32729]: election_strategy: 1 2024-09-21T09:46:55.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:54 smithi129 ceph-mon[32729]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-09-21T09:46:55.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:54 smithi129 ceph-mon[32729]: 1: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.b 2024-09-21T09:46:55.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:54 smithi129 ceph-mon[32729]: 2: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.c 2024-09-21T09:46:55.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:54 smithi129 ceph-mon[32729]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:46:55.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:54 smithi129 ceph-mon[32729]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T09:46:55.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:54 smithi129 ceph-mon[32729]: mgrmap e40: y(active, since 28s), standbys: x 2024-09-21T09:46:55.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:54 smithi129 ceph-mon[32729]: overall HEALTH_OK 2024-09-21T09:46:55.447 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:46:55.681 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 27 16 60331 60315 139.601 85.4375 0.0143013 0.00715834 2024-09-21T09:46:56.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:55 smithi107 ceph-mon[102221]: from='client.25357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:56.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:55 smithi107 ceph-mon[105022]: from='client.25357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:56.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:55 smithi129 ceph-mon[32729]: from='client.25357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:56.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:55 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:55.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:56.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:55 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:55.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:56.682 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 28 16 61820 61804 137.939 93.0625 0.00734169 0.00724473 2024-09-21T09:46:57.105 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:56 smithi107 ceph-mon[102221]: pgmap v19: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-21T09:46:57.105 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:56 smithi107 ceph-mon[102221]: from='client.25363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:57.105 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:46:57.106 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:56 smithi107 ceph-mon[105022]: pgmap v19: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-21T09:46:57.106 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:56 smithi107 ceph-mon[105022]: from='client.25363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:57.106 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:46:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:56 smithi129 ceph-mon[32729]: pgmap v19: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-21T09:46:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:56 smithi129 ceph-mon[32729]: from='client.25363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:46:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:56 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:46:57.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:46:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:46:56.950Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:46:57.682 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 29 16 63335 63319 136.447 94.6875 0.00588377 0.0073236 2024-09-21T09:46:58.585 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T09:46:58.682 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 30 16 64576 64560 134.484 77.5625 0.0132182 0.00743107 2024-09-21T09:46:58.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:46:58 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:46:58.473+0000 7efdb5a17640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-21T09:46:59.682 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 31 16 66086 66070 133.19 94.375 0.00497164 0.00750309 2024-09-21T09:46:59.841 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:59 smithi107 ceph-mon[102221]: pgmap v20: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-09-21T09:46:59.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:46:59 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2769161420' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T09:46:59.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:59 smithi107 ceph-mon[105022]: pgmap v20: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-09-21T09:46:59.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:46:59 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2769161420' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T09:46:59.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:59 smithi129 ceph-mon[32729]: pgmap v20: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-09-21T09:46:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:46:59 smithi129 ceph-mon[32729]: from='client.? 172.21.15.107:0/2769161420' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T09:47:00.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch ps' 2024-09-21T09:47:00.668 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:00.682 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 32 16 67548 67532 131.883 91.375 0.00712173 0.0075776 2024-09-21T09:47:00.899 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:00 smithi107 ceph-mon[102221]: pgmap v21: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T09:47:00.899 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:00 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:01.153 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:47:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:00 smithi129 ceph-mon[32729]: pgmap v21: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T09:47:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:00 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:00 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:01.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:00 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:01.291 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:00 smithi107 ceph-mon[105022]: pgmap v21: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T09:47:01.291 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:00 smithi107 ceph-mon[105022]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:01.291 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:00 smithi107 ceph-mon[105022]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:01.682 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 33 16 68771 68755 130.203 76.4375 0.0224179 0.00767486 2024-09-21T09:47:02.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:47:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:47:01] "GET /metrics HTTP/1.1" 200 36402 "" "Prometheus/2.51.0" 2024-09-21T09:47:02.414 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:02 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:02.415 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:02 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:02 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:02 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:02.683 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 34 16 70233 70217 129.056 91.375 0.0234901 0.00774348 2024-09-21T09:47:02.785 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:02 smithi107 ceph-mon[105022]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:02.786 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:02 smithi107 ceph-mon[105022]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:03.684 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 35 16 71476 71460 127.588 77.6875 0.0453798 0.00783251 2024-09-21T09:47:03.752 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:03 smithi107 ceph-mon[102221]: pgmap v22: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T09:47:03.753 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:03 smithi107 ceph-mon[105022]: pgmap v22: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T09:47:03.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:03 smithi129 ceph-mon[32729]: pgmap v22: 89 pgs: 89 active+clean; 3.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T09:47:04.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:04 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:04.146Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:04.684 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 36 15 72700 72685 126.17 76.5625 0.0164888 0.00792038 2024-09-21T09:47:05.684 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 37 16 73735 73719 124.507 64.625 0.0107372 0.00802646 2024-09-21T09:47:05.742 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:05 smithi129 ceph-mon[32729]: pgmap v23: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 124 MiB/s wr, 1.99k op/s 2024-09-21T09:47:05.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:05 smithi107 ceph-mon[102221]: pgmap v23: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 124 MiB/s wr, 1.99k op/s 2024-09-21T09:47:05.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:05 smithi107 ceph-mon[105022]: pgmap v23: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 124 MiB/s wr, 1.99k op/s 2024-09-21T09:47:06.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:05 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:06.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:05 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:05.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:06.684 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 38 16 74854 74838 123.071 69.9375 0.0137707 0.00811952 2024-09-21T09:47:07.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:06 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:06.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:07.466 INFO:teuthology.orchestra.run.smithi107.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T09:47:07.466 INFO:teuthology.orchestra.run.smithi107.stdout:alertmanager.a smithi107 *:9093,9094 running (2m) 6s ago 7m 19.1M - 0.27.0 11f11916f8cd 6fa4ebf1f69d 2024-09-21T09:47:07.467 INFO:teuthology.orchestra.run.smithi107.stdout:grafana.a smithi129 *:3000 running (2m) 35s ago 7m 75.1M - 9.4.7 954c08fa6188 72cc8ff672f5 2024-09-21T09:47:07.467 INFO:teuthology.orchestra.run.smithi107.stdout:mds.foo.smithi107.sedied smithi107 running (6m) 6s ago 6m 19.7M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 83d74619fe7a 2024-09-21T09:47:07.467 INFO:teuthology.orchestra.run.smithi107.stdout:mds.foo.smithi129.zdzmqr smithi129 running (6m) 35s ago 6m 15.0M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 2133ce1b60f9 2024-09-21T09:47:07.467 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.x smithi129 *:8443,9283,8765 running (68s) 35s ago 11m 453M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 59e80c9df333 2024-09-21T09:47:07.467 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.y smithi107 *:8443,9283,8765 running (108s) 6s ago 12m 506M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 849409c95398 2024-09-21T09:47:07.467 INFO:teuthology.orchestra.run.smithi107.stdout:mon.a smithi107 running (48s) 6s ago 13m 39.1M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 34ae11d9ef1b 2024-09-21T09:47:07.467 INFO:teuthology.orchestra.run.smithi107.stdout:mon.b smithi129 running (11m) 35s ago 11m 49.9M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 589a4d4f8d6b 2024-09-21T09:47:07.467 INFO:teuthology.orchestra.run.smithi107.stdout:mon.c smithi107 running (18s) 6s ago 11m 21.7M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 04f7b77f1500 2024-09-21T09:47:07.467 INFO:teuthology.orchestra.run.smithi107.stdout:node-exporter.a smithi107 *:9100 running (2m) 6s ago 6m 11.3M - 1.7.0 72c9c2088986 2444eabb7c7e 2024-09-21T09:47:07.467 INFO:teuthology.orchestra.run.smithi107.stdout:node-exporter.b smithi129 *:9100 running (2m) 35s ago 6m 8254k - 1.7.0 72c9c2088986 ce5db8669c3d 2024-09-21T09:47:07.467 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 smithi107 running (10m) 6s ago 10m 323M 2479M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 96047e77ae78 2024-09-21T09:47:07.467 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1 smithi107 running (10m) 6s ago 10m 235M 2479M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 508718820c25 2024-09-21T09:47:07.467 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2 smithi107 running (10m) 6s ago 10m 311M 2479M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be ae11dfe6a988 2024-09-21T09:47:07.467 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3 smithi107 running (9m) 6s ago 9m 271M 2479M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be cf6bac271e88 2024-09-21T09:47:07.467 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4 smithi129 running (9m) 35s ago 9m 285M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 7c6ca82e1ac1 2024-09-21T09:47:07.468 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5 smithi129 running (8m) 35s ago 8m 296M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 9ae131a2ef13 2024-09-21T09:47:07.468 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6 smithi129 running (8m) 35s ago 8m 167M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be a5ccfa3e24c2 2024-09-21T09:47:07.468 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7 smithi129 running (8m) 35s ago 8m 208M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be a32c9bd56124 2024-09-21T09:47:07.468 INFO:teuthology.orchestra.run.smithi107.stdout:prometheus.a smithi129 *:9095 running (76s) 35s ago 7m 44.9M - 2.51.0 1d3b7f56885b 24d10324469f 2024-09-21T09:47:07.684 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 39 16 76210 76194 122.088 84.75 0.00493876 0.00818575 2024-09-21T09:47:07.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:07 smithi107 ceph-mon[102221]: pgmap v24: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-21T09:47:07.777 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:07 smithi107 ceph-mon[105022]: pgmap v24: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-21T09:47:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:07 smithi129 ceph-mon[32729]: pgmap v24: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-21T09:47:08.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade pause' 2024-09-21T09:47:08.684 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T09:47:08.683375+0000 min lat: 0.00113027 max lat: 0.18541 avg lat: 0.00825257 2024-09-21T09:47:08.684 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T09:47:08.684 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 40 16 77531 77515 121.1 82.5625 0.0157358 0.00825257 2024-09-21T09:47:08.729 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:08 smithi129 ceph-mon[32729]: from='client.34266 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:47:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:08 smithi107 ceph-mon[102221]: from='client.34266 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:47:08.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:08 smithi107 ceph-mon[105022]: from='client.34266 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:47:09.402 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:47:09.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:09 smithi107 ceph-mon[102221]: pgmap v25: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-21T09:47:09.544 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:09 smithi107 ceph-mon[105022]: pgmap v25: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-21T09:47:09.685 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 41 16 78782 78766 120.05 78.1875 0.0151684 0.00832399 2024-09-21T09:47:09.842 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:09 smithi129 ceph-mon[32729]: pgmap v25: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-21T09:47:10.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:10.685 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 42 16 80071 80055 119.11 80.5625 0.01717 0.00839069 2024-09-21T09:47:11.685 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 43 16 81312 81296 118.144 77.5625 0.00482192 0.00845951 2024-09-21T09:47:11.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:11 smithi107 ceph-mon[102221]: pgmap v26: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-21T09:47:11.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:47:11.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:11 smithi107 ceph-mon[105022]: pgmap v26: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-21T09:47:11.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:47:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:11 smithi129 ceph-mon[32729]: pgmap v26: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-21T09:47:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:11 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:47:12.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:47:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:47:11] "GET /metrics HTTP/1.1" 200 36398 "" "Prometheus/2.51.0" 2024-09-21T09:47:12.685 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 44 16 82529 82513 117.187 76.0625 0.0110025 0.00852887 2024-09-21T09:47:13.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:13 smithi107 ceph-mon[102221]: pgmap v27: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-21T09:47:13.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:13 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:13.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:13 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:13.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:13.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:13.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:13 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:13.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:13.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:47:13.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T09:47:13.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-21T09:47:13.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:13 smithi107 ceph-mon[105022]: pgmap v27: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-21T09:47:13.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:13 smithi107 ceph-mon[105022]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:13.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:13 smithi107 ceph-mon[105022]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:13.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:13.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:13.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:13 smithi107 ceph-mon[105022]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:13.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:13.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:47:13.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T09:47:13.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-21T09:47:13.685 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 45 16 83730 83714 116.251 75.0625 0.0172535 0.00859737 2024-09-21T09:47:13.807 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:13 smithi129 ceph-mon[32729]: pgmap v27: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-21T09:47:13.807 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:13 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:13.807 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:13 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:13.807 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:13 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:13.807 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:13 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:47:13.807 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:13 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:13.807 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:13 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:47:13.807 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:13 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T09:47:13.807 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:13 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T09:47:13.808 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:13 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-21T09:47:13.846 INFO:teuthology.orchestra.run.smithi107.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:47:14.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:14 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:14 smithi107 ceph-mon[102221]: Upgrade: It appears safe to stop mon.b 2024-09-21T09:47:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:14 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:14 smithi107 ceph-mon[105022]: Upgrade: It appears safe to stop mon.b 2024-09-21T09:47:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:14 smithi107 ceph-mon[105022]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:14.686 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 46 16 85023 85007 115.48 80.8125 0.0118472 0.00865417 2024-09-21T09:47:14.713 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'sleep 60' 2024-09-21T09:47:14.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:14 smithi129 ceph-mon[32729]: Upgrade: It appears safe to stop mon.b 2024-09-21T09:47:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:14 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:15.109 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:47:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:15 smithi107 ceph-mon[102221]: pgmap v28: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T09:47:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:15 smithi107 ceph-mon[102221]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:47:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:15 smithi107 ceph-mon[102221]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:47:15.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:15 smithi107 ceph-mon[105022]: pgmap v28: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T09:47:15.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:15 smithi107 ceph-mon[105022]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:47:15.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:15 smithi107 ceph-mon[105022]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:47:15.686 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 47 16 86101 86085 114.457 67.375 0.0138275 0.00873159 2024-09-21T09:47:15.723 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:15 smithi129 ceph-mon[32729]: pgmap v28: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T09:47:15.723 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:15 smithi129 ceph-mon[32729]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:47:15.723 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:15 smithi129 ceph-mon[32729]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:47:16.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:15.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:16.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:15.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:16.686 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 48 16 87363 87347 113.715 78.875 0.0116594 0.00878894 2024-09-21T09:47:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:16 smithi107 ceph-mon[102221]: Upgrade: Updating mon.b 2024-09-21T09:47:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:16 smithi107 ceph-mon[102221]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:16 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:47:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:16 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:47:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:16 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:16 smithi107 ceph-mon[102221]: Deploying daemon mon.b on smithi129 2024-09-21T09:47:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:16 smithi107 ceph-mon[102221]: pgmap v29: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T09:47:16.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:16 smithi107 ceph-mon[105022]: Upgrade: Updating mon.b 2024-09-21T09:47:16.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:16 smithi107 ceph-mon[105022]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:16.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:16 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:47:16.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:16 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:47:16.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:16 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:16.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:16 smithi107 ceph-mon[105022]: Deploying daemon mon.b on smithi129 2024-09-21T09:47:16.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:16 smithi107 ceph-mon[105022]: pgmap v29: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T09:47:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:16 smithi129 ceph-mon[32729]: Upgrade: Updating mon.b 2024-09-21T09:47:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:16 smithi129 ceph-mon[32729]: from='mgr.25219 ' entity='mgr.y' 2024-09-21T09:47:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:16 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:47:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:16 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:47:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:16 smithi129 ceph-mon[32729]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:47:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:16 smithi129 ceph-mon[32729]: Deploying daemon mon.b on smithi129 2024-09-21T09:47:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:16 smithi129 ceph-mon[32729]: pgmap v29: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T09:47:17.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:16 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:16.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:17.686 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 49 16 88670 88654 113.062 81.6875 0.0175244 0.00883991 2024-09-21T09:47:18.375 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:17 smithi129 systemd[1]: Stopping Ceph mon.b for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:47:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:18 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-b[32706]: 2024-09-21T09:47:18.372+0000 7f9d09f04640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T09:47:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:18 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-b[32706]: 2024-09-21T09:47:18.372+0000 7f9d09f04640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-21T09:47:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:18 smithi129 podman[88298]: 2024-09-21 09:47:18.528324742 +0000 UTC m=+0.277987181 container died 589a4d4f8d6b5cf8cc4f00d3c6e5428d27a4737214535881eda37d7d74850081 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a) 2024-09-21T09:47:18.686 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 50 16 89926 89910 112.37 78.5 0.0279678 0.00889375 2024-09-21T09:47:18.780 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:47:18 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:47:18] ENGINE Bus STOPPING 2024-09-21T09:47:18.781 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:47:18 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:47:18] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:47:19.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:47:18 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:47:18] ENGINE Bus STOPPED 2024-09-21T09:47:19.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:47:18 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:47:18] ENGINE Bus STARTING 2024-09-21T09:47:19.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:47:18 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:47:18] ENGINE Serving on http://:::9283 2024-09-21T09:47:19.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:47:18 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:09:47:18] ENGINE Bus STARTED 2024-09-21T09:47:19.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:19 smithi129 podman[88298]: 2024-09-21 09:47:19.379460454 +0000 UTC m=+1.129122887 container cleanup 589a4d4f8d6b5cf8cc4f00d3c6e5428d27a4737214535881eda37d7d74850081 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-b, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-21T09:47:19.394 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:19 smithi129 bash[88298]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-b 2024-09-21T09:47:19.686 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 51 16 91135 91119 111.649 75.5625 0.011278 0.00895177 2024-09-21T09:47:19.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:19 smithi129 podman[88314]: 2024-09-21 09:47:19.698305213 +0000 UTC m=+1.173061784 container remove 589a4d4f8d6b5cf8cc4f00d3c6e5428d27a4737214535881eda37d7d74850081 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-b, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-21T09:47:19.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:19 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.b.service: Deactivated successfully. 2024-09-21T09:47:19.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:19 smithi129 systemd[1]: Stopped Ceph mon.b for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:47:19.956 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:19 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.b.service: Consumed 10.019s CPU time. 2024-09-21T09:47:20.528 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:20 smithi129 systemd[1]: Starting Ceph mon.b for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:47:20.528 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:20.686 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 52 16 92377 92361 110.994 77.625 0.0107666 0.00900462 2024-09-21T09:47:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:20 smithi129 podman[88435]: 2024-09-21 09:47:20.435354349 +0000 UTC m=+0.037200252 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:47:21.438 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 podman[88435]: 2024-09-21 09:47:21.175145273 +0000 UTC m=+0.776991176 container create b01038fb71fdf70d509f346955879d8e3df264048d40a1f71577033c05f29130 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-b, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-09-21T09:47:21.686 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 53 16 93560 93544 110.295 73.9375 0.0170858 0.00906179 2024-09-21T09:47:21.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T09:47:21.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T09:47:21.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T09:47:21.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T09:47:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T09:47:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T09:47:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T09:47:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T09:47:21.860 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:47:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:47:21] "GET /metrics HTTP/1.1" 200 36398 "" "Prometheus/2.51.0" 2024-09-21T09:47:21.924 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 podman[88435]: 2024-09-21 09:47:21.436703601 +0000 UTC m=+1.038549507 container init b01038fb71fdf70d509f346955879d8e3df264048d40a1f71577033c05f29130 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-b, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T09:47:21.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 podman[88435]: 2024-09-21 09:47:21.445287939 +0000 UTC m=+1.047133839 container start b01038fb71fdf70d509f346955879d8e3df264048d40a1f71577033c05f29130 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-b, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-21T09:47:21.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T09:47:21.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev), process ceph-mon, pid 7 2024-09-21T09:47:21.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: pidfile_write: ignore empty --pid-file 2024-09-21T09:47:21.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: load: jerasure load: lrc 2024-09-21T09:47:21.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: RocksDB version: 7.9.2 2024-09-21T09:47:21.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Git sha 0 2024-09-21T09:47:21.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Compile date 2024-09-20 20:32:28 2024-09-21T09:47:21.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: DB SUMMARY 2024-09-21T09:47:21.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: DB Session ID: D0E0NPZTG7G8MG3M9ZDG 2024-09-21T09:47:21.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: CURRENT file: CURRENT 2024-09-21T09:47:21.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T09:47:21.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1103 Bytes 2024-09-21T09:47:21.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-09-21T09:47:21.925 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 5631906 ; 2024-09-21T09:47:21.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.error_if_exists: 0 2024-09-21T09:47:21.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.create_if_missing: 0 2024-09-21T09:47:21.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.paranoid_checks: 1 2024-09-21T09:47:21.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T09:47:21.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T09:47:21.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T09:47:21.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.env: 0x55782d9bc120 2024-09-21T09:47:21.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T09:47:21.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.info_log: 0x55782ecbf700 2024-09-21T09:47:21.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T09:47:21.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.statistics: (nil) 2024-09-21T09:47:21.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.use_fsync: 0 2024-09-21T09:47:21.926 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_log_file_size: 0 2024-09-21T09:47:21.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T09:47:21.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T09:47:21.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T09:47:21.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T09:47:21.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.allow_fallocate: 1 2024-09-21T09:47:21.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T09:47:21.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T09:47:21.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.use_direct_reads: 0 2024-09-21T09:47:21.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T09:47:21.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T09:47:21.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.db_log_dir: 2024-09-21T09:47:21.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.wal_dir: 2024-09-21T09:47:21.927 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T09:47:21.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T09:47:21.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T09:47:21.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T09:47:21.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T09:47:21.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T09:47:21.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T09:47:21.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T09:47:21.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.write_buffer_manager: 0x55782ecc21e0 2024-09-21T09:47:21.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T09:47:21.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T09:47:21.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T09:47:21.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T09:47:21.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T09:47:21.928 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T09:47:21.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T09:47:21.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T09:47:21.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.unordered_write: 0 2024-09-21T09:47:21.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T09:47:21.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T09:47:21.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T09:47:21.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T09:47:21.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.row_cache: None 2024-09-21T09:47:21.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.wal_filter: None 2024-09-21T09:47:21.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T09:47:21.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T09:47:21.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.two_write_queues: 0 2024-09-21T09:47:21.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T09:47:21.929 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.wal_compression: 0 2024-09-21T09:47:21.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.atomic_flush: 0 2024-09-21T09:47:21.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T09:47:21.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T09:47:21.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T09:47:21.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.log_readahead_size: 0 2024-09-21T09:47:21.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T09:47:21.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T09:47:21.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T09:47:21.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T09:47:21.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T09:47:21.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T09:47:21.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T09:47:21.930 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_background_jobs: 2 2024-09-21T09:47:21.931 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_background_compactions: -1 2024-09-21T09:47:21.931 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_subcompactions: 1 2024-09-21T09:47:21.931 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T09:47:21.931 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T09:47:21.931 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T09:47:21.931 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T09:47:21.931 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T09:47:21.931 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T09:47:21.931 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T09:47:21.931 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T09:47:21.931 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_open_files: -1 2024-09-21T09:47:21.931 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T09:47:21.931 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T09:47:21.932 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T09:47:21.932 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T09:47:21.932 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_background_flushes: -1 2024-09-21T09:47:21.932 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Compression algorithms supported: 2024-09-21T09:47:21.932 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: kZSTD supported: 0 2024-09-21T09:47:21.932 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: kXpressCompression supported: 0 2024-09-21T09:47:21.932 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: kBZip2Compression supported: 0 2024-09-21T09:47:21.932 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T09:47:21.932 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: kLZ4Compression supported: 1 2024-09-21T09:47:21.932 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: kZlibCompression supported: 1 2024-09-21T09:47:21.932 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T09:47:21.932 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: kSnappyCompression supported: 1 2024-09-21T09:47:21.932 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T09:47:21.932 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T09:47:21.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-09-21T09:47:21.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T09:47:21.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T09:47:21.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.merge_operator: 2024-09-21T09:47:21.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compaction_filter: None 2024-09-21T09:47:21.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T09:47:21.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T09:47:21.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T09:47:21.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T09:47:21.933 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55782ecbf320) 2024-09-21T09:47:21.933 INFO:journalctl@ceph.mon.b.smithi129.stdout: cache_index_and_filter_blocks: 1 2024-09-21T09:47:21.933 INFO:journalctl@ceph.mon.b.smithi129.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T09:47:21.933 INFO:journalctl@ceph.mon.b.smithi129.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T09:47:21.934 INFO:journalctl@ceph.mon.b.smithi129.stdout: pin_top_level_index_and_filter: 1 2024-09-21T09:47:21.934 INFO:journalctl@ceph.mon.b.smithi129.stdout: index_type: 0 2024-09-21T09:47:21.934 INFO:journalctl@ceph.mon.b.smithi129.stdout: data_block_index_type: 0 2024-09-21T09:47:21.934 INFO:journalctl@ceph.mon.b.smithi129.stdout: index_shortening: 1 2024-09-21T09:47:21.934 INFO:journalctl@ceph.mon.b.smithi129.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T09:47:21.934 INFO:journalctl@ceph.mon.b.smithi129.stdout: checksum: 4 2024-09-21T09:47:21.934 INFO:journalctl@ceph.mon.b.smithi129.stdout: no_block_cache: 0 2024-09-21T09:47:21.934 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_cache: 0x55782ecb3610 2024-09-21T09:47:21.934 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_cache_name: BinnedLRUCache 2024-09-21T09:47:21.934 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_cache_options: 2024-09-21T09:47:21.934 INFO:journalctl@ceph.mon.b.smithi129.stdout: capacity : 536870912 2024-09-21T09:47:21.934 INFO:journalctl@ceph.mon.b.smithi129.stdout: num_shard_bits : 4 2024-09-21T09:47:21.934 INFO:journalctl@ceph.mon.b.smithi129.stdout: strict_capacity_limit : 0 2024-09-21T09:47:21.934 INFO:journalctl@ceph.mon.b.smithi129.stdout: high_pri_pool_ratio: 0.000 2024-09-21T09:47:21.935 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_cache_compressed: (nil) 2024-09-21T09:47:21.935 INFO:journalctl@ceph.mon.b.smithi129.stdout: persistent_cache: (nil) 2024-09-21T09:47:21.935 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_size: 4096 2024-09-21T09:47:21.935 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_size_deviation: 10 2024-09-21T09:47:21.935 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_restart_interval: 16 2024-09-21T09:47:21.935 INFO:journalctl@ceph.mon.b.smithi129.stdout: index_block_restart_interval: 1 2024-09-21T09:47:21.935 INFO:journalctl@ceph.mon.b.smithi129.stdout: metadata_block_size: 4096 2024-09-21T09:47:21.935 INFO:journalctl@ceph.mon.b.smithi129.stdout: partition_filters: 0 2024-09-21T09:47:21.935 INFO:journalctl@ceph.mon.b.smithi129.stdout: use_delta_encoding: 1 2024-09-21T09:47:21.935 INFO:journalctl@ceph.mon.b.smithi129.stdout: filter_policy: bloomfilter 2024-09-21T09:47:21.935 INFO:journalctl@ceph.mon.b.smithi129.stdout: whole_key_filtering: 1 2024-09-21T09:47:21.935 INFO:journalctl@ceph.mon.b.smithi129.stdout: verify_compression: 0 2024-09-21T09:47:21.936 INFO:journalctl@ceph.mon.b.smithi129.stdout: read_amp_bytes_per_bit: 0 2024-09-21T09:47:21.936 INFO:journalctl@ceph.mon.b.smithi129.stdout: format_version: 5 2024-09-21T09:47:21.936 INFO:journalctl@ceph.mon.b.smithi129.stdout: enable_index_compression: 1 2024-09-21T09:47:21.936 INFO:journalctl@ceph.mon.b.smithi129.stdout: block_align: 0 2024-09-21T09:47:21.936 INFO:journalctl@ceph.mon.b.smithi129.stdout: max_auto_readahead_size: 262144 2024-09-21T09:47:21.936 INFO:journalctl@ceph.mon.b.smithi129.stdout: prepopulate_block_cache: 0 2024-09-21T09:47:21.936 INFO:journalctl@ceph.mon.b.smithi129.stdout: initial_auto_readahead_size: 8192 2024-09-21T09:47:21.936 INFO:journalctl@ceph.mon.b.smithi129.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T09:47:21.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T09:47:21.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T09:47:21.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compression: NoCompression 2024-09-21T09:47:21.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T09:47:21.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T09:47:21.936 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T09:47:21.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.num_levels: 7 2024-09-21T09:47:21.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T09:47:21.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T09:47:21.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T09:47:21.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T09:47:21.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T09:47:21.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T09:47:21.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T09:47:21.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:47:21.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T09:47:21.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T09:47:21.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T09:47:21.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T09:47:21.937 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T09:47:21.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T09:47:21.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T09:47:21.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T09:47:21.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:47:21.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T09:47:21.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T09:47:21.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T09:47:21.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T09:47:21.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T09:47:21.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T09:47:21.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T09:47:21.938 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T09:47:21.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T09:47:21.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T09:47:21.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T09:47:21.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T09:47:21.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T09:47:21.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T09:47:21.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T09:47:21.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T09:47:21.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T09:47:21.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T09:47:21.939 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T09:47:21.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T09:47:21.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T09:47:21.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T09:47:21.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T09:47:21.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T09:47:21.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T09:47:21.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T09:47:21.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T09:47:21.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T09:47:21.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T09:47:21.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T09:47:21.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T09:47:21.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T09:47:21.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T09:47:21.940 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T09:47:21.941 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T09:47:21.941 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T09:47:21.941 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T09:47:21.941 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.inplace_update_support: 0 2024-09-21T09:47:21.941 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T09:47:21.941 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T09:47:21.941 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T09:47:21.941 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T09:47:21.941 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.bloom_locality: 0 2024-09-21T09:47:21.941 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.max_successive_merges: 0 2024-09-21T09:47:21.941 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T09:47:21.941 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T09:47:21.941 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T09:47:21.941 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T09:47:21.942 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.ttl: 2592000 2024-09-21T09:47:21.942 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T09:47:21.942 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T09:47:21.942 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T09:47:21.942 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.enable_blob_files: false 2024-09-21T09:47:21.942 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.min_blob_size: 0 2024-09-21T09:47:21.942 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T09:47:21.942 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T09:47:21.942 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T09:47:21.942 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T09:47:21.942 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T09:47:21.942 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T09:47:21.942 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T09:47:21.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T09:47:21.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-09-21T09:47:21.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T09:47:21.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 12816, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T09:47:21.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-21T09:47:21.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ce344556-834e-436c-bce9-a300bc6eb641 2024-09-21T09:47:21.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912041489205, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-21T09:47:21.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-21T09:47:21.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912041512904, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 3477223, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12821, "largest_seqno": 14613, "table_properties": {"data_size": 3469439, "index_size": 4818, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1925, "raw_key_size": 17318, "raw_average_key_size": 23, "raw_value_size": 3453712, "raw_average_value_size": 4629, "num_data_blocks": 218, "num_entries": 746, "num_filter_entries": 746, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726912041, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ce344556-834e-436c-bce9-a300bc6eb641", "db_session_id": "D0E0NPZTG7G8MG3M9ZDG", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-21T09:47:21.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912041513052, "job": 1, "event": "recovery_finished"} 2024-09-21T09:47:21.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-21T09:47:21.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T09:47:21.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T09:47:21.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55782ece2e00 2024-09-21T09:47:21.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: DB pointer 0x55782edf6000 2024-09-21T09:47:21.943 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T09:47:21.944 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T09:47:21.944 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** DB Stats ** 2024-09-21T09:47:21.944 INFO:journalctl@ceph.mon.b.smithi129.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:47:21.944 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T09:47:21.944 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:47:21.944 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:47:21.944 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T09:47:21.944 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:47:21.944 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:47:21.944 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T09:47:21.944 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** Compaction Stats [default] ** 2024-09-21T09:47:21.944 INFO:journalctl@ceph.mon.b.smithi129.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T09:47:21.944 INFO:journalctl@ceph.mon.b.smithi129.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T09:47:21.944 INFO:journalctl@ceph.mon.b.smithi129.stdout: L0 1/0 3.32 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 457.3 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-09-21T09:47:21.945 INFO:journalctl@ceph.mon.b.smithi129.stdout: L6 1/0 7.39 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-21T09:47:21.945 INFO:journalctl@ceph.mon.b.smithi129.stdout: Sum 2/0 10.71 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 457.3 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-09-21T09:47:21.945 INFO:journalctl@ceph.mon.b.smithi129.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 457.3 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-09-21T09:47:21.945 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T09:47:21.945 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** Compaction Stats [default] ** 2024-09-21T09:47:21.945 INFO:journalctl@ceph.mon.b.smithi129.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T09:47:21.945 INFO:journalctl@ceph.mon.b.smithi129.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:47:21.945 INFO:journalctl@ceph.mon.b.smithi129.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 457.3 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-09-21T09:47:21.945 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T09:47:21.945 INFO:journalctl@ceph.mon.b.smithi129.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T09:47:21.945 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T09:47:21.946 INFO:journalctl@ceph.mon.b.smithi129.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:47:21.946 INFO:journalctl@ceph.mon.b.smithi129.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-09-21T09:47:21.946 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:47:21.946 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:47:21.946 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:47:21.946 INFO:journalctl@ceph.mon.b.smithi129.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:47:21.946 INFO:journalctl@ceph.mon.b.smithi129.stdout: Cumulative compaction: 0.00 GB write, 110.17 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:47:21.946 INFO:journalctl@ceph.mon.b.smithi129.stdout: Interval compaction: 0.00 GB write, 110.17 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:47:21.946 INFO:journalctl@ceph.mon.b.smithi129.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:47:21.946 INFO:journalctl@ceph.mon.b.smithi129.stdout: Block cache BinnedLRUCache@0x55782ecb3610#7 capacity: 512.00 MB usage: 7.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.7e-05 secs_since: 0 2024-09-21T09:47:21.946 INFO:journalctl@ceph.mon.b.smithi129.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,5.08 KB,0.000968575%) Misc(1,0.00 KB,0%) 2024-09-21T09:47:21.946 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T09:47:21.946 INFO:journalctl@ceph.mon.b.smithi129.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:47:21.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: starting mon.b rank 1 at public addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] at bind addrs [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:47:21.946 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: mon.b@-1(???) e3 preinit fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:47:21.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: mon.b@-1(???).mds e9 new map 2024-09-21T09:47:21.947 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: mon.b@-1(???).mds e9 print_map 2024-09-21T09:47:21.947 INFO:journalctl@ceph.mon.b.smithi129.stdout: e9 2024-09-21T09:47:21.947 INFO:journalctl@ceph.mon.b.smithi129.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-21T09:47:21.947 INFO:journalctl@ceph.mon.b.smithi129.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T09:47:21.947 INFO:journalctl@ceph.mon.b.smithi129.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:47:21.947 INFO:journalctl@ceph.mon.b.smithi129.stdout: legacy client fscid: 1 2024-09-21T09:47:21.947 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T09:47:21.947 INFO:journalctl@ceph.mon.b.smithi129.stdout: Filesystem 'foo' (1) 2024-09-21T09:47:21.947 INFO:journalctl@ceph.mon.b.smithi129.stdout: fs_name foo 2024-09-21T09:47:21.947 INFO:journalctl@ceph.mon.b.smithi129.stdout: epoch 8 2024-09-21T09:47:21.947 INFO:journalctl@ceph.mon.b.smithi129.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-21T09:47:21.947 INFO:journalctl@ceph.mon.b.smithi129.stdout: created 2024-09-21T09:40:49.090560+0000 2024-09-21T09:47:21.947 INFO:journalctl@ceph.mon.b.smithi129.stdout: modified 2024-09-21T09:41:07.064557+0000 2024-09-21T09:47:21.948 INFO:journalctl@ceph.mon.b.smithi129.stdout: tableserver 0 2024-09-21T09:47:21.948 INFO:journalctl@ceph.mon.b.smithi129.stdout: root 0 2024-09-21T09:47:21.948 INFO:journalctl@ceph.mon.b.smithi129.stdout: session_timeout 60 2024-09-21T09:47:21.948 INFO:journalctl@ceph.mon.b.smithi129.stdout: session_autoclose 300 2024-09-21T09:47:21.948 INFO:journalctl@ceph.mon.b.smithi129.stdout: max_file_size 1099511627776 2024-09-21T09:47:21.948 INFO:journalctl@ceph.mon.b.smithi129.stdout: max_xattr_size 65536 2024-09-21T09:47:21.948 INFO:journalctl@ceph.mon.b.smithi129.stdout: required_client_features {} 2024-09-21T09:47:21.948 INFO:journalctl@ceph.mon.b.smithi129.stdout: last_failure 0 2024-09-21T09:47:21.948 INFO:journalctl@ceph.mon.b.smithi129.stdout: last_failure_osd_epoch 0 2024-09-21T09:47:21.948 INFO:journalctl@ceph.mon.b.smithi129.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:47:21.948 INFO:journalctl@ceph.mon.b.smithi129.stdout: max_mds 1 2024-09-21T09:47:21.948 INFO:journalctl@ceph.mon.b.smithi129.stdout: in 0 2024-09-21T09:47:21.948 INFO:journalctl@ceph.mon.b.smithi129.stdout: up {0=24557} 2024-09-21T09:47:21.948 INFO:journalctl@ceph.mon.b.smithi129.stdout: failed 2024-09-21T09:47:21.949 INFO:journalctl@ceph.mon.b.smithi129.stdout: damaged 2024-09-21T09:47:21.949 INFO:journalctl@ceph.mon.b.smithi129.stdout: stopped 2024-09-21T09:47:21.949 INFO:journalctl@ceph.mon.b.smithi129.stdout: data_pools [4] 2024-09-21T09:47:21.949 INFO:journalctl@ceph.mon.b.smithi129.stdout: metadata_pool 3 2024-09-21T09:47:21.949 INFO:journalctl@ceph.mon.b.smithi129.stdout: inline_data disabled 2024-09-21T09:47:21.949 INFO:journalctl@ceph.mon.b.smithi129.stdout: balancer 2024-09-21T09:47:21.949 INFO:journalctl@ceph.mon.b.smithi129.stdout: bal_rank_mask -1 2024-09-21T09:47:21.949 INFO:journalctl@ceph.mon.b.smithi129.stdout: standby_count_wanted 1 2024-09-21T09:47:21.949 INFO:journalctl@ceph.mon.b.smithi129.stdout: qdb_cluster leader: 0 members: 2024-09-21T09:47:21.949 INFO:journalctl@ceph.mon.b.smithi129.stdout: [mds.foo.smithi107.sedied{0:24557} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.107:6834/2570718287,v1:172.21.15.107:6835/2570718287] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T09:47:21.949 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T09:47:21.949 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T09:47:21.949 INFO:journalctl@ceph.mon.b.smithi129.stdout: Standby daemons: 2024-09-21T09:47:21.949 INFO:journalctl@ceph.mon.b.smithi129.stdout: 2024-09-21T09:47:21.950 INFO:journalctl@ceph.mon.b.smithi129.stdout: [mds.foo.smithi129.zdzmqr{-1:24526} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.129:6832/194235284,v1:172.21.15.129:6833/194235284] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T09:47:21.950 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: mon.b@-1(???).osd e77 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-21T09:47:21.950 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: mon.b@-1(???).osd e77 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T09:47:21.950 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: mon.b@-1(???).osd e77 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T09:47:21.950 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: mon.b@-1(???).osd e77 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T09:47:21.950 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: mon.b@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-09-21T09:47:21.950 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 bash[88435]: b01038fb71fdf70d509f346955879d8e3df264048d40a1f71577033c05f29130 2024-09-21T09:47:21.950 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 systemd[1]: Started Ceph mon.b for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:47:21.950 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T09:47:21.950 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T09:47:21.950 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T09:47:21.950 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T09:47:22.686 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 54 16 94773 94757 109.656 75.8125 0.00859343 0.00911485 2024-09-21T09:47:22.751 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T09:47:22.751 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: mon.b calling monitor election 2024-09-21T09:47:22.751 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: mon.c calling monitor election 2024-09-21T09:47:22.751 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: mon.a calling monitor election 2024-09-21T09:47:22.751 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T09:47:22.751 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T09:47:22.751 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:47:22.751 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: monmap epoch 4 2024-09-21T09:47:22.751 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:47:22.751 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: last_changed 2024-09-21T09:47:21.567722+0000 2024-09-21T09:47:22.752 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: created 2024-09-21T09:34:00.747992+0000 2024-09-21T09:47:22.752 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: min_mon_release 19 (squid) 2024-09-21T09:47:22.752 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: election_strategy: 1 2024-09-21T09:47:22.752 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-09-21T09:47:22.752 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: 1: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.b 2024-09-21T09:47:22.752 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: 2: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.c 2024-09-21T09:47:22.752 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:47:22.752 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T09:47:22.752 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: mgrmap e40: y(active, since 55s), standbys: x 2024-09-21T09:47:22.752 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: overall HEALTH_OK 2024-09-21T09:47:22.752 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:22.752 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:22.752 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:22 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:47:22.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T09:47:22.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: mon.b calling monitor election 2024-09-21T09:47:22.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: mon.c calling monitor election 2024-09-21T09:47:22.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: mon.a calling monitor election 2024-09-21T09:47:22.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T09:47:22.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T09:47:22.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:47:22.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: monmap epoch 4 2024-09-21T09:47:22.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:47:22.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: last_changed 2024-09-21T09:47:21.567722+0000 2024-09-21T09:47:22.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: created 2024-09-21T09:34:00.747992+0000 2024-09-21T09:47:22.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: min_mon_release 19 (squid) 2024-09-21T09:47:22.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: election_strategy: 1 2024-09-21T09:47:22.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-09-21T09:47:22.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: 1: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.b 2024-09-21T09:47:22.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: 2: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.c 2024-09-21T09:47:22.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:47:22.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T09:47:22.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: mgrmap e40: y(active, since 55s), standbys: x 2024-09-21T09:47:22.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: overall HEALTH_OK 2024-09-21T09:47:22.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:22.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:22.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:47:22.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T09:47:22.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: mon.b calling monitor election 2024-09-21T09:47:22.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: mon.c calling monitor election 2024-09-21T09:47:22.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: mon.a calling monitor election 2024-09-21T09:47:22.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T09:47:22.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T09:47:22.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:47:22.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: monmap epoch 4 2024-09-21T09:47:22.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T09:47:22.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: last_changed 2024-09-21T09:47:21.567722+0000 2024-09-21T09:47:22.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: created 2024-09-21T09:34:00.747992+0000 2024-09-21T09:47:22.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: min_mon_release 19 (squid) 2024-09-21T09:47:22.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: election_strategy: 1 2024-09-21T09:47:22.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: 0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-09-21T09:47:22.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: 1: [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] mon.b 2024-09-21T09:47:22.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: 2: [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] mon.c 2024-09-21T09:47:22.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: fsmap foo:1 {0=foo.smithi107.sedied=up:active} 1 up:standby 2024-09-21T09:47:22.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T09:47:22.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: mgrmap e40: y(active, since 55s), standbys: x 2024-09-21T09:47:22.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: overall HEALTH_OK 2024-09-21T09:47:22.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:22.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:22.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:22 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:47:23.686 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 55 16 95991 95975 109.047 76.125 0.00922922 0.00916532 2024-09-21T09:47:23.723 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:23 smithi129 ceph-mon[88471]: pgmap v32: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T09:47:23.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:23 smithi107 ceph-mon[102221]: pgmap v32: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T09:47:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:23 smithi107 ceph-mon[105022]: pgmap v32: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T09:47:24.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:24 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:24.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:24.687 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 56 16 97060 97044 108.292 66.8125 0.0225033 0.00922881 2024-09-21T09:47:24.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:24 smithi107 ceph-mon[102221]: pgmap v33: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-21T09:47:24.897 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:24 smithi107 ceph-mon[105022]: pgmap v33: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-21T09:47:24.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:24 smithi129 ceph-mon[88471]: pgmap v33: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-21T09:47:25.686 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 57 16 98145 98129 107.582 67.8125 0.0180276 0.00928957 2024-09-21T09:47:26.038 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:25.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:26.038 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:25.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:26.687 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 58 16 99299 99283 106.971 72.125 0.00561267 0.00934325 2024-09-21T09:47:26.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:47:26 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:47:26.531+0000 7efdb5a17640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-21T09:47:27.023 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:26 smithi129 ceph-mon[88471]: pgmap v34: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T09:47:27.023 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:47:27.023 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:26.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:26 smithi107 ceph-mon[102221]: pgmap v34: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T09:47:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:47:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:26 smithi107 ceph-mon[105022]: pgmap v34: 89 pgs: 89 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T09:47:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:47:27.687 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 59 16 100429 100413 106.355 70.625 0.00750409 0.00939747 2024-09-21T09:47:28.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:28 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:28.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:28 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:28.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:28 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:28.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:28 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:28.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:28 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:28.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:28 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:28.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:28 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:28.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:28 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:28.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:28 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:28.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:28 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:28.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:28 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:28.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:28 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:28.687 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T09:47:28.686080+0000 min lat: 0.00113027 max lat: 0.18541 avg lat: 0.00944909 2024-09-21T09:47:28.687 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T09:47:28.687 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 60 16 101574 101558 105.775 71.5625 0.00641877 0.00944909 2024-09-21T09:47:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:29 smithi107 ceph-mon[102221]: pgmap v35: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T09:47:29.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:29 smithi107 ceph-mon[105022]: pgmap v35: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T09:47:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:29 smithi129 ceph-mon[88471]: pgmap v35: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T09:47:29.687 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 61 16 102732 102716 105.227 72.375 0.0122745 0.00949866 2024-09-21T09:47:30.379 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:30.687 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 62 16 103876 103860 104.683 71.5 0.0096783 0.00954763 2024-09-21T09:47:31.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:31 smithi107 ceph-mon[102221]: pgmap v36: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 62 MiB/s wr, 993 op/s 2024-09-21T09:47:31.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:31 smithi107 ceph-mon[105022]: pgmap v36: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 62 MiB/s wr, 993 op/s 2024-09-21T09:47:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:31 smithi129 ceph-mon[88471]: pgmap v36: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 62 MiB/s wr, 993 op/s 2024-09-21T09:47:31.687 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 63 16 105009 104993 104.145 70.8125 0.0133009 0.00959666 2024-09-21T09:47:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:47:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:47:31] "GET /metrics HTTP/1.1" 200 36329 "" "Prometheus/2.51.0" 2024-09-21T09:47:32.687 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 64 16 106128 106112 103.611 69.9375 0.0151707 0.00964708 2024-09-21T09:47:33.661 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:33 smithi107 ceph-mon[102221]: pgmap v37: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 62 MiB/s wr, 993 op/s 2024-09-21T09:47:33.662 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:33 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:33.662 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:33 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:33.662 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:33 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:33.662 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:33 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:47:33.662 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:33 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:33.662 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:33 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:47:33.662 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:33 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:47:33.662 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:33 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:33.662 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:33 smithi107 ceph-mon[105022]: pgmap v37: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 62 MiB/s wr, 993 op/s 2024-09-21T09:47:33.663 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:33 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:33.663 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:33 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:33.663 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:33 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:33.663 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:33 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:47:33.663 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:33 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:33.663 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:33 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:47:33.663 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:33 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:47:33.663 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:33 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:33.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:33 smithi129 ceph-mon[88471]: pgmap v37: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 62 MiB/s wr, 993 op/s 2024-09-21T09:47:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:33 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:33 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:33 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:33 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:47:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:33 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:33 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:47:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:33 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:47:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:33 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:33.687 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 65 16 107177 107161 103.025 65.5625 0.00751478 0.00970177 2024-09-21T09:47:34.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:34 smithi129 ceph-mon[88471]: Reconfiguring mon.a (monmap changed)... 2024-09-21T09:47:34.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:34 smithi129 ceph-mon[88471]: Reconfiguring daemon mon.a on smithi107 2024-09-21T09:47:34.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:34 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:34.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:34.687 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 66 16 108225 108209 102.457 65.5 0.0192756 0.00975536 2024-09-21T09:47:34.796 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:34 smithi107 ceph-mon[105022]: Reconfiguring mon.a (monmap changed)... 2024-09-21T09:47:34.796 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:34 smithi107 ceph-mon[105022]: Reconfiguring daemon mon.a on smithi107 2024-09-21T09:47:34.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:34 smithi107 ceph-mon[102221]: Reconfiguring mon.a (monmap changed)... 2024-09-21T09:47:34.796 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:34 smithi107 ceph-mon[102221]: Reconfiguring daemon mon.a on smithi107 2024-09-21T09:47:35.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:35 smithi129 ceph-mon[88471]: pgmap v38: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-21T09:47:35.687 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 67 16 109293 109277 101.924 66.75 0.0133912 0.00980599 2024-09-21T09:47:35.802 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:35 smithi107 ceph-mon[102221]: pgmap v38: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-21T09:47:35.803 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:35 smithi107 ceph-mon[105022]: pgmap v38: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-21T09:47:36.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:35 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:35.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:36.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:35 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:35.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:36.688 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 68 16 110298 110282 101.348 62.8125 0.00974506 0.00986209 2024-09-21T09:47:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:36 smithi107 ceph-mon[102221]: pgmap v39: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 59 MiB/s wr, 951 op/s 2024-09-21T09:47:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:36 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:36 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:36 smithi107 ceph-mon[102221]: Reconfiguring mgr.y (monmap changed)... 2024-09-21T09:47:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:36 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T09:47:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:36 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T09:47:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:36 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:36 smithi107 ceph-mon[102221]: Reconfiguring daemon mgr.y on smithi107 2024-09-21T09:47:37.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:36 smithi107 ceph-mon[105022]: pgmap v39: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 59 MiB/s wr, 951 op/s 2024-09-21T09:47:37.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:36 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:37.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:36 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:37.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:36 smithi107 ceph-mon[105022]: Reconfiguring mgr.y (monmap changed)... 2024-09-21T09:47:37.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:36 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T09:47:37.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:36 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T09:47:37.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:36 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:37.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:36 smithi107 ceph-mon[105022]: Reconfiguring daemon mgr.y on smithi107 2024-09-21T09:47:37.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:36 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:36.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:37.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:36 smithi129 ceph-mon[88471]: pgmap v39: 89 pgs: 89 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 59 MiB/s wr, 951 op/s 2024-09-21T09:47:37.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:36 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:37.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:36 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:37.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:36 smithi129 ceph-mon[88471]: Reconfiguring mgr.y (monmap changed)... 2024-09-21T09:47:37.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:36 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T09:47:37.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:36 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T09:47:37.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:36 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:37.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:36 smithi129 ceph-mon[88471]: Reconfiguring daemon mgr.y on smithi107 2024-09-21T09:47:37.687 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 69 16 111219 111203 100.714 57.5625 0.0123737 0.009924 2024-09-21T09:47:38.688 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 70 16 112257 112241 100.202 64.875 0.0153466 0.00997453 2024-09-21T09:47:39.688 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 71 16 113238 113222 99.6541 61.3125 0.0109758 0.0100296 2024-09-21T09:47:39.836 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:39 smithi129 ceph-mon[88471]: pgmap v40: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-21T09:47:39.836 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:39 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:39.836 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:39 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:39.836 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:39 smithi129 ceph-mon[88471]: Reconfiguring mon.c (monmap changed)... 2024-09-21T09:47:39.836 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:39 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:47:39.836 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:39 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:47:39.837 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:39 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:39.837 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:39 smithi129 ceph-mon[88471]: Reconfiguring daemon mon.c on smithi107 2024-09-21T09:47:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:39 smithi107 ceph-mon[102221]: pgmap v40: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-21T09:47:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:39 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:39 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:39 smithi107 ceph-mon[102221]: Reconfiguring mon.c (monmap changed)... 2024-09-21T09:47:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:39 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:47:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:39 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:47:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:39 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:39 smithi107 ceph-mon[102221]: Reconfiguring daemon mon.c on smithi107 2024-09-21T09:47:39.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:39 smithi107 ceph-mon[105022]: pgmap v40: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-21T09:47:39.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:39 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:39.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:39 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:39.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:39 smithi107 ceph-mon[105022]: Reconfiguring mon.c (monmap changed)... 2024-09-21T09:47:39.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:39 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:47:39.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:39 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:47:39.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:39 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:39.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:39 smithi107 ceph-mon[105022]: Reconfiguring daemon mon.c on smithi107 2024-09-21T09:47:40.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:40.690 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 72 16 114214 114198 99.114 61 0.0179737 0.0100841 2024-09-21T09:47:41.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:41 smithi107 ceph-mon[102221]: pgmap v41: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 916 op/s 2024-09-21T09:47:41.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:41.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:41.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:41 smithi107 ceph-mon[102221]: Reconfiguring osd.0 (monmap changed)... 2024-09-21T09:47:41.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T09:47:41.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:41.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:41 smithi107 ceph-mon[102221]: Reconfiguring daemon osd.0 on smithi107 2024-09-21T09:47:41.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:47:41.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:41 smithi107 ceph-mon[105022]: pgmap v41: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 916 op/s 2024-09-21T09:47:41.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:41.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:41.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:41 smithi107 ceph-mon[105022]: Reconfiguring osd.0 (monmap changed)... 2024-09-21T09:47:41.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T09:47:41.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:41.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:41 smithi107 ceph-mon[105022]: Reconfiguring daemon osd.0 on smithi107 2024-09-21T09:47:41.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:47:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:41 smithi129 ceph-mon[88471]: pgmap v41: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 916 op/s 2024-09-21T09:47:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:41 smithi129 ceph-mon[88471]: Reconfiguring osd.0 (monmap changed)... 2024-09-21T09:47:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T09:47:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:41 smithi129 ceph-mon[88471]: Reconfiguring daemon osd.0 on smithi107 2024-09-21T09:47:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:47:41.690 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 73 16 115227 115211 98.6235 63.3125 0.0134576 0.0101346 2024-09-21T09:47:41.841 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:47:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:47:41] "GET /metrics HTTP/1.1" 200 36393 "" "Prometheus/2.51.0" 2024-09-21T09:47:42.690 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 74 16 116198 116182 98.1108 60.6875 0.0141353 0.0101872 2024-09-21T09:47:43.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:42 smithi107 ceph-mon[102221]: pgmap v42: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 916 op/s 2024-09-21T09:47:43.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:43.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:43.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:42 smithi107 ceph-mon[102221]: Reconfiguring osd.1 (monmap changed)... 2024-09-21T09:47:43.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T09:47:43.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:43.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:42 smithi107 ceph-mon[102221]: Reconfiguring daemon osd.1 on smithi107 2024-09-21T09:47:43.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:42 smithi107 ceph-mon[105022]: pgmap v42: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 916 op/s 2024-09-21T09:47:43.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:43.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:43.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:42 smithi107 ceph-mon[105022]: Reconfiguring osd.1 (monmap changed)... 2024-09-21T09:47:43.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T09:47:43.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:43.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:42 smithi107 ceph-mon[105022]: Reconfiguring daemon osd.1 on smithi107 2024-09-21T09:47:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:42 smithi129 ceph-mon[88471]: pgmap v42: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 916 op/s 2024-09-21T09:47:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:42 smithi129 ceph-mon[88471]: Reconfiguring osd.1 (monmap changed)... 2024-09-21T09:47:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T09:47:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:42 smithi129 ceph-mon[88471]: Reconfiguring daemon osd.1 on smithi107 2024-09-21T09:47:43.690 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 75 16 117216 117200 97.651 63.625 0.00878769 0.0102356 2024-09-21T09:47:44.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:44 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:44.150Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:44.690 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 76 16 118216 118200 97.1884 62.5 0.00762202 0.0102839 2024-09-21T09:47:45.253 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:44 smithi107 ceph-mon[102221]: pgmap v43: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T09:47:45.253 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:44 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:45.254 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:44 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:45.254 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:45 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T09:47:45.254 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:45 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:45.254 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:44 smithi107 ceph-mon[105022]: pgmap v43: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T09:47:45.254 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:44 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:45.254 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:44 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:45.254 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:45 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T09:47:45.254 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:45 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:44 smithi129 ceph-mon[88471]: pgmap v43: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T09:47:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:44 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:44 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:44 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T09:47:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:44 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:45.691 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 77 16 119149 119133 96.6835 58.3125 0.00523394 0.010338 2024-09-21T09:47:46.014 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:45 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:46.014 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:45 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:45.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:46.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:46 smithi107 ceph-mon[102221]: Reconfiguring osd.2 (monmap changed)... 2024-09-21T09:47:46.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:46 smithi107 ceph-mon[102221]: Reconfiguring daemon osd.2 on smithi107 2024-09-21T09:47:46.291 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:46 smithi107 ceph-mon[105022]: Reconfiguring osd.2 (monmap changed)... 2024-09-21T09:47:46.291 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:46 smithi107 ceph-mon[105022]: Reconfiguring daemon osd.2 on smithi107 2024-09-21T09:47:46.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:46 smithi129 ceph-mon[88471]: Reconfiguring osd.2 (monmap changed)... 2024-09-21T09:47:46.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:46 smithi129 ceph-mon[88471]: Reconfiguring daemon osd.2 on smithi107 2024-09-21T09:47:46.691 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 78 16 120098 120082 96.2044 59.3125 0.00768704 0.0103891 2024-09-21T09:47:47.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:47 smithi107 ceph-mon[102221]: pgmap v44: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 53 MiB/s wr, 851 op/s 2024-09-21T09:47:47.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:47 smithi107 ceph-mon[105022]: pgmap v44: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 53 MiB/s wr, 851 op/s 2024-09-21T09:47:47.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:47 smithi129 ceph-mon[88471]: pgmap v44: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 53 MiB/s wr, 851 op/s 2024-09-21T09:47:47.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:46 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:46.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:47.691 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 79 16 121050 121034 95.7397 59.5 0.01023 0.01044 2024-09-21T09:47:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:48 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:48 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:48 smithi129 ceph-mon[88471]: Reconfiguring osd.3 (monmap changed)... 2024-09-21T09:47:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:48 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T09:47:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:48 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:48 smithi129 ceph-mon[88471]: Reconfiguring daemon osd.3 on smithi107 2024-09-21T09:47:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:48 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:48 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:48 smithi107 ceph-mon[102221]: Reconfiguring osd.3 (monmap changed)... 2024-09-21T09:47:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:48 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T09:47:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:48 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:48 smithi107 ceph-mon[102221]: Reconfiguring daemon osd.3 on smithi107 2024-09-21T09:47:48.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:48 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:48.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:48 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:48.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:48 smithi107 ceph-mon[105022]: Reconfiguring osd.3 (monmap changed)... 2024-09-21T09:47:48.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:48 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T09:47:48.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:48 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:48.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:48 smithi107 ceph-mon[105022]: Reconfiguring daemon osd.3 on smithi107 2024-09-21T09:47:48.691 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T09:47:48.690085+0000 min lat: 0.00113027 max lat: 0.201372 avg lat: 0.0104856 2024-09-21T09:47:48.691 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T09:47:48.691 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 80 16 122052 122036 95.3258 62.625 0.0157225 0.0104856 2024-09-21T09:47:49.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:49 smithi107 ceph-mon[102221]: pgmap v45: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T09:47:49.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:49 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:49.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:49 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:49.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:49 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi107.sedied", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T09:47:49.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:49 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:49.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:49 smithi107 ceph-mon[105022]: pgmap v45: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T09:47:49.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:49 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:49.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:49 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:49.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:49 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi107.sedied", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T09:47:49.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:49 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:49.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:49 smithi129 ceph-mon[88471]: pgmap v45: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T09:47:49.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:49 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:49.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:49 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:49.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:49 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi107.sedied", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T09:47:49.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:49 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:49.691 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 81 16 122979 122963 94.8641 57.9375 0.0250777 0.0105367 2024-09-21T09:47:50.215 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:50 smithi129 ceph-mon[88471]: Reconfiguring mds.foo.smithi107.sedied (monmap changed)... 2024-09-21T09:47:50.215 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:50 smithi129 ceph-mon[88471]: Reconfiguring daemon mds.foo.smithi107.sedied on smithi107 2024-09-21T09:47:50.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:50 smithi107 ceph-mon[102221]: Reconfiguring mds.foo.smithi107.sedied (monmap changed)... 2024-09-21T09:47:50.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:50 smithi107 ceph-mon[102221]: Reconfiguring daemon mds.foo.smithi107.sedied on smithi107 2024-09-21T09:47:50.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:50 smithi107 ceph-mon[105022]: Reconfiguring mds.foo.smithi107.sedied (monmap changed)... 2024-09-21T09:47:50.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:50 smithi107 ceph-mon[105022]: Reconfiguring daemon mds.foo.smithi107.sedied on smithi107 2024-09-21T09:47:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:50.691 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 82 16 123952 123936 94.4488 60.8125 0.0102725 0.0105829 2024-09-21T09:47:51.439 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:51 smithi129 ceph-mon[88471]: pgmap v46: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 51 MiB/s wr, 815 op/s 2024-09-21T09:47:51.439 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:51 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:51.439 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:51 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:51.439 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:51 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:47:51.439 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:51 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:47:51.439 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:51 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:51.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:51 smithi107 ceph-mon[102221]: pgmap v46: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 51 MiB/s wr, 815 op/s 2024-09-21T09:47:51.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:51 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:51.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:51 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:51.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:51 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:47:51.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:51 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:47:51.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:51 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:51.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:51 smithi107 ceph-mon[105022]: pgmap v46: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 51 MiB/s wr, 815 op/s 2024-09-21T09:47:51.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:51 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:51.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:51 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:51.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:51 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T09:47:51.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:51 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T09:47:51.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:51 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:51.691 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 83 16 124927 124911 94.045 60.9375 0.0101216 0.0106281 2024-09-21T09:47:52.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:47:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:47:51] "GET /metrics HTTP/1.1" 200 36393 "" "Prometheus/2.51.0" 2024-09-21T09:47:52.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:52 smithi107 ceph-mon[102221]: Reconfiguring mon.b (monmap changed)... 2024-09-21T09:47:52.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:52 smithi107 ceph-mon[102221]: Reconfiguring daemon mon.b on smithi129 2024-09-21T09:47:52.531 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:52 smithi107 ceph-mon[105022]: Reconfiguring mon.b (monmap changed)... 2024-09-21T09:47:52.531 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:52 smithi107 ceph-mon[105022]: Reconfiguring daemon mon.b on smithi129 2024-09-21T09:47:52.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:52 smithi129 ceph-mon[88471]: Reconfiguring mon.b (monmap changed)... 2024-09-21T09:47:52.603 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:52 smithi129 ceph-mon[88471]: Reconfiguring daemon mon.b on smithi129 2024-09-21T09:47:52.691 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 84 16 125837 125821 93.6025 56.875 0.0173983 0.0106784 2024-09-21T09:47:53.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:53 smithi129 ceph-mon[88471]: pgmap v47: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 51 MiB/s wr, 815 op/s 2024-09-21T09:47:53.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:53 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:53.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:53 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:53.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:53 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T09:47:53.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:53 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T09:47:53.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:53 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:53 smithi107 ceph-mon[102221]: pgmap v47: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 51 MiB/s wr, 815 op/s 2024-09-21T09:47:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:53 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:53 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:53 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T09:47:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:53 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T09:47:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:53 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:53 smithi107 ceph-mon[105022]: pgmap v47: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 51 MiB/s wr, 815 op/s 2024-09-21T09:47:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:53 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:53 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:53 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T09:47:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:53 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T09:47:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:53 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:53.691 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 85 16 126686 126670 93.1255 53.0625 0.0127505 0.0107333 2024-09-21T09:47:54.283 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:54 smithi129 ceph-mon[88471]: Reconfiguring mgr.x (monmap changed)... 2024-09-21T09:47:54.283 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:54 smithi129 ceph-mon[88471]: Reconfiguring daemon mgr.x on smithi129 2024-09-21T09:47:54.283 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:54 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:54.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:54.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:54 smithi107 ceph-mon[102221]: Reconfiguring mgr.x (monmap changed)... 2024-09-21T09:47:54.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:54 smithi107 ceph-mon[102221]: Reconfiguring daemon mgr.x on smithi129 2024-09-21T09:47:54.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:54 smithi107 ceph-mon[105022]: Reconfiguring mgr.x (monmap changed)... 2024-09-21T09:47:54.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:54 smithi107 ceph-mon[105022]: Reconfiguring daemon mgr.x on smithi129 2024-09-21T09:47:54.691 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 86 16 127611 127595 92.7149 57.8125 0.0242909 0.010781 2024-09-21T09:47:55.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:55 smithi129 ceph-mon[88471]: pgmap v48: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-21T09:47:55.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:55 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:55.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:55 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:55.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:55 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T09:47:55.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:55 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:55.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:55 smithi107 ceph-mon[102221]: pgmap v48: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-21T09:47:55.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:55 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:55.680 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:55 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:55.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:55 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T09:47:55.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:55 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:55.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:55 smithi107 ceph-mon[105022]: pgmap v48: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-21T09:47:55.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:55 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:55.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:55 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:55.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:55 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T09:47:55.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:55 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:55.691 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 87 16 128573 128557 92.3403 60.125 0.012694 0.0108247 2024-09-21T09:47:55.984 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:55 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:55.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:55.984 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:55 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:55.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:56.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:56 smithi107 ceph-mon[105022]: Reconfiguring osd.4 (monmap changed)... 2024-09-21T09:47:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:56 smithi107 ceph-mon[105022]: Reconfiguring daemon osd.4 on smithi129 2024-09-21T09:47:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:47:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:56 smithi107 ceph-mon[102221]: Reconfiguring osd.4 (monmap changed)... 2024-09-21T09:47:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:56 smithi107 ceph-mon[102221]: Reconfiguring daemon osd.4 on smithi129 2024-09-21T09:47:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:47:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:56 smithi129 ceph-mon[88471]: Reconfiguring osd.4 (monmap changed)... 2024-09-21T09:47:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:56 smithi129 ceph-mon[88471]: Reconfiguring daemon osd.4 on smithi129 2024-09-21T09:47:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:47:56.691 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 88 16 129541 129525 91.9784 60.5 0.0113434 0.0108671 2024-09-21T09:47:57.244 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:47:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:47:56.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:47:57.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:57 smithi107 ceph-mon[105022]: pgmap v49: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 50 MiB/s wr, 807 op/s 2024-09-21T09:47:57.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:57 smithi107 ceph-mon[102221]: pgmap v49: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 50 MiB/s wr, 807 op/s 2024-09-21T09:47:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:57 smithi129 ceph-mon[88471]: pgmap v49: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 50 MiB/s wr, 807 op/s 2024-09-21T09:47:57.691 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 89 16 130509 130493 91.6247 60.5 0.0208224 0.0109095 2024-09-21T09:47:58.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:58 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:58.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:58 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:58.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:58 smithi129 ceph-mon[88471]: Reconfiguring osd.5 (monmap changed)... 2024-09-21T09:47:58.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:58 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T09:47:58.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:58 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:58.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:58 smithi129 ceph-mon[88471]: Reconfiguring daemon osd.5 on smithi129 2024-09-21T09:47:58.692 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total time run: 90.01 2024-09-21T09:47:58.692 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total writes made: 131456 2024-09-21T09:47:58.692 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Write size: 65536 2024-09-21T09:47:58.692 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object size: 65536 2024-09-21T09:47:58.692 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Bandwidth (MB/sec): 91.2787 2024-09-21T09:47:58.692 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Bandwidth: 39.3742 2024-09-21T09:47:58.692 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max bandwidth (MB/sec): 236.875 2024-09-21T09:47:58.692 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min bandwidth (MB/sec): 53.0625 2024-09-21T09:47:58.692 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average IOPS: 1460 2024-09-21T09:47:58.692 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev IOPS: 629.988 2024-09-21T09:47:58.692 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max IOPS: 3790 2024-09-21T09:47:58.693 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min IOPS: 849 2024-09-21T09:47:58.693 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average Latency(s): 0.0109513 2024-09-21T09:47:58.693 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Latency(s): 0.00804345 2024-09-21T09:47:58.693 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max latency(s): 0.201372 2024-09-21T09:47:58.693 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min latency(s): 0.00113027 2024-09-21T09:47:58.696 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T09:47:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:58 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:58 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:58 smithi107 ceph-mon[102221]: Reconfiguring osd.5 (monmap changed)... 2024-09-21T09:47:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:58 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T09:47:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:58 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:58 smithi107 ceph-mon[102221]: Reconfiguring daemon osd.5 on smithi129 2024-09-21T09:47:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:58 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:58 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:58 smithi107 ceph-mon[105022]: Reconfiguring osd.5 (monmap changed)... 2024-09-21T09:47:58.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:58 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T09:47:58.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:58 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:58.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:58 smithi107 ceph-mon[105022]: Reconfiguring daemon osd.5 on smithi129 2024-09-21T09:47:59.843 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:59 smithi129 ceph-mon[88471]: pgmap v50: 89 pgs: 89 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-21T09:47:59.843 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:59 smithi129 ceph-mon[88471]: mgrmap e41: y(active, since 92s), standbys: x 2024-09-21T09:47:59.843 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:59 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:59.843 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:59 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:59.843 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:59 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T09:47:59.843 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:47:59 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:59 smithi107 ceph-mon[102221]: pgmap v50: 89 pgs: 89 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-21T09:47:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:59 smithi107 ceph-mon[102221]: mgrmap e41: y(active, since 92s), standbys: x 2024-09-21T09:47:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:59 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:59 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:59 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T09:47:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:47:59 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:47:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:59 smithi107 ceph-mon[105022]: pgmap v50: 89 pgs: 89 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-21T09:47:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:59 smithi107 ceph-mon[105022]: mgrmap e41: y(active, since 92s), standbys: x 2024-09-21T09:47:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:59 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:59 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:47:59.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:59 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T09:47:59.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:47:59 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:00.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:00.809 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:00 smithi129 ceph-mon[88471]: Reconfiguring osd.6 (monmap changed)... 2024-09-21T09:48:00.809 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:00 smithi129 ceph-mon[88471]: Reconfiguring daemon osd.6 on smithi129 2024-09-21T09:48:00.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:00 smithi107 ceph-mon[102221]: Reconfiguring osd.6 (monmap changed)... 2024-09-21T09:48:00.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:00 smithi107 ceph-mon[102221]: Reconfiguring daemon osd.6 on smithi129 2024-09-21T09:48:00.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:00 smithi107 ceph-mon[105022]: Reconfiguring osd.6 (monmap changed)... 2024-09-21T09:48:00.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:00 smithi107 ceph-mon[105022]: Reconfiguring daemon osd.6 on smithi129 2024-09-21T09:48:01.747 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:01 smithi129 ceph-mon[88471]: pgmap v51: 89 pgs: 89 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s 2024-09-21T09:48:01.747 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:01 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:01.747 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:01 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:01.747 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:01 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T09:48:01.747 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:01 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:01.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:01 smithi107 ceph-mon[105022]: pgmap v51: 89 pgs: 89 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s 2024-09-21T09:48:01.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:01 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:01.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:01 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:01.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:01 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T09:48:01.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:01 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:01.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:01 smithi107 ceph-mon[102221]: pgmap v51: 89 pgs: 89 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s 2024-09-21T09:48:01.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:01 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:01.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:01 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:01.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:01 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T09:48:01.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:01 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:02.108 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:48:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:48:01] "GET /metrics HTTP/1.1" 200 36391 "" "Prometheus/2.51.0" 2024-09-21T09:48:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:02 smithi129 ceph-mon[88471]: Reconfiguring osd.7 (monmap changed)... 2024-09-21T09:48:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:02 smithi129 ceph-mon[88471]: Reconfiguring daemon osd.7 on smithi129 2024-09-21T09:48:02.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:02 smithi107 ceph-mon[105022]: Reconfiguring osd.7 (monmap changed)... 2024-09-21T09:48:02.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:02 smithi107 ceph-mon[105022]: Reconfiguring daemon osd.7 on smithi129 2024-09-21T09:48:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:02 smithi107 ceph-mon[102221]: Reconfiguring osd.7 (monmap changed)... 2024-09-21T09:48:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:02 smithi107 ceph-mon[102221]: Reconfiguring daemon osd.7 on smithi129 2024-09-21T09:48:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:03 smithi107 ceph-mon[102221]: pgmap v52: 89 pgs: 89 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s 2024-09-21T09:48:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi129.zdzmqr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T09:48:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:03.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:03 smithi107 ceph-mon[105022]: pgmap v52: 89 pgs: 89 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s 2024-09-21T09:48:03.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:03.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:03.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi129.zdzmqr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T09:48:03.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:03.758 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:03 smithi129 ceph-mon[88471]: pgmap v52: 89 pgs: 89 active+clean; 7.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s 2024-09-21T09:48:03.758 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:03.758 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:03.758 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi129.zdzmqr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T09:48:03.758 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:04.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:04 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:04.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:04 smithi107 ceph-mon[102221]: Reconfiguring mds.foo.smithi129.zdzmqr (monmap changed)... 2024-09-21T09:48:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:04 smithi107 ceph-mon[102221]: Reconfiguring daemon mds.foo.smithi129.zdzmqr on smithi129 2024-09-21T09:48:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:04 smithi107 ceph-mon[105022]: Reconfiguring mds.foo.smithi129.zdzmqr (monmap changed)... 2024-09-21T09:48:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:04 smithi107 ceph-mon[105022]: Reconfiguring daemon mds.foo.smithi129.zdzmqr on smithi129 2024-09-21T09:48:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:04 smithi129 ceph-mon[88471]: Reconfiguring mds.foo.smithi129.zdzmqr (monmap changed)... 2024-09-21T09:48:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:04 smithi129 ceph-mon[88471]: Reconfiguring daemon mds.foo.smithi129.zdzmqr on smithi129 2024-09-21T09:48:05.742 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:05 smithi129 ceph-mon[88471]: pgmap v53: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 58 MiB/s wr, 1.20k op/s 2024-09-21T09:48:05.742 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:05 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:05.742 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:05 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:05.742 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:05 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:48:05.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:05 smithi107 ceph-mon[105022]: pgmap v53: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 58 MiB/s wr, 1.20k op/s 2024-09-21T09:48:05.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:05 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:05.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:05 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:05.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:05 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:48:05.864 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:05 smithi107 ceph-mon[102221]: pgmap v53: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 58 MiB/s wr, 1.20k op/s 2024-09-21T09:48:05.864 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:05 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:05.864 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:05 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:05.865 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:05 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:48:06.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:05 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:06.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:05 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:05.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:07.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:06 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:06.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:07.719 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:07 smithi129 ceph-mon[88471]: pgmap v54: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 33 MiB/s wr, 795 op/s 2024-09-21T09:48:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:07 smithi107 ceph-mon[102221]: pgmap v54: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 33 MiB/s wr, 795 op/s 2024-09-21T09:48:07.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:07 smithi107 ceph-mon[105022]: pgmap v54: 89 pgs: 89 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 85 B/s rd, 33 MiB/s wr, 795 op/s 2024-09-21T09:48:09.784 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:09 smithi107 ceph-mon[102221]: pgmap v55: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 33 MiB/s wr, 1.19k op/s 2024-09-21T09:48:09.784 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:09 smithi107 ceph-mon[105022]: pgmap v55: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 33 MiB/s wr, 1.19k op/s 2024-09-21T09:48:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:09 smithi129 ceph-mon[88471]: pgmap v55: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 33 MiB/s wr, 1.19k op/s 2024-09-21T09:48:10.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:10.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:11.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:11 smithi107 ceph-mon[105022]: pgmap v56: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 9.0 MiB/s wr, 805 op/s 2024-09-21T09:48:11.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:48:11.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:11.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:11.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:11 smithi107 ceph-mon[102221]: pgmap v56: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 9.0 MiB/s wr, 805 op/s 2024-09-21T09:48:11.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:48:11.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:11.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:11 smithi129 ceph-mon[88471]: pgmap v56: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 9.0 MiB/s wr, 805 op/s 2024-09-21T09:48:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:48:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:12.108 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:48:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:48:11] "GET /metrics HTTP/1.1" 200 36394 "" "Prometheus/2.51.0" 2024-09-21T09:48:12.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:12.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:48:12.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:12.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:12.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:48:12.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:12.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:12.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:48:12.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:13.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:13 smithi107 ceph-mon[102221]: pgmap v57: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 9.0 MiB/s wr, 805 op/s 2024-09-21T09:48:13.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:13 smithi107 ceph-mon[105022]: pgmap v57: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 9.0 MiB/s wr, 805 op/s 2024-09-21T09:48:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:13 smithi129 ceph-mon[88471]: pgmap v57: 89 pgs: 89 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 85 B/s rd, 9.0 MiB/s wr, 805 op/s 2024-09-21T09:48:14.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:14 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:15.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:15 smithi107 ceph-mon[102221]: pgmap v58: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 9.0 MiB/s wr, 1.18k op/s 2024-09-21T09:48:15.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:15 smithi107 ceph-mon[105022]: pgmap v58: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 9.0 MiB/s wr, 1.18k op/s 2024-09-21T09:48:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:15 smithi129 ceph-mon[88471]: pgmap v58: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 9.0 MiB/s wr, 1.18k op/s 2024-09-21T09:48:15.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:15.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:15.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:15.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:17.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:16 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:16.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:17.759 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:17 smithi107 ceph-mon[102221]: pgmap v59: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 771 op/s 2024-09-21T09:48:17.760 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:17 smithi107 ceph-mon[105022]: pgmap v59: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 771 op/s 2024-09-21T09:48:17.798 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade resume' 2024-09-21T09:48:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:17 smithi129 ceph-mon[88471]: pgmap v59: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 771 op/s 2024-09-21T09:48:18.223 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:48:19.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:19 smithi107 ceph-mon[102221]: pgmap v60: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-09-21T09:48:19.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:19 smithi107 ceph-mon[105022]: pgmap v60: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-09-21T09:48:19.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:19 smithi129 ceph-mon[88471]: pgmap v60: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s 2024-09-21T09:48:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:20.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:20.754 INFO:teuthology.orchestra.run.smithi107.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:48:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[105022]: pgmap v61: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 781 op/s 2024-09-21T09:48:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:48:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:48:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:21.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:48:21.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:48:21.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:48:21.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:21.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T09:48:21.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-21T09:48:21.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T09:48:21.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-21T09:48:21.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T09:48:21.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-21T09:48:21.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:48:21.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:21.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-21T09:48:21.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[102221]: pgmap v61: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 781 op/s 2024-09-21T09:48:21.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:21.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:48:21.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:21.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:48:21.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:21.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:48:21.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:48:21.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:48:21.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:21.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T09:48:21.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-21T09:48:21.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T09:48:21.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-21T09:48:21.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T09:48:21.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-21T09:48:21.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:48:21.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:21.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-21T09:48:21.862 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:48:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:48:21] "GET /metrics HTTP/1.1" 200 36394 "" "Prometheus/2.51.0" 2024-09-21T09:48:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:21 smithi129 ceph-mon[88471]: pgmap v61: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 781 op/s 2024-09-21T09:48:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:48:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:48:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:48:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:48:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:48:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T09:48:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-21T09:48:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T09:48:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-21T09:48:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T09:48:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-21T09:48:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:48:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-21T09:48:21.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'echo wait for all mons to upgrade' 2024-09-21T09:48:22.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:22 smithi107 ceph-mon[102221]: from='client.44111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:22.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:22 smithi107 ceph-mon[102221]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:48:22.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:22 smithi107 ceph-mon[102221]: Upgrade: Setting container_image for all mon 2024-09-21T09:48:22.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:22 smithi107 ceph-mon[102221]: Upgrade: Setting container_image for all crash 2024-09-21T09:48:22.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:22 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-21T09:48:22.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:22 smithi107 ceph-mon[102221]: Upgrade: osd.0 is safe to restart 2024-09-21T09:48:22.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:22 smithi107 ceph-mon[105022]: from='client.44111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:22.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:22 smithi107 ceph-mon[105022]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:48:22.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:22 smithi107 ceph-mon[105022]: Upgrade: Setting container_image for all mon 2024-09-21T09:48:22.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:22 smithi107 ceph-mon[105022]: Upgrade: Setting container_image for all crash 2024-09-21T09:48:22.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:22 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-21T09:48:22.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:22 smithi107 ceph-mon[105022]: Upgrade: osd.0 is safe to restart 2024-09-21T09:48:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:22 smithi129 ceph-mon[88471]: from='client.44111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:22 smithi129 ceph-mon[88471]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:48:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:22 smithi129 ceph-mon[88471]: Upgrade: Setting container_image for all mon 2024-09-21T09:48:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:22 smithi129 ceph-mon[88471]: Upgrade: Setting container_image for all crash 2024-09-21T09:48:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:22 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-21T09:48:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:22 smithi129 ceph-mon[88471]: Upgrade: osd.0 is safe to restart 2024-09-21T09:48:23.049 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:48:23.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:23 smithi107 ceph-mon[102221]: pgmap v62: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 781 op/s 2024-09-21T09:48:23.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:23 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:23.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:23 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T09:48:23.779 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:23 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:23.780 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:23 smithi107 ceph-mon[105022]: pgmap v62: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 781 op/s 2024-09-21T09:48:23.780 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:23 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:23.780 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:23 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T09:48:23.780 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:23 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:23 smithi129 ceph-mon[88471]: pgmap v62: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 781 op/s 2024-09-21T09:48:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:23 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:23 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T09:48:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:23 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:24.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:24 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:24.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:24.827 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:24 smithi107 ceph-mon[102221]: Upgrade: Updating osd.0 2024-09-21T09:48:24.827 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:24 smithi107 ceph-mon[102221]: Deploying daemon osd.0 on smithi107 2024-09-21T09:48:24.828 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:24 smithi107 ceph-mon[105022]: Upgrade: Updating osd.0 2024-09-21T09:48:24.828 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:24 smithi107 ceph-mon[105022]: Deploying daemon osd.0 on smithi107 2024-09-21T09:48:24.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:24 smithi129 ceph-mon[88471]: Upgrade: Updating osd.0 2024-09-21T09:48:24.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:24 smithi129 ceph-mon[88471]: Deploying daemon osd.0 on smithi107 2024-09-21T09:48:25.835 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:25 smithi107 ceph-mon[102221]: pgmap v63: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-09-21T09:48:25.835 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:25 smithi107 ceph-mon[105022]: pgmap v63: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-09-21T09:48:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:25 smithi129 ceph-mon[88471]: pgmap v63: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s 2024-09-21T09:48:25.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:25.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:25.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:25.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:26.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:48:26.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:48:26.837 INFO:teuthology.orchestra.run.smithi107.stdout:wait for all mons to upgrade 2024-09-21T09:48:26.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:48:27.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:26.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:27.835 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:27 smithi107 ceph-mon[102221]: pgmap v64: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 757 op/s 2024-09-21T09:48:27.835 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:27 smithi107 ceph-mon[105022]: pgmap v64: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 757 op/s 2024-09-21T09:48:27.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:27 smithi129 ceph-mon[88471]: pgmap v64: 89 pgs: 89 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 757 op/s 2024-09-21T09:48:28.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'while ! ceph mon versions | grep $sha1 | grep '"'"': 3'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-21T09:48:29.567 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:48:29.591 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:29 smithi107 ceph-mon[102221]: pgmap v65: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-21T09:48:29.591 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:29 smithi107 ceph-mon[105022]: pgmap v65: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-21T09:48:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:29 smithi129 ceph-mon[88471]: pgmap v65: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s 2024-09-21T09:48:30.168 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:29 smithi107 systemd[1]: Stopping Ceph osd.0 for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:48:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:31.160 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0[45557]: 2024-09-21T09:48:31.047+0000 7ff80cb42640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T09:48:31.160 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0[45557]: 2024-09-21T09:48:31.047+0000 7ff80cb42640 -1 osd.0 77 *** Got signal Terminated *** 2024-09-21T09:48:31.160 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0[45557]: 2024-09-21T09:48:31.047+0000 7ff80cb42640 -1 osd.0 77 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T09:48:31.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:31 smithi107 ceph-mon[102221]: pgmap v66: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 697 op/s 2024-09-21T09:48:31.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:31 smithi107 ceph-mon[102221]: osd.0 marked itself down and dead 2024-09-21T09:48:31.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:31 smithi107 ceph-mon[105022]: pgmap v66: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 697 op/s 2024-09-21T09:48:31.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:31 smithi107 ceph-mon[105022]: osd.0 marked itself down and dead 2024-09-21T09:48:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:31 smithi129 ceph-mon[88471]: pgmap v66: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 697 op/s 2024-09-21T09:48:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:31 smithi129 ceph-mon[88471]: osd.0 marked itself down and dead 2024-09-21T09:48:31.930 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 3 2024-09-21T09:48:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:48:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:48:31] "GET /metrics HTTP/1.1" 200 36397 "" "Prometheus/2.51.0" 2024-09-21T09:48:32.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:32 smithi107 ceph-mon[102221]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T09:48:32.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:32 smithi107 ceph-mon[102221]: osdmap e78: 8 total, 7 up, 8 in 2024-09-21T09:48:32.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:32 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2365435544' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T09:48:32.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:32 smithi107 ceph-mon[105022]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T09:48:32.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:32 smithi107 ceph-mon[105022]: osdmap e78: 8 total, 7 up, 8 in 2024-09-21T09:48:32.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:32 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2365435544' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T09:48:32.610 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:32 smithi107 podman[110976]: 2024-09-21 09:48:32.329349725 +0000 UTC m=+1.870948333 container died 96047e77ae7844e0f6a08a11ba3b7b4395b098104644deaaca7c73e0a9b221fc (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T09:48:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:32 smithi129 ceph-mon[88471]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T09:48:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:32 smithi129 ceph-mon[88471]: osdmap e78: 8 total, 7 up, 8 in 2024-09-21T09:48:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:32 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2365435544' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T09:48:33.109 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:32 smithi107 podman[110976]: 2024-09-21 09:48:32.889479853 +0000 UTC m=+2.431078464 container cleanup 96047e77ae7844e0f6a08a11ba3b7b4395b098104644deaaca7c73e0a9b221fc (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T09:48:33.109 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:32 smithi107 bash[110976]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0 2024-09-21T09:48:33.416 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:33 smithi107 podman[111081]: 2024-09-21 09:48:33.124606074 +0000 UTC m=+0.970242109 container remove 96047e77ae7844e0f6a08a11ba3b7b4395b098104644deaaca7c73e0a9b221fc (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a, ceph=True) 2024-09-21T09:48:33.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch ps' 2024-09-21T09:48:33.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:33 smithi107 ceph-mon[102221]: pgmap v68: 89 pgs: 13 stale+active+clean, 76 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 837 op/s 2024-09-21T09:48:33.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:33 smithi107 ceph-mon[102221]: osdmap e79: 8 total, 7 up, 8 in 2024-09-21T09:48:33.704 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:33 smithi107 ceph-mon[105022]: pgmap v68: 89 pgs: 13 stale+active+clean, 76 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 837 op/s 2024-09-21T09:48:33.705 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:33 smithi107 ceph-mon[105022]: osdmap e79: 8 total, 7 up, 8 in 2024-09-21T09:48:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:33 smithi129 ceph-mon[88471]: pgmap v68: 89 pgs: 13 stale+active+clean, 76 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 837 op/s 2024-09-21T09:48:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:33 smithi129 ceph-mon[88471]: osdmap e79: 8 total, 7 up, 8 in 2024-09-21T09:48:34.116 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:48:34.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:34 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:34.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:34.573 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:34 smithi107 podman[111219]: 2024-09-21 09:48:34.160783491 +0000 UTC m=+0.035754717 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:48:34.573 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:34 smithi107 podman[111219]: 2024-09-21 09:48:34.431000716 +0000 UTC m=+0.305971941 container create 47a7216e1f91a0d5699d84d5d1bb69bec53507165e04f3668afdcabf5f7d8971 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-21T09:48:34.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:34 smithi107 ceph-mon[102221]: pgmap v70: 89 pgs: 30 peering, 59 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-21T09:48:34.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:34 smithi107 ceph-mon[102221]: Health check failed: Reduced data availability: 9 pgs inactive, 26 pgs peering (PG_AVAILABILITY) 2024-09-21T09:48:34.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:34 smithi107 ceph-mon[105022]: pgmap v70: 89 pgs: 30 peering, 59 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-21T09:48:34.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:34 smithi107 ceph-mon[105022]: Health check failed: Reduced data availability: 9 pgs inactive, 26 pgs peering (PG_AVAILABILITY) 2024-09-21T09:48:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:34 smithi129 ceph-mon[88471]: pgmap v70: 89 pgs: 30 peering, 59 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-21T09:48:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:34 smithi129 ceph-mon[88471]: Health check failed: Reduced data availability: 9 pgs inactive, 26 pgs peering (PG_AVAILABILITY) 2024-09-21T09:48:35.579 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:35 smithi107 podman[111219]: 2024-09-21 09:48:35.280774365 +0000 UTC m=+1.155745604 container init 47a7216e1f91a0d5699d84d5d1bb69bec53507165e04f3668afdcabf5f7d8971 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b) 2024-09-21T09:48:35.579 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:35 smithi107 podman[111219]: 2024-09-21 09:48:35.291239786 +0000 UTC m=+1.166211011 container start 47a7216e1f91a0d5699d84d5d1bb69bec53507165e04f3668afdcabf5f7d8971 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=main-62da54b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True) 2024-09-21T09:48:35.579 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:35 smithi107 podman[111219]: 2024-09-21 09:48:35.380737939 +0000 UTC m=+1.255709165 container attach 47a7216e1f91a0d5699d84d5d1bb69bec53507165e04f3668afdcabf5f7d8971 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, ceph=True, CEPH_POINT_RELEASE=) 2024-09-21T09:48:36.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:35 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:35.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:36.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:35 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:35.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:36.341 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:36 smithi107 podman[111219]: 2024-09-21 09:48:36.009733811 +0000 UTC m=+1.884705039 container died 47a7216e1f91a0d5699d84d5d1bb69bec53507165e04f3668afdcabf5f7d8971 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.build-date=20240731, RELEASE=main-62da54b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T09:48:36.912 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:36 smithi107 podman[111313]: 2024-09-21 09:48:36.631505243 +0000 UTC m=+1.046586709 container remove 47a7216e1f91a0d5699d84d5d1bb69bec53507165e04f3668afdcabf5f7d8971 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-21T09:48:36.912 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:36 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.0.service: Deactivated successfully. 2024-09-21T09:48:36.912 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:36 smithi107 systemd[1]: Stopped Ceph osd.0 for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:48:36.912 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:36 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.0.service: Consumed 10min 10.465s CPU time. 2024-09-21T09:48:37.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:36 smithi129 ceph-mon[88471]: pgmap v71: 89 pgs: 30 peering, 59 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 503 op/s 2024-09-21T09:48:37.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:36 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:36.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:37.359 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:37 smithi107 systemd[1]: Starting Ceph osd.0 for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:48:37.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:36 smithi107 ceph-mon[102221]: pgmap v71: 89 pgs: 30 peering, 59 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 503 op/s 2024-09-21T09:48:37.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:36 smithi107 ceph-mon[105022]: pgmap v71: 89 pgs: 30 peering, 59 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 503 op/s 2024-09-21T09:48:38.122 INFO:teuthology.orchestra.run.smithi107.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T09:48:38.122 INFO:teuthology.orchestra.run.smithi107.stdout:alertmanager.a smithi107 *:9093,9094 running (4m) 26s ago 8m 19.1M - 0.27.0 11f11916f8cd 6fa4ebf1f69d 2024-09-21T09:48:38.122 INFO:teuthology.orchestra.run.smithi107.stdout:grafana.a smithi129 *:3000 running (3m) 70s ago 8m 82.2M - 9.4.7 954c08fa6188 72cc8ff672f5 2024-09-21T09:48:38.122 INFO:teuthology.orchestra.run.smithi107.stdout:mds.foo.smithi107.sedied smithi107 running (7m) 26s ago 7m 20.3M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 83d74619fe7a 2024-09-21T09:48:38.122 INFO:teuthology.orchestra.run.smithi107.stdout:mds.foo.smithi129.zdzmqr smithi129 running (7m) 70s ago 7m 15.4M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 2133ce1b60f9 2024-09-21T09:48:38.122 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.x smithi129 *:8443,9283,8765 running (2m) 70s ago 12m 455M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 59e80c9df333 2024-09-21T09:48:38.122 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.y smithi107 *:8443,9283,8765 running (3m) 26s ago 14m 511M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 849409c95398 2024-09-21T09:48:38.122 INFO:teuthology.orchestra.run.smithi107.stdout:mon.a smithi107 running (2m) 26s ago 14m 43.6M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 34ae11d9ef1b 2024-09-21T09:48:38.122 INFO:teuthology.orchestra.run.smithi107.stdout:mon.b smithi129 running (77s) 70s ago 13m 22.6M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 b01038fb71fd 2024-09-21T09:48:38.122 INFO:teuthology.orchestra.run.smithi107.stdout:mon.c smithi107 running (110s) 26s ago 12m 36.7M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 04f7b77f1500 2024-09-21T09:48:38.123 INFO:teuthology.orchestra.run.smithi107.stdout:node-exporter.a smithi107 *:9100 running (4m) 26s ago 8m 11.2M - 1.7.0 72c9c2088986 2444eabb7c7e 2024-09-21T09:48:38.123 INFO:teuthology.orchestra.run.smithi107.stdout:node-exporter.b smithi129 *:9100 running (3m) 70s ago 8m 10.5M - 1.7.0 72c9c2088986 ce5db8669c3d 2024-09-21T09:48:38.123 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 smithi107 running (12m) 26s ago 12m 373M 2479M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 96047e77ae78 2024-09-21T09:48:38.123 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1 smithi107 running (12m) 26s ago 11m 328M 2479M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 508718820c25 2024-09-21T09:48:38.123 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2 smithi107 running (11m) 26s ago 11m 457M 2479M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be ae11dfe6a988 2024-09-21T09:48:38.123 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3 smithi107 running (11m) 26s ago 11m 477M 2479M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be cf6bac271e88 2024-09-21T09:48:38.123 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4 smithi129 running (10m) 70s ago 10m 355M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 7c6ca82e1ac1 2024-09-21T09:48:38.123 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5 smithi129 running (10m) 70s ago 10m 377M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 9ae131a2ef13 2024-09-21T09:48:38.123 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6 smithi129 running (10m) 70s ago 10m 247M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be a5ccfa3e24c2 2024-09-21T09:48:38.123 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7 smithi129 running (9m) 70s ago 9m 316M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be a32c9bd56124 2024-09-21T09:48:38.123 INFO:teuthology.orchestra.run.smithi107.stdout:prometheus.a smithi129 *:9095 running (2m) 70s ago 9m 45.4M - 2.51.0 1d3b7f56885b 24d10324469f 2024-09-21T09:48:38.220 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:37 smithi107 ceph-mon[102221]: Health check failed: Degraded data redundancy: 24271/291924 objects degraded (8.314%), 11 pgs degraded (PG_DEGRADED) 2024-09-21T09:48:38.220 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:37 smithi107 ceph-mon[102221]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 26 pgs peering) 2024-09-21T09:48:38.221 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:37 smithi107 ceph-mon[105022]: Health check failed: Degraded data redundancy: 24271/291924 objects degraded (8.314%), 11 pgs degraded (PG_DEGRADED) 2024-09-21T09:48:38.221 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:37 smithi107 ceph-mon[105022]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 26 pgs peering) 2024-09-21T09:48:38.221 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:38 smithi107 podman[111473]: 2024-09-21 09:48:37.979867622 +0000 UTC m=+0.034633180 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:48:38.221 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:38 smithi107 podman[111473]: 2024-09-21 09:48:38.105020757 +0000 UTC m=+0.159786307 container create 32629837bbedc94b68b903cb69b3091b08e146dbac5fcc29d40665594c5f9e3c (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-21T09:48:38.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:37 smithi129 ceph-mon[88471]: Health check failed: Degraded data redundancy: 24271/291924 objects degraded (8.314%), 11 pgs degraded (PG_DEGRADED) 2024-09-21T09:48:38.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:37 smithi129 ceph-mon[88471]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 26 pgs peering) 2024-09-21T09:48:38.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade pause' 2024-09-21T09:48:39.107 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:38 smithi107 ceph-mon[102221]: pgmap v72: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 24271/291924 objects degraded (8.314%) 2024-09-21T09:48:39.107 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:38 smithi107 ceph-mon[102221]: from='client.44117 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:39.107 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:38 smithi107 podman[111473]: 2024-09-21 09:48:38.997843906 +0000 UTC m=+1.052609462 container init 32629837bbedc94b68b903cb69b3091b08e146dbac5fcc29d40665594c5f9e3c (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=main-62da54b) 2024-09-21T09:48:39.108 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:39 smithi107 podman[111473]: 2024-09-21 09:48:39.01081339 +0000 UTC m=+1.065578938 container start 32629837bbedc94b68b903cb69b3091b08e146dbac5fcc29d40665594c5f9e3c (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, ceph=True) 2024-09-21T09:48:39.108 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:38 smithi107 ceph-mon[105022]: pgmap v72: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 24271/291924 objects degraded (8.314%) 2024-09-21T09:48:39.108 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:38 smithi107 ceph-mon[105022]: from='client.44117 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:39.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:38 smithi129 ceph-mon[88471]: pgmap v72: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 24271/291924 objects degraded (8.314%) 2024-09-21T09:48:39.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:38 smithi129 ceph-mon[88471]: from='client.44117 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:39.303 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:48:39.360 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:39 smithi107 podman[111473]: 2024-09-21 09:48:39.106037276 +0000 UTC m=+1.160802823 container attach 32629837bbedc94b68b903cb69b3091b08e146dbac5fcc29d40665594c5f9e3c (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate, RELEASE=main-62da54b, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-21T09:48:39.361 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate[111500]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T09:48:39.361 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:39 smithi107 bash[111473]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T09:48:39.361 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate[111500]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T09:48:39.361 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:39 smithi107 bash[111473]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T09:48:40.109 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate[111500]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T09:48:40.109 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:39 smithi107 bash[111473]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T09:48:40.109 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate[111500]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-21T09:48:40.109 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:39 smithi107 bash[111473]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-21T09:48:40.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate[111500]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-21T09:48:40.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:39 smithi107 bash[111473]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-21T09:48:40.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate[111500]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-21T09:48:40.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:39 smithi107 bash[111473]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-21T09:48:40.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate[111500]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-21T09:48:40.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:39 smithi107 bash[111473]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-21T09:48:40.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate[111500]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T09:48:40.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:39 smithi107 bash[111473]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T09:48:40.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:39 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate[111500]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-21T09:48:40.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:39 smithi107 bash[111473]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-21T09:48:40.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:40.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:40.843 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:40 smithi107 podman[111473]: 2024-09-21 09:48:40.473576567 +0000 UTC m=+2.528342116 container died 32629837bbedc94b68b903cb69b3091b08e146dbac5fcc29d40665594c5f9e3c (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-21T09:48:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:40 smithi107 ceph-mon[102221]: pgmap v73: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 24271/291924 objects degraded (8.314%) 2024-09-21T09:48:41.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:40 smithi107 ceph-mon[105022]: pgmap v73: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 24271/291924 objects degraded (8.314%) 2024-09-21T09:48:41.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:40 smithi129 ceph-mon[88471]: pgmap v73: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 24271/291924 objects degraded (8.314%) 2024-09-21T09:48:41.609 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:41 smithi107 podman[111737]: 2024-09-21 09:48:41.134026021 +0000 UTC m=+1.172198028 container remove 32629837bbedc94b68b903cb69b3091b08e146dbac5fcc29d40665594c5f9e3c (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-09-21T09:48:42.008 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:48:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:48:41] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-21T09:48:42.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:42.279 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:48:42.280 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:42.280 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:48:42.280 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:42 smithi107 podman[111831]: 2024-09-21 09:48:42.03383799 +0000 UTC m=+0.264158182 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:48:42.280 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:42 smithi107 podman[111831]: 2024-09-21 09:48:42.175547976 +0000 UTC m=+0.405868163 container create 378577ff0aedbcd86ce6c31c663fb74a9731f523a43c0d88baf7211c935b2aff (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-62da54b, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-21T09:48:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:48:42.827 INFO:teuthology.orchestra.run.smithi107.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:48:43.109 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:42 smithi107 podman[111831]: 2024-09-21 09:48:42.68084526 +0000 UTC m=+0.911165448 container init 378577ff0aedbcd86ce6c31c663fb74a9731f523a43c0d88baf7211c935b2aff (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-62da54b, GIT_BRANCH=HEAD) 2024-09-21T09:48:43.109 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:42 smithi107 podman[111831]: 2024-09-21 09:48:42.688088787 +0000 UTC m=+0.918408976 container start 378577ff0aedbcd86ce6c31c663fb74a9731f523a43c0d88baf7211c935b2aff (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731) 2024-09-21T09:48:43.109 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:42 smithi107 bash[111831]: 378577ff0aedbcd86ce6c31c663fb74a9731f523a43c0d88baf7211c935b2aff 2024-09-21T09:48:43.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:42 smithi107 systemd[1]: Started Ceph osd.0 for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:48:43.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:43 smithi129 ceph-mon[88471]: pgmap v74: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 847 op/s; 24271/291924 objects degraded (8.314%) 2024-09-21T09:48:43.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:43 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:43.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:43 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:43.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:43 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:43.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:43 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:48:43.513 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:43 smithi107 ceph-mon[105022]: pgmap v74: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 847 op/s; 24271/291924 objects degraded (8.314%) 2024-09-21T09:48:43.513 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:43 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:43.513 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:43 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:43.513 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:43 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:43.513 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:43 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:48:43.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:43 smithi107 ceph-mon[102221]: pgmap v74: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 847 op/s; 24271/291924 objects degraded (8.314%) 2024-09-21T09:48:43.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:43 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:43.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:43 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:43.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:43 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:43.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:43 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:48:43.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'sleep 60' 2024-09-21T09:48:44.359 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:44 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0[111891]: 2024-09-21T09:48:44.077+0000 7efd20191640 -1 Falling back to public interface 2024-09-21T09:48:44.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:44 smithi107 ceph-mon[105022]: from='client.44120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:44.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:44 smithi107 ceph-mon[105022]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:48:44.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:44 smithi107 ceph-mon[102221]: from='client.44120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:44.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:44 smithi107 ceph-mon[102221]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:48:44.420 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:44 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:44.147Z caller=alerting.go:391 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-09-21T09:48:44.420 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:44 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:44.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:44.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:44 smithi129 ceph-mon[88471]: from='client.44120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:48:44.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:44 smithi129 ceph-mon[88471]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:48:44.672 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:48:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:45 smithi129 ceph-mon[88471]: pgmap v75: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 842 op/s; 22985/276798 objects degraded (8.304%) 2024-09-21T09:48:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:45 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 22985/276798 objects degraded (8.304%), 11 pgs degraded (PG_DEGRADED) 2024-09-21T09:48:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:45 smithi107 ceph-mon[105022]: pgmap v75: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 842 op/s; 22985/276798 objects degraded (8.304%) 2024-09-21T09:48:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:45 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 22985/276798 objects degraded (8.304%), 11 pgs degraded (PG_DEGRADED) 2024-09-21T09:48:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:45 smithi107 ceph-mon[102221]: pgmap v75: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 842 op/s; 22985/276798 objects degraded (8.304%) 2024-09-21T09:48:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:45 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 22985/276798 objects degraded (8.304%), 11 pgs degraded (PG_DEGRADED) 2024-09-21T09:48:46.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:45 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:45.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:46.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:45 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:45.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:47.330 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:47 smithi129 ceph-mon[88471]: pgmap v76: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 790 op/s; 22985/276798 objects degraded (8.304%) 2024-09-21T09:48:47.330 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:46 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:46.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:47.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:47 smithi107 ceph-mon[102221]: pgmap v76: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 790 op/s; 22985/276798 objects degraded (8.304%) 2024-09-21T09:48:47.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:47 smithi107 ceph-mon[105022]: pgmap v76: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 790 op/s; 22985/276798 objects degraded (8.304%) 2024-09-21T09:48:49.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:49 smithi129 ceph-mon[88471]: pgmap v77: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s; 21852/262674 objects degraded (8.319%) 2024-09-21T09:48:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:49 smithi107 ceph-mon[102221]: pgmap v77: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s; 21852/262674 objects degraded (8.319%) 2024-09-21T09:48:49.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:49 smithi107 ceph-mon[105022]: pgmap v77: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.18k op/s; 21852/262674 objects degraded (8.319%) 2024-09-21T09:48:50.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:51.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:50 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:51.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:50 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:51.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:50 smithi107 ceph-mon[102221]: pgmap v78: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 812 op/s; 21852/262674 objects degraded (8.319%) 2024-09-21T09:48:51.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:50 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:51.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:50 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:51.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:50 smithi107 ceph-mon[105022]: pgmap v78: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 812 op/s; 21852/262674 objects degraded (8.319%) 2024-09-21T09:48:51.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:50 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:51.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:50 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:51.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:50 smithi129 ceph-mon[88471]: pgmap v78: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 812 op/s; 21852/262674 objects degraded (8.319%) 2024-09-21T09:48:51.609 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0[111891]: 2024-09-21T09:48:51.190+0000 7efd20191640 -1 osd.0 77 log_to_monitors true 2024-09-21T09:48:52.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:51 smithi107 ceph-mon[102221]: from='osd.0 [v2:172.21.15.107:6802/1236110780,v1:172.21.15.107:6803/1236110780]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T09:48:52.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:51 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:52.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:51 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:52.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:51 smithi107 ceph-mon[105022]: from='osd.0 [v2:172.21.15.107:6802/1236110780,v1:172.21.15.107:6803/1236110780]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T09:48:52.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:51 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:52.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:51 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:52.110 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:48:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:48:51] "GET /metrics HTTP/1.1" 200 36541 "" "Prometheus/2.51.0" 2024-09-21T09:48:52.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:51 smithi129 ceph-mon[88471]: from='osd.0 [v2:172.21.15.107:6802/1236110780,v1:172.21.15.107:6803/1236110780]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T09:48:52.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:51 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:52.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:51 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:52.830 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 09:48:52 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0[111891]: 2024-09-21T09:48:52.378+0000 7efd13f33640 -1 osd.0 77 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:48:53.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:52 smithi107 ceph-mon[102221]: from='osd.0 [v2:172.21.15.107:6802/1236110780,v1:172.21.15.107:6803/1236110780]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T09:48:53.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:52 smithi107 ceph-mon[102221]: osdmap e80: 8 total, 7 up, 8 in 2024-09-21T09:48:53.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:52 smithi107 ceph-mon[102221]: from='osd.0 [v2:172.21.15.107:6802/1236110780,v1:172.21.15.107:6803/1236110780]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T09:48:53.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:52 smithi107 ceph-mon[102221]: pgmap v80: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 974 op/s; 21852/262674 objects degraded (8.319%) 2024-09-21T09:48:53.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:52 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:53.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:52 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:53.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:52 smithi107 ceph-mon[102221]: from='osd.0 [v2:172.21.15.107:6802/1236110780,v1:172.21.15.107:6803/1236110780]' entity='osd.0' 2024-09-21T09:48:53.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:52 smithi107 ceph-mon[105022]: from='osd.0 [v2:172.21.15.107:6802/1236110780,v1:172.21.15.107:6803/1236110780]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T09:48:53.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:52 smithi107 ceph-mon[105022]: osdmap e80: 8 total, 7 up, 8 in 2024-09-21T09:48:53.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:52 smithi107 ceph-mon[105022]: from='osd.0 [v2:172.21.15.107:6802/1236110780,v1:172.21.15.107:6803/1236110780]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T09:48:53.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:52 smithi107 ceph-mon[105022]: pgmap v80: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 974 op/s; 21852/262674 objects degraded (8.319%) 2024-09-21T09:48:53.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:52 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:53.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:52 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:53.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:52 smithi107 ceph-mon[105022]: from='osd.0 [v2:172.21.15.107:6802/1236110780,v1:172.21.15.107:6803/1236110780]' entity='osd.0' 2024-09-21T09:48:53.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:52 smithi129 ceph-mon[88471]: from='osd.0 [v2:172.21.15.107:6802/1236110780,v1:172.21.15.107:6803/1236110780]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T09:48:53.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:52 smithi129 ceph-mon[88471]: osdmap e80: 8 total, 7 up, 8 in 2024-09-21T09:48:53.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:52 smithi129 ceph-mon[88471]: from='osd.0 [v2:172.21.15.107:6802/1236110780,v1:172.21.15.107:6803/1236110780]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T09:48:53.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:52 smithi129 ceph-mon[88471]: pgmap v80: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.3 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 974 op/s; 21852/262674 objects degraded (8.319%) 2024-09-21T09:48:53.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:52 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:53.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:52 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:53.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:52 smithi129 ceph-mon[88471]: from='osd.0 [v2:172.21.15.107:6802/1236110780,v1:172.21.15.107:6803/1236110780]' entity='osd.0' 2024-09-21T09:48:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:53 smithi107 ceph-mon[102221]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T09:48:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:53 smithi107 ceph-mon[102221]: osd.0 [v2:172.21.15.107:6802/1236110780,v1:172.21.15.107:6803/1236110780] boot 2024-09-21T09:48:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:53 smithi107 ceph-mon[102221]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T09:48:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:53 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T09:48:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:53 smithi107 ceph-mon[105022]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T09:48:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:53 smithi107 ceph-mon[105022]: osd.0 [v2:172.21.15.107:6802/1236110780,v1:172.21.15.107:6803/1236110780] boot 2024-09-21T09:48:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:53 smithi107 ceph-mon[105022]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T09:48:53.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:53 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T09:48:54.148 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:53 smithi129 ceph-mon[88471]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T09:48:54.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:53 smithi129 ceph-mon[88471]: osd.0 [v2:172.21.15.107:6802/1236110780,v1:172.21.15.107:6803/1236110780] boot 2024-09-21T09:48:54.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:53 smithi129 ceph-mon[88471]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T09:48:54.171 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:53 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T09:48:54.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:54 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:54.146Z caller=alerting.go:391 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-09-21T09:48:54.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:54 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:54.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:54.956 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:54 smithi107 ceph-mon[102221]: pgmap v82: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s; 20724/248832 objects degraded (8.329%) 2024-09-21T09:48:54.956 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:54 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 21852/262674 objects degraded (8.319%), 11 pgs degraded (PG_DEGRADED) 2024-09-21T09:48:54.957 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:54 smithi107 ceph-mon[102221]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T09:48:54.957 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:54 smithi107 ceph-mon[105022]: pgmap v82: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s; 20724/248832 objects degraded (8.329%) 2024-09-21T09:48:54.957 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:54 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 21852/262674 objects degraded (8.319%), 11 pgs degraded (PG_DEGRADED) 2024-09-21T09:48:54.957 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:54 smithi107 ceph-mon[105022]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T09:48:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:54 smithi129 ceph-mon[88471]: pgmap v82: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.16k op/s; 20724/248832 objects degraded (8.329%) 2024-09-21T09:48:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:54 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 21852/262674 objects degraded (8.319%), 11 pgs degraded (PG_DEGRADED) 2024-09-21T09:48:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:54 smithi129 ceph-mon[88471]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T09:48:56.121 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:55 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:55.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:56.121 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:55 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:55.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:56.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:56 smithi129 ceph-mon[88471]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T09:48:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:56 smithi107 ceph-mon[102221]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T09:48:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:56 smithi107 ceph-mon[105022]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T09:48:57.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:57 smithi107 ceph-mon[102221]: pgmap v85: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s; 20724/248832 objects degraded (8.329%) 2024-09-21T09:48:57.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:57 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:57.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:57 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:48:57.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:57 smithi107 ceph-mon[105022]: pgmap v85: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s; 20724/248832 objects degraded (8.329%) 2024-09-21T09:48:57.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:57 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:57.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:57 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:48:57.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:57 smithi129 ceph-mon[88471]: pgmap v85: 89 pgs: 19 active+undersized, 11 active+undersized+degraded, 59 active+clean; 5.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s; 20724/248832 objects degraded (8.329%) 2024-09-21T09:48:57.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:57 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:57.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:57 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:48:57.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:48:56 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:48:56.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:48:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:58 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:58 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:58 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:58 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:48:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:58 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:58.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:58 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:48:58.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:58 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:58.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:58 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:48:58.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:58 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:58.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:58 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:58.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:58 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:58.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:58 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:58.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:58 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:48:58.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:58 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:58.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:58 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:48:58.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:58 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:58.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:58 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:48:58.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:58 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:58 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:58 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:58 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:58 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:48:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:58 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:58 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:48:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:58 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:48:58.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:58 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:48:58.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:58 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:48:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:59 smithi107 ceph-mon[102221]: pgmap v86: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s; 3739/240864 objects degraded (1.552%); 0 B/s, 14 objects/s recovering 2024-09-21T09:48:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:48:59 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 3739/240864 objects degraded (1.552%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T09:48:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:59 smithi107 ceph-mon[105022]: pgmap v86: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s; 3739/240864 objects degraded (1.552%); 0 B/s, 14 objects/s recovering 2024-09-21T09:48:59.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:48:59 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 3739/240864 objects degraded (1.552%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T09:48:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:59 smithi129 ceph-mon[88471]: pgmap v86: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s; 3739/240864 objects degraded (1.552%); 0 B/s, 14 objects/s recovering 2024-09-21T09:48:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:48:59 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 3739/240864 objects degraded (1.552%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T09:49:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:01.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:01 smithi107 ceph-mon[102221]: pgmap v87: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 410 op/s; 3739/240864 objects degraded (1.552%); 0 B/s, 13 objects/s recovering 2024-09-21T09:49:01.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:49:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:49:01] "GET /metrics HTTP/1.1" 200 36391 "" "Prometheus/2.51.0" 2024-09-21T09:49:01.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:01 smithi107 ceph-mon[105022]: pgmap v87: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 410 op/s; 3739/240864 objects degraded (1.552%); 0 B/s, 13 objects/s recovering 2024-09-21T09:49:01.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:01 smithi129 ceph-mon[88471]: pgmap v87: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 410 op/s; 3739/240864 objects degraded (1.552%); 0 B/s, 13 objects/s recovering 2024-09-21T09:49:03.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:03 smithi107 ceph-mon[102221]: pgmap v88: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 331 op/s; 3739/240864 objects degraded (1.552%); 0 B/s, 10 objects/s recovering 2024-09-21T09:49:03.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:03 smithi107 ceph-mon[105022]: pgmap v88: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 331 op/s; 3739/240864 objects degraded (1.552%); 0 B/s, 10 objects/s recovering 2024-09-21T09:49:03.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:03 smithi129 ceph-mon[88471]: pgmap v88: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 331 op/s; 3739/240864 objects degraded (1.552%); 0 B/s, 10 objects/s recovering 2024-09-21T09:49:04.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:04 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:04.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.1\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.1\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.1\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:04 smithi129 ceph-mon[88471]: pgmap v89: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 3739/227538 objects degraded (1.643%); 0 B/s, 32 objects/s recovering 2024-09-21T09:49:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:04 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 3739/227538 objects degraded (1.643%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T09:49:05.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:04 smithi107 ceph-mon[102221]: pgmap v89: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 3739/227538 objects degraded (1.643%); 0 B/s, 32 objects/s recovering 2024-09-21T09:49:05.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:04 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 3739/227538 objects degraded (1.643%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T09:49:05.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:04 smithi107 ceph-mon[105022]: pgmap v89: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 3739/227538 objects degraded (1.643%); 0 B/s, 32 objects/s recovering 2024-09-21T09:49:05.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:04 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 3739/227538 objects degraded (1.643%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T09:49:06.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:05 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:06.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:05 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:05.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:07.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:06 smithi129 ceph-mon[88471]: pgmap v90: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 660 op/s; 3739/227538 objects degraded (1.643%); 0 B/s, 29 objects/s recovering 2024-09-21T09:49:07.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:06 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:06.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:07.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:06 smithi107 ceph-mon[102221]: pgmap v90: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 660 op/s; 3739/227538 objects degraded (1.643%); 0 B/s, 29 objects/s recovering 2024-09-21T09:49:07.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:06 smithi107 ceph-mon[105022]: pgmap v90: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 660 op/s; 3739/227538 objects degraded (1.643%); 0 B/s, 29 objects/s recovering 2024-09-21T09:49:09.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:08 smithi107 ceph-mon[102221]: pgmap v91: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 960 op/s; 3739/214239 objects degraded (1.745%); 0 B/s, 47 objects/s recovering 2024-09-21T09:49:09.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:08 smithi107 ceph-mon[105022]: pgmap v91: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 960 op/s; 3739/214239 objects degraded (1.745%); 0 B/s, 47 objects/s recovering 2024-09-21T09:49:09.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:08 smithi129 ceph-mon[88471]: pgmap v91: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 960 op/s; 3739/214239 objects degraded (1.745%); 0 B/s, 47 objects/s recovering 2024-09-21T09:49:10.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:11.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:10 smithi107 ceph-mon[102221]: pgmap v92: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 3739/214239 objects degraded (1.745%); 0 B/s, 40 objects/s recovering 2024-09-21T09:49:11.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:10 smithi107 ceph-mon[105022]: pgmap v92: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 3739/214239 objects degraded (1.745%); 0 B/s, 40 objects/s recovering 2024-09-21T09:49:11.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:10 smithi129 ceph-mon[88471]: pgmap v92: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 3739/214239 objects degraded (1.745%); 0 B/s, 40 objects/s recovering 2024-09-21T09:49:12.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:49:12.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:49:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:49:11] "GET /metrics HTTP/1.1" 200 36538 "" "Prometheus/2.51.0" 2024-09-21T09:49:12.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:49:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:49:13.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:12 smithi107 ceph-mon[102221]: pgmap v93: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 3739/214239 objects degraded (1.745%); 0 B/s, 40 objects/s recovering 2024-09-21T09:49:13.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:12 smithi107 ceph-mon[105022]: pgmap v93: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 3739/214239 objects degraded (1.745%); 0 B/s, 40 objects/s recovering 2024-09-21T09:49:13.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:12 smithi129 ceph-mon[88471]: pgmap v93: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 3739/214239 objects degraded (1.745%); 0 B/s, 40 objects/s recovering 2024-09-21T09:49:14.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:13 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 3739/201129 objects degraded (1.859%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T09:49:14.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:13 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 3739/201129 objects degraded (1.859%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T09:49:14.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:13 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 3739/201129 objects degraded (1.859%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T09:49:14.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:14 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.1\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.1\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.1\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:15.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:14 smithi107 ceph-mon[102221]: pgmap v94: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 3739/201129 objects degraded (1.859%); 0 B/s, 61 objects/s recovering 2024-09-21T09:49:15.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:14 smithi107 ceph-mon[105022]: pgmap v94: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 3739/201129 objects degraded (1.859%); 0 B/s, 61 objects/s recovering 2024-09-21T09:49:15.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:14 smithi129 ceph-mon[88471]: pgmap v94: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 3739/201129 objects degraded (1.859%); 0 B/s, 61 objects/s recovering 2024-09-21T09:49:16.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:15.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:16.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:15.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:17.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:16 smithi107 ceph-mon[102221]: pgmap v95: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 733 op/s; 3739/201129 objects degraded (1.859%); 0 B/s, 42 objects/s recovering 2024-09-21T09:49:17.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:16 smithi107 ceph-mon[105022]: pgmap v95: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 733 op/s; 3739/201129 objects degraded (1.859%); 0 B/s, 42 objects/s recovering 2024-09-21T09:49:17.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:16 smithi129 ceph-mon[88471]: pgmap v95: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 733 op/s; 3739/201129 objects degraded (1.859%); 0 B/s, 42 objects/s recovering 2024-09-21T09:49:17.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:16 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:16.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:19.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:19 smithi107 ceph-mon[102221]: pgmap v96: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 3739/187872 objects degraded (1.990%); 0 B/s, 65 objects/s recovering 2024-09-21T09:49:19.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:19 smithi107 ceph-mon[105022]: pgmap v96: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 3739/187872 objects degraded (1.990%); 0 B/s, 65 objects/s recovering 2024-09-21T09:49:19.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:19 smithi129 ceph-mon[88471]: pgmap v96: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 3739/187872 objects degraded (1.990%); 0 B/s, 65 objects/s recovering 2024-09-21T09:49:20.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:20 smithi107 ceph-mon[102221]: osdmap e84: 8 total, 8 up, 8 in 2024-09-21T09:49:20.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:20 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 3739/187872 objects degraded (1.990%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T09:49:20.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:20 smithi107 ceph-mon[105022]: osdmap e84: 8 total, 8 up, 8 in 2024-09-21T09:49:20.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:20 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 3739/187872 objects degraded (1.990%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T09:49:20.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:20 smithi129 ceph-mon[88471]: osdmap e84: 8 total, 8 up, 8 in 2024-09-21T09:49:20.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:20 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 3739/187872 objects degraded (1.990%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T09:49:20.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:21.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:21 smithi107 ceph-mon[102221]: pgmap v98: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 3739/187872 objects degraded (1.990%); 0 B/s, 54 objects/s recovering 2024-09-21T09:49:21.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:21 smithi107 ceph-mon[102221]: osdmap e85: 8 total, 8 up, 8 in 2024-09-21T09:49:21.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:21 smithi107 ceph-mon[105022]: pgmap v98: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 3739/187872 objects degraded (1.990%); 0 B/s, 54 objects/s recovering 2024-09-21T09:49:21.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:21 smithi107 ceph-mon[105022]: osdmap e85: 8 total, 8 up, 8 in 2024-09-21T09:49:21.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:21 smithi129 ceph-mon[88471]: pgmap v98: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 3739/187872 objects degraded (1.990%); 0 B/s, 54 objects/s recovering 2024-09-21T09:49:21.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:21 smithi129 ceph-mon[88471]: osdmap e85: 8 total, 8 up, 8 in 2024-09-21T09:49:22.108 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:49:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:49:21] "GET /metrics HTTP/1.1" 200 36538 "" "Prometheus/2.51.0" 2024-09-21T09:49:23.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:23 smithi107 ceph-mon[102221]: pgmap v100: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 552 op/s; 3739/187872 objects degraded (1.990%); 0 B/s, 34 objects/s recovering 2024-09-21T09:49:23.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:23 smithi107 ceph-mon[105022]: pgmap v100: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 552 op/s; 3739/187872 objects degraded (1.990%); 0 B/s, 34 objects/s recovering 2024-09-21T09:49:23.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:23 smithi129 ceph-mon[88471]: pgmap v100: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 552 op/s; 3739/187872 objects degraded (1.990%); 0 B/s, 34 objects/s recovering 2024-09-21T09:49:24.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:24 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:24.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.1\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.1\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.1\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:25.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:25 smithi107 ceph-mon[102221]: pgmap v101: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 2468/176775 objects degraded (1.396%); 0 B/s, 93 objects/s recovering 2024-09-21T09:49:25.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:25 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2468/176775 objects degraded (1.396%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T09:49:25.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:25 smithi107 ceph-mon[105022]: pgmap v101: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 2468/176775 objects degraded (1.396%); 0 B/s, 93 objects/s recovering 2024-09-21T09:49:25.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:25 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2468/176775 objects degraded (1.396%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T09:49:25.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:25 smithi129 ceph-mon[88471]: pgmap v101: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 2468/176775 objects degraded (1.396%); 0 B/s, 93 objects/s recovering 2024-09-21T09:49:25.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:25 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2468/176775 objects degraded (1.396%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T09:49:26.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:25.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:26.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:25.742Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:27 smithi107 ceph-mon[102221]: pgmap v102: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 462 op/s; 2468/176775 objects degraded (1.396%); 0 B/s, 58 objects/s recovering 2024-09-21T09:49:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:49:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:27 smithi107 ceph-mon[105022]: pgmap v102: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 462 op/s; 2468/176775 objects degraded (1.396%); 0 B/s, 58 objects/s recovering 2024-09-21T09:49:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:49:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:27 smithi129 ceph-mon[88471]: pgmap v102: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 462 op/s; 2468/176775 objects degraded (1.396%); 0 B/s, 58 objects/s recovering 2024-09-21T09:49:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:49:27.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:26.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi107\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi107\", job=\"node\", machine=\"x86_64\", nodename=\"smithi107\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:29.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:29 smithi107 ceph-mon[102221]: pgmap v103: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 2468/164145 objects degraded (1.504%); 0 B/s, 91 objects/s recovering 2024-09-21T09:49:29.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:29 smithi107 ceph-mon[105022]: pgmap v103: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 2468/164145 objects degraded (1.504%); 0 B/s, 91 objects/s recovering 2024-09-21T09:49:29.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:29 smithi129 ceph-mon[88471]: pgmap v103: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 2468/164145 objects degraded (1.504%); 0 B/s, 91 objects/s recovering 2024-09-21T09:49:30.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:30 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2468/164145 objects degraded (1.504%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T09:49:30.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:30 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2468/164145 objects degraded (1.504%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T09:49:30.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:30 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2468/164145 objects degraded (1.504%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T09:49:30.422 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:31.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:31 smithi107 ceph-mon[102221]: pgmap v104: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 790 op/s; 2468/164145 objects degraded (1.504%); 0 B/s, 80 objects/s recovering 2024-09-21T09:49:31.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:31 smithi107 ceph-mon[105022]: pgmap v104: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 790 op/s; 2468/164145 objects degraded (1.504%); 0 B/s, 80 objects/s recovering 2024-09-21T09:49:31.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:31 smithi129 ceph-mon[88471]: pgmap v104: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 790 op/s; 2468/164145 objects degraded (1.504%); 0 B/s, 80 objects/s recovering 2024-09-21T09:49:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:49:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:49:31] "GET /metrics HTTP/1.1" 200 36533 "" "Prometheus/2.51.0" 2024-09-21T09:49:33.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:33 smithi107 ceph-mon[102221]: pgmap v105: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 2468/164145 objects degraded (1.504%); 0 B/s, 68 objects/s recovering 2024-09-21T09:49:33.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:33 smithi107 ceph-mon[105022]: pgmap v105: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 2468/164145 objects degraded (1.504%); 0 B/s, 68 objects/s recovering 2024-09-21T09:49:33.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:33 smithi129 ceph-mon[88471]: pgmap v105: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 2468/164145 objects degraded (1.504%); 0 B/s, 68 objects/s recovering 2024-09-21T09:49:34.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:34 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:34.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.1\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.1\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.1\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.107\", device_class=\"ssd\", hostname=\"smithi107\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.107\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:35.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:35 smithi107 ceph-mon[102221]: pgmap v106: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2468/151014 objects degraded (1.634%); 0 B/s, 97 objects/s recovering 2024-09-21T09:49:35.362 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:35 smithi107 ceph-mon[102221]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T09:49:35.362 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:35 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2468/151014 objects degraded (1.634%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T09:49:35.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:35 smithi107 ceph-mon[105022]: pgmap v106: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2468/151014 objects degraded (1.634%); 0 B/s, 97 objects/s recovering 2024-09-21T09:49:35.363 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:35 smithi107 ceph-mon[105022]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T09:49:35.363 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:35 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2468/151014 objects degraded (1.634%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T09:49:35.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:35 smithi129 ceph-mon[88471]: pgmap v106: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2468/151014 objects degraded (1.634%); 0 B/s, 97 objects/s recovering 2024-09-21T09:49:35.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:35 smithi129 ceph-mon[88471]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T09:49:35.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:35 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2468/151014 objects degraded (1.634%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T09:49:36.091 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:35 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:35.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:36.092 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:35 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:35.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:36.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:36 smithi107 ceph-mon[102221]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T09:49:36.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:36 smithi107 ceph-mon[105022]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T09:49:36.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:36 smithi129 ceph-mon[88471]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T09:49:37.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:37 smithi107 ceph-mon[102221]: pgmap v109: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 546 op/s; 2468/151014 objects degraded (1.634%); 0 B/s, 45 objects/s recovering 2024-09-21T09:49:37.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:37 smithi107 ceph-mon[105022]: pgmap v109: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 546 op/s; 2468/151014 objects degraded (1.634%); 0 B/s, 45 objects/s recovering 2024-09-21T09:49:37.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:37 smithi129 ceph-mon[88471]: pgmap v109: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 546 op/s; 2468/151014 objects degraded (1.634%); 0 B/s, 45 objects/s recovering 2024-09-21T09:49:39.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:39 smithi107 ceph-mon[102221]: pgmap v110: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 994 op/s; 1232/140277 objects degraded (0.878%); 0 B/s, 104 objects/s recovering 2024-09-21T09:49:39.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:39 smithi107 ceph-mon[105022]: pgmap v110: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 994 op/s; 1232/140277 objects degraded (0.878%); 0 B/s, 104 objects/s recovering 2024-09-21T09:49:39.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:39 smithi129 ceph-mon[88471]: pgmap v110: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 994 op/s; 1232/140277 objects degraded (0.878%); 0 B/s, 104 objects/s recovering 2024-09-21T09:49:40.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:40 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 1232/140277 objects degraded (0.878%), 1 pg degraded (PG_DEGRADED) 2024-09-21T09:49:40.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:40 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 1232/140277 objects degraded (0.878%), 1 pg degraded (PG_DEGRADED) 2024-09-21T09:49:40.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:40 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 1232/140277 objects degraded (0.878%), 1 pg degraded (PG_DEGRADED) 2024-09-21T09:49:41.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:41 smithi129 ceph-mon[88471]: pgmap v111: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 994 op/s; 1232/140277 objects degraded (0.878%); 0 B/s, 104 objects/s recovering 2024-09-21T09:49:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:41 smithi107 ceph-mon[102221]: pgmap v111: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 994 op/s; 1232/140277 objects degraded (0.878%); 0 B/s, 104 objects/s recovering 2024-09-21T09:49:41.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:41 smithi107 ceph-mon[105022]: pgmap v111: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 994 op/s; 1232/140277 objects degraded (0.878%); 0 B/s, 104 objects/s recovering 2024-09-21T09:49:42.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:49:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:49:41] "GET /metrics HTTP/1.1" 200 36535 "" "Prometheus/2.51.0" 2024-09-21T09:49:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:49:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:49:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:49:43.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:43 smithi129 ceph-mon[88471]: pgmap v112: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 447 op/s; 1232/140277 objects degraded (0.878%); 0 B/s, 58 objects/s recovering 2024-09-21T09:49:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:43 smithi107 ceph-mon[102221]: pgmap v112: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 447 op/s; 1232/140277 objects degraded (0.878%); 0 B/s, 58 objects/s recovering 2024-09-21T09:49:43.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:43 smithi107 ceph-mon[105022]: pgmap v112: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 447 op/s; 1232/140277 objects degraded (0.878%); 0 B/s, 58 objects/s recovering 2024-09-21T09:49:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:44 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 1232/127167 objects degraded (0.969%), 1 pg degraded (PG_DEGRADED) 2024-09-21T09:49:44.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:44 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 1232/127167 objects degraded (0.969%), 1 pg degraded (PG_DEGRADED) 2024-09-21T09:49:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:44 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 1232/127167 objects degraded (0.969%), 1 pg degraded (PG_DEGRADED) 2024-09-21T09:49:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:45 smithi107 ceph-mon[102221]: pgmap v113: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 1232/127167 objects degraded (0.969%); 0 B/s, 71 objects/s recovering 2024-09-21T09:49:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:45 smithi107 ceph-mon[105022]: pgmap v113: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 1232/127167 objects degraded (0.969%); 0 B/s, 71 objects/s recovering 2024-09-21T09:49:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:45 smithi129 ceph-mon[88471]: pgmap v113: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 810 op/s; 1232/127167 objects degraded (0.969%); 0 B/s, 71 objects/s recovering 2024-09-21T09:49:47.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:47 smithi107 ceph-mon[102221]: pgmap v114: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 1232/127167 objects degraded (0.969%); 0 B/s, 64 objects/s recovering 2024-09-21T09:49:47.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:47 smithi107 ceph-mon[105022]: pgmap v114: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 1232/127167 objects degraded (0.969%); 0 B/s, 64 objects/s recovering 2024-09-21T09:49:47.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:47 smithi129 ceph-mon[88471]: pgmap v114: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 1232/127167 objects degraded (0.969%); 0 B/s, 64 objects/s recovering 2024-09-21T09:49:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:49 smithi107 ceph-mon[102221]: pgmap v115: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 1232/114078 objects degraded (1.080%); 0 B/s, 77 objects/s recovering 2024-09-21T09:49:49.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:49 smithi107 ceph-mon[105022]: pgmap v115: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 1232/114078 objects degraded (1.080%); 0 B/s, 77 objects/s recovering 2024-09-21T09:49:49.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:49 smithi129 ceph-mon[88471]: pgmap v115: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 1232/114078 objects degraded (1.080%); 0 B/s, 77 objects/s recovering 2024-09-21T09:49:49.749 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade resume' 2024-09-21T09:49:50.218 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:49:50.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:49:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:49:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:49:51.553 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:51 smithi107 ceph-mon[102221]: pgmap v116: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 1232/114078 objects degraded (1.080%); 0 B/s, 38 objects/s recovering 2024-09-21T09:49:51.554 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:51 smithi107 ceph-mon[105022]: pgmap v116: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 1232/114078 objects degraded (1.080%); 0 B/s, 38 objects/s recovering 2024-09-21T09:49:51.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:51 smithi129 ceph-mon[88471]: pgmap v116: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 1232/114078 objects degraded (1.080%); 0 B/s, 38 objects/s recovering 2024-09-21T09:49:51.820 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:49:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:49:51] "GET /metrics HTTP/1.1" 200 36535 "" "Prometheus/2.51.0" 2024-09-21T09:49:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:53 smithi107 ceph-mon[102221]: pgmap v117: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 1232/114078 objects degraded (1.080%); 0 B/s, 38 objects/s recovering 2024-09-21T09:49:53.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:53 smithi107 ceph-mon[105022]: pgmap v117: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 1232/114078 objects degraded (1.080%); 0 B/s, 38 objects/s recovering 2024-09-21T09:49:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:53 smithi129 ceph-mon[88471]: pgmap v117: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 1232/114078 objects degraded (1.080%); 0 B/s, 38 objects/s recovering 2024-09-21T09:49:53.738 INFO:teuthology.orchestra.run.smithi107.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:49:54.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'echo wait for half of osds to upgrade' 2024-09-21T09:49:54.756 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:54 smithi107 ceph-mon[102221]: from='client.54121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:54.756 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:54 smithi107 ceph-mon[102221]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:49:54.756 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:54 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:49:54.756 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:54 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:49:54.756 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:54 smithi107 ceph-mon[102221]: pgmap v118: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 1232/101187 objects degraded (1.218%); 0 B/s, 64 objects/s recovering 2024-09-21T09:49:54.756 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:54 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 1232/114078 objects degraded (1.080%), 1 pg degraded (PG_DEGRADED) 2024-09-21T09:49:55.072 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:54 smithi107 ceph-mon[105022]: from='client.54121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:55.072 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:54 smithi107 ceph-mon[105022]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:49:55.072 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:54 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:49:55.072 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:54 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:49:55.072 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:54 smithi107 ceph-mon[105022]: pgmap v118: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 1232/101187 objects degraded (1.218%); 0 B/s, 64 objects/s recovering 2024-09-21T09:49:55.072 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:54 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 1232/114078 objects degraded (1.080%), 1 pg degraded (PG_DEGRADED) 2024-09-21T09:49:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:54 smithi129 ceph-mon[88471]: from='client.54121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:49:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:54 smithi129 ceph-mon[88471]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:49:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:54 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:49:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:54 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:49:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:54 smithi129 ceph-mon[88471]: pgmap v118: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 1232/101187 objects degraded (1.218%); 0 B/s, 64 objects/s recovering 2024-09-21T09:49:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:54 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 1232/114078 objects degraded (1.080%), 1 pg degraded (PG_DEGRADED) 2024-09-21T09:49:55.647 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:49:56.947 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:56 smithi107 ceph-mon[105022]: osdmap e88: 8 total, 8 up, 8 in 2024-09-21T09:49:56.947 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:56 smithi107 ceph-mon[105022]: pgmap v120: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 1232/101187 objects degraded (1.218%); 0 B/s, 54 objects/s recovering 2024-09-21T09:49:56.948 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:49:56.949 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:56 smithi107 ceph-mon[102221]: osdmap e88: 8 total, 8 up, 8 in 2024-09-21T09:49:56.949 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:56 smithi107 ceph-mon[102221]: pgmap v120: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 1232/101187 objects degraded (1.218%); 0 B/s, 54 objects/s recovering 2024-09-21T09:49:56.949 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:49:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:56 smithi129 ceph-mon[88471]: osdmap e88: 8 total, 8 up, 8 in 2024-09-21T09:49:57.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:56 smithi129 ceph-mon[88471]: pgmap v120: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 87 active+clean; 2.1 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 1232/101187 objects degraded (1.218%); 0 B/s, 54 objects/s recovering 2024-09-21T09:49:57.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:49:57.856 INFO:teuthology.orchestra.run.smithi107.stdout:wait for half of osds to upgrade 2024-09-21T09:49:57.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:57 smithi129 ceph-mon[88471]: osdmap e89: 8 total, 8 up, 8 in 2024-09-21T09:49:58.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:57 smithi107 ceph-mon[102221]: osdmap e89: 8 total, 8 up, 8 in 2024-09-21T09:49:58.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:57 smithi107 ceph-mon[105022]: osdmap e89: 8 total, 8 up, 8 in 2024-09-21T09:49:59.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:58 smithi107 ceph-mon[102221]: pgmap v122: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.9 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 934 op/s; 0 B/s, 113 objects/s recovering 2024-09-21T09:49:59.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:58 smithi107 ceph-mon[105022]: pgmap v122: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.9 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 934 op/s; 0 B/s, 113 objects/s recovering 2024-09-21T09:49:59.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:58 smithi129 ceph-mon[88471]: pgmap v122: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.9 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 934 op/s; 0 B/s, 113 objects/s recovering 2024-09-21T09:49:59.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'while ! ceph osd versions | grep $sha1 | egrep '"'"': [45678]'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-21T09:50:00.074 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:59 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-21T09:50:00.074 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:59 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:00.074 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:49:59 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:00.076 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:59 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-21T09:50:00.076 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:59 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:00.076 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:49:59 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:00.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:59 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-21T09:50:00.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:59 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:00.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:49:59 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:00.633 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:50:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:50:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:50:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:50:01.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:00 smithi107 ceph-mon[102221]: pgmap v123: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.9 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 934 op/s; 0 B/s, 113 objects/s recovering 2024-09-21T09:50:01.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:00 smithi107 ceph-mon[102221]: Health detail: HEALTH_WARN Degraded data redundancy: 1 pg undersized 2024-09-21T09:50:01.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:00 smithi107 ceph-mon[102221]: [WRN] PG_DEGRADED: Degraded data redundancy: 1 pg undersized 2024-09-21T09:50:01.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:00 smithi107 ceph-mon[102221]: pg 6.d is stuck undersized for 62s, current state active+recovering+undersized+remapped, last acting [5,1] 2024-09-21T09:50:01.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:00 smithi107 ceph-mon[105022]: pgmap v123: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.9 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 934 op/s; 0 B/s, 113 objects/s recovering 2024-09-21T09:50:01.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:00 smithi107 ceph-mon[105022]: Health detail: HEALTH_WARN Degraded data redundancy: 1 pg undersized 2024-09-21T09:50:01.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:00 smithi107 ceph-mon[105022]: [WRN] PG_DEGRADED: Degraded data redundancy: 1 pg undersized 2024-09-21T09:50:01.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:00 smithi107 ceph-mon[105022]: pg 6.d is stuck undersized for 62s, current state active+recovering+undersized+remapped, last acting [5,1] 2024-09-21T09:50:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:00 smithi129 ceph-mon[88471]: pgmap v123: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.9 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 934 op/s; 0 B/s, 113 objects/s recovering 2024-09-21T09:50:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:00 smithi129 ceph-mon[88471]: Health detail: HEALTH_WARN Degraded data redundancy: 1 pg undersized 2024-09-21T09:50:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:00 smithi129 ceph-mon[88471]: [WRN] PG_DEGRADED: Degraded data redundancy: 1 pg undersized 2024-09-21T09:50:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:00 smithi129 ceph-mon[88471]: pg 6.d is stuck undersized for 62s, current state active+recovering+undersized+remapped, last acting [5,1] 2024-09-21T09:50:01.818 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:50:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:50:01] "GET /metrics HTTP/1.1" 200 36530 "" "Prometheus/2.51.0" 2024-09-21T09:50:02.984 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:02 smithi107 ceph-mon[102221]: pgmap v124: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.9 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 397 op/s; 0 B/s, 74 objects/s recovering 2024-09-21T09:50:02.984 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:02 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:02.984 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:02 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:02.984 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:02 smithi107 ceph-mon[105022]: pgmap v124: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.9 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 397 op/s; 0 B/s, 74 objects/s recovering 2024-09-21T09:50:02.984 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:02 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:02.984 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:02 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:03.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:02 smithi129 ceph-mon[88471]: pgmap v124: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.9 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 397 op/s; 0 B/s, 74 objects/s recovering 2024-09-21T09:50:03.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:02 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:03.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:02 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:03.588 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:50:03.537+0000 7efdb5a17640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-21T09:50:04.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:50:04.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:50:04.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:04.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:50:04.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:50:04.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:50:04.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:50:04.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T09:50:04.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T09:50:04.281 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-mon[102221]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-21T09:50:04.282 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:50:04.282 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:50:04.282 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:04.282 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:50:04.282 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:50:04.282 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:50:04.282 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:50:04.282 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T09:50:04.282 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T09:50:04.282 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:03 smithi107 ceph-mon[105022]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-21T09:50:04.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:50:04.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:50:04.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:04.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:50:04.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:50:04.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:50:04.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:50:04.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T09:50:04.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:03 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T09:50:04.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:03 smithi129 ceph-mon[88471]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-21T09:50:04.702 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:50:05.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:04 smithi107 ceph-mon[102221]: pgmap v125: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.6 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 904 op/s; 0 B/s, 116 objects/s recovering 2024-09-21T09:50:05.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:04 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1445448880' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:05.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:04 smithi107 ceph-mon[105022]: pgmap v125: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.6 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 904 op/s; 0 B/s, 116 objects/s recovering 2024-09-21T09:50:05.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:04 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1445448880' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:05.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:04 smithi129 ceph-mon[88471]: pgmap v125: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.6 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 904 op/s; 0 B/s, 116 objects/s recovering 2024-09-21T09:50:05.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:04 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1445448880' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:06.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:05 smithi107 ceph-mon[102221]: from='client.54130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:06.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:05 smithi107 ceph-mon[102221]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:06.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:05 smithi107 ceph-mon[105022]: from='client.54130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:06.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:05 smithi107 ceph-mon[105022]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:06.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:05 smithi129 ceph-mon[88471]: from='client.54130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:06.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:05 smithi129 ceph-mon[88471]: from='client.34317 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:07.319 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:06 smithi129 ceph-mon[88471]: pgmap v126: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.6 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 734 op/s; 0 B/s, 94 objects/s recovering 2024-09-21T09:50:07.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:06 smithi107 ceph-mon[102221]: pgmap v126: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.6 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 734 op/s; 0 B/s, 94 objects/s recovering 2024-09-21T09:50:07.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:06 smithi107 ceph-mon[105022]: pgmap v126: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.6 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 734 op/s; 0 B/s, 94 objects/s recovering 2024-09-21T09:50:08.289 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:50:08.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:07 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1074424004' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:08.289 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:07 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1074424004' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:08.334 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:07 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1074424004' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:09.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:08 smithi107 ceph-mon[102221]: pgmap v127: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 0 B/s, 116 objects/s recovering 2024-09-21T09:50:09.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:08 smithi107 ceph-mon[102221]: from='client.54142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:09.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:08 smithi107 ceph-mon[105022]: pgmap v127: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 0 B/s, 116 objects/s recovering 2024-09-21T09:50:09.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:08 smithi107 ceph-mon[105022]: from='client.54142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:09.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:08 smithi129 ceph-mon[88471]: pgmap v127: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 0 B/s, 116 objects/s recovering 2024-09-21T09:50:09.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:08 smithi129 ceph-mon[88471]: from='client.54142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:10.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:09 smithi129 ceph-mon[88471]: from='client.34326 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:10.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:09 smithi129 ceph-mon[88471]: osdmap e90: 8 total, 8 up, 8 in 2024-09-21T09:50:10.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:09 smithi107 ceph-mon[102221]: from='client.34326 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:10.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:09 smithi107 ceph-mon[102221]: osdmap e90: 8 total, 8 up, 8 in 2024-09-21T09:50:10.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:09 smithi107 ceph-mon[105022]: from='client.34326 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:10.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:09 smithi107 ceph-mon[105022]: osdmap e90: 8 total, 8 up, 8 in 2024-09-21T09:50:10.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:50:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:50:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:50:11.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:10 smithi107 ceph-mon[102221]: pgmap v129: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 826 op/s; 0 B/s, 69 objects/s recovering 2024-09-21T09:50:11.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:10 smithi107 ceph-mon[102221]: osdmap e91: 8 total, 8 up, 8 in 2024-09-21T09:50:11.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:10 smithi107 ceph-mon[105022]: pgmap v129: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 826 op/s; 0 B/s, 69 objects/s recovering 2024-09-21T09:50:11.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:10 smithi107 ceph-mon[105022]: osdmap e91: 8 total, 8 up, 8 in 2024-09-21T09:50:11.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:10 smithi129 ceph-mon[88471]: pgmap v129: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 826 op/s; 0 B/s, 69 objects/s recovering 2024-09-21T09:50:11.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:10 smithi129 ceph-mon[88471]: osdmap e91: 8 total, 8 up, 8 in 2024-09-21T09:50:11.891 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:50:12.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:50:12.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:12 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/349490546' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:12.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:50:12.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:12 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/349490546' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:12.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:50:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:50:11] "GET /metrics HTTP/1.1" 200 36521 "" "Prometheus/2.51.0" 2024-09-21T09:50:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:50:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:12 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/349490546' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:13.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:13 smithi107 ceph-mon[102221]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:13.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:13 smithi107 ceph-mon[102221]: pgmap v131: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 513 op/s; 0 B/s, 43 objects/s recovering 2024-09-21T09:50:13.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:13 smithi107 ceph-mon[102221]: from='client.34341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:13.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:13 smithi107 ceph-mon[105022]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:13.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:13 smithi107 ceph-mon[105022]: pgmap v131: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 513 op/s; 0 B/s, 43 objects/s recovering 2024-09-21T09:50:13.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:13 smithi107 ceph-mon[105022]: from='client.34341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:13.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:13 smithi129 ceph-mon[88471]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:13.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:13 smithi129 ceph-mon[88471]: pgmap v131: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.4 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 513 op/s; 0 B/s, 43 objects/s recovering 2024-09-21T09:50:13.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:13 smithi129 ceph-mon[88471]: from='client.34341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:14.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:14 smithi107 ceph-mon[102221]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-21T09:50:14.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:14 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T09:50:14.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:14 smithi107 ceph-mon[105022]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-21T09:50:14.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:14 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T09:50:14.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:14 smithi129 ceph-mon[88471]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-21T09:50:14.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:14 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T09:50:15.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:15 smithi107 ceph-mon[102221]: pgmap v132: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 952 op/s; 0 B/s, 56 objects/s recovering 2024-09-21T09:50:15.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:15 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3481318290' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:15.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:15 smithi107 ceph-mon[105022]: pgmap v132: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 952 op/s; 0 B/s, 56 objects/s recovering 2024-09-21T09:50:15.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:15 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3481318290' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:15.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:15 smithi129 ceph-mon[88471]: pgmap v132: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 952 op/s; 0 B/s, 56 objects/s recovering 2024-09-21T09:50:15.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:15 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3481318290' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:15.496 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:50:16.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:16 smithi107 ceph-mon[102221]: from='client.34353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:16.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:16 smithi107 ceph-mon[105022]: from='client.34353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:16.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:16 smithi129 ceph-mon[88471]: from='client.34353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:17.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:17 smithi107 ceph-mon[102221]: pgmap v133: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 438 op/s; 0 B/s, 12 objects/s recovering 2024-09-21T09:50:17.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:17 smithi107 ceph-mon[102221]: from='client.54166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:17.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:17 smithi107 ceph-mon[105022]: pgmap v133: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 438 op/s; 0 B/s, 12 objects/s recovering 2024-09-21T09:50:17.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:17 smithi107 ceph-mon[105022]: from='client.54166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:17.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:17 smithi129 ceph-mon[88471]: pgmap v133: 89 pgs: 89 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 438 op/s; 0 B/s, 12 objects/s recovering 2024-09-21T09:50:17.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:17 smithi129 ceph-mon[88471]: from='client.54166 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:19.110 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:50:19.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:19 smithi107 ceph-mon[102221]: pgmap v134: 89 pgs: 89 active+clean; 897 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 887 op/s; 0 B/s, 11 objects/s recovering 2024-09-21T09:50:19.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:19 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1600695742' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:19.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:19 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T09:50:19.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:19 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T09:50:19.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:19 smithi107 ceph-mon[102221]: Upgrade: osd.1 is safe to restart 2024-09-21T09:50:19.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:19 smithi107 ceph-mon[105022]: pgmap v134: 89 pgs: 89 active+clean; 897 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 887 op/s; 0 B/s, 11 objects/s recovering 2024-09-21T09:50:19.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:19 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1600695742' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:19.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:19 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T09:50:19.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:19 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T09:50:19.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:19 smithi107 ceph-mon[105022]: Upgrade: osd.1 is safe to restart 2024-09-21T09:50:19.360 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:19 smithi129 ceph-mon[88471]: pgmap v134: 89 pgs: 89 active+clean; 897 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 887 op/s; 0 B/s, 11 objects/s recovering 2024-09-21T09:50:19.360 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:19 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1600695742' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:19.360 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:19 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T09:50:19.360 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:19 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T09:50:19.360 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:19 smithi129 ceph-mon[88471]: Upgrade: osd.1 is safe to restart 2024-09-21T09:50:20.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:20 smithi129 ceph-mon[88471]: from='client.54178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:20.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:20 smithi107 ceph-mon[102221]: from='client.54178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:20.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:20 smithi107 ceph-mon[105022]: from='client.54178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:50:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:50:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:50:21.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:21 smithi107 ceph-mon[102221]: from='client.54181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:21.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:21 smithi107 ceph-mon[102221]: pgmap v135: 89 pgs: 89 active+clean; 897 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 789 op/s; 0 B/s, 10 objects/s recovering 2024-09-21T09:50:21.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:21 smithi107 ceph-mon[105022]: from='client.54181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:21.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:21 smithi107 ceph-mon[105022]: pgmap v135: 89 pgs: 89 active+clean; 897 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 789 op/s; 0 B/s, 10 objects/s recovering 2024-09-21T09:50:21.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:21 smithi129 ceph-mon[88471]: from='client.54181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:21.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:21 smithi129 ceph-mon[88471]: pgmap v135: 89 pgs: 89 active+clean; 897 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 789 op/s; 0 B/s, 10 objects/s recovering 2024-09-21T09:50:22.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:50:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:50:21] "GET /metrics HTTP/1.1" 200 36521 "" "Prometheus/2.51.0" 2024-09-21T09:50:22.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:22 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:22.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:22 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3704115291' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:22.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:22 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:22.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:22 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3704115291' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:22.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:22 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:22.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:22 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3704115291' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:22.729 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:50:23.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:23 smithi107 ceph-mon[102221]: pgmap v136: 89 pgs: 89 active+clean; 897 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 663 op/s; 0 B/s, 8 objects/s recovering 2024-09-21T09:50:23.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:23 smithi107 ceph-mon[102221]: Upgrade: Updating osd.1 2024-09-21T09:50:23.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:23 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:23.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:23 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T09:50:23.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:23 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:50:23.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:23 smithi107 ceph-mon[102221]: Deploying daemon osd.1 on smithi107 2024-09-21T09:50:23.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:23 smithi107 ceph-mon[105022]: pgmap v136: 89 pgs: 89 active+clean; 897 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 663 op/s; 0 B/s, 8 objects/s recovering 2024-09-21T09:50:23.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:23 smithi107 ceph-mon[105022]: Upgrade: Updating osd.1 2024-09-21T09:50:23.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:23 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:23.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:23 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T09:50:23.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:23 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:50:23.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:23 smithi107 ceph-mon[105022]: Deploying daemon osd.1 on smithi107 2024-09-21T09:50:23.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:23 smithi129 ceph-mon[88471]: pgmap v136: 89 pgs: 89 active+clean; 897 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 663 op/s; 0 B/s, 8 objects/s recovering 2024-09-21T09:50:23.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:23 smithi129 ceph-mon[88471]: Upgrade: Updating osd.1 2024-09-21T09:50:23.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:23 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:23.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:23 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T09:50:23.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:23 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:50:23.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:23 smithi129 ceph-mon[88471]: Deploying daemon osd.1 on smithi107 2024-09-21T09:50:24.563 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:24 smithi107 ceph-mon[102221]: from='client.34374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:24.563 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:24 smithi107 ceph-mon[102221]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:24.564 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:24 smithi107 ceph-mon[105022]: from='client.34374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:24.564 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:24 smithi107 ceph-mon[105022]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:24.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:24 smithi129 ceph-mon[88471]: from='client.34374 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:24.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:24 smithi129 ceph-mon[88471]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:25.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:25 smithi107 ceph-mon[102221]: pgmap v137: 89 pgs: 89 active+clean; 635 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 0 B/s, 8 objects/s recovering 2024-09-21T09:50:25.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:25 smithi107 ceph-mon[105022]: pgmap v137: 89 pgs: 89 active+clean; 635 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 0 B/s, 8 objects/s recovering 2024-09-21T09:50:25.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:25 smithi129 ceph-mon[88471]: pgmap v137: 89 pgs: 89 active+clean; 635 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 0 B/s, 8 objects/s recovering 2024-09-21T09:50:26.373 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:50:26.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:26 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1145590747' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:26.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:26.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:50:26.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:26 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1145590747' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:26.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:26.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:50:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:26 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1145590747' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:50:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:27 smithi107 ceph-mon[102221]: pgmap v138: 89 pgs: 89 active+clean; 635 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 715 op/s 2024-09-21T09:50:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:27 smithi107 ceph-mon[102221]: from='client.44210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:27 smithi107 ceph-mon[105022]: pgmap v138: 89 pgs: 89 active+clean; 635 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 715 op/s 2024-09-21T09:50:27.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:27 smithi107 ceph-mon[105022]: from='client.44210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:27.360 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:27 smithi107 systemd[1]: Stopping Ceph osd.1 for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:50:27.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:27 smithi129 ceph-mon[88471]: pgmap v138: 89 pgs: 89 active+clean; 635 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 715 op/s 2024-09-21T09:50:27.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:27 smithi129 ceph-mon[88471]: from='client.44210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:27.859 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:27 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1[51795]: 2024-09-21T09:50:27.555+0000 7fec6e3ba640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T09:50:27.859 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:27 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1[51795]: 2024-09-21T09:50:27.555+0000 7fec6e3ba640 -1 osd.1 91 *** Got signal Terminated *** 2024-09-21T09:50:27.859 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:27 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1[51795]: 2024-09-21T09:50:27.555+0000 7fec6e3ba640 -1 osd.1 91 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T09:50:28.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:28 smithi107 ceph-mon[102221]: from='client.54205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:28.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:28 smithi107 ceph-mon[102221]: osd.1 marked itself down and dead 2024-09-21T09:50:28.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:28 smithi107 ceph-mon[105022]: from='client.54205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:28.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:28 smithi107 ceph-mon[105022]: osd.1 marked itself down and dead 2024-09-21T09:50:28.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:28 smithi129 ceph-mon[88471]: from='client.54205 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:28.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:28 smithi129 ceph-mon[88471]: osd.1 marked itself down and dead 2024-09-21T09:50:29.170 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:28 smithi107 podman[117575]: 2024-09-21 09:50:28.858779431 +0000 UTC m=+1.440607474 container died 508718820c253f133925ce3cbdcac2f7fce4458234fbf73a413d53e6df825779 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-21T09:50:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:29 smithi107 ceph-mon[102221]: pgmap v139: 89 pgs: 89 active+clean; 399 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-21T09:50:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:29 smithi107 ceph-mon[102221]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T09:50:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:29 smithi107 ceph-mon[102221]: osdmap e92: 8 total, 7 up, 8 in 2024-09-21T09:50:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:29 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1353249052' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:29.609 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:29 smithi107 podman[117575]: 2024-09-21 09:50:29.343081888 +0000 UTC m=+1.924909926 container cleanup 508718820c253f133925ce3cbdcac2f7fce4458234fbf73a413d53e6df825779 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T09:50:29.609 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:29 smithi107 bash[117575]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1 2024-09-21T09:50:29.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:29 smithi107 ceph-mon[105022]: pgmap v139: 89 pgs: 89 active+clean; 399 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-21T09:50:29.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:29 smithi107 ceph-mon[105022]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T09:50:29.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:29 smithi107 ceph-mon[105022]: osdmap e92: 8 total, 7 up, 8 in 2024-09-21T09:50:29.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:29 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1353249052' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:29.632 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:50:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:29 smithi129 ceph-mon[88471]: pgmap v139: 89 pgs: 89 active+clean; 399 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-21T09:50:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:29 smithi129 ceph-mon[88471]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T09:50:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:29 smithi129 ceph-mon[88471]: osdmap e92: 8 total, 7 up, 8 in 2024-09-21T09:50:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:29 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1353249052' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:30.109 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:29 smithi107 podman[117598]: 2024-09-21 09:50:29.61210744 +0000 UTC m=+0.747675801 container remove 508718820c253f133925ce3cbdcac2f7fce4458234fbf73a413d53e6df825779 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-21T09:50:30.468 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:30 smithi107 ceph-mon[102221]: osdmap e93: 8 total, 7 up, 8 in 2024-09-21T09:50:30.468 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:30 smithi107 ceph-mon[102221]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:30.468 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:30 smithi107 podman[117744]: 2024-09-21 09:50:30.097676471 +0000 UTC m=+0.032277604 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:50:30.468 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:30 smithi107 podman[117744]: 2024-09-21 09:50:30.213567404 +0000 UTC m=+0.148168533 container create f48963c026b283360732e96891f9b0ef633558b0796cc418b9bacb85e549aa51 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b) 2024-09-21T09:50:30.468 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:30 smithi107 podman[117744]: 2024-09-21 09:50:30.466605714 +0000 UTC m=+0.401206849 container init f48963c026b283360732e96891f9b0ef633558b0796cc418b9bacb85e549aa51 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-62da54b) 2024-09-21T09:50:30.469 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:30 smithi107 ceph-mon[105022]: osdmap e93: 8 total, 7 up, 8 in 2024-09-21T09:50:30.469 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:30 smithi107 ceph-mon[105022]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:30 smithi129 ceph-mon[88471]: osdmap e93: 8 total, 7 up, 8 in 2024-09-21T09:50:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:30 smithi129 ceph-mon[88471]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:50:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:50:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:50:30.778 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:30 smithi107 podman[117744]: 2024-09-21 09:50:30.481639306 +0000 UTC m=+0.416240430 container start f48963c026b283360732e96891f9b0ef633558b0796cc418b9bacb85e549aa51 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b) 2024-09-21T09:50:30.778 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:30 smithi107 podman[117744]: 2024-09-21 09:50:30.574950713 +0000 UTC m=+0.509551847 container attach f48963c026b283360732e96891f9b0ef633558b0796cc418b9bacb85e549aa51 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=main-62da54b, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-21T09:50:30.778 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:30 smithi107 podman[117744]: 2024-09-21 09:50:30.776022209 +0000 UTC m=+0.710623343 container died f48963c026b283360732e96891f9b0ef633558b0796cc418b9bacb85e549aa51 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True) 2024-09-21T09:50:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:31 smithi129 ceph-mon[88471]: pgmap v142: 89 pgs: 11 stale+active+clean, 78 active+clean; 399 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 995 op/s 2024-09-21T09:50:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:31 smithi129 ceph-mon[88471]: from='client.54220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:31.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:31 smithi107 ceph-mon[102221]: pgmap v142: 89 pgs: 11 stale+active+clean, 78 active+clean; 399 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 995 op/s 2024-09-21T09:50:31.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:31 smithi107 ceph-mon[102221]: from='client.54220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:31.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:31 smithi107 ceph-mon[105022]: pgmap v142: 89 pgs: 11 stale+active+clean, 78 active+clean; 399 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 995 op/s 2024-09-21T09:50:31.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:31 smithi107 ceph-mon[105022]: from='client.54220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:32.109 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:31 smithi107 podman[117785]: 2024-09-21 09:50:31.812765468 +0000 UTC m=+1.029312226 container remove f48963c026b283360732e96891f9b0ef633558b0796cc418b9bacb85e549aa51 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T09:50:32.109 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:31 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.1.service: Deactivated successfully. 2024-09-21T09:50:32.109 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:31 smithi107 systemd[1]: Stopped Ceph osd.1 for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:50:32.109 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:31 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.1.service: Consumed 10min 9.663s CPU time. 2024-09-21T09:50:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:50:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:50:31] "GET /metrics HTTP/1.1" 200 36520 "" "Prometheus/2.51.0" 2024-09-21T09:50:32.460 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:32 smithi107 systemd[1]: Starting Ceph osd.1 for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:50:32.806 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:32 smithi107 podman[117892]: 2024-09-21 09:50:32.361600793 +0000 UTC m=+0.036367791 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:50:32.807 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:32 smithi107 podman[117892]: 2024-09-21 09:50:32.51658535 +0000 UTC m=+0.191352338 container create e8940e486cca6d2bf7f80c456cbc45eee166224e8b4391eb348521025a474841 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-activate, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-21T09:50:33.109 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:32 smithi107 podman[117892]: 2024-09-21 09:50:32.805202483 +0000 UTC m=+0.479969473 container init e8940e486cca6d2bf7f80c456cbc45eee166224e8b4391eb348521025a474841 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-activate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-09-21T09:50:33.109 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:32 smithi107 podman[117892]: 2024-09-21 09:50:32.816798764 +0000 UTC m=+0.491565753 container start e8940e486cca6d2bf7f80c456cbc45eee166224e8b4391eb348521025a474841 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-activate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T09:50:33.109 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:32 smithi107 podman[117892]: 2024-09-21 09:50:32.928513764 +0000 UTC m=+0.603280754 container attach e8940e486cca6d2bf7f80c456cbc45eee166224e8b4391eb348521025a474841 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-activate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=main-62da54b, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T09:50:33.109 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-activate[117912]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T09:50:33.110 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:33 smithi107 bash[117892]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T09:50:33.110 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-activate[117912]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T09:50:33.110 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:33 smithi107 bash[117892]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T09:50:33.287 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:50:33.609 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-activate[117912]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T09:50:33.609 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:33 smithi107 bash[117892]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T09:50:33.609 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-activate[117912]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-21T09:50:33.609 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:33 smithi107 bash[117892]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-21T09:50:33.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:33 smithi107 ceph-mon[102221]: pgmap v143: 89 pgs: 11 stale+active+clean, 78 active+clean; 399 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 470 op/s 2024-09-21T09:50:33.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:33 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1532689274' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:33.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:33 smithi107 ceph-mon[105022]: pgmap v143: 89 pgs: 11 stale+active+clean, 78 active+clean; 399 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 470 op/s 2024-09-21T09:50:33.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:33 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1532689274' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:33 smithi129 ceph-mon[88471]: pgmap v143: 89 pgs: 11 stale+active+clean, 78 active+clean; 399 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 470 op/s 2024-09-21T09:50:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:33 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1532689274' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:34.109 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-activate[117912]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-21T09:50:34.109 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:33 smithi107 bash[117892]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-21T09:50:34.109 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-activate[117912]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-21T09:50:34.109 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:33 smithi107 bash[117892]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-21T09:50:34.109 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-activate[117912]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-21T09:50:34.109 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:33 smithi107 bash[117892]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-21T09:50:34.110 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-activate[117912]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T09:50:34.110 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:33 smithi107 bash[117892]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T09:50:34.110 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:33 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-activate[117912]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-21T09:50:34.110 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:33 smithi107 bash[117892]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-21T09:50:34.110 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:33 smithi107 conmon[117912]: conmon e8940e486cca6d2bf7f8 : Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-e8940e486cca6d2bf7f80c456cbc45eee166224e8b4391eb348521025a474841.scope/memory.events 2024-09-21T09:50:34.110 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:33 smithi107 podman[117892]: 2024-09-21 09:50:33.804169164 +0000 UTC m=+1.478936154 container died e8940e486cca6d2bf7f80c456cbc45eee166224e8b4391eb348521025a474841 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-activate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-21T09:50:34.469 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:34 smithi107 ceph-mon[102221]: from='client.54226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:34.469 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:34 smithi107 ceph-mon[105022]: from='client.54226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:34.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:34 smithi129 ceph-mon[88471]: from='client.54226 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:34.784 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:34 smithi107 podman[118111]: 2024-09-21 09:50:34.46719201 +0000 UTC m=+0.647391518 container remove e8940e486cca6d2bf7f80c456cbc45eee166224e8b4391eb348521025a474841 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-62da54b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-09-21T09:50:34.859 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Removed 131456 objects 2024-09-21T09:50:34.859 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Clean up completed and total clean up time :156.163 2024-09-21T09:50:35.048 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:34 smithi107 podman[118169]: 2024-09-21 09:50:34.688345841 +0000 UTC m=+0.038877297 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:50:35.049 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:34 smithi107 podman[118169]: 2024-09-21 09:50:34.818438781 +0000 UTC m=+0.168970223 container create c2b756a1ca12591146d0427a1da635e48ad1342f299b90003a2581d91c4281eb (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1, org.label-schema.license=GPLv2, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T09:50:35.359 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:35 smithi107 podman[118169]: 2024-09-21 09:50:35.046528807 +0000 UTC m=+0.397060249 container init c2b756a1ca12591146d0427a1da635e48ad1342f299b90003a2581d91c4281eb (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b) 2024-09-21T09:50:35.359 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:35 smithi107 podman[118169]: 2024-09-21 09:50:35.051635757 +0000 UTC m=+0.402167200 container start c2b756a1ca12591146d0427a1da635e48ad1342f299b90003a2581d91c4281eb (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-62da54b) 2024-09-21T09:50:35.359 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:35 smithi107 ceph-osd[118207]: -- 172.21.15.107:0/2884087598 <== mon.2 v2:172.21.15.107:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563f60a91a40 con 0x563f60a6fc00 2024-09-21T09:50:35.359 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:35 smithi107 bash[118169]: c2b756a1ca12591146d0427a1da635e48ad1342f299b90003a2581d91c4281eb 2024-09-21T09:50:35.359 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:35 smithi107 systemd[1]: Started Ceph osd.1 for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:50:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:35 smithi129 ceph-mon[88471]: from='client.34419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:35 smithi129 ceph-mon[88471]: pgmap v144: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 163 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 943 op/s; 829/7872 objects degraded (10.531%) 2024-09-21T09:50:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:35 smithi129 ceph-mon[88471]: Health check failed: Degraded data redundancy: 829/7872 objects degraded (10.531%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:50:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:35 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:35 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:35 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:50:35.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:35 smithi107 ceph-mon[102221]: from='client.34419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:35.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:35 smithi107 ceph-mon[102221]: pgmap v144: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 163 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 943 op/s; 829/7872 objects degraded (10.531%) 2024-09-21T09:50:35.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:35 smithi107 ceph-mon[102221]: Health check failed: Degraded data redundancy: 829/7872 objects degraded (10.531%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:50:35.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:35 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:35.846 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:35 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:35.846 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:35 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:50:35.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:35 smithi107 ceph-mon[105022]: from='client.34419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:35.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:35 smithi107 ceph-mon[105022]: pgmap v144: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 163 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 943 op/s; 829/7872 objects degraded (10.531%) 2024-09-21T09:50:35.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:35 smithi107 ceph-mon[105022]: Health check failed: Degraded data redundancy: 829/7872 objects degraded (10.531%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:50:35.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:35 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:35.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:35 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:35.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:35 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:50:36.428 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:50:36.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:36 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2139635161' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:36.788 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:36 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2139635161' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:36.788 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:36 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2139635161' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:37.109 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1[118184]: 2024-09-21T09:50:36.953+0000 7f640295a640 -1 Falling back to public interface 2024-09-21T09:50:37.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:37 smithi129 ceph-mon[88471]: pgmap v145: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 163 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 472 op/s; 829/7872 objects degraded (10.531%) 2024-09-21T09:50:37.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:37 smithi129 ceph-mon[88471]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:37.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:37 smithi107 ceph-mon[102221]: pgmap v145: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 163 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 472 op/s; 829/7872 objects degraded (10.531%) 2024-09-21T09:50:37.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:37 smithi107 ceph-mon[102221]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:37.706 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:37 smithi107 ceph-mon[105022]: pgmap v145: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 163 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 472 op/s; 829/7872 objects degraded (10.531%) 2024-09-21T09:50:37.706 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:37 smithi107 ceph-mon[105022]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:38.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:38 smithi107 ceph-mon[102221]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:38 smithi107 ceph-mon[105022]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:38.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:38 smithi129 ceph-mon[88471]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:38.786 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-21T09:50:38.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-21T09:50:39.062 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:50:39.386 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:39 smithi107 ceph-mon[102221]: pgmap v146: 89 pgs: 31 active+undersized, 6 active+undersized+degraded, 52 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 4/78 objects degraded (5.128%) 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:39 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:39 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:39 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4052270571' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:39 smithi107 ceph-mon[105022]: pgmap v146: 89 pgs: 31 active+undersized, 6 active+undersized+degraded, 52 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 4/78 objects degraded (5.128%) 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:39 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:39 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:39 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4052270571' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:39 smithi129 ceph-mon[88471]: pgmap v146: 89 pgs: 31 active+undersized, 6 active+undersized+degraded, 52 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 4/78 objects degraded (5.128%) 2024-09-21T09:50:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:39 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:39 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:39 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4052270571' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:40.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:40 smithi129 ceph-mon[88471]: from='client.44267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:40.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:40 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:40.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:40 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:40.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:50:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:50:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:50:40.670 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:40 smithi107 ceph-mon[102221]: from='client.44267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:40.670 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:40 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:40.670 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:40 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:40.670 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:40 smithi107 ceph-mon[105022]: from='client.44267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:40.670 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:40 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:40.670 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:40 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:41 smithi107 ceph-mon[102221]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:41 smithi107 ceph-mon[102221]: pgmap v147: 89 pgs: 31 active+undersized, 6 active+undersized+degraded, 52 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 604 op/s; 4/78 objects degraded (5.128%) 2024-09-21T09:50:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:50:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:41 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4260071805' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T09:50:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:41 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T09:50:41.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:41 smithi107 ceph-mon[105022]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:41.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:41 smithi107 ceph-mon[105022]: pgmap v147: 89 pgs: 31 active+undersized, 6 active+undersized+degraded, 52 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 604 op/s; 4/78 objects degraded (5.128%) 2024-09-21T09:50:41.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:41.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:50:41.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:41 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4260071805' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T09:50:41.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:41 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T09:50:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:41 smithi129 ceph-mon[88471]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:41 smithi129 ceph-mon[88471]: pgmap v147: 89 pgs: 31 active+undersized, 6 active+undersized+degraded, 52 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 604 op/s; 4/78 objects degraded (5.128%) 2024-09-21T09:50:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:50:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:41 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4260071805' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T09:50:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:41 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T09:50:42.010 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:50:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:50:41] "GET /metrics HTTP/1.1" 200 36448 "" "Prometheus/2.51.0" 2024-09-21T09:50:42.140 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_1' removed 2024-09-21T09:50:42.332 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:50:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:42 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2885623233' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:42 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T09:50:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:42 smithi107 ceph-mon[102221]: osdmap e94: 8 total, 7 up, 8 in 2024-09-21T09:50:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:42 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2885623233' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:42 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T09:50:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:42 smithi107 ceph-mon[105022]: osdmap e94: 8 total, 7 up, 8 in 2024-09-21T09:50:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:42 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2885623233' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:42 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T09:50:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:42 smithi129 ceph-mon[88471]: osdmap e94: 8 total, 7 up, 8 in 2024-09-21T09:50:42.958 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T09:50:42.958 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T09:50:42.959 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-21T09:50:42.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_2 16 2024-09-21T09:50:42.981 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:42 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1[118184]: 2024-09-21T09:50:42.723+0000 7f640295a640 -1 osd.1 91 log_to_monitors true 2024-09-21T09:50:43.394 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:50:43.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:43 smithi107 ceph-mon[102221]: pgmap v148: 89 pgs: 31 active+undersized, 6 active+undersized+degraded, 52 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 531 op/s; 4/78 objects degraded (5.128%) 2024-09-21T09:50:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:43 smithi107 ceph-mon[102221]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:43 smithi107 ceph-mon[102221]: from='osd.1 [v2:172.21.15.107:6810/2988029448,v1:172.21.15.107:6811/2988029448]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T09:50:43.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:43 smithi107 ceph-mon[105022]: pgmap v148: 89 pgs: 31 active+undersized, 6 active+undersized+degraded, 52 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 531 op/s; 4/78 objects degraded (5.128%) 2024-09-21T09:50:43.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:43 smithi107 ceph-mon[105022]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:43.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:43 smithi107 ceph-mon[105022]: from='osd.1 [v2:172.21.15.107:6810/2988029448,v1:172.21.15.107:6811/2988029448]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T09:50:43.609 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 09:50:43 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1[118184]: 2024-09-21T09:50:43.494+0000 7f63fa704640 -1 osd.1 91 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:50:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:43 smithi129 ceph-mon[88471]: pgmap v148: 89 pgs: 31 active+undersized, 6 active+undersized+degraded, 52 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 531 op/s; 4/78 objects degraded (5.128%) 2024-09-21T09:50:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:43 smithi129 ceph-mon[88471]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:43 smithi129 ceph-mon[88471]: from='osd.1 [v2:172.21.15.107:6810/2988029448,v1:172.21.15.107:6811/2988029448]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T09:50:44.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:44 smithi107 ceph-mon[102221]: from='client.44297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:44.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:44 smithi107 ceph-mon[102221]: from='osd.1 [v2:172.21.15.107:6810/2988029448,v1:172.21.15.107:6811/2988029448]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T09:50:44.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:44 smithi107 ceph-mon[102221]: osdmap e95: 8 total, 7 up, 8 in 2024-09-21T09:50:44.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:44 smithi107 ceph-mon[102221]: from='osd.1 [v2:172.21.15.107:6810/2988029448,v1:172.21.15.107:6811/2988029448]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T09:50:44.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:44 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 4/78 objects degraded (5.128%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:50:44.746 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:44 smithi107 ceph-mon[105022]: from='client.44297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:44.747 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:44 smithi107 ceph-mon[105022]: from='osd.1 [v2:172.21.15.107:6810/2988029448,v1:172.21.15.107:6811/2988029448]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T09:50:44.747 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:44 smithi107 ceph-mon[105022]: osdmap e95: 8 total, 7 up, 8 in 2024-09-21T09:50:44.747 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:44 smithi107 ceph-mon[105022]: from='osd.1 [v2:172.21.15.107:6810/2988029448,v1:172.21.15.107:6811/2988029448]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T09:50:44.747 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:44 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 4/78 objects degraded (5.128%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:50:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:44 smithi129 ceph-mon[88471]: from='client.44297 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:44 smithi129 ceph-mon[88471]: from='osd.1 [v2:172.21.15.107:6810/2988029448,v1:172.21.15.107:6811/2988029448]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T09:50:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:44 smithi129 ceph-mon[88471]: osdmap e95: 8 total, 7 up, 8 in 2024-09-21T09:50:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:44 smithi129 ceph-mon[88471]: from='osd.1 [v2:172.21.15.107:6810/2988029448,v1:172.21.15.107:6811/2988029448]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T09:50:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:44 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 4/78 objects degraded (5.128%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:50:45.310 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:50:45.481 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:50:45.329+0000 7efdb5a17640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-21T09:50:45.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[102221]: pgmap v151: 73 pgs: 28 active+undersized, 4 active+undersized+degraded, 41 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4/78 objects degraded (5.128%) 2024-09-21T09:50:45.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[102221]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T09:50:45.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[102221]: osd.1 [v2:172.21.15.107:6810/2988029448,v1:172.21.15.107:6811/2988029448] boot 2024-09-21T09:50:45.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[102221]: osdmap e96: 8 total, 8 up, 8 in 2024-09-21T09:50:45.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T09:50:45.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3617280892' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:45.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:45.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:45.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:50:45.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:50:45.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:45.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:50:45.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:50:45.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:50:45.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:50:45.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T09:50:45.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[105022]: pgmap v151: 73 pgs: 28 active+undersized, 4 active+undersized+degraded, 41 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4/78 objects degraded (5.128%) 2024-09-21T09:50:45.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[105022]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T09:50:45.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[105022]: osd.1 [v2:172.21.15.107:6810/2988029448,v1:172.21.15.107:6811/2988029448] boot 2024-09-21T09:50:45.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[105022]: osdmap e96: 8 total, 8 up, 8 in 2024-09-21T09:50:45.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T09:50:45.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3617280892' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:45.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:45.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:45.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:50:45.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:50:45.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:45.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:50:45.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:50:45.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:50:45.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:50:45.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:45 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T09:50:45.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:45 smithi129 ceph-mon[88471]: pgmap v151: 73 pgs: 28 active+undersized, 4 active+undersized+degraded, 41 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4/78 objects degraded (5.128%) 2024-09-21T09:50:45.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:45 smithi129 ceph-mon[88471]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T09:50:45.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:45 smithi129 ceph-mon[88471]: osd.1 [v2:172.21.15.107:6810/2988029448,v1:172.21.15.107:6811/2988029448] boot 2024-09-21T09:50:45.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:45 smithi129 ceph-mon[88471]: osdmap e96: 8 total, 8 up, 8 in 2024-09-21T09:50:45.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:45 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T09:50:45.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:45 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3617280892' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:45.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:45 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:45.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:45 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:45.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:45 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:50:45.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:45 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:50:45.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:45 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:45.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:45 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:50:45.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:45 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:50:45.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:45 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:50:45.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:45 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:50:45.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:45 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T09:50:46.500 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_2' created 2024-09-21T09:50:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:46 smithi107 ceph-mon[105022]: from='client.44312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:46 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T09:50:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:46 smithi107 ceph-mon[105022]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-21T09:50:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:46 smithi107 ceph-mon[105022]: osdmap e97: 8 total, 8 up, 8 in 2024-09-21T09:50:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:46 smithi107 ceph-mon[105022]: from='client.34503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:46 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/728638684' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T09:50:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:46 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T09:50:46.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:46 smithi107 ceph-mon[102221]: from='client.44312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:46.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:46 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T09:50:46.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:46 smithi107 ceph-mon[102221]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-21T09:50:46.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:46 smithi107 ceph-mon[102221]: osdmap e97: 8 total, 8 up, 8 in 2024-09-21T09:50:46.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:46 smithi107 ceph-mon[102221]: from='client.34503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:46.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:46 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/728638684' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T09:50:46.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:46 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T09:50:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:46 smithi129 ceph-mon[88471]: from='client.44312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:46 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T09:50:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:46 smithi129 ceph-mon[88471]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-21T09:50:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:46 smithi129 ceph-mon[88471]: osdmap e97: 8 total, 8 up, 8 in 2024-09-21T09:50:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:46 smithi129 ceph-mon[88471]: from='client.34503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:46 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/728638684' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T09:50:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:46 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T09:50:47.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:47 smithi107 ceph-mon[102221]: pgmap v154: 73 pgs: 28 active+undersized, 4 active+undersized+degraded, 41 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4/78 objects degraded (5.128%) 2024-09-21T09:50:47.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:47 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T09:50:47.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:47 smithi107 ceph-mon[102221]: osdmap e98: 8 total, 8 up, 8 in 2024-09-21T09:50:47.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:47 smithi107 ceph-mon[105022]: pgmap v154: 73 pgs: 28 active+undersized, 4 active+undersized+degraded, 41 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4/78 objects degraded (5.128%) 2024-09-21T09:50:47.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:47 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T09:50:47.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:47 smithi107 ceph-mon[105022]: osdmap e98: 8 total, 8 up, 8 in 2024-09-21T09:50:47.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:47 smithi129 ceph-mon[88471]: pgmap v154: 73 pgs: 28 active+undersized, 4 active+undersized+degraded, 41 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4/78 objects degraded (5.128%) 2024-09-21T09:50:47.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:47 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T09:50:47.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:47 smithi129 ceph-mon[88471]: osdmap e98: 8 total, 8 up, 8 in 2024-09-21T09:50:47.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-21T09:50:48.200 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:50:48.254 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:50:48.653 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:48 smithi107 ceph-mon[102221]: osdmap e99: 8 total, 8 up, 8 in 2024-09-21T09:50:48.653 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:48 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1067057848' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:48.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:48 smithi107 ceph-mon[105022]: osdmap e99: 8 total, 8 up, 8 in 2024-09-21T09:50:48.653 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:48 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1067057848' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:48.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:48 smithi129 ceph-mon[88471]: osdmap e99: 8 total, 8 up, 8 in 2024-09-21T09:50:48.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:48 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1067057848' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:49.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:49 smithi107 ceph-mon[102221]: pgmap v157: 89 pgs: 12 creating+peering, 1 unknown, 9 active+undersized, 67 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:49.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:49 smithi107 ceph-mon[102221]: from='client.44336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:49.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:49 smithi107 ceph-mon[102221]: from='client.34527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:49.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:49 smithi107 ceph-mon[102221]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:50:49.683 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:49 smithi107 ceph-mon[102221]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/78 objects degraded (5.128%), 4 pgs degraded) 2024-09-21T09:50:49.684 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:49 smithi107 ceph-mon[102221]: osdmap e100: 8 total, 8 up, 8 in 2024-09-21T09:50:49.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:49 smithi107 ceph-mon[105022]: pgmap v157: 89 pgs: 12 creating+peering, 1 unknown, 9 active+undersized, 67 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:49.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:49 smithi107 ceph-mon[105022]: from='client.44336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:49.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:49 smithi107 ceph-mon[105022]: from='client.34527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:49.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:49 smithi107 ceph-mon[105022]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:50:49.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:49 smithi107 ceph-mon[105022]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/78 objects degraded (5.128%), 4 pgs degraded) 2024-09-21T09:50:49.684 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:49 smithi107 ceph-mon[105022]: osdmap e100: 8 total, 8 up, 8 in 2024-09-21T09:50:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:49 smithi129 ceph-mon[88471]: pgmap v157: 89 pgs: 12 creating+peering, 1 unknown, 9 active+undersized, 67 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:49 smithi129 ceph-mon[88471]: from='client.44336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:49 smithi129 ceph-mon[88471]: from='client.34527 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:49 smithi129 ceph-mon[88471]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:50:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:49 smithi129 ceph-mon[88471]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/78 objects degraded (5.128%), 4 pgs degraded) 2024-09-21T09:50:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:49 smithi129 ceph-mon[88471]: osdmap e100: 8 total, 8 up, 8 in 2024-09-21T09:50:50.508 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:50:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:50:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:50:50.514 INFO:teuthology.orchestra.run.smithi107.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-21T09:50:50.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:50 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2740532913' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T09:50:50.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:50 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2740532913' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T09:50:50.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:50 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2740532913' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T09:50:51.073 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:50:51.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:51 smithi107 ceph-mon[102221]: pgmap v159: 89 pgs: 12 creating+peering, 77 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:51.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:51 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2740532913' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:50:51.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:51 smithi107 ceph-mon[102221]: osdmap e101: 8 total, 8 up, 8 in 2024-09-21T09:50:51.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:51 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2790087455' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:51.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:51 smithi107 ceph-mon[105022]: pgmap v159: 89 pgs: 12 creating+peering, 77 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:51.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:51 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2740532913' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:50:51.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:51 smithi107 ceph-mon[105022]: osdmap e101: 8 total, 8 up, 8 in 2024-09-21T09:50:51.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:51 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2790087455' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:51.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:51 smithi129 ceph-mon[88471]: pgmap v159: 89 pgs: 12 creating+peering, 77 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:51.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:51 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2740532913' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:50:51.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:51 smithi129 ceph-mon[88471]: osdmap e101: 8 total, 8 up, 8 in 2024-09-21T09:50:51.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:51 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2790087455' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:52.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:50:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:50:51] "GET /metrics HTTP/1.1" 200 36448 "" "Prometheus/2.51.0" 2024-09-21T09:50:52.224 DEBUG:teuthology.orchestra.run.smithi107:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-21T09:50:52.226 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T09:50:52.226 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T09:50:52.272 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:hints = 1 2024-09-21T09:50:52.272 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T09:50:52.272 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object prefix: benchmark_data_smithi107_123142 2024-09-21T09:50:52.272 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T09:50:52.272 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 0 0 0 0 0 0 - 0 2024-09-21T09:50:52.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:52 smithi107 ceph-mon[102221]: from='client.34545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:52.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:52 smithi107 ceph-mon[102221]: from='client.34551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:52.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:52 smithi107 ceph-mon[105022]: from='client.34545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:52.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:52 smithi107 ceph-mon[105022]: from='client.34551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:52 smithi129 ceph-mon[88471]: from='client.34545 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:52 smithi129 ceph-mon[88471]: from='client.34551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:53.272 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1 16 3446 3430 214.364 214.375 0.0046212 0.004653 2024-09-21T09:50:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:53 smithi107 ceph-mon[102221]: pgmap v161: 89 pgs: 12 creating+peering, 77 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:53 smithi107 ceph-mon[102221]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:50:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:53 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T09:50:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:53 smithi107 ceph-mon[105022]: pgmap v161: 89 pgs: 12 creating+peering, 77 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:53 smithi107 ceph-mon[105022]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:50:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:53 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T09:50:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:53 smithi129 ceph-mon[88471]: pgmap v161: 89 pgs: 12 creating+peering, 77 active+clean; 582 KiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:50:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:53 smithi129 ceph-mon[88471]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:50:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:53 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T09:50:54.272 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2 16 6269 6253 195.392 176.438 0.00526698 0.00510463 2024-09-21T09:50:54.518 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:50:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:54 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1749206555' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:54 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1749206555' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:54 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1749206555' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:55.272 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3 16 9380 9364 195.068 194.438 0.00493711 0.00511795 2024-09-21T09:50:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:55 smithi107 ceph-mon[102221]: pgmap v162: 89 pgs: 89 active+clean; 32 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s wr, 79 op/s 2024-09-21T09:50:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:55 smithi107 ceph-mon[102221]: from='client.34569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:55.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:55 smithi107 ceph-mon[105022]: pgmap v162: 89 pgs: 89 active+clean; 32 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s wr, 79 op/s 2024-09-21T09:50:55.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:55 smithi107 ceph-mon[105022]: from='client.34569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:55.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:55 smithi129 ceph-mon[88471]: pgmap v162: 89 pgs: 89 active+clean; 32 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s wr, 79 op/s 2024-09-21T09:50:55.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:55 smithi129 ceph-mon[88471]: from='client.34569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:56.273 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 4 16 12632 12616 197.11 203.25 0.00520907 0.00506615 2024-09-21T09:50:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:56 smithi107 ceph-mon[105022]: from='client.34575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:50:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:56 smithi107 ceph-mon[102221]: from='client.34575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:50:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:56 smithi129 ceph-mon[88471]: from='client.34575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:50:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:50:57.273 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 5 15 15576 15561 194.497 184.062 0.00361749 0.00513332 2024-09-21T09:50:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:57 smithi107 ceph-mon[102221]: pgmap v163: 89 pgs: 89 active+clean; 32 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.0 MiB/s wr, 63 op/s 2024-09-21T09:50:57.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:57 smithi107 ceph-mon[105022]: pgmap v163: 89 pgs: 89 active+clean; 32 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.0 MiB/s wr, 63 op/s 2024-09-21T09:50:57.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:57 smithi129 ceph-mon[88471]: pgmap v163: 89 pgs: 89 active+clean; 32 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.0 MiB/s wr, 63 op/s 2024-09-21T09:50:58.241 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:50:58.273 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 6 16 18134 18118 188.714 159.812 0.00831525 0.00529249 2024-09-21T09:50:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:58 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/985550501' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:58 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/985550501' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:58 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/985550501' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:50:59.273 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 7 16 20712 20696 184.77 161.125 0.00590092 0.00540593 2024-09-21T09:50:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:59 smithi107 ceph-mon[102221]: pgmap v164: 89 pgs: 89 active+clean; 915 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-09-21T09:50:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:50:59 smithi107 ceph-mon[102221]: from='client.54340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:59 smithi107 ceph-mon[105022]: pgmap v164: 89 pgs: 89 active+clean; 915 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-09-21T09:50:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:50:59 smithi107 ceph-mon[105022]: from='client.54340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:59 smithi129 ceph-mon[88471]: pgmap v164: 89 pgs: 89 active+clean; 915 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 97 MiB/s wr, 1.56k op/s 2024-09-21T09:50:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:50:59 smithi129 ceph-mon[88471]: from='client.54340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:00.273 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 8 16 23398 23382 182.657 167.875 0.00487349 0.00546906 2024-09-21T09:51:00.567 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:51:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:51:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:51:00.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:00 smithi107 ceph-mon[105022]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:00.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:00 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T09:51:00.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:00 smithi107 ceph-mon[102221]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:00.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:00 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T09:51:00.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:00 smithi129 ceph-mon[88471]: from='client.34590 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:00.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:00 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T09:51:01.273 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 9 16 25584 25568 177.541 136.625 0.0103652 0.00562575 2024-09-21T09:51:01.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:01 smithi107 ceph-mon[102221]: pgmap v165: 89 pgs: 89 active+clean; 970 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-21T09:51:01.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:01 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T09:51:01.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:01 smithi107 ceph-mon[102221]: Upgrade: osd.2 is safe to restart 2024-09-21T09:51:01.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:01 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:01.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:01 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3792916565' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:01.860 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:51:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:51:01] "GET /metrics HTTP/1.1" 200 36498 "" "Prometheus/2.51.0" 2024-09-21T09:51:01.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:01 smithi107 ceph-mon[105022]: pgmap v165: 89 pgs: 89 active+clean; 970 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-21T09:51:01.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:01 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T09:51:01.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:01 smithi107 ceph-mon[105022]: Upgrade: osd.2 is safe to restart 2024-09-21T09:51:01.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:01 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:01.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:01 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3792916565' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:01.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:01 smithi129 ceph-mon[88471]: pgmap v165: 89 pgs: 89 active+clean; 970 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-21T09:51:01.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:01 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T09:51:01.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:01 smithi129 ceph-mon[88471]: Upgrade: osd.2 is safe to restart 2024-09-21T09:51:01.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:01 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:01.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:01 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3792916565' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:02.004 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:51:02.273 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 10 16 27659 27643 172.754 129.688 0.00833239 0.00578195 2024-09-21T09:51:03.273 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 11 16 29787 29771 169.137 133 0.00672234 0.00590634 2024-09-21T09:51:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:03 smithi107 ceph-mon[102221]: pgmap v166: 89 pgs: 89 active+clean; 970 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-21T09:51:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:03 smithi107 ceph-mon[102221]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:03 smithi107 ceph-mon[102221]: from='client.54361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T09:51:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:51:03.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:03 smithi107 ceph-mon[105022]: pgmap v166: 89 pgs: 89 active+clean; 970 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-21T09:51:03.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:03 smithi107 ceph-mon[105022]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:03.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:03 smithi107 ceph-mon[105022]: from='client.54361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:03.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:03.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T09:51:03.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:51:03.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:03 smithi129 ceph-mon[88471]: pgmap v166: 89 pgs: 89 active+clean; 970 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-21T09:51:03.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:03 smithi129 ceph-mon[88471]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:03.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:03 smithi129 ceph-mon[88471]: from='client.54361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:03.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:03.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T09:51:03.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:51:04.273 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 12 16 31878 31862 165.932 130.688 0.00442187 0.00602112 2024-09-21T09:51:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:04 smithi107 ceph-mon[102221]: Upgrade: Updating osd.2 2024-09-21T09:51:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:04 smithi107 ceph-mon[102221]: Deploying daemon osd.2 on smithi107 2024-09-21T09:51:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:04 smithi107 ceph-mon[105022]: Upgrade: Updating osd.2 2024-09-21T09:51:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:04 smithi107 ceph-mon[105022]: Deploying daemon osd.2 on smithi107 2024-09-21T09:51:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:04 smithi129 ceph-mon[88471]: Upgrade: Updating osd.2 2024-09-21T09:51:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:04 smithi129 ceph-mon[88471]: Deploying daemon osd.2 on smithi107 2024-09-21T09:51:05.273 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 13 16 33784 33768 162.33 119.125 0.00744731 0.00615496 2024-09-21T09:51:05.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:05 smithi107 ceph-mon[102221]: pgmap v167: 89 pgs: 89 active+clean; 1.7 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-09-21T09:51:05.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:05 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/355381362' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:05.747 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:51:05.747 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:05 smithi107 ceph-mon[105022]: pgmap v167: 89 pgs: 89 active+clean; 1.7 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-09-21T09:51:05.747 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:05 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/355381362' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:05 smithi129 ceph-mon[88471]: pgmap v167: 89 pgs: 89 active+clean; 1.7 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-09-21T09:51:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:05 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/355381362' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:06.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 14 16 35539 35523 158.569 109.688 0.0101658 0.00630103 2024-09-21T09:51:07.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 15 16 37436 37420 155.902 118.562 0.00490175 0.00640874 2024-09-21T09:51:07.603 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:07 smithi107 systemd[1]: Stopping Ceph osd.2 for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:51:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:07 smithi107 ceph-mon[102221]: from='client.34611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:07 smithi107 ceph-mon[102221]: pgmap v168: 89 pgs: 89 active+clean; 1.7 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-09-21T09:51:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:07 smithi107 ceph-mon[102221]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:07.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:07 smithi107 ceph-mon[105022]: from='client.34611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:07.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:07 smithi107 ceph-mon[105022]: pgmap v168: 89 pgs: 89 active+clean; 1.7 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-09-21T09:51:07.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:07 smithi107 ceph-mon[105022]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:07.860 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:07 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2[57889]: 2024-09-21T09:51:07.608+0000 7f0fbafec640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T09:51:07.860 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:07 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2[57889]: 2024-09-21T09:51:07.608+0000 7f0fbafec640 -1 osd.2 101 *** Got signal Terminated *** 2024-09-21T09:51:07.861 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:07 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2[57889]: 2024-09-21T09:51:07.608+0000 7f0fbafec640 -1 osd.2 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T09:51:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:07 smithi129 ceph-mon[88471]: from='client.34611 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:07 smithi129 ceph-mon[88471]: pgmap v168: 89 pgs: 89 active+clean; 1.7 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-09-21T09:51:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:07 smithi129 ceph-mon[88471]: from='client.34614 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:08.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 16 16 39453 39437 154.036 126.062 0.00456786 0.00648661 2024-09-21T09:51:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:08 smithi107 ceph-mon[102221]: osd.2 marked itself down and dead 2024-09-21T09:51:08.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:08 smithi107 ceph-mon[105022]: osd.2 marked itself down and dead 2024-09-21T09:51:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:08 smithi129 ceph-mon[88471]: osd.2 marked itself down and dead 2024-09-21T09:51:09.046 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:51:09.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 17 16 40101 40085 147.357 40.5 0.00190959 0.00652456 2024-09-21T09:51:09.359 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:09 smithi107 podman[123841]: 2024-09-21 09:51:09.242633733 +0000 UTC m=+1.805317493 container died ae11dfe6a988be62b79a17aad0395fdeb156ffc5c859e43d641b14cefe3579cb (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=quincy-fbd1a6a, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-21T09:51:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:09 smithi129 ceph-mon[88471]: pgmap v169: 89 pgs: 89 active+clean; 2.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 189 MiB/s wr, 3.03k op/s 2024-09-21T09:51:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:09 smithi129 ceph-mon[88471]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T09:51:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:09 smithi129 ceph-mon[88471]: osdmap e102: 8 total, 7 up, 8 in 2024-09-21T09:51:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:09 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2995501132' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:10.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:09 smithi107 ceph-mon[102221]: pgmap v169: 89 pgs: 89 active+clean; 2.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 189 MiB/s wr, 3.03k op/s 2024-09-21T09:51:10.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:09 smithi107 ceph-mon[102221]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T09:51:10.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:09 smithi107 ceph-mon[102221]: osdmap e102: 8 total, 7 up, 8 in 2024-09-21T09:51:10.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:09 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2995501132' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:10.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:09 smithi107 ceph-mon[105022]: pgmap v169: 89 pgs: 89 active+clean; 2.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 189 MiB/s wr, 3.03k op/s 2024-09-21T09:51:10.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:09 smithi107 ceph-mon[105022]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T09:51:10.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:09 smithi107 ceph-mon[105022]: osdmap e102: 8 total, 7 up, 8 in 2024-09-21T09:51:10.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:09 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2995501132' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:10.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 18 16 41620 41604 144.445 94.9375 0.00375787 0.00691799 2024-09-21T09:51:10.638 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:51:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:51:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:51:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:10 smithi129 ceph-mon[88471]: from='client.34623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:10 smithi129 ceph-mon[88471]: from='client.44438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:10 smithi129 ceph-mon[88471]: osdmap e103: 8 total, 7 up, 8 in 2024-09-21T09:51:11.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:10 smithi107 ceph-mon[102221]: from='client.34623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:11.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:10 smithi107 ceph-mon[102221]: from='client.44438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:11.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:10 smithi107 ceph-mon[102221]: osdmap e103: 8 total, 7 up, 8 in 2024-09-21T09:51:11.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:10 smithi107 ceph-mon[105022]: from='client.34623 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:11.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:10 smithi107 ceph-mon[105022]: from='client.44438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:11.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:10 smithi107 ceph-mon[105022]: osdmap e103: 8 total, 7 up, 8 in 2024-09-21T09:51:11.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 19 16 43807 43791 144.036 136.688 0.00639222 0.00693805 2024-09-21T09:51:11.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:11 smithi107 ceph-mon[102221]: pgmap v172: 89 pgs: 5 stale+active+clean, 84 active+clean; 2.3 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 171 MiB/s wr, 2.74k op/s 2024-09-21T09:51:11.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:11.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:51:11.860 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:51:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:51:11] "GET /metrics HTTP/1.1" 200 36510 "" "Prometheus/2.51.0" 2024-09-21T09:51:11.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:11 smithi107 ceph-mon[105022]: pgmap v172: 89 pgs: 5 stale+active+clean, 84 active+clean; 2.3 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 171 MiB/s wr, 2.74k op/s 2024-09-21T09:51:11.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:11.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:51:11.861 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:11 smithi107 podman[123841]: 2024-09-21 09:51:11.612548052 +0000 UTC m=+4.175231804 container cleanup ae11dfe6a988be62b79a17aad0395fdeb156ffc5c859e43d641b14cefe3579cb (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=quincy-fbd1a6a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-21T09:51:11.861 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:11 smithi107 bash[123841]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2 2024-09-21T09:51:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:11 smithi129 ceph-mon[88471]: pgmap v172: 89 pgs: 5 stale+active+clean, 84 active+clean; 2.3 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 171 MiB/s wr, 2.74k op/s 2024-09-21T09:51:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:51:12.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T09:51:12.273411+0000 min lat: 0.00122166 max lat: 1.03241 avg lat: 0.00700638 2024-09-21T09:51:12.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T09:51:12.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 20 16 45656 45640 142.612 115.562 0.0094757 0.00700638 2024-09-21T09:51:12.359 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:12 smithi107 podman[123930]: 2024-09-21 09:51:12.056180824 +0000 UTC m=+2.817464435 container remove ae11dfe6a988be62b79a17aad0395fdeb156ffc5c859e43d641b14cefe3579cb (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=quincy-fbd1a6a, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-21T09:51:12.421 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:51:12.820 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:12 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2609936647' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:12.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:12 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2609936647' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:12.821 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:12 smithi107 podman[124067]: 2024-09-21 09:51:12.513454558 +0000 UTC m=+0.048680916 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:51:12.821 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:12 smithi107 podman[124067]: 2024-09-21 09:51:12.666156599 +0000 UTC m=+0.201382954 container create 4223f1e8659b56a7c31be45b216dbe992ae06f6e763e7b154fa8e312e0d2ce65 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T09:51:12.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:12 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2609936647' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:13.109 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:12 smithi107 podman[124067]: 2024-09-21 09:51:12.936208596 +0000 UTC m=+0.471434963 container init 4223f1e8659b56a7c31be45b216dbe992ae06f6e763e7b154fa8e312e0d2ce65 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T09:51:13.109 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:12 smithi107 podman[124067]: 2024-09-21 09:51:12.947314589 +0000 UTC m=+0.482540948 container start 4223f1e8659b56a7c31be45b216dbe992ae06f6e763e7b154fa8e312e0d2ce65 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-21T09:51:13.109 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:13 smithi107 podman[124067]: 2024-09-21 09:51:13.036113785 +0000 UTC m=+0.571340140 container attach 4223f1e8659b56a7c31be45b216dbe992ae06f6e763e7b154fa8e312e0d2ce65 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T09:51:13.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 21 16 47525 47509 141.382 116.812 0.00568742 0.00706812 2024-09-21T09:51:13.609 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:13 smithi107 podman[124131]: 2024-09-21 09:51:13.312000523 +0000 UTC m=+0.040641896 container died 4223f1e8659b56a7c31be45b216dbe992ae06f6e763e7b154fa8e312e0d2ce65 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-deactivate, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T09:51:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:13 smithi129 ceph-mon[88471]: pgmap v173: 89 pgs: 5 stale+active+clean, 84 active+clean; 2.3 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T09:51:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:13 smithi129 ceph-mon[88471]: from='client.34641 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:14.004 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:13 smithi107 ceph-mon[102221]: pgmap v173: 89 pgs: 5 stale+active+clean, 84 active+clean; 2.3 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T09:51:14.004 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:13 smithi107 ceph-mon[102221]: from='client.34641 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:14.004 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:13 smithi107 ceph-mon[105022]: pgmap v173: 89 pgs: 5 stale+active+clean, 84 active+clean; 2.3 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T09:51:14.005 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:13 smithi107 ceph-mon[105022]: from='client.34641 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:14.005 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:14 smithi107 podman[124131]: 2024-09-21 09:51:14.002676838 +0000 UTC m=+0.731318198 container remove 4223f1e8659b56a7c31be45b216dbe992ae06f6e763e7b154fa8e312e0d2ce65 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-21T09:51:14.274 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 22 16 49616 49600 140.896 130.688 0.00686568 0.00709243 2024-09-21T09:51:14.339 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:14 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.2.service: Deactivated successfully. 2024-09-21T09:51:14.339 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:14 smithi107 systemd[1]: Stopped Ceph osd.2 for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:51:14.339 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:14 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.2.service: Consumed 15min 18.117s CPU time. 2024-09-21T09:51:14.609 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:14 smithi107 systemd[1]: Starting Ceph osd.2 for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:51:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:14 smithi129 ceph-mon[88471]: from='client.44453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:14.992 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:14 smithi107 ceph-mon[102221]: from='client.44453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:14.993 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:14 smithi107 ceph-mon[105022]: from='client.44453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:14.993 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:14 smithi107 podman[124227]: 2024-09-21 09:51:14.572529111 +0000 UTC m=+0.035214195 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:51:14.993 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:14 smithi107 podman[124227]: 2024-09-21 09:51:14.712448305 +0000 UTC m=+0.175133368 container create b67c468a612db78f3b74444d27c5b55f68e76420b7341f4344cbbd9b2a42cd96 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-activate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-21T09:51:14.993 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:14 smithi107 podman[124227]: 2024-09-21 09:51:14.990816256 +0000 UTC m=+0.453501330 container init b67c468a612db78f3b74444d27c5b55f68e76420b7341f4344cbbd9b2a42cd96 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-activate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-21T09:51:15.252 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:15 smithi107 podman[124227]: 2024-09-21 09:51:15.002622563 +0000 UTC m=+0.465307641 container start b67c468a612db78f3b74444d27c5b55f68e76420b7341f4344cbbd9b2a42cd96 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-activate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-21T09:51:15.252 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:15 smithi107 podman[124227]: 2024-09-21 09:51:15.107435188 +0000 UTC m=+0.570120251 container attach b67c468a612db78f3b74444d27c5b55f68e76420b7341f4344cbbd9b2a42cd96 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-activate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=main-62da54b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS) 2024-09-21T09:51:15.252 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:15 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-activate[124240]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T09:51:15.252 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:15 smithi107 bash[124227]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T09:51:15.253 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:15 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-activate[124240]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T09:51:15.253 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:15 smithi107 bash[124227]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T09:51:15.275 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 23 16 51504 51488 139.896 118 0.00886288 0.00714235 2024-09-21T09:51:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:15 smithi129 ceph-mon[88471]: pgmap v174: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 2.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s; 17155/136731 objects degraded (12.547%); 8.0 KiB/s, 0 objects/s recovering 2024-09-21T09:51:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:15 smithi129 ceph-mon[88471]: Health check failed: Degraded data redundancy: 17155/136731 objects degraded (12.547%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:15 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2318223081' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:16.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:15 smithi107 ceph-mon[102221]: pgmap v174: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 2.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s; 17155/136731 objects degraded (12.547%); 8.0 KiB/s, 0 objects/s recovering 2024-09-21T09:51:16.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:15 smithi107 ceph-mon[102221]: Health check failed: Degraded data redundancy: 17155/136731 objects degraded (12.547%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:16.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:15 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2318223081' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:16.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:15 smithi107 ceph-mon[105022]: pgmap v174: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 2.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s; 17155/136731 objects degraded (12.547%); 8.0 KiB/s, 0 objects/s recovering 2024-09-21T09:51:16.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:15 smithi107 ceph-mon[105022]: Health check failed: Degraded data redundancy: 17155/136731 objects degraded (12.547%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:16.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:15 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2318223081' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:16.110 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:15 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-activate[124240]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T09:51:16.110 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:15 smithi107 bash[124227]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T09:51:16.110 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:15 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-activate[124240]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-21T09:51:16.110 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:15 smithi107 bash[124227]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-21T09:51:16.124 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:51:16.275 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 24 16 53249 53233 138.611 109.062 0.00972666 0.00720928 2024-09-21T09:51:16.607 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:16 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-activate[124240]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-21T09:51:16.699 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:16 smithi107 bash[124227]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-21T09:51:16.699 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:16 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-activate[124240]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-21T09:51:16.699 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:16 smithi107 bash[124227]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-21T09:51:16.699 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:16 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-activate[124240]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-21T09:51:16.699 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:16 smithi107 bash[124227]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-21T09:51:16.699 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:16 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-activate[124240]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T09:51:16.699 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:16 smithi107 bash[124227]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T09:51:16.700 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:16 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-activate[124240]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-21T09:51:16.700 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:16 smithi107 bash[124227]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-21T09:51:16.700 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:16 smithi107 podman[124227]: 2024-09-21 09:51:16.248712989 +0000 UTC m=+1.711398064 container died b67c468a612db78f3b74444d27c5b55f68e76420b7341f4344cbbd9b2a42cd96 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-activate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T09:51:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:16 smithi107 ceph-mon[102221]: pgmap v175: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 2.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s; 17155/136731 objects degraded (12.547%); 8.0 KiB/s, 0 objects/s recovering 2024-09-21T09:51:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:16 smithi107 ceph-mon[102221]: from='client.44462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:16 smithi107 ceph-mon[105022]: pgmap v175: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 2.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s; 17155/136731 objects degraded (12.547%); 8.0 KiB/s, 0 objects/s recovering 2024-09-21T09:51:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:16 smithi107 ceph-mon[105022]: from='client.44462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:16 smithi129 ceph-mon[88471]: pgmap v175: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 2.8 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s; 17155/136731 objects degraded (12.547%); 8.0 KiB/s, 0 objects/s recovering 2024-09-21T09:51:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:16 smithi129 ceph-mon[88471]: from='client.44462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:17.275 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 25 16 55185 55169 137.906 121 0.00698356 0.00724644 2024-09-21T09:51:17.471 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:17 smithi107 podman[124445]: 2024-09-21 09:51:17.171870944 +0000 UTC m=+0.909200047 container remove b67c468a612db78f3b74444d27c5b55f68e76420b7341f4344cbbd9b2a42cd96 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-activate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-21T09:51:17.819 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:17 smithi107 ceph-mon[102221]: from='client.54409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:17.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:17 smithi107 ceph-mon[105022]: from='client.54409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:17.820 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:17 smithi107 podman[124516]: 2024-09-21 09:51:17.377329434 +0000 UTC m=+0.030201784 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:51:17.820 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:17 smithi107 podman[124516]: 2024-09-21 09:51:17.514635955 +0000 UTC m=+0.167508290 container create 4e385b5d2d2818f498c6b13d6b6ac379e03f8b2be82eeb8237d17416971d11a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-21T09:51:17.820 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:17 smithi107 podman[124516]: 2024-09-21 09:51:17.817848868 +0000 UTC m=+0.470721214 container init 4e385b5d2d2818f498c6b13d6b6ac379e03f8b2be82eeb8237d17416971d11a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=main-62da54b, CEPH_POINT_RELEASE=) 2024-09-21T09:51:18.109 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:17 smithi107 podman[124516]: 2024-09-21 09:51:17.826687442 +0000 UTC m=+0.479559781 container start 4e385b5d2d2818f498c6b13d6b6ac379e03f8b2be82eeb8237d17416971d11a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=main-62da54b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T09:51:18.109 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:17 smithi107 ceph-osd[124551]: -- 172.21.15.107:0/2452726144 <== mon.0 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55a56d38c1a0 con 0x55a56d350000 2024-09-21T09:51:18.109 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:17 smithi107 ceph-osd[124551]: -- 172.21.15.107:0/2452726144 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a56c579a40 con 0x55a56d350000 2024-09-21T09:51:18.109 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:17 smithi107 bash[124516]: 4e385b5d2d2818f498c6b13d6b6ac379e03f8b2be82eeb8237d17416971d11a2 2024-09-21T09:51:18.109 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:17 smithi107 systemd[1]: Started Ceph osd.2 for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:51:18.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:17 smithi129 ceph-mon[88471]: from='client.54409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:18.275 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 26 16 57062 57046 137.114 117.312 0.0184641 0.00728747 2024-09-21T09:51:19.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:19 smithi129 ceph-mon[88471]: pgmap v176: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s; 20778/163602 objects degraded (12.700%); 6.9 KiB/s, 0 objects/s recovering 2024-09-21T09:51:19.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:19 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:19.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:19 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:19.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:19 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:51:19.275 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 27 16 58683 58667 135.787 101.312 0.00743836 0.00735884 2024-09-21T09:51:19.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:19 smithi107 ceph-mon[102221]: pgmap v176: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s; 20778/163602 objects degraded (12.700%); 6.9 KiB/s, 0 objects/s recovering 2024-09-21T09:51:19.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:19 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:19.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:19 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:19.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:19 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:51:19.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:19 smithi107 ceph-mon[105022]: pgmap v176: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s; 20778/163602 objects degraded (12.700%); 6.9 KiB/s, 0 objects/s recovering 2024-09-21T09:51:19.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:19 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:19.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:19 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:19.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:19 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:51:19.360 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:19 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2[124527]: 2024-09-21T09:51:19.230+0000 7fb992273640 -1 Falling back to public interface 2024-09-21T09:51:19.754 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:51:20.275 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 28 16 60356 60340 134.672 104.562 0.0117778 0.0074204 2024-09-21T09:51:20.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:20 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4160532477' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:20.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:20 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4160532477' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:20.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:20 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4160532477' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:20.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:51:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:51:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:51:21.276 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 29 16 61836 61820 133.218 92.5 0.00568256 0.00750167 2024-09-21T09:51:21.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:21 smithi107 ceph-mon[102221]: from='client.34671 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:21.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:21 smithi107 ceph-mon[102221]: pgmap v177: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 107 MiB/s wr, 1.72k op/s; 20778/165483 objects degraded (12.556%); 6.2 KiB/s, 0 objects/s recovering 2024-09-21T09:51:21.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:21 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 20778/165483 objects degraded (12.556%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:21.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:21 smithi107 ceph-mon[102221]: from='client.34674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:21.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:21 smithi107 ceph-mon[105022]: from='client.34671 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:21.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:21 smithi107 ceph-mon[105022]: pgmap v177: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 107 MiB/s wr, 1.72k op/s; 20778/165483 objects degraded (12.556%); 6.2 KiB/s, 0 objects/s recovering 2024-09-21T09:51:21.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:21 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 20778/165483 objects degraded (12.556%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:21.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:21 smithi107 ceph-mon[105022]: from='client.34674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:21.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:21 smithi129 ceph-mon[88471]: from='client.34671 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:21.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:21 smithi129 ceph-mon[88471]: pgmap v177: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 107 MiB/s wr, 1.72k op/s; 20778/165483 objects degraded (12.556%); 6.2 KiB/s, 0 objects/s recovering 2024-09-21T09:51:21.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:21 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 20778/165483 objects degraded (12.556%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:21.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:21 smithi129 ceph-mon[88471]: from='client.34674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:22.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:51:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:51:21] "GET /metrics HTTP/1.1" 200 36510 "" "Prometheus/2.51.0" 2024-09-21T09:51:22.277 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 30 16 63509 63493 132.258 104.562 0.012949 0.00755574 2024-09-21T09:51:23.277 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 31 16 64580 64564 130.151 66.9375 0.0106058 0.00767786 2024-09-21T09:51:23.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:23 smithi107 ceph-mon[102221]: pgmap v178: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s; 20778/165483 objects degraded (12.556%); 5.3 KiB/s, 0 objects/s recovering 2024-09-21T09:51:23.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:23 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1608645293' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:23.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:23 smithi107 ceph-mon[105022]: pgmap v178: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s; 20778/165483 objects degraded (12.556%); 5.3 KiB/s, 0 objects/s recovering 2024-09-21T09:51:23.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:23 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1608645293' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:23.406 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:51:23.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:23 smithi129 ceph-mon[88471]: pgmap v178: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s; 20778/165483 objects degraded (12.556%); 5.3 KiB/s, 0 objects/s recovering 2024-09-21T09:51:23.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:23 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1608645293' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:24.277 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 32 16 65720 65704 128.31 71.25 0.00569102 0.00778899 2024-09-21T09:51:24.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:24 smithi107 ceph-mon[102221]: from='client.44486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:24.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:24 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:24.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:24 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:24.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:24 smithi107 ceph-mon[105022]: from='client.44486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:24.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:24 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:24.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:24 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:24.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:24 smithi129 ceph-mon[88471]: from='client.44486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:24.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:24 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:24.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:24 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:25.277 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 33 16 67284 67268 127.384 97.75 0.00839136 0.00784526 2024-09-21T09:51:25.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:25 smithi107 ceph-mon[102221]: from='client.54430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:25.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:25 smithi107 ceph-mon[102221]: pgmap v179: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 135 MiB/s wr, 2.16k op/s; 23949/190317 objects degraded (12.584%); 5.3 KiB/s, 0 objects/s recovering 2024-09-21T09:51:25.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:25 smithi107 ceph-mon[105022]: from='client.54430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:25.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:25 smithi107 ceph-mon[105022]: pgmap v179: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 135 MiB/s wr, 2.16k op/s; 23949/190317 objects degraded (12.584%); 5.3 KiB/s, 0 objects/s recovering 2024-09-21T09:51:25.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:25 smithi129 ceph-mon[88471]: from='client.54430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:25.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:25 smithi129 ceph-mon[88471]: pgmap v179: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 135 MiB/s wr, 2.16k op/s; 23949/190317 objects degraded (12.584%); 5.3 KiB/s, 0 objects/s recovering 2024-09-21T09:51:26.277 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 34 16 68726 68710 126.288 90.125 0.00476581 0.00791256 2024-09-21T09:51:27.047 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:51:27.277 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 35 16 69837 69821 124.663 69.4375 0.0115677 0.00801585 2024-09-21T09:51:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:27 smithi107 ceph-mon[102221]: pgmap v180: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s; 23949/190317 objects degraded (12.584%) 2024-09-21T09:51:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:51:27.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:27 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1935563310' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:27.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:27.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:27.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:27 smithi107 ceph-mon[105022]: pgmap v180: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s; 23949/190317 objects degraded (12.584%) 2024-09-21T09:51:27.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:27.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:51:27.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:27 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1935563310' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:27.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:27.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:27 smithi129 ceph-mon[88471]: pgmap v180: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 93 MiB/s wr, 1.49k op/s; 23949/190317 objects degraded (12.584%) 2024-09-21T09:51:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:51:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:27 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1935563310' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:28.277 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 36 16 71073 71057 123.345 77.25 0.00571159 0.00810209 2024-09-21T09:51:28.595 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:28 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2[124527]: 2024-09-21T09:51:28.455+0000 7fb992273640 -1 osd.2 101 log_to_monitors true 2024-09-21T09:51:28.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:28 smithi107 ceph-mon[102221]: from='client.44501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:28.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:28 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:28.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:28 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:28.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:28 smithi107 ceph-mon[102221]: from='client.44507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:28.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:28 smithi107 ceph-mon[102221]: from='osd.2 [v2:172.21.15.107:6818/1583228557,v1:172.21.15.107:6819/1583228557]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T09:51:28.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:28 smithi107 ceph-mon[102221]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T09:51:28.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:28 smithi107 ceph-mon[105022]: from='client.44501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:28.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:28 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:28.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:28 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:28.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:28 smithi107 ceph-mon[105022]: from='client.44507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:28.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:28 smithi107 ceph-mon[105022]: from='osd.2 [v2:172.21.15.107:6818/1583228557,v1:172.21.15.107:6819/1583228557]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T09:51:28.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:28 smithi107 ceph-mon[105022]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T09:51:28.860 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 09:51:28 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2[124527]: 2024-09-21T09:51:28.690+0000 7fb98a01d640 -1 osd.2 101 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:51:28.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:28 smithi129 ceph-mon[88471]: from='client.44501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:28.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:28 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:28.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:28 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:28.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:28 smithi129 ceph-mon[88471]: from='client.44507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:28.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:28 smithi129 ceph-mon[88471]: from='osd.2 [v2:172.21.15.107:6818/1583228557,v1:172.21.15.107:6819/1583228557]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T09:51:28.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:28 smithi129 ceph-mon[88471]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T09:51:29.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 37 16 72522 72506 122.459 90.5625 0.0140041 0.00816114 2024-09-21T09:51:29.809 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:29 smithi107 ceph-mon[102221]: pgmap v181: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 124 MiB/s wr, 1.99k op/s; 26339/208314 objects degraded (12.644%) 2024-09-21T09:51:29.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:29 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 26339/208314 objects degraded (12.644%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:29.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:29 smithi107 ceph-mon[102221]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T09:51:29.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:29 smithi107 ceph-mon[102221]: from='osd.2 [v2:172.21.15.107:6818/1583228557,v1:172.21.15.107:6819/1583228557]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T09:51:29.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:29 smithi107 ceph-mon[102221]: osdmap e104: 8 total, 7 up, 8 in 2024-09-21T09:51:29.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:29 smithi107 ceph-mon[102221]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T09:51:29.810 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:29 smithi107 ceph-mon[105022]: pgmap v181: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 124 MiB/s wr, 1.99k op/s; 26339/208314 objects degraded (12.644%) 2024-09-21T09:51:29.810 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:29 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 26339/208314 objects degraded (12.644%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:29.810 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:29 smithi107 ceph-mon[105022]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T09:51:29.810 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:29 smithi107 ceph-mon[105022]: from='osd.2 [v2:172.21.15.107:6818/1583228557,v1:172.21.15.107:6819/1583228557]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T09:51:29.811 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:29 smithi107 ceph-mon[105022]: osdmap e104: 8 total, 7 up, 8 in 2024-09-21T09:51:29.811 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:29 smithi107 ceph-mon[105022]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T09:51:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:29 smithi129 ceph-mon[88471]: pgmap v181: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 124 MiB/s wr, 1.99k op/s; 26339/208314 objects degraded (12.644%) 2024-09-21T09:51:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:29 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 26339/208314 objects degraded (12.644%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:29 smithi129 ceph-mon[88471]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T09:51:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:29 smithi129 ceph-mon[88471]: from='osd.2 [v2:172.21.15.107:6818/1583228557,v1:172.21.15.107:6819/1583228557]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T09:51:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:29 smithi129 ceph-mon[88471]: osdmap e104: 8 total, 7 up, 8 in 2024-09-21T09:51:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:29 smithi129 ceph-mon[88471]: from='osd.2 ' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T09:51:30.215 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:51:30.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 38 16 73132 73116 120.239 38.125 0.00288476 0.00817808 2024-09-21T09:51:30.597 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:51:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:51:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:51:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:30 smithi107 ceph-mon[105022]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T09:51:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:30 smithi107 ceph-mon[105022]: osd.2 [v2:172.21.15.107:6818/1583228557,v1:172.21.15.107:6819/1583228557] boot 2024-09-21T09:51:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:30 smithi107 ceph-mon[105022]: osdmap e105: 8 total, 8 up, 8 in 2024-09-21T09:51:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:30 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T09:51:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:30 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/846838915' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:30.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:30 smithi107 ceph-mon[102221]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T09:51:30.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:30 smithi107 ceph-mon[102221]: osd.2 [v2:172.21.15.107:6818/1583228557,v1:172.21.15.107:6819/1583228557] boot 2024-09-21T09:51:30.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:30 smithi107 ceph-mon[102221]: osdmap e105: 8 total, 8 up, 8 in 2024-09-21T09:51:30.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:30 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T09:51:30.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:30 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/846838915' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:30.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:30 smithi129 ceph-mon[88471]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T09:51:30.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:30 smithi129 ceph-mon[88471]: osd.2 [v2:172.21.15.107:6818/1583228557,v1:172.21.15.107:6819/1583228557] boot 2024-09-21T09:51:30.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:30 smithi129 ceph-mon[88471]: osdmap e105: 8 total, 8 up, 8 in 2024-09-21T09:51:30.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:30 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T09:51:30.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:30 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/846838915' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:31.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 39 16 73132 73116 117.156 0 - 0.00817808 2024-09-21T09:51:31.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:31 smithi107 ceph-mon[102221]: pgmap v184: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 115 MiB/s wr, 1.83k op/s; 26339/209544 objects degraded (12.570%) 2024-09-21T09:51:31.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:31 smithi107 ceph-mon[102221]: from='client.44522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:31.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:31 smithi107 ceph-mon[102221]: from='client.34704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:31.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:31 smithi107 ceph-mon[102221]: osdmap e106: 8 total, 8 up, 8 in 2024-09-21T09:51:31.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:31 smithi107 ceph-mon[105022]: pgmap v184: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 115 MiB/s wr, 1.83k op/s; 26339/209544 objects degraded (12.570%) 2024-09-21T09:51:31.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:31 smithi107 ceph-mon[105022]: from='client.44522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:31.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:31 smithi107 ceph-mon[105022]: from='client.34704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:31.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:31 smithi107 ceph-mon[105022]: osdmap e106: 8 total, 8 up, 8 in 2024-09-21T09:51:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:31 smithi129 ceph-mon[88471]: pgmap v184: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 115 MiB/s wr, 1.83k op/s; 26339/209544 objects degraded (12.570%) 2024-09-21T09:51:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:31 smithi129 ceph-mon[88471]: from='client.44522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:31 smithi129 ceph-mon[88471]: from='client.34704 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:31 smithi129 ceph-mon[88471]: osdmap e106: 8 total, 8 up, 8 in 2024-09-21T09:51:32.051 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:51:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:51:31] "GET /metrics HTTP/1.1" 200 36522 "" "Prometheus/2.51.0" 2024-09-21T09:51:32.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T09:51:32.277214+0000 min lat: 0.00122166 max lat: 2.22402 avg lat: 0.00868548 2024-09-21T09:51:32.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T09:51:32.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 40 16 73660 73644 115.052 16.5 0.0142511 0.00868548 2024-09-21T09:51:32.359 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:51:32.048+0000 7efdb5a17640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-21T09:51:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:32 smithi129 ceph-mon[88471]: osdmap e107: 8 total, 8 up, 8 in 2024-09-21T09:51:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:32 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:32 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:32 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:51:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:32 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:51:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:32 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:32 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:51:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:32 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:51:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:32 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:51:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:32 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:51:32.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:32 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:51:33.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[102221]: osdmap e107: 8 total, 8 up, 8 in 2024-09-21T09:51:33.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:33.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:33.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:51:33.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:51:33.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:33.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:51:33.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:51:33.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:51:33.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:51:33.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:51:33.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[105022]: osdmap e107: 8 total, 8 up, 8 in 2024-09-21T09:51:33.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:33.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:33.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:51:33.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:51:33.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:33.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:51:33.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:51:33.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:51:33.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:51:33.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:32 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:51:33.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 41 16 74691 74675 113.818 64.4375 0.0091497 0.00878019 2024-09-21T09:51:33.610 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:51:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:33 smithi129 ceph-mon[88471]: pgmap v187: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 6.4 MiB/s wr, 102 op/s; 26339/209544 objects degraded (12.570%) 2024-09-21T09:51:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:33 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:51:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:33 smithi129 ceph-mon[88471]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-21T09:51:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:33 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1430506806' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:34.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:33 smithi107 ceph-mon[102221]: pgmap v187: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 6.4 MiB/s wr, 102 op/s; 26339/209544 objects degraded (12.570%) 2024-09-21T09:51:34.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:33 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:51:34.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:33 smithi107 ceph-mon[102221]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-21T09:51:34.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:33 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1430506806' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:34.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:33 smithi107 ceph-mon[105022]: pgmap v187: 89 pgs: 16 active+undersized, 9 active+undersized+degraded, 64 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 6.4 MiB/s wr, 102 op/s; 26339/209544 objects degraded (12.570%) 2024-09-21T09:51:34.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:33 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:51:34.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:33 smithi107 ceph-mon[105022]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-21T09:51:34.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:33 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1430506806' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:34.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 42 16 75652 75636 112.538 60.0625 0.00889225 0.00888039 2024-09-21T09:51:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:34 smithi129 ceph-mon[88471]: from='client.44540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:34 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 26339/209544 objects degraded (12.570%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:35.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:34 smithi107 ceph-mon[102221]: from='client.44540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:34 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 26339/209544 objects degraded (12.570%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:35.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:34 smithi107 ceph-mon[105022]: from='client.44540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:35.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:34 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 26339/209544 objects degraded (12.570%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:35.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 43 16 76767 76751 111.541 69.6875 0.00428302 0.00896004 2024-09-21T09:51:35.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:35 smithi129 ceph-mon[88471]: pgmap v188: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 83 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 45 MiB/s wr, 719 op/s; 12483/221007 objects degraded (5.648%); 374 KiB/s, 5 objects/s recovering 2024-09-21T09:51:35.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:35 smithi129 ceph-mon[88471]: from='client.34713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:36.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:35 smithi107 ceph-mon[102221]: pgmap v188: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 83 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 45 MiB/s wr, 719 op/s; 12483/221007 objects degraded (5.648%); 374 KiB/s, 5 objects/s recovering 2024-09-21T09:51:36.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:35 smithi107 ceph-mon[102221]: from='client.34713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:36.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:35 smithi107 ceph-mon[105022]: pgmap v188: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 83 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 45 MiB/s wr, 719 op/s; 12483/221007 objects degraded (5.648%); 374 KiB/s, 5 objects/s recovering 2024-09-21T09:51:36.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:35 smithi107 ceph-mon[105022]: from='client.34713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:36.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 44 16 77865 77849 110.566 68.625 0.0193276 0.00903945 2024-09-21T09:51:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:36 smithi107 ceph-mon[102221]: pgmap v189: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 83 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 38 MiB/s wr, 607 op/s; 12483/221007 objects degraded (5.648%); 315 KiB/s, 4 objects/s recovering 2024-09-21T09:51:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:36 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1947395333' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:37.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:36 smithi107 ceph-mon[105022]: pgmap v189: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 83 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 38 MiB/s wr, 607 op/s; 12483/221007 objects degraded (5.648%); 315 KiB/s, 4 objects/s recovering 2024-09-21T09:51:37.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:36 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1947395333' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:37.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:36 smithi129 ceph-mon[88471]: pgmap v189: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 83 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 38 MiB/s wr, 607 op/s; 12483/221007 objects degraded (5.648%); 315 KiB/s, 4 objects/s recovering 2024-09-21T09:51:37.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:36 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1947395333' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:37.264 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:51:37.279 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 45 16 78812 78796 109.424 59.1875 0.0270346 0.00913239 2024-09-21T09:51:38.069 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:37 smithi129 ceph-mon[88471]: from='client.34722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:38.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:37 smithi107 ceph-mon[102221]: from='client.34722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:38.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:37 smithi107 ceph-mon[105022]: from='client.34722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:38.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 46 16 79834 79818 108.434 63.875 0.0302142 0.00921684 2024-09-21T09:51:39.080 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:38 smithi129 ceph-mon[88471]: from='client.44558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:39.080 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:38 smithi129 ceph-mon[88471]: pgmap v190: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s; 12367/235560 objects degraded (5.250%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T09:51:39.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:38 smithi107 ceph-mon[105022]: from='client.44558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:39.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:38 smithi107 ceph-mon[105022]: pgmap v190: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s; 12367/235560 objects degraded (5.250%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T09:51:39.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:38 smithi107 ceph-mon[102221]: from='client.44558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:39.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:38 smithi107 ceph-mon[102221]: pgmap v190: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s; 12367/235560 objects degraded (5.250%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T09:51:39.278 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 47 16 80899 80883 107.543 66.5625 0.0116964 0.00929275 2024-09-21T09:51:40.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:39 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 12367/235560 objects degraded (5.250%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:40.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:39 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 12367/235560 objects degraded (5.250%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:40.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:39 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 12367/235560 objects degraded (5.250%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:40.279 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 48 16 81914 81898 106.624 63.4375 0.0390014 0.00937246 2024-09-21T09:51:40.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:51:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:51:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:51:40.926 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:51:41.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:40 smithi107 ceph-mon[105022]: pgmap v191: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 61 MiB/s wr, 968 op/s; 12367/236550 objects degraded (5.228%); 1013 KiB/s, 15 objects/s recovering 2024-09-21T09:51:41.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:40 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1760499868' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:40 smithi107 ceph-mon[102221]: pgmap v191: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 61 MiB/s wr, 968 op/s; 12367/236550 objects degraded (5.228%); 1013 KiB/s, 15 objects/s recovering 2024-09-21T09:51:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:40 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1760499868' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:40 smithi129 ceph-mon[88471]: pgmap v191: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 61 MiB/s wr, 968 op/s; 12367/236550 objects degraded (5.228%); 1013 KiB/s, 15 objects/s recovering 2024-09-21T09:51:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:40 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1760499868' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:41.279 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 49 16 82817 82801 105.6 56.4375 0.0089308 0.00946478 2024-09-21T09:51:42.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:51:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:51:41] "GET /metrics HTTP/1.1" 200 36533 "" "Prometheus/2.51.0" 2024-09-21T09:51:42.279 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 50 16 83846 83830 104.774 64.3125 0.013017 0.00953816 2024-09-21T09:51:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:42 smithi129 ceph-mon[88471]: from='client.54493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:51:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:42 smithi129 ceph-mon[88471]: from='client.54499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:42 smithi107 ceph-mon[102221]: from='client.54493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:51:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:42 smithi107 ceph-mon[102221]: from='client.54499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:42 smithi107 ceph-mon[105022]: from='client.54493 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:51:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:51:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:42 smithi107 ceph-mon[105022]: from='client.54499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:43.279 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 51 16 84867 84851 103.97 63.8125 0.0120584 0.00961286 2024-09-21T09:51:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:43 smithi129 ceph-mon[88471]: pgmap v192: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 55 MiB/s wr, 875 op/s; 12367/236550 objects degraded (5.228%); 915 KiB/s, 14 objects/s recovering 2024-09-21T09:51:43.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:43 smithi107 ceph-mon[105022]: pgmap v192: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 55 MiB/s wr, 875 op/s; 12367/236550 objects degraded (5.228%); 915 KiB/s, 14 objects/s recovering 2024-09-21T09:51:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:43 smithi107 ceph-mon[102221]: pgmap v192: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 55 MiB/s wr, 875 op/s; 12367/236550 objects degraded (5.228%); 915 KiB/s, 14 objects/s recovering 2024-09-21T09:51:44.279 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 52 16 85802 85786 103.095 58.4375 0.00799302 0.00969375 2024-09-21T09:51:44.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:44 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4236194641' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:44.594 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:51:44.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:44 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4236194641' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:44.594 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:44 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4236194641' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:45.279 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 53 16 86699 86683 102.207 56.0625 0.0202123 0.00977813 2024-09-21T09:51:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:45 smithi129 ceph-mon[88471]: pgmap v193: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 12228/251709 objects degraded (4.858%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T09:51:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:45 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 12367/236550 objects degraded (5.228%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:45 smithi129 ceph-mon[88471]: from='client.34752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:45 smithi107 ceph-mon[102221]: pgmap v193: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 12228/251709 objects degraded (4.858%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T09:51:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:45 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 12367/236550 objects degraded (5.228%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:45 smithi107 ceph-mon[102221]: from='client.34752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:45 smithi107 ceph-mon[105022]: pgmap v193: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 12228/251709 objects degraded (4.858%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T09:51:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:45 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 12367/236550 objects degraded (5.228%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:45 smithi107 ceph-mon[105022]: from='client.34752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:46.279 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 54 16 87711 87695 101.486 63.25 0.0131521 0.0098481 2024-09-21T09:51:46.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:46 smithi107 ceph-mon[105022]: from='client.54511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:46.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:46 smithi107 ceph-mon[102221]: from='client.54511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:46.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:46 smithi129 ceph-mon[88471]: from='client.54511 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:47.279 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 55 16 88713 88697 100.779 62.625 0.0142875 0.00991664 2024-09-21T09:51:47.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:47 smithi107 ceph-mon[105022]: pgmap v194: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s; 12228/251709 objects degraded (4.858%); 1.3 MiB/s, 21 objects/s recovering 2024-09-21T09:51:47.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:47 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:51:47.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:47 smithi107 ceph-mon[102221]: pgmap v194: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s; 12228/251709 objects degraded (4.858%); 1.3 MiB/s, 21 objects/s recovering 2024-09-21T09:51:47.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:47 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:51:47.360 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:51:47 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:51:47.064+0000 7efdb5a17640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-21T09:51:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:47 smithi129 ceph-mon[88471]: pgmap v194: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s; 12228/251709 objects degraded (4.858%); 1.3 MiB/s, 21 objects/s recovering 2024-09-21T09:51:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:47 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:51:48.251 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:51:48.279 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 56 16 89611 89595 99.9817 56.125 0.00721083 0.00999538 2024-09-21T09:51:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:48 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:51:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:48 smithi107 ceph-mon[102221]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-21T09:51:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:48 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1114454665' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:48.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:48 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:51:48.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:48 smithi107 ceph-mon[105022]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-21T09:51:48.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:48 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1114454665' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:48.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:48 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:51:48.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:48 smithi129 ceph-mon[88471]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-21T09:51:48.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:48 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1114454665' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:49.279 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 57 16 90553 90537 99.2605 58.875 0.00876509 0.0100686 2024-09-21T09:51:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:49 smithi107 ceph-mon[102221]: pgmap v195: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s; 12098/265410 objects degraded (4.558%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T09:51:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:49 smithi107 ceph-mon[102221]: from='client.44597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:49 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 12098/265410 objects degraded (4.558%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:49.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:49 smithi107 ceph-mon[105022]: pgmap v195: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s; 12098/265410 objects degraded (4.558%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T09:51:49.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:49 smithi107 ceph-mon[105022]: from='client.44597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:49.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:49 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 12098/265410 objects degraded (4.558%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:49.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:49 smithi129 ceph-mon[88471]: pgmap v195: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s; 12098/265410 objects degraded (4.558%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T09:51:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:49 smithi129 ceph-mon[88471]: from='client.44597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:49 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 12098/265410 objects degraded (4.558%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:50.279 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 58 16 91526 91510 98.5975 60.8125 0.00583807 0.0101363 2024-09-21T09:51:50.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:50 smithi129 ceph-mon[88471]: from='client.34773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:50.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:51:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:51:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:51:50.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:50 smithi107 ceph-mon[102221]: from='client.34773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:50.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:50 smithi107 ceph-mon[105022]: from='client.34773 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:51.280 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 59 16 92484 92468 97.9411 59.875 0.0208348 0.0102038 2024-09-21T09:51:51.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:51 smithi107 ceph-mon[102221]: pgmap v196: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s; 12098/266259 objects degraded (4.544%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T09:51:51.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:51 smithi107 ceph-mon[105022]: pgmap v196: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s; 12098/266259 objects degraded (4.544%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T09:51:51.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:51 smithi129 ceph-mon[88471]: pgmap v196: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s; 12098/266259 objects degraded (4.544%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T09:51:51.922 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:51:52.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:51:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:51:51] "GET /metrics HTTP/1.1" 200 36533 "" "Prometheus/2.51.0" 2024-09-21T09:51:52.280 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T09:51:52.279014+0000 min lat: 0.00122166 max lat: 2.22402 avg lat: 0.010286 2024-09-21T09:51:52.280 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T09:51:52.280 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 60 16 93299 93283 97.1577 50.9375 0.028304 0.010286 2024-09-21T09:51:52.543 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:52 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/510562811' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:52.544 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:52 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/510562811' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:52.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:52 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/510562811' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:53.280 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 61 16 94136 94120 96.4224 52.3125 0.0133754 0.0103647 2024-09-21T09:51:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:53 smithi107 ceph-mon[102221]: from='client.54526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:53 smithi107 ceph-mon[102221]: pgmap v197: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 52 MiB/s wr, 824 op/s; 12098/266259 objects degraded (4.544%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T09:51:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:53 smithi107 ceph-mon[102221]: from='client.34782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:53 smithi107 ceph-mon[105022]: from='client.54526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:53 smithi107 ceph-mon[105022]: pgmap v197: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 52 MiB/s wr, 824 op/s; 12098/266259 objects degraded (4.544%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T09:51:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:53 smithi107 ceph-mon[105022]: from='client.34782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:53 smithi129 ceph-mon[88471]: from='client.54526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:53 smithi129 ceph-mon[88471]: pgmap v197: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 52 MiB/s wr, 824 op/s; 12098/266259 objects degraded (4.544%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T09:51:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:53 smithi129 ceph-mon[88471]: from='client.34782 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:54.280 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 62 16 94987 94971 95.725 53.1875 0.007705 0.0104394 2024-09-21T09:51:54.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:54 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 12098/266259 objects degraded (4.544%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:54.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:54 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 12098/266259 objects degraded (4.544%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:54 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 12098/266259 objects degraded (4.544%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:55.280 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 63 16 95846 95830 95.0577 53.6875 0.015141 0.0105141 2024-09-21T09:51:55.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:55 smithi107 ceph-mon[102221]: pgmap v198: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s; 11944/280053 objects degraded (4.265%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T09:51:55.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:55 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3627659151' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:55.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:55 smithi107 ceph-mon[105022]: pgmap v198: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s; 11944/280053 objects degraded (4.265%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T09:51:55.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:55 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3627659151' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:55.611 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:51:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:55 smithi129 ceph-mon[88471]: pgmap v198: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s; 11944/280053 objects degraded (4.265%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T09:51:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:55 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3627659151' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:56.280 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 64 16 96699 96683 94.4054 53.3125 0.00737988 0.0105862 2024-09-21T09:51:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:56 smithi107 ceph-mon[102221]: from='client.54541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:51:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:56 smithi107 ceph-mon[105022]: from='client.54541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:51:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:56 smithi129 ceph-mon[88471]: from='client.54541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:51:57.280 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 65 16 97650 97634 93.8673 59.4375 0.0109868 0.0106472 2024-09-21T09:51:57.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:57 smithi107 ceph-mon[102221]: pgmap v199: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 49 MiB/s wr, 787 op/s; 11944/280053 objects degraded (4.265%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T09:51:57.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:57 smithi107 ceph-mon[102221]: from='client.54544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:57.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:57 smithi107 ceph-mon[105022]: pgmap v199: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 49 MiB/s wr, 787 op/s; 11944/280053 objects degraded (4.265%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T09:51:57.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:57 smithi107 ceph-mon[105022]: from='client.54544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:57 smithi129 ceph-mon[88471]: pgmap v199: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 49 MiB/s wr, 787 op/s; 11944/280053 objects degraded (4.265%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T09:51:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:57 smithi129 ceph-mon[88471]: from='client.54544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:58.280 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 66 16 98592 98576 93.3371 58.875 0.0123186 0.0107068 2024-09-21T09:51:59.280 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 67 16 99423 99407 92.7191 51.9375 0.0373534 0.0107792 2024-09-21T09:51:59.295 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:51:59.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:59 smithi107 ceph-mon[102221]: pgmap v200: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s; 11818/292335 objects degraded (4.043%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T09:51:59.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:59 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2715279310' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:59.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:51:59 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 11818/292335 objects degraded (4.043%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:59.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:59 smithi107 ceph-mon[105022]: pgmap v200: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s; 11818/292335 objects degraded (4.043%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T09:51:59.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:59 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2715279310' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:59.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:51:59 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 11818/292335 objects degraded (4.043%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:51:59.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:59 smithi129 ceph-mon[88471]: pgmap v200: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s; 11818/292335 objects degraded (4.043%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T09:51:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:59 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2715279310' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:51:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:51:59 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 11818/292335 objects degraded (4.043%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:52:00.280 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 68 16 100298 100282 92.1598 54.6875 0.0155662 0.010845 2024-09-21T09:52:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:00 smithi107 ceph-mon[102221]: from='client.34803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:00.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:00 smithi107 ceph-mon[105022]: from='client.34803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:00 smithi129 ceph-mon[88471]: from='client.34803 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:52:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:52:00.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:52:01.280 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 69 16 101255 101239 91.691 59.8125 0.0225695 0.0109 2024-09-21T09:52:01.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:01 smithi107 ceph-mon[102221]: from='client.34806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:01.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:01 smithi107 ceph-mon[102221]: pgmap v201: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 48 MiB/s wr, 771 op/s; 11818/293193 objects degraded (4.031%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T09:52:01.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:01 smithi107 ceph-mon[105022]: from='client.34806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:01.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:01 smithi107 ceph-mon[105022]: pgmap v201: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 48 MiB/s wr, 771 op/s; 11818/293193 objects degraded (4.031%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T09:52:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:01 smithi129 ceph-mon[88471]: from='client.34806 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:01 smithi129 ceph-mon[88471]: pgmap v201: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 48 MiB/s wr, 771 op/s; 11818/293193 objects degraded (4.031%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T09:52:02.088 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:52:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:52:01] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-09-21T09:52:02.281 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 70 16 102135 102119 91.1667 55 0.00859993 0.010963 2024-09-21T09:52:02.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:02 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:52:02.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:02 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:52:02.359 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:52:02 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:52:02.085+0000 7efdb5a17640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-21T09:52:02.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:02 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:52:02.965 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:52:03.281 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 71 16 102941 102925 90.5921 50.375 0.0152629 0.0110322 2024-09-21T09:52:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:03 smithi107 ceph-mon[102221]: pgmap v202: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 47 MiB/s wr, 747 op/s; 11818/293193 objects degraded (4.031%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T09:52:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:03 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:52:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:03 smithi107 ceph-mon[102221]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-21T09:52:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:03 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3912601084' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:03.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:03 smithi107 ceph-mon[105022]: pgmap v202: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 47 MiB/s wr, 747 op/s; 11818/293193 objects degraded (4.031%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T09:52:03.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:03 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:52:03.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:03 smithi107 ceph-mon[105022]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-21T09:52:03.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:03 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3912601084' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:03 smithi129 ceph-mon[88471]: pgmap v202: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 47 MiB/s wr, 747 op/s; 11818/293193 objects degraded (4.031%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T09:52:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:03 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:52:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:03 smithi129 ceph-mon[88471]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-21T09:52:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:03 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3912601084' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:04.281 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 72 16 103814 103798 90.0917 54.5625 0.0267725 0.011094 2024-09-21T09:52:04.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:04 smithi107 ceph-mon[102221]: from='client.44648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:04.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:04 smithi107 ceph-mon[102221]: from='client.34821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:04.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:04 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 11818/293193 objects degraded (4.031%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:52:04.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:04 smithi107 ceph-mon[105022]: from='client.44648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:04.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:04 smithi107 ceph-mon[105022]: from='client.34821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:04.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:04 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 11818/293193 objects degraded (4.031%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:52:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:04 smithi129 ceph-mon[88471]: from='client.44648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:04 smithi129 ceph-mon[88471]: from='client.34821 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:04 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 11818/293193 objects degraded (4.031%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:52:05.281 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 73 16 104729 104713 89.6409 57.1875 0.00732294 0.0111501 2024-09-21T09:52:05.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:05 smithi107 ceph-mon[102221]: pgmap v203: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 11677/306495 objects degraded (3.810%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T09:52:05.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:05 smithi107 ceph-mon[105022]: pgmap v203: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 11677/306495 objects degraded (3.810%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T09:52:05.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:05 smithi129 ceph-mon[88471]: pgmap v203: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 11677/306495 objects degraded (3.810%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T09:52:06.281 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 74 16 105546 105530 89.1195 51.0625 0.0142243 0.0112147 2024-09-21T09:52:06.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:06 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2850983223' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:06.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:06 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2850983223' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:06.623 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:52:06.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:06 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2850983223' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:07.281 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 75 16 106342 106326 88.5945 49.75 0.0140546 0.0112814 2024-09-21T09:52:07.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:07 smithi107 ceph-mon[102221]: pgmap v204: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 46 MiB/s wr, 734 op/s; 11677/306495 objects degraded (3.810%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T09:52:07.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:07 smithi107 ceph-mon[102221]: from='client.44660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:07.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:07 smithi107 ceph-mon[105022]: pgmap v204: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 46 MiB/s wr, 734 op/s; 11677/306495 objects degraded (3.810%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T09:52:07.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:07 smithi107 ceph-mon[105022]: from='client.44660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:07.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:07 smithi129 ceph-mon[88471]: pgmap v204: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 46 MiB/s wr, 734 op/s; 11677/306495 objects degraded (3.810%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T09:52:07.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:07 smithi129 ceph-mon[88471]: from='client.44660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:08.281 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 76 16 107260 107244 88.1837 57.375 0.00774152 0.0113347 2024-09-21T09:52:08.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:08 smithi107 ceph-mon[102221]: from='client.34830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:08.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:08 smithi107 ceph-mon[105022]: from='client.34830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:08.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:08 smithi129 ceph-mon[88471]: from='client.34830 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:09.281 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 77 15 108136 108121 87.7502 54.8125 0.0194813 0.0113903 2024-09-21T09:52:09.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:09 smithi107 ceph-mon[102221]: pgmap v205: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 11520/318450 objects degraded (3.618%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T09:52:09.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:09 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 11520/318450 objects degraded (3.618%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:52:09.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:09 smithi107 ceph-mon[105022]: pgmap v205: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 11520/318450 objects degraded (3.618%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T09:52:09.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:09 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 11520/318450 objects degraded (3.618%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:52:09.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:09 smithi129 ceph-mon[88471]: pgmap v205: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 11520/318450 objects degraded (3.618%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T09:52:09.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:09 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 11520/318450 objects degraded (3.618%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:52:10.281 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 78 16 108911 108895 87.2454 48.375 0.0180136 0.011456 2024-09-21T09:52:10.282 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:52:10.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:10 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3302434591' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:10.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:10 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3302434591' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:10.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:10 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3302434591' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:10.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:52:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:52:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:52:11.281 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 79 16 109731 109715 86.7897 51.25 0.017035 0.0115167 2024-09-21T09:52:11.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:11 smithi129 ceph-mon[88471]: pgmap v206: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 47 MiB/s wr, 749 op/s; 11520/319308 objects degraded (3.608%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T09:52:11.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:11 smithi129 ceph-mon[88471]: from='client.34842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:11.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:52:11.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:11 smithi107 ceph-mon[102221]: pgmap v206: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 47 MiB/s wr, 749 op/s; 11520/319308 objects degraded (3.608%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T09:52:11.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:11 smithi107 ceph-mon[102221]: from='client.34842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:11.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:52:11.768 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:11 smithi107 ceph-mon[105022]: pgmap v206: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 47 MiB/s wr, 749 op/s; 11520/319308 objects degraded (3.608%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T09:52:11.768 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:11 smithi107 ceph-mon[105022]: from='client.34842 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:11.768 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:52:12.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:52:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:52:11] "GET /metrics HTTP/1.1" 200 36536 "" "Prometheus/2.51.0" 2024-09-21T09:52:12.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T09:52:12.280831+0000 min lat: 0.00122166 max lat: 2.22402 avg lat: 0.0115689 2024-09-21T09:52:12.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T09:52:12.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 80 16 110615 110599 86.3954 55.25 0.0333662 0.0115689 2024-09-21T09:52:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:12 smithi129 ceph-mon[88471]: from='client.54586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:12.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:12 smithi107 ceph-mon[102221]: from='client.54586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:12.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:12 smithi107 ceph-mon[105022]: from='client.54586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:13.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 81 16 111451 111435 85.9738 52.25 0.0423764 0.011625 2024-09-21T09:52:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:13 smithi129 ceph-mon[88471]: pgmap v207: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 45 MiB/s wr, 725 op/s; 11520/319308 objects degraded (3.608%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T09:52:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:13 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/736572214' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:13.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:13 smithi107 ceph-mon[102221]: pgmap v207: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 45 MiB/s wr, 725 op/s; 11520/319308 objects degraded (3.608%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T09:52:13.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:13 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/736572214' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:13.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:13 smithi107 ceph-mon[105022]: pgmap v207: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 45 MiB/s wr, 725 op/s; 11520/319308 objects degraded (3.608%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T09:52:13.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:13 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/736572214' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:13.946 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:52:14.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 82 16 112214 112198 85.5069 47.6875 0.00659016 0.0116893 2024-09-21T09:52:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:14 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 11520/319308 objects degraded (3.608%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:52:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:14 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 11520/319308 objects degraded (3.608%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:52:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:14 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 11520/319308 objects degraded (3.608%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:52:15.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 83 16 113063 113047 85.1159 53.0625 0.022693 0.0117424 2024-09-21T09:52:15.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:15 smithi129 ceph-mon[88471]: from='client.34860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:15.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:15 smithi129 ceph-mon[88471]: pgmap v208: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s; 11372/332073 objects degraded (3.425%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T09:52:15.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:15 smithi129 ceph-mon[88471]: from='client.44687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:15.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:15 smithi107 ceph-mon[102221]: from='client.34860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:15.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:15 smithi107 ceph-mon[102221]: pgmap v208: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s; 11372/332073 objects degraded (3.425%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T09:52:15.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:15 smithi107 ceph-mon[102221]: from='client.44687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:15.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:15 smithi107 ceph-mon[105022]: from='client.34860 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:15.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:15 smithi107 ceph-mon[105022]: pgmap v208: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.08k op/s; 11372/332073 objects degraded (3.425%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T09:52:15.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:15 smithi107 ceph-mon[105022]: from='client.44687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:16.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 84 16 113934 113918 84.7507 54.4375 0.0102118 0.0117932 2024-09-21T09:52:17.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 85 16 114729 114713 84.3381 49.6875 0.021984 0.0118516 2024-09-21T09:52:17.359 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:52:17 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:52:17.104+0000 7efdb5a17640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-21T09:52:17.605 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:52:17.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:17 smithi129 ceph-mon[88471]: pgmap v209: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 44 MiB/s wr, 710 op/s; 11372/332073 objects degraded (3.425%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T09:52:17.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:17 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/408337164' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:17.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:17 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:52:17.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:17 smithi107 ceph-mon[102221]: pgmap v209: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 44 MiB/s wr, 710 op/s; 11372/332073 objects degraded (3.425%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T09:52:17.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:17 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/408337164' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:17.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:17 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:52:17.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:17 smithi107 ceph-mon[105022]: pgmap v209: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 44 MiB/s wr, 710 op/s; 11372/332073 objects degraded (3.425%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T09:52:17.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:17 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/408337164' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:17.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:17 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:52:18.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 86 16 115492 115476 83.9119 47.6875 0.00853717 0.0119116 2024-09-21T09:52:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:18 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:52:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:18 smithi129 ceph-mon[88471]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-21T09:52:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:18 smithi129 ceph-mon[88471]: from='client.44696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:18.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:18 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:52:18.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:18 smithi107 ceph-mon[102221]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-21T09:52:18.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:18 smithi107 ceph-mon[102221]: from='client.44696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:18.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:18 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:52:18.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:18 smithi107 ceph-mon[105022]: Upgrade: unsafe to stop osd(s) at this time (3 PGs are or would become offline) 2024-09-21T09:52:18.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:18 smithi107 ceph-mon[105022]: from='client.44696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:19.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 87 16 116351 116335 83.5645 53.6875 0.0217318 0.011961 2024-09-21T09:52:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:19 smithi129 ceph-mon[88471]: pgmap v210: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 11216/343584 objects degraded (3.264%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T09:52:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:19 smithi129 ceph-mon[88471]: from='client.54607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:19 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 11216/343584 objects degraded (3.264%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:52:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:19 smithi107 ceph-mon[102221]: pgmap v210: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 11216/343584 objects degraded (3.264%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T09:52:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:19 smithi107 ceph-mon[102221]: from='client.54607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:19 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 11216/343584 objects degraded (3.264%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:52:19.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:19 smithi107 ceph-mon[105022]: pgmap v210: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 11216/343584 objects degraded (3.264%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T09:52:19.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:19 smithi107 ceph-mon[105022]: from='client.54607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:19.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:19 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 11216/343584 objects degraded (3.264%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:52:20.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 88 16 117198 117182 83.2164 52.9375 0.0210737 0.0120108 2024-09-21T09:52:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:52:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:52:20.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:52:21.265 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:52:21.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 89 16 117946 117930 82.8066 46.75 0.0100373 0.0120712 2024-09-21T09:52:21.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:21 smithi107 ceph-mon[102221]: pgmap v211: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 45 MiB/s wr, 719 op/s; 11216/344352 objects degraded (3.257%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T09:52:21.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:21 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3467915620' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:21.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:21 smithi107 ceph-mon[105022]: pgmap v211: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 45 MiB/s wr, 719 op/s; 11216/344352 objects degraded (3.257%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T09:52:21.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:21 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3467915620' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:21 smithi129 ceph-mon[88471]: pgmap v211: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 45 MiB/s wr, 719 op/s; 11216/344352 objects degraded (3.257%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T09:52:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:21 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3467915620' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:22.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:52:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:52:21] "GET /metrics HTTP/1.1" 200 36536 "" "Prometheus/2.51.0" 2024-09-21T09:52:22.282 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 90 4 118704 118700 82.4212 48.125 0.029952 0.0121276 2024-09-21T09:52:22.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:22 smithi107 ceph-mon[102221]: from='client.34887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:22.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:22 smithi107 ceph-mon[105022]: from='client.34887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:22 smithi129 ceph-mon[88471]: from='client.34887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:23.283 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total time run: 90.0123 2024-09-21T09:52:23.283 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total writes made: 118704 2024-09-21T09:52:23.283 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Write size: 65536 2024-09-21T09:52:23.283 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object size: 65536 2024-09-21T09:52:23.283 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Bandwidth (MB/sec): 82.4221 2024-09-21T09:52:23.284 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Bandwidth: 42.9081 2024-09-21T09:52:23.284 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max bandwidth (MB/sec): 214.375 2024-09-21T09:52:23.284 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min bandwidth (MB/sec): 0 2024-09-21T09:52:23.284 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average IOPS: 1318 2024-09-21T09:52:23.284 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev IOPS: 686.53 2024-09-21T09:52:23.284 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max IOPS: 3430 2024-09-21T09:52:23.285 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min IOPS: 0 2024-09-21T09:52:23.285 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average Latency(s): 0.0121277 2024-09-21T09:52:23.285 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Latency(s): 0.0285824 2024-09-21T09:52:23.285 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max latency(s): 2.22402 2024-09-21T09:52:23.285 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min latency(s): 0.00122166 2024-09-21T09:52:23.291 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T09:52:23.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:23 smithi107 ceph-mon[102221]: from='client.54610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:23.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:23 smithi107 ceph-mon[102221]: pgmap v212: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 43 MiB/s wr, 695 op/s; 11216/344352 objects degraded (3.257%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T09:52:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:23 smithi107 ceph-mon[105022]: from='client.54610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:23.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:23 smithi107 ceph-mon[105022]: pgmap v212: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 43 MiB/s wr, 695 op/s; 11216/344352 objects degraded (3.257%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T09:52:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:23 smithi129 ceph-mon[88471]: from='client.54610 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:23 smithi129 ceph-mon[88471]: pgmap v212: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 43 MiB/s wr, 695 op/s; 11216/344352 objects degraded (3.257%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T09:52:24.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:24 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 11216/344352 objects degraded (3.257%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:52:24.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:24 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/203947534' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:24.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:24 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 11216/344352 objects degraded (3.257%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:52:24.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:24 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/203947534' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:24.885 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:52:24.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:24 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 11216/344352 objects degraded (3.257%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:52:24.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:24 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/203947534' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:25 smithi107 ceph-mon[102221]: pgmap v213: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 63 MiB/s wr, 1.02k op/s; 10908/355842 objects degraded (3.065%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T09:52:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:25 smithi107 ceph-mon[105022]: pgmap v213: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 63 MiB/s wr, 1.02k op/s; 10908/355842 objects degraded (3.065%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T09:52:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:25 smithi129 ceph-mon[88471]: pgmap v213: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 63 MiB/s wr, 1.02k op/s; 10908/355842 objects degraded (3.065%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T09:52:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:26 smithi107 ceph-mon[102221]: from='client.54616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:26 smithi107 ceph-mon[102221]: from='client.54622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:52:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:26 smithi107 ceph-mon[105022]: from='client.54616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:26 smithi107 ceph-mon[105022]: from='client.54622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:52:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:26 smithi129 ceph-mon[88471]: from='client.54616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:26 smithi129 ceph-mon[88471]: from='client.54622 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:52:27.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:27 smithi107 ceph-mon[102221]: pgmap v214: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 41 MiB/s wr, 662 op/s; 10908/355842 objects degraded (3.065%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T09:52:27.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:27 smithi107 ceph-mon[105022]: pgmap v214: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 41 MiB/s wr, 662 op/s; 10908/355842 objects degraded (3.065%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T09:52:27.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:27 smithi129 ceph-mon[88471]: pgmap v214: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 41 MiB/s wr, 662 op/s; 10908/355842 objects degraded (3.065%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T09:52:28.512 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:52:28.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:28 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4150262' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:28.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:28 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4150262' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:28.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:28 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4150262' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:29.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:29 smithi107 ceph-mon[102221]: pgmap v215: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 85 B/s rd, 42 MiB/s wr, 930 op/s; 10594/346812 objects degraded (3.055%); 4.0 MiB/s, 64 objects/s recovering 2024-09-21T09:52:29.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:29 smithi107 ceph-mon[102221]: from='client.54631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:29.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:29 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 10594/346812 objects degraded (3.055%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:52:29.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:29 smithi107 ceph-mon[105022]: pgmap v215: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 85 B/s rd, 42 MiB/s wr, 930 op/s; 10594/346812 objects degraded (3.055%); 4.0 MiB/s, 64 objects/s recovering 2024-09-21T09:52:29.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:29 smithi107 ceph-mon[105022]: from='client.54631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:29.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:29 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 10594/346812 objects degraded (3.055%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:52:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:29 smithi129 ceph-mon[88471]: pgmap v215: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 85 B/s rd, 42 MiB/s wr, 930 op/s; 10594/346812 objects degraded (3.055%); 4.0 MiB/s, 64 objects/s recovering 2024-09-21T09:52:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:29 smithi129 ceph-mon[88471]: from='client.54631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:29 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 10594/346812 objects degraded (3.055%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:52:30.498 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:52:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:52:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:52:30.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:30 smithi107 ceph-mon[102221]: from='client.44729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:30 smithi107 ceph-mon[105022]: from='client.44729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:30.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:30 smithi129 ceph-mon[88471]: from='client.44729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:31.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:31 smithi107 ceph-mon[102221]: pgmap v216: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 85 B/s rd, 22 MiB/s wr, 631 op/s; 10594/346074 objects degraded (3.061%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T09:52:31.768 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:31 smithi107 ceph-mon[105022]: pgmap v216: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 85 B/s rd, 22 MiB/s wr, 631 op/s; 10594/346074 objects degraded (3.061%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T09:52:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:31 smithi129 ceph-mon[88471]: pgmap v216: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 85 B/s rd, 22 MiB/s wr, 631 op/s; 10594/346074 objects degraded (3.061%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T09:52:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:52:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:52:31] "GET /metrics HTTP/1.1" 200 36536 "" "Prometheus/2.51.0" 2024-09-21T09:52:32.183 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:52:32.854 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:32 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2600407800' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:32.854 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:32 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:52:32.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:32 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2600407800' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:32.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:32 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:52:32.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:32 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2600407800' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:32.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:32 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:52:33.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:33 smithi107 ceph-mon[102221]: pgmap v217: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 85 B/s rd, 21 MiB/s wr, 609 op/s; 10594/346074 objects degraded (3.061%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T09:52:33.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:33 smithi107 ceph-mon[102221]: from='client.44741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:33.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:33 smithi107 ceph-mon[102221]: osdmap e108: 8 total, 8 up, 8 in 2024-09-21T09:52:33.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:33 smithi107 ceph-mon[105022]: pgmap v217: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 85 B/s rd, 21 MiB/s wr, 609 op/s; 10594/346074 objects degraded (3.061%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T09:52:33.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:33 smithi107 ceph-mon[105022]: from='client.44741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:33.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:33 smithi107 ceph-mon[105022]: osdmap e108: 8 total, 8 up, 8 in 2024-09-21T09:52:33.899 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:33 smithi129 ceph-mon[88471]: pgmap v217: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 85 B/s rd, 21 MiB/s wr, 609 op/s; 10594/346074 objects degraded (3.061%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T09:52:33.899 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:33 smithi129 ceph-mon[88471]: from='client.44741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:33.899 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:33 smithi129 ceph-mon[88471]: osdmap e108: 8 total, 8 up, 8 in 2024-09-21T09:52:34.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:34 smithi107 ceph-mon[105022]: from='client.54643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:34.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:34 smithi107 ceph-mon[105022]: osdmap e109: 8 total, 8 up, 8 in 2024-09-21T09:52:34.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:34 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 10594/346074 objects degraded (3.061%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T09:52:34.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:34 smithi107 ceph-mon[102221]: from='client.54643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:34.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:34 smithi107 ceph-mon[102221]: osdmap e109: 8 total, 8 up, 8 in 2024-09-21T09:52:34.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:34 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 10594/346074 objects degraded (3.061%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T09:52:34.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:34 smithi129 ceph-mon[88471]: from='client.54643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:34.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:34 smithi129 ceph-mon[88471]: osdmap e109: 8 total, 8 up, 8 in 2024-09-21T09:52:34.896 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:34 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 10594/346074 objects degraded (3.061%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-09-21T09:52:35.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:35 smithi107 ceph-mon[102221]: pgmap v220: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 127 B/s rd, 800 KiB/s wr, 924 op/s; 10434/334257 objects degraded (3.122%); 3.7 MiB/s, 59 objects/s recovering 2024-09-21T09:52:35.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:35 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3396665439' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:35.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:35 smithi107 ceph-mon[105022]: pgmap v220: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 127 B/s rd, 800 KiB/s wr, 924 op/s; 10434/334257 objects degraded (3.122%); 3.7 MiB/s, 59 objects/s recovering 2024-09-21T09:52:35.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:35 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3396665439' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:35.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:35 smithi129 ceph-mon[88471]: pgmap v220: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 127 B/s rd, 800 KiB/s wr, 924 op/s; 10434/334257 objects degraded (3.122%); 3.7 MiB/s, 59 objects/s recovering 2024-09-21T09:52:35.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:35 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3396665439' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:35.827 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:52:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:37 smithi107 ceph-mon[102221]: from='client.54652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:37 smithi107 ceph-mon[102221]: pgmap v221: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 522 op/s; 10434/334257 objects degraded (3.122%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T09:52:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:37 smithi107 ceph-mon[102221]: from='client.44756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:37 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:52:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:37 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:52:37.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:37 smithi107 ceph-mon[105022]: from='client.54652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:37.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:37 smithi107 ceph-mon[105022]: pgmap v221: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 522 op/s; 10434/334257 objects degraded (3.122%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T09:52:37.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:37 smithi107 ceph-mon[105022]: from='client.44756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:37.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:37 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:52:37.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:37 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:52:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:37 smithi129 ceph-mon[88471]: from='client.54652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:37 smithi129 ceph-mon[88471]: pgmap v221: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 522 op/s; 10434/334257 objects degraded (3.122%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T09:52:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:37 smithi129 ceph-mon[88471]: from='client.44756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:37 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:52:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:37 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:52:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:52:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:52:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:52:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:52:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:52:38.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:52:38.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:52:38.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:52:38.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:52:38.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:52:38.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:52:38.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:52:38.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:52:38.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:52:38.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:52:38.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:52:38.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:52:38.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:52:38.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:52:38.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:52:38.861 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:52:38 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:52:38.381+0000 7efdb5a17640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:52:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:52:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:52:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:52:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:52:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:52:38.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:52:38.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:52:38.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:52:38.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:52:38.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:52:39.462 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:52:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:39 smithi107 ceph-mon[102221]: pgmap v222: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 845 op/s; 10063/325767 objects degraded (3.089%); 4.1 MiB/s, 66 objects/s recovering 2024-09-21T09:52:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:39 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:52:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:39 smithi107 ceph-mon[102221]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:52:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:39 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3537482436' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:39 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 10063/325767 objects degraded (3.089%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:52:39.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:39 smithi107 ceph-mon[105022]: pgmap v222: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 845 op/s; 10063/325767 objects degraded (3.089%); 4.1 MiB/s, 66 objects/s recovering 2024-09-21T09:52:39.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:39 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:52:39.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:39 smithi107 ceph-mon[105022]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:52:39.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:39 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3537482436' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:39.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:39 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 10063/325767 objects degraded (3.089%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:52:39.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:39 smithi129 ceph-mon[88471]: pgmap v222: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 845 op/s; 10063/325767 objects degraded (3.089%); 4.1 MiB/s, 66 objects/s recovering 2024-09-21T09:52:39.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:39 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:52:39.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:39 smithi129 ceph-mon[88471]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:52:39.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:39 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3537482436' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:39.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:39 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 10063/325767 objects degraded (3.089%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:52:40.447 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:52:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:52:40.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:52:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:40 smithi129 ceph-mon[88471]: from='client.34935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:40 smithi107 ceph-mon[102221]: from='client.34935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:41.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:40 smithi107 ceph-mon[105022]: from='client.34935 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:41 smithi129 ceph-mon[88471]: pgmap v223: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 874 op/s; 10063/325074 objects degraded (3.096%); 4.1 MiB/s, 66 objects/s recovering 2024-09-21T09:52:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:41 smithi129 ceph-mon[88471]: from='client.54673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:52:42.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:41 smithi107 ceph-mon[102221]: pgmap v223: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 874 op/s; 10063/325074 objects degraded (3.096%); 4.1 MiB/s, 66 objects/s recovering 2024-09-21T09:52:42.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:41 smithi107 ceph-mon[102221]: from='client.54673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:42.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:52:42.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:41 smithi107 ceph-mon[105022]: pgmap v223: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 874 op/s; 10063/325074 objects degraded (3.096%); 4.1 MiB/s, 66 objects/s recovering 2024-09-21T09:52:42.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:41 smithi107 ceph-mon[105022]: from='client.54673 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:42.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:52:42.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:52:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:52:41] "GET /metrics HTTP/1.1" 200 36537 "" "Prometheus/2.51.0" 2024-09-21T09:52:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:42 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3380779354' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:43.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:42 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3380779354' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:43.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:42 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3380779354' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:43.115 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:52:44.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:43 smithi107 ceph-mon[102221]: pgmap v224: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 421 op/s; 10063/325074 objects degraded (3.096%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T09:52:44.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:43 smithi107 ceph-mon[105022]: pgmap v224: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 421 op/s; 10063/325074 objects degraded (3.096%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T09:52:44.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:43 smithi129 ceph-mon[88471]: pgmap v224: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 421 op/s; 10063/325074 objects degraded (3.096%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T09:52:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:44 smithi107 ceph-mon[102221]: from='client.34941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:44 smithi107 ceph-mon[102221]: from='client.34944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:44 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 10063/325074 objects degraded (3.096%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:52:45.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:44 smithi107 ceph-mon[105022]: from='client.34941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:45.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:44 smithi107 ceph-mon[105022]: from='client.34944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:45.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:44 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 10063/325074 objects degraded (3.096%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:52:45.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:44 smithi129 ceph-mon[88471]: from='client.34941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:45.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:44 smithi129 ceph-mon[88471]: from='client.34944 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:45.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:44 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 10063/325074 objects degraded (3.096%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:52:46.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:45 smithi107 ceph-mon[102221]: pgmap v225: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 685 op/s; 9796/313398 objects degraded (3.126%); 3.8 MiB/s, 61 objects/s recovering 2024-09-21T09:52:47.126 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:45 smithi107 ceph-mon[105022]: pgmap v225: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 685 op/s; 9796/313398 objects degraded (3.126%); 3.8 MiB/s, 61 objects/s recovering 2024-09-21T09:52:47.127 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:52:47.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:46 smithi107 ceph-mon[102221]: pgmap v226: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 579 op/s; 9796/313398 objects degraded (3.126%); 3.3 MiB/s, 53 objects/s recovering 2024-09-21T09:52:47.127 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:46 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2350974071' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:47.127 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:46 smithi107 ceph-mon[105022]: pgmap v226: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 579 op/s; 9796/313398 objects degraded (3.126%); 3.3 MiB/s, 53 objects/s recovering 2024-09-21T09:52:47.128 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:46 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2350974071' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:47.128 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:45 smithi129 ceph-mon[88471]: pgmap v225: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 685 op/s; 9796/313398 objects degraded (3.126%); 3.8 MiB/s, 61 objects/s recovering 2024-09-21T09:52:47.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:46 smithi129 ceph-mon[88471]: pgmap v226: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 579 op/s; 9796/313398 objects degraded (3.126%); 3.3 MiB/s, 53 objects/s recovering 2024-09-21T09:52:47.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:46 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2350974071' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:48.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:47 smithi107 ceph-mon[102221]: from='client.54682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:48.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:47 smithi107 ceph-mon[102221]: from='client.44789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:48.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:47 smithi107 ceph-mon[105022]: from='client.54682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:48.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:47 smithi107 ceph-mon[105022]: from='client.44789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:48.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:47 smithi129 ceph-mon[88471]: from='client.54682 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:48.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:47 smithi129 ceph-mon[88471]: from='client.44789 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:49.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:48 smithi107 ceph-mon[102221]: pgmap v227: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 9516/302613 objects degraded (3.145%); 4.8 MiB/s, 76 objects/s recovering 2024-09-21T09:52:49.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:48 smithi107 ceph-mon[105022]: pgmap v227: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 9516/302613 objects degraded (3.145%); 4.8 MiB/s, 76 objects/s recovering 2024-09-21T09:52:49.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:48 smithi129 ceph-mon[88471]: pgmap v227: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 9516/302613 objects degraded (3.145%); 4.8 MiB/s, 76 objects/s recovering 2024-09-21T09:52:50.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:49 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 9516/302613 objects degraded (3.145%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:52:50.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:49 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 9516/302613 objects degraded (3.145%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:52:50.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:49 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 9516/302613 objects degraded (3.145%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:52:50.379 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:52:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:52:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:52:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:52:51.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:50 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2002572472' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:51.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:50 smithi107 ceph-mon[102221]: pgmap v228: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 662 op/s; 9516/301899 objects degraded (3.152%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T09:52:51.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:50 smithi107 ceph-mon[102221]: from='client.44795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:51.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:50 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2002572472' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:51.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:50 smithi107 ceph-mon[105022]: pgmap v228: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 662 op/s; 9516/301899 objects degraded (3.152%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T09:52:51.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:50 smithi107 ceph-mon[105022]: from='client.44795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:51.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:50 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2002572472' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:51.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:50 smithi129 ceph-mon[88471]: pgmap v228: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 662 op/s; 9516/301899 objects degraded (3.152%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T09:52:51.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:50 smithi129 ceph-mon[88471]: from='client.44795 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:51.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:51 smithi129 ceph-mon[88471]: from='client.54697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:52.010 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:51 smithi107 ceph-mon[102221]: from='client.54697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:52.011 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:52:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:52:51] "GET /metrics HTTP/1.1" 200 36537 "" "Prometheus/2.51.0" 2024-09-21T09:52:52.011 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:51 smithi107 ceph-mon[105022]: from='client.54697 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:53.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:52 smithi107 ceph-mon[102221]: pgmap v229: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 643 op/s; 9516/301899 objects degraded (3.152%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T09:52:53.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:52 smithi107 ceph-mon[105022]: pgmap v229: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 643 op/s; 9516/301899 objects degraded (3.152%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T09:52:53.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:52 smithi129 ceph-mon[88471]: pgmap v229: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 643 op/s; 9516/301899 objects degraded (3.152%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T09:52:53.759 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:52:53 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:52:53.400+0000 7efdb5a17640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:52:54.006 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:52:54.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:53 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:52:54.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:53 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:52:54.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:53 smithi107 ceph-mon[102221]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:52:54.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:53 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1135001375' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:54.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:53 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:52:54.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:53 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:52:54.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:53 smithi107 ceph-mon[105022]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:52:54.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:53 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1135001375' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:54.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:53 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:52:54.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:53 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:52:54.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:53 smithi129 ceph-mon[88471]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:52:54.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:53 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1135001375' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:55.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:54 smithi107 ceph-mon[102221]: pgmap v230: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 965 op/s; 9220/290289 objects degraded (3.176%); 4.4 MiB/s, 70 objects/s recovering 2024-09-21T09:52:55.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:54 smithi107 ceph-mon[102221]: from='client.44804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:55.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:54 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 9516/301899 objects degraded (3.152%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:52:55.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:54 smithi107 ceph-mon[102221]: from='client.34980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:55.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:54 smithi107 ceph-mon[105022]: pgmap v230: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 965 op/s; 9220/290289 objects degraded (3.176%); 4.4 MiB/s, 70 objects/s recovering 2024-09-21T09:52:55.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:54 smithi107 ceph-mon[105022]: from='client.44804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:55.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:54 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 9516/301899 objects degraded (3.152%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:52:55.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:54 smithi107 ceph-mon[105022]: from='client.34980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:54 smithi129 ceph-mon[88471]: pgmap v230: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 965 op/s; 9220/290289 objects degraded (3.176%); 4.4 MiB/s, 70 objects/s recovering 2024-09-21T09:52:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:54 smithi129 ceph-mon[88471]: from='client.44804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:54 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 9516/301899 objects degraded (3.152%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:52:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:54 smithi129 ceph-mon[88471]: from='client.34980 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:57.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:57 smithi107 ceph-mon[102221]: pgmap v231: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 641 op/s; 9220/290289 objects degraded (3.176%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T09:52:57.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:57 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:52:57.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:57 smithi107 ceph-mon[105022]: pgmap v231: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 641 op/s; 9220/290289 objects degraded (3.176%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T09:52:57.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:57 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:52:57.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:57 smithi129 ceph-mon[88471]: pgmap v231: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 641 op/s; 9220/290289 objects degraded (3.176%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T09:52:57.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:57 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:52:57.625 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:52:58.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:58 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/27359157' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:58.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:58 smithi107 ceph-mon[102221]: from='client.44816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:58.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:58 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/27359157' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:58.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:58 smithi107 ceph-mon[105022]: from='client.44816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:58.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:58 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/27359157' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:52:58.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:58 smithi129 ceph-mon[88471]: from='client.44816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:59.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:59 smithi107 ceph-mon[102221]: pgmap v232: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 924 op/s; 8919/280092 objects degraded (3.184%); 4.6 MiB/s, 73 objects/s recovering 2024-09-21T09:52:59.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:52:59 smithi107 ceph-mon[102221]: from='client.54724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:59.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:59 smithi107 ceph-mon[105022]: pgmap v232: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 924 op/s; 8919/280092 objects degraded (3.184%); 4.6 MiB/s, 73 objects/s recovering 2024-09-21T09:52:59.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:52:59 smithi107 ceph-mon[105022]: from='client.54724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:59.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:59 smithi129 ceph-mon[88471]: pgmap v232: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 924 op/s; 8919/280092 objects degraded (3.184%); 4.6 MiB/s, 73 objects/s recovering 2024-09-21T09:52:59.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:52:59 smithi129 ceph-mon[88471]: from='client.54724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:00.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:00 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 8919/280092 objects degraded (3.184%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:00.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:00 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 8919/280092 objects degraded (3.184%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:00.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:00 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 8919/280092 objects degraded (3.184%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:00.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:53:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:53:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:53:01.252 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:53:01.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:01 smithi107 ceph-mon[102221]: pgmap v233: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 644 op/s; 8919/279390 objects degraded (3.192%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T09:53:01.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:01 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3092123664' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:01.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:01 smithi107 ceph-mon[105022]: pgmap v233: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 644 op/s; 8919/279390 objects degraded (3.192%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T09:53:01.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:01 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3092123664' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:01.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:01 smithi129 ceph-mon[88471]: pgmap v233: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 644 op/s; 8919/279390 objects degraded (3.192%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T09:53:01.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:01 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3092123664' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:02.045 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:53:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:53:01] "GET /metrics HTTP/1.1" 200 36529 "" "Prometheus/2.51.0" 2024-09-21T09:53:02.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:02 smithi107 ceph-mon[102221]: from='client.54736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:02.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:02 smithi107 ceph-mon[105022]: from='client.54736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:02.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:02 smithi129 ceph-mon[88471]: from='client.54736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:03.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:03 smithi107 ceph-mon[102221]: from='client.54742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:03.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:03 smithi107 ceph-mon[102221]: pgmap v234: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 625 op/s; 8919/279390 objects degraded (3.192%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T09:53:03.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:03 smithi107 ceph-mon[105022]: from='client.54742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:03.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:03 smithi107 ceph-mon[105022]: pgmap v234: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 625 op/s; 8919/279390 objects degraded (3.192%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T09:53:03.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:03 smithi129 ceph-mon[88471]: from='client.54742 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:03.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:03 smithi129 ceph-mon[88471]: pgmap v234: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 625 op/s; 8919/279390 objects degraded (3.192%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T09:53:04.902 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:53:05.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:05 smithi107 ceph-mon[102221]: pgmap v235: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 935 op/s; 8570/268221 objects degraded (3.195%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T09:53:05.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:05 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 8570/268221 objects degraded (3.195%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:05.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:05 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/730789108' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:05.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:05 smithi107 ceph-mon[105022]: pgmap v235: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 935 op/s; 8570/268221 objects degraded (3.195%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T09:53:05.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:05 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 8570/268221 objects degraded (3.195%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:05.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:05 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/730789108' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:05.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:05 smithi129 ceph-mon[88471]: pgmap v235: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 935 op/s; 8570/268221 objects degraded (3.195%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T09:53:05.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:05 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 8570/268221 objects degraded (3.195%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:05.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:05 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/730789108' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:06.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:06 smithi107 ceph-mon[102221]: from='client.54754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:06.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:06 smithi107 ceph-mon[102221]: from='client.35010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:06.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:06 smithi107 ceph-mon[105022]: from='client.54754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:06.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:06 smithi107 ceph-mon[105022]: from='client.35010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:06.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:06 smithi129 ceph-mon[88471]: from='client.54754 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:06.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:06 smithi129 ceph-mon[88471]: from='client.35010 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:07.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:07 smithi107 ceph-mon[102221]: pgmap v236: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 612 op/s; 8570/268221 objects degraded (3.195%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T09:53:07.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:07 smithi107 ceph-mon[102221]: osdmap e110: 8 total, 8 up, 8 in 2024-09-21T09:53:07.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:07 smithi107 ceph-mon[105022]: pgmap v236: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 612 op/s; 8570/268221 objects degraded (3.195%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T09:53:07.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:07 smithi107 ceph-mon[105022]: osdmap e110: 8 total, 8 up, 8 in 2024-09-21T09:53:07.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:07 smithi129 ceph-mon[88471]: pgmap v236: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 612 op/s; 8570/268221 objects degraded (3.195%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T09:53:07.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:07 smithi129 ceph-mon[88471]: osdmap e110: 8 total, 8 up, 8 in 2024-09-21T09:53:08.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:08 smithi107 ceph-mon[102221]: osdmap e111: 8 total, 8 up, 8 in 2024-09-21T09:53:08.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:08 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3499616371' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:08.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:08 smithi107 ceph-mon[105022]: osdmap e111: 8 total, 8 up, 8 in 2024-09-21T09:53:08.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:08 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3499616371' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:08.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:08 smithi129 ceph-mon[88471]: osdmap e111: 8 total, 8 up, 8 in 2024-09-21T09:53:08.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:08 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3499616371' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:08.522 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:53:08.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:53:08 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:53:08.420+0000 7efdb5a17640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:53:09.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:09 smithi107 ceph-mon[102221]: pgmap v239: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 767 op/s; 8127/260970 objects degraded (3.114%); 6.2 MiB/s, 98 objects/s recovering 2024-09-21T09:53:09.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:09 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:53:09.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:09 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:53:09.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:09 smithi107 ceph-mon[102221]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:53:09.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:09 smithi107 ceph-mon[102221]: from='client.54769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:09.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:09 smithi107 ceph-mon[105022]: pgmap v239: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 767 op/s; 8127/260970 objects degraded (3.114%); 6.2 MiB/s, 98 objects/s recovering 2024-09-21T09:53:09.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:09 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:53:09.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:09 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:53:09.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:09 smithi107 ceph-mon[105022]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:53:09.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:09 smithi107 ceph-mon[105022]: from='client.54769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:09.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:09 smithi129 ceph-mon[88471]: pgmap v239: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 767 op/s; 8127/260970 objects degraded (3.114%); 6.2 MiB/s, 98 objects/s recovering 2024-09-21T09:53:09.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:09 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:53:09.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:09 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:53:09.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:09 smithi129 ceph-mon[88471]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:53:09.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:09 smithi129 ceph-mon[88471]: from='client.54769 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:10.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:10 smithi107 ceph-mon[102221]: from='client.35028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:10.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:10 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 8127/260970 objects degraded (3.114%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:10.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:10 smithi107 ceph-mon[105022]: from='client.35028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:10.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:10 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 8127/260970 objects degraded (3.114%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:10.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:10 smithi129 ceph-mon[88471]: from='client.35028 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:10.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:10 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 8127/260970 objects degraded (3.114%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:10.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:53:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:53:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:53:11.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:11 smithi107 ceph-mon[102221]: pgmap v240: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 864 op/s; 8127/258639 objects degraded (3.142%); 6.2 MiB/s, 98 objects/s recovering 2024-09-21T09:53:11.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:11 smithi107 ceph-mon[105022]: pgmap v240: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 864 op/s; 8127/258639 objects degraded (3.142%); 6.2 MiB/s, 98 objects/s recovering 2024-09-21T09:53:11.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:11 smithi129 ceph-mon[88471]: pgmap v240: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 864 op/s; 8127/258639 objects degraded (3.142%); 6.2 MiB/s, 98 objects/s recovering 2024-09-21T09:53:12.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:53:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:53:11] "GET /metrics HTTP/1.1" 200 36538 "" "Prometheus/2.51.0" 2024-09-21T09:53:12.129 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:53:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:53:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:12 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1378987082' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:12.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:53:12.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:12 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1378987082' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:12.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:53:12.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:12 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1378987082' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:13.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:13 smithi129 ceph-mon[88471]: pgmap v241: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 399 op/s; 8127/258639 objects degraded (3.142%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T09:53:13.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:13 smithi129 ceph-mon[88471]: from='client.54778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:13.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:13 smithi129 ceph-mon[88471]: from='client.35046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:13.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:13 smithi107 ceph-mon[102221]: pgmap v241: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 399 op/s; 8127/258639 objects degraded (3.142%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T09:53:13.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:13 smithi107 ceph-mon[102221]: from='client.54778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:13.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:13 smithi107 ceph-mon[102221]: from='client.35046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:13.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:13 smithi107 ceph-mon[105022]: pgmap v241: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 399 op/s; 8127/258639 objects degraded (3.142%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T09:53:13.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:13 smithi107 ceph-mon[105022]: from='client.54778 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:13.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:13 smithi107 ceph-mon[105022]: from='client.35046 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:15.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:15 smithi129 ceph-mon[88471]: pgmap v242: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 7989/247704 objects degraded (3.225%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T09:53:15.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:15 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 7989/247704 objects degraded (3.225%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:15 smithi107 ceph-mon[102221]: pgmap v242: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 7989/247704 objects degraded (3.225%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T09:53:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:15 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 7989/247704 objects degraded (3.225%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:15.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:15 smithi107 ceph-mon[105022]: pgmap v242: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 7989/247704 objects degraded (3.225%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T09:53:15.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:15 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 7989/247704 objects degraded (3.225%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:15.760 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:53:16.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:16 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4002546655' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:16.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:16 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4002546655' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:16.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:16 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4002546655' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:17.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:17 smithi129 ceph-mon[88471]: from='client.54787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:17.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:17 smithi129 ceph-mon[88471]: pgmap v243: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 7989/247704 objects degraded (3.225%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T09:53:17.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:17 smithi129 ceph-mon[88471]: from='client.35061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:17.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:17 smithi107 ceph-mon[102221]: from='client.54787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:17.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:17 smithi107 ceph-mon[102221]: pgmap v243: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 7989/247704 objects degraded (3.225%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T09:53:17.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:17 smithi107 ceph-mon[102221]: from='client.35061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:17.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:17 smithi107 ceph-mon[105022]: from='client.54787 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:17.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:17 smithi107 ceph-mon[105022]: pgmap v243: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 7989/247704 objects degraded (3.225%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T09:53:17.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:17 smithi107 ceph-mon[105022]: from='client.35061 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:19.418 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:19 smithi107 ceph-mon[102221]: pgmap v244: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 775 op/s; 7812/239283 objects degraded (3.265%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T09:53:19.418 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:19 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2150805235' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:19.419 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:53:19.419 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:19 smithi107 ceph-mon[105022]: pgmap v244: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 775 op/s; 7812/239283 objects degraded (3.265%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T09:53:19.419 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:19 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2150805235' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:19.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:19 smithi129 ceph-mon[88471]: pgmap v244: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 775 op/s; 7812/239283 objects degraded (3.265%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T09:53:19.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:19 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2150805235' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:20.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:20 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 7812/239283 objects degraded (3.265%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:20.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:20 smithi129 ceph-mon[88471]: from='client.35067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:20.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:53:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:53:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:53:20.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:20 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 7812/239283 objects degraded (3.265%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:20.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:20 smithi107 ceph-mon[105022]: from='client.35067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:20.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:20 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 7812/239283 objects degraded (3.265%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:20.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:20 smithi107 ceph-mon[102221]: from='client.35067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:21.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:21 smithi129 ceph-mon[88471]: from='client.54799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:21.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:21 smithi129 ceph-mon[88471]: pgmap v245: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 681 op/s; 7812/236409 objects degraded (3.304%); 1.6 MiB/s, 26 objects/s recovering 2024-09-21T09:53:21.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:21 smithi107 ceph-mon[105022]: from='client.54799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:21.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:21 smithi107 ceph-mon[105022]: pgmap v245: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 681 op/s; 7812/236409 objects degraded (3.304%); 1.6 MiB/s, 26 objects/s recovering 2024-09-21T09:53:21.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:21 smithi107 ceph-mon[102221]: from='client.54799 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:21.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:21 smithi107 ceph-mon[102221]: pgmap v245: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 681 op/s; 7812/236409 objects degraded (3.304%); 1.6 MiB/s, 26 objects/s recovering 2024-09-21T09:53:22.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:53:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:53:21] "GET /metrics HTTP/1.1" 200 36538 "" "Prometheus/2.51.0" 2024-09-21T09:53:23.054 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:53:23.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:23 smithi107 ceph-mon[102221]: pgmap v246: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 617 op/s; 7812/236409 objects degraded (3.304%); 1.6 MiB/s, 26 objects/s recovering 2024-09-21T09:53:23.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:23 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/194046193' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:23.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:23 smithi107 ceph-mon[105022]: pgmap v246: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 617 op/s; 7812/236409 objects degraded (3.304%); 1.6 MiB/s, 26 objects/s recovering 2024-09-21T09:53:23.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:23 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/194046193' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:23.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:23 smithi129 ceph-mon[88471]: pgmap v246: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 617 op/s; 7812/236409 objects degraded (3.304%); 1.6 MiB/s, 26 objects/s recovering 2024-09-21T09:53:23.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:23 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/194046193' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:23.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:53:23 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:53:23.437+0000 7efdb5a17640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:53:24.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:24 smithi107 ceph-mon[102221]: from='client.44894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:24.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:24 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:53:24.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:24 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:53:24.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:24 smithi107 ceph-mon[102221]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:53:24.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:24 smithi107 ceph-mon[102221]: from='client.44897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:24.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:24 smithi107 ceph-mon[105022]: from='client.44894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:24.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:24 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:53:24.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:24 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:53:24.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:24 smithi107 ceph-mon[105022]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:53:24.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:24 smithi107 ceph-mon[105022]: from='client.44897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:24.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:24 smithi129 ceph-mon[88471]: from='client.44894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:24.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:24 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:53:24.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:24 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:53:24.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:24 smithi129 ceph-mon[88471]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:53:24.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:24 smithi129 ceph-mon[88471]: from='client.44897 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:25.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:25 smithi107 ceph-mon[102221]: pgmap v247: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 927 op/s; 7640/225243 objects degraded (3.392%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T09:53:25.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:25 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 7640/225243 objects degraded (3.392%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:25.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:25 smithi107 ceph-mon[105022]: pgmap v247: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 927 op/s; 7640/225243 objects degraded (3.392%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T09:53:25.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:25 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 7640/225243 objects degraded (3.392%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:25.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:25 smithi129 ceph-mon[88471]: pgmap v247: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 927 op/s; 7640/225243 objects degraded (3.392%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T09:53:25.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:25 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 7640/225243 objects degraded (3.392%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:26.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:53:26.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:26 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/532922617' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:26.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:53:26.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:26 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/532922617' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:53:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:26 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/532922617' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:26.700 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:53:27.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:27 smithi107 ceph-mon[102221]: pgmap v248: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 623 op/s; 7640/225243 objects degraded (3.392%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T09:53:27.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:27 smithi107 ceph-mon[102221]: from='client.54823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:27.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:27 smithi107 ceph-mon[105022]: pgmap v248: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 623 op/s; 7640/225243 objects degraded (3.392%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T09:53:27.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:27 smithi107 ceph-mon[105022]: from='client.54823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:27.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:27 smithi129 ceph-mon[88471]: pgmap v248: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 623 op/s; 7640/225243 objects degraded (3.392%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T09:53:27.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:27 smithi129 ceph-mon[88471]: from='client.54823 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:28.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:28 smithi107 ceph-mon[102221]: from='client.35097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:28.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:28 smithi107 ceph-mon[105022]: from='client.35097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:28.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:28 smithi129 ceph-mon[88471]: from='client.35097 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:29 smithi107 ceph-mon[102221]: pgmap v249: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 7486/216942 objects degraded (3.451%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T09:53:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:29 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 7486/216942 objects degraded (3.451%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:29.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:29 smithi107 ceph-mon[105022]: pgmap v249: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 7486/216942 objects degraded (3.451%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T09:53:29.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:29 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 7486/216942 objects degraded (3.451%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:29 smithi129 ceph-mon[88471]: pgmap v249: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 854 op/s; 7486/216942 objects degraded (3.451%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T09:53:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:29 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 7486/216942 objects degraded (3.451%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:30.374 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:53:30.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:30 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/61679145' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:30.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:30 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/61679145' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:30 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/61679145' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:53:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:53:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:53:31.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:31 smithi107 ceph-mon[102221]: pgmap v250: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 7486/214149 objects degraded (3.496%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T09:53:31.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:31 smithi107 ceph-mon[102221]: from='client.44909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:31.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:31 smithi107 ceph-mon[105022]: pgmap v250: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 7486/214149 objects degraded (3.496%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T09:53:31.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:31 smithi107 ceph-mon[105022]: from='client.44909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:31 smithi129 ceph-mon[88471]: pgmap v250: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 7486/214149 objects degraded (3.496%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T09:53:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:31 smithi129 ceph-mon[88471]: from='client.44909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:53:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:53:31] "GET /metrics HTTP/1.1" 200 36536 "" "Prometheus/2.51.0" 2024-09-21T09:53:32.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:32 smithi107 ceph-mon[102221]: from='client.35112 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:32.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:32 smithi107 ceph-mon[105022]: from='client.35112 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:32.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:32 smithi129 ceph-mon[88471]: from='client.35112 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:33.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:33 smithi107 ceph-mon[102221]: pgmap v251: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 618 op/s; 7486/214149 objects degraded (3.496%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T09:53:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:33 smithi107 ceph-mon[105022]: pgmap v251: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 618 op/s; 7486/214149 objects degraded (3.496%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T09:53:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:33 smithi129 ceph-mon[88471]: pgmap v251: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 618 op/s; 7486/214149 objects degraded (3.496%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T09:53:34.026 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:53:34.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:34 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1510964884' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:34.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:34 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 7486/214149 objects degraded (3.496%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:34.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:34 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1510964884' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:34.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:34 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 7486/214149 objects degraded (3.496%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:34.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:34 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1510964884' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:34.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:34 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 7486/214149 objects degraded (3.496%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:35.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:35 smithi107 ceph-mon[102221]: pgmap v252: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 933 op/s; 7316/202791 objects degraded (3.608%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T09:53:35.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:35 smithi107 ceph-mon[102221]: from='client.35121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:35.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:35 smithi107 ceph-mon[102221]: from='client.35127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:35.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:35 smithi107 ceph-mon[105022]: pgmap v252: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 933 op/s; 7316/202791 objects degraded (3.608%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T09:53:35.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:35 smithi107 ceph-mon[105022]: from='client.35121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:35.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:35 smithi107 ceph-mon[105022]: from='client.35127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:35 smithi129 ceph-mon[88471]: pgmap v252: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 933 op/s; 7316/202791 objects degraded (3.608%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T09:53:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:35 smithi129 ceph-mon[88471]: from='client.35121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:35 smithi129 ceph-mon[88471]: from='client.35127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:37.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:37 smithi107 ceph-mon[105022]: pgmap v253: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 623 op/s; 7316/202791 objects degraded (3.608%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T09:53:37.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:37 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/815622968' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:37.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:37 smithi107 ceph-mon[102221]: pgmap v253: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 623 op/s; 7316/202791 objects degraded (3.608%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T09:53:37.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:37 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/815622968' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:37.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:37 smithi129 ceph-mon[88471]: pgmap v253: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 623 op/s; 7316/202791 objects degraded (3.608%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T09:53:37.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:37 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/815622968' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:37.669 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:53:38.570 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:38 smithi129 ceph-mon[88471]: from='client.35136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:38.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:38 smithi107 ceph-mon[102221]: from='client.35136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:38.651 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:38 smithi107 ceph-mon[105022]: from='client.35136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:39.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:39 smithi129 ceph-mon[88471]: pgmap v254: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 7143/194625 objects degraded (3.670%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T09:53:39.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:39 smithi129 ceph-mon[88471]: from='client.54847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:39.410 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:39 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:53:39.411 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:39 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 7143/194625 objects degraded (3.670%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:39.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:39 smithi107 ceph-mon[102221]: pgmap v254: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 7143/194625 objects degraded (3.670%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T09:53:39.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:39 smithi107 ceph-mon[102221]: from='client.54847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:39.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:39 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:53:39.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:39 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 7143/194625 objects degraded (3.670%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:39.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:39 smithi107 ceph-mon[105022]: pgmap v254: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 850 op/s; 7143/194625 objects degraded (3.670%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T09:53:39.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:39 smithi107 ceph-mon[105022]: from='client.54847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:39.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:39 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:53:39.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:39 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 7143/194625 objects degraded (3.670%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:40.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:53:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:53:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:53:41.246 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:53:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:41 smithi107 ceph-mon[102221]: pgmap v255: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 7143/191718 objects degraded (3.726%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:53:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:41 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/727169261' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:41.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:53:41.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:41 smithi107 ceph-mon[105022]: pgmap v255: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 7143/191718 objects degraded (3.726%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:53:41.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:41 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/727169261' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:41.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:53:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:41 smithi129 ceph-mon[88471]: pgmap v255: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 7143/191718 objects degraded (3.726%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:53:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:41 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/727169261' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:53:42.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:53:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:53:41] "GET /metrics HTTP/1.1" 200 36540 "" "Prometheus/2.51.0" 2024-09-21T09:53:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:42 smithi107 ceph-mon[105022]: from='client.35148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:42.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:42 smithi107 ceph-mon[102221]: from='client.35148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:42 smithi129 ceph-mon[88471]: from='client.35148 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:43.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:43 smithi107 ceph-mon[102221]: from='client.54859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:43.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:43 smithi107 ceph-mon[102221]: pgmap v256: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 622 op/s; 7143/191718 objects degraded (3.726%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:53:43.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:43 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:53:43.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:43 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:53:43.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:43 smithi107 ceph-mon[105022]: from='client.54859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:43.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:43 smithi107 ceph-mon[105022]: pgmap v256: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 622 op/s; 7143/191718 objects degraded (3.726%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:53:43.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:43 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:53:43.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:43 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:53:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:43 smithi129 ceph-mon[88471]: from='client.54859 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:43 smithi129 ceph-mon[88471]: pgmap v256: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 622 op/s; 7143/191718 objects degraded (3.726%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:53:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:43 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:53:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:43 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:53:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:44 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 7143/191718 objects degraded (3.726%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:44.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:44 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/630584642' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:44.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:44 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 7143/191718 objects degraded (3.726%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:44.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:44 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/630584642' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:44 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 7143/191718 objects degraded (3.726%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:44 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/630584642' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:44.867 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:53:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:45 smithi107 ceph-mon[102221]: pgmap v257: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 905 op/s; 6970/181524 objects degraded (3.840%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T09:53:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:45 smithi107 ceph-mon[105022]: pgmap v257: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 905 op/s; 6970/181524 objects degraded (3.840%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T09:53:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:45 smithi129 ceph-mon[88471]: pgmap v257: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 905 op/s; 6970/181524 objects degraded (3.840%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T09:53:46.522 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:53:46.253+0000 7efdb5a17640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:53:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[105022]: from='client.54868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[105022]: from='client.35163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:53:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:53:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:53:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:53:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:53:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:53:46.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:53:46.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:53:46.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:53:46.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:53:46.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[102221]: from='client.54868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:46.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[102221]: from='client.35163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:46.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:53:46.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:53:46.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:53:46.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:53:46.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:53:46.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:53:46.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:53:46.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:53:46.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:53:46.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:46 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:53:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:46 smithi129 ceph-mon[88471]: from='client.54868 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:46 smithi129 ceph-mon[88471]: from='client.35163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:46 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:53:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:46 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:53:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:46 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:53:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:46 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:53:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:46 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:53:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:46 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:53:46.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:46 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:53:46.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:46 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:53:46.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:46 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:53:46.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:46 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:53:47.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:47 smithi107 ceph-mon[102221]: pgmap v258: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 590 op/s; 6970/181524 objects degraded (3.840%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:53:47.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:47 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:53:47.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:47 smithi107 ceph-mon[102221]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:53:47.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:47 smithi107 ceph-mon[105022]: pgmap v258: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 590 op/s; 6970/181524 objects degraded (3.840%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:53:47.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:47 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:53:47.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:47 smithi107 ceph-mon[105022]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:53:47.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:47 smithi129 ceph-mon[88471]: pgmap v258: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 590 op/s; 6970/181524 objects degraded (3.840%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:53:47.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:47 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:53:47.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:47 smithi129 ceph-mon[88471]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:53:48.483 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:53:48.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:48 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1876834578' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:48.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:48 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1876834578' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:48.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:48 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1876834578' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:49.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:49 smithi107 ceph-mon[102221]: pgmap v259: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 807 op/s; 6804/173712 objects degraded (3.917%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T09:53:49.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:49 smithi107 ceph-mon[102221]: from='client.44960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:49.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:49 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 6804/173712 objects degraded (3.917%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:49.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:49 smithi107 ceph-mon[105022]: pgmap v259: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 807 op/s; 6804/173712 objects degraded (3.917%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T09:53:49.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:49 smithi107 ceph-mon[105022]: from='client.44960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:49.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:49 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 6804/173712 objects degraded (3.917%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:49 smithi129 ceph-mon[88471]: pgmap v259: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 807 op/s; 6804/173712 objects degraded (3.917%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T09:53:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:49 smithi129 ceph-mon[88471]: from='client.44960 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:49 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 6804/173712 objects degraded (3.917%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:50.565 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:53:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:53:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:53:50.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:50 smithi107 ceph-mon[102221]: from='client.54886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:50.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:50 smithi107 ceph-mon[105022]: from='client.54886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:50.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:50 smithi129 ceph-mon[88471]: from='client.54886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:51.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:51 smithi107 ceph-mon[102221]: pgmap v260: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 656 op/s; 6804/170961 objects degraded (3.980%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:53:51.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:51 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1259547692' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:51.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:51 smithi107 ceph-mon[105022]: pgmap v260: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 656 op/s; 6804/170961 objects degraded (3.980%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:53:51.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:51 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1259547692' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:51.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:53:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:53:51] "GET /metrics HTTP/1.1" 200 36540 "" "Prometheus/2.51.0" 2024-09-21T09:53:51.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:51 smithi129 ceph-mon[88471]: pgmap v260: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 656 op/s; 6804/170961 objects degraded (3.980%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:53:51.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:51 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1259547692' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:52.105 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:53:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:53 smithi107 ceph-mon[102221]: pgmap v261: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 576 op/s; 6804/170961 objects degraded (3.980%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:53:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:53 smithi107 ceph-mon[102221]: from='client.44978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:53 smithi107 ceph-mon[102221]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:53 smithi107 ceph-mon[105022]: pgmap v261: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 576 op/s; 6804/170961 objects degraded (3.980%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:53:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:53 smithi107 ceph-mon[105022]: from='client.44978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:53 smithi107 ceph-mon[105022]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:53 smithi129 ceph-mon[88471]: pgmap v261: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 576 op/s; 6804/170961 objects degraded (3.980%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:53:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:53 smithi129 ceph-mon[88471]: from='client.44978 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:53 smithi129 ceph-mon[88471]: from='client.35184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:54.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:54 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 6804/170961 objects degraded (3.980%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:54.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:54 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 6804/170961 objects degraded (3.980%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:54 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 6804/170961 objects degraded (3.980%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:55.801 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:53:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:55 smithi107 ceph-mon[102221]: pgmap v262: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 880 op/s; 6633/159993 objects degraded (4.146%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T09:53:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:55 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3266467876' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:55.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:55 smithi107 ceph-mon[105022]: pgmap v262: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 880 op/s; 6633/159993 objects degraded (4.146%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T09:53:55.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:55 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3266467876' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:55.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:55 smithi129 ceph-mon[88471]: pgmap v262: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 880 op/s; 6633/159993 objects degraded (4.146%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T09:53:55.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:55 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3266467876' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:53:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:53:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:53:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:57 smithi107 ceph-mon[102221]: from='client.35193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:57 smithi107 ceph-mon[102221]: pgmap v263: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 6633/159993 objects degraded (4.146%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:53:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:57 smithi107 ceph-mon[102221]: from='client.54922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:57.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:57 smithi107 ceph-mon[105022]: from='client.35193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:57.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:57 smithi107 ceph-mon[105022]: pgmap v263: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 6633/159993 objects degraded (4.146%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:53:57.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:57 smithi107 ceph-mon[105022]: from='client.54922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:57.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:57 smithi129 ceph-mon[88471]: from='client.35193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:57.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:57 smithi129 ceph-mon[88471]: pgmap v263: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 6633/159993 objects degraded (4.146%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:53:57.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:57 smithi129 ceph-mon[88471]: from='client.54922 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:59.419 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:53:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:59 smithi107 ceph-mon[102221]: pgmap v264: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 819 op/s; 6465/151986 objects degraded (4.254%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T09:53:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:59 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/51954411' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:53:59 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 6465/151986 objects degraded (4.254%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:59 smithi107 ceph-mon[105022]: pgmap v264: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 819 op/s; 6465/151986 objects degraded (4.254%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T09:53:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:59 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/51954411' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:53:59 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 6465/151986 objects degraded (4.254%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:53:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:59 smithi129 ceph-mon[88471]: pgmap v264: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 819 op/s; 6465/151986 objects degraded (4.254%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T09:53:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:59 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/51954411' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:53:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:53:59 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 6465/151986 objects degraded (4.254%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:00.621 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:54:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:54:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:54:00.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:00 smithi129 ceph-mon[88471]: from='client.44999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:01.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:00 smithi107 ceph-mon[102221]: from='client.44999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:01.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:00 smithi107 ceph-mon[105022]: from='client.44999 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:01.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:54:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:54:01.271+0000 7efdb5a17640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:54:01.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:01 smithi129 ceph-mon[88471]: from='client.35202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:01.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:01 smithi129 ceph-mon[88471]: pgmap v265: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 680 op/s; 6465/149214 objects degraded (4.333%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:54:01.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:01 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:54:02.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:01 smithi107 ceph-mon[102221]: from='client.35202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:02.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:01 smithi107 ceph-mon[102221]: pgmap v265: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 680 op/s; 6465/149214 objects degraded (4.333%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:54:02.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:01 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:54:02.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:01 smithi107 ceph-mon[105022]: from='client.35202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:02.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:01 smithi107 ceph-mon[105022]: pgmap v265: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 680 op/s; 6465/149214 objects degraded (4.333%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:54:02.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:01 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:54:02.110 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:54:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:54:01] "GET /metrics HTTP/1.1" 200 36537 "" "Prometheus/2.51.0" 2024-09-21T09:54:02.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:02 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:54:02.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:02 smithi129 ceph-mon[88471]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:54:02.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:02 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1536649410' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:03.035 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:54:03.036 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:02 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:54:03.036 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:02 smithi107 ceph-mon[102221]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:54:03.036 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:02 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1536649410' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:03.036 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:02 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:54:03.036 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:02 smithi107 ceph-mon[105022]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:54:03.036 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:02 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1536649410' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:03.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:03 smithi107 ceph-mon[102221]: pgmap v266: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 603 op/s; 6465/149214 objects degraded (4.333%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:54:03.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:03 smithi107 ceph-mon[105022]: pgmap v266: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 603 op/s; 6465/149214 objects degraded (4.333%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:54:03.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:03 smithi129 ceph-mon[88471]: pgmap v266: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 603 op/s; 6465/149214 objects degraded (4.333%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T09:54:04.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:04 smithi107 ceph-mon[102221]: from='client.54943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:04 smithi107 ceph-mon[102221]: from='client.45014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:04 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 6465/149214 objects degraded (4.333%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:04 smithi107 ceph-mon[102221]: osdmap e112: 8 total, 8 up, 8 in 2024-09-21T09:54:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:04 smithi107 ceph-mon[105022]: from='client.54943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:04 smithi107 ceph-mon[105022]: from='client.45014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:04 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 6465/149214 objects degraded (4.333%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:04 smithi107 ceph-mon[105022]: osdmap e112: 8 total, 8 up, 8 in 2024-09-21T09:54:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:04 smithi129 ceph-mon[88471]: from='client.54943 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:04 smithi129 ceph-mon[88471]: from='client.45014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:04 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 6465/149214 objects degraded (4.333%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:04 smithi129 ceph-mon[88471]: osdmap e112: 8 total, 8 up, 8 in 2024-09-21T09:54:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:05 smithi129 ceph-mon[88471]: pgmap v267: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 921 op/s; 6266/137790 objects degraded (4.547%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T09:54:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:05 smithi129 ceph-mon[88471]: osdmap e113: 8 total, 8 up, 8 in 2024-09-21T09:54:06.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:05 smithi107 ceph-mon[102221]: pgmap v267: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 921 op/s; 6266/137790 objects degraded (4.547%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T09:54:06.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:05 smithi107 ceph-mon[102221]: osdmap e113: 8 total, 8 up, 8 in 2024-09-21T09:54:06.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:05 smithi107 ceph-mon[105022]: pgmap v267: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 921 op/s; 6266/137790 objects degraded (4.547%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T09:54:06.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:05 smithi107 ceph-mon[105022]: osdmap e113: 8 total, 8 up, 8 in 2024-09-21T09:54:06.645 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:54:07.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:06 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2824956904' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:07.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:06 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2824956904' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:07.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:06 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2824956904' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:08.069 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:07 smithi129 ceph-mon[88471]: pgmap v270: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 591 op/s; 6266/137790 objects degraded (4.547%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T09:54:08.070 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:07 smithi129 ceph-mon[88471]: from='client.35220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:08.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:07 smithi107 ceph-mon[102221]: pgmap v270: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 591 op/s; 6266/137790 objects degraded (4.547%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T09:54:08.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:07 smithi107 ceph-mon[102221]: from='client.35220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:08.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:07 smithi107 ceph-mon[105022]: pgmap v270: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 591 op/s; 6266/137790 objects degraded (4.547%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T09:54:08.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:07 smithi107 ceph-mon[105022]: from='client.35220 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:09.081 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:08 smithi129 ceph-mon[88471]: from='client.45026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:08 smithi107 ceph-mon[102221]: from='client.45026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:09.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:08 smithi107 ceph-mon[105022]: from='client.45026 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:10.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:09 smithi107 ceph-mon[102221]: pgmap v271: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 632 op/s; 5893/134037 objects degraded (4.397%); 1.6 MiB/s, 71 objects/s recovering 2024-09-21T09:54:10.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:09 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 5893/134037 objects degraded (4.397%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:10.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:09 smithi107 ceph-mon[105022]: pgmap v271: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 632 op/s; 5893/134037 objects degraded (4.397%); 1.6 MiB/s, 71 objects/s recovering 2024-09-21T09:54:10.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:09 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 5893/134037 objects degraded (4.397%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:10.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:09 smithi129 ceph-mon[88471]: pgmap v271: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 632 op/s; 5893/134037 objects degraded (4.397%); 1.6 MiB/s, 71 objects/s recovering 2024-09-21T09:54:10.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:09 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 5893/134037 objects degraded (4.397%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:10.281 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:54:10.656 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:54:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:54:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:54:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:10 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3613083203' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:11.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:10 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3613083203' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:11.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:10 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3613083203' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:12.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:11 smithi107 ceph-mon[102221]: pgmap v272: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 843 op/s; 5831/128955 objects degraded (4.522%); 1.6 MiB/s, 79 objects/s recovering 2024-09-21T09:54:12.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:11 smithi107 ceph-mon[102221]: from='client.35235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:12.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:54:12.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:54:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:54:11] "GET /metrics HTTP/1.1" 200 36535 "" "Prometheus/2.51.0" 2024-09-21T09:54:12.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:11 smithi107 ceph-mon[105022]: pgmap v272: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 843 op/s; 5831/128955 objects degraded (4.522%); 1.6 MiB/s, 79 objects/s recovering 2024-09-21T09:54:12.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:11 smithi107 ceph-mon[105022]: from='client.35235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:12.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:54:12.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:11 smithi129 ceph-mon[88471]: pgmap v272: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 843 op/s; 5831/128955 objects degraded (4.522%); 1.6 MiB/s, 79 objects/s recovering 2024-09-21T09:54:12.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:11 smithi129 ceph-mon[88471]: from='client.35235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:12.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:54:13.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:12 smithi107 ceph-mon[102221]: from='client.35241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:13.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:12 smithi107 ceph-mon[105022]: from='client.35241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:13.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:12 smithi129 ceph-mon[88471]: from='client.35241 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:13.893 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:54:14.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:13 smithi107 ceph-mon[102221]: pgmap v273: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 368 op/s; 5831/128955 objects degraded (4.522%); 0 B/s, 54 objects/s recovering 2024-09-21T09:54:14.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:13 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3183507614' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:14.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:13 smithi107 ceph-mon[105022]: pgmap v273: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 368 op/s; 5831/128955 objects degraded (4.522%); 0 B/s, 54 objects/s recovering 2024-09-21T09:54:14.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:13 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3183507614' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:14.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:13 smithi129 ceph-mon[88471]: pgmap v273: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 368 op/s; 5831/128955 objects degraded (4.522%); 0 B/s, 54 objects/s recovering 2024-09-21T09:54:14.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:13 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3183507614' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:15.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:14 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 5831/128955 objects degraded (4.522%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:15.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:14 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 5831/128955 objects degraded (4.522%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:15.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:14 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 5831/128955 objects degraded (4.522%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:16.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:15 smithi107 ceph-mon[102221]: from='client.45035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:16.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:15 smithi107 ceph-mon[102221]: pgmap v274: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 665 op/s; 5582/118266 objects degraded (4.720%); 0 B/s, 70 objects/s recovering 2024-09-21T09:54:16.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:15 smithi107 ceph-mon[102221]: from='client.54982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:16.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:15 smithi107 ceph-mon[105022]: from='client.45035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:16.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:15 smithi107 ceph-mon[105022]: pgmap v274: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 665 op/s; 5582/118266 objects degraded (4.720%); 0 B/s, 70 objects/s recovering 2024-09-21T09:54:16.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:15 smithi107 ceph-mon[105022]: from='client.54982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:16.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:15 smithi129 ceph-mon[88471]: from='client.45035 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:16.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:15 smithi129 ceph-mon[88471]: pgmap v274: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 665 op/s; 5582/118266 objects degraded (4.720%); 0 B/s, 70 objects/s recovering 2024-09-21T09:54:16.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:15 smithi129 ceph-mon[88471]: from='client.54982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:16.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:54:16 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:54:16.287+0000 7efdb5a17640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:54:17.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:16 smithi107 ceph-mon[102221]: pgmap v275: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 624 op/s; 5582/117567 objects degraded (4.748%); 0 B/s, 63 objects/s recovering 2024-09-21T09:54:17.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:16 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:54:17.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:16 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:54:17.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:16 smithi107 ceph-mon[102221]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:54:17.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:16 smithi107 ceph-mon[105022]: pgmap v275: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 624 op/s; 5582/117567 objects degraded (4.748%); 0 B/s, 63 objects/s recovering 2024-09-21T09:54:17.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:16 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:54:17.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:16 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:54:17.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:16 smithi107 ceph-mon[105022]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:54:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:16 smithi129 ceph-mon[88471]: pgmap v275: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 624 op/s; 5582/117567 objects degraded (4.748%); 0 B/s, 63 objects/s recovering 2024-09-21T09:54:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:16 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:54:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:16 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:54:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:16 smithi129 ceph-mon[88471]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:54:17.512 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:54:18.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:17 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2311140073' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:18.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:17 smithi107 ceph-mon[102221]: from='client.45044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:18.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:17 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2311140073' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:18.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:17 smithi107 ceph-mon[105022]: from='client.45044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:18.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:17 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2311140073' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:18.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:17 smithi129 ceph-mon[88471]: from='client.45044 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:19.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:18 smithi107 ceph-mon[102221]: pgmap v276: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 5311/112671 objects degraded (4.714%); 0 B/s, 79 objects/s recovering 2024-09-21T09:54:19.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:18 smithi107 ceph-mon[102221]: from='client.45050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:19.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:18 smithi107 ceph-mon[105022]: pgmap v276: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 5311/112671 objects degraded (4.714%); 0 B/s, 79 objects/s recovering 2024-09-21T09:54:19.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:18 smithi107 ceph-mon[105022]: from='client.45050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:19.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:18 smithi129 ceph-mon[88471]: pgmap v276: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.3 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 5311/112671 objects degraded (4.714%); 0 B/s, 79 objects/s recovering 2024-09-21T09:54:19.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:18 smithi129 ceph-mon[88471]: from='client.45050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:20.025 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:19 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 5311/112671 objects degraded (4.714%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:20.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:19 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 5311/112671 objects degraded (4.714%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:20.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:19 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 5311/112671 objects degraded (4.714%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:20.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:54:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:54:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:54:21.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:20 smithi107 ceph-mon[102221]: pgmap v277: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 768 op/s; 5311/106356 objects degraded (4.994%); 0 B/s, 48 objects/s recovering 2024-09-21T09:54:21.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:20 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/130635126' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:21.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:20 smithi107 ceph-mon[105022]: pgmap v277: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 768 op/s; 5311/106356 objects degraded (4.994%); 0 B/s, 48 objects/s recovering 2024-09-21T09:54:21.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:20 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/130635126' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:21.141 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:54:21.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:20 smithi129 ceph-mon[88471]: pgmap v277: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 768 op/s; 5311/106356 objects degraded (4.994%); 0 B/s, 48 objects/s recovering 2024-09-21T09:54:21.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:20 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/130635126' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:22.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:21 smithi107 ceph-mon[102221]: from='client.35274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:22.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:21 smithi107 ceph-mon[102221]: from='client.55009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:22.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:54:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:54:21] "GET /metrics HTTP/1.1" 200 36535 "" "Prometheus/2.51.0" 2024-09-21T09:54:22.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:21 smithi107 ceph-mon[105022]: from='client.35274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:22.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:21 smithi107 ceph-mon[105022]: from='client.55009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:22.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:21 smithi129 ceph-mon[88471]: from='client.35274 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:22.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:21 smithi129 ceph-mon[88471]: from='client.55009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:23.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:22 smithi107 ceph-mon[102221]: pgmap v278: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 627 op/s; 5311/106356 objects degraded (4.994%); 0 B/s, 43 objects/s recovering 2024-09-21T09:54:23.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:22 smithi107 ceph-mon[105022]: pgmap v278: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 627 op/s; 5311/106356 objects degraded (4.994%); 0 B/s, 43 objects/s recovering 2024-09-21T09:54:23.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:22 smithi129 ceph-mon[88471]: pgmap v278: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 627 op/s; 5311/106356 objects degraded (4.994%); 0 B/s, 43 objects/s recovering 2024-09-21T09:54:24.802 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:54:25.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:25 smithi107 ceph-mon[102221]: pgmap v279: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 925 op/s; 5041/95634 objects degraded (5.271%); 0 B/s, 65 objects/s recovering 2024-09-21T09:54:25.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:25 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 5041/95634 objects degraded (5.271%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:25.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:25 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1446331076' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:25.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:25 smithi107 ceph-mon[105022]: pgmap v279: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 925 op/s; 5041/95634 objects degraded (5.271%); 0 B/s, 65 objects/s recovering 2024-09-21T09:54:25.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:25 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 5041/95634 objects degraded (5.271%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:25.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:25 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1446331076' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:25.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:25 smithi129 ceph-mon[88471]: pgmap v279: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 925 op/s; 5041/95634 objects degraded (5.271%); 0 B/s, 65 objects/s recovering 2024-09-21T09:54:25.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:25 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 5041/95634 objects degraded (5.271%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:25.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:25 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1446331076' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:26.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:26 smithi107 ceph-mon[102221]: from='client.45074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:26.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:26 smithi107 ceph-mon[102221]: from='client.45080 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:26.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:26 smithi107 ceph-mon[105022]: from='client.45074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:26.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:26 smithi107 ceph-mon[105022]: from='client.45080 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:26.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:26 smithi129 ceph-mon[88471]: from='client.45074 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:26.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:26 smithi129 ceph-mon[88471]: from='client.45080 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:27 smithi107 ceph-mon[102221]: pgmap v280: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 647 op/s; 5041/94944 objects degraded (5.309%); 0 B/s, 45 objects/s recovering 2024-09-21T09:54:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:54:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:27 smithi107 ceph-mon[105022]: pgmap v280: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 647 op/s; 5041/94944 objects degraded (5.309%); 0 B/s, 45 objects/s recovering 2024-09-21T09:54:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:54:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:27 smithi129 ceph-mon[88471]: pgmap v280: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 647 op/s; 5041/94944 objects degraded (5.309%); 0 B/s, 45 objects/s recovering 2024-09-21T09:54:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:54:28.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:28 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2200884101' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:28.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:28 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2200884101' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:28.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:28 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2200884101' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:28.431 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:54:29.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:29 smithi107 ceph-mon[102221]: pgmap v281: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 766 op/s; 4766/89958 objects degraded (5.298%); 0 B/s, 67 objects/s recovering 2024-09-21T09:54:29.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:29 smithi107 ceph-mon[102221]: from='client.35289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:29.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:29 smithi107 ceph-mon[105022]: pgmap v281: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 766 op/s; 4766/89958 objects degraded (5.298%); 0 B/s, 67 objects/s recovering 2024-09-21T09:54:29.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:29 smithi107 ceph-mon[105022]: from='client.35289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:29.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:29 smithi129 ceph-mon[88471]: pgmap v281: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 766 op/s; 4766/89958 objects degraded (5.298%); 0 B/s, 67 objects/s recovering 2024-09-21T09:54:29.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:29 smithi129 ceph-mon[88471]: from='client.35289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:30.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:30 smithi107 ceph-mon[102221]: from='client.55024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:30.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:30 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 4766/89958 objects degraded (5.298%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:30.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:30 smithi107 ceph-mon[105022]: from='client.55024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:30.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:30 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 4766/89958 objects degraded (5.298%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:30.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:30 smithi129 ceph-mon[88471]: from='client.55024 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:30.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:30 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 4766/89958 objects degraded (5.298%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:30.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:54:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:54:30.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:54:31.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:31 smithi107 ceph-mon[102221]: pgmap v282: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 4766/83760 objects degraded (5.690%); 0 B/s, 45 objects/s recovering 2024-09-21T09:54:31.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:31 smithi107 ceph-mon[105022]: pgmap v282: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 4766/83760 objects degraded (5.690%); 0 B/s, 45 objects/s recovering 2024-09-21T09:54:31.359 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:54:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:54:31.313+0000 7efdb5a17640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:54:31.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:31 smithi129 ceph-mon[88471]: pgmap v282: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 4766/83760 objects degraded (5.690%); 0 B/s, 45 objects/s recovering 2024-09-21T09:54:32.087 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:54:32.087 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:54:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:54:31] "GET /metrics HTTP/1.1" 200 36517 "" "Prometheus/2.51.0" 2024-09-21T09:54:32.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:32 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:54:32.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:32 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:54:32.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:32 smithi107 ceph-mon[102221]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:54:32.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:32 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/356496864' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:32.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:32 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:54:32.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:32 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:54:32.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:32 smithi107 ceph-mon[105022]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:54:32.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:32 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/356496864' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:32.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:32 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:54:32.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:32 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:54:32.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:32 smithi129 ceph-mon[88471]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:54:32.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:32 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/356496864' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:33.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:33 smithi129 ceph-mon[88471]: pgmap v283: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 627 op/s; 4766/83760 objects degraded (5.690%); 0 B/s, 45 objects/s recovering 2024-09-21T09:54:33.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:33 smithi129 ceph-mon[88471]: from='client.55027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:33.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:33 smithi129 ceph-mon[88471]: from='client.35304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:33.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:33 smithi107 ceph-mon[102221]: pgmap v283: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 627 op/s; 4766/83760 objects degraded (5.690%); 0 B/s, 45 objects/s recovering 2024-09-21T09:54:33.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:33 smithi107 ceph-mon[102221]: from='client.55027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:33.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:33 smithi107 ceph-mon[102221]: from='client.35304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:33 smithi107 ceph-mon[105022]: pgmap v283: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.7 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 627 op/s; 4766/83760 objects degraded (5.690%); 0 B/s, 45 objects/s recovering 2024-09-21T09:54:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:33 smithi107 ceph-mon[105022]: from='client.55027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:33 smithi107 ceph-mon[105022]: from='client.35304 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:35.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:35 smithi129 ceph-mon[88471]: pgmap v284: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 922 op/s; 4471/73128 objects degraded (6.114%); 0 B/s, 69 objects/s recovering 2024-09-21T09:54:35.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:35 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 4471/73128 objects degraded (6.114%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:35.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:35 smithi107 ceph-mon[102221]: pgmap v284: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 922 op/s; 4471/73128 objects degraded (6.114%); 0 B/s, 69 objects/s recovering 2024-09-21T09:54:35.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:35 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 4471/73128 objects degraded (6.114%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:35.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:35 smithi107 ceph-mon[105022]: pgmap v284: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 922 op/s; 4471/73128 objects degraded (6.114%); 0 B/s, 69 objects/s recovering 2024-09-21T09:54:35.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:35 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 4471/73128 objects degraded (6.114%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:35.743 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:54:36.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:36 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2384864742' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:36.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:36 smithi129 ceph-mon[88471]: from='client.45113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:36.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:36 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2384864742' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:36.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:36 smithi107 ceph-mon[102221]: from='client.45113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:36.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:36 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2384864742' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:36.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:36 smithi107 ceph-mon[105022]: from='client.45113 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:37.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:37 smithi129 ceph-mon[88471]: pgmap v285: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 644 op/s; 4471/72423 objects degraded (6.173%); 0 B/s, 47 objects/s recovering 2024-09-21T09:54:37.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:37 smithi129 ceph-mon[88471]: from='client.55042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:37.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:37 smithi107 ceph-mon[102221]: pgmap v285: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 644 op/s; 4471/72423 objects degraded (6.173%); 0 B/s, 47 objects/s recovering 2024-09-21T09:54:37.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:37 smithi107 ceph-mon[102221]: from='client.55042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:37.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:37 smithi107 ceph-mon[105022]: pgmap v285: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 644 op/s; 4471/72423 objects degraded (6.173%); 0 B/s, 47 objects/s recovering 2024-09-21T09:54:37.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:37 smithi107 ceph-mon[105022]: from='client.55042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:38.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:38 smithi129 ceph-mon[88471]: osdmap e114: 8 total, 8 up, 8 in 2024-09-21T09:54:38.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:38 smithi107 ceph-mon[102221]: osdmap e114: 8 total, 8 up, 8 in 2024-09-21T09:54:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:38 smithi107 ceph-mon[105022]: osdmap e114: 8 total, 8 up, 8 in 2024-09-21T09:54:39.398 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:54:39.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:39 smithi129 ceph-mon[88471]: pgmap v287: 89 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 4170/67839 objects degraded (6.147%); 0 B/s, 59 objects/s recovering 2024-09-21T09:54:39.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:39 smithi129 ceph-mon[88471]: osdmap e115: 8 total, 8 up, 8 in 2024-09-21T09:54:39.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:39 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2531739324' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:39.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:39 smithi107 ceph-mon[102221]: pgmap v287: 89 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 4170/67839 objects degraded (6.147%); 0 B/s, 59 objects/s recovering 2024-09-21T09:54:39.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:39 smithi107 ceph-mon[102221]: osdmap e115: 8 total, 8 up, 8 in 2024-09-21T09:54:39.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:39 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2531739324' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:39.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:39 smithi107 ceph-mon[105022]: pgmap v287: 89 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.4 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 4170/67839 objects degraded (6.147%); 0 B/s, 59 objects/s recovering 2024-09-21T09:54:39.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:39 smithi107 ceph-mon[105022]: osdmap e115: 8 total, 8 up, 8 in 2024-09-21T09:54:39.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:39 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2531739324' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:40.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:40 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 4170/67839 objects degraded (6.147%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:40.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:40 smithi129 ceph-mon[88471]: from='client.45128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:40.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:54:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:54:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:54:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:40 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 4170/67839 objects degraded (6.147%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:40 smithi107 ceph-mon[102221]: from='client.45128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:40.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:40 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 4170/67839 objects degraded (6.147%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:40.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:40 smithi107 ceph-mon[105022]: from='client.45128 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:41.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:41 smithi129 ceph-mon[88471]: from='client.55048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:41.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:41 smithi129 ceph-mon[88471]: pgmap v289: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 3972/62307 objects degraded (6.375%); 0 B/s, 99 objects/s recovering 2024-09-21T09:54:41.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:54:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:41 smithi107 ceph-mon[102221]: from='client.55048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:41 smithi107 ceph-mon[102221]: pgmap v289: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 3972/62307 objects degraded (6.375%); 0 B/s, 99 objects/s recovering 2024-09-21T09:54:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:54:41.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:41 smithi107 ceph-mon[105022]: from='client.55048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:41.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:41 smithi107 ceph-mon[105022]: pgmap v289: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 3972/62307 objects degraded (6.375%); 0 B/s, 99 objects/s recovering 2024-09-21T09:54:41.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:54:42.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:54:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:54:41] "GET /metrics HTTP/1.1" 200 36518 "" "Prometheus/2.51.0" 2024-09-21T09:54:43.045 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:54:43.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:43 smithi107 ceph-mon[102221]: pgmap v290: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 450 op/s; 3972/62307 objects degraded (6.375%); 0 B/s, 62 objects/s recovering 2024-09-21T09:54:43.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:43 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1608446513' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:43.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:43 smithi107 ceph-mon[105022]: pgmap v290: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 450 op/s; 3972/62307 objects degraded (6.375%); 0 B/s, 62 objects/s recovering 2024-09-21T09:54:43.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:43 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1608446513' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:43 smithi129 ceph-mon[88471]: pgmap v290: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 450 op/s; 3972/62307 objects degraded (6.375%); 0 B/s, 62 objects/s recovering 2024-09-21T09:54:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:43 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1608446513' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:44 smithi107 ceph-mon[102221]: from='client.45146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:44 smithi107 ceph-mon[102221]: from='client.55060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:44.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:44 smithi107 ceph-mon[105022]: from='client.45146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:44.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:44 smithi107 ceph-mon[105022]: from='client.55060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:44 smithi129 ceph-mon[88471]: from='client.45146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:44 smithi129 ceph-mon[88471]: from='client.55060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:45 smithi107 ceph-mon[102221]: pgmap v291: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 789 op/s; 3772/53469 objects degraded (7.055%); 0 B/s, 87 objects/s recovering 2024-09-21T09:54:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:45 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 3772/53469 objects degraded (7.055%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:45 smithi107 ceph-mon[105022]: pgmap v291: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 789 op/s; 3772/53469 objects degraded (7.055%); 0 B/s, 87 objects/s recovering 2024-09-21T09:54:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:45 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 3772/53469 objects degraded (7.055%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:45.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:45 smithi129 ceph-mon[88471]: pgmap v291: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 789 op/s; 3772/53469 objects degraded (7.055%); 0 B/s, 87 objects/s recovering 2024-09-21T09:54:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:45 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 3772/53469 objects degraded (7.055%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:46.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:46 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/499714493' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:46.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:46 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/499714493' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:46.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:46 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/499714493' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:46.722 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:54:47.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:47 smithi129 ceph-mon[88471]: pgmap v292: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 3772/52083 objects degraded (7.242%); 0 B/s, 44 objects/s recovering 2024-09-21T09:54:47.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:47 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:54:47.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:47 smithi129 ceph-mon[88471]: from='client.55069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:47.512 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:47 smithi107 ceph-mon[102221]: pgmap v292: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 3772/52083 objects degraded (7.242%); 0 B/s, 44 objects/s recovering 2024-09-21T09:54:47.512 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:47 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:54:47.512 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:47 smithi107 ceph-mon[102221]: from='client.55069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:47.512 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:47 smithi107 ceph-mon[105022]: pgmap v292: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 619 op/s; 3772/52083 objects degraded (7.242%); 0 B/s, 44 objects/s recovering 2024-09-21T09:54:47.512 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:47 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:54:47.512 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:47 smithi107 ceph-mon[105022]: from='client.55069 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:48.460 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:48 smithi129 ceph-mon[88471]: from='client.35349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:48 smithi107 ceph-mon[102221]: from='client.35349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:48.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:48 smithi107 ceph-mon[105022]: from='client.35349 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:49.552 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:49 smithi107 ceph-mon[102221]: pgmap v293: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1001 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 3772/48123 objects degraded (7.838%); 0 B/s, 39 objects/s recovering 2024-09-21T09:54:49.552 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:49 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 3772/48123 objects degraded (7.838%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:49.552 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:49 smithi107 ceph-mon[105022]: pgmap v293: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1001 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 3772/48123 objects degraded (7.838%); 0 B/s, 39 objects/s recovering 2024-09-21T09:54:49.552 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:49 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 3772/48123 objects degraded (7.838%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:49 smithi129 ceph-mon[88471]: pgmap v293: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1001 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 657 op/s; 3772/48123 objects degraded (7.838%); 0 B/s, 39 objects/s recovering 2024-09-21T09:54:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:49 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 3772/48123 objects degraded (7.838%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:50.364 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:54:50.514 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:50 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/350738381' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:50.514 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:50 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/350738381' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:54:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:54:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:54:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:50 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/350738381' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:51.580 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:51 smithi129 ceph-mon[88471]: pgmap v294: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 861 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 676 op/s; 3572/41361 objects degraded (8.636%); 0 B/s, 50 objects/s recovering 2024-09-21T09:54:51.580 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:51 smithi129 ceph-mon[88471]: from='client.55084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:51.581 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:51 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:54:51.581 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:51 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:54:51.581 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 09:54:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=cleanup t=2024-09-21T09:54:51.433490658Z level=info msg="Completed cleanup jobs" duration=11.785222ms 2024-09-21T09:54:51.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:51 smithi107 ceph-mon[102221]: pgmap v294: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 861 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 676 op/s; 3572/41361 objects degraded (8.636%); 0 B/s, 50 objects/s recovering 2024-09-21T09:54:51.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:51 smithi107 ceph-mon[102221]: from='client.55084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:51.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:51 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:54:51.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:51 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:54:51.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:51 smithi107 ceph-mon[105022]: pgmap v294: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 861 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 676 op/s; 3572/41361 objects degraded (8.636%); 0 B/s, 50 objects/s recovering 2024-09-21T09:54:51.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:51 smithi107 ceph-mon[105022]: from='client.55084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:51.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:51 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:54:51.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:51 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:54:52.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:54:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:54:51] "GET /metrics HTTP/1.1" 200 36518 "" "Prometheus/2.51.0" 2024-09-21T09:54:52.523 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:52 smithi107 ceph-mon[105022]: from='client.45155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:52.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:52 smithi107 ceph-mon[102221]: from='client.45155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:52.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:52 smithi129 ceph-mon[88471]: from='client.45155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:53.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:54:52 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:54:52.952+0000 7efdb5a17640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:54:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[102221]: pgmap v295: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 861 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 581 op/s; 3572/41361 objects degraded (8.636%); 0 B/s, 33 objects/s recovering 2024-09-21T09:54:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:54:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:54:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:54:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:54:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:54:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:54:53.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:54:53.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:54:53.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:54:53.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:54:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[105022]: pgmap v295: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 861 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 581 op/s; 3572/41361 objects degraded (8.636%); 0 B/s, 33 objects/s recovering 2024-09-21T09:54:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:54:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:54:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:54:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:54:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:54:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:54:53.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:54:53.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:54:53.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:54:53.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:53 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:54:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:53 smithi129 ceph-mon[88471]: pgmap v295: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 861 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 581 op/s; 3572/41361 objects degraded (8.636%); 0 B/s, 33 objects/s recovering 2024-09-21T09:54:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:53 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:54:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:53 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:54:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:53 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:54:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:53 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:54:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:53 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:54:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:53 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:54:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:53 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:54:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:53 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:54:53.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:53 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:54:53.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:53 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:54:54.027 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:54:54.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:54 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:54:54.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:54 smithi107 ceph-mon[102221]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:54:54.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:54 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2315503584' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:54.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:54 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 3572/41361 objects degraded (8.636%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:54.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:54 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:54:54.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:54 smithi107 ceph-mon[105022]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:54:54.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:54 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2315503584' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:54.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:54 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 3572/41361 objects degraded (8.636%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:54 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:54:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:54 smithi129 ceph-mon[88471]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:54:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:54 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2315503584' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:54 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 3572/41361 objects degraded (8.636%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:55.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:55 smithi129 ceph-mon[88471]: pgmap v296: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 676 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 827 op/s; 3379/32487 objects degraded (10.401%); 0 B/s, 49 objects/s recovering 2024-09-21T09:54:55.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:55 smithi129 ceph-mon[88471]: from='client.55096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:55.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:55 smithi129 ceph-mon[88471]: from='client.45167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:55 smithi107 ceph-mon[102221]: pgmap v296: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 676 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 827 op/s; 3379/32487 objects degraded (10.401%); 0 B/s, 49 objects/s recovering 2024-09-21T09:54:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:55 smithi107 ceph-mon[102221]: from='client.55096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:55 smithi107 ceph-mon[102221]: from='client.45167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:55.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:55 smithi107 ceph-mon[105022]: pgmap v296: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 676 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 827 op/s; 3379/32487 objects degraded (10.401%); 0 B/s, 49 objects/s recovering 2024-09-21T09:54:55.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:55 smithi107 ceph-mon[105022]: from='client.55096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:55.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:55 smithi107 ceph-mon[105022]: from='client.45167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:54:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:54:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:54:57.645 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:54:57.645 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:57 smithi107 ceph-mon[102221]: pgmap v297: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 650 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 616 op/s; 3379/31275 objects degraded (10.804%); 0 B/s, 32 objects/s recovering 2024-09-21T09:54:57.645 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:57 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1887837947' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:57.645 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:57 smithi107 ceph-mon[105022]: pgmap v297: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 650 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 616 op/s; 3379/31275 objects degraded (10.804%); 0 B/s, 32 objects/s recovering 2024-09-21T09:54:57.645 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:57 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1887837947' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:57.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:57 smithi129 ceph-mon[88471]: pgmap v297: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 650 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 616 op/s; 3379/31275 objects degraded (10.804%); 0 B/s, 32 objects/s recovering 2024-09-21T09:54:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:57 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1887837947' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:54:58.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:58 smithi129 ceph-mon[88471]: from='client.35385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:58 smithi107 ceph-mon[102221]: from='client.35385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:58 smithi107 ceph-mon[105022]: from='client.35385 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:59 smithi129 ceph-mon[88471]: pgmap v298: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 568 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 687 op/s; 3379/27330 objects degraded (12.364%); 0 B/s, 32 objects/s recovering 2024-09-21T09:54:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:59 smithi129 ceph-mon[88471]: from='client.35391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:54:59 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 3379/27330 objects degraded (12.364%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:59 smithi107 ceph-mon[102221]: pgmap v298: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 568 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 687 op/s; 3379/27330 objects degraded (12.364%); 0 B/s, 32 objects/s recovering 2024-09-21T09:54:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:59 smithi107 ceph-mon[102221]: from='client.35391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:54:59 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 3379/27330 objects degraded (12.364%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:54:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:59 smithi107 ceph-mon[105022]: pgmap v298: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 568 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 687 op/s; 3379/27330 objects degraded (12.364%); 0 B/s, 32 objects/s recovering 2024-09-21T09:54:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:59 smithi107 ceph-mon[105022]: from='client.35391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:54:59 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 3379/27330 objects degraded (12.364%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:55:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:55:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:55:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:55:01.299 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:55:01.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:01 smithi107 ceph-mon[102221]: pgmap v299: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 425 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 768 op/s; 3177/20454 objects degraded (15.532%); 0 B/s, 49 objects/s recovering 2024-09-21T09:55:01.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:01 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3626548980' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:01.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:01 smithi107 ceph-mon[105022]: pgmap v299: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 425 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 768 op/s; 3177/20454 objects degraded (15.532%); 0 B/s, 49 objects/s recovering 2024-09-21T09:55:01.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:01 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3626548980' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:01 smithi129 ceph-mon[88471]: pgmap v299: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 425 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 768 op/s; 3177/20454 objects degraded (15.532%); 0 B/s, 49 objects/s recovering 2024-09-21T09:55:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:01 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3626548980' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:02.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:55:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:55:01] "GET /metrics HTTP/1.1" 200 36508 "" "Prometheus/2.51.0" 2024-09-21T09:55:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:02 smithi129 ceph-mon[88471]: from='client.35394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:02 smithi107 ceph-mon[102221]: from='client.35394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:02.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:02 smithi107 ceph-mon[105022]: from='client.35394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:03 smithi107 ceph-mon[102221]: from='client.55126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:03 smithi107 ceph-mon[102221]: pgmap v300: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 425 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 580 op/s; 3177/20454 objects degraded (15.532%); 0 B/s, 32 objects/s recovering 2024-09-21T09:55:03.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:03 smithi107 ceph-mon[105022]: from='client.55126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:03.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:03 smithi107 ceph-mon[105022]: pgmap v300: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 425 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 580 op/s; 3177/20454 objects degraded (15.532%); 0 B/s, 32 objects/s recovering 2024-09-21T09:55:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:03 smithi129 ceph-mon[88471]: from='client.55126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:03 smithi129 ceph-mon[88471]: pgmap v300: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 425 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 580 op/s; 3177/20454 objects degraded (15.532%); 0 B/s, 32 objects/s recovering 2024-09-21T09:55:04.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:04 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 3177/20454 objects degraded (15.532%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:55:04.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:04 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2315608707' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:04.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:04 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 3177/20454 objects degraded (15.532%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:55:04.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:04 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2315608707' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:04 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 3177/20454 objects degraded (15.532%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:55:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:04 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2315608707' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:04.936 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:55:05.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:05 smithi107 ceph-mon[102221]: pgmap v301: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 225 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 2952/10854 objects degraded (27.197%); 0 B/s, 51 objects/s recovering 2024-09-21T09:55:05.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:05 smithi107 ceph-mon[105022]: pgmap v301: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 225 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 2952/10854 objects degraded (27.197%); 0 B/s, 51 objects/s recovering 2024-09-21T09:55:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:05 smithi129 ceph-mon[88471]: pgmap v301: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 225 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 846 op/s; 2952/10854 objects degraded (27.197%); 0 B/s, 51 objects/s recovering 2024-09-21T09:55:06.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:06 smithi107 ceph-mon[102221]: from='client.35409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:06.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:06 smithi107 ceph-mon[102221]: from='client.45203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:06.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:06 smithi107 ceph-mon[105022]: from='client.35409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:06.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:06 smithi107 ceph-mon[105022]: from='client.45203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:06 smithi129 ceph-mon[88471]: from='client.35409 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:06 smithi129 ceph-mon[88471]: from='client.45203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:07.175 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Removed 118704 objects 2024-09-21T09:55:07.175 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Clean up completed and total clean up time :163.884 2024-09-21T09:55:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:07 smithi107 ceph-mon[102221]: pgmap v302: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 198 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 637 op/s; 2952/9540 objects degraded (30.943%); 0 B/s, 35 objects/s recovering 2024-09-21T09:55:07.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:07 smithi107 ceph-mon[105022]: pgmap v302: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 198 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 637 op/s; 2952/9540 objects degraded (30.943%); 0 B/s, 35 objects/s recovering 2024-09-21T09:55:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:07 smithi129 ceph-mon[88471]: pgmap v302: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 198 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 637 op/s; 2952/9540 objects degraded (30.943%); 0 B/s, 35 objects/s recovering 2024-09-21T09:55:08.120 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:55:08.359 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:55:07 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:55:07.963+0000 7efdb5a17640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:55:08.553 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-21T09:55:08.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-21T09:55:08.820 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:08 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3330774826' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:08.820 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:08 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:55:08.821 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:08 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3330774826' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:08 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:55:08.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:08 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3330774826' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:08.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:08 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:55:09.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:09 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:55:09.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:09 smithi107 ceph-mon[102221]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:55:09.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:09 smithi107 ceph-mon[102221]: pgmap v303: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 109 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 721 op/s; 2952/5298 objects degraded (55.719%); 0 B/s, 35 objects/s recovering 2024-09-21T09:55:09.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:09 smithi107 ceph-mon[102221]: from='client.35421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:09.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:09 smithi107 ceph-mon[102221]: from='client.45221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:09.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:09 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2952/5298 objects degraded (55.719%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:55:09.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:09 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:55:09.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:09 smithi107 ceph-mon[105022]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:55:09.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:09 smithi107 ceph-mon[105022]: pgmap v303: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 109 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 721 op/s; 2952/5298 objects degraded (55.719%); 0 B/s, 35 objects/s recovering 2024-09-21T09:55:09.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:09 smithi107 ceph-mon[105022]: from='client.35421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:09.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:09 smithi107 ceph-mon[105022]: from='client.45221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:09.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:09 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2952/5298 objects degraded (55.719%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:55:09.839 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:09 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:55:09.839 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:09 smithi129 ceph-mon[88471]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-09-21T09:55:09.839 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:09 smithi129 ceph-mon[88471]: pgmap v303: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 109 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 721 op/s; 2952/5298 objects degraded (55.719%); 0 B/s, 35 objects/s recovering 2024-09-21T09:55:09.839 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:09 smithi129 ceph-mon[88471]: from='client.35421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:09.839 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:09 smithi129 ceph-mon[88471]: from='client.45221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:09.839 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:09 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2952/5298 objects degraded (55.719%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:55:10.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:55:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:55:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:55:10.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:10 smithi107 ceph-mon[102221]: osdmap e116: 8 total, 8 up, 8 in 2024-09-21T09:55:10.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:10 smithi107 ceph-mon[105022]: osdmap e116: 8 total, 8 up, 8 in 2024-09-21T09:55:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:10 smithi129 ceph-mon[88471]: osdmap e116: 8 total, 8 up, 8 in 2024-09-21T09:55:11.122 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:55:11.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:11 smithi107 ceph-mon[102221]: pgmap v305: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 2.3 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 676 op/s; 2435/162 objects degraded (1503.086%); 0 B/s, 74 objects/s recovering 2024-09-21T09:55:11.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:11 smithi107 ceph-mon[102221]: osdmap e117: 8 total, 8 up, 8 in 2024-09-21T09:55:11.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:11 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/883847867' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:11.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:55:11.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:11 smithi107 ceph-mon[105022]: pgmap v305: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 2.3 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 676 op/s; 2435/162 objects degraded (1503.086%); 0 B/s, 74 objects/s recovering 2024-09-21T09:55:11.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:11 smithi107 ceph-mon[105022]: osdmap e117: 8 total, 8 up, 8 in 2024-09-21T09:55:11.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:11 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/883847867' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:11.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:55:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:11 smithi129 ceph-mon[88471]: pgmap v305: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 2.3 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 676 op/s; 2435/162 objects degraded (1503.086%); 0 B/s, 74 objects/s recovering 2024-09-21T09:55:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:11 smithi129 ceph-mon[88471]: osdmap e117: 8 total, 8 up, 8 in 2024-09-21T09:55:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:11 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/883847867' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:55:12.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:55:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:55:11] "GET /metrics HTTP/1.1" 200 36505 "" "Prometheus/2.51.0" 2024-09-21T09:55:12.585 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_2' removed 2024-09-21T09:55:12.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:12 smithi107 ceph-mon[102221]: from='client.45233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:12.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:12 smithi107 ceph-mon[102221]: from='client.45239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:12.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:12 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/419576294' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T09:55:12.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:12 smithi107 ceph-mon[105022]: from='client.45233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:12.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:12 smithi107 ceph-mon[105022]: from='client.45239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:12.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:12 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/419576294' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T09:55:12.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:12 smithi129 ceph-mon[88471]: from='client.45233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:12.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:12 smithi129 ceph-mon[88471]: from='client.45239 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:12.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:12 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/419576294' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T09:55:13.472 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T09:55:13.472 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T09:55:13.472 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-21T09:55:13.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_3 16 2024-09-21T09:55:13.730 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:13.851 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:13 smithi107 ceph-mon[102221]: pgmap v307: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 2.3 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 445 op/s; 2435/162 objects degraded (1503.086%); 0 B/s, 64 objects/s recovering 2024-09-21T09:55:13.851 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:13 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/419576294' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T09:55:13.851 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:13 smithi107 ceph-mon[102221]: osdmap e118: 8 total, 8 up, 8 in 2024-09-21T09:55:13.851 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:13 smithi107 ceph-mon[105022]: pgmap v307: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 2.3 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 445 op/s; 2435/162 objects degraded (1503.086%); 0 B/s, 64 objects/s recovering 2024-09-21T09:55:13.851 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:13 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/419576294' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T09:55:13.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:13 smithi107 ceph-mon[105022]: osdmap e118: 8 total, 8 up, 8 in 2024-09-21T09:55:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:13 smithi129 ceph-mon[88471]: pgmap v307: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 2.3 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 445 op/s; 2435/162 objects degraded (1503.086%); 0 B/s, 64 objects/s recovering 2024-09-21T09:55:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:13 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/419576294' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T09:55:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:13 smithi129 ceph-mon[88471]: osdmap e118: 8 total, 8 up, 8 in 2024-09-21T09:55:14.088 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:55:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:14 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/391516604' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:14 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2435/162 objects degraded (1503.086%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:55:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:14 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/391516604' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:14 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2435/162 objects degraded (1503.086%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:55:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:14 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/391516604' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:14 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2435/162 objects degraded (1503.086%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:55:15.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:15 smithi107 ceph-mon[105022]: pgmap v309: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:15.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:15 smithi107 ceph-mon[105022]: from='client.45257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:15.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:15 smithi107 ceph-mon[105022]: from='client.45263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:15.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:15 smithi107 ceph-mon[105022]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2435/162 objects degraded (1503.086%), 2 pgs degraded, 2 pgs undersized) 2024-09-21T09:55:15.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:15 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T09:55:15.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:15 smithi107 ceph-mon[102221]: pgmap v309: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:15.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:15 smithi107 ceph-mon[102221]: from='client.45257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:15.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:15 smithi107 ceph-mon[102221]: from='client.45263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:15.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:15 smithi107 ceph-mon[102221]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2435/162 objects degraded (1503.086%), 2 pgs degraded, 2 pgs undersized) 2024-09-21T09:55:15.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:15 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T09:55:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:15 smithi129 ceph-mon[88471]: pgmap v309: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:15 smithi129 ceph-mon[88471]: from='client.45257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:15 smithi129 ceph-mon[88471]: from='client.45263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:15 smithi129 ceph-mon[88471]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2435/162 objects degraded (1503.086%), 2 pgs degraded, 2 pgs undersized) 2024-09-21T09:55:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:15 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T09:55:16.487 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_3' created 2024-09-21T09:55:16.804 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:16 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2913617511' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T09:55:16.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:16 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T09:55:16.805 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:16 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2913617511' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T09:55:16.805 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:16 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T09:55:16.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:16 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2913617511' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T09:55:16.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:16 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T09:55:16.958 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:55:17.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-21T09:55:17.340 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:17.742 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:17 smithi107 ceph-mon[102221]: pgmap v310: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:17.742 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:17 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T09:55:17.742 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:17 smithi107 ceph-mon[102221]: osdmap e119: 8 total, 8 up, 8 in 2024-09-21T09:55:17.742 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:17 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1676922259' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:17.743 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:17 smithi107 ceph-mon[105022]: pgmap v310: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:17.743 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:17 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T09:55:17.743 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:17 smithi107 ceph-mon[105022]: osdmap e119: 8 total, 8 up, 8 in 2024-09-21T09:55:17.743 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:17 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1676922259' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:17 smithi129 ceph-mon[88471]: pgmap v310: 73 pgs: 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:17 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T09:55:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:17 smithi129 ceph-mon[88471]: osdmap e119: 8 total, 8 up, 8 in 2024-09-21T09:55:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:17 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1676922259' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:18.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:18 smithi107 ceph-mon[102221]: from='client.35487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:18.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:18 smithi107 ceph-mon[102221]: from='client.45287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:18.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:18 smithi107 ceph-mon[102221]: osdmap e120: 8 total, 8 up, 8 in 2024-09-21T09:55:18.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:18 smithi107 ceph-mon[102221]: osdmap e121: 8 total, 8 up, 8 in 2024-09-21T09:55:18.767 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:18 smithi107 ceph-mon[105022]: from='client.35487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:18.767 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:18 smithi107 ceph-mon[105022]: from='client.45287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:18.767 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:18 smithi107 ceph-mon[105022]: osdmap e120: 8 total, 8 up, 8 in 2024-09-21T09:55:18.767 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:18 smithi107 ceph-mon[105022]: osdmap e121: 8 total, 8 up, 8 in 2024-09-21T09:55:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:18 smithi129 ceph-mon[88471]: from='client.35487 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:18 smithi129 ceph-mon[88471]: from='client.45287 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:18 smithi129 ceph-mon[88471]: osdmap e120: 8 total, 8 up, 8 in 2024-09-21T09:55:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:18 smithi129 ceph-mon[88471]: osdmap e121: 8 total, 8 up, 8 in 2024-09-21T09:55:19.510 INFO:teuthology.orchestra.run.smithi107.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-21T09:55:19.815 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:19 smithi107 ceph-mon[102221]: pgmap v313: 89 pgs: 6 creating+peering, 10 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:19.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:19 smithi107 ceph-mon[102221]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:55:19.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:19 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2338818682' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T09:55:19.816 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:19 smithi107 ceph-mon[105022]: pgmap v313: 89 pgs: 6 creating+peering, 10 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:19.816 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:19 smithi107 ceph-mon[105022]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:55:19.816 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:19 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2338818682' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T09:55:19.817 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:55:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:19 smithi129 ceph-mon[88471]: pgmap v313: 89 pgs: 6 creating+peering, 10 unknown, 73 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:19 smithi129 ceph-mon[88471]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:55:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:19 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2338818682' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T09:55:20.507 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:55:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:55:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:55:20.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:20 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/217555966' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:20.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:20 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2338818682' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:55:20.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:20 smithi107 ceph-mon[105022]: osdmap e122: 8 total, 8 up, 8 in 2024-09-21T09:55:20.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:20 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/217555966' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:20.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:20 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2338818682' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:55:20.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:20 smithi107 ceph-mon[102221]: osdmap e122: 8 total, 8 up, 8 in 2024-09-21T09:55:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:20 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/217555966' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:20.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:20 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2338818682' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:55:20.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:20 smithi129 ceph-mon[88471]: osdmap e122: 8 total, 8 up, 8 in 2024-09-21T09:55:21.187 DEBUG:teuthology.orchestra.run.smithi107:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-21T09:55:21.190 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T09:55:21.190 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T09:55:21.240 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:hints = 1 2024-09-21T09:55:21.240 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T09:55:21.240 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object prefix: benchmark_data_smithi107_135256 2024-09-21T09:55:21.240 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T09:55:21.241 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 0 0 0 0 0 0 - 0 2024-09-21T09:55:21.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:21 smithi107 ceph-mon[102221]: from='client.45305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:21.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:21 smithi107 ceph-mon[102221]: pgmap v316: 89 pgs: 6 creating+peering, 83 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:21.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:21 smithi107 ceph-mon[102221]: from='client.35517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:21.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:21 smithi107 ceph-mon[102221]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:55:21.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:21 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T09:55:21.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:21.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:21 smithi107 ceph-mon[105022]: from='client.45305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:21.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:21 smithi107 ceph-mon[105022]: pgmap v316: 89 pgs: 6 creating+peering, 83 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:21.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:21 smithi107 ceph-mon[105022]: from='client.35517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:21.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:21 smithi107 ceph-mon[105022]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:55:21.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:21 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T09:55:21.772 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:21 smithi129 ceph-mon[88471]: from='client.45305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:21 smithi129 ceph-mon[88471]: pgmap v316: 89 pgs: 6 creating+peering, 83 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:21 smithi129 ceph-mon[88471]: from='client.35517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:21 smithi129 ceph-mon[88471]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:55:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:21 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T09:55:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:22.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:55:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:55:21] "GET /metrics HTTP/1.1" 200 36505 "" "Prometheus/2.51.0" 2024-09-21T09:55:22.241 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1 16 4092 4076 254.733 254.75 0.00530611 0.00391212 2024-09-21T09:55:23.241 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2 16 7143 7127 222.704 190.688 0.00297879 0.00447595 2024-09-21T09:55:23.306 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:55:23.306 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:55:22 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:55:22.979+0000 7efdb5a17640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-21T09:55:23.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:23 smithi107 ceph-mon[102221]: pgmap v317: 89 pgs: 6 creating+peering, 83 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:23.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:23 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3758200244' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:23.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:23 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:55:23.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:23 smithi107 ceph-mon[105022]: pgmap v317: 89 pgs: 6 creating+peering, 83 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:23.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:23 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3758200244' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:23.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:23 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:55:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:23 smithi129 ceph-mon[88471]: pgmap v317: 89 pgs: 6 creating+peering, 83 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:23 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3758200244' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:23 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:55:24.241 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3 16 10088 10072 209.818 184.062 0.0050818 0.0047564 2024-09-21T09:55:24.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:24 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:55:24.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:24 smithi107 ceph-mon[102221]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-21T09:55:24.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:24 smithi107 ceph-mon[102221]: from='client.55231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:24.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:24 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:55:24.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:24 smithi107 ceph-mon[105022]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-21T09:55:24.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:24 smithi107 ceph-mon[105022]: from='client.55231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:24.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:24 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:55:24.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:24 smithi129 ceph-mon[88471]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-21T09:55:24.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:24 smithi129 ceph-mon[88471]: from='client.55231 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:25.241 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 4 16 13509 13493 210.812 213.812 0.00870282 0.00473448 2024-09-21T09:55:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:25 smithi107 ceph-mon[102221]: from='client.45335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:25 smithi107 ceph-mon[102221]: pgmap v318: 89 pgs: 89 active+clean; 399 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 61 MiB/s wr, 976 op/s 2024-09-21T09:55:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:25 smithi107 ceph-mon[105022]: from='client.45335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:25 smithi107 ceph-mon[105022]: pgmap v318: 89 pgs: 89 active+clean; 399 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 61 MiB/s wr, 976 op/s 2024-09-21T09:55:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:25 smithi129 ceph-mon[88471]: from='client.45335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:25 smithi129 ceph-mon[88471]: pgmap v318: 89 pgs: 89 active+clean; 399 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 61 MiB/s wr, 976 op/s 2024-09-21T09:55:26.241 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 5 16 16622 16606 207.559 194.562 0.00410184 0.00481228 2024-09-21T09:55:27.018 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:55:27.241 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 6 16 19378 19362 201.671 172.25 0.00582578 0.0049527 2024-09-21T09:55:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:27 smithi107 ceph-mon[102221]: pgmap v319: 89 pgs: 89 active+clean; 399 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 50 MiB/s wr, 795 op/s 2024-09-21T09:55:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:55:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:27 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/146702902' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:27 smithi107 ceph-mon[105022]: pgmap v319: 89 pgs: 89 active+clean; 399 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 50 MiB/s wr, 795 op/s 2024-09-21T09:55:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:27.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:55:27.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:27 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/146702902' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:27 smithi129 ceph-mon[88471]: pgmap v319: 89 pgs: 89 active+clean; 399 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 50 MiB/s wr, 795 op/s 2024-09-21T09:55:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:55:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:27 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/146702902' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:28.241 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 7 16 22206 22190 198.109 176.75 0.00614545 0.00504257 2024-09-21T09:55:28.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:28 smithi129 ceph-mon[88471]: from='client.55246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:28.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:28 smithi129 ceph-mon[88471]: from='client.35553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:28.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:28 smithi107 ceph-mon[102221]: from='client.55246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:28.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:28 smithi107 ceph-mon[102221]: from='client.35553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:28.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:28 smithi107 ceph-mon[105022]: from='client.55246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:28.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:28 smithi107 ceph-mon[105022]: from='client.35553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:29.241 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 8 16 25030 25014 195.406 176.5 0.00804046 0.00511133 2024-09-21T09:55:29.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:29 smithi129 ceph-mon[88471]: pgmap v320: 89 pgs: 89 active+clean; 754 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-21T09:55:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:29 smithi107 ceph-mon[102221]: pgmap v320: 89 pgs: 89 active+clean; 754 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-21T09:55:29.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:29 smithi107 ceph-mon[105022]: pgmap v320: 89 pgs: 89 active+clean; 754 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-21T09:55:30.241 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 9 16 27520 27504 190.984 155.625 0.00982096 0.0052304 2024-09-21T09:55:30.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:30 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3988879137' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:30.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:55:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:55:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:55:30.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:30 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3988879137' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:30.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:30 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3988879137' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:30.708 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:55:31.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 10 16 29657 29641 185.241 133.562 0.00422222 0.00539115 2024-09-21T09:55:31.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:31 smithi129 ceph-mon[88471]: pgmap v321: 89 pgs: 89 active+clean; 1.3 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-09-21T09:55:31.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:31 smithi129 ceph-mon[88471]: from='client.35565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:31.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:31 smithi107 ceph-mon[102221]: pgmap v321: 89 pgs: 89 active+clean; 1.3 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-09-21T09:55:31.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:31 smithi107 ceph-mon[102221]: from='client.35565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:31.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:31 smithi107 ceph-mon[105022]: pgmap v321: 89 pgs: 89 active+clean; 1.3 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 128 MiB/s wr, 2.05k op/s 2024-09-21T09:55:31.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:31 smithi107 ceph-mon[105022]: from='client.35565 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:55:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:55:31] "GET /metrics HTTP/1.1" 200 36505 "" "Prometheus/2.51.0" 2024-09-21T09:55:32.241 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 11 16 32082 32066 182.178 151.562 0.0070075 0.00548325 2024-09-21T09:55:32.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:32 smithi107 ceph-mon[102221]: from='client.45365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:32.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:32 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:32.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:32 smithi107 ceph-mon[105022]: from='client.45365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:32.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:32 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:32.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:32 smithi129 ceph-mon[88471]: from='client.45365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:32.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:32 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:33.241 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 12 16 34329 34313 178.698 140.438 0.00624729 0.00559088 2024-09-21T09:55:33.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:33 smithi107 ceph-mon[102221]: pgmap v322: 89 pgs: 89 active+clean; 1.3 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-21T09:55:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:33 smithi107 ceph-mon[105022]: pgmap v322: 89 pgs: 89 active+clean; 1.3 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-21T09:55:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:33 smithi129 ceph-mon[88471]: pgmap v322: 89 pgs: 89 active+clean; 1.3 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-21T09:55:34.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 13 16 36322 36306 174.533 124.562 0.00543837 0.00572362 2024-09-21T09:55:34.297 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:55:34.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:34 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1407158219' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:34.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:34 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1407158219' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:34.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:34 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1407158219' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:35.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 14 16 38293 38277 170.865 123.188 0.00414874 0.00584731 2024-09-21T09:55:35.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:35 smithi107 ceph-mon[102221]: pgmap v323: 89 pgs: 89 active+clean; 2.1 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 176 MiB/s wr, 2.81k op/s 2024-09-21T09:55:35.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:35 smithi107 ceph-mon[102221]: from='client.35580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:35.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:35 smithi107 ceph-mon[105022]: pgmap v323: 89 pgs: 89 active+clean; 2.1 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 176 MiB/s wr, 2.81k op/s 2024-09-21T09:55:35.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:35 smithi107 ceph-mon[105022]: from='client.35580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:35 smithi129 ceph-mon[88471]: pgmap v323: 89 pgs: 89 active+clean; 2.1 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 176 MiB/s wr, 2.81k op/s 2024-09-21T09:55:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:35 smithi129 ceph-mon[88471]: from='client.35580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:36.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 15 16 40515 40499 168.731 138.875 0.00513704 0.00592139 2024-09-21T09:55:36.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:36 smithi107 ceph-mon[102221]: from='client.55282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:36.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:36 smithi107 ceph-mon[105022]: from='client.55282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:36.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:36 smithi129 ceph-mon[88471]: from='client.55282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:37.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 16 16 42453 42437 165.755 121.125 0.0167325 0.00602818 2024-09-21T09:55:37.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:37 smithi107 ceph-mon[102221]: pgmap v324: 89 pgs: 89 active+clean; 2.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 143 MiB/s wr, 2.28k op/s 2024-09-21T09:55:37.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:37 smithi107 ceph-mon[105022]: pgmap v324: 89 pgs: 89 active+clean; 2.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 143 MiB/s wr, 2.28k op/s 2024-09-21T09:55:37.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:37 smithi129 ceph-mon[88471]: pgmap v324: 89 pgs: 89 active+clean; 2.1 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 143 MiB/s wr, 2.28k op/s 2024-09-21T09:55:37.904 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:55:38.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 17 16 44133 44117 162.181 105 0.0203193 0.00615914 2024-09-21T09:55:38.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:38 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2896075600' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:38.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:55:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:38 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2896075600' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:55:38.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:38 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2896075600' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:38.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T09:55:39.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 18 16 45448 45432 157.736 82.1875 0.0123895 0.00633402 2024-09-21T09:55:39.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:39 smithi107 ceph-mon[102221]: from='client.55291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:39.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:39 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:55:39.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:39 smithi107 ceph-mon[102221]: Upgrade: osd.3 is safe to restart 2024-09-21T09:55:39.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:39 smithi107 ceph-mon[102221]: pgmap v325: 89 pgs: 89 active+clean; 2.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-09-21T09:55:39.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:39 smithi107 ceph-mon[102221]: from='client.45392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:39.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:39 smithi107 ceph-mon[105022]: from='client.55291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:39.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:39 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:55:39.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:39 smithi107 ceph-mon[105022]: Upgrade: osd.3 is safe to restart 2024-09-21T09:55:39.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:39 smithi107 ceph-mon[105022]: pgmap v325: 89 pgs: 89 active+clean; 2.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-09-21T09:55:39.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:39 smithi107 ceph-mon[105022]: from='client.45392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:39 smithi129 ceph-mon[88471]: from='client.55291 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:39 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T09:55:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:39 smithi129 ceph-mon[88471]: Upgrade: osd.3 is safe to restart 2024-09-21T09:55:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:39 smithi129 ceph-mon[88471]: pgmap v325: 89 pgs: 89 active+clean; 2.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-09-21T09:55:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:39 smithi129 ceph-mon[88471]: from='client.45392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:40.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 19 16 47388 47372 155.815 121.25 0.00500647 0.00641194 2024-09-21T09:55:40.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:55:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:55:40.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:55:41.155 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:55:41.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T09:55:41.241442+0000 min lat: 0.00118684 max lat: 0.270817 avg lat: 0.00655644 2024-09-21T09:55:41.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T09:55:41.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 20 16 48320 48304 150.937 58.25 0.00199425 0.00655644 2024-09-21T09:55:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:41 smithi107 ceph-mon[102221]: pgmap v326: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 165 MiB/s wr, 2.63k op/s 2024-09-21T09:55:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:41 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/750917928' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:55:41.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:41 smithi107 ceph-mon[105022]: pgmap v326: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 165 MiB/s wr, 2.63k op/s 2024-09-21T09:55:41.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:41 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/750917928' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:41.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:55:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:41 smithi129 ceph-mon[88471]: pgmap v326: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 165 MiB/s wr, 2.63k op/s 2024-09-21T09:55:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:41 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/750917928' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:55:42.065 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:55:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:55:41] "GET /metrics HTTP/1.1" 200 36511 "" "Prometheus/2.51.0" 2024-09-21T09:55:42.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 21 16 50099 50083 149.043 111.188 0.0050947 0.00670458 2024-09-21T09:55:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:42 smithi107 ceph-mon[102221]: from='client.45404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:42 smithi107 ceph-mon[102221]: from='client.35613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T09:55:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:55:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:42 smithi107 ceph-mon[105022]: from='client.45404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:42 smithi107 ceph-mon[105022]: from='client.35613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:42.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T09:55:42.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:55:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:42 smithi129 ceph-mon[88471]: from='client.45404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:42 smithi129 ceph-mon[88471]: from='client.35613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T09:55:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:55:43.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 22 16 51860 51844 147.271 110.062 0.00709869 0.00678506 2024-09-21T09:55:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:43 smithi107 ceph-mon[102221]: pgmap v327: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-09-21T09:55:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:43 smithi107 ceph-mon[102221]: Upgrade: Updating osd.3 2024-09-21T09:55:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:43 smithi107 ceph-mon[102221]: Deploying daemon osd.3 on smithi107 2024-09-21T09:55:43.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:43 smithi107 ceph-mon[105022]: pgmap v327: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-09-21T09:55:43.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:43 smithi107 ceph-mon[105022]: Upgrade: Updating osd.3 2024-09-21T09:55:43.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:43 smithi107 ceph-mon[105022]: Deploying daemon osd.3 on smithi107 2024-09-21T09:55:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:43 smithi129 ceph-mon[88471]: pgmap v327: 89 pgs: 89 active+clean; 2.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-09-21T09:55:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:43 smithi129 ceph-mon[88471]: Upgrade: Updating osd.3 2024-09-21T09:55:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:43 smithi129 ceph-mon[88471]: Deploying daemon osd.3 on smithi107 2024-09-21T09:55:44.242 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 23 16 53546 53530 145.449 105.375 0.00717737 0.00687093 2024-09-21T09:55:44.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:44 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4176536765' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:44.489 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:44 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4176536765' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:44 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4176536765' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:44.740 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:55:45.243 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 24 16 54989 54973 143.146 90.1875 0.009038 0.00697995 2024-09-21T09:55:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:45 smithi129 ceph-mon[88471]: pgmap v328: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 156 MiB/s wr, 2.49k op/s 2024-09-21T09:55:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:45 smithi129 ceph-mon[88471]: from='client.55315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:45.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:45 smithi107 ceph-mon[102221]: pgmap v328: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 156 MiB/s wr, 2.49k op/s 2024-09-21T09:55:45.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:45 smithi107 ceph-mon[102221]: from='client.55315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:45.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:45 smithi107 ceph-mon[105022]: pgmap v328: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 156 MiB/s wr, 2.49k op/s 2024-09-21T09:55:45.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:45 smithi107 ceph-mon[105022]: from='client.55315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:46.243 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 25 15 56708 56693 141.72 107.5 0.00347577 0.00705083 2024-09-21T09:55:46.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:46 smithi107 ceph-mon[105022]: from='client.35625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:46.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:46 smithi107 ceph-mon[102221]: from='client.35625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:46.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:46 smithi129 ceph-mon[88471]: from='client.35625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:47.243 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 26 16 58339 58323 140.187 101.875 0.0042378 0.00712687 2024-09-21T09:55:47.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:47 smithi107 ceph-mon[105022]: pgmap v329: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-21T09:55:47.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:47 smithi107 ceph-mon[102221]: pgmap v329: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-21T09:55:47.609 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:47 smithi107 systemd[1]: Stopping Ceph osd.3 for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:55:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:47 smithi129 ceph-mon[88471]: pgmap v329: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-21T09:55:48.235 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:55:48.243 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 27 16 59495 59479 137.671 72.25 0.00350195 0.00724488 2024-09-21T09:55:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:48 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2324880792' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:48.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:48 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2324880792' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:48.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:48 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2324880792' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:49.243 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 28 16 60595 60579 135.209 68.75 0.00733234 0.00739069 2024-09-21T09:55:49.359 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:49 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3[64322]: 2024-09-21T09:55:49.001+0000 7f898096f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T09:55:49.359 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:49 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3[64322]: 2024-09-21T09:55:49.001+0000 7f898096f640 -1 osd.3 122 *** Got signal Terminated *** 2024-09-21T09:55:49.359 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:49 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3[64322]: 2024-09-21T09:55:49.001+0000 7f898096f640 -1 osd.3 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T09:55:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:49 smithi129 ceph-mon[88471]: pgmap v330: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-21T09:55:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:49 smithi129 ceph-mon[88471]: from='client.45431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:49 smithi129 ceph-mon[88471]: from='client.35637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:49 smithi129 ceph-mon[88471]: osd.3 marked itself down and dead 2024-09-21T09:55:49.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:49 smithi107 ceph-mon[105022]: pgmap v330: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-21T09:55:49.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:49 smithi107 ceph-mon[105022]: from='client.45431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:49.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:49 smithi107 ceph-mon[105022]: from='client.35637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:49.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:49 smithi107 ceph-mon[105022]: osd.3 marked itself down and dead 2024-09-21T09:55:49.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:49 smithi107 ceph-mon[102221]: pgmap v330: 89 pgs: 89 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-21T09:55:49.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:49 smithi107 ceph-mon[102221]: from='client.45431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:49.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:49 smithi107 ceph-mon[102221]: from='client.35637 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:49.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:49 smithi107 ceph-mon[102221]: osd.3 marked itself down and dead 2024-09-21T09:55:50.243 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 29 16 60904 60888 131.212 19.3125 0.00276292 0.00740236 2024-09-21T09:55:50.359 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:50 smithi107 podman[136216]: 2024-09-21 09:55:50.061406615 +0000 UTC m=+2.539661598 container died cf6bac271e88244e743fd5a5edb189cda9e68f4bf29b3322fae3946943b89480 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3, RELEASE=quincy-fbd1a6a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-21T09:55:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:50 smithi129 ceph-mon[88471]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T09:55:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:50 smithi129 ceph-mon[88471]: osdmap e123: 8 total, 7 up, 8 in 2024-09-21T09:55:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:55:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:55:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:55:50.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:50 smithi107 ceph-mon[102221]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T09:55:50.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:50 smithi107 ceph-mon[102221]: osdmap e123: 8 total, 7 up, 8 in 2024-09-21T09:55:50.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:50 smithi107 ceph-mon[105022]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T09:55:50.854 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:50 smithi107 ceph-mon[105022]: osdmap e123: 8 total, 7 up, 8 in 2024-09-21T09:55:50.854 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:50 smithi107 podman[136216]: 2024-09-21 09:55:50.599977971 +0000 UTC m=+3.078232941 container cleanup cf6bac271e88244e743fd5a5edb189cda9e68f4bf29b3322fae3946943b89480 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True) 2024-09-21T09:55:50.854 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:50 smithi107 bash[136216]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3 2024-09-21T09:55:51.109 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:50 smithi107 podman[136286]: 2024-09-21 09:55:50.851946475 +0000 UTC m=+0.749012400 container remove cf6bac271e88244e743fd5a5edb189cda9e68f4bf29b3322fae3946943b89480 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T09:55:51.243 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 30 16 62353 62337 129.857 90.5625 0.00569288 0.0076958 2024-09-21T09:55:51.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:51 smithi129 ceph-mon[88471]: pgmap v332: 89 pgs: 16 stale+active+clean, 73 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-09-21T09:55:51.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:51 smithi129 ceph-mon[88471]: osdmap e124: 8 total, 7 up, 8 in 2024-09-21T09:55:51.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:51 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3357630899' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:51.714 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:55:51.714 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:51 smithi107 ceph-mon[102221]: pgmap v332: 89 pgs: 16 stale+active+clean, 73 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-09-21T09:55:51.714 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:51 smithi107 ceph-mon[102221]: osdmap e124: 8 total, 7 up, 8 in 2024-09-21T09:55:51.715 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:51 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3357630899' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:51.715 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:51 smithi107 ceph-mon[105022]: pgmap v332: 89 pgs: 16 stale+active+clean, 73 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-09-21T09:55:51.715 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:51 smithi107 ceph-mon[105022]: osdmap e124: 8 total, 7 up, 8 in 2024-09-21T09:55:51.715 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:51 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3357630899' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:51.715 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:51 smithi107 podman[136413]: 2024-09-21 09:55:51.47118699 +0000 UTC m=+0.034844489 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:55:51.715 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:51 smithi107 podman[136413]: 2024-09-21 09:55:51.603259712 +0000 UTC m=+0.166917200 container create 6ffe807d21e1c304495c6f312f9daa53154aa98b5bf6455e46eb0da755d04e6a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T09:55:51.991 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:51 smithi107 podman[136413]: 2024-09-21 09:55:51.882013377 +0000 UTC m=+0.445670860 container init 6ffe807d21e1c304495c6f312f9daa53154aa98b5bf6455e46eb0da755d04e6a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-deactivate, RELEASE=main-62da54b, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-21T09:55:51.992 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:51 smithi107 podman[136413]: 2024-09-21 09:55:51.891451814 +0000 UTC m=+0.455109301 container start 6ffe807d21e1c304495c6f312f9daa53154aa98b5bf6455e46eb0da755d04e6a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.build-date=20240731, RELEASE=main-62da54b, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T09:55:51.992 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:55:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:55:51] "GET /metrics HTTP/1.1" 200 36511 "" "Prometheus/2.51.0" 2024-09-21T09:55:52.243 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 31 16 64043 64027 129.075 105.625 0.00556721 0.00774271 2024-09-21T09:55:52.359 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:51 smithi107 podman[136413]: 2024-09-21 09:55:51.990207605 +0000 UTC m=+0.553865088 container attach 6ffe807d21e1c304495c6f312f9daa53154aa98b5bf6455e46eb0da755d04e6a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-deactivate, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-21T09:55:52.359 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:52 smithi107 podman[136413]: 2024-09-21 09:55:52.195370851 +0000 UTC m=+0.759028351 container died 6ffe807d21e1c304495c6f312f9daa53154aa98b5bf6455e46eb0da755d04e6a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T09:55:52.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:52 smithi107 ceph-mon[102221]: from='client.35646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:52.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:52 smithi107 ceph-mon[105022]: from='client.35646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:52 smithi129 ceph-mon[88471]: from='client.35646 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:53.109 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:52 smithi107 podman[136475]: 2024-09-21 09:55:52.828481693 +0000 UTC m=+0.625184835 container remove 6ffe807d21e1c304495c6f312f9daa53154aa98b5bf6455e46eb0da755d04e6a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-21T09:55:53.109 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:52 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.3.service: Deactivated successfully. 2024-09-21T09:55:53.109 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:52 smithi107 systemd[1]: Stopped Ceph osd.3 for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:55:53.109 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:52 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.3.service: Consumed 22min 44.159s CPU time. 2024-09-21T09:55:53.243 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 32 16 65809 65793 128.491 110.375 0.00539399 0.00777747 2024-09-21T09:55:53.432 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:53 smithi107 systemd[1]: Starting Ceph osd.3 for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T09:55:53.743 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:53 smithi107 ceph-mon[102221]: pgmap v334: 89 pgs: 16 stale+active+clean, 73 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 60 MiB/s wr, 962 op/s 2024-09-21T09:55:53.743 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:53 smithi107 ceph-mon[102221]: from='client.35652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:53.744 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:53 smithi107 ceph-mon[105022]: pgmap v334: 89 pgs: 16 stale+active+clean, 73 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 60 MiB/s wr, 962 op/s 2024-09-21T09:55:53.744 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:53 smithi107 ceph-mon[105022]: from='client.35652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:53.744 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:53 smithi107 podman[136574]: 2024-09-21 09:55:53.343875906 +0000 UTC m=+0.029178202 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:55:53.744 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:53 smithi107 podman[136574]: 2024-09-21 09:55:53.479989331 +0000 UTC m=+0.165291620 container create f473d6bcb438bdb7df23695c57e8aa724b98459bb469c23016a4d000ae9e14eb (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-activate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-21T09:55:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:53 smithi129 ceph-mon[88471]: pgmap v334: 89 pgs: 16 stale+active+clean, 73 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 60 MiB/s wr, 962 op/s 2024-09-21T09:55:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:53 smithi129 ceph-mon[88471]: from='client.35652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:54.109 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:53 smithi107 podman[136574]: 2024-09-21 09:55:53.741953178 +0000 UTC m=+0.427255473 container init f473d6bcb438bdb7df23695c57e8aa724b98459bb469c23016a4d000ae9e14eb (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-activate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-21T09:55:54.109 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:53 smithi107 podman[136574]: 2024-09-21 09:55:53.753075534 +0000 UTC m=+0.438377818 container start f473d6bcb438bdb7df23695c57e8aa724b98459bb469c23016a4d000ae9e14eb (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-activate, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T09:55:54.109 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:53 smithi107 podman[136574]: 2024-09-21 09:55:53.850159167 +0000 UTC m=+0.535461453 container attach f473d6bcb438bdb7df23695c57e8aa724b98459bb469c23016a4d000ae9e14eb (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-activate, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T09:55:54.109 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:53 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-activate[136588]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T09:55:54.109 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:53 smithi107 bash[136574]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T09:55:54.109 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:53 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-activate[136588]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T09:55:54.110 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:53 smithi107 bash[136574]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T09:55:54.243 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 33 16 67547 67531 127.888 108.625 0.00956986 0.00781345 2024-09-21T09:55:54.859 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:54 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-activate[136588]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T09:55:54.859 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:54 smithi107 bash[136574]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T09:55:54.859 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:54 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-activate[136588]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-21T09:55:54.859 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:54 smithi107 bash[136574]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-21T09:55:55.217 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:54 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-activate[136588]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-21T09:55:55.217 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:54 smithi107 bash[136574]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-21T09:55:55.217 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:54 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-activate[136588]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-21T09:55:55.217 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:54 smithi107 bash[136574]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-21T09:55:55.217 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:54 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-activate[136588]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-21T09:55:55.217 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:54 smithi107 bash[136574]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-21T09:55:55.217 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:54 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-activate[136588]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T09:55:55.217 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:54 smithi107 bash[136574]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T09:55:55.217 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:54 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-activate[136588]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-21T09:55:55.217 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:54 smithi107 bash[136574]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-21T09:55:55.217 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:54 smithi107 podman[136574]: 2024-09-21 09:55:54.92469098 +0000 UTC m=+1.609993280 container died f473d6bcb438bdb7df23695c57e8aa724b98459bb469c23016a4d000ae9e14eb (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-activate, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T09:55:55.243 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 34 16 69092 69076 126.967 96.5625 0.00890506 0.00787034 2024-09-21T09:55:55.354 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:55:55.584 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:55 smithi107 ceph-mon[102221]: pgmap v335: 89 pgs: 25 active+undersized, 2 stale+active+clean, 8 active+undersized+degraded, 54 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s; 20541/194436 objects degraded (10.564%); 8.0 KiB/s, 0 objects/s recovering 2024-09-21T09:55:55.585 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:55 smithi107 ceph-mon[102221]: Health check failed: Degraded data redundancy: 20541/194436 objects degraded (10.564%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T09:55:55.585 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:55 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4010164481' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:55.585 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:55 smithi107 ceph-mon[105022]: pgmap v335: 89 pgs: 25 active+undersized, 2 stale+active+clean, 8 active+undersized+degraded, 54 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s; 20541/194436 objects degraded (10.564%); 8.0 KiB/s, 0 objects/s recovering 2024-09-21T09:55:55.585 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:55 smithi107 ceph-mon[105022]: Health check failed: Degraded data redundancy: 20541/194436 objects degraded (10.564%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T09:55:55.585 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:55 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4010164481' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:55 smithi129 ceph-mon[88471]: pgmap v335: 89 pgs: 25 active+undersized, 2 stale+active+clean, 8 active+undersized+degraded, 54 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s; 20541/194436 objects degraded (10.564%); 8.0 KiB/s, 0 objects/s recovering 2024-09-21T09:55:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:55 smithi129 ceph-mon[88471]: Health check failed: Degraded data redundancy: 20541/194436 objects degraded (10.564%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T09:55:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:55 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4010164481' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:55.859 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:55 smithi107 podman[136781]: 2024-09-21 09:55:55.582486556 +0000 UTC m=+0.647853426 container remove f473d6bcb438bdb7df23695c57e8aa724b98459bb469c23016a4d000ae9e14eb (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-21T09:55:56.197 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:55 smithi107 podman[136877]: 2024-09-21 09:55:55.813944758 +0000 UTC m=+0.033306696 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:55:56.197 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:55 smithi107 podman[136877]: 2024-09-21 09:55:55.951388861 +0000 UTC m=+0.170750796 container create 687d481b0402fe04ad0e09f4019d97ba7beb5a2bf62e84b64610a90eef23e58d (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-21T09:55:56.197 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:56 smithi107 podman[136877]: 2024-09-21 09:55:56.195758415 +0000 UTC m=+0.415120367 container init 687d481b0402fe04ad0e09f4019d97ba7beb5a2bf62e84b64610a90eef23e58d (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T09:55:56.244 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 35 16 70575 70559 125.987 92.6875 0.00971669 0.00793264 2024-09-21T09:55:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:56 smithi107 ceph-mon[102221]: from='client.35658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:55:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:56 smithi107 ceph-mon[105022]: from='client.35658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:55:56.610 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:56 smithi107 podman[136877]: 2024-09-21 09:55:56.20469421 +0000 UTC m=+0.424056144 container start 687d481b0402fe04ad0e09f4019d97ba7beb5a2bf62e84b64610a90eef23e58d (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-21T09:55:56.610 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:56 smithi107 bash[136877]: 687d481b0402fe04ad0e09f4019d97ba7beb5a2bf62e84b64610a90eef23e58d 2024-09-21T09:55:56.610 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:56 smithi107 systemd[1]: Started Ceph osd.3 for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T09:55:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:56 smithi129 ceph-mon[88471]: from='client.35658 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:55:57.244 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 36 16 72036 72020 125.021 91.3125 0.00738726 0.00799382 2024-09-21T09:55:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:57 smithi129 ceph-mon[88471]: from='client.35664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:57 smithi129 ceph-mon[88471]: pgmap v336: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s; 24708/195978 objects degraded (12.608%); 16 KiB/s, 0 objects/s recovering 2024-09-21T09:55:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:57 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:57 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:57 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:55:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:57 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:57.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:57 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:57.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:57 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:57.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:57 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:57.717 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:57 smithi107 ceph-mon[102221]: from='client.35664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:57.717 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:57 smithi107 ceph-mon[102221]: pgmap v336: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s; 24708/195978 objects degraded (12.608%); 16 KiB/s, 0 objects/s recovering 2024-09-21T09:55:57.717 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:57 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:57.717 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:57 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:57.717 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:57 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:55:57.717 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:57 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:57.717 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:57 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:57.718 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:57 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:57.718 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:57 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:57.718 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:57 smithi107 ceph-mon[105022]: from='client.35664 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:57.718 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:57 smithi107 ceph-mon[105022]: pgmap v336: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s; 24708/195978 objects degraded (12.608%); 16 KiB/s, 0 objects/s recovering 2024-09-21T09:55:57.718 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:57 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:57.718 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:57 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:57.718 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:57 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:55:57.718 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:57 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:57.718 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:57 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:57.718 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:57 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:57.718 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:57 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:55:57.986 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:55:57 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3[136895]: 2024-09-21T09:55:57.842+0000 7fe15a4cf640 -1 Falling back to public interface 2024-09-21T09:55:58.245 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 37 16 73541 73525 124.182 94.0625 0.00861124 0.00804788 2024-09-21T09:55:58.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:58 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/908303857' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:58 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/908303857' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:58 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/908303857' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:55:58.922 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:55:59.245 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 38 16 74963 74947 123.253 88.875 0.0174405 0.00810909 2024-09-21T09:55:59.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:55:59 smithi107 ceph-mon[102221]: pgmap v337: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 26222/207855 objects degraded (12.616%); 15 KiB/s, 0 objects/s recovering 2024-09-21T09:55:59.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:55:59 smithi107 ceph-mon[105022]: pgmap v337: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 26222/207855 objects degraded (12.616%); 15 KiB/s, 0 objects/s recovering 2024-09-21T09:55:59.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:55:59 smithi129 ceph-mon[88471]: pgmap v337: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 26222/207855 objects degraded (12.616%); 15 KiB/s, 0 objects/s recovering 2024-09-21T09:56:00.245 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 39 16 76451 76435 122.477 93 0.0114248 0.00815951 2024-09-21T09:56:00.303 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:56:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:56:00.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:00 smithi107 ceph-mon[102221]: from='client.35676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:00 smithi107 ceph-mon[102221]: from='client.55360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:00.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:00 smithi107 ceph-mon[105022]: from='client.35676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:00.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:00 smithi107 ceph-mon[105022]: from='client.55360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:00 smithi129 ceph-mon[88471]: from='client.35676 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:00 smithi129 ceph-mon[88471]: from='client.55360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:01.245 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T09:56:01.244660+0000 min lat: 0.00118684 max lat: 1.03919 avg lat: 0.00821355 2024-09-21T09:56:01.246 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T09:56:01.246 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 40 16 77898 77882 121.676 90.4375 0.0181082 0.00821355 2024-09-21T09:56:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:01 smithi129 ceph-mon[88471]: pgmap v338: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s; 27665/219300 objects degraded (12.615%); 13 KiB/s, 0 objects/s recovering 2024-09-21T09:56:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:01 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 27665/219300 objects degraded (12.615%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:01 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:01 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:01.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:01 smithi107 ceph-mon[102221]: pgmap v338: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s; 27665/219300 objects degraded (12.615%); 13 KiB/s, 0 objects/s recovering 2024-09-21T09:56:01.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:01 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 27665/219300 objects degraded (12.615%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:01.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:01 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:01.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:01 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:01.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:01 smithi107 ceph-mon[105022]: pgmap v338: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 87 MiB/s wr, 1.39k op/s; 27665/219300 objects degraded (12.615%); 13 KiB/s, 0 objects/s recovering 2024-09-21T09:56:01.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:01 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 27665/219300 objects degraded (12.615%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:01.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:01 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:01.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:01 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:02.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:56:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:56:01] "GET /metrics HTTP/1.1" 200 36523 "" "Prometheus/2.51.0" 2024-09-21T09:56:02.247 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 41 16 79122 79106 120.57 76.5 0.00829272 0.00828795 2024-09-21T09:56:02.544 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:56:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:02 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/897610270' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:56:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:02 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:02 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:02.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:02 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/897610270' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:56:02.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:02 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:02.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:02 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:02.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:02 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/897610270' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:56:02.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:02 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:02.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:02 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:03.247 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 42 16 80336 80320 119.506 75.875 0.00569638 0.00836294 2024-09-21T09:56:03.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:03 smithi107 ceph-mon[105022]: pgmap v339: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s; 27665/219300 objects degraded (12.615%); 11 KiB/s, 0 objects/s recovering 2024-09-21T09:56:03.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:03.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:03.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:03 smithi107 ceph-mon[105022]: from='client.45485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:03.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:03 smithi107 ceph-mon[102221]: pgmap v339: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s; 27665/219300 objects degraded (12.615%); 11 KiB/s, 0 objects/s recovering 2024-09-21T09:56:03.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:03.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:03.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:03 smithi107 ceph-mon[102221]: from='client.45485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:03.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:03 smithi129 ceph-mon[88471]: pgmap v339: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s; 27665/219300 objects degraded (12.615%); 11 KiB/s, 0 objects/s recovering 2024-09-21T09:56:03.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:03.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:03.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:03 smithi129 ceph-mon[88471]: from='client.45485 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:04.247 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 43 16 81837 81821 118.908 93.8125 0.00606647 0.00840514 2024-09-21T09:56:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:04 smithi107 ceph-mon[105022]: from='client.55372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:04 smithi107 ceph-mon[102221]: from='client.55372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:04 smithi129 ceph-mon[88471]: from='client.55372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:05.248 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 44 16 83195 83179 118.133 84.875 0.00456366 0.0084593 2024-09-21T09:56:05.813 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:05 smithi107 ceph-mon[102221]: pgmap v340: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s; 29828/238932 objects degraded (12.484%); 11 KiB/s, 0 objects/s recovering 2024-09-21T09:56:05.813 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:05 smithi107 ceph-mon[105022]: pgmap v340: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s; 29828/238932 objects degraded (12.484%); 11 KiB/s, 0 objects/s recovering 2024-09-21T09:56:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:05 smithi129 ceph-mon[88471]: pgmap v340: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s; 29828/238932 objects degraded (12.484%); 11 KiB/s, 0 objects/s recovering 2024-09-21T09:56:06.181 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:56:06.248 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 45 16 84372 84356 117.142 73.5625 0.0308278 0.0085311 2024-09-21T09:56:06.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:06 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2069904344' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:56:06.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:06 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2069904344' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:56:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:06 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2069904344' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:56:07.248 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 46 16 85668 85652 116.357 81 0.0116692 0.00858936 2024-09-21T09:56:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:07 smithi107 ceph-mon[102221]: pgmap v341: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s; 30264/240240 objects degraded (12.597%); 5.3 KiB/s, 0 objects/s recovering 2024-09-21T09:56:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:07 smithi107 ceph-mon[102221]: from='client.35703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:07 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 30264/240240 objects degraded (12.597%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:07 smithi107 ceph-mon[102221]: from='client.45503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:07.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:07 smithi107 ceph-mon[105022]: pgmap v341: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s; 30264/240240 objects degraded (12.597%); 5.3 KiB/s, 0 objects/s recovering 2024-09-21T09:56:07.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:07 smithi107 ceph-mon[105022]: from='client.35703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:07.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:07 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 30264/240240 objects degraded (12.597%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:07.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:07 smithi107 ceph-mon[105022]: from='client.45503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:07 smithi129 ceph-mon[88471]: pgmap v341: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s; 30264/240240 objects degraded (12.597%); 5.3 KiB/s, 0 objects/s recovering 2024-09-21T09:56:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:07 smithi129 ceph-mon[88471]: from='client.35703 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:07 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 30264/240240 objects degraded (12.597%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:07 smithi129 ceph-mon[88471]: from='client.45503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:08.248 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 47 15 87103 87088 115.79 89.75 0.00888673 0.00863171 2024-09-21T09:56:08.360 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:56:08 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T09:56:08.184+0000 7efdb5a17640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-21T09:56:09.248 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 48 16 88389 88373 115.051 80.3125 0.0125199 0.00868659 2024-09-21T09:56:09.366 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:09.366 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:09.366 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:56:09.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:56:09.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[102221]: pgmap v342: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s; 31537/250140 objects degraded (12.608%) 2024-09-21T09:56:09.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:09.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:56:09.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:56:09.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:56:09.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:56:09.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-21T09:56:09.367 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[102221]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-21T09:56:09.368 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[102221]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-21T09:56:09.368 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3[136895]: 2024-09-21T09:56:09.045+0000 7fe15a4cf640 -1 osd.3 122 log_to_monitors true 2024-09-21T09:56:09.368 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:09.368 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:09.368 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:56:09.369 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:56:09.369 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[105022]: pgmap v342: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s; 31537/250140 objects degraded (12.608%) 2024-09-21T09:56:09.369 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:09.369 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:56:09.369 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:56:09.369 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:56:09.369 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:56:09.369 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-21T09:56:09.369 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[105022]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-21T09:56:09.369 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:09 smithi107 ceph-mon[105022]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-21T09:56:09.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:09 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:09.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:09 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:09.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:09 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:56:09.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:09 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:56:09.423 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:09 smithi129 ceph-mon[88471]: pgmap v342: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s; 31537/250140 objects degraded (12.608%) 2024-09-21T09:56:09.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:09 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:09.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:09 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:56:09.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:09 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:56:09.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:09 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:56:09.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:09 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:56:09.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:09 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-21T09:56:09.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:09 smithi129 ceph-mon[88471]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-21T09:56:09.424 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:09 smithi129 ceph-mon[88471]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-09-21T09:56:09.741 INFO:teuthology.orchestra.run.smithi107.stdout:true 2024-09-21T09:56:10.148 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:10 smithi129 ceph-mon[88471]: from='osd.3 [v2:172.21.15.107:6826/860985313,v1:172.21.15.107:6827/860985313]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T09:56:10.148 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:10 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4261461973' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:56:10.148 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:10 smithi129 ceph-mon[88471]: from='client.55390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:10.248 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 49 16 89502 89486 114.123 69.5625 0.00333527 0.00874925 2024-09-21T09:56:10.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:10 smithi107 ceph-mon[102221]: from='osd.3 [v2:172.21.15.107:6826/860985313,v1:172.21.15.107:6827/860985313]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T09:56:10.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:10 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4261461973' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:56:10.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:10 smithi107 ceph-mon[102221]: from='client.55390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:10.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:10 smithi107 ceph-mon[105022]: from='osd.3 [v2:172.21.15.107:6826/860985313,v1:172.21.15.107:6827/860985313]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T09:56:10.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:10 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4261461973' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:56:10.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:10 smithi107 ceph-mon[105022]: from='client.55390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:10.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:56:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:56:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:10.859 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 09:56:10 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3[136895]: 2024-09-21T09:56:10.556+0000 7fe14e271640 -1 osd.3 122 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:56:11.248 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 50 16 90584 90568 113.193 67.625 0.00710269 0.008829 2024-09-21T09:56:11.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:11 smithi107 ceph-mon[102221]: from='osd.3 [v2:172.21.15.107:6826/860985313,v1:172.21.15.107:6827/860985313]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T09:56:11.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:11 smithi107 ceph-mon[102221]: osdmap e125: 8 total, 7 up, 8 in 2024-09-21T09:56:11.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:11 smithi107 ceph-mon[102221]: pgmap v343: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s; 32776/259944 objects degraded (12.609%) 2024-09-21T09:56:11.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:11 smithi107 ceph-mon[102221]: from='osd.3 [v2:172.21.15.107:6826/860985313,v1:172.21.15.107:6827/860985313]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T09:56:11.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:11 smithi107 ceph-mon[102221]: from='client.35727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:11.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:11 smithi107 ceph-mon[102221]: from='osd.3 [v2:172.21.15.107:6826/860985313,v1:172.21.15.107:6827/860985313]' entity='osd.3' 2024-09-21T09:56:11.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:11 smithi107 ceph-mon[105022]: from='osd.3 [v2:172.21.15.107:6826/860985313,v1:172.21.15.107:6827/860985313]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T09:56:11.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:11 smithi107 ceph-mon[105022]: osdmap e125: 8 total, 7 up, 8 in 2024-09-21T09:56:11.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:11 smithi107 ceph-mon[105022]: pgmap v343: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s; 32776/259944 objects degraded (12.609%) 2024-09-21T09:56:11.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:11 smithi107 ceph-mon[105022]: from='osd.3 [v2:172.21.15.107:6826/860985313,v1:172.21.15.107:6827/860985313]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T09:56:11.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:11 smithi107 ceph-mon[105022]: from='client.35727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:11.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:11 smithi107 ceph-mon[105022]: from='osd.3 [v2:172.21.15.107:6826/860985313,v1:172.21.15.107:6827/860985313]' entity='osd.3' 2024-09-21T09:56:11.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:11 smithi129 ceph-mon[88471]: from='osd.3 [v2:172.21.15.107:6826/860985313,v1:172.21.15.107:6827/860985313]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T09:56:11.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:11 smithi129 ceph-mon[88471]: osdmap e125: 8 total, 7 up, 8 in 2024-09-21T09:56:11.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:11 smithi129 ceph-mon[88471]: pgmap v343: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s; 32776/259944 objects degraded (12.609%) 2024-09-21T09:56:11.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:11 smithi129 ceph-mon[88471]: from='osd.3 [v2:172.21.15.107:6826/860985313,v1:172.21.15.107:6827/860985313]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T09:56:11.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:11 smithi129 ceph-mon[88471]: from='client.35727 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:11.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:11 smithi129 ceph-mon[88471]: from='osd.3 [v2:172.21.15.107:6826/860985313,v1:172.21.15.107:6827/860985313]' entity='osd.3' 2024-09-21T09:56:12.043 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:56:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:56:11] "GET /metrics HTTP/1.1" 200 36370 "" "Prometheus/2.51.0" 2024-09-21T09:56:12.248 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 51 16 91073 91057 111.573 30.5625 0.00304689 0.00884195 2024-09-21T09:56:12.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:56:12.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:12 smithi107 ceph-mon[105022]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T09:56:12.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:12 smithi107 ceph-mon[105022]: osd.3 [v2:172.21.15.107:6826/860985313,v1:172.21.15.107:6827/860985313] boot 2024-09-21T09:56:12.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:12 smithi107 ceph-mon[105022]: osdmap e126: 8 total, 8 up, 8 in 2024-09-21T09:56:12.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T09:56:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:56:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:12 smithi107 ceph-mon[102221]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T09:56:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:12 smithi107 ceph-mon[102221]: osd.3 [v2:172.21.15.107:6826/860985313,v1:172.21.15.107:6827/860985313] boot 2024-09-21T09:56:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:12 smithi107 ceph-mon[102221]: osdmap e126: 8 total, 8 up, 8 in 2024-09-21T09:56:12.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T09:56:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:56:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:12 smithi129 ceph-mon[88471]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T09:56:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:12 smithi129 ceph-mon[88471]: osd.3 [v2:172.21.15.107:6826/860985313,v1:172.21.15.107:6827/860985313] boot 2024-09-21T09:56:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:12 smithi129 ceph-mon[88471]: osdmap e126: 8 total, 8 up, 8 in 2024-09-21T09:56:12.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T09:56:12.465 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 4 2024-09-21T09:56:13.248 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 52 16 91073 91057 109.427 0 - 0.00884195 2024-09-21T09:56:13.324 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:13 smithi107 ceph-mon[102221]: pgmap v346: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 55 MiB/s wr, 875 op/s; 32776/259944 objects degraded (12.609%) 2024-09-21T09:56:13.325 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:13 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3394575074' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:56:13.325 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:13 smithi107 ceph-mon[102221]: osdmap e127: 8 total, 8 up, 8 in 2024-09-21T09:56:13.325 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:13 smithi107 ceph-mon[105022]: pgmap v346: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 55 MiB/s wr, 875 op/s; 32776/259944 objects degraded (12.609%) 2024-09-21T09:56:13.325 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:13 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3394575074' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:56:13.325 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:13 smithi107 ceph-mon[105022]: osdmap e127: 8 total, 8 up, 8 in 2024-09-21T09:56:13.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:13 smithi129 ceph-mon[88471]: pgmap v346: 89 pgs: 28 active+undersized, 9 active+undersized+degraded, 52 active+clean; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 55 MiB/s wr, 875 op/s; 32776/259944 objects degraded (12.609%) 2024-09-21T09:56:13.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:13 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3394575074' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T09:56:13.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:13 smithi129 ceph-mon[88471]: osdmap e127: 8 total, 8 up, 8 in 2024-09-21T09:56:13.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade pause' 2024-09-21T09:56:14.217 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:14.248 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 53 16 91443 91427 107.799 11.5625 0.00878588 0.00927088 2024-09-21T09:56:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:14 smithi107 ceph-mon[105022]: osdmap e128: 8 total, 8 up, 8 in 2024-09-21T09:56:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:14 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 32776/259944 objects degraded (12.609%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:14 smithi107 ceph-mon[102221]: osdmap e128: 8 total, 8 up, 8 in 2024-09-21T09:56:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:14 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 32776/259944 objects degraded (12.609%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:14 smithi129 ceph-mon[88471]: osdmap e128: 8 total, 8 up, 8 in 2024-09-21T09:56:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:14 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 32776/259944 objects degraded (12.609%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:15.248 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 54 16 92513 92497 107.041 66.875 0.018442 0.00933763 2024-09-21T09:56:15.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:15 smithi107 ceph-mon[102221]: pgmap v349: 89 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 9 peering, 13 active+undersized, 2 active+undersized+degraded, 60 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 7686/273381 objects degraded (2.811%) 2024-09-21T09:56:15.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:15 smithi107 ceph-mon[105022]: pgmap v349: 89 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 9 peering, 13 active+undersized, 2 active+undersized+degraded, 60 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 7686/273381 objects degraded (2.811%) 2024-09-21T09:56:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:15 smithi129 ceph-mon[88471]: pgmap v349: 89 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 9 peering, 13 active+undersized, 2 active+undersized+degraded, 60 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 7686/273381 objects degraded (2.811%) 2024-09-21T09:56:16.249 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 55 15 93531 93516 106.252 63.6875 0.0158639 0.00940674 2024-09-21T09:56:16.610 INFO:teuthology.orchestra.run.smithi107.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:56:17.249 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 56 16 94487 94471 105.421 59.6875 0.0188171 0.0094808 2024-09-21T09:56:17.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch ps' 2024-09-21T09:56:17.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:17 smithi107 ceph-mon[102221]: pgmap v350: 89 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 9 peering, 75 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 47 MiB/s wr, 746 op/s; 7684/273381 objects degraded (2.811%) 2024-09-21T09:56:17.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:17 smithi107 ceph-mon[102221]: from='client.45530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:17.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:17 smithi107 ceph-mon[102221]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:56:17.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:17 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:17.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:17 smithi107 ceph-mon[105022]: pgmap v350: 89 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 9 peering, 75 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 47 MiB/s wr, 746 op/s; 7684/273381 objects degraded (2.811%) 2024-09-21T09:56:17.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:17 smithi107 ceph-mon[105022]: from='client.45530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:17.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:17 smithi107 ceph-mon[105022]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:56:17.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:17 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:17 smithi129 ceph-mon[88471]: pgmap v350: 89 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 9 peering, 75 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 47 MiB/s wr, 746 op/s; 7684/273381 objects degraded (2.811%) 2024-09-21T09:56:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:17 smithi129 ceph-mon[88471]: from='client.45530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:17 smithi129 ceph-mon[88471]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:56:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:17 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:18.249 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 57 16 95473 95457 104.652 61.625 0.0104021 0.00954897 2024-09-21T09:56:19.249 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 58 16 96502 96486 103.957 64.3125 0.0224696 0.00961448 2024-09-21T09:56:19.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:19 smithi107 ceph-mon[102221]: pgmap v351: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 83 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 55 MiB/s wr, 876 op/s; 9493/277005 objects degraded (3.427%) 2024-09-21T09:56:19.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:19 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 9493/277005 objects degraded (3.427%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:19.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:19 smithi107 ceph-mon[105022]: pgmap v351: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 83 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 55 MiB/s wr, 876 op/s; 9493/277005 objects degraded (3.427%) 2024-09-21T09:56:19.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:19 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 9493/277005 objects degraded (3.427%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:19 smithi129 ceph-mon[88471]: pgmap v351: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 83 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 55 MiB/s wr, 876 op/s; 9493/277005 objects degraded (3.427%) 2024-09-21T09:56:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:19 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 9493/277005 objects degraded (3.427%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:20.249 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 59 16 97453 97437 103.202 59.4375 0.00877313 0.00968511 2024-09-21T09:56:20.652 INFO:teuthology.orchestra.run.smithi107.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T09:56:20.652 INFO:teuthology.orchestra.run.smithi107.stdout:alertmanager.a smithi107 *:9093,9094 running (12m) 19s ago 16m 19.2M - 0.27.0 11f11916f8cd 6fa4ebf1f69d 2024-09-21T09:56:20.652 INFO:teuthology.orchestra.run.smithi107.stdout:grafana.a smithi129 *:3000 running (11m) 18s ago 16m 86.7M - 9.4.7 954c08fa6188 72cc8ff672f5 2024-09-21T09:56:20.652 INFO:teuthology.orchestra.run.smithi107.stdout:mds.foo.smithi107.sedied smithi107 running (15m) 19s ago 15m 22.5M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 83d74619fe7a 2024-09-21T09:56:20.652 INFO:teuthology.orchestra.run.smithi107.stdout:mds.foo.smithi129.zdzmqr smithi129 running (15m) 18s ago 15m 16.6M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 2133ce1b60f9 2024-09-21T09:56:20.652 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.x smithi129 *:8443,9283,8765 running (10m) 18s ago 20m 460M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 59e80c9df333 2024-09-21T09:56:20.652 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.y smithi107 *:8443,9283,8765 running (11m) 19s ago 22m 523M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 849409c95398 2024-09-21T09:56:20.652 INFO:teuthology.orchestra.run.smithi107.stdout:mon.a smithi107 running (10m) 19s ago 22m 60.2M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 34ae11d9ef1b 2024-09-21T09:56:20.652 INFO:teuthology.orchestra.run.smithi107.stdout:mon.b smithi129 running (9m) 18s ago 20m 46.9M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 b01038fb71fd 2024-09-21T09:56:20.652 INFO:teuthology.orchestra.run.smithi107.stdout:mon.c smithi107 running (9m) 19s ago 20m 51.1M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 04f7b77f1500 2024-09-21T09:56:20.653 INFO:teuthology.orchestra.run.smithi107.stdout:node-exporter.a smithi107 *:9100 running (11m) 19s ago 16m 11.7M - 1.7.0 72c9c2088986 2444eabb7c7e 2024-09-21T09:56:20.653 INFO:teuthology.orchestra.run.smithi107.stdout:node-exporter.b smithi129 *:9100 running (11m) 18s ago 16m 11.2M - 1.7.0 72c9c2088986 ce5db8669c3d 2024-09-21T09:56:20.653 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 smithi107 running (7m) 19s ago 20m 605M 2479M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 378577ff0aed 2024-09-21T09:56:20.653 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1 smithi107 running (5m) 19s ago 19m 364M 2479M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 c2b756a1ca12 2024-09-21T09:56:20.653 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2 smithi107 running (5m) 19s ago 19m 263M 2479M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 4e385b5d2d28 2024-09-21T09:56:20.653 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3 smithi107 running (24s) 19s ago 18m 12.3M 2479M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 687d481b0402 2024-09-21T09:56:20.653 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4 smithi129 running (18m) 18s ago 18m 532M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 7c6ca82e1ac1 2024-09-21T09:56:20.653 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5 smithi129 running (18m) 18s ago 18m 807M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 9ae131a2ef13 2024-09-21T09:56:20.653 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6 smithi129 running (17m) 18s ago 17m 458M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be a5ccfa3e24c2 2024-09-21T09:56:20.653 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7 smithi129 running (17m) 18s ago 17m 922M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be a32c9bd56124 2024-09-21T09:56:20.653 INFO:teuthology.orchestra.run.smithi107.stdout:prometheus.a smithi129 *:9095 running (10m) 18s ago 16m 57.6M - 2.51.0 1d3b7f56885b 24d10324469f 2024-09-21T09:56:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:56:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:56:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:21.250 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T09:56:21.248300+0000 min lat: 0.00118684 max lat: 2.3214 avg lat: 0.00975577 2024-09-21T09:56:21.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T09:56:21.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 60 16 98380 98364 102.448 57.9375 0.029163 0.00975577 2024-09-21T09:56:21.664 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:21 smithi107 ceph-mon[102221]: pgmap v352: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 11156/286317 objects degraded (3.896%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T09:56:21.664 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:21 smithi107 ceph-mon[102221]: from='client.55402 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:21.665 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:21 smithi107 ceph-mon[105022]: pgmap v352: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 11156/286317 objects degraded (3.896%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T09:56:21.665 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:21 smithi107 ceph-mon[105022]: from='client.55402 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:21.709 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch ps' 2024-09-21T09:56:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:21 smithi129 ceph-mon[88471]: pgmap v352: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 11156/286317 objects degraded (3.896%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T09:56:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:21 smithi129 ceph-mon[88471]: from='client.55402 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:22.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:56:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:56:21] "GET /metrics HTTP/1.1" 200 36370 "" "Prometheus/2.51.0" 2024-09-21T09:56:22.178 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:22.250 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 61 16 99336 99320 101.748 59.75 0.0216526 0.00982132 2024-09-21T09:56:23.249 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 62 16 100198 100182 100.976 53.875 0.0162107 0.00989801 2024-09-21T09:56:23.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:23 smithi107 ceph-mon[102221]: pgmap v353: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 44 MiB/s wr, 707 op/s; 11156/286317 objects degraded (3.896%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T09:56:23.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:23 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:56:23.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:23 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:56:23.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:23 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:56:23.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:23 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:23.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:23 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:56:23.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:23 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:56:23.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:23 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:56:23.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:23 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:23.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:23 smithi107 ceph-mon[105022]: pgmap v353: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 44 MiB/s wr, 707 op/s; 11156/286317 objects degraded (3.896%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T09:56:23.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:23 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:56:23.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:23 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:56:23.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:23 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:56:23.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:23 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:23.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:23 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:56:23.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:23 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:56:23.796 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:23 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:56:23.796 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:23 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:23 smithi129 ceph-mon[88471]: pgmap v353: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 44 MiB/s wr, 707 op/s; 11156/286317 objects degraded (3.896%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T09:56:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:23 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:56:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:23 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:56:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:23 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:56:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:23 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:23 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:56:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:23 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:56:23.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:23 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:56:23.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:23 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:24.250 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 63 16 101002 100986 100.17 50.25 0.00706869 0.00997714 2024-09-21T09:56:24.637 INFO:teuthology.orchestra.run.smithi107.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T09:56:24.637 INFO:teuthology.orchestra.run.smithi107.stdout:alertmanager.a smithi107 *:9093,9094 running (12m) 23s ago 16m 19.2M - 0.27.0 11f11916f8cd 6fa4ebf1f69d 2024-09-21T09:56:24.637 INFO:teuthology.orchestra.run.smithi107.stdout:grafana.a smithi129 *:3000 running (11m) 22s ago 16m 86.7M - 9.4.7 954c08fa6188 72cc8ff672f5 2024-09-21T09:56:24.637 INFO:teuthology.orchestra.run.smithi107.stdout:mds.foo.smithi107.sedied smithi107 running (15m) 23s ago 15m 22.5M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 83d74619fe7a 2024-09-21T09:56:24.637 INFO:teuthology.orchestra.run.smithi107.stdout:mds.foo.smithi129.zdzmqr smithi129 running (15m) 22s ago 15m 16.6M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 2133ce1b60f9 2024-09-21T09:56:24.637 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.x smithi129 *:8443,9283,8765 running (10m) 22s ago 20m 460M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 59e80c9df333 2024-09-21T09:56:24.637 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.y smithi107 *:8443,9283,8765 running (11m) 23s ago 22m 523M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 849409c95398 2024-09-21T09:56:24.637 INFO:teuthology.orchestra.run.smithi107.stdout:mon.a smithi107 running (10m) 23s ago 22m 60.2M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 34ae11d9ef1b 2024-09-21T09:56:24.638 INFO:teuthology.orchestra.run.smithi107.stdout:mon.b smithi129 running (9m) 22s ago 20m 46.9M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 b01038fb71fd 2024-09-21T09:56:24.638 INFO:teuthology.orchestra.run.smithi107.stdout:mon.c smithi107 running (9m) 23s ago 20m 51.1M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 04f7b77f1500 2024-09-21T09:56:24.638 INFO:teuthology.orchestra.run.smithi107.stdout:node-exporter.a smithi107 *:9100 running (11m) 23s ago 16m 11.7M - 1.7.0 72c9c2088986 2444eabb7c7e 2024-09-21T09:56:24.638 INFO:teuthology.orchestra.run.smithi107.stdout:node-exporter.b smithi129 *:9100 running (11m) 22s ago 16m 11.2M - 1.7.0 72c9c2088986 ce5db8669c3d 2024-09-21T09:56:24.638 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 smithi107 running (7m) 23s ago 20m 605M 2479M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 378577ff0aed 2024-09-21T09:56:24.638 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1 smithi107 running (5m) 23s ago 19m 364M 2479M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 c2b756a1ca12 2024-09-21T09:56:24.638 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2 smithi107 running (5m) 23s ago 19m 263M 2479M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 4e385b5d2d28 2024-09-21T09:56:24.638 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3 smithi107 running (28s) 23s ago 18m 12.3M 2479M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 687d481b0402 2024-09-21T09:56:24.638 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4 smithi129 running (18m) 22s ago 18m 532M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 7c6ca82e1ac1 2024-09-21T09:56:24.638 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5 smithi129 running (18m) 22s ago 18m 807M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 9ae131a2ef13 2024-09-21T09:56:24.638 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6 smithi129 running (17m) 22s ago 17m 458M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be a5ccfa3e24c2 2024-09-21T09:56:24.638 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7 smithi129 running (17m) 22s ago 17m 922M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be a32c9bd56124 2024-09-21T09:56:24.638 INFO:teuthology.orchestra.run.smithi107.stdout:prometheus.a smithi129 *:9095 running (10m) 22s ago 16m 57.6M - 2.51.0 1d3b7f56885b 24d10324469f 2024-09-21T09:56:24.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:24 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 11156/286317 objects degraded (3.896%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:25.006 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:24 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 11156/286317 objects degraded (3.896%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:25.006 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:24 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 11156/286317 objects degraded (3.896%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:25.249 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 64 16 101948 101932 99.529 59.125 0.00501129 0.0100421 2024-09-21T09:56:25.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph versions' 2024-09-21T09:56:25.806 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:25 smithi129 ceph-mon[88471]: pgmap v354: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 10949/300495 objects degraded (3.644%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T09:56:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:25 smithi129 ceph-mon[88471]: from='client.45542 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:25.947 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:25 smithi107 ceph-mon[102221]: pgmap v354: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 10949/300495 objects degraded (3.644%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T09:56:25.947 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:25 smithi107 ceph-mon[102221]: from='client.45542 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:25.947 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:25 smithi107 ceph-mon[105022]: pgmap v354: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 10949/300495 objects degraded (3.644%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T09:56:25.947 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:25 smithi107 ceph-mon[105022]: from='client.45542 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:26.249 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 65 16 102889 102873 98.9026 58.8125 0.0102366 0.0101054 2024-09-21T09:56:27.250 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 66 16 103869 103853 98.332 61.25 0.00884982 0.0101646 2024-09-21T09:56:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:27 smithi107 ceph-mon[102221]: pgmap v355: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 47 MiB/s wr, 752 op/s; 10949/300495 objects degraded (3.644%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T09:56:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:27.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:56:27.292 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:27 smithi107 ceph-mon[105022]: pgmap v355: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 47 MiB/s wr, 752 op/s; 10949/300495 objects degraded (3.644%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T09:56:27.292 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:27.292 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:56:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:27 smithi129 ceph-mon[88471]: pgmap v355: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 47 MiB/s wr, 752 op/s; 10949/300495 objects degraded (3.644%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T09:56:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:56:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:56:28.250 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 67 16 104780 104764 97.7142 56.9375 0.0204416 0.0102287 2024-09-21T09:56:28.466 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T09:56:28.467 INFO:teuthology.orchestra.run.smithi107.stdout: "mon": { 2024-09-21T09:56:28.467 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 3 2024-09-21T09:56:28.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T09:56:28.467 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr": { 2024-09-21T09:56:28.467 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T09:56:28.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T09:56:28.467 INFO:teuthology.orchestra.run.smithi107.stdout: "osd": { 2024-09-21T09:56:28.467 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)": 4, 2024-09-21T09:56:28.467 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 4 2024-09-21T09:56:28.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T09:56:28.467 INFO:teuthology.orchestra.run.smithi107.stdout: "mds": { 2024-09-21T09:56:28.467 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)": 2 2024-09-21T09:56:28.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T09:56:28.468 INFO:teuthology.orchestra.run.smithi107.stdout: "overall": { 2024-09-21T09:56:28.468 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)": 6, 2024-09-21T09:56:28.468 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 9 2024-09-21T09:56:28.468 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T09:56:28.468 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T09:56:29.199 DEBUG:teuthology.parallel:result is None 2024-09-21T09:56:29.250 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 68 16 105724 105708 97.1448 59 0.018405 0.0102879 2024-09-21T09:56:29.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:29 smithi129 ceph-mon[88471]: pgmap v356: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 56 MiB/s wr, 891 op/s; 10949/305496 objects degraded (3.584%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T09:56:29.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:29 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/957349726' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:56:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:29 smithi107 ceph-mon[102221]: pgmap v356: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 56 MiB/s wr, 891 op/s; 10949/305496 objects degraded (3.584%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T09:56:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:29 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/957349726' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:56:29.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:29 smithi107 ceph-mon[105022]: pgmap v356: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 56 MiB/s wr, 891 op/s; 10949/305496 objects degraded (3.584%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T09:56:29.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:29 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/957349726' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T09:56:30.250 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 69 16 106733 106717 96.6508 63.0625 0.0109398 0.0103414 2024-09-21T09:56:30.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:30 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 10949/305496 objects degraded (3.584%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:30.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:56:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:56:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:30.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:30 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 10949/305496 objects degraded (3.584%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:30.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:30 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 10949/305496 objects degraded (3.584%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:31.250 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 70 16 107748 107732 96.1763 63.4375 0.0342181 0.0103921 2024-09-21T09:56:31.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:31 smithi129 ceph-mon[88471]: pgmap v357: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 10709/314361 objects degraded (3.407%); 3.3 MiB/s, 53 objects/s recovering 2024-09-21T09:56:31.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:31 smithi107 ceph-mon[102221]: pgmap v357: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 10709/314361 objects degraded (3.407%); 3.3 MiB/s, 53 objects/s recovering 2024-09-21T09:56:31.899 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:31 smithi107 ceph-mon[105022]: pgmap v357: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s; 10709/314361 objects degraded (3.407%); 3.3 MiB/s, 53 objects/s recovering 2024-09-21T09:56:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:56:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:56:31] "GET /metrics HTTP/1.1" 200 36533 "" "Prometheus/2.51.0" 2024-09-21T09:56:32.250 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 71 16 108730 108714 95.686 61.375 0.00947371 0.0104445 2024-09-21T09:56:33.250 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 72 16 109735 109719 95.2294 62.8125 0.0205242 0.0104961 2024-09-21T09:56:33.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:33 smithi129 ceph-mon[88471]: pgmap v358: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s; 10709/314361 objects degraded (3.407%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T09:56:33.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:33 smithi107 ceph-mon[102221]: pgmap v358: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s; 10709/314361 objects degraded (3.407%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T09:56:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:33 smithi107 ceph-mon[105022]: pgmap v358: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s; 10709/314361 objects degraded (3.407%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T09:56:34.250 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 73 16 110670 110654 94.7254 58.4375 0.0108913 0.0105514 2024-09-21T09:56:35.250 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 74 16 111632 111616 94.2577 60.125 0.0066757 0.0106039 2024-09-21T09:56:35.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:35 smithi129 ceph-mon[88471]: pgmap v359: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s; 10435/322518 objects degraded (3.235%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T09:56:35.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:35 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 10435/322518 objects degraded (3.235%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:35.616 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:35 smithi107 ceph-mon[102221]: pgmap v359: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s; 10435/322518 objects degraded (3.235%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T09:56:35.617 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:35 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 10435/322518 objects degraded (3.235%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:35.617 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:35 smithi107 ceph-mon[105022]: pgmap v359: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 63 MiB/s wr, 1.00k op/s; 10435/322518 objects degraded (3.235%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T09:56:35.617 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:35 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 10435/322518 objects degraded (3.235%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:36.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 75 16 112610 112594 93.8159 61.125 0.0109739 0.0106533 2024-09-21T09:56:37.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 76 16 113591 113575 93.3882 61.3125 0.0127401 0.0107034 2024-09-21T09:56:37.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:37 smithi107 ceph-mon[102221]: pgmap v360: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 50 MiB/s wr, 798 op/s; 10435/329247 objects degraded (3.169%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T09:56:37.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:37 smithi107 ceph-mon[105022]: pgmap v360: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 50 MiB/s wr, 798 op/s; 10435/329247 objects degraded (3.169%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T09:56:37.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:37 smithi129 ceph-mon[88471]: pgmap v360: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 50 MiB/s wr, 798 op/s; 10435/329247 objects degraded (3.169%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T09:56:38.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 77 16 114534 114518 92.9407 58.9375 0.00482125 0.010753 2024-09-21T09:56:39.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 78 16 115408 115392 92.4494 54.625 0.0111072 0.010808 2024-09-21T09:56:39.470 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:39 smithi129 ceph-mon[88471]: pgmap v361: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 59 MiB/s wr, 946 op/s; 10435/334581 objects degraded (3.119%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T09:56:39.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:39 smithi107 ceph-mon[102221]: pgmap v361: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 59 MiB/s wr, 946 op/s; 10435/334581 objects degraded (3.119%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T09:56:39.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:39 smithi107 ceph-mon[105022]: pgmap v361: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 59 MiB/s wr, 946 op/s; 10435/334581 objects degraded (3.119%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T09:56:40.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 79 16 116349 116333 92.0236 58.8125 0.00517237 0.0108612 2024-09-21T09:56:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:40 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 10435/334581 objects degraded (3.119%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:40.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:40 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 10435/334581 objects degraded (3.119%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:40.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:40 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 10435/334581 objects degraded (3.119%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:40.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:56:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:56:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:41.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T09:56:41.250111+0000 min lat: 0.00118684 max lat: 2.3214 avg lat: 0.0109069 2024-09-21T09:56:41.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T09:56:41.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 80 16 117250 117234 91.5772 56.3125 0.00490411 0.0109069 2024-09-21T09:56:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:41 smithi107 ceph-mon[102221]: pgmap v362: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 10150/343662 objects degraded (2.953%); 4.2 MiB/s, 66 objects/s recovering 2024-09-21T09:56:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:56:41.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:41 smithi107 ceph-mon[105022]: pgmap v362: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 10150/343662 objects degraded (2.953%); 4.2 MiB/s, 66 objects/s recovering 2024-09-21T09:56:41.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:56:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:41 smithi129 ceph-mon[88471]: pgmap v362: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 10150/343662 objects degraded (2.953%); 4.2 MiB/s, 66 objects/s recovering 2024-09-21T09:56:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:56:42.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:56:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:56:41] "GET /metrics HTTP/1.1" 200 36533 "" "Prometheus/2.51.0" 2024-09-21T09:56:42.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 81 16 118155 118139 91.1448 56.5625 0.0245894 0.0109665 2024-09-21T09:56:43.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 82 16 119102 119086 90.755 59.1875 0.0115748 0.0110137 2024-09-21T09:56:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:43 smithi107 ceph-mon[102221]: pgmap v363: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 51 MiB/s wr, 813 op/s; 10150/343662 objects degraded (2.953%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T09:56:43.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:43 smithi107 ceph-mon[105022]: pgmap v363: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 51 MiB/s wr, 813 op/s; 10150/343662 objects degraded (2.953%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T09:56:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:43 smithi129 ceph-mon[88471]: pgmap v363: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 51 MiB/s wr, 813 op/s; 10150/343662 objects degraded (2.953%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T09:56:44.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 83 16 120070 120054 90.3905 60.5 0.00632667 0.0110577 2024-09-21T09:56:45.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 84 16 121005 120989 90.01 58.4375 0.0222876 0.0111046 2024-09-21T09:56:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:45 smithi107 ceph-mon[102221]: pgmap v364: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 9816/351309 objects degraded (2.794%); 4.6 MiB/s, 74 objects/s recovering 2024-09-21T09:56:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:45 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 9816/351309 objects degraded (2.794%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:45 smithi107 ceph-mon[105022]: pgmap v364: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 9816/351309 objects degraded (2.794%); 4.6 MiB/s, 74 objects/s recovering 2024-09-21T09:56:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:45 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 9816/351309 objects degraded (2.794%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:45 smithi129 ceph-mon[88471]: pgmap v364: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 9816/351309 objects degraded (2.794%); 4.6 MiB/s, 74 objects/s recovering 2024-09-21T09:56:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:45 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 9816/351309 objects degraded (2.794%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:46.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 85 16 121937 121921 89.6363 58.25 0.0063586 0.0111507 2024-09-21T09:56:47.251 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 86 16 122876 122860 89.2764 58.6875 0.0328886 0.0111962 2024-09-21T09:56:47.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:47 smithi107 ceph-mon[102221]: pgmap v365: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 60 MiB/s wr, 964 op/s; 9816/357243 objects degraded (2.748%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T09:56:47.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:47 smithi107 ceph-mon[105022]: pgmap v365: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 60 MiB/s wr, 964 op/s; 9816/357243 objects degraded (2.748%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T09:56:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:47 smithi129 ceph-mon[88471]: pgmap v365: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 60 MiB/s wr, 964 op/s; 9816/357243 objects degraded (2.748%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T09:56:48.252 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 87 16 123782 123766 88.9011 56.625 0.023187 0.0112427 2024-09-21T09:56:49.252 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 88 16 124058 124042 88.0869 17.25 0.00399919 0.0112448 2024-09-21T09:56:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:49 smithi107 ceph-mon[102221]: pgmap v366: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 58 MiB/s wr, 931 op/s; 9816/362808 objects degraded (2.706%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T09:56:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:49 smithi107 ceph-mon[102221]: osdmap e129: 8 total, 8 up, 8 in 2024-09-21T09:56:49.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:49 smithi107 ceph-mon[105022]: pgmap v366: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 58 MiB/s wr, 931 op/s; 9816/362808 objects degraded (2.706%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T09:56:49.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:49 smithi107 ceph-mon[105022]: osdmap e129: 8 total, 8 up, 8 in 2024-09-21T09:56:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:49 smithi129 ceph-mon[88471]: pgmap v366: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 58 MiB/s wr, 931 op/s; 9816/362808 objects degraded (2.706%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T09:56:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:49 smithi129 ceph-mon[88471]: osdmap e129: 8 total, 8 up, 8 in 2024-09-21T09:56:50.252 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 89 16 124964 124948 87.7333 56.625 0.0266312 0.0113926 2024-09-21T09:56:50.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:50 smithi107 ceph-mon[102221]: osdmap e130: 8 total, 8 up, 8 in 2024-09-21T09:56:50.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:50 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 9816/362808 objects degraded (2.706%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:50.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:50 smithi107 ceph-mon[105022]: osdmap e130: 8 total, 8 up, 8 in 2024-09-21T09:56:50.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:50 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 9816/362808 objects degraded (2.706%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:50 smithi129 ceph-mon[88471]: osdmap e130: 8 total, 8 up, 8 in 2024-09-21T09:56:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:50 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 9816/362808 objects degraded (2.706%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:56:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:56:50.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:56:51.252 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total time run: 90.0108 2024-09-21T09:56:51.252 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total writes made: 125881 2024-09-21T09:56:51.252 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Write size: 65536 2024-09-21T09:56:51.252 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object size: 65536 2024-09-21T09:56:51.252 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Bandwidth (MB/sec): 87.4069 2024-09-21T09:56:51.252 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Bandwidth: 45.2966 2024-09-21T09:56:51.252 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max bandwidth (MB/sec): 254.75 2024-09-21T09:56:51.252 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min bandwidth (MB/sec): 0 2024-09-21T09:56:51.252 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average IOPS: 1398 2024-09-21T09:56:51.253 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev IOPS: 724.745 2024-09-21T09:56:51.253 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max IOPS: 4076 2024-09-21T09:56:51.253 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min IOPS: 0 2024-09-21T09:56:51.253 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average Latency(s): 0.0114363 2024-09-21T09:56:51.253 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Latency(s): 0.0310696 2024-09-21T09:56:51.253 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max latency(s): 2.3214 2024-09-21T09:56:51.253 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min latency(s): 0.00118684 2024-09-21T09:56:51.258 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T09:56:51.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:51 smithi107 ceph-mon[102221]: pgmap v369: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s; 9361/370794 objects degraded (2.525%); 6.2 MiB/s, 98 objects/s recovering 2024-09-21T09:56:51.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:51 smithi107 ceph-mon[105022]: pgmap v369: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s; 9361/370794 objects degraded (2.525%); 6.2 MiB/s, 98 objects/s recovering 2024-09-21T09:56:51.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:51 smithi129 ceph-mon[88471]: pgmap v369: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s; 9361/370794 objects degraded (2.525%); 6.2 MiB/s, 98 objects/s recovering 2024-09-21T09:56:52.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:56:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:56:51] "GET /metrics HTTP/1.1" 200 36533 "" "Prometheus/2.51.0" 2024-09-21T09:56:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:53 smithi107 ceph-mon[102221]: pgmap v370: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 51 MiB/s wr, 811 op/s; 9361/370794 objects degraded (2.525%); 3.6 MiB/s, 56 objects/s recovering 2024-09-21T09:56:53.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:53 smithi107 ceph-mon[105022]: pgmap v370: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 51 MiB/s wr, 811 op/s; 9361/370794 objects degraded (2.525%); 3.6 MiB/s, 56 objects/s recovering 2024-09-21T09:56:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:53 smithi129 ceph-mon[88471]: pgmap v370: 89 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 51 MiB/s wr, 811 op/s; 9361/370794 objects degraded (2.525%); 3.6 MiB/s, 56 objects/s recovering 2024-09-21T09:56:55.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:55 smithi107 ceph-mon[102221]: pgmap v371: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 47 MiB/s wr, 844 op/s; 9361/373170 objects degraded (2.509%); 3.6 MiB/s, 56 objects/s recovering 2024-09-21T09:56:55.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:55 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 9361/373170 objects degraded (2.509%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:55.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:55 smithi107 ceph-mon[105022]: pgmap v371: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 47 MiB/s wr, 844 op/s; 9361/373170 objects degraded (2.509%); 3.6 MiB/s, 56 objects/s recovering 2024-09-21T09:56:55.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:55 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 9361/373170 objects degraded (2.509%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:55 smithi129 ceph-mon[88471]: pgmap v371: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 47 MiB/s wr, 844 op/s; 9361/373170 objects degraded (2.509%); 3.6 MiB/s, 56 objects/s recovering 2024-09-21T09:56:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:55 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 9361/373170 objects degraded (2.509%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:56:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:56:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:56:57.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:57 smithi107 ceph-mon[105022]: pgmap v372: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 39 MiB/s wr, 858 op/s; 9153/372021 objects degraded (2.460%); 5.2 MiB/s, 82 objects/s recovering 2024-09-21T09:56:57.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:57 smithi107 ceph-mon[102221]: pgmap v372: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 39 MiB/s wr, 858 op/s; 9153/372021 objects degraded (2.460%); 5.2 MiB/s, 82 objects/s recovering 2024-09-21T09:56:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:57 smithi129 ceph-mon[88471]: pgmap v372: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 39 MiB/s wr, 858 op/s; 9153/372021 objects degraded (2.460%); 5.2 MiB/s, 82 objects/s recovering 2024-09-21T09:56:59.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:59 smithi107 ceph-mon[102221]: pgmap v373: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 104 B/s rd, 32 MiB/s wr, 863 op/s; 9153/367140 objects degraded (2.493%); 4.2 MiB/s, 67 objects/s recovering 2024-09-21T09:56:59.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:56:59 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 9153/367140 objects degraded (2.493%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:59.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:59 smithi107 ceph-mon[105022]: pgmap v373: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 104 B/s rd, 32 MiB/s wr, 863 op/s; 9153/367140 objects degraded (2.493%); 4.2 MiB/s, 67 objects/s recovering 2024-09-21T09:56:59.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:56:59 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 9153/367140 objects degraded (2.493%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T09:56:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:59 smithi129 ceph-mon[88471]: pgmap v373: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 104 B/s rd, 32 MiB/s wr, 863 op/s; 9153/367140 objects degraded (2.493%); 4.2 MiB/s, 67 objects/s recovering 2024-09-21T09:56:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:56:59 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 9153/367140 objects degraded (2.493%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T09:57:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:57:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:57:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:01.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:01 smithi107 ceph-mon[102221]: pgmap v374: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 94 B/s rd, 13 MiB/s wr, 792 op/s; 8975/358905 objects degraded (2.501%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T09:57:01.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:01 smithi107 ceph-mon[105022]: pgmap v374: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 94 B/s rd, 13 MiB/s wr, 792 op/s; 8975/358905 objects degraded (2.501%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T09:57:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:01 smithi129 ceph-mon[88471]: pgmap v374: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 94 B/s rd, 13 MiB/s wr, 792 op/s; 8975/358905 objects degraded (2.501%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T09:57:02.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:57:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:57:01] "GET /metrics HTTP/1.1" 200 36527 "" "Prometheus/2.51.0" 2024-09-21T09:57:03.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:03 smithi107 ceph-mon[102221]: pgmap v375: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 12 MiB/s wr, 714 op/s; 8975/358905 objects degraded (2.501%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T09:57:03.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:03 smithi107 ceph-mon[105022]: pgmap v375: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 12 MiB/s wr, 714 op/s; 8975/358905 objects degraded (2.501%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T09:57:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:03 smithi129 ceph-mon[88471]: pgmap v375: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 85 B/s rd, 12 MiB/s wr, 714 op/s; 8975/358905 objects degraded (2.501%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T09:57:04.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:04 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 8975/358905 objects degraded (2.501%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T09:57:04.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:04 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 8975/358905 objects degraded (2.501%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T09:57:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:04 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 8975/358905 objects degraded (2.501%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T09:57:05.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:05 smithi107 ceph-mon[105022]: pgmap v376: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 85 B/s rd, 12 MiB/s wr, 931 op/s; 8975/351105 objects degraded (2.556%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T09:57:05.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:05 smithi107 ceph-mon[102221]: pgmap v376: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 85 B/s rd, 12 MiB/s wr, 931 op/s; 8975/351105 objects degraded (2.556%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T09:57:05.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:05 smithi129 ceph-mon[88471]: pgmap v376: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 85 B/s rd, 12 MiB/s wr, 931 op/s; 8975/351105 objects degraded (2.556%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T09:57:07.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:07 smithi107 ceph-mon[102221]: pgmap v377: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 4.1 MiB/s wr, 916 op/s; 8769/344919 objects degraded (2.542%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T09:57:07.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:07 smithi107 ceph-mon[105022]: pgmap v377: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 4.1 MiB/s wr, 916 op/s; 8769/344919 objects degraded (2.542%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T09:57:07.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:07 smithi129 ceph-mon[88471]: pgmap v377: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 4.1 MiB/s wr, 916 op/s; 8769/344919 objects degraded (2.542%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T09:57:09.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:09 smithi107 ceph-mon[102221]: pgmap v378: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 8769/339852 objects degraded (2.580%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T09:57:09.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:09 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 8769/339852 objects degraded (2.580%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T09:57:09.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:09 smithi107 ceph-mon[105022]: pgmap v378: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 8769/339852 objects degraded (2.580%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T09:57:09.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:09 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 8769/339852 objects degraded (2.580%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T09:57:09.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:09 smithi129 ceph-mon[88471]: pgmap v378: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 8769/339852 objects degraded (2.580%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T09:57:09.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:09 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 8769/339852 objects degraded (2.580%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T09:57:10.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:57:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:57:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:11.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:11 smithi107 ceph-mon[102221]: pgmap v379: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 999 op/s; 8560/331152 objects degraded (2.585%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T09:57:11.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:57:11.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:11 smithi107 ceph-mon[105022]: pgmap v379: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 999 op/s; 8560/331152 objects degraded (2.585%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T09:57:11.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:57:11.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:11 smithi129 ceph-mon[88471]: pgmap v379: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 999 op/s; 8560/331152 objects degraded (2.585%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T09:57:11.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:57:12.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:57:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:57:11] "GET /metrics HTTP/1.1" 200 36530 "" "Prometheus/2.51.0" 2024-09-21T09:57:13.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:13 smithi107 ceph-mon[102221]: pgmap v380: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 770 op/s; 8560/331152 objects degraded (2.585%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T09:57:13.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:13 smithi107 ceph-mon[105022]: pgmap v380: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 770 op/s; 8560/331152 objects degraded (2.585%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T09:57:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:13 smithi129 ceph-mon[88471]: pgmap v380: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 770 op/s; 8560/331152 objects degraded (2.585%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T09:57:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:14 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 8560/331152 objects degraded (2.585%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T09:57:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:14 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 8560/331152 objects degraded (2.585%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T09:57:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:14 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 8560/331152 objects degraded (2.585%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T09:57:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:15 smithi107 ceph-mon[102221]: pgmap v381: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 988 op/s; 8560/323313 objects degraded (2.648%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T09:57:15.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:15 smithi107 ceph-mon[105022]: pgmap v381: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 988 op/s; 8560/323313 objects degraded (2.648%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T09:57:15.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:15 smithi129 ceph-mon[88471]: pgmap v381: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 988 op/s; 8560/323313 objects degraded (2.648%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T09:57:17.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:17 smithi107 ceph-mon[102221]: pgmap v382: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 929 op/s; 8298/317628 objects degraded (2.612%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T09:57:17.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:17 smithi107 ceph-mon[105022]: pgmap v382: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 929 op/s; 8298/317628 objects degraded (2.612%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T09:57:17.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:17 smithi129 ceph-mon[88471]: pgmap v382: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 929 op/s; 8298/317628 objects degraded (2.612%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T09:57:19.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:19 smithi107 ceph-mon[102221]: pgmap v383: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 903 op/s; 8298/312399 objects degraded (2.656%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T09:57:19.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:19 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 8298/312399 objects degraded (2.656%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:19.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:19 smithi107 ceph-mon[105022]: pgmap v383: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 903 op/s; 8298/312399 objects degraded (2.656%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T09:57:19.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:19 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 8298/312399 objects degraded (2.656%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:19 smithi129 ceph-mon[88471]: pgmap v383: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 903 op/s; 8298/312399 objects degraded (2.656%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T09:57:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:19 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 8298/312399 objects degraded (2.656%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:20.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:57:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:57:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:21.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:21 smithi107 ceph-mon[105022]: pgmap v384: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 997 op/s; 8024/303936 objects degraded (2.640%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T09:57:21.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:21 smithi107 ceph-mon[102221]: pgmap v384: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 997 op/s; 8024/303936 objects degraded (2.640%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T09:57:21.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:21 smithi129 ceph-mon[88471]: pgmap v384: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 997 op/s; 8024/303936 objects degraded (2.640%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T09:57:22.110 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:57:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:57:21] "GET /metrics HTTP/1.1" 200 36530 "" "Prometheus/2.51.0" 2024-09-21T09:57:23.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:23 smithi107 ceph-mon[102221]: pgmap v385: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 755 op/s; 8024/303936 objects degraded (2.640%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T09:57:23.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:23 smithi107 ceph-mon[105022]: pgmap v385: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 755 op/s; 8024/303936 objects degraded (2.640%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T09:57:23.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:23 smithi129 ceph-mon[88471]: pgmap v385: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 755 op/s; 8024/303936 objects degraded (2.640%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T09:57:24.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:24 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:57:24.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:24 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 8024/303936 objects degraded (2.640%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:24.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:24 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:57:24.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:24 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 8024/303936 objects degraded (2.640%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:24.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:24 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:57:24.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:24 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 8024/303936 objects degraded (2.640%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:25.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:25 smithi107 ceph-mon[105022]: pgmap v386: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 970 op/s; 8024/296190 objects degraded (2.709%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T09:57:25.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:25 smithi107 ceph-mon[102221]: pgmap v386: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 970 op/s; 8024/296190 objects degraded (2.709%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T09:57:25.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:25 smithi129 ceph-mon[88471]: pgmap v386: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 970 op/s; 8024/296190 objects degraded (2.709%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T09:57:26.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:57:26.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:57:26.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:57:27.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:27 smithi129 ceph-mon[88471]: pgmap v387: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 915 op/s; 7724/290361 objects degraded (2.660%); 4.4 MiB/s, 69 objects/s recovering 2024-09-21T09:57:27.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:27 smithi107 ceph-mon[102221]: pgmap v387: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 915 op/s; 7724/290361 objects degraded (2.660%); 4.4 MiB/s, 69 objects/s recovering 2024-09-21T09:57:27.701 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:27 smithi107 ceph-mon[105022]: pgmap v387: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 915 op/s; 7724/290361 objects degraded (2.660%); 4.4 MiB/s, 69 objects/s recovering 2024-09-21T09:57:28.627 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:28 smithi107 ceph-mon[102221]: osdmap e131: 8 total, 8 up, 8 in 2024-09-21T09:57:28.628 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:28 smithi107 ceph-mon[105022]: osdmap e131: 8 total, 8 up, 8 in 2024-09-21T09:57:28.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:28 smithi129 ceph-mon[88471]: osdmap e131: 8 total, 8 up, 8 in 2024-09-21T09:57:29.374 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:29 smithi129 ceph-mon[88471]: pgmap v389: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 887 op/s; 7700/285759 objects degraded (2.695%); 3.7 MiB/s, 59 objects/s recovering 2024-09-21T09:57:29.374 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:29 smithi129 ceph-mon[88471]: osdmap e132: 8 total, 8 up, 8 in 2024-09-21T09:57:29.374 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:29 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 7700/285759 objects degraded (2.695%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:29.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:29 smithi107 ceph-mon[102221]: pgmap v389: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 887 op/s; 7700/285759 objects degraded (2.695%); 3.7 MiB/s, 59 objects/s recovering 2024-09-21T09:57:29.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:29 smithi107 ceph-mon[102221]: osdmap e132: 8 total, 8 up, 8 in 2024-09-21T09:57:29.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:29 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 7700/285759 objects degraded (2.695%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:29.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:29 smithi107 ceph-mon[105022]: pgmap v389: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 887 op/s; 7700/285759 objects degraded (2.695%); 3.7 MiB/s, 59 objects/s recovering 2024-09-21T09:57:29.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:29 smithi107 ceph-mon[105022]: osdmap e132: 8 total, 8 up, 8 in 2024-09-21T09:57:29.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:29 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 7700/285759 objects degraded (2.695%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:30 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:57:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:30 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:57:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:30 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:57:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:30 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:57:30.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:57:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:57:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:30.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:30 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:57:30.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:30 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:57:30.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:30 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:57:30.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:30 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:57:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:30 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:57:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:30 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:57:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:30 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:57:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:30 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:57:31.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:31 smithi107 ceph-mon[102221]: pgmap v391: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 7543/279051 objects degraded (2.703%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T09:57:31.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:31 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:57:31.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:31 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:57:31.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:31 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:57:31.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:31 smithi107 ceph-mon[105022]: pgmap v391: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 7543/279051 objects degraded (2.703%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T09:57:31.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:31 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:57:31.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:31 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:57:31.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:31 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:57:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:31 smithi129 ceph-mon[88471]: pgmap v391: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 7543/279051 objects degraded (2.703%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T09:57:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:31 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:57:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:31 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:57:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:31 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:57:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:57:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:57:31] "GET /metrics HTTP/1.1" 200 36529 "" "Prometheus/2.51.0" 2024-09-21T09:57:33.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:33 smithi107 ceph-mon[102221]: pgmap v392: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 7543/279051 objects degraded (2.703%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T09:57:33.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:33 smithi107 ceph-mon[105022]: pgmap v392: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 7543/279051 objects degraded (2.703%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T09:57:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:33 smithi129 ceph-mon[88471]: pgmap v392: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 7543/279051 objects degraded (2.703%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T09:57:34.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:34 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 7543/279051 objects degraded (2.703%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:34.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:34 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 7543/279051 objects degraded (2.703%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:34.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:34 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 7543/279051 objects degraded (2.703%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:35.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:35 smithi107 ceph-mon[102221]: pgmap v393: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 738 op/s; 7257/272619 objects degraded (2.662%); 3.6 MiB/s, 58 objects/s recovering 2024-09-21T09:57:35.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:35 smithi107 ceph-mon[105022]: pgmap v393: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 738 op/s; 7257/272619 objects degraded (2.662%); 3.6 MiB/s, 58 objects/s recovering 2024-09-21T09:57:35.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:35 smithi129 ceph-mon[88471]: pgmap v393: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 738 op/s; 7257/272619 objects degraded (2.662%); 3.6 MiB/s, 58 objects/s recovering 2024-09-21T09:57:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:37 smithi107 ceph-mon[102221]: pgmap v394: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 725 op/s; 7257/266826 objects degraded (2.720%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T09:57:37.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:37 smithi107 ceph-mon[105022]: pgmap v394: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 725 op/s; 7257/266826 objects degraded (2.720%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T09:57:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:37 smithi129 ceph-mon[88471]: pgmap v394: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 725 op/s; 7257/266826 objects degraded (2.720%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T09:57:39.720 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:39 smithi129 ceph-mon[88471]: pgmap v395: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 806 op/s; 7063/261558 objects degraded (2.700%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T09:57:39.721 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:39 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 7063/261558 objects degraded (2.700%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:39.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:39 smithi107 ceph-mon[102221]: pgmap v395: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 806 op/s; 7063/261558 objects degraded (2.700%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T09:57:39.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:39 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 7063/261558 objects degraded (2.700%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:39.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:39 smithi107 ceph-mon[105022]: pgmap v395: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 806 op/s; 7063/261558 objects degraded (2.700%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T09:57:39.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:39 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 7063/261558 objects degraded (2.700%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:40.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:57:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:57:40.216Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:41.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:41 smithi107 ceph-mon[102221]: pgmap v396: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 931 op/s; 7063/253056 objects degraded (2.791%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T09:57:41.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:57:41.768 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:41 smithi107 ceph-mon[105022]: pgmap v396: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 931 op/s; 7063/253056 objects degraded (2.791%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T09:57:41.768 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:57:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:41 smithi129 ceph-mon[88471]: pgmap v396: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 931 op/s; 7063/253056 objects degraded (2.791%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T09:57:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:57:42.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:57:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:57:41] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-09-21T09:57:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:43 smithi107 ceph-mon[102221]: pgmap v397: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 721 op/s; 7063/253056 objects degraded (2.791%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T09:57:43.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:43 smithi107 ceph-mon[105022]: pgmap v397: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 721 op/s; 7063/253056 objects degraded (2.791%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T09:57:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:43 smithi129 ceph-mon[88471]: pgmap v397: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 721 op/s; 7063/253056 objects degraded (2.791%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T09:57:44.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:44 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 7063/253056 objects degraded (2.791%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:44.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:44 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 7063/253056 objects degraded (2.791%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:44 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 7063/253056 objects degraded (2.791%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:45.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:45 smithi107 ceph-mon[102221]: pgmap v398: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 926 op/s; 6848/245688 objects degraded (2.787%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T09:57:45.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:45 smithi107 ceph-mon[105022]: pgmap v398: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 926 op/s; 6848/245688 objects degraded (2.787%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T09:57:45.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:45 smithi129 ceph-mon[88471]: pgmap v398: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 926 op/s; 6848/245688 objects degraded (2.787%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T09:57:47.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:47 smithi107 ceph-mon[102221]: pgmap v399: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 906 op/s; 6848/239964 objects degraded (2.854%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T09:57:47.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:47 smithi107 ceph-mon[105022]: pgmap v399: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 906 op/s; 6848/239964 objects degraded (2.854%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T09:57:47.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:47 smithi129 ceph-mon[88471]: pgmap v399: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 906 op/s; 6848/239964 objects degraded (2.854%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T09:57:49.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:49 smithi107 ceph-mon[102221]: pgmap v400: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 889 op/s; 6610/234780 objects degraded (2.815%); 3.4 MiB/s, 53 objects/s recovering 2024-09-21T09:57:49.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:49 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 6610/234780 objects degraded (2.815%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:49.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:49 smithi107 ceph-mon[105022]: pgmap v400: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 889 op/s; 6610/234780 objects degraded (2.815%); 3.4 MiB/s, 53 objects/s recovering 2024-09-21T09:57:49.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:49 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 6610/234780 objects degraded (2.815%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:49 smithi129 ceph-mon[88471]: pgmap v400: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 889 op/s; 6610/234780 objects degraded (2.815%); 3.4 MiB/s, 53 objects/s recovering 2024-09-21T09:57:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:49 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 6610/234780 objects degraded (2.815%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:57:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:57:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:57:51.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:51 smithi107 ceph-mon[102221]: pgmap v401: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 967 op/s; 6610/226716 objects degraded (2.916%); 2.4 MiB/s, 37 objects/s recovering 2024-09-21T09:57:51.768 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:51 smithi107 ceph-mon[105022]: pgmap v401: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 967 op/s; 6610/226716 objects degraded (2.916%); 2.4 MiB/s, 37 objects/s recovering 2024-09-21T09:57:51.836 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:51 smithi129 ceph-mon[88471]: pgmap v401: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 967 op/s; 6610/226716 objects degraded (2.916%); 2.4 MiB/s, 37 objects/s recovering 2024-09-21T09:57:52.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:57:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:57:51] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-09-21T09:57:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:53 smithi107 ceph-mon[102221]: pgmap v402: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 6610/226716 objects degraded (2.916%); 2.4 MiB/s, 37 objects/s recovering 2024-09-21T09:57:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:53 smithi107 ceph-mon[105022]: pgmap v402: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 6610/226716 objects degraded (2.916%); 2.4 MiB/s, 37 objects/s recovering 2024-09-21T09:57:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:53 smithi129 ceph-mon[88471]: pgmap v402: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 6610/226716 objects degraded (2.916%); 2.4 MiB/s, 37 objects/s recovering 2024-09-21T09:57:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:54 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 6610/226716 objects degraded (2.916%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:54 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 6610/226716 objects degraded (2.916%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:54 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 6610/226716 objects degraded (2.916%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:55 smithi107 ceph-mon[102221]: pgmap v403: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 939 op/s; 6376/219240 objects degraded (2.908%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T09:57:55.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:55 smithi107 ceph-mon[105022]: pgmap v403: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 939 op/s; 6376/219240 objects degraded (2.908%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T09:57:55.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:55 smithi129 ceph-mon[88471]: pgmap v403: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 939 op/s; 6376/219240 objects degraded (2.908%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T09:57:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:57:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:57:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:57:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:57 smithi107 ceph-mon[102221]: pgmap v404: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 6376/213513 objects degraded (2.986%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T09:57:57.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:57 smithi107 ceph-mon[105022]: pgmap v404: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 6376/213513 objects degraded (2.986%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T09:57:57.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:57 smithi129 ceph-mon[88471]: pgmap v404: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.3 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 893 op/s; 6376/213513 objects degraded (2.986%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T09:57:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:59 smithi107 ceph-mon[102221]: pgmap v405: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 872 op/s; 6130/208554 objects degraded (2.939%); 3.7 MiB/s, 59 objects/s recovering 2024-09-21T09:57:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:57:59 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 6130/208554 objects degraded (2.939%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:59 smithi107 ceph-mon[105022]: pgmap v405: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 872 op/s; 6130/208554 objects degraded (2.939%); 3.7 MiB/s, 59 objects/s recovering 2024-09-21T09:57:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:57:59 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 6130/208554 objects degraded (2.939%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:57:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:59 smithi129 ceph-mon[88471]: pgmap v405: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 872 op/s; 6130/208554 objects degraded (2.939%); 3.7 MiB/s, 59 objects/s recovering 2024-09-21T09:57:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:57:59 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 6130/208554 objects degraded (2.939%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:58:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:58:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:01.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:01 smithi107 ceph-mon[102221]: pgmap v406: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 961 op/s; 6130/200154 objects degraded (3.063%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T09:58:01.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:58:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:58:01] "GET /metrics HTTP/1.1" 200 36530 "" "Prometheus/2.51.0" 2024-09-21T09:58:01.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:01 smithi107 ceph-mon[105022]: pgmap v406: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 961 op/s; 6130/200154 objects degraded (3.063%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T09:58:01.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:01 smithi129 ceph-mon[88471]: pgmap v406: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 961 op/s; 6130/200154 objects degraded (3.063%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T09:58:03.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:03 smithi107 ceph-mon[102221]: pgmap v407: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 737 op/s; 6130/200154 objects degraded (3.063%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T09:58:03.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:03 smithi107 ceph-mon[105022]: pgmap v407: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 737 op/s; 6130/200154 objects degraded (3.063%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T09:58:03.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:03 smithi129 ceph-mon[88471]: pgmap v407: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 737 op/s; 6130/200154 objects degraded (3.063%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T09:58:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:04 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 6130/200154 objects degraded (3.063%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:04 smithi107 ceph-mon[102221]: osdmap e133: 8 total, 8 up, 8 in 2024-09-21T09:58:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:04 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 6130/200154 objects degraded (3.063%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:04 smithi107 ceph-mon[105022]: osdmap e133: 8 total, 8 up, 8 in 2024-09-21T09:58:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:04 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 6130/200154 objects degraded (3.063%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:04 smithi129 ceph-mon[88471]: osdmap e133: 8 total, 8 up, 8 in 2024-09-21T09:58:05.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:05 smithi107 ceph-mon[102221]: pgmap v408: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 953 op/s; 5872/192369 objects degraded (3.052%); 3.8 MiB/s, 61 objects/s recovering 2024-09-21T09:58:05.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:05 smithi107 ceph-mon[102221]: osdmap e134: 8 total, 8 up, 8 in 2024-09-21T09:58:05.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:05 smithi107 ceph-mon[105022]: pgmap v408: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 953 op/s; 5872/192369 objects degraded (3.052%); 3.8 MiB/s, 61 objects/s recovering 2024-09-21T09:58:05.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:05 smithi107 ceph-mon[105022]: osdmap e134: 8 total, 8 up, 8 in 2024-09-21T09:58:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:05 smithi129 ceph-mon[88471]: pgmap v408: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 953 op/s; 5872/192369 objects degraded (3.052%); 3.8 MiB/s, 61 objects/s recovering 2024-09-21T09:58:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:05 smithi129 ceph-mon[88471]: osdmap e134: 8 total, 8 up, 8 in 2024-09-21T09:58:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:07 smithi107 ceph-mon[102221]: pgmap v411: 89 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 917 op/s; 5857/186519 objects degraded (3.140%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T09:58:07.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:07 smithi107 ceph-mon[105022]: pgmap v411: 89 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 917 op/s; 5857/186519 objects degraded (3.140%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T09:58:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:07 smithi129 ceph-mon[88471]: pgmap v411: 89 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 917 op/s; 5857/186519 objects degraded (3.140%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T09:58:09.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:09 smithi107 ceph-mon[102221]: pgmap v412: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 5743/182391 objects degraded (3.149%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T09:58:09.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:09 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 5743/182391 objects degraded (3.149%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:09.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:09 smithi107 ceph-mon[105022]: pgmap v412: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 5743/182391 objects degraded (3.149%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T09:58:09.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:09 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 5743/182391 objects degraded (3.149%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:09 smithi129 ceph-mon[88471]: pgmap v412: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 5743/182391 objects degraded (3.149%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T09:58:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:09 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 5743/182391 objects degraded (3.149%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:10.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:58:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:58:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:11.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:11 smithi107 ceph-mon[102221]: pgmap v413: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 5415/175371 objects degraded (3.088%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T09:58:11.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:11.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:58:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:58:11] "GET /metrics HTTP/1.1" 200 36531 "" "Prometheus/2.51.0" 2024-09-21T09:58:11.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:11 smithi107 ceph-mon[105022]: pgmap v413: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 5415/175371 objects degraded (3.088%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T09:58:11.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:11 smithi129 ceph-mon[88471]: pgmap v413: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 5415/175371 objects degraded (3.088%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T09:58:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:13.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:13 smithi107 ceph-mon[102221]: pgmap v414: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 707 op/s; 5415/175371 objects degraded (3.088%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T09:58:13.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:13 smithi107 ceph-mon[105022]: pgmap v414: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 707 op/s; 5415/175371 objects degraded (3.088%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T09:58:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:13 smithi129 ceph-mon[88471]: pgmap v414: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 707 op/s; 5415/175371 objects degraded (3.088%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T09:58:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:14 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 5415/175371 objects degraded (3.088%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:14 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 5415/175371 objects degraded (3.088%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:14 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 5415/175371 objects degraded (3.088%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:15.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:15 smithi107 ceph-mon[102221]: pgmap v415: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 826 op/s; 5415/168066 objects degraded (3.222%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T09:58:15.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:15 smithi107 ceph-mon[105022]: pgmap v415: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 826 op/s; 5415/168066 objects degraded (3.222%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T09:58:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:15 smithi129 ceph-mon[88471]: pgmap v415: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 826 op/s; 5415/168066 objects degraded (3.222%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T09:58:17.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:17 smithi107 ceph-mon[102221]: pgmap v416: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 755 op/s; 5187/162030 objects degraded (3.201%); 3.9 MiB/s, 61 objects/s recovering 2024-09-21T09:58:17.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:17 smithi107 ceph-mon[105022]: pgmap v416: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 755 op/s; 5187/162030 objects degraded (3.201%); 3.9 MiB/s, 61 objects/s recovering 2024-09-21T09:58:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:17 smithi129 ceph-mon[88471]: pgmap v416: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 755 op/s; 5187/162030 objects degraded (3.201%); 3.9 MiB/s, 61 objects/s recovering 2024-09-21T09:58:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:19 smithi107 ceph-mon[102221]: pgmap v417: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 818 op/s; 5187/157035 objects degraded (3.303%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T09:58:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:19 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 5187/157035 objects degraded (3.303%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:19.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:19 smithi107 ceph-mon[105022]: pgmap v417: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 818 op/s; 5187/157035 objects degraded (3.303%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T09:58:19.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:19 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 5187/157035 objects degraded (3.303%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:19 smithi129 ceph-mon[88471]: pgmap v417: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 818 op/s; 5187/157035 objects degraded (3.303%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T09:58:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:19 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 5187/157035 objects degraded (3.303%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:58:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:58:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:21.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:21 smithi107 ceph-mon[102221]: pgmap v418: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 934 op/s; 4955/148716 objects degraded (3.332%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T09:58:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:21 smithi107 ceph-mon[105022]: pgmap v418: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 934 op/s; 4955/148716 objects degraded (3.332%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T09:58:21.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:58:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:58:21] "GET /metrics HTTP/1.1" 200 36531 "" "Prometheus/2.51.0" 2024-09-21T09:58:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:21 smithi129 ceph-mon[88471]: pgmap v418: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 934 op/s; 4955/148716 objects degraded (3.332%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T09:58:23.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:23 smithi107 ceph-mon[102221]: pgmap v419: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 4955/148716 objects degraded (3.332%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T09:58:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:23 smithi107 ceph-mon[105022]: pgmap v419: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 4955/148716 objects degraded (3.332%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T09:58:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:23 smithi129 ceph-mon[88471]: pgmap v419: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 4955/148716 objects degraded (3.332%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T09:58:24.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:24 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 4955/148716 objects degraded (3.332%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:25.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:24 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 4955/148716 objects degraded (3.332%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:25.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:24 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 4955/148716 objects degraded (3.332%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:25 smithi129 ceph-mon[88471]: pgmap v420: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 949 op/s; 4955/141156 objects degraded (3.510%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T09:58:26.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:25 smithi107 ceph-mon[102221]: pgmap v420: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 949 op/s; 4955/141156 objects degraded (3.510%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T09:58:26.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:25 smithi107 ceph-mon[105022]: pgmap v420: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 949 op/s; 4955/141156 objects degraded (3.510%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T09:58:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:27.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:27.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:27 smithi129 ceph-mon[88471]: pgmap v421: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 4721/135252 objects degraded (3.491%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T09:58:28.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:27 smithi107 ceph-mon[102221]: pgmap v421: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 4721/135252 objects degraded (3.491%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T09:58:28.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:27 smithi107 ceph-mon[105022]: pgmap v421: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 4721/135252 objects degraded (3.491%); 3.6 MiB/s, 57 objects/s recovering 2024-09-21T09:58:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:29 smithi129 ceph-mon[88471]: pgmap v422: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 882 op/s; 4721/130257 objects degraded (3.624%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T09:58:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:29 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 4721/130257 objects degraded (3.624%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:30.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:29 smithi107 ceph-mon[102221]: pgmap v422: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 882 op/s; 4721/130257 objects degraded (3.624%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T09:58:30.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:29 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 4721/130257 objects degraded (3.624%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:30.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:29 smithi107 ceph-mon[105022]: pgmap v422: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 882 op/s; 4721/130257 objects degraded (3.624%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T09:58:30.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:29 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 4721/130257 objects degraded (3.624%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:58:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:58:30.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:31.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:30 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:58:31.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:30 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:58:31.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:30 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:58:31.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:31 smithi107 ceph-mon[105022]: pgmap v423: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 976 op/s; 4464/121857 objects degraded (3.663%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T09:58:31.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:31 smithi107 ceph-mon[102221]: pgmap v423: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 976 op/s; 4464/121857 objects degraded (3.663%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T09:58:31.860 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:58:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:58:31] "GET /metrics HTTP/1.1" 200 36529 "" "Prometheus/2.51.0" 2024-09-21T09:58:31.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:31 smithi129 ceph-mon[88471]: pgmap v423: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 976 op/s; 4464/121857 objects degraded (3.663%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T09:58:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:33 smithi129 ceph-mon[88471]: pgmap v424: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 745 op/s; 4464/121857 objects degraded (3.663%); 2.6 MiB/s, 40 objects/s recovering 2024-09-21T09:58:34.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:33 smithi107 ceph-mon[102221]: pgmap v424: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 745 op/s; 4464/121857 objects degraded (3.663%); 2.6 MiB/s, 40 objects/s recovering 2024-09-21T09:58:34.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:33 smithi107 ceph-mon[105022]: pgmap v424: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 745 op/s; 4464/121857 objects degraded (3.663%); 2.6 MiB/s, 40 objects/s recovering 2024-09-21T09:58:34.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:34 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 4464/121857 objects degraded (3.663%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:34.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:34 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 4464/121857 objects degraded (3.663%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:34 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 4464/121857 objects degraded (3.663%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:36.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:35 smithi107 ceph-mon[102221]: pgmap v425: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.3 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 954 op/s; 4464/114336 objects degraded (3.904%); 2.6 MiB/s, 40 objects/s recovering 2024-09-21T09:58:36.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:35 smithi107 ceph-mon[105022]: pgmap v425: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.3 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 954 op/s; 4464/114336 objects degraded (3.904%); 2.6 MiB/s, 40 objects/s recovering 2024-09-21T09:58:36.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:35 smithi129 ceph-mon[88471]: pgmap v425: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.3 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 954 op/s; 4464/114336 objects degraded (3.904%); 2.6 MiB/s, 40 objects/s recovering 2024-09-21T09:58:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:37 smithi107 ceph-mon[102221]: pgmap v426: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 4188/108726 objects degraded (3.852%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T09:58:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:37 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:58:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:37 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:58:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:37 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:58:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:37 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:58:37.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:37 smithi107 ceph-mon[105022]: pgmap v426: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 4188/108726 objects degraded (3.852%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T09:58:37.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:37 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:58:37.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:37 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:58:37.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:37 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:58:37.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:37 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:58:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:37 smithi129 ceph-mon[88471]: pgmap v426: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 4188/108726 objects degraded (3.852%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T09:58:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:37 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:58:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:37 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:58:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:37 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:58:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:37 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:58:38.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:38.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:58:38.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:58:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:58:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:58:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T09:58:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T09:58:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:58:39.820 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:39 smithi129 ceph-mon[88471]: pgmap v427: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 868 op/s; 4188/103956 objects degraded (4.029%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T09:58:39.820 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:39 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 4188/103956 objects degraded (4.029%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:39.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:39 smithi107 ceph-mon[105022]: pgmap v427: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 868 op/s; 4188/103956 objects degraded (4.029%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T09:58:39.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:39 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 4188/103956 objects degraded (4.029%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:39 smithi107 ceph-mon[102221]: pgmap v427: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.1 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 868 op/s; 4188/103956 objects degraded (4.029%); 2.8 MiB/s, 44 objects/s recovering 2024-09-21T09:58:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:39 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 4188/103956 objects degraded (4.029%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:40.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:58:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:58:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:41.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:41 smithi107 ceph-mon[102221]: pgmap v428: 89 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 956 op/s; 3854/95823 objects degraded (4.022%); 2.9 MiB/s, 72 objects/s recovering 2024-09-21T09:58:41.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:41 smithi107 ceph-mon[102221]: osdmap e135: 8 total, 8 up, 8 in 2024-09-21T09:58:41.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:41.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:41 smithi107 ceph-mon[105022]: pgmap v428: 89 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 956 op/s; 3854/95823 objects degraded (4.022%); 2.9 MiB/s, 72 objects/s recovering 2024-09-21T09:58:41.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:41 smithi107 ceph-mon[105022]: osdmap e135: 8 total, 8 up, 8 in 2024-09-21T09:58:41.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:41 smithi129 ceph-mon[88471]: pgmap v428: 89 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 956 op/s; 3854/95823 objects degraded (4.022%); 2.9 MiB/s, 72 objects/s recovering 2024-09-21T09:58:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:41 smithi129 ceph-mon[88471]: osdmap e135: 8 total, 8 up, 8 in 2024-09-21T09:58:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:42.108 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:58:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:58:41] "GET /metrics HTTP/1.1" 200 36523 "" "Prometheus/2.51.0" 2024-09-21T09:58:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:42 smithi129 ceph-mon[88471]: osdmap e136: 8 total, 8 up, 8 in 2024-09-21T09:58:42.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:42 smithi107 ceph-mon[105022]: osdmap e136: 8 total, 8 up, 8 in 2024-09-21T09:58:42.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:42 smithi107 ceph-mon[102221]: osdmap e136: 8 total, 8 up, 8 in 2024-09-21T09:58:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:43 smithi107 ceph-mon[102221]: pgmap v431: 89 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 771 op/s; 3854/95823 objects degraded (4.022%); 2.3 MiB/s, 76 objects/s recovering 2024-09-21T09:58:43.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:43 smithi107 ceph-mon[105022]: pgmap v431: 89 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 771 op/s; 3854/95823 objects degraded (4.022%); 2.3 MiB/s, 76 objects/s recovering 2024-09-21T09:58:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:43 smithi129 ceph-mon[88471]: pgmap v431: 89 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 771 op/s; 3854/95823 objects degraded (4.022%); 2.3 MiB/s, 76 objects/s recovering 2024-09-21T09:58:44.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:44 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 3854/95823 objects degraded (4.022%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:44.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:44 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 3854/95823 objects degraded (4.022%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:44 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 3854/95823 objects degraded (4.022%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:45.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:45 smithi107 ceph-mon[102221]: pgmap v432: 89 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 788 op/s; 3854/89802 objects degraded (4.292%); 120 KiB/s, 41 objects/s recovering 2024-09-21T09:58:45.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:45 smithi107 ceph-mon[105022]: pgmap v432: 89 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 788 op/s; 3854/89802 objects degraded (4.292%); 120 KiB/s, 41 objects/s recovering 2024-09-21T09:58:45.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:45 smithi129 ceph-mon[88471]: pgmap v432: 89 pgs: 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 788 op/s; 3854/89802 objects degraded (4.292%); 120 KiB/s, 41 objects/s recovering 2024-09-21T09:58:47.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:47 smithi107 ceph-mon[102221]: pgmap v433: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.7 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 3472/84672 objects degraded (4.101%); 120 KiB/s, 89 objects/s recovering 2024-09-21T09:58:47.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:47 smithi107 ceph-mon[105022]: pgmap v433: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.7 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 3472/84672 objects degraded (4.101%); 120 KiB/s, 89 objects/s recovering 2024-09-21T09:58:47.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:47 smithi129 ceph-mon[88471]: pgmap v433: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.7 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 3472/84672 objects degraded (4.101%); 120 KiB/s, 89 objects/s recovering 2024-09-21T09:58:49.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:49 smithi107 ceph-mon[102221]: pgmap v434: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 672 op/s; 3472/79689 objects degraded (4.357%); 0 B/s, 47 objects/s recovering 2024-09-21T09:58:49.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:49 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 3472/79689 objects degraded (4.357%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:49.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:49 smithi107 ceph-mon[105022]: pgmap v434: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 672 op/s; 3472/79689 objects degraded (4.357%); 0 B/s, 47 objects/s recovering 2024-09-21T09:58:49.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:49 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 3472/79689 objects degraded (4.357%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:49 smithi129 ceph-mon[88471]: pgmap v434: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 672 op/s; 3472/79689 objects degraded (4.357%); 0 B/s, 47 objects/s recovering 2024-09-21T09:58:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:49 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 3472/79689 objects degraded (4.357%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:58:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:58:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:58:51.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:51 smithi107 ceph-mon[102221]: pgmap v435: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 830 op/s; 3241/71904 objects degraded (4.507%); 0 B/s, 63 objects/s recovering 2024-09-21T09:58:51.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:58:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:58:51] "GET /metrics HTTP/1.1" 200 36523 "" "Prometheus/2.51.0" 2024-09-21T09:58:51.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:51 smithi107 ceph-mon[105022]: pgmap v435: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 830 op/s; 3241/71904 objects degraded (4.507%); 0 B/s, 63 objects/s recovering 2024-09-21T09:58:51.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:51 smithi129 ceph-mon[88471]: pgmap v435: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 830 op/s; 3241/71904 objects degraded (4.507%); 0 B/s, 63 objects/s recovering 2024-09-21T09:58:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:53 smithi107 ceph-mon[102221]: pgmap v436: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 776 op/s; 3241/71103 objects degraded (4.558%); 0 B/s, 57 objects/s recovering 2024-09-21T09:58:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:53 smithi107 ceph-mon[105022]: pgmap v436: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 776 op/s; 3241/71103 objects degraded (4.558%); 0 B/s, 57 objects/s recovering 2024-09-21T09:58:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:53 smithi129 ceph-mon[88471]: pgmap v436: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.4 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 776 op/s; 3241/71103 objects degraded (4.558%); 0 B/s, 57 objects/s recovering 2024-09-21T09:58:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:54 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 3241/71103 objects degraded (4.558%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:54 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 3241/71103 objects degraded (4.558%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:54 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 3241/71103 objects degraded (4.558%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:55 smithi107 ceph-mon[102221]: pgmap v437: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 895 op/s; 3241/63555 objects degraded (5.100%); 0 B/s, 51 objects/s recovering 2024-09-21T09:58:55.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:55 smithi107 ceph-mon[105022]: pgmap v437: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 895 op/s; 3241/63555 objects degraded (5.100%); 0 B/s, 51 objects/s recovering 2024-09-21T09:58:55.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:55 smithi129 ceph-mon[88471]: pgmap v437: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.3 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 895 op/s; 3241/63555 objects degraded (5.100%); 0 B/s, 51 objects/s recovering 2024-09-21T09:58:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:58:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:57 smithi107 ceph-mon[102221]: pgmap v438: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.2 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 891 op/s; 2994/57678 objects degraded (5.191%); 0 B/s, 71 objects/s recovering 2024-09-21T09:58:57.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:57 smithi107 ceph-mon[105022]: pgmap v438: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.2 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 891 op/s; 2994/57678 objects degraded (5.191%); 0 B/s, 71 objects/s recovering 2024-09-21T09:58:57.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:57 smithi129 ceph-mon[88471]: pgmap v438: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.2 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 891 op/s; 2994/57678 objects degraded (5.191%); 0 B/s, 71 objects/s recovering 2024-09-21T09:58:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:59 smithi107 ceph-mon[102221]: pgmap v439: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 881 op/s; 2994/52911 objects degraded (5.659%); 0 B/s, 39 objects/s recovering 2024-09-21T09:58:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:58:59 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2994/52911 objects degraded (5.659%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:59 smithi107 ceph-mon[105022]: pgmap v439: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 881 op/s; 2994/52911 objects degraded (5.659%); 0 B/s, 39 objects/s recovering 2024-09-21T09:58:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:58:59 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2994/52911 objects degraded (5.659%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:58:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:59 smithi129 ceph-mon[88471]: pgmap v439: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 881 op/s; 2994/52911 objects degraded (5.659%); 0 B/s, 39 objects/s recovering 2024-09-21T09:58:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:58:59 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2994/52911 objects degraded (5.659%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:59:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:59:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:59:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:01.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:01 smithi107 ceph-mon[102221]: pgmap v440: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 943 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 953 op/s; 2750/45339 objects degraded (6.065%); 0 B/s, 60 objects/s recovering 2024-09-21T09:59:01.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:59:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:59:01] "GET /metrics HTTP/1.1" 200 36511 "" "Prometheus/2.51.0" 2024-09-21T09:59:01.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:01 smithi107 ceph-mon[105022]: pgmap v440: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 943 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 953 op/s; 2750/45339 objects degraded (6.065%); 0 B/s, 60 objects/s recovering 2024-09-21T09:59:01.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:01 smithi129 ceph-mon[88471]: pgmap v440: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 943 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 953 op/s; 2750/45339 objects degraded (6.065%); 0 B/s, 60 objects/s recovering 2024-09-21T09:59:03.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:03 smithi107 ceph-mon[102221]: pgmap v441: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 927 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 758 op/s; 2750/44568 objects degraded (6.170%); 0 B/s, 40 objects/s recovering 2024-09-21T09:59:03.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:03 smithi107 ceph-mon[105022]: pgmap v441: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 927 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 758 op/s; 2750/44568 objects degraded (6.170%); 0 B/s, 40 objects/s recovering 2024-09-21T09:59:03.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:03 smithi129 ceph-mon[88471]: pgmap v441: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 927 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 758 op/s; 2750/44568 objects degraded (6.170%); 0 B/s, 40 objects/s recovering 2024-09-21T09:59:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:04 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2750/44568 objects degraded (6.170%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:59:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:04 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2750/44568 objects degraded (6.170%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:59:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:04 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2750/44568 objects degraded (6.170%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:59:05.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:05 smithi107 ceph-mon[102221]: pgmap v442: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 768 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 949 op/s; 2750/36903 objects degraded (7.452%); 0 B/s, 40 objects/s recovering 2024-09-21T09:59:05.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:05 smithi107 ceph-mon[105022]: pgmap v442: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 768 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 949 op/s; 2750/36903 objects degraded (7.452%); 0 B/s, 40 objects/s recovering 2024-09-21T09:59:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:05 smithi129 ceph-mon[88471]: pgmap v442: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 768 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 949 op/s; 2750/36903 objects degraded (7.452%); 0 B/s, 40 objects/s recovering 2024-09-21T09:59:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:07 smithi129 ceph-mon[88471]: pgmap v443: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 646 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 901 op/s; 2453/31068 objects degraded (7.896%); 0 B/s, 65 objects/s recovering 2024-09-21T09:59:08.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:07 smithi107 ceph-mon[102221]: pgmap v443: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 646 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 901 op/s; 2453/31068 objects degraded (7.896%); 0 B/s, 65 objects/s recovering 2024-09-21T09:59:08.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:07 smithi107 ceph-mon[105022]: pgmap v443: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 646 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 901 op/s; 2453/31068 objects degraded (7.896%); 0 B/s, 65 objects/s recovering 2024-09-21T09:59:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:09 smithi129 ceph-mon[88471]: pgmap v444: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 540 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 879 op/s; 2453/25995 objects degraded (9.436%); 0 B/s, 45 objects/s recovering 2024-09-21T09:59:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:09 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2453/25995 objects degraded (9.436%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:59:10.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:09 smithi107 ceph-mon[102221]: pgmap v444: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 540 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 879 op/s; 2453/25995 objects degraded (9.436%); 0 B/s, 45 objects/s recovering 2024-09-21T09:59:10.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:09 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2453/25995 objects degraded (9.436%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:59:10.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:09 smithi107 ceph-mon[105022]: pgmap v444: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 540 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 879 op/s; 2453/25995 objects degraded (9.436%); 0 B/s, 45 objects/s recovering 2024-09-21T09:59:10.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:09 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2453/25995 objects degraded (9.436%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:59:10.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:59:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:59:10.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:11.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:11 smithi129 ceph-mon[88471]: pgmap v445: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 385 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 954 op/s; 2146/18519 objects degraded (11.588%); 0 B/s, 70 objects/s recovering 2024-09-21T09:59:11.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:12.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:11 smithi107 ceph-mon[102221]: pgmap v445: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 385 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 954 op/s; 2146/18519 objects degraded (11.588%); 0 B/s, 70 objects/s recovering 2024-09-21T09:59:12.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:12.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:59:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:59:11] "GET /metrics HTTP/1.1" 200 36502 "" "Prometheus/2.51.0" 2024-09-21T09:59:12.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:11 smithi107 ceph-mon[105022]: pgmap v445: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 385 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 954 op/s; 2146/18519 objects degraded (11.588%); 0 B/s, 70 objects/s recovering 2024-09-21T09:59:12.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:13 smithi129 ceph-mon[88471]: pgmap v446: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 366 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 769 op/s; 2146/17628 objects degraded (12.174%); 0 B/s, 50 objects/s recovering 2024-09-21T09:59:14.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:13 smithi107 ceph-mon[102221]: pgmap v446: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 366 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 769 op/s; 2146/17628 objects degraded (12.174%); 0 B/s, 50 objects/s recovering 2024-09-21T09:59:14.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:13 smithi107 ceph-mon[105022]: pgmap v446: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 366 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 769 op/s; 2146/17628 objects degraded (12.174%); 0 B/s, 50 objects/s recovering 2024-09-21T09:59:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:14 smithi129 ceph-mon[88471]: osdmap e137: 8 total, 8 up, 8 in 2024-09-21T09:59:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:14 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2146/17628 objects degraded (12.174%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:59:15.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:14 smithi107 ceph-mon[102221]: osdmap e137: 8 total, 8 up, 8 in 2024-09-21T09:59:15.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:14 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2146/17628 objects degraded (12.174%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:59:15.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:14 smithi107 ceph-mon[105022]: osdmap e137: 8 total, 8 up, 8 in 2024-09-21T09:59:15.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:14 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2146/17628 objects degraded (12.174%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T09:59:15.804 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Removed 125881 objects 2024-09-21T09:59:15.804 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Clean up completed and total clean up time :144.546 2024-09-21T09:59:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:15 smithi129 ceph-mon[88471]: pgmap v448: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 203 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 903 op/s; 2146/9774 objects degraded (21.956%); 0 B/s, 60 objects/s recovering 2024-09-21T09:59:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:15 smithi129 ceph-mon[88471]: osdmap e138: 8 total, 8 up, 8 in 2024-09-21T09:59:16.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:15 smithi107 ceph-mon[102221]: pgmap v448: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 203 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 903 op/s; 2146/9774 objects degraded (21.956%); 0 B/s, 60 objects/s recovering 2024-09-21T09:59:16.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:15 smithi107 ceph-mon[102221]: osdmap e138: 8 total, 8 up, 8 in 2024-09-21T09:59:16.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:15 smithi107 ceph-mon[105022]: pgmap v448: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 203 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 903 op/s; 2146/9774 objects degraded (21.956%); 0 B/s, 60 objects/s recovering 2024-09-21T09:59:16.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:15 smithi107 ceph-mon[105022]: osdmap e138: 8 total, 8 up, 8 in 2024-09-21T09:59:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:17 smithi129 ceph-mon[88471]: pgmap v450: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 95 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 890 op/s; 1724/4626 objects degraded (37.268%); 0 B/s, 91 objects/s recovering 2024-09-21T09:59:18.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:17 smithi107 ceph-mon[102221]: pgmap v450: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 95 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 890 op/s; 1724/4626 objects degraded (37.268%); 0 B/s, 91 objects/s recovering 2024-09-21T09:59:18.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:17 smithi107 ceph-mon[105022]: pgmap v450: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 95 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 890 op/s; 1724/4626 objects degraded (37.268%); 0 B/s, 91 objects/s recovering 2024-09-21T09:59:19.143 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-21T09:59:19.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-21T09:59:19.420 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:19 smithi107 ceph-mon[102221]: pgmap v451: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 37 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 1724/1818 objects degraded (94.829%); 0 B/s, 52 objects/s recovering 2024-09-21T09:59:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:19 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 1724/1818 objects degraded (94.829%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T09:59:19.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:19 smithi107 ceph-mon[105022]: pgmap v451: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 37 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 1724/1818 objects degraded (94.829%); 0 B/s, 52 objects/s recovering 2024-09-21T09:59:19.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:19 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 1724/1818 objects degraded (94.829%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T09:59:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:19 smithi129 ceph-mon[88471]: pgmap v451: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 37 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 1724/1818 objects degraded (94.829%); 0 B/s, 52 objects/s recovering 2024-09-21T09:59:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:19 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 1724/1818 objects degraded (94.829%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T09:59:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:59:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:59:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:21.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:21 smithi107 ceph-mon[102221]: pgmap v452: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 3.1 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 725 op/s; 601/201 objects degraded (299.005%); 0 B/s, 193 objects/s recovering 2024-09-21T09:59:21.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:21 smithi107 ceph-mon[105022]: pgmap v452: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 3.1 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 725 op/s; 601/201 objects degraded (299.005%); 0 B/s, 193 objects/s recovering 2024-09-21T09:59:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:21 smithi129 ceph-mon[88471]: pgmap v452: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 3.1 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 725 op/s; 601/201 objects degraded (299.005%); 0 B/s, 193 objects/s recovering 2024-09-21T09:59:22.091 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:59:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:59:21] "GET /metrics HTTP/1.1" 200 36502 "" "Prometheus/2.51.0" 2024-09-21T09:59:22.679 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_3' removed 2024-09-21T09:59:23.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:22 smithi107 ceph-mon[102221]: osdmap e139: 8 total, 8 up, 8 in 2024-09-21T09:59:23.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:22 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1169182352' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T09:59:23.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:22 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T09:59:23.030 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:22 smithi107 ceph-mon[105022]: osdmap e139: 8 total, 8 up, 8 in 2024-09-21T09:59:23.030 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:22 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1169182352' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T09:59:23.030 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:22 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T09:59:23.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:22 smithi129 ceph-mon[88471]: osdmap e139: 8 total, 8 up, 8 in 2024-09-21T09:59:23.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:22 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1169182352' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T09:59:23.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:22 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T09:59:23.396 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-21T09:59:23.396 INFO:teuthology.task.print:**** done end radosbench.yaml 2024-09-21T09:59:23.396 DEBUG:teuthology.parallel:result is None 2024-09-21T09:59:23.396 INFO:teuthology.run_tasks:Running task print... 2024-09-21T09:59:23.412 INFO:teuthology.task.print:**** done upgrading first half of cluster **** 2024-09-21T09:59:23.412 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-09-21T09:59:23.463 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T09:59:23.463 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T09:59:23.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T09:59:23.708 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:23.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:23 smithi107 ceph-mon[102221]: pgmap v454: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 403 op/s; 601/78 objects degraded (770.513%); 0 B/s, 193 objects/s recovering 2024-09-21T09:59:23.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:23 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T09:59:23.982 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:23 smithi107 ceph-mon[102221]: osdmap e140: 8 total, 8 up, 8 in 2024-09-21T09:59:23.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:23 smithi107 ceph-mon[105022]: pgmap v454: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 403 op/s; 601/78 objects degraded (770.513%); 0 B/s, 193 objects/s recovering 2024-09-21T09:59:23.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:23 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T09:59:23.982 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:23 smithi107 ceph-mon[105022]: osdmap e140: 8 total, 8 up, 8 in 2024-09-21T09:59:24.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:23 smithi129 ceph-mon[88471]: pgmap v454: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 88 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 403 op/s; 601/78 objects degraded (770.513%); 0 B/s, 193 objects/s recovering 2024-09-21T09:59:24.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:23 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T09:59:24.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:23 smithi129 ceph-mon[88471]: osdmap e140: 8 total, 8 up, 8 in 2024-09-21T09:59:25.058 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:24 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 601/78 objects degraded (770.513%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T09:59:25.058 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:24 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 601/78 objects degraded (770.513%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T09:59:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:24 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 601/78 objects degraded (770.513%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T09:59:25.804 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:59:25.805 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:25 smithi107 ceph-mon[102221]: pgmap v456: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:25.806 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:25 smithi107 ceph-mon[102221]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 601/78 objects degraded (770.513%), 1 pg degraded, 1 pg undersized) 2024-09-21T09:59:25.806 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:25 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T09:59:25.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:25 smithi107 ceph-mon[105022]: pgmap v456: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:25.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:25 smithi107 ceph-mon[105022]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 601/78 objects degraded (770.513%), 1 pg degraded, 1 pg undersized) 2024-09-21T09:59:25.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:25 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T09:59:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:25 smithi129 ceph-mon[88471]: pgmap v456: 73 pgs: 73 active+clean; 582 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:25 smithi129 ceph-mon[88471]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 601/78 objects degraded (770.513%), 1 pg degraded, 1 pg undersized) 2024-09-21T09:59:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:25 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T09:59:26.432 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":41,"flags":0,"active_gid":25219,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1060510503},{"type":"v1","addr":"172.21.15.107:6801","nonce":1060510503}]},"active_addr":"172.21.15.107:6801/1060510503","active_change":"2024-09-21T09:46:25.797449+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":25216,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.107:8443/","prometheus":"http://172.21.15.107:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":76,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":3176015646}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":1640345110}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":3728874377}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":479311957}]}]} 2024-09-21T09:59:26.436 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T09:59:26.437 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T09:59:26.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:59:26.679 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:26.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:26 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2823740277' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T09:59:26.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:59:26.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:26.944 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:26 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2823740277' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T09:59:26.944 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:59:26.945 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:26 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2823740277' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T09:59:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:59:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:27.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:27 smithi107 ceph-mon[102221]: pgmap v457: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:27.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:27 smithi107 ceph-mon[105022]: pgmap v457: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:27 smithi129 ceph-mon[88471]: pgmap v457: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:28.491 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:59:28.491 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":140,"fsid":"86c49514-77fc-11ef-baf6-efdc52797490","created":"2024-09-21T09:34:03.635940+0000","modified":"2024-09-21T09:59:22.662912+0000","last_up_change":"2024-09-21T09:56:11.548115+0000","last_in_change":"2024-09-21T09:38:46.049422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:37:08.948207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:39:33.152023+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:40:47.823974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:40:48.313916+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"635c1eb0-29b1-4fdd-bd47-881ef05e49bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":77,"up_from":81,"up_thru":129,"down_at":78,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1236110780},{"type":"v1","addr":"172.21.15.107:6803","nonce":1236110780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1236110780},{"type":"v1","addr":"172.21.15.107:6805","nonce":1236110780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1236110780},{"type":"v1","addr":"172.21.15.107:6809","nonce":1236110780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1236110780},{"type":"v1","addr":"172.21.15.107:6807","nonce":1236110780}]},"public_addr":"172.21.15.107:6803/1236110780","cluster_addr":"172.21.15.107:6805/1236110780","heartbeat_back_addr":"172.21.15.107:6809/1236110780","heartbeat_front_addr":"172.21.15.107:6807/1236110780","state":["exists","up"]},{"osd":1,"uuid":"a39b6a18-939d-4999-89b2-1126c1b2d1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":91,"up_from":96,"up_thru":138,"down_at":92,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2988029448},{"type":"v1","addr":"172.21.15.107:6811","nonce":2988029448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2988029448},{"type":"v1","addr":"172.21.15.107:6813","nonce":2988029448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2988029448},{"type":"v1","addr":"172.21.15.107:6817","nonce":2988029448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2988029448},{"type":"v1","addr":"172.21.15.107:6815","nonce":2988029448}]},"public_addr":"172.21.15.107:6811/2988029448","cluster_addr":"172.21.15.107:6813/2988029448","heartbeat_back_addr":"172.21.15.107:6817/2988029448","heartbeat_front_addr":"172.21.15.107:6815/2988029448","state":["exists","up"]},{"osd":2,"uuid":"b01c5f1b-77e4-462f-80d0-fd1d3afb786b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":101,"up_from":105,"up_thru":126,"down_at":102,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1583228557},{"type":"v1","addr":"172.21.15.107:6819","nonce":1583228557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":1583228557},{"type":"v1","addr":"172.21.15.107:6821","nonce":1583228557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1583228557},{"type":"v1","addr":"172.21.15.107:6825","nonce":1583228557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":1583228557},{"type":"v1","addr":"172.21.15.107:6823","nonce":1583228557}]},"public_addr":"172.21.15.107:6819/1583228557","cluster_addr":"172.21.15.107:6821/1583228557","heartbeat_back_addr":"172.21.15.107:6825/1583228557","heartbeat_front_addr":"172.21.15.107:6823/1583228557","state":["exists","up"]},{"osd":3,"uuid":"5135d269-b6c4-4a9d-bcf9-ffe19bc5c9ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":122,"up_from":126,"up_thru":139,"down_at":123,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":860985313},{"type":"v1","addr":"172.21.15.107:6827","nonce":860985313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":860985313},{"type":"v1","addr":"172.21.15.107:6829","nonce":860985313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":860985313},{"type":"v1","addr":"172.21.15.107:6833","nonce":860985313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":860985313},{"type":"v1","addr":"172.21.15.107:6831","nonce":860985313}]},"public_addr":"172.21.15.107:6827/860985313","cluster_addr":"172.21.15.107:6829/860985313","heartbeat_back_addr":"172.21.15.107:6833/860985313","heartbeat_front_addr":"172.21.15.107:6831/860985313","state":["exists","up"]},{"osd":4,"uuid":"23242972-e341-4773-809f-299aabeacd56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6801","nonce":4101249335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6803","nonce":4101249335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6807","nonce":4101249335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6805","nonce":4101249335}]},"public_addr":"172.21.15.129:6801/4101249335","cluster_addr":"172.21.15.129:6803/4101249335","heartbeat_back_addr":"172.21.15.129:6807/4101249335","heartbeat_front_addr":"172.21.15.129:6805/4101249335","state":["exists","up"]},{"osd":5,"uuid":"f9da25cd-1680-4d97-a9d1-98dded10766f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6809","nonce":2733195233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6811","nonce":2733195233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6815","nonce":2733195233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6813","nonce":2733195233}]},"public_addr":"172.21.15.129:6809/2733195233","cluster_addr":"172.21.15.129:6811/2733195233","heartbeat_back_addr":"172.21.15.129:6815/2733195233","heartbeat_front_addr":"172.21.15.129:6813/2733195233","state":["exists","up"]},{"osd":6,"uuid":"d6d2bd99-0178-409c-a548-cbc30b8b76fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6817","nonce":3400496608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6819","nonce":3400496608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6823","nonce":3400496608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6821","nonce":3400496608}]},"public_addr":"172.21.15.129:6817/3400496608","cluster_addr":"172.21.15.129:6819/3400496608","heartbeat_back_addr":"172.21.15.129:6823/3400496608","heartbeat_front_addr":"172.21.15.129:6821/3400496608","state":["exists","up"]},{"osd":7,"uuid":"2c0a9723-0328-4134-b5d4-cf307839018a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6825","nonce":101668996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6827","nonce":101668996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6831","nonce":101668996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6829","nonce":101668996}]},"public_addr":"172.21.15.129:6825/101668996","cluster_addr":"172.21.15.129:6827/101668996","heartbeat_back_addr":"172.21.15.129:6831/101668996","heartbeat_front_addr":"172.21.15.129:6829/101668996","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:36:18.300841+0000","dead_epoch":77},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:36:42.334925+0000","dead_epoch":91},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:06.141467+0000","dead_epoch":101},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:30.344630+0000","dead_epoch":122},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:51.661657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:38:14.654068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:38:38.089961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:39:01.119878+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/3807929340":"2024-09-22T09:46:25.797234+0000","172.21.15.107:6800/1580746173":"2024-09-22T09:46:25.797234+0000","172.21.15.107:0/3989069453":"2024-09-22T09:46:25.797234+0000","172.21.15.107:0/4262440050":"2024-09-22T09:46:25.797234+0000","172.21.15.107:0/1261754960":"2024-09-22T09:46:25.797234+0000","172.21.15.107:0/1942117255":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/627572584":"2024-09-22T09:46:25.797234+0000","172.21.15.107:0/51270168":"2024-09-22T09:43:44.074024+0000","172.21.15.107:0/3025710247":"2024-09-22T09:43:44.074024+0000","172.21.15.107:6801/4208383830":"2024-09-22T09:34:56.643317+0000","172.21.15.129:0/306463122":"2024-09-22T09:45:31.607526+0000","172.21.15.107:6800/4208383830":"2024-09-22T09:34:56.643317+0000","172.21.15.107:6800/4055614859":"2024-09-22T09:43:44.074024+0000","172.21.15.107:0/4103644615":"2024-09-22T09:39:30.975484+0000","172.21.15.107:6801/1580746173":"2024-09-22T09:46:25.797234+0000","172.21.15.107:6801/2716631102":"2024-09-22T09:39:30.975484+0000","172.21.15.107:6800/2716631102":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/637981220":"2024-09-22T09:34:34.697433+0000","172.21.15.107:6801/1686228827":"2024-09-22T09:34:34.697433+0000","172.21.15.129:6835/2445466552":"2024-09-22T09:45:31.607526+0000","172.21.15.129:6834/2445466552":"2024-09-22T09:45:31.607526+0000","172.21.15.107:0/2526044436":"2024-09-22T09:39:30.975484+0000","172.21.15.129:0/3127898221":"2024-09-22T09:45:31.607526+0000","172.21.15.107:6800/1686228827":"2024-09-22T09:34:34.697433+0000","172.21.15.129:0/4112718236":"2024-09-22T09:45:31.607526+0000","172.21.15.107:0/2156530655":"2024-09-22T09:39:30.975484+0000","172.21.15.107:6801/4055614859":"2024-09-22T09:43:44.074024+0000","172.21.15.129:0/2239836304":"2024-09-22T09:45:31.607526+0000","172.21.15.107:0/1526680749":"2024-09-22T09:43:44.074024+0000","172.21.15.107:0/837896416":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/3424783814":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/2701254891":"2024-09-22T09:43:44.074024+0000","172.21.15.107:0/782384657":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/2492859118":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/218684017":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/2945318176":"2024-09-22T09:46:25.797234+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:59:28.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:28 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2620896223' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T09:59:28.781 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:28 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2620896223' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T09:59:29.106 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T09:59:29.107 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:59:29.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:28 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2620896223' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T09:59:29.350 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:29.724 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:29 smithi107 ceph-mon[102221]: pgmap v458: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:29.724 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:29 smithi107 ceph-mon[105022]: pgmap v458: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:30.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:29 smithi129 ceph-mon[88471]: pgmap v458: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:59:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:59:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:31.284 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T09:59:31.284 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":140,"fsid":"86c49514-77fc-11ef-baf6-efdc52797490","created":"2024-09-21T09:34:03.635940+0000","modified":"2024-09-21T09:59:22.662912+0000","last_up_change":"2024-09-21T09:56:11.548115+0000","last_in_change":"2024-09-21T09:38:46.049422+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:37:08.948207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:39:33.152023+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:40:47.823974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:40:48.313916+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"635c1eb0-29b1-4fdd-bd47-881ef05e49bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":77,"up_from":81,"up_thru":129,"down_at":78,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1236110780},{"type":"v1","addr":"172.21.15.107:6803","nonce":1236110780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1236110780},{"type":"v1","addr":"172.21.15.107:6805","nonce":1236110780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1236110780},{"type":"v1","addr":"172.21.15.107:6809","nonce":1236110780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1236110780},{"type":"v1","addr":"172.21.15.107:6807","nonce":1236110780}]},"public_addr":"172.21.15.107:6803/1236110780","cluster_addr":"172.21.15.107:6805/1236110780","heartbeat_back_addr":"172.21.15.107:6809/1236110780","heartbeat_front_addr":"172.21.15.107:6807/1236110780","state":["exists","up"]},{"osd":1,"uuid":"a39b6a18-939d-4999-89b2-1126c1b2d1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":91,"up_from":96,"up_thru":138,"down_at":92,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2988029448},{"type":"v1","addr":"172.21.15.107:6811","nonce":2988029448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2988029448},{"type":"v1","addr":"172.21.15.107:6813","nonce":2988029448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2988029448},{"type":"v1","addr":"172.21.15.107:6817","nonce":2988029448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2988029448},{"type":"v1","addr":"172.21.15.107:6815","nonce":2988029448}]},"public_addr":"172.21.15.107:6811/2988029448","cluster_addr":"172.21.15.107:6813/2988029448","heartbeat_back_addr":"172.21.15.107:6817/2988029448","heartbeat_front_addr":"172.21.15.107:6815/2988029448","state":["exists","up"]},{"osd":2,"uuid":"b01c5f1b-77e4-462f-80d0-fd1d3afb786b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":101,"up_from":105,"up_thru":126,"down_at":102,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1583228557},{"type":"v1","addr":"172.21.15.107:6819","nonce":1583228557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":1583228557},{"type":"v1","addr":"172.21.15.107:6821","nonce":1583228557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1583228557},{"type":"v1","addr":"172.21.15.107:6825","nonce":1583228557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":1583228557},{"type":"v1","addr":"172.21.15.107:6823","nonce":1583228557}]},"public_addr":"172.21.15.107:6819/1583228557","cluster_addr":"172.21.15.107:6821/1583228557","heartbeat_back_addr":"172.21.15.107:6825/1583228557","heartbeat_front_addr":"172.21.15.107:6823/1583228557","state":["exists","up"]},{"osd":3,"uuid":"5135d269-b6c4-4a9d-bcf9-ffe19bc5c9ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":122,"up_from":126,"up_thru":139,"down_at":123,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":860985313},{"type":"v1","addr":"172.21.15.107:6827","nonce":860985313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":860985313},{"type":"v1","addr":"172.21.15.107:6829","nonce":860985313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":860985313},{"type":"v1","addr":"172.21.15.107:6833","nonce":860985313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":860985313},{"type":"v1","addr":"172.21.15.107:6831","nonce":860985313}]},"public_addr":"172.21.15.107:6827/860985313","cluster_addr":"172.21.15.107:6829/860985313","heartbeat_back_addr":"172.21.15.107:6833/860985313","heartbeat_front_addr":"172.21.15.107:6831/860985313","state":["exists","up"]},{"osd":4,"uuid":"23242972-e341-4773-809f-299aabeacd56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6801","nonce":4101249335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6803","nonce":4101249335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6807","nonce":4101249335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6805","nonce":4101249335}]},"public_addr":"172.21.15.129:6801/4101249335","cluster_addr":"172.21.15.129:6803/4101249335","heartbeat_back_addr":"172.21.15.129:6807/4101249335","heartbeat_front_addr":"172.21.15.129:6805/4101249335","state":["exists","up"]},{"osd":5,"uuid":"f9da25cd-1680-4d97-a9d1-98dded10766f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":133,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6809","nonce":2733195233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6811","nonce":2733195233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6815","nonce":2733195233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6813","nonce":2733195233}]},"public_addr":"172.21.15.129:6809/2733195233","cluster_addr":"172.21.15.129:6811/2733195233","heartbeat_back_addr":"172.21.15.129:6815/2733195233","heartbeat_front_addr":"172.21.15.129:6813/2733195233","state":["exists","up"]},{"osd":6,"uuid":"d6d2bd99-0178-409c-a548-cbc30b8b76fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6817","nonce":3400496608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6819","nonce":3400496608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6823","nonce":3400496608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6821","nonce":3400496608}]},"public_addr":"172.21.15.129:6817/3400496608","cluster_addr":"172.21.15.129:6819/3400496608","heartbeat_back_addr":"172.21.15.129:6823/3400496608","heartbeat_front_addr":"172.21.15.129:6821/3400496608","state":["exists","up"]},{"osd":7,"uuid":"2c0a9723-0328-4134-b5d4-cf307839018a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6825","nonce":101668996}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6827","nonce":101668996}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6831","nonce":101668996}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":101668996},{"type":"v1","addr":"172.21.15.129:6829","nonce":101668996}]},"public_addr":"172.21.15.129:6825/101668996","cluster_addr":"172.21.15.129:6827/101668996","heartbeat_back_addr":"172.21.15.129:6831/101668996","heartbeat_front_addr":"172.21.15.129:6829/101668996","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:36:18.300841+0000","dead_epoch":77},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:36:42.334925+0000","dead_epoch":91},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:06.141467+0000","dead_epoch":101},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:30.344630+0000","dead_epoch":122},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:51.661657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:38:14.654068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:38:38.089961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:39:01.119878+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/3807929340":"2024-09-22T09:46:25.797234+0000","172.21.15.107:6800/1580746173":"2024-09-22T09:46:25.797234+0000","172.21.15.107:0/3989069453":"2024-09-22T09:46:25.797234+0000","172.21.15.107:0/4262440050":"2024-09-22T09:46:25.797234+0000","172.21.15.107:0/1261754960":"2024-09-22T09:46:25.797234+0000","172.21.15.107:0/1942117255":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/627572584":"2024-09-22T09:46:25.797234+0000","172.21.15.107:0/51270168":"2024-09-22T09:43:44.074024+0000","172.21.15.107:0/3025710247":"2024-09-22T09:43:44.074024+0000","172.21.15.107:6801/4208383830":"2024-09-22T09:34:56.643317+0000","172.21.15.129:0/306463122":"2024-09-22T09:45:31.607526+0000","172.21.15.107:6800/4208383830":"2024-09-22T09:34:56.643317+0000","172.21.15.107:6800/4055614859":"2024-09-22T09:43:44.074024+0000","172.21.15.107:0/4103644615":"2024-09-22T09:39:30.975484+0000","172.21.15.107:6801/1580746173":"2024-09-22T09:46:25.797234+0000","172.21.15.107:6801/2716631102":"2024-09-22T09:39:30.975484+0000","172.21.15.107:6800/2716631102":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/637981220":"2024-09-22T09:34:34.697433+0000","172.21.15.107:6801/1686228827":"2024-09-22T09:34:34.697433+0000","172.21.15.129:6835/2445466552":"2024-09-22T09:45:31.607526+0000","172.21.15.129:6834/2445466552":"2024-09-22T09:45:31.607526+0000","172.21.15.107:0/2526044436":"2024-09-22T09:39:30.975484+0000","172.21.15.129:0/3127898221":"2024-09-22T09:45:31.607526+0000","172.21.15.107:6800/1686228827":"2024-09-22T09:34:34.697433+0000","172.21.15.129:0/4112718236":"2024-09-22T09:45:31.607526+0000","172.21.15.107:0/2156530655":"2024-09-22T09:39:30.975484+0000","172.21.15.107:6801/4055614859":"2024-09-22T09:43:44.074024+0000","172.21.15.129:0/2239836304":"2024-09-22T09:45:31.607526+0000","172.21.15.107:0/1526680749":"2024-09-22T09:43:44.074024+0000","172.21.15.107:0/837896416":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/3424783814":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/2701254891":"2024-09-22T09:43:44.074024+0000","172.21.15.107:0/782384657":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/2492859118":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/218684017":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/2945318176":"2024-09-22T09:46:25.797234+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:59:31.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:31 smithi107 ceph-mon[102221]: pgmap v459: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:31.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:31 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3434933186' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T09:59:31.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:31 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:59:31.860 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:59:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:59:31] "GET /metrics HTTP/1.1" 200 34159 "" "Prometheus/2.51.0" 2024-09-21T09:59:31.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:31 smithi107 ceph-mon[105022]: pgmap v459: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:31.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:31 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3434933186' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T09:59:31.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:31 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:59:31.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T09:59:31.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T09:59:31.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T09:59:31.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T09:59:31.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T09:59:31.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T09:59:31.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T09:59:31.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T09:59:32.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:31 smithi129 ceph-mon[88471]: pgmap v459: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:32.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:31 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3434933186' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T09:59:32.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:31 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:59:32.407 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:32.411 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:32.691 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:32.691 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:32.692 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:32.692 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:33.431 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:33.431 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:33.803 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:33 smithi107 ceph-mon[102221]: pgmap v460: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:33.803 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:33 smithi107 ceph-mon[105022]: pgmap v460: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:34.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:33 smithi129 ceph-mon[88471]: pgmap v460: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:35.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:35 smithi107 ceph-mon[102221]: pgmap v461: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:35.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:35 smithi107 ceph-mon[105022]: pgmap v461: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:36.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:35 smithi129 ceph-mon[88471]: pgmap v461: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:37 smithi107 ceph-mon[102221]: pgmap v462: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:37.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:37 smithi107 ceph-mon[105022]: pgmap v462: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:38.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:37 smithi129 ceph-mon[88471]: pgmap v462: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:38.496 INFO:teuthology.orchestra.run.smithi107.stdout:541165879339 2024-09-21T09:59:38.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T09:59:38.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:59:38.816 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:59:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T09:59:40.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:39 smithi107 ceph-mon[102221]: pgmap v463: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:40.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:39 smithi107 ceph-mon[105022]: pgmap v463: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:40.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:39 smithi129 ceph-mon[88471]: pgmap v463: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:40.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:59:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:59:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:40.722 INFO:teuthology.orchestra.run.smithi107.stdout:206158430459 2024-09-21T09:59:40.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T09:59:41.768 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:41 smithi107 ceph-mon[105022]: pgmap v464: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:41.768 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:41.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:41 smithi107 ceph-mon[102221]: pgmap v464: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:41.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:42.001 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:41 smithi129 ceph-mon[88471]: pgmap v464: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:42.001 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:42.018 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:59:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:59:41] "GET /metrics HTTP/1.1" 200 34152 "" "Prometheus/2.51.0" 2024-09-21T09:59:43.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:43 smithi107 ceph-mon[102221]: pgmap v465: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:43.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:43 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:59:43.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:43 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:59:43.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:43 smithi107 ceph-mon[105022]: pgmap v465: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:43.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:43 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:59:43.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:43 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:59:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:43 smithi129 ceph-mon[88471]: pgmap v465: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:43 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:59:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:43 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T09:59:44.108 INFO:teuthology.orchestra.run.smithi107.stdout:347892351107 2024-09-21T09:59:44.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T09:59:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:45 smithi107 ceph-mon[102221]: pgmap v466: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:45.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:45 smithi107 ceph-mon[105022]: pgmap v466: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:45 smithi129 ceph-mon[88471]: pgmap v466: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:46.807 INFO:teuthology.orchestra.run.smithi107.stdout:412316860526 2024-09-21T09:59:46.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T09:59:47.184 INFO:teuthology.orchestra.run.smithi107.stdout:180388626688 2024-09-21T09:59:47.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T09:59:47.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:47 smithi107 ceph-mon[102221]: pgmap v467: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:47.531 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:47 smithi107 ceph-mon[105022]: pgmap v467: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:47.557 INFO:teuthology.orchestra.run.smithi107.stdout:128849019146 2024-09-21T09:59:47.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T09:59:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:47 smithi129 ceph-mon[88471]: pgmap v467: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:48.737 INFO:teuthology.orchestra.run.smithi107.stdout:450971566181 2024-09-21T09:59:48.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T09:59:49.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:49 smithi107 ceph-mon[102221]: pgmap v468: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:49.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:49 smithi107 ceph-mon[105022]: pgmap v468: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:49 smithi129 ceph-mon[88471]: pgmap v468: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:49.848 INFO:teuthology.orchestra.run.smithi107.stdout:154618822918 2024-09-21T09:59:49.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T09:59:49.992 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:49.998 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:50.000 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:50.001 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:50.005 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:50.008 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:50.008 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:59:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 09:59:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T09:59:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T09:59:51.488 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:51 smithi129 ceph-mon[88471]: pgmap v469: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:51.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:51 smithi107 ceph-mon[102221]: pgmap v469: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:51.564 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:51 smithi107 ceph-mon[105022]: pgmap v469: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:51.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 09:59:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:09:59:51] "GET /metrics HTTP/1.1" 200 34152 "" "Prometheus/2.51.0" 2024-09-21T09:59:53.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:53 smithi107 ceph-mon[102221]: pgmap v470: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:53.530 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:53 smithi107 ceph-mon[105022]: pgmap v470: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:53 smithi129 ceph-mon[88471]: pgmap v470: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:55.438 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:55 smithi107 ceph-mon[102221]: pgmap v471: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:55.439 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:55 smithi107 ceph-mon[105022]: pgmap v471: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:55 smithi129 ceph-mon[88471]: pgmap v471: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:56.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:56 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1056171575' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T09:59:56.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:56.261 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:56 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1056171575' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T09:59:56.262 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:56 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1056171575' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T09:59:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T09:59:57.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:57 smithi107 ceph-mon[102221]: pgmap v472: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:57.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:57 smithi107 ceph-mon[105022]: pgmap v472: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:57.589 INFO:teuthology.orchestra.run.smithi107.stdout:412316860527 2024-09-21T09:59:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:57 smithi129 ceph-mon[88471]: pgmap v472: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:58.146 INFO:tasks.cephadm.ceph_manager.ceph:need seq 412316860526 got 412316860527 for osd.1 2024-09-21T09:59:58.146 DEBUG:teuthology.parallel:result is None 2024-09-21T09:59:58.923 INFO:teuthology.orchestra.run.smithi107.stdout:128849019149 2024-09-21T09:59:59.492 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:59 smithi107 ceph-mon[102221]: pgmap v473: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:59.492 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 09:59:59 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4111504061' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T09:59:59.492 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:59 smithi107 ceph-mon[105022]: pgmap v473: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:59.492 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 09:59:59 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4111504061' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T09:59:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:59 smithi129 ceph-mon[88471]: pgmap v473: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 09:59:59 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4111504061' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T10:00:00.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:00 smithi107 ceph-mon[102221]: overall HEALTH_OK 2024-09-21T10:00:00.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:00 smithi107 ceph-mon[105022]: overall HEALTH_OK 2024-09-21T10:00:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:00 smithi129 ceph-mon[88471]: overall HEALTH_OK 2024-09-21T10:00:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:00:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:00:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:00.887 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849019146 got 128849019149 for osd.4 2024-09-21T10:00:00.887 DEBUG:teuthology.parallel:result is None 2024-09-21T10:00:01.244 INFO:teuthology.orchestra.run.smithi107.stdout:347892351111 2024-09-21T10:00:01.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:01 smithi107 ceph-mon[102221]: pgmap v474: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:01.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:01 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/967176323' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:00:01.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:01 smithi107 ceph-mon[105022]: pgmap v474: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:01.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:01 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/967176323' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:00:01.448 INFO:teuthology.orchestra.run.smithi107.stdout:206158430463 2024-09-21T10:00:01.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:01 smithi129 ceph-mon[88471]: pgmap v474: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:01.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:01 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/967176323' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:00:02.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:00:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:00:01] "GET /metrics HTTP/1.1" 200 34153 "" "Prometheus/2.51.0" 2024-09-21T10:00:02.141 INFO:teuthology.orchestra.run.smithi107.stdout:180388626691 2024-09-21T10:00:02.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:02 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3441228142' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:00:02.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:02 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3287441108' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:00:02.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:02 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2886757822' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:00:02.515 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:02 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3441228142' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:00:02.515 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:02 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3287441108' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:00:02.515 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:02 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2886757822' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:00:02.538 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892351107 got 347892351111 for osd.0 2024-09-21T10:00:02.538 DEBUG:teuthology.parallel:result is None 2024-09-21T10:00:02.615 INFO:teuthology.orchestra.run.smithi107.stdout:541165879343 2024-09-21T10:00:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:02 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3441228142' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:00:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:02 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3287441108' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:00:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:02 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2886757822' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:00:03.572 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:03 smithi107 ceph-mon[102221]: pgmap v475: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:03.572 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:03 smithi107 ceph-mon[105022]: pgmap v475: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:03.597 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430459 got 206158430463 for osd.7 2024-09-21T10:00:03.598 DEBUG:teuthology.parallel:result is None 2024-09-21T10:00:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:03 smithi129 ceph-mon[88471]: pgmap v475: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:04.146 INFO:tasks.cephadm.ceph_manager.ceph:need seq 541165879339 got 541165879343 for osd.3 2024-09-21T10:00:04.146 DEBUG:teuthology.parallel:result is None 2024-09-21T10:00:04.370 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:04.505 INFO:teuthology.orchestra.run.smithi107.stdout:450971566184 2024-09-21T10:00:05.182 INFO:tasks.cephadm.ceph_manager.ceph:need seq 450971566181 got 450971566184 for osd.2 2024-09-21T10:00:05.182 DEBUG:teuthology.parallel:result is None 2024-09-21T10:00:05.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:05 smithi107 ceph-mon[105022]: pgmap v476: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:05.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:05 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2796993410' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:00:05.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:05 smithi107 ceph-mon[102221]: pgmap v476: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:05.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:05 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2796993410' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:00:05.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:05 smithi129 ceph-mon[88471]: pgmap v476: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:05.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:05 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2796993410' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:00:06.663 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626688 got 180388626691 for osd.6 2024-09-21T10:00:06.663 DEBUG:teuthology.parallel:result is None 2024-09-21T10:00:07.008 INFO:teuthology.orchestra.run.smithi107.stdout:154618822921 2024-09-21T10:00:07.569 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:07 smithi107 ceph-mon[102221]: pgmap v477: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:07.569 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:07 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/415571286' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:00:07.569 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:07 smithi107 ceph-mon[105022]: pgmap v477: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:07.570 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:07 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/415571286' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:00:07.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:07 smithi129 ceph-mon[88471]: pgmap v477: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:07.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:07 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/415571286' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:00:07.963 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822918 got 154618822921 for osd.5 2024-09-21T10:00:07.963 DEBUG:teuthology.parallel:result is None 2024-09-21T10:00:07.963 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T10:00:07.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:00:08.355 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:09.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:09 smithi107 ceph-mon[102221]: pgmap v478: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:09.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:09 smithi107 ceph-mon[105022]: pgmap v478: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:09.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:09 smithi129 ceph-mon[88471]: pgmap v478: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:10.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:00:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:00:10.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:11.022 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:00:11.023 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:00:11.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:11 smithi107 ceph-mon[102221]: pgmap v479: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:11.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:11.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:00:11.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:00:11.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:11 smithi107 ceph-mon[105022]: pgmap v479: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:11.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:11.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:00:11.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:00:11.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:11 smithi129 ceph-mon[88471]: pgmap v479: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:11.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:11.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:00:11.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:00:11.789 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-09-21T10:00:10.128283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":213,"num_write_kb":1660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3644428,"kb_used_data":14348,"kb_used_omap":29,"kb_used_meta":3629922,"kb_avail":746316788,"statfs":{"total":767960285184,"available":764228390912,"internally_reserved":0,"allocated":14692352,"data_stored":9175278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30226,"internal_metadata":3717040622},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002706"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":103,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.685385+0000","last_change":"2024-09-21T09:56:12.582314+0000","last_active":"2024-09-21T09:59:22.685385+0000","last_peered":"2024-09-21T09:59:22.685385+0000","last_clean":"2024-09-21T09:59:22.685385+0000","last_became_active":"2024-09-21T09:56:12.581458+0000","last_became_peered":"2024-09-21T09:56:12.581458+0000","last_unstale":"2024-09-21T09:59:22.685385+0000","last_undegraded":"2024-09-21T09:59:22.685385+0000","last_fullsized":"2024-09-21T09:59:22.685385+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:17:21.624171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":106,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843561+0000","last_change":"2024-09-21T09:51:30.640091+0000","last_active":"2024-09-21T09:59:21.843561+0000","last_peered":"2024-09-21T09:59:21.843561+0000","last_clean":"2024-09-21T09:59:21.843561+0000","last_became_active":"2024-09-21T09:51:30.639739+0000","last_became_peered":"2024-09-21T09:51:30.639739+0000","last_unstale":"2024-09-21T09:59:21.843561+0000","last_undegraded":"2024-09-21T09:59:21.843561+0000","last_fullsized":"2024-09-21T09:59:21.843561+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:41.958415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660114+0000","last_change":"2024-09-21T09:56:12.583291+0000","last_active":"2024-09-21T09:59:21.660114+0000","last_peered":"2024-09-21T09:59:21.660114+0000","last_clean":"2024-09-21T09:59:21.660114+0000","last_became_active":"2024-09-21T09:56:12.582908+0000","last_became_peered":"2024-09-21T09:56:12.582908+0000","last_unstale":"2024-09-21T09:59:21.660114+0000","last_undegraded":"2024-09-21T09:59:21.660114+0000","last_fullsized":"2024-09-21T09:59:21.660114+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:04:07.985942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":153,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672057+0000","last_change":"2024-09-21T09:56:12.573112+0000","last_active":"2024-09-21T09:59:22.672057+0000","last_peered":"2024-09-21T09:59:22.672057+0000","last_clean":"2024-09-21T09:59:22.672057+0000","last_became_active":"2024-09-21T09:56:12.572835+0000","last_became_peered":"2024-09-21T09:56:12.572835+0000","last_unstale":"2024-09-21T09:59:22.672057+0000","last_undegraded":"2024-09-21T09:59:22.672057+0000","last_fullsized":"2024-09-21T09:59:22.672057+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:53.493530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":154,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.677271+0000","last_change":"2024-09-21T09:56:12.574922+0000","last_active":"2024-09-21T09:59:22.677271+0000","last_peered":"2024-09-21T09:59:22.677271+0000","last_clean":"2024-09-21T09:59:22.677271+0000","last_became_active":"2024-09-21T09:56:12.574450+0000","last_became_peered":"2024-09-21T09:56:12.574450+0000","last_unstale":"2024-09-21T09:59:22.677271+0000","last_undegraded":"2024-09-21T09:59:22.677271+0000","last_fullsized":"2024-09-21T09:59:22.677271+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:48.408302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":87,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843305+0000","last_change":"2024-09-21T09:40:49.101911+0000","last_active":"2024-09-21T09:59:21.843305+0000","last_peered":"2024-09-21T09:59:21.843305+0000","last_clean":"2024-09-21T09:59:21.843305+0000","last_became_active":"2024-09-21T09:40:49.101629+0000","last_became_peered":"2024-09-21T09:40:49.101629+0000","last_unstale":"2024-09-21T09:59:21.843305+0000","last_undegraded":"2024-09-21T09:59:21.843305+0000","last_fullsized":"2024-09-21T09:59:21.843305+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:07:19.215551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":103,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.686040+0000","last_change":"2024-09-21T09:56:12.601652+0000","last_active":"2024-09-21T09:59:22.686040+0000","last_peered":"2024-09-21T09:59:22.686040+0000","last_clean":"2024-09-21T09:59:22.686040+0000","last_became_active":"2024-09-21T09:56:12.583693+0000","last_became_peered":"2024-09-21T09:56:12.583693+0000","last_unstale":"2024-09-21T09:59:22.686040+0000","last_undegraded":"2024-09-21T09:59:22.686040+0000","last_fullsized":"2024-09-21T09:59:22.686040+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:24.992107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":112,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843253+0000","last_change":"2024-09-21T09:48:54.140948+0000","last_active":"2024-09-21T09:59:21.843253+0000","last_peered":"2024-09-21T09:59:21.843253+0000","last_clean":"2024-09-21T09:59:21.843253+0000","last_became_active":"2024-09-21T09:48:54.140641+0000","last_became_peered":"2024-09-21T09:48:54.140641+0000","last_unstale":"2024-09-21T09:59:21.843253+0000","last_undegraded":"2024-09-21T09:59:21.843253+0000","last_fullsized":"2024-09-21T09:59:21.843253+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:25.157599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":105,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843654+0000","last_change":"2024-09-21T09:48:54.127687+0000","last_active":"2024-09-21T09:59:21.843654+0000","last_peered":"2024-09-21T09:59:21.843654+0000","last_clean":"2024-09-21T09:59:21.843654+0000","last_became_active":"2024-09-21T09:48:54.127325+0000","last_became_peered":"2024-09-21T09:48:54.127325+0000","last_unstale":"2024-09-21T09:59:21.843654+0000","last_undegraded":"2024-09-21T09:59:21.843654+0000","last_fullsized":"2024-09-21T09:59:21.843654+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:30.004794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":87,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.659704+0000","last_change":"2024-09-21T09:40:49.397673+0000","last_active":"2024-09-21T09:59:21.659704+0000","last_peered":"2024-09-21T09:59:21.659704+0000","last_clean":"2024-09-21T09:59:21.659704+0000","last_became_active":"2024-09-21T09:40:49.397365+0000","last_became_peered":"2024-09-21T09:40:49.397365+0000","last_unstale":"2024-09-21T09:59:21.659704+0000","last_undegraded":"2024-09-21T09:59:21.659704+0000","last_fullsized":"2024-09-21T09:59:21.659704+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:48:55.191816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":106,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843931+0000","last_change":"2024-09-21T09:50:45.508896+0000","last_active":"2024-09-21T09:59:21.843931+0000","last_peered":"2024-09-21T09:59:21.843931+0000","last_clean":"2024-09-21T09:59:21.843931+0000","last_became_active":"2024-09-21T09:50:45.508504+0000","last_became_peered":"2024-09-21T09:50:45.508504+0000","last_unstale":"2024-09-21T09:59:21.843931+0000","last_undegraded":"2024-09-21T09:59:21.843931+0000","last_fullsized":"2024-09-21T09:59:21.843931+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:51:23.098434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":120,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665055+0000","last_change":"2024-09-21T09:56:12.576294+0000","last_active":"2024-09-21T09:59:21.665055+0000","last_peered":"2024-09-21T09:59:21.665055+0000","last_clean":"2024-09-21T09:59:21.665055+0000","last_became_active":"2024-09-21T09:56:12.575928+0000","last_became_peered":"2024-09-21T09:56:12.575928+0000","last_unstale":"2024-09-21T09:59:21.665055+0000","last_undegraded":"2024-09-21T09:59:21.665055+0000","last_fullsized":"2024-09-21T09:59:21.665055+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:23.227435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":88,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843761+0000","last_change":"2024-09-21T09:40:49.096812+0000","last_active":"2024-09-21T09:59:21.843761+0000","last_peered":"2024-09-21T09:59:21.843761+0000","last_clean":"2024-09-21T09:59:21.843761+0000","last_became_active":"2024-09-21T09:40:49.096549+0000","last_became_peered":"2024-09-21T09:40:49.096549+0000","last_unstale":"2024-09-21T09:59:21.843761+0000","last_undegraded":"2024-09-21T09:59:21.843761+0000","last_fullsized":"2024-09-21T09:59:21.843761+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:24.512887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843208+0000","last_change":"2024-09-21T09:50:45.500221+0000","last_active":"2024-09-21T09:59:21.843208+0000","last_peered":"2024-09-21T09:59:21.843208+0000","last_clean":"2024-09-21T09:59:21.843208+0000","last_became_active":"2024-09-21T09:50:45.499939+0000","last_became_peered":"2024-09-21T09:50:45.499939+0000","last_unstale":"2024-09-21T09:59:21.843208+0000","last_undegraded":"2024-09-21T09:59:21.843208+0000","last_fullsized":"2024-09-21T09:59:21.843208+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:08:12.782227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":105,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660058+0000","last_change":"2024-09-21T09:56:12.590219+0000","last_active":"2024-09-21T09:59:21.660058+0000","last_peered":"2024-09-21T09:59:21.660058+0000","last_clean":"2024-09-21T09:59:21.660058+0000","last_became_active":"2024-09-21T09:56:12.589946+0000","last_became_peered":"2024-09-21T09:56:12.589946+0000","last_unstale":"2024-09-21T09:59:21.660058+0000","last_undegraded":"2024-09-21T09:59:21.660058+0000","last_fullsized":"2024-09-21T09:59:21.660058+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:48.919929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":155,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672048+0000","last_change":"2024-09-21T09:50:45.503336+0000","last_active":"2024-09-21T09:59:22.672048+0000","last_peered":"2024-09-21T09:59:22.672048+0000","last_clean":"2024-09-21T09:59:22.672048+0000","last_became_active":"2024-09-21T09:50:45.503045+0000","last_became_peered":"2024-09-21T09:50:45.503045+0000","last_unstale":"2024-09-21T09:59:22.672048+0000","last_undegraded":"2024-09-21T09:59:22.672048+0000","last_fullsized":"2024-09-21T09:59:22.672048+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:46.728347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":91,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.685711+0000","last_change":"2024-09-21T09:56:12.575897+0000","last_active":"2024-09-21T09:59:22.685711+0000","last_peered":"2024-09-21T09:59:22.685711+0000","last_clean":"2024-09-21T09:59:22.685711+0000","last_became_active":"2024-09-21T09:56:12.575532+0000","last_became_peered":"2024-09-21T09:56:12.575532+0000","last_unstale":"2024-09-21T09:59:22.685711+0000","last_undegraded":"2024-09-21T09:59:22.685711+0000","last_fullsized":"2024-09-21T09:59:22.685711+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:01.302499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665473+0000","last_change":"2024-09-21T09:56:12.588824+0000","last_active":"2024-09-21T09:59:21.665473+0000","last_peered":"2024-09-21T09:59:21.665473+0000","last_clean":"2024-09-21T09:59:21.665473+0000","last_became_active":"2024-09-21T09:56:12.588561+0000","last_became_peered":"2024-09-21T09:56:12.588561+0000","last_unstale":"2024-09-21T09:59:21.665473+0000","last_undegraded":"2024-09-21T09:59:21.665473+0000","last_fullsized":"2024-09-21T09:59:21.665473+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:31:48.900623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843805+0000","last_change":"2024-09-21T09:50:45.501596+0000","last_active":"2024-09-21T09:59:21.843805+0000","last_peered":"2024-09-21T09:59:21.843805+0000","last_clean":"2024-09-21T09:59:21.843805+0000","last_became_active":"2024-09-21T09:50:45.501186+0000","last_became_peered":"2024-09-21T09:50:45.501186+0000","last_unstale":"2024-09-21T09:59:21.843805+0000","last_undegraded":"2024-09-21T09:59:21.843805+0000","last_fullsized":"2024-09-21T09:59:21.843805+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:43:16.449425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":113,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665698+0000","last_change":"2024-09-21T09:51:30.639545+0000","last_active":"2024-09-21T09:59:21.665698+0000","last_peered":"2024-09-21T09:59:21.665698+0000","last_clean":"2024-09-21T09:59:21.665698+0000","last_became_active":"2024-09-21T09:51:30.639106+0000","last_became_peered":"2024-09-21T09:51:30.639106+0000","last_unstale":"2024-09-21T09:59:21.665698+0000","last_undegraded":"2024-09-21T09:59:21.665698+0000","last_fullsized":"2024-09-21T09:59:21.665698+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:19:19.607327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665245+0000","last_change":"2024-09-21T09:51:30.639276+0000","last_active":"2024-09-21T09:59:21.665245+0000","last_peered":"2024-09-21T09:59:21.665245+0000","last_clean":"2024-09-21T09:59:21.665245+0000","last_became_active":"2024-09-21T09:51:30.638273+0000","last_became_peered":"2024-09-21T09:51:30.638273+0000","last_unstale":"2024-09-21T09:59:21.665245+0000","last_undegraded":"2024-09-21T09:59:21.665245+0000","last_fullsized":"2024-09-21T09:59:21.665245+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:19.093920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665822+0000","last_change":"2024-09-21T09:50:45.510764+0000","last_active":"2024-09-21T09:59:21.665822+0000","last_peered":"2024-09-21T09:59:21.665822+0000","last_clean":"2024-09-21T09:59:21.665822+0000","last_became_active":"2024-09-21T09:50:45.510226+0000","last_became_peered":"2024-09-21T09:50:45.510226+0000","last_unstale":"2024-09-21T09:59:21.665822+0000","last_undegraded":"2024-09-21T09:59:21.665822+0000","last_fullsized":"2024-09-21T09:59:21.665822+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:31:13.491427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":104,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.689113+0000","last_change":"2024-09-21T09:56:12.578433+0000","last_active":"2024-09-21T09:59:22.689113+0000","last_peered":"2024-09-21T09:59:22.689113+0000","last_clean":"2024-09-21T09:59:22.689113+0000","last_became_active":"2024-09-21T09:56:12.578012+0000","last_became_peered":"2024-09-21T09:56:12.578012+0000","last_unstale":"2024-09-21T09:59:22.689113+0000","last_undegraded":"2024-09-21T09:59:22.689113+0000","last_fullsized":"2024-09-21T09:59:22.689113+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:07:59.598935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":125,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.768359+0000","last_change":"2024-09-21T09:56:12.575396+0000","last_active":"2024-09-21T09:59:22.768359+0000","last_peered":"2024-09-21T09:59:22.768359+0000","last_clean":"2024-09-21T09:59:22.768359+0000","last_became_active":"2024-09-21T09:56:12.574913+0000","last_became_peered":"2024-09-21T09:56:12.574913+0000","last_unstale":"2024-09-21T09:59:22.768359+0000","last_undegraded":"2024-09-21T09:59:22.768359+0000","last_fullsized":"2024-09-21T09:59:22.768359+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:05.462880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":221,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843628+0000","last_change":"2024-09-21T09:48:54.128018+0000","last_active":"2024-09-21T09:59:21.843628+0000","last_peered":"2024-09-21T09:59:21.843628+0000","last_clean":"2024-09-21T09:59:21.843628+0000","last_became_active":"2024-09-21T09:48:54.127533+0000","last_became_peered":"2024-09-21T09:48:54.127533+0000","last_unstale":"2024-09-21T09:59:21.843628+0000","last_undegraded":"2024-09-21T09:59:21.843628+0000","last_fullsized":"2024-09-21T09:59:21.843628+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:42.410506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":118,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.659885+0000","last_change":"2024-09-21T09:51:30.636796+0000","last_active":"2024-09-21T09:59:21.659885+0000","last_peered":"2024-09-21T09:59:21.659885+0000","last_clean":"2024-09-21T09:59:21.659885+0000","last_became_active":"2024-09-21T09:51:30.636052+0000","last_became_peered":"2024-09-21T09:51:30.636052+0000","last_unstale":"2024-09-21T09:59:21.659885+0000","last_undegraded":"2024-09-21T09:59:21.659885+0000","last_fullsized":"2024-09-21T09:59:21.659885+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:21:36.694847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039581700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":92,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.685870+0000","last_change":"2024-09-21T09:56:12.580057+0000","last_active":"2024-09-21T09:59:22.685870+0000","last_peered":"2024-09-21T09:59:22.685870+0000","last_clean":"2024-09-21T09:59:22.685870+0000","last_became_active":"2024-09-21T09:56:12.579700+0000","last_became_peered":"2024-09-21T09:56:12.579700+0000","last_unstale":"2024-09-21T09:59:22.685870+0000","last_undegraded":"2024-09-21T09:59:22.685870+0000","last_fullsized":"2024-09-21T09:59:22.685870+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:49.768872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":86,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.844142+0000","last_change":"2024-09-21T09:40:50.115273+0000","last_active":"2024-09-21T09:59:21.844142+0000","last_peered":"2024-09-21T09:59:21.844142+0000","last_clean":"2024-09-21T09:59:21.844142+0000","last_became_active":"2024-09-21T09:40:50.114998+0000","last_became_peered":"2024-09-21T09:40:50.114998+0000","last_unstale":"2024-09-21T09:59:21.844142+0000","last_undegraded":"2024-09-21T09:59:21.844142+0000","last_fullsized":"2024-09-21T09:59:21.844142+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:42:39.208914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":133,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843492+0000","last_change":"2024-09-21T09:50:45.501687+0000","last_active":"2024-09-21T09:59:21.843492+0000","last_peered":"2024-09-21T09:59:21.843492+0000","last_clean":"2024-09-21T09:59:21.843492+0000","last_became_active":"2024-09-21T09:50:45.501368+0000","last_became_peered":"2024-09-21T09:50:45.501368+0000","last_unstale":"2024-09-21T09:59:21.843492+0000","last_undegraded":"2024-09-21T09:59:21.843492+0000","last_fullsized":"2024-09-21T09:59:21.843492+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:51.187511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":154,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672126+0000","last_change":"2024-09-21T09:56:12.573698+0000","last_active":"2024-09-21T09:59:22.672126+0000","last_peered":"2024-09-21T09:59:22.672126+0000","last_clean":"2024-09-21T09:59:22.672126+0000","last_became_active":"2024-09-21T09:56:12.573390+0000","last_became_peered":"2024-09-21T09:56:12.573390+0000","last_unstale":"2024-09-21T09:59:22.672126+0000","last_undegraded":"2024-09-21T09:59:22.672126+0000","last_fullsized":"2024-09-21T09:59:22.672126+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:55:12.937878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":166,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.677131+0000","last_change":"2024-09-21T09:51:30.666462+0000","last_active":"2024-09-21T09:59:22.677131+0000","last_peered":"2024-09-21T09:59:22.677131+0000","last_clean":"2024-09-21T09:59:22.677131+0000","last_became_active":"2024-09-21T09:51:30.666006+0000","last_became_peered":"2024-09-21T09:51:30.666006+0000","last_unstale":"2024-09-21T09:59:22.677131+0000","last_undegraded":"2024-09-21T09:59:22.677131+0000","last_fullsized":"2024-09-21T09:59:22.677131+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:51.481698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":156,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.674729+0000","last_change":"2024-09-21T09:56:12.580659+0000","last_active":"2024-09-21T09:59:22.674729+0000","last_peered":"2024-09-21T09:59:22.674729+0000","last_clean":"2024-09-21T09:59:22.674729+0000","last_became_active":"2024-09-21T09:56:12.580300+0000","last_became_peered":"2024-09-21T09:56:12.580300+0000","last_unstale":"2024-09-21T09:59:22.674729+0000","last_undegraded":"2024-09-21T09:59:22.674729+0000","last_fullsized":"2024-09-21T09:59:22.674729+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:04.945404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":124,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.746911+0000","last_change":"2024-09-21T09:51:30.646466+0000","last_active":"2024-09-21T09:59:22.746911+0000","last_peered":"2024-09-21T09:59:22.746911+0000","last_clean":"2024-09-21T09:59:22.746911+0000","last_became_active":"2024-09-21T09:51:30.646209+0000","last_became_peered":"2024-09-21T09:51:30.646209+0000","last_unstale":"2024-09-21T09:59:22.746911+0000","last_undegraded":"2024-09-21T09:59:22.746911+0000","last_fullsized":"2024-09-21T09:59:22.746911+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:44:41.465418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":129,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.675687+0000","last_change":"2024-09-21T09:51:30.649056+0000","last_active":"2024-09-21T09:59:22.675687+0000","last_peered":"2024-09-21T09:59:22.675687+0000","last_clean":"2024-09-21T09:59:22.675687+0000","last_became_active":"2024-09-21T09:51:30.647245+0000","last_became_peered":"2024-09-21T09:51:30.647245+0000","last_unstale":"2024-09-21T09:59:22.675687+0000","last_undegraded":"2024-09-21T09:59:22.675687+0000","last_fullsized":"2024-09-21T09:59:22.675687+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:32.661847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":119,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.659804+0000","last_change":"2024-09-21T09:50:45.502094+0000","last_active":"2024-09-21T09:59:21.659804+0000","last_peered":"2024-09-21T09:59:21.659804+0000","last_clean":"2024-09-21T09:59:21.659804+0000","last_became_active":"2024-09-21T09:50:45.501603+0000","last_became_peered":"2024-09-21T09:50:45.501603+0000","last_unstale":"2024-09-21T09:59:21.659804+0000","last_undegraded":"2024-09-21T09:59:21.659804+0000","last_fullsized":"2024-09-21T09:59:21.659804+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:01.747067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091939799999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.666081+0000","last_change":"2024-09-21T09:48:54.128617+0000","last_active":"2024-09-21T09:59:21.666081+0000","last_peered":"2024-09-21T09:59:21.666081+0000","last_clean":"2024-09-21T09:59:21.666081+0000","last_became_active":"2024-09-21T09:48:54.127878+0000","last_became_peered":"2024-09-21T09:48:54.127878+0000","last_unstale":"2024-09-21T09:59:21.666081+0000","last_undegraded":"2024-09-21T09:59:21.666081+0000","last_fullsized":"2024-09-21T09:59:21.666081+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:07.305638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":136,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.762089+0000","last_change":"2024-09-21T09:56:12.588992+0000","last_active":"2024-09-21T09:59:22.762089+0000","last_peered":"2024-09-21T09:59:22.762089+0000","last_clean":"2024-09-21T09:59:22.762089+0000","last_became_active":"2024-09-21T09:56:12.588665+0000","last_became_peered":"2024-09-21T09:56:12.588665+0000","last_unstale":"2024-09-21T09:59:22.762089+0000","last_undegraded":"2024-09-21T09:59:22.762089+0000","last_fullsized":"2024-09-21T09:59:22.762089+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:19:06.160456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":122,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660179+0000","last_change":"2024-09-21T09:56:12.583280+0000","last_active":"2024-09-21T09:59:21.660179+0000","last_peered":"2024-09-21T09:59:21.660179+0000","last_clean":"2024-09-21T09:59:21.660179+0000","last_became_active":"2024-09-21T09:56:12.583007+0000","last_became_peered":"2024-09-21T09:56:12.583007+0000","last_unstale":"2024-09-21T09:59:21.660179+0000","last_undegraded":"2024-09-21T09:59:21.660179+0000","last_fullsized":"2024-09-21T09:59:21.660179+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:26.991596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":163,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.762196+0000","last_change":"2024-09-21T09:50:45.509978+0000","last_active":"2024-09-21T09:59:22.762196+0000","last_peered":"2024-09-21T09:59:22.762196+0000","last_clean":"2024-09-21T09:59:22.762196+0000","last_became_active":"2024-09-21T09:50:45.509391+0000","last_became_peered":"2024-09-21T09:50:45.509391+0000","last_unstale":"2024-09-21T09:59:22.762196+0000","last_undegraded":"2024-09-21T09:59:22.762196+0000","last_fullsized":"2024-09-21T09:59:22.762196+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:50.360847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":114,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.751096+0000","last_change":"2024-09-21T09:50:45.501349+0000","last_active":"2024-09-21T09:59:22.751096+0000","last_peered":"2024-09-21T09:59:22.751096+0000","last_clean":"2024-09-21T09:59:22.751096+0000","last_became_active":"2024-09-21T09:50:45.501065+0000","last_became_peered":"2024-09-21T09:50:45.501065+0000","last_unstale":"2024-09-21T09:59:22.751096+0000","last_undegraded":"2024-09-21T09:59:22.751096+0000","last_fullsized":"2024-09-21T09:59:22.751096+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:28:58.570586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":124,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.747028+0000","last_change":"2024-09-21T09:51:30.646531+0000","last_active":"2024-09-21T09:59:22.747028+0000","last_peered":"2024-09-21T09:59:22.747028+0000","last_clean":"2024-09-21T09:59:22.747028+0000","last_became_active":"2024-09-21T09:51:30.646248+0000","last_became_peered":"2024-09-21T09:51:30.646248+0000","last_unstale":"2024-09-21T09:59:22.747028+0000","last_undegraded":"2024-09-21T09:59:22.747028+0000","last_fullsized":"2024-09-21T09:59:22.747028+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:00.312924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843891+0000","last_change":"2024-09-21T09:48:54.125569+0000","last_active":"2024-09-21T09:59:21.843891+0000","last_peered":"2024-09-21T09:59:21.843891+0000","last_clean":"2024-09-21T09:59:21.843891+0000","last_became_active":"2024-09-21T09:48:54.125212+0000","last_became_peered":"2024-09-21T09:48:54.125212+0000","last_unstale":"2024-09-21T09:59:21.843891+0000","last_undegraded":"2024-09-21T09:59:21.843891+0000","last_fullsized":"2024-09-21T09:59:21.843891+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:01.434278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":144,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672095+0000","last_change":"2024-09-21T09:48:54.131373+0000","last_active":"2024-09-21T09:59:22.672095+0000","last_peered":"2024-09-21T09:59:22.672095+0000","last_clean":"2024-09-21T09:59:22.672095+0000","last_became_active":"2024-09-21T09:48:54.130391+0000","last_became_peered":"2024-09-21T09:48:54.130391+0000","last_unstale":"2024-09-21T09:59:22.672095+0000","last_undegraded":"2024-09-21T09:59:22.672095+0000","last_fullsized":"2024-09-21T09:59:22.672095+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:46.136344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":91,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.686169+0000","last_change":"2024-09-21T09:56:12.579726+0000","last_active":"2024-09-21T09:59:22.686169+0000","last_peered":"2024-09-21T09:59:22.686169+0000","last_clean":"2024-09-21T09:59:22.686169+0000","last_became_active":"2024-09-21T09:56:12.579304+0000","last_became_peered":"2024-09-21T09:56:12.579304+0000","last_unstale":"2024-09-21T09:59:22.686169+0000","last_undegraded":"2024-09-21T09:59:22.686169+0000","last_fullsized":"2024-09-21T09:59:22.686169+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:44.605640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":124,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.768137+0000","last_change":"2024-09-21T09:50:45.513017+0000","last_active":"2024-09-21T09:59:22.768137+0000","last_peered":"2024-09-21T09:59:22.768137+0000","last_clean":"2024-09-21T09:59:22.768137+0000","last_became_active":"2024-09-21T09:50:45.512629+0000","last_became_peered":"2024-09-21T09:50:45.512629+0000","last_unstale":"2024-09-21T09:59:22.768137+0000","last_undegraded":"2024-09-21T09:59:22.768137+0000","last_fullsized":"2024-09-21T09:59:22.768137+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:47:08.421575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":88,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.688413+0000","last_change":"2024-09-21T09:56:12.578501+0000","last_active":"2024-09-21T09:59:22.688413+0000","last_peered":"2024-09-21T09:59:22.688413+0000","last_clean":"2024-09-21T09:59:22.688413+0000","last_became_active":"2024-09-21T09:56:12.578060+0000","last_became_peered":"2024-09-21T09:56:12.578060+0000","last_unstale":"2024-09-21T09:59:22.688413+0000","last_undegraded":"2024-09-21T09:59:22.688413+0000","last_fullsized":"2024-09-21T09:59:22.688413+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:46:42.441923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":158,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.677411+0000","last_change":"2024-09-21T09:50:45.510812+0000","last_active":"2024-09-21T09:59:22.677411+0000","last_peered":"2024-09-21T09:59:22.677411+0000","last_clean":"2024-09-21T09:59:22.677411+0000","last_became_active":"2024-09-21T09:50:45.510480+0000","last_became_peered":"2024-09-21T09:50:45.510480+0000","last_unstale":"2024-09-21T09:59:22.677411+0000","last_undegraded":"2024-09-21T09:59:22.677411+0000","last_fullsized":"2024-09-21T09:59:22.677411+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:05.408194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.844139+0000","last_change":"2024-09-21T09:51:30.646536+0000","last_active":"2024-09-21T09:59:21.844139+0000","last_peered":"2024-09-21T09:59:21.844139+0000","last_clean":"2024-09-21T09:59:21.844139+0000","last_became_active":"2024-09-21T09:51:30.646204+0000","last_became_peered":"2024-09-21T09:51:30.646204+0000","last_unstale":"2024-09-21T09:59:21.844139+0000","last_undegraded":"2024-09-21T09:59:21.844139+0000","last_fullsized":"2024-09-21T09:59:21.844139+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:20.327494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":86,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665138+0000","last_change":"2024-09-21T09:40:49.190763+0000","last_active":"2024-09-21T09:59:21.665138+0000","last_peered":"2024-09-21T09:59:21.665138+0000","last_clean":"2024-09-21T09:59:21.665138+0000","last_became_active":"2024-09-21T09:40:49.190463+0000","last_became_peered":"2024-09-21T09:40:49.190463+0000","last_unstale":"2024-09-21T09:59:21.665138+0000","last_undegraded":"2024-09-21T09:59:21.665138+0000","last_fullsized":"2024-09-21T09:59:21.665138+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:15.669404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":113,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665996+0000","last_change":"2024-09-21T09:51:30.642074+0000","last_active":"2024-09-21T09:59:21.665996+0000","last_peered":"2024-09-21T09:59:21.665996+0000","last_clean":"2024-09-21T09:59:21.665996+0000","last_became_active":"2024-09-21T09:51:30.641176+0000","last_became_peered":"2024-09-21T09:51:30.641176+0000","last_unstale":"2024-09-21T09:59:21.665996+0000","last_undegraded":"2024-09-21T09:59:21.665996+0000","last_fullsized":"2024-09-21T09:59:21.665996+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:18.766865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":110,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665580+0000","last_change":"2024-09-21T09:40:49.096338+0000","last_active":"2024-09-21T09:59:21.665580+0000","last_peered":"2024-09-21T09:59:21.665580+0000","last_clean":"2024-09-21T09:59:21.665580+0000","last_became_active":"2024-09-21T09:40:49.096078+0000","last_became_peered":"2024-09-21T09:40:49.096078+0000","last_unstale":"2024-09-21T09:59:21.665580+0000","last_undegraded":"2024-09-21T09:59:21.665580+0000","last_fullsized":"2024-09-21T09:59:21.665580+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:56.500555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":125,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.746249+0000","last_change":"2024-09-21T09:51:30.642334+0000","last_active":"2024-09-21T09:59:22.746249+0000","last_peered":"2024-09-21T09:59:22.746249+0000","last_clean":"2024-09-21T09:59:22.746249+0000","last_became_active":"2024-09-21T09:51:30.641267+0000","last_became_peered":"2024-09-21T09:51:30.641267+0000","last_unstale":"2024-09-21T09:59:22.746249+0000","last_undegraded":"2024-09-21T09:59:22.746249+0000","last_fullsized":"2024-09-21T09:59:22.746249+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:24.201971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":107,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843634+0000","last_change":"2024-09-21T09:56:12.580601+0000","last_active":"2024-09-21T09:59:21.843634+0000","last_peered":"2024-09-21T09:59:21.843634+0000","last_clean":"2024-09-21T09:59:21.843634+0000","last_became_active":"2024-09-21T09:56:12.580162+0000","last_became_peered":"2024-09-21T09:56:12.580162+0000","last_unstale":"2024-09-21T09:59:21.843634+0000","last_undegraded":"2024-09-21T09:59:21.843634+0000","last_fullsized":"2024-09-21T09:59:21.843634+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:49:16.381068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":105,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660053+0000","last_change":"2024-09-21T09:50:45.510379+0000","last_active":"2024-09-21T09:59:21.660053+0000","last_peered":"2024-09-21T09:59:21.660053+0000","last_clean":"2024-09-21T09:59:21.660053+0000","last_became_active":"2024-09-21T09:50:45.510082+0000","last_became_peered":"2024-09-21T09:50:45.510082+0000","last_unstale":"2024-09-21T09:59:21.660053+0000","last_undegraded":"2024-09-21T09:59:21.660053+0000","last_fullsized":"2024-09-21T09:59:21.660053+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:50.140708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":93,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.684782+0000","last_change":"2024-09-21T09:56:12.579297+0000","last_active":"2024-09-21T09:59:22.684782+0000","last_peered":"2024-09-21T09:59:22.684782+0000","last_clean":"2024-09-21T09:59:22.684782+0000","last_became_active":"2024-09-21T09:56:12.578321+0000","last_became_peered":"2024-09-21T09:56:12.578321+0000","last_unstale":"2024-09-21T09:59:22.684782+0000","last_undegraded":"2024-09-21T09:59:22.684782+0000","last_fullsized":"2024-09-21T09:59:22.684782+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:27:33.794391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":153,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672638+0000","last_change":"2024-09-21T09:56:12.574472+0000","last_active":"2024-09-21T09:59:22.672638+0000","last_peered":"2024-09-21T09:59:22.672638+0000","last_clean":"2024-09-21T09:59:22.672638+0000","last_became_active":"2024-09-21T09:56:12.573966+0000","last_became_peered":"2024-09-21T09:56:12.573966+0000","last_unstale":"2024-09-21T09:59:22.672638+0000","last_undegraded":"2024-09-21T09:59:22.672638+0000","last_fullsized":"2024-09-21T09:59:22.672638+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:37:21.415933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":103,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.687919+0000","last_change":"2024-09-21T09:56:12.576756+0000","last_active":"2024-09-21T09:59:22.687919+0000","last_peered":"2024-09-21T09:59:22.687919+0000","last_clean":"2024-09-21T09:59:22.687919+0000","last_became_active":"2024-09-21T09:56:12.575752+0000","last_became_peered":"2024-09-21T09:56:12.575752+0000","last_unstale":"2024-09-21T09:59:22.687919+0000","last_undegraded":"2024-09-21T09:59:22.687919+0000","last_fullsized":"2024-09-21T09:59:22.687919+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:55.224839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.844056+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T09:59:21.844056+0000","last_peered":"2024-09-21T09:59:21.844056+0000","last_clean":"2024-09-21T09:59:21.844056+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T09:59:21.844056+0000","last_undegraded":"2024-09-21T09:59:21.844056+0000","last_fullsized":"2024-09-21T09:59:21.844056+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":121,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.688491+0000","last_change":"2024-09-21T09:56:12.571495+0000","last_active":"2024-09-21T09:59:22.688491+0000","last_peered":"2024-09-21T09:59:22.688491+0000","last_clean":"2024-09-21T09:59:22.688491+0000","last_became_active":"2024-09-21T09:56:12.571200+0000","last_became_peered":"2024-09-21T09:56:12.571200+0000","last_unstale":"2024-09-21T09:59:22.688491+0000","last_undegraded":"2024-09-21T09:59:22.688491+0000","last_fullsized":"2024-09-21T09:59:22.688491+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:53:58.851300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":155,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.677181+0000","last_change":"2024-09-21T09:56:12.574888+0000","last_active":"2024-09-21T09:59:22.677181+0000","last_peered":"2024-09-21T09:59:22.677181+0000","last_clean":"2024-09-21T09:59:22.677181+0000","last_became_active":"2024-09-21T09:56:12.574215+0000","last_became_peered":"2024-09-21T09:56:12.574215+0000","last_unstale":"2024-09-21T09:59:22.677181+0000","last_undegraded":"2024-09-21T09:59:22.677181+0000","last_fullsized":"2024-09-21T09:59:22.677181+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:48.599320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":123,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.747399+0000","last_change":"2024-09-21T09:56:12.580276+0000","last_active":"2024-09-21T09:59:22.747399+0000","last_peered":"2024-09-21T09:59:22.747399+0000","last_clean":"2024-09-21T09:59:22.747399+0000","last_became_active":"2024-09-21T09:56:12.579989+0000","last_became_peered":"2024-09-21T09:56:12.579989+0000","last_unstale":"2024-09-21T09:59:22.747399+0000","last_undegraded":"2024-09-21T09:59:22.747399+0000","last_fullsized":"2024-09-21T09:59:22.747399+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:14:56.361873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":103,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.688111+0000","last_change":"2024-09-21T09:56:12.582383+0000","last_active":"2024-09-21T09:59:22.688111+0000","last_peered":"2024-09-21T09:59:22.688111+0000","last_clean":"2024-09-21T09:59:22.688111+0000","last_became_active":"2024-09-21T09:56:12.581427+0000","last_became_peered":"2024-09-21T09:56:12.581427+0000","last_unstale":"2024-09-21T09:59:22.688111+0000","last_undegraded":"2024-09-21T09:59:22.688111+0000","last_fullsized":"2024-09-21T09:59:22.688111+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:24.275633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":122,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665267+0000","last_change":"2024-09-21T09:51:30.645644+0000","last_active":"2024-09-21T09:59:21.665267+0000","last_peered":"2024-09-21T09:59:21.665267+0000","last_clean":"2024-09-21T09:59:21.665267+0000","last_became_active":"2024-09-21T09:51:30.644883+0000","last_became_peered":"2024-09-21T09:51:30.644883+0000","last_unstale":"2024-09-21T09:59:21.665267+0000","last_undegraded":"2024-09-21T09:59:21.665267+0000","last_fullsized":"2024-09-21T09:59:21.665267+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:39.331936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":157,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.675764+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T09:59:22.675764+0000","last_peered":"2024-09-21T09:59:22.675764+0000","last_clean":"2024-09-21T09:59:22.675764+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T09:59:22.675764+0000","last_undegraded":"2024-09-21T09:59:22.675764+0000","last_fullsized":"2024-09-21T09:59:22.675764+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":146,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672236+0000","last_change":"2024-09-21T09:48:54.130327+0000","last_active":"2024-09-21T09:59:22.672236+0000","last_peered":"2024-09-21T09:59:22.672236+0000","last_clean":"2024-09-21T09:59:22.672236+0000","last_became_active":"2024-09-21T09:48:54.128350+0000","last_became_peered":"2024-09-21T09:48:54.128350+0000","last_unstale":"2024-09-21T09:59:22.672236+0000","last_undegraded":"2024-09-21T09:59:22.672236+0000","last_fullsized":"2024-09-21T09:59:22.672236+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:31:40.490861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":120,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665946+0000","last_change":"2024-09-21T09:56:12.576148+0000","last_active":"2024-09-21T09:59:21.665946+0000","last_peered":"2024-09-21T09:59:21.665946+0000","last_clean":"2024-09-21T09:59:21.665946+0000","last_became_active":"2024-09-21T09:56:12.575813+0000","last_became_peered":"2024-09-21T09:56:12.575813+0000","last_unstale":"2024-09-21T09:59:21.665946+0000","last_undegraded":"2024-09-21T09:59:21.665946+0000","last_fullsized":"2024-09-21T09:59:21.665946+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:53:01.367816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":107,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660030+0000","last_change":"2024-09-21T09:50:45.502447+0000","last_active":"2024-09-21T09:59:21.660030+0000","last_peered":"2024-09-21T09:59:21.660030+0000","last_clean":"2024-09-21T09:59:21.660030+0000","last_became_active":"2024-09-21T09:50:45.500800+0000","last_became_peered":"2024-09-21T09:50:45.500800+0000","last_unstale":"2024-09-21T09:59:21.660030+0000","last_undegraded":"2024-09-21T09:59:21.660030+0000","last_fullsized":"2024-09-21T09:59:21.660030+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:20:51.819410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":120,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665155+0000","last_change":"2024-09-21T09:56:12.576457+0000","last_active":"2024-09-21T09:59:21.665155+0000","last_peered":"2024-09-21T09:59:21.665155+0000","last_clean":"2024-09-21T09:59:21.665155+0000","last_became_active":"2024-09-21T09:56:12.576061+0000","last_became_peered":"2024-09-21T09:56:12.576061+0000","last_unstale":"2024-09-21T09:59:21.665155+0000","last_undegraded":"2024-09-21T09:59:21.665155+0000","last_fullsized":"2024-09-21T09:59:21.665155+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:51:46.676699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":90,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660124+0000","last_change":"2024-09-21T09:40:49.094916+0000","last_active":"2024-09-21T09:59:21.660124+0000","last_peered":"2024-09-21T09:59:21.660124+0000","last_clean":"2024-09-21T09:59:21.660124+0000","last_became_active":"2024-09-21T09:40:49.094264+0000","last_became_peered":"2024-09-21T09:40:49.094264+0000","last_unstale":"2024-09-21T09:59:21.660124+0000","last_undegraded":"2024-09-21T09:59:21.660124+0000","last_fullsized":"2024-09-21T09:59:21.660124+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:53.663226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":129,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.685081+0000","last_change":"2024-09-21T09:56:12.577329+0000","last_active":"2024-09-21T09:59:22.685081+0000","last_peered":"2024-09-21T09:59:22.685081+0000","last_clean":"2024-09-21T09:59:22.685081+0000","last_became_active":"2024-09-21T09:56:12.576941+0000","last_became_peered":"2024-09-21T09:56:12.576941+0000","last_unstale":"2024-09-21T09:59:22.685081+0000","last_undegraded":"2024-09-21T09:59:22.685081+0000","last_fullsized":"2024-09-21T09:59:22.685081+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:35.235768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":114,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.688977+0000","last_change":"2024-09-21T09:56:12.577137+0000","last_active":"2024-09-21T09:59:22.688977+0000","last_peered":"2024-09-21T09:59:22.688977+0000","last_clean":"2024-09-21T09:59:22.688977+0000","last_became_active":"2024-09-21T09:56:12.576668+0000","last_became_peered":"2024-09-21T09:56:12.576668+0000","last_unstale":"2024-09-21T09:59:22.688977+0000","last_undegraded":"2024-09-21T09:59:22.688977+0000","last_fullsized":"2024-09-21T09:59:22.688977+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:56.772427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.664974+0000","last_change":"2024-09-21T09:50:45.502819+0000","last_active":"2024-09-21T09:59:21.664974+0000","last_peered":"2024-09-21T09:59:21.664974+0000","last_clean":"2024-09-21T09:59:21.664974+0000","last_became_active":"2024-09-21T09:50:45.502483+0000","last_became_peered":"2024-09-21T09:50:45.502483+0000","last_unstale":"2024-09-21T09:59:21.664974+0000","last_undegraded":"2024-09-21T09:59:21.664974+0000","last_fullsized":"2024-09-21T09:59:21.664974+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:26.549958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":158,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672143+0000","last_change":"2024-09-21T09:51:30.650116+0000","last_active":"2024-09-21T09:59:22.672143+0000","last_peered":"2024-09-21T09:59:22.672143+0000","last_clean":"2024-09-21T09:59:22.672143+0000","last_became_active":"2024-09-21T09:51:30.649798+0000","last_became_peered":"2024-09-21T09:51:30.649798+0000","last_unstale":"2024-09-21T09:59:22.672143+0000","last_undegraded":"2024-09-21T09:59:22.672143+0000","last_fullsized":"2024-09-21T09:59:22.672143+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:40.222590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13748,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626693,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299524,"kb_used_data":2164,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93445628,"statfs":{"total":95995035648,"available":95688323072,"internally_reserved":0,"allocated":2215936,"data_stored":1518744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019151,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":587332,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93157820,"statfs":{"total":95995035648,"available":95393607680,"internally_reserved":0,"allocated":1626112,"data_stored":928433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":81,"seq":347892351113,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":406520,"kb_used_data":2152,"kb_used_omap":5,"kb_used_meta":404346,"kb_avail":93338632,"statfs":{"total":95995035648,"available":95578759168,"internally_reserved":0,"allocated":2203648,"data_stored":1514539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":414050364},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430465,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":596236,"kb_used_data":2172,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93148916,"statfs":{"total":95995035648,"available":95384489984,"internally_reserved":0,"allocated":2224128,"data_stored":1518887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822922,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":910832,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":909246,"kb_avail":92834320,"statfs":{"total":95995035648,"available":95062343680,"internally_reserved":0,"allocated":1605632,"data_stored":923580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931068248},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879345,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":222768,"kb_used_data":1568,"kb_used_omap":4,"kb_used_meta":221179,"kb_avail":93522384,"statfs":{"total":95995035648,"available":95766921216,"internally_reserved":0,"allocated":1605632,"data_stored":923653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":226487740},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566186,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":291252,"kb_used_data":1572,"kb_used_omap":6,"kb_used_meta":289657,"kb_avail":93453900,"statfs":{"total":95995035648,"available":95696793600,"internally_reserved":0,"allocated":1609728,"data_stored":924175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":296609191},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860531,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":329964,"kb_used_data":1564,"kb_used_omap":5,"kb_used_meta":328378,"kb_avail":93415188,"statfs":{"total":95995035648,"available":95657152512,"internally_reserved":0,"allocated":1601536,"data_stored":923267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":336259311},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:00:11.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:00:12.034 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:12.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:00:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:00:11] "GET /metrics HTTP/1.1" 200 34150 "" "Prometheus/2.51.0" 2024-09-21T10:00:12.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:12 smithi107 ceph-mon[102221]: from='client.35874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:00:12.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:00:12.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:00:12.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:00:12.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:12 smithi107 ceph-mon[105022]: from='client.35874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:00:12.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:00:12.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:00:12.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:00:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:12 smithi129 ceph-mon[88471]: from='client.35874 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:00:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:00:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:00:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:00:13.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:13 smithi107 ceph-mon[102221]: pgmap v480: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:13.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:13 smithi107 ceph-mon[105022]: pgmap v480: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:13 smithi129 ceph-mon[88471]: pgmap v480: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:13.929 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:00:13.930 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:00:14.636 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2024-09-21T10:00:12.128755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":213,"num_write_kb":1660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3644428,"kb_used_data":14348,"kb_used_omap":29,"kb_used_meta":3629922,"kb_avail":746316788,"statfs":{"total":767960285184,"available":764228390912,"internally_reserved":0,"allocated":14692352,"data_stored":9175278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30226,"internal_metadata":3717040622},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002684"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":103,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.685385+0000","last_change":"2024-09-21T09:56:12.582314+0000","last_active":"2024-09-21T09:59:22.685385+0000","last_peered":"2024-09-21T09:59:22.685385+0000","last_clean":"2024-09-21T09:59:22.685385+0000","last_became_active":"2024-09-21T09:56:12.581458+0000","last_became_peered":"2024-09-21T09:56:12.581458+0000","last_unstale":"2024-09-21T09:59:22.685385+0000","last_undegraded":"2024-09-21T09:59:22.685385+0000","last_fullsized":"2024-09-21T09:59:22.685385+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:17:21.624171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":106,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843561+0000","last_change":"2024-09-21T09:51:30.640091+0000","last_active":"2024-09-21T09:59:21.843561+0000","last_peered":"2024-09-21T09:59:21.843561+0000","last_clean":"2024-09-21T09:59:21.843561+0000","last_became_active":"2024-09-21T09:51:30.639739+0000","last_became_peered":"2024-09-21T09:51:30.639739+0000","last_unstale":"2024-09-21T09:59:21.843561+0000","last_undegraded":"2024-09-21T09:59:21.843561+0000","last_fullsized":"2024-09-21T09:59:21.843561+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:41.958415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660114+0000","last_change":"2024-09-21T09:56:12.583291+0000","last_active":"2024-09-21T09:59:21.660114+0000","last_peered":"2024-09-21T09:59:21.660114+0000","last_clean":"2024-09-21T09:59:21.660114+0000","last_became_active":"2024-09-21T09:56:12.582908+0000","last_became_peered":"2024-09-21T09:56:12.582908+0000","last_unstale":"2024-09-21T09:59:21.660114+0000","last_undegraded":"2024-09-21T09:59:21.660114+0000","last_fullsized":"2024-09-21T09:59:21.660114+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:04:07.985942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":153,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672057+0000","last_change":"2024-09-21T09:56:12.573112+0000","last_active":"2024-09-21T09:59:22.672057+0000","last_peered":"2024-09-21T09:59:22.672057+0000","last_clean":"2024-09-21T09:59:22.672057+0000","last_became_active":"2024-09-21T09:56:12.572835+0000","last_became_peered":"2024-09-21T09:56:12.572835+0000","last_unstale":"2024-09-21T09:59:22.672057+0000","last_undegraded":"2024-09-21T09:59:22.672057+0000","last_fullsized":"2024-09-21T09:59:22.672057+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:53.493530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":154,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.677271+0000","last_change":"2024-09-21T09:56:12.574922+0000","last_active":"2024-09-21T09:59:22.677271+0000","last_peered":"2024-09-21T09:59:22.677271+0000","last_clean":"2024-09-21T09:59:22.677271+0000","last_became_active":"2024-09-21T09:56:12.574450+0000","last_became_peered":"2024-09-21T09:56:12.574450+0000","last_unstale":"2024-09-21T09:59:22.677271+0000","last_undegraded":"2024-09-21T09:59:22.677271+0000","last_fullsized":"2024-09-21T09:59:22.677271+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:48.408302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":87,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843305+0000","last_change":"2024-09-21T09:40:49.101911+0000","last_active":"2024-09-21T09:59:21.843305+0000","last_peered":"2024-09-21T09:59:21.843305+0000","last_clean":"2024-09-21T09:59:21.843305+0000","last_became_active":"2024-09-21T09:40:49.101629+0000","last_became_peered":"2024-09-21T09:40:49.101629+0000","last_unstale":"2024-09-21T09:59:21.843305+0000","last_undegraded":"2024-09-21T09:59:21.843305+0000","last_fullsized":"2024-09-21T09:59:21.843305+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:07:19.215551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":103,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.686040+0000","last_change":"2024-09-21T09:56:12.601652+0000","last_active":"2024-09-21T09:59:22.686040+0000","last_peered":"2024-09-21T09:59:22.686040+0000","last_clean":"2024-09-21T09:59:22.686040+0000","last_became_active":"2024-09-21T09:56:12.583693+0000","last_became_peered":"2024-09-21T09:56:12.583693+0000","last_unstale":"2024-09-21T09:59:22.686040+0000","last_undegraded":"2024-09-21T09:59:22.686040+0000","last_fullsized":"2024-09-21T09:59:22.686040+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:24.992107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":112,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843253+0000","last_change":"2024-09-21T09:48:54.140948+0000","last_active":"2024-09-21T09:59:21.843253+0000","last_peered":"2024-09-21T09:59:21.843253+0000","last_clean":"2024-09-21T09:59:21.843253+0000","last_became_active":"2024-09-21T09:48:54.140641+0000","last_became_peered":"2024-09-21T09:48:54.140641+0000","last_unstale":"2024-09-21T09:59:21.843253+0000","last_undegraded":"2024-09-21T09:59:21.843253+0000","last_fullsized":"2024-09-21T09:59:21.843253+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:25.157599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":105,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843654+0000","last_change":"2024-09-21T09:48:54.127687+0000","last_active":"2024-09-21T09:59:21.843654+0000","last_peered":"2024-09-21T09:59:21.843654+0000","last_clean":"2024-09-21T09:59:21.843654+0000","last_became_active":"2024-09-21T09:48:54.127325+0000","last_became_peered":"2024-09-21T09:48:54.127325+0000","last_unstale":"2024-09-21T09:59:21.843654+0000","last_undegraded":"2024-09-21T09:59:21.843654+0000","last_fullsized":"2024-09-21T09:59:21.843654+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:30.004794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":87,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.659704+0000","last_change":"2024-09-21T09:40:49.397673+0000","last_active":"2024-09-21T09:59:21.659704+0000","last_peered":"2024-09-21T09:59:21.659704+0000","last_clean":"2024-09-21T09:59:21.659704+0000","last_became_active":"2024-09-21T09:40:49.397365+0000","last_became_peered":"2024-09-21T09:40:49.397365+0000","last_unstale":"2024-09-21T09:59:21.659704+0000","last_undegraded":"2024-09-21T09:59:21.659704+0000","last_fullsized":"2024-09-21T09:59:21.659704+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:48:55.191816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":106,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843931+0000","last_change":"2024-09-21T09:50:45.508896+0000","last_active":"2024-09-21T09:59:21.843931+0000","last_peered":"2024-09-21T09:59:21.843931+0000","last_clean":"2024-09-21T09:59:21.843931+0000","last_became_active":"2024-09-21T09:50:45.508504+0000","last_became_peered":"2024-09-21T09:50:45.508504+0000","last_unstale":"2024-09-21T09:59:21.843931+0000","last_undegraded":"2024-09-21T09:59:21.843931+0000","last_fullsized":"2024-09-21T09:59:21.843931+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:51:23.098434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":120,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665055+0000","last_change":"2024-09-21T09:56:12.576294+0000","last_active":"2024-09-21T09:59:21.665055+0000","last_peered":"2024-09-21T09:59:21.665055+0000","last_clean":"2024-09-21T09:59:21.665055+0000","last_became_active":"2024-09-21T09:56:12.575928+0000","last_became_peered":"2024-09-21T09:56:12.575928+0000","last_unstale":"2024-09-21T09:59:21.665055+0000","last_undegraded":"2024-09-21T09:59:21.665055+0000","last_fullsized":"2024-09-21T09:59:21.665055+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:23.227435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":88,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843761+0000","last_change":"2024-09-21T09:40:49.096812+0000","last_active":"2024-09-21T09:59:21.843761+0000","last_peered":"2024-09-21T09:59:21.843761+0000","last_clean":"2024-09-21T09:59:21.843761+0000","last_became_active":"2024-09-21T09:40:49.096549+0000","last_became_peered":"2024-09-21T09:40:49.096549+0000","last_unstale":"2024-09-21T09:59:21.843761+0000","last_undegraded":"2024-09-21T09:59:21.843761+0000","last_fullsized":"2024-09-21T09:59:21.843761+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:24.512887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843208+0000","last_change":"2024-09-21T09:50:45.500221+0000","last_active":"2024-09-21T09:59:21.843208+0000","last_peered":"2024-09-21T09:59:21.843208+0000","last_clean":"2024-09-21T09:59:21.843208+0000","last_became_active":"2024-09-21T09:50:45.499939+0000","last_became_peered":"2024-09-21T09:50:45.499939+0000","last_unstale":"2024-09-21T09:59:21.843208+0000","last_undegraded":"2024-09-21T09:59:21.843208+0000","last_fullsized":"2024-09-21T09:59:21.843208+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:08:12.782227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":105,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660058+0000","last_change":"2024-09-21T09:56:12.590219+0000","last_active":"2024-09-21T09:59:21.660058+0000","last_peered":"2024-09-21T09:59:21.660058+0000","last_clean":"2024-09-21T09:59:21.660058+0000","last_became_active":"2024-09-21T09:56:12.589946+0000","last_became_peered":"2024-09-21T09:56:12.589946+0000","last_unstale":"2024-09-21T09:59:21.660058+0000","last_undegraded":"2024-09-21T09:59:21.660058+0000","last_fullsized":"2024-09-21T09:59:21.660058+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:48.919929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":155,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672048+0000","last_change":"2024-09-21T09:50:45.503336+0000","last_active":"2024-09-21T09:59:22.672048+0000","last_peered":"2024-09-21T09:59:22.672048+0000","last_clean":"2024-09-21T09:59:22.672048+0000","last_became_active":"2024-09-21T09:50:45.503045+0000","last_became_peered":"2024-09-21T09:50:45.503045+0000","last_unstale":"2024-09-21T09:59:22.672048+0000","last_undegraded":"2024-09-21T09:59:22.672048+0000","last_fullsized":"2024-09-21T09:59:22.672048+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:46.728347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":91,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.685711+0000","last_change":"2024-09-21T09:56:12.575897+0000","last_active":"2024-09-21T09:59:22.685711+0000","last_peered":"2024-09-21T09:59:22.685711+0000","last_clean":"2024-09-21T09:59:22.685711+0000","last_became_active":"2024-09-21T09:56:12.575532+0000","last_became_peered":"2024-09-21T09:56:12.575532+0000","last_unstale":"2024-09-21T09:59:22.685711+0000","last_undegraded":"2024-09-21T09:59:22.685711+0000","last_fullsized":"2024-09-21T09:59:22.685711+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:01.302499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665473+0000","last_change":"2024-09-21T09:56:12.588824+0000","last_active":"2024-09-21T09:59:21.665473+0000","last_peered":"2024-09-21T09:59:21.665473+0000","last_clean":"2024-09-21T09:59:21.665473+0000","last_became_active":"2024-09-21T09:56:12.588561+0000","last_became_peered":"2024-09-21T09:56:12.588561+0000","last_unstale":"2024-09-21T09:59:21.665473+0000","last_undegraded":"2024-09-21T09:59:21.665473+0000","last_fullsized":"2024-09-21T09:59:21.665473+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:31:48.900623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843805+0000","last_change":"2024-09-21T09:50:45.501596+0000","last_active":"2024-09-21T09:59:21.843805+0000","last_peered":"2024-09-21T09:59:21.843805+0000","last_clean":"2024-09-21T09:59:21.843805+0000","last_became_active":"2024-09-21T09:50:45.501186+0000","last_became_peered":"2024-09-21T09:50:45.501186+0000","last_unstale":"2024-09-21T09:59:21.843805+0000","last_undegraded":"2024-09-21T09:59:21.843805+0000","last_fullsized":"2024-09-21T09:59:21.843805+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:43:16.449425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":113,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665698+0000","last_change":"2024-09-21T09:51:30.639545+0000","last_active":"2024-09-21T09:59:21.665698+0000","last_peered":"2024-09-21T09:59:21.665698+0000","last_clean":"2024-09-21T09:59:21.665698+0000","last_became_active":"2024-09-21T09:51:30.639106+0000","last_became_peered":"2024-09-21T09:51:30.639106+0000","last_unstale":"2024-09-21T09:59:21.665698+0000","last_undegraded":"2024-09-21T09:59:21.665698+0000","last_fullsized":"2024-09-21T09:59:21.665698+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:19:19.607327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665245+0000","last_change":"2024-09-21T09:51:30.639276+0000","last_active":"2024-09-21T09:59:21.665245+0000","last_peered":"2024-09-21T09:59:21.665245+0000","last_clean":"2024-09-21T09:59:21.665245+0000","last_became_active":"2024-09-21T09:51:30.638273+0000","last_became_peered":"2024-09-21T09:51:30.638273+0000","last_unstale":"2024-09-21T09:59:21.665245+0000","last_undegraded":"2024-09-21T09:59:21.665245+0000","last_fullsized":"2024-09-21T09:59:21.665245+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:19.093920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665822+0000","last_change":"2024-09-21T09:50:45.510764+0000","last_active":"2024-09-21T09:59:21.665822+0000","last_peered":"2024-09-21T09:59:21.665822+0000","last_clean":"2024-09-21T09:59:21.665822+0000","last_became_active":"2024-09-21T09:50:45.510226+0000","last_became_peered":"2024-09-21T09:50:45.510226+0000","last_unstale":"2024-09-21T09:59:21.665822+0000","last_undegraded":"2024-09-21T09:59:21.665822+0000","last_fullsized":"2024-09-21T09:59:21.665822+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:31:13.491427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":104,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.689113+0000","last_change":"2024-09-21T09:56:12.578433+0000","last_active":"2024-09-21T09:59:22.689113+0000","last_peered":"2024-09-21T09:59:22.689113+0000","last_clean":"2024-09-21T09:59:22.689113+0000","last_became_active":"2024-09-21T09:56:12.578012+0000","last_became_peered":"2024-09-21T09:56:12.578012+0000","last_unstale":"2024-09-21T09:59:22.689113+0000","last_undegraded":"2024-09-21T09:59:22.689113+0000","last_fullsized":"2024-09-21T09:59:22.689113+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:07:59.598935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":125,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.768359+0000","last_change":"2024-09-21T09:56:12.575396+0000","last_active":"2024-09-21T09:59:22.768359+0000","last_peered":"2024-09-21T09:59:22.768359+0000","last_clean":"2024-09-21T09:59:22.768359+0000","last_became_active":"2024-09-21T09:56:12.574913+0000","last_became_peered":"2024-09-21T09:56:12.574913+0000","last_unstale":"2024-09-21T09:59:22.768359+0000","last_undegraded":"2024-09-21T09:59:22.768359+0000","last_fullsized":"2024-09-21T09:59:22.768359+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:05.462880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":221,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843628+0000","last_change":"2024-09-21T09:48:54.128018+0000","last_active":"2024-09-21T09:59:21.843628+0000","last_peered":"2024-09-21T09:59:21.843628+0000","last_clean":"2024-09-21T09:59:21.843628+0000","last_became_active":"2024-09-21T09:48:54.127533+0000","last_became_peered":"2024-09-21T09:48:54.127533+0000","last_unstale":"2024-09-21T09:59:21.843628+0000","last_undegraded":"2024-09-21T09:59:21.843628+0000","last_fullsized":"2024-09-21T09:59:21.843628+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:42.410506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":118,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.659885+0000","last_change":"2024-09-21T09:51:30.636796+0000","last_active":"2024-09-21T09:59:21.659885+0000","last_peered":"2024-09-21T09:59:21.659885+0000","last_clean":"2024-09-21T09:59:21.659885+0000","last_became_active":"2024-09-21T09:51:30.636052+0000","last_became_peered":"2024-09-21T09:51:30.636052+0000","last_unstale":"2024-09-21T09:59:21.659885+0000","last_undegraded":"2024-09-21T09:59:21.659885+0000","last_fullsized":"2024-09-21T09:59:21.659885+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:21:36.694847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039581700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":92,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.685870+0000","last_change":"2024-09-21T09:56:12.580057+0000","last_active":"2024-09-21T09:59:22.685870+0000","last_peered":"2024-09-21T09:59:22.685870+0000","last_clean":"2024-09-21T09:59:22.685870+0000","last_became_active":"2024-09-21T09:56:12.579700+0000","last_became_peered":"2024-09-21T09:56:12.579700+0000","last_unstale":"2024-09-21T09:59:22.685870+0000","last_undegraded":"2024-09-21T09:59:22.685870+0000","last_fullsized":"2024-09-21T09:59:22.685870+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:49.768872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":86,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.844142+0000","last_change":"2024-09-21T09:40:50.115273+0000","last_active":"2024-09-21T09:59:21.844142+0000","last_peered":"2024-09-21T09:59:21.844142+0000","last_clean":"2024-09-21T09:59:21.844142+0000","last_became_active":"2024-09-21T09:40:50.114998+0000","last_became_peered":"2024-09-21T09:40:50.114998+0000","last_unstale":"2024-09-21T09:59:21.844142+0000","last_undegraded":"2024-09-21T09:59:21.844142+0000","last_fullsized":"2024-09-21T09:59:21.844142+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:42:39.208914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":133,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843492+0000","last_change":"2024-09-21T09:50:45.501687+0000","last_active":"2024-09-21T09:59:21.843492+0000","last_peered":"2024-09-21T09:59:21.843492+0000","last_clean":"2024-09-21T09:59:21.843492+0000","last_became_active":"2024-09-21T09:50:45.501368+0000","last_became_peered":"2024-09-21T09:50:45.501368+0000","last_unstale":"2024-09-21T09:59:21.843492+0000","last_undegraded":"2024-09-21T09:59:21.843492+0000","last_fullsized":"2024-09-21T09:59:21.843492+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:51.187511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":154,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672126+0000","last_change":"2024-09-21T09:56:12.573698+0000","last_active":"2024-09-21T09:59:22.672126+0000","last_peered":"2024-09-21T09:59:22.672126+0000","last_clean":"2024-09-21T09:59:22.672126+0000","last_became_active":"2024-09-21T09:56:12.573390+0000","last_became_peered":"2024-09-21T09:56:12.573390+0000","last_unstale":"2024-09-21T09:59:22.672126+0000","last_undegraded":"2024-09-21T09:59:22.672126+0000","last_fullsized":"2024-09-21T09:59:22.672126+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:55:12.937878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":166,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.677131+0000","last_change":"2024-09-21T09:51:30.666462+0000","last_active":"2024-09-21T09:59:22.677131+0000","last_peered":"2024-09-21T09:59:22.677131+0000","last_clean":"2024-09-21T09:59:22.677131+0000","last_became_active":"2024-09-21T09:51:30.666006+0000","last_became_peered":"2024-09-21T09:51:30.666006+0000","last_unstale":"2024-09-21T09:59:22.677131+0000","last_undegraded":"2024-09-21T09:59:22.677131+0000","last_fullsized":"2024-09-21T09:59:22.677131+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:51.481698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":156,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.674729+0000","last_change":"2024-09-21T09:56:12.580659+0000","last_active":"2024-09-21T09:59:22.674729+0000","last_peered":"2024-09-21T09:59:22.674729+0000","last_clean":"2024-09-21T09:59:22.674729+0000","last_became_active":"2024-09-21T09:56:12.580300+0000","last_became_peered":"2024-09-21T09:56:12.580300+0000","last_unstale":"2024-09-21T09:59:22.674729+0000","last_undegraded":"2024-09-21T09:59:22.674729+0000","last_fullsized":"2024-09-21T09:59:22.674729+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:04.945404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":124,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.746911+0000","last_change":"2024-09-21T09:51:30.646466+0000","last_active":"2024-09-21T09:59:22.746911+0000","last_peered":"2024-09-21T09:59:22.746911+0000","last_clean":"2024-09-21T09:59:22.746911+0000","last_became_active":"2024-09-21T09:51:30.646209+0000","last_became_peered":"2024-09-21T09:51:30.646209+0000","last_unstale":"2024-09-21T09:59:22.746911+0000","last_undegraded":"2024-09-21T09:59:22.746911+0000","last_fullsized":"2024-09-21T09:59:22.746911+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:44:41.465418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":129,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.675687+0000","last_change":"2024-09-21T09:51:30.649056+0000","last_active":"2024-09-21T09:59:22.675687+0000","last_peered":"2024-09-21T09:59:22.675687+0000","last_clean":"2024-09-21T09:59:22.675687+0000","last_became_active":"2024-09-21T09:51:30.647245+0000","last_became_peered":"2024-09-21T09:51:30.647245+0000","last_unstale":"2024-09-21T09:59:22.675687+0000","last_undegraded":"2024-09-21T09:59:22.675687+0000","last_fullsized":"2024-09-21T09:59:22.675687+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:32.661847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":119,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.659804+0000","last_change":"2024-09-21T09:50:45.502094+0000","last_active":"2024-09-21T09:59:21.659804+0000","last_peered":"2024-09-21T09:59:21.659804+0000","last_clean":"2024-09-21T09:59:21.659804+0000","last_became_active":"2024-09-21T09:50:45.501603+0000","last_became_peered":"2024-09-21T09:50:45.501603+0000","last_unstale":"2024-09-21T09:59:21.659804+0000","last_undegraded":"2024-09-21T09:59:21.659804+0000","last_fullsized":"2024-09-21T09:59:21.659804+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:01.747067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091939799999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.666081+0000","last_change":"2024-09-21T09:48:54.128617+0000","last_active":"2024-09-21T09:59:21.666081+0000","last_peered":"2024-09-21T09:59:21.666081+0000","last_clean":"2024-09-21T09:59:21.666081+0000","last_became_active":"2024-09-21T09:48:54.127878+0000","last_became_peered":"2024-09-21T09:48:54.127878+0000","last_unstale":"2024-09-21T09:59:21.666081+0000","last_undegraded":"2024-09-21T09:59:21.666081+0000","last_fullsized":"2024-09-21T09:59:21.666081+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:07.305638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":136,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.762089+0000","last_change":"2024-09-21T09:56:12.588992+0000","last_active":"2024-09-21T09:59:22.762089+0000","last_peered":"2024-09-21T09:59:22.762089+0000","last_clean":"2024-09-21T09:59:22.762089+0000","last_became_active":"2024-09-21T09:56:12.588665+0000","last_became_peered":"2024-09-21T09:56:12.588665+0000","last_unstale":"2024-09-21T09:59:22.762089+0000","last_undegraded":"2024-09-21T09:59:22.762089+0000","last_fullsized":"2024-09-21T09:59:22.762089+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:19:06.160456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":122,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660179+0000","last_change":"2024-09-21T09:56:12.583280+0000","last_active":"2024-09-21T09:59:21.660179+0000","last_peered":"2024-09-21T09:59:21.660179+0000","last_clean":"2024-09-21T09:59:21.660179+0000","last_became_active":"2024-09-21T09:56:12.583007+0000","last_became_peered":"2024-09-21T09:56:12.583007+0000","last_unstale":"2024-09-21T09:59:21.660179+0000","last_undegraded":"2024-09-21T09:59:21.660179+0000","last_fullsized":"2024-09-21T09:59:21.660179+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:26.991596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":163,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.762196+0000","last_change":"2024-09-21T09:50:45.509978+0000","last_active":"2024-09-21T09:59:22.762196+0000","last_peered":"2024-09-21T09:59:22.762196+0000","last_clean":"2024-09-21T09:59:22.762196+0000","last_became_active":"2024-09-21T09:50:45.509391+0000","last_became_peered":"2024-09-21T09:50:45.509391+0000","last_unstale":"2024-09-21T09:59:22.762196+0000","last_undegraded":"2024-09-21T09:59:22.762196+0000","last_fullsized":"2024-09-21T09:59:22.762196+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:50.360847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":114,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.751096+0000","last_change":"2024-09-21T09:50:45.501349+0000","last_active":"2024-09-21T09:59:22.751096+0000","last_peered":"2024-09-21T09:59:22.751096+0000","last_clean":"2024-09-21T09:59:22.751096+0000","last_became_active":"2024-09-21T09:50:45.501065+0000","last_became_peered":"2024-09-21T09:50:45.501065+0000","last_unstale":"2024-09-21T09:59:22.751096+0000","last_undegraded":"2024-09-21T09:59:22.751096+0000","last_fullsized":"2024-09-21T09:59:22.751096+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:28:58.570586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":124,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.747028+0000","last_change":"2024-09-21T09:51:30.646531+0000","last_active":"2024-09-21T09:59:22.747028+0000","last_peered":"2024-09-21T09:59:22.747028+0000","last_clean":"2024-09-21T09:59:22.747028+0000","last_became_active":"2024-09-21T09:51:30.646248+0000","last_became_peered":"2024-09-21T09:51:30.646248+0000","last_unstale":"2024-09-21T09:59:22.747028+0000","last_undegraded":"2024-09-21T09:59:22.747028+0000","last_fullsized":"2024-09-21T09:59:22.747028+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:00.312924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843891+0000","last_change":"2024-09-21T09:48:54.125569+0000","last_active":"2024-09-21T09:59:21.843891+0000","last_peered":"2024-09-21T09:59:21.843891+0000","last_clean":"2024-09-21T09:59:21.843891+0000","last_became_active":"2024-09-21T09:48:54.125212+0000","last_became_peered":"2024-09-21T09:48:54.125212+0000","last_unstale":"2024-09-21T09:59:21.843891+0000","last_undegraded":"2024-09-21T09:59:21.843891+0000","last_fullsized":"2024-09-21T09:59:21.843891+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:01.434278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":144,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672095+0000","last_change":"2024-09-21T09:48:54.131373+0000","last_active":"2024-09-21T09:59:22.672095+0000","last_peered":"2024-09-21T09:59:22.672095+0000","last_clean":"2024-09-21T09:59:22.672095+0000","last_became_active":"2024-09-21T09:48:54.130391+0000","last_became_peered":"2024-09-21T09:48:54.130391+0000","last_unstale":"2024-09-21T09:59:22.672095+0000","last_undegraded":"2024-09-21T09:59:22.672095+0000","last_fullsized":"2024-09-21T09:59:22.672095+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:46.136344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":91,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.686169+0000","last_change":"2024-09-21T09:56:12.579726+0000","last_active":"2024-09-21T09:59:22.686169+0000","last_peered":"2024-09-21T09:59:22.686169+0000","last_clean":"2024-09-21T09:59:22.686169+0000","last_became_active":"2024-09-21T09:56:12.579304+0000","last_became_peered":"2024-09-21T09:56:12.579304+0000","last_unstale":"2024-09-21T09:59:22.686169+0000","last_undegraded":"2024-09-21T09:59:22.686169+0000","last_fullsized":"2024-09-21T09:59:22.686169+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:44.605640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":124,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.768137+0000","last_change":"2024-09-21T09:50:45.513017+0000","last_active":"2024-09-21T09:59:22.768137+0000","last_peered":"2024-09-21T09:59:22.768137+0000","last_clean":"2024-09-21T09:59:22.768137+0000","last_became_active":"2024-09-21T09:50:45.512629+0000","last_became_peered":"2024-09-21T09:50:45.512629+0000","last_unstale":"2024-09-21T09:59:22.768137+0000","last_undegraded":"2024-09-21T09:59:22.768137+0000","last_fullsized":"2024-09-21T09:59:22.768137+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:47:08.421575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":89,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.688449+0000","last_change":"2024-09-21T09:56:12.578501+0000","last_active":"2024-09-21T09:59:22.688449+0000","last_peered":"2024-09-21T09:59:22.688449+0000","last_clean":"2024-09-21T09:59:22.688449+0000","last_became_active":"2024-09-21T09:56:12.578060+0000","last_became_peered":"2024-09-21T09:56:12.578060+0000","last_unstale":"2024-09-21T09:59:22.688449+0000","last_undegraded":"2024-09-21T09:59:22.688449+0000","last_fullsized":"2024-09-21T09:59:22.688449+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:46:42.441923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":158,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.677411+0000","last_change":"2024-09-21T09:50:45.510812+0000","last_active":"2024-09-21T09:59:22.677411+0000","last_peered":"2024-09-21T09:59:22.677411+0000","last_clean":"2024-09-21T09:59:22.677411+0000","last_became_active":"2024-09-21T09:50:45.510480+0000","last_became_peered":"2024-09-21T09:50:45.510480+0000","last_unstale":"2024-09-21T09:59:22.677411+0000","last_undegraded":"2024-09-21T09:59:22.677411+0000","last_fullsized":"2024-09-21T09:59:22.677411+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:05.408194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.844139+0000","last_change":"2024-09-21T09:51:30.646536+0000","last_active":"2024-09-21T09:59:21.844139+0000","last_peered":"2024-09-21T09:59:21.844139+0000","last_clean":"2024-09-21T09:59:21.844139+0000","last_became_active":"2024-09-21T09:51:30.646204+0000","last_became_peered":"2024-09-21T09:51:30.646204+0000","last_unstale":"2024-09-21T09:59:21.844139+0000","last_undegraded":"2024-09-21T09:59:21.844139+0000","last_fullsized":"2024-09-21T09:59:21.844139+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:20.327494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":86,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665138+0000","last_change":"2024-09-21T09:40:49.190763+0000","last_active":"2024-09-21T09:59:21.665138+0000","last_peered":"2024-09-21T09:59:21.665138+0000","last_clean":"2024-09-21T09:59:21.665138+0000","last_became_active":"2024-09-21T09:40:49.190463+0000","last_became_peered":"2024-09-21T09:40:49.190463+0000","last_unstale":"2024-09-21T09:59:21.665138+0000","last_undegraded":"2024-09-21T09:59:21.665138+0000","last_fullsized":"2024-09-21T09:59:21.665138+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:15.669404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":113,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665996+0000","last_change":"2024-09-21T09:51:30.642074+0000","last_active":"2024-09-21T09:59:21.665996+0000","last_peered":"2024-09-21T09:59:21.665996+0000","last_clean":"2024-09-21T09:59:21.665996+0000","last_became_active":"2024-09-21T09:51:30.641176+0000","last_became_peered":"2024-09-21T09:51:30.641176+0000","last_unstale":"2024-09-21T09:59:21.665996+0000","last_undegraded":"2024-09-21T09:59:21.665996+0000","last_fullsized":"2024-09-21T09:59:21.665996+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:18.766865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":110,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665580+0000","last_change":"2024-09-21T09:40:49.096338+0000","last_active":"2024-09-21T09:59:21.665580+0000","last_peered":"2024-09-21T09:59:21.665580+0000","last_clean":"2024-09-21T09:59:21.665580+0000","last_became_active":"2024-09-21T09:40:49.096078+0000","last_became_peered":"2024-09-21T09:40:49.096078+0000","last_unstale":"2024-09-21T09:59:21.665580+0000","last_undegraded":"2024-09-21T09:59:21.665580+0000","last_fullsized":"2024-09-21T09:59:21.665580+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:56.500555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":125,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.746249+0000","last_change":"2024-09-21T09:51:30.642334+0000","last_active":"2024-09-21T09:59:22.746249+0000","last_peered":"2024-09-21T09:59:22.746249+0000","last_clean":"2024-09-21T09:59:22.746249+0000","last_became_active":"2024-09-21T09:51:30.641267+0000","last_became_peered":"2024-09-21T09:51:30.641267+0000","last_unstale":"2024-09-21T09:59:22.746249+0000","last_undegraded":"2024-09-21T09:59:22.746249+0000","last_fullsized":"2024-09-21T09:59:22.746249+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:24.201971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":107,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843634+0000","last_change":"2024-09-21T09:56:12.580601+0000","last_active":"2024-09-21T09:59:21.843634+0000","last_peered":"2024-09-21T09:59:21.843634+0000","last_clean":"2024-09-21T09:59:21.843634+0000","last_became_active":"2024-09-21T09:56:12.580162+0000","last_became_peered":"2024-09-21T09:56:12.580162+0000","last_unstale":"2024-09-21T09:59:21.843634+0000","last_undegraded":"2024-09-21T09:59:21.843634+0000","last_fullsized":"2024-09-21T09:59:21.843634+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:49:16.381068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":105,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660053+0000","last_change":"2024-09-21T09:50:45.510379+0000","last_active":"2024-09-21T09:59:21.660053+0000","last_peered":"2024-09-21T09:59:21.660053+0000","last_clean":"2024-09-21T09:59:21.660053+0000","last_became_active":"2024-09-21T09:50:45.510082+0000","last_became_peered":"2024-09-21T09:50:45.510082+0000","last_unstale":"2024-09-21T09:59:21.660053+0000","last_undegraded":"2024-09-21T09:59:21.660053+0000","last_fullsized":"2024-09-21T09:59:21.660053+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:50.140708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":93,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.684782+0000","last_change":"2024-09-21T09:56:12.579297+0000","last_active":"2024-09-21T09:59:22.684782+0000","last_peered":"2024-09-21T09:59:22.684782+0000","last_clean":"2024-09-21T09:59:22.684782+0000","last_became_active":"2024-09-21T09:56:12.578321+0000","last_became_peered":"2024-09-21T09:56:12.578321+0000","last_unstale":"2024-09-21T09:59:22.684782+0000","last_undegraded":"2024-09-21T09:59:22.684782+0000","last_fullsized":"2024-09-21T09:59:22.684782+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:27:33.794391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":153,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672638+0000","last_change":"2024-09-21T09:56:12.574472+0000","last_active":"2024-09-21T09:59:22.672638+0000","last_peered":"2024-09-21T09:59:22.672638+0000","last_clean":"2024-09-21T09:59:22.672638+0000","last_became_active":"2024-09-21T09:56:12.573966+0000","last_became_peered":"2024-09-21T09:56:12.573966+0000","last_unstale":"2024-09-21T09:59:22.672638+0000","last_undegraded":"2024-09-21T09:59:22.672638+0000","last_fullsized":"2024-09-21T09:59:22.672638+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:37:21.415933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":103,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.687919+0000","last_change":"2024-09-21T09:56:12.576756+0000","last_active":"2024-09-21T09:59:22.687919+0000","last_peered":"2024-09-21T09:59:22.687919+0000","last_clean":"2024-09-21T09:59:22.687919+0000","last_became_active":"2024-09-21T09:56:12.575752+0000","last_became_peered":"2024-09-21T09:56:12.575752+0000","last_unstale":"2024-09-21T09:59:22.687919+0000","last_undegraded":"2024-09-21T09:59:22.687919+0000","last_fullsized":"2024-09-21T09:59:22.687919+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:55.224839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.844056+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T09:59:21.844056+0000","last_peered":"2024-09-21T09:59:21.844056+0000","last_clean":"2024-09-21T09:59:21.844056+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T09:59:21.844056+0000","last_undegraded":"2024-09-21T09:59:21.844056+0000","last_fullsized":"2024-09-21T09:59:21.844056+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":121,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.688491+0000","last_change":"2024-09-21T09:56:12.571495+0000","last_active":"2024-09-21T09:59:22.688491+0000","last_peered":"2024-09-21T09:59:22.688491+0000","last_clean":"2024-09-21T09:59:22.688491+0000","last_became_active":"2024-09-21T09:56:12.571200+0000","last_became_peered":"2024-09-21T09:56:12.571200+0000","last_unstale":"2024-09-21T09:59:22.688491+0000","last_undegraded":"2024-09-21T09:59:22.688491+0000","last_fullsized":"2024-09-21T09:59:22.688491+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:53:58.851300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":155,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.677181+0000","last_change":"2024-09-21T09:56:12.574888+0000","last_active":"2024-09-21T09:59:22.677181+0000","last_peered":"2024-09-21T09:59:22.677181+0000","last_clean":"2024-09-21T09:59:22.677181+0000","last_became_active":"2024-09-21T09:56:12.574215+0000","last_became_peered":"2024-09-21T09:56:12.574215+0000","last_unstale":"2024-09-21T09:59:22.677181+0000","last_undegraded":"2024-09-21T09:59:22.677181+0000","last_fullsized":"2024-09-21T09:59:22.677181+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:48.599320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":123,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.747399+0000","last_change":"2024-09-21T09:56:12.580276+0000","last_active":"2024-09-21T09:59:22.747399+0000","last_peered":"2024-09-21T09:59:22.747399+0000","last_clean":"2024-09-21T09:59:22.747399+0000","last_became_active":"2024-09-21T09:56:12.579989+0000","last_became_peered":"2024-09-21T09:56:12.579989+0000","last_unstale":"2024-09-21T09:59:22.747399+0000","last_undegraded":"2024-09-21T09:59:22.747399+0000","last_fullsized":"2024-09-21T09:59:22.747399+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:14:56.361873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":103,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.688111+0000","last_change":"2024-09-21T09:56:12.582383+0000","last_active":"2024-09-21T09:59:22.688111+0000","last_peered":"2024-09-21T09:59:22.688111+0000","last_clean":"2024-09-21T09:59:22.688111+0000","last_became_active":"2024-09-21T09:56:12.581427+0000","last_became_peered":"2024-09-21T09:56:12.581427+0000","last_unstale":"2024-09-21T09:59:22.688111+0000","last_undegraded":"2024-09-21T09:59:22.688111+0000","last_fullsized":"2024-09-21T09:59:22.688111+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:24.275633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":122,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665267+0000","last_change":"2024-09-21T09:51:30.645644+0000","last_active":"2024-09-21T09:59:21.665267+0000","last_peered":"2024-09-21T09:59:21.665267+0000","last_clean":"2024-09-21T09:59:21.665267+0000","last_became_active":"2024-09-21T09:51:30.644883+0000","last_became_peered":"2024-09-21T09:51:30.644883+0000","last_unstale":"2024-09-21T09:59:21.665267+0000","last_undegraded":"2024-09-21T09:59:21.665267+0000","last_fullsized":"2024-09-21T09:59:21.665267+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:39.331936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":157,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.675764+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T09:59:22.675764+0000","last_peered":"2024-09-21T09:59:22.675764+0000","last_clean":"2024-09-21T09:59:22.675764+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T09:59:22.675764+0000","last_undegraded":"2024-09-21T09:59:22.675764+0000","last_fullsized":"2024-09-21T09:59:22.675764+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":146,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672236+0000","last_change":"2024-09-21T09:48:54.130327+0000","last_active":"2024-09-21T09:59:22.672236+0000","last_peered":"2024-09-21T09:59:22.672236+0000","last_clean":"2024-09-21T09:59:22.672236+0000","last_became_active":"2024-09-21T09:48:54.128350+0000","last_became_peered":"2024-09-21T09:48:54.128350+0000","last_unstale":"2024-09-21T09:59:22.672236+0000","last_undegraded":"2024-09-21T09:59:22.672236+0000","last_fullsized":"2024-09-21T09:59:22.672236+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:31:40.490861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":120,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665946+0000","last_change":"2024-09-21T09:56:12.576148+0000","last_active":"2024-09-21T09:59:21.665946+0000","last_peered":"2024-09-21T09:59:21.665946+0000","last_clean":"2024-09-21T09:59:21.665946+0000","last_became_active":"2024-09-21T09:56:12.575813+0000","last_became_peered":"2024-09-21T09:56:12.575813+0000","last_unstale":"2024-09-21T09:59:21.665946+0000","last_undegraded":"2024-09-21T09:59:21.665946+0000","last_fullsized":"2024-09-21T09:59:21.665946+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:53:01.367816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":107,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660030+0000","last_change":"2024-09-21T09:50:45.502447+0000","last_active":"2024-09-21T09:59:21.660030+0000","last_peered":"2024-09-21T09:59:21.660030+0000","last_clean":"2024-09-21T09:59:21.660030+0000","last_became_active":"2024-09-21T09:50:45.500800+0000","last_became_peered":"2024-09-21T09:50:45.500800+0000","last_unstale":"2024-09-21T09:59:21.660030+0000","last_undegraded":"2024-09-21T09:59:21.660030+0000","last_fullsized":"2024-09-21T09:59:21.660030+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:20:51.819410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":120,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665155+0000","last_change":"2024-09-21T09:56:12.576457+0000","last_active":"2024-09-21T09:59:21.665155+0000","last_peered":"2024-09-21T09:59:21.665155+0000","last_clean":"2024-09-21T09:59:21.665155+0000","last_became_active":"2024-09-21T09:56:12.576061+0000","last_became_peered":"2024-09-21T09:56:12.576061+0000","last_unstale":"2024-09-21T09:59:21.665155+0000","last_undegraded":"2024-09-21T09:59:21.665155+0000","last_fullsized":"2024-09-21T09:59:21.665155+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:51:46.676699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":90,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660124+0000","last_change":"2024-09-21T09:40:49.094916+0000","last_active":"2024-09-21T09:59:21.660124+0000","last_peered":"2024-09-21T09:59:21.660124+0000","last_clean":"2024-09-21T09:59:21.660124+0000","last_became_active":"2024-09-21T09:40:49.094264+0000","last_became_peered":"2024-09-21T09:40:49.094264+0000","last_unstale":"2024-09-21T09:59:21.660124+0000","last_undegraded":"2024-09-21T09:59:21.660124+0000","last_fullsized":"2024-09-21T09:59:21.660124+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:53.663226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":129,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.685081+0000","last_change":"2024-09-21T09:56:12.577329+0000","last_active":"2024-09-21T09:59:22.685081+0000","last_peered":"2024-09-21T09:59:22.685081+0000","last_clean":"2024-09-21T09:59:22.685081+0000","last_became_active":"2024-09-21T09:56:12.576941+0000","last_became_peered":"2024-09-21T09:56:12.576941+0000","last_unstale":"2024-09-21T09:59:22.685081+0000","last_undegraded":"2024-09-21T09:59:22.685081+0000","last_fullsized":"2024-09-21T09:59:22.685081+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:35.235768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":115,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.689026+0000","last_change":"2024-09-21T09:56:12.577137+0000","last_active":"2024-09-21T09:59:22.689026+0000","last_peered":"2024-09-21T09:59:22.689026+0000","last_clean":"2024-09-21T09:59:22.689026+0000","last_became_active":"2024-09-21T09:56:12.576668+0000","last_became_peered":"2024-09-21T09:56:12.576668+0000","last_unstale":"2024-09-21T09:59:22.689026+0000","last_undegraded":"2024-09-21T09:59:22.689026+0000","last_fullsized":"2024-09-21T09:59:22.689026+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:56.772427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.664974+0000","last_change":"2024-09-21T09:50:45.502819+0000","last_active":"2024-09-21T09:59:21.664974+0000","last_peered":"2024-09-21T09:59:21.664974+0000","last_clean":"2024-09-21T09:59:21.664974+0000","last_became_active":"2024-09-21T09:50:45.502483+0000","last_became_peered":"2024-09-21T09:50:45.502483+0000","last_unstale":"2024-09-21T09:59:21.664974+0000","last_undegraded":"2024-09-21T09:59:21.664974+0000","last_fullsized":"2024-09-21T09:59:21.664974+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:26.549958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":158,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672143+0000","last_change":"2024-09-21T09:51:30.650116+0000","last_active":"2024-09-21T09:59:22.672143+0000","last_peered":"2024-09-21T09:59:22.672143+0000","last_clean":"2024-09-21T09:59:22.672143+0000","last_became_active":"2024-09-21T09:51:30.649798+0000","last_became_peered":"2024-09-21T09:51:30.649798+0000","last_unstale":"2024-09-21T09:59:22.672143+0000","last_undegraded":"2024-09-21T09:59:22.672143+0000","last_fullsized":"2024-09-21T09:59:22.672143+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:40.222590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13748,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626693,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299524,"kb_used_data":2164,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93445628,"statfs":{"total":95995035648,"available":95688323072,"internally_reserved":0,"allocated":2215936,"data_stored":1518744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019151,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":587332,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93157820,"statfs":{"total":95995035648,"available":95393607680,"internally_reserved":0,"allocated":1626112,"data_stored":928433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":81,"seq":347892351113,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":406520,"kb_used_data":2152,"kb_used_omap":5,"kb_used_meta":404346,"kb_avail":93338632,"statfs":{"total":95995035648,"available":95578759168,"internally_reserved":0,"allocated":2203648,"data_stored":1514539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":414050364},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430465,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":596236,"kb_used_data":2172,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93148916,"statfs":{"total":95995035648,"available":95384489984,"internally_reserved":0,"allocated":2224128,"data_stored":1518887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822922,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":910832,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":909246,"kb_avail":92834320,"statfs":{"total":95995035648,"available":95062343680,"internally_reserved":0,"allocated":1605632,"data_stored":923580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931068248},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879346,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":222768,"kb_used_data":1568,"kb_used_omap":4,"kb_used_meta":221179,"kb_avail":93522384,"statfs":{"total":95995035648,"available":95766921216,"internally_reserved":0,"allocated":1605632,"data_stored":923653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":226487740},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566186,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":291252,"kb_used_data":1572,"kb_used_omap":6,"kb_used_meta":289657,"kb_avail":93453900,"statfs":{"total":95995035648,"available":95696793600,"internally_reserved":0,"allocated":1609728,"data_stored":924175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":296609191},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860531,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":329964,"kb_used_data":1564,"kb_used_omap":5,"kb_used_meta":328378,"kb_avail":93415188,"statfs":{"total":95995035648,"available":95657152512,"internally_reserved":0,"allocated":1601536,"data_stored":923267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":336259311},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:00:14.639 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T10:00:14.639 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T10:00:14.639 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T10:00:14.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph health --format=json 2024-09-21T10:00:14.890 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:15.696 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:15 smithi107 ceph-mon[102221]: from='client.45680 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:00:15.696 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:15 smithi107 ceph-mon[102221]: pgmap v481: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:15.696 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:15 smithi107 ceph-mon[105022]: from='client.45680 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:00:15.696 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:15 smithi107 ceph-mon[105022]: pgmap v481: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:15 smithi129 ceph-mon[88471]: from='client.45680 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:00:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:15 smithi129 ceph-mon[88471]: pgmap v481: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:16.737 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:00:16.737 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T10:00:17.457 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T10:00:17.457 INFO:teuthology.run_tasks:Running task print... 2024-09-21T10:00:17.473 INFO:teuthology.task.print:**** applying stress + thrashing to mixed-version cluster **** 2024-09-21T10:00:17.473 INFO:teuthology.run_tasks:Running task parallel... 2024-09-21T10:00:17.491 INFO:teuthology.task.parallel:starting parallel... 2024-09-21T10:00:17.492 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-21T10:00:17.492 INFO:teuthology.task.sequential:In sequential, running task thrashosds... 2024-09-21T10:00:17.493 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T10:00:17.494 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'OSD_DOWN', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSDMAP_FLAGS', 'PG_AVAILABILITY'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'main', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}} 2024-09-21T10:00:17.494 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T10:00:17.494 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-21T10:00:17.494 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T10:00:17.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:00:17.743 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:17.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:17 smithi107 ceph-mon[102221]: pgmap v482: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:17.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:17 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/484170622' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T10:00:17.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:17 smithi107 ceph-mon[105022]: pgmap v482: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:17.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:17 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/484170622' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T10:00:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:17 smithi129 ceph-mon[88471]: pgmap v482: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:17 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/484170622' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T10:00:19.597 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:00:19.597 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:00:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:19 smithi107 ceph-mon[102221]: pgmap v483: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:19.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:19 smithi107 ceph-mon[105022]: pgmap v483: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:19 smithi129 ceph-mon[88471]: pgmap v483: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:20.365 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-21T10:00:18.130435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":213,"num_write_kb":1660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3644428,"kb_used_data":14348,"kb_used_omap":29,"kb_used_meta":3629922,"kb_avail":746316788,"statfs":{"total":767960285184,"available":764228390912,"internally_reserved":0,"allocated":14692352,"data_stored":9175278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30226,"internal_metadata":3717040622},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002977"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":103,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.685385+0000","last_change":"2024-09-21T09:56:12.582314+0000","last_active":"2024-09-21T09:59:22.685385+0000","last_peered":"2024-09-21T09:59:22.685385+0000","last_clean":"2024-09-21T09:59:22.685385+0000","last_became_active":"2024-09-21T09:56:12.581458+0000","last_became_peered":"2024-09-21T09:56:12.581458+0000","last_unstale":"2024-09-21T09:59:22.685385+0000","last_undegraded":"2024-09-21T09:59:22.685385+0000","last_fullsized":"2024-09-21T09:59:22.685385+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:17:21.624171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":106,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843561+0000","last_change":"2024-09-21T09:51:30.640091+0000","last_active":"2024-09-21T09:59:21.843561+0000","last_peered":"2024-09-21T09:59:21.843561+0000","last_clean":"2024-09-21T09:59:21.843561+0000","last_became_active":"2024-09-21T09:51:30.639739+0000","last_became_peered":"2024-09-21T09:51:30.639739+0000","last_unstale":"2024-09-21T09:59:21.843561+0000","last_undegraded":"2024-09-21T09:59:21.843561+0000","last_fullsized":"2024-09-21T09:59:21.843561+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:41.958415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660114+0000","last_change":"2024-09-21T09:56:12.583291+0000","last_active":"2024-09-21T09:59:21.660114+0000","last_peered":"2024-09-21T09:59:21.660114+0000","last_clean":"2024-09-21T09:59:21.660114+0000","last_became_active":"2024-09-21T09:56:12.582908+0000","last_became_peered":"2024-09-21T09:56:12.582908+0000","last_unstale":"2024-09-21T09:59:21.660114+0000","last_undegraded":"2024-09-21T09:59:21.660114+0000","last_fullsized":"2024-09-21T09:59:21.660114+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:04:07.985942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":153,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672057+0000","last_change":"2024-09-21T09:56:12.573112+0000","last_active":"2024-09-21T09:59:22.672057+0000","last_peered":"2024-09-21T09:59:22.672057+0000","last_clean":"2024-09-21T09:59:22.672057+0000","last_became_active":"2024-09-21T09:56:12.572835+0000","last_became_peered":"2024-09-21T09:56:12.572835+0000","last_unstale":"2024-09-21T09:59:22.672057+0000","last_undegraded":"2024-09-21T09:59:22.672057+0000","last_fullsized":"2024-09-21T09:59:22.672057+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:53.493530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":154,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.677271+0000","last_change":"2024-09-21T09:56:12.574922+0000","last_active":"2024-09-21T09:59:22.677271+0000","last_peered":"2024-09-21T09:59:22.677271+0000","last_clean":"2024-09-21T09:59:22.677271+0000","last_became_active":"2024-09-21T09:56:12.574450+0000","last_became_peered":"2024-09-21T09:56:12.574450+0000","last_unstale":"2024-09-21T09:59:22.677271+0000","last_undegraded":"2024-09-21T09:59:22.677271+0000","last_fullsized":"2024-09-21T09:59:22.677271+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:48.408302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":87,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843305+0000","last_change":"2024-09-21T09:40:49.101911+0000","last_active":"2024-09-21T09:59:21.843305+0000","last_peered":"2024-09-21T09:59:21.843305+0000","last_clean":"2024-09-21T09:59:21.843305+0000","last_became_active":"2024-09-21T09:40:49.101629+0000","last_became_peered":"2024-09-21T09:40:49.101629+0000","last_unstale":"2024-09-21T09:59:21.843305+0000","last_undegraded":"2024-09-21T09:59:21.843305+0000","last_fullsized":"2024-09-21T09:59:21.843305+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:07:19.215551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":103,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.686040+0000","last_change":"2024-09-21T09:56:12.601652+0000","last_active":"2024-09-21T09:59:22.686040+0000","last_peered":"2024-09-21T09:59:22.686040+0000","last_clean":"2024-09-21T09:59:22.686040+0000","last_became_active":"2024-09-21T09:56:12.583693+0000","last_became_peered":"2024-09-21T09:56:12.583693+0000","last_unstale":"2024-09-21T09:59:22.686040+0000","last_undegraded":"2024-09-21T09:59:22.686040+0000","last_fullsized":"2024-09-21T09:59:22.686040+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:24.992107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":112,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843253+0000","last_change":"2024-09-21T09:48:54.140948+0000","last_active":"2024-09-21T09:59:21.843253+0000","last_peered":"2024-09-21T09:59:21.843253+0000","last_clean":"2024-09-21T09:59:21.843253+0000","last_became_active":"2024-09-21T09:48:54.140641+0000","last_became_peered":"2024-09-21T09:48:54.140641+0000","last_unstale":"2024-09-21T09:59:21.843253+0000","last_undegraded":"2024-09-21T09:59:21.843253+0000","last_fullsized":"2024-09-21T09:59:21.843253+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:25.157599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":105,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843654+0000","last_change":"2024-09-21T09:48:54.127687+0000","last_active":"2024-09-21T09:59:21.843654+0000","last_peered":"2024-09-21T09:59:21.843654+0000","last_clean":"2024-09-21T09:59:21.843654+0000","last_became_active":"2024-09-21T09:48:54.127325+0000","last_became_peered":"2024-09-21T09:48:54.127325+0000","last_unstale":"2024-09-21T09:59:21.843654+0000","last_undegraded":"2024-09-21T09:59:21.843654+0000","last_fullsized":"2024-09-21T09:59:21.843654+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:30.004794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":87,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.659704+0000","last_change":"2024-09-21T09:40:49.397673+0000","last_active":"2024-09-21T09:59:21.659704+0000","last_peered":"2024-09-21T09:59:21.659704+0000","last_clean":"2024-09-21T09:59:21.659704+0000","last_became_active":"2024-09-21T09:40:49.397365+0000","last_became_peered":"2024-09-21T09:40:49.397365+0000","last_unstale":"2024-09-21T09:59:21.659704+0000","last_undegraded":"2024-09-21T09:59:21.659704+0000","last_fullsized":"2024-09-21T09:59:21.659704+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:48:55.191816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":106,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843931+0000","last_change":"2024-09-21T09:50:45.508896+0000","last_active":"2024-09-21T09:59:21.843931+0000","last_peered":"2024-09-21T09:59:21.843931+0000","last_clean":"2024-09-21T09:59:21.843931+0000","last_became_active":"2024-09-21T09:50:45.508504+0000","last_became_peered":"2024-09-21T09:50:45.508504+0000","last_unstale":"2024-09-21T09:59:21.843931+0000","last_undegraded":"2024-09-21T09:59:21.843931+0000","last_fullsized":"2024-09-21T09:59:21.843931+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:51:23.098434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":120,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665055+0000","last_change":"2024-09-21T09:56:12.576294+0000","last_active":"2024-09-21T09:59:21.665055+0000","last_peered":"2024-09-21T09:59:21.665055+0000","last_clean":"2024-09-21T09:59:21.665055+0000","last_became_active":"2024-09-21T09:56:12.575928+0000","last_became_peered":"2024-09-21T09:56:12.575928+0000","last_unstale":"2024-09-21T09:59:21.665055+0000","last_undegraded":"2024-09-21T09:59:21.665055+0000","last_fullsized":"2024-09-21T09:59:21.665055+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:23.227435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":88,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843761+0000","last_change":"2024-09-21T09:40:49.096812+0000","last_active":"2024-09-21T09:59:21.843761+0000","last_peered":"2024-09-21T09:59:21.843761+0000","last_clean":"2024-09-21T09:59:21.843761+0000","last_became_active":"2024-09-21T09:40:49.096549+0000","last_became_peered":"2024-09-21T09:40:49.096549+0000","last_unstale":"2024-09-21T09:59:21.843761+0000","last_undegraded":"2024-09-21T09:59:21.843761+0000","last_fullsized":"2024-09-21T09:59:21.843761+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:24.512887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843208+0000","last_change":"2024-09-21T09:50:45.500221+0000","last_active":"2024-09-21T09:59:21.843208+0000","last_peered":"2024-09-21T09:59:21.843208+0000","last_clean":"2024-09-21T09:59:21.843208+0000","last_became_active":"2024-09-21T09:50:45.499939+0000","last_became_peered":"2024-09-21T09:50:45.499939+0000","last_unstale":"2024-09-21T09:59:21.843208+0000","last_undegraded":"2024-09-21T09:59:21.843208+0000","last_fullsized":"2024-09-21T09:59:21.843208+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:08:12.782227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":105,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660058+0000","last_change":"2024-09-21T09:56:12.590219+0000","last_active":"2024-09-21T09:59:21.660058+0000","last_peered":"2024-09-21T09:59:21.660058+0000","last_clean":"2024-09-21T09:59:21.660058+0000","last_became_active":"2024-09-21T09:56:12.589946+0000","last_became_peered":"2024-09-21T09:56:12.589946+0000","last_unstale":"2024-09-21T09:59:21.660058+0000","last_undegraded":"2024-09-21T09:59:21.660058+0000","last_fullsized":"2024-09-21T09:59:21.660058+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:48.919929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":155,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672048+0000","last_change":"2024-09-21T09:50:45.503336+0000","last_active":"2024-09-21T09:59:22.672048+0000","last_peered":"2024-09-21T09:59:22.672048+0000","last_clean":"2024-09-21T09:59:22.672048+0000","last_became_active":"2024-09-21T09:50:45.503045+0000","last_became_peered":"2024-09-21T09:50:45.503045+0000","last_unstale":"2024-09-21T09:59:22.672048+0000","last_undegraded":"2024-09-21T09:59:22.672048+0000","last_fullsized":"2024-09-21T09:59:22.672048+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:46.728347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":91,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.685711+0000","last_change":"2024-09-21T09:56:12.575897+0000","last_active":"2024-09-21T09:59:22.685711+0000","last_peered":"2024-09-21T09:59:22.685711+0000","last_clean":"2024-09-21T09:59:22.685711+0000","last_became_active":"2024-09-21T09:56:12.575532+0000","last_became_peered":"2024-09-21T09:56:12.575532+0000","last_unstale":"2024-09-21T09:59:22.685711+0000","last_undegraded":"2024-09-21T09:59:22.685711+0000","last_fullsized":"2024-09-21T09:59:22.685711+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:01.302499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665473+0000","last_change":"2024-09-21T09:56:12.588824+0000","last_active":"2024-09-21T09:59:21.665473+0000","last_peered":"2024-09-21T09:59:21.665473+0000","last_clean":"2024-09-21T09:59:21.665473+0000","last_became_active":"2024-09-21T09:56:12.588561+0000","last_became_peered":"2024-09-21T09:56:12.588561+0000","last_unstale":"2024-09-21T09:59:21.665473+0000","last_undegraded":"2024-09-21T09:59:21.665473+0000","last_fullsized":"2024-09-21T09:59:21.665473+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:31:48.900623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843805+0000","last_change":"2024-09-21T09:50:45.501596+0000","last_active":"2024-09-21T09:59:21.843805+0000","last_peered":"2024-09-21T09:59:21.843805+0000","last_clean":"2024-09-21T09:59:21.843805+0000","last_became_active":"2024-09-21T09:50:45.501186+0000","last_became_peered":"2024-09-21T09:50:45.501186+0000","last_unstale":"2024-09-21T09:59:21.843805+0000","last_undegraded":"2024-09-21T09:59:21.843805+0000","last_fullsized":"2024-09-21T09:59:21.843805+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:43:16.449425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":113,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665698+0000","last_change":"2024-09-21T09:51:30.639545+0000","last_active":"2024-09-21T09:59:21.665698+0000","last_peered":"2024-09-21T09:59:21.665698+0000","last_clean":"2024-09-21T09:59:21.665698+0000","last_became_active":"2024-09-21T09:51:30.639106+0000","last_became_peered":"2024-09-21T09:51:30.639106+0000","last_unstale":"2024-09-21T09:59:21.665698+0000","last_undegraded":"2024-09-21T09:59:21.665698+0000","last_fullsized":"2024-09-21T09:59:21.665698+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:19:19.607327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665245+0000","last_change":"2024-09-21T09:51:30.639276+0000","last_active":"2024-09-21T09:59:21.665245+0000","last_peered":"2024-09-21T09:59:21.665245+0000","last_clean":"2024-09-21T09:59:21.665245+0000","last_became_active":"2024-09-21T09:51:30.638273+0000","last_became_peered":"2024-09-21T09:51:30.638273+0000","last_unstale":"2024-09-21T09:59:21.665245+0000","last_undegraded":"2024-09-21T09:59:21.665245+0000","last_fullsized":"2024-09-21T09:59:21.665245+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:19.093920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665822+0000","last_change":"2024-09-21T09:50:45.510764+0000","last_active":"2024-09-21T09:59:21.665822+0000","last_peered":"2024-09-21T09:59:21.665822+0000","last_clean":"2024-09-21T09:59:21.665822+0000","last_became_active":"2024-09-21T09:50:45.510226+0000","last_became_peered":"2024-09-21T09:50:45.510226+0000","last_unstale":"2024-09-21T09:59:21.665822+0000","last_undegraded":"2024-09-21T09:59:21.665822+0000","last_fullsized":"2024-09-21T09:59:21.665822+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:31:13.491427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":104,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.689113+0000","last_change":"2024-09-21T09:56:12.578433+0000","last_active":"2024-09-21T09:59:22.689113+0000","last_peered":"2024-09-21T09:59:22.689113+0000","last_clean":"2024-09-21T09:59:22.689113+0000","last_became_active":"2024-09-21T09:56:12.578012+0000","last_became_peered":"2024-09-21T09:56:12.578012+0000","last_unstale":"2024-09-21T09:59:22.689113+0000","last_undegraded":"2024-09-21T09:59:22.689113+0000","last_fullsized":"2024-09-21T09:59:22.689113+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:07:59.598935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":125,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.768359+0000","last_change":"2024-09-21T09:56:12.575396+0000","last_active":"2024-09-21T09:59:22.768359+0000","last_peered":"2024-09-21T09:59:22.768359+0000","last_clean":"2024-09-21T09:59:22.768359+0000","last_became_active":"2024-09-21T09:56:12.574913+0000","last_became_peered":"2024-09-21T09:56:12.574913+0000","last_unstale":"2024-09-21T09:59:22.768359+0000","last_undegraded":"2024-09-21T09:59:22.768359+0000","last_fullsized":"2024-09-21T09:59:22.768359+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:05.462880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":221,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843628+0000","last_change":"2024-09-21T09:48:54.128018+0000","last_active":"2024-09-21T09:59:21.843628+0000","last_peered":"2024-09-21T09:59:21.843628+0000","last_clean":"2024-09-21T09:59:21.843628+0000","last_became_active":"2024-09-21T09:48:54.127533+0000","last_became_peered":"2024-09-21T09:48:54.127533+0000","last_unstale":"2024-09-21T09:59:21.843628+0000","last_undegraded":"2024-09-21T09:59:21.843628+0000","last_fullsized":"2024-09-21T09:59:21.843628+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:42.410506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":118,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.659885+0000","last_change":"2024-09-21T09:51:30.636796+0000","last_active":"2024-09-21T09:59:21.659885+0000","last_peered":"2024-09-21T09:59:21.659885+0000","last_clean":"2024-09-21T09:59:21.659885+0000","last_became_active":"2024-09-21T09:51:30.636052+0000","last_became_peered":"2024-09-21T09:51:30.636052+0000","last_unstale":"2024-09-21T09:59:21.659885+0000","last_undegraded":"2024-09-21T09:59:21.659885+0000","last_fullsized":"2024-09-21T09:59:21.659885+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:21:36.694847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039581700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":92,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.685870+0000","last_change":"2024-09-21T09:56:12.580057+0000","last_active":"2024-09-21T09:59:22.685870+0000","last_peered":"2024-09-21T09:59:22.685870+0000","last_clean":"2024-09-21T09:59:22.685870+0000","last_became_active":"2024-09-21T09:56:12.579700+0000","last_became_peered":"2024-09-21T09:56:12.579700+0000","last_unstale":"2024-09-21T09:59:22.685870+0000","last_undegraded":"2024-09-21T09:59:22.685870+0000","last_fullsized":"2024-09-21T09:59:22.685870+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:49.768872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":86,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.844142+0000","last_change":"2024-09-21T09:40:50.115273+0000","last_active":"2024-09-21T09:59:21.844142+0000","last_peered":"2024-09-21T09:59:21.844142+0000","last_clean":"2024-09-21T09:59:21.844142+0000","last_became_active":"2024-09-21T09:40:50.114998+0000","last_became_peered":"2024-09-21T09:40:50.114998+0000","last_unstale":"2024-09-21T09:59:21.844142+0000","last_undegraded":"2024-09-21T09:59:21.844142+0000","last_fullsized":"2024-09-21T09:59:21.844142+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:42:39.208914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":133,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843492+0000","last_change":"2024-09-21T09:50:45.501687+0000","last_active":"2024-09-21T09:59:21.843492+0000","last_peered":"2024-09-21T09:59:21.843492+0000","last_clean":"2024-09-21T09:59:21.843492+0000","last_became_active":"2024-09-21T09:50:45.501368+0000","last_became_peered":"2024-09-21T09:50:45.501368+0000","last_unstale":"2024-09-21T09:59:21.843492+0000","last_undegraded":"2024-09-21T09:59:21.843492+0000","last_fullsized":"2024-09-21T09:59:21.843492+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:51.187511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":154,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672126+0000","last_change":"2024-09-21T09:56:12.573698+0000","last_active":"2024-09-21T09:59:22.672126+0000","last_peered":"2024-09-21T09:59:22.672126+0000","last_clean":"2024-09-21T09:59:22.672126+0000","last_became_active":"2024-09-21T09:56:12.573390+0000","last_became_peered":"2024-09-21T09:56:12.573390+0000","last_unstale":"2024-09-21T09:59:22.672126+0000","last_undegraded":"2024-09-21T09:59:22.672126+0000","last_fullsized":"2024-09-21T09:59:22.672126+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:55:12.937878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":166,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.677131+0000","last_change":"2024-09-21T09:51:30.666462+0000","last_active":"2024-09-21T09:59:22.677131+0000","last_peered":"2024-09-21T09:59:22.677131+0000","last_clean":"2024-09-21T09:59:22.677131+0000","last_became_active":"2024-09-21T09:51:30.666006+0000","last_became_peered":"2024-09-21T09:51:30.666006+0000","last_unstale":"2024-09-21T09:59:22.677131+0000","last_undegraded":"2024-09-21T09:59:22.677131+0000","last_fullsized":"2024-09-21T09:59:22.677131+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:51.481698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":156,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.674729+0000","last_change":"2024-09-21T09:56:12.580659+0000","last_active":"2024-09-21T09:59:22.674729+0000","last_peered":"2024-09-21T09:59:22.674729+0000","last_clean":"2024-09-21T09:59:22.674729+0000","last_became_active":"2024-09-21T09:56:12.580300+0000","last_became_peered":"2024-09-21T09:56:12.580300+0000","last_unstale":"2024-09-21T09:59:22.674729+0000","last_undegraded":"2024-09-21T09:59:22.674729+0000","last_fullsized":"2024-09-21T09:59:22.674729+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:04.945404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":124,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.746911+0000","last_change":"2024-09-21T09:51:30.646466+0000","last_active":"2024-09-21T09:59:22.746911+0000","last_peered":"2024-09-21T09:59:22.746911+0000","last_clean":"2024-09-21T09:59:22.746911+0000","last_became_active":"2024-09-21T09:51:30.646209+0000","last_became_peered":"2024-09-21T09:51:30.646209+0000","last_unstale":"2024-09-21T09:59:22.746911+0000","last_undegraded":"2024-09-21T09:59:22.746911+0000","last_fullsized":"2024-09-21T09:59:22.746911+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:44:41.465418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":129,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.675687+0000","last_change":"2024-09-21T09:51:30.649056+0000","last_active":"2024-09-21T09:59:22.675687+0000","last_peered":"2024-09-21T09:59:22.675687+0000","last_clean":"2024-09-21T09:59:22.675687+0000","last_became_active":"2024-09-21T09:51:30.647245+0000","last_became_peered":"2024-09-21T09:51:30.647245+0000","last_unstale":"2024-09-21T09:59:22.675687+0000","last_undegraded":"2024-09-21T09:59:22.675687+0000","last_fullsized":"2024-09-21T09:59:22.675687+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:32.661847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":119,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.659804+0000","last_change":"2024-09-21T09:50:45.502094+0000","last_active":"2024-09-21T09:59:21.659804+0000","last_peered":"2024-09-21T09:59:21.659804+0000","last_clean":"2024-09-21T09:59:21.659804+0000","last_became_active":"2024-09-21T09:50:45.501603+0000","last_became_peered":"2024-09-21T09:50:45.501603+0000","last_unstale":"2024-09-21T09:59:21.659804+0000","last_undegraded":"2024-09-21T09:59:21.659804+0000","last_fullsized":"2024-09-21T09:59:21.659804+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:01.747067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091939799999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.666081+0000","last_change":"2024-09-21T09:48:54.128617+0000","last_active":"2024-09-21T09:59:21.666081+0000","last_peered":"2024-09-21T09:59:21.666081+0000","last_clean":"2024-09-21T09:59:21.666081+0000","last_became_active":"2024-09-21T09:48:54.127878+0000","last_became_peered":"2024-09-21T09:48:54.127878+0000","last_unstale":"2024-09-21T09:59:21.666081+0000","last_undegraded":"2024-09-21T09:59:21.666081+0000","last_fullsized":"2024-09-21T09:59:21.666081+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:07.305638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":136,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.762089+0000","last_change":"2024-09-21T09:56:12.588992+0000","last_active":"2024-09-21T09:59:22.762089+0000","last_peered":"2024-09-21T09:59:22.762089+0000","last_clean":"2024-09-21T09:59:22.762089+0000","last_became_active":"2024-09-21T09:56:12.588665+0000","last_became_peered":"2024-09-21T09:56:12.588665+0000","last_unstale":"2024-09-21T09:59:22.762089+0000","last_undegraded":"2024-09-21T09:59:22.762089+0000","last_fullsized":"2024-09-21T09:59:22.762089+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:19:06.160456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":122,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660179+0000","last_change":"2024-09-21T09:56:12.583280+0000","last_active":"2024-09-21T09:59:21.660179+0000","last_peered":"2024-09-21T09:59:21.660179+0000","last_clean":"2024-09-21T09:59:21.660179+0000","last_became_active":"2024-09-21T09:56:12.583007+0000","last_became_peered":"2024-09-21T09:56:12.583007+0000","last_unstale":"2024-09-21T09:59:21.660179+0000","last_undegraded":"2024-09-21T09:59:21.660179+0000","last_fullsized":"2024-09-21T09:59:21.660179+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:26.991596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":163,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.762196+0000","last_change":"2024-09-21T09:50:45.509978+0000","last_active":"2024-09-21T09:59:22.762196+0000","last_peered":"2024-09-21T09:59:22.762196+0000","last_clean":"2024-09-21T09:59:22.762196+0000","last_became_active":"2024-09-21T09:50:45.509391+0000","last_became_peered":"2024-09-21T09:50:45.509391+0000","last_unstale":"2024-09-21T09:59:22.762196+0000","last_undegraded":"2024-09-21T09:59:22.762196+0000","last_fullsized":"2024-09-21T09:59:22.762196+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:50.360847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":114,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.751096+0000","last_change":"2024-09-21T09:50:45.501349+0000","last_active":"2024-09-21T09:59:22.751096+0000","last_peered":"2024-09-21T09:59:22.751096+0000","last_clean":"2024-09-21T09:59:22.751096+0000","last_became_active":"2024-09-21T09:50:45.501065+0000","last_became_peered":"2024-09-21T09:50:45.501065+0000","last_unstale":"2024-09-21T09:59:22.751096+0000","last_undegraded":"2024-09-21T09:59:22.751096+0000","last_fullsized":"2024-09-21T09:59:22.751096+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:28:58.570586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":124,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.747028+0000","last_change":"2024-09-21T09:51:30.646531+0000","last_active":"2024-09-21T09:59:22.747028+0000","last_peered":"2024-09-21T09:59:22.747028+0000","last_clean":"2024-09-21T09:59:22.747028+0000","last_became_active":"2024-09-21T09:51:30.646248+0000","last_became_peered":"2024-09-21T09:51:30.646248+0000","last_unstale":"2024-09-21T09:59:22.747028+0000","last_undegraded":"2024-09-21T09:59:22.747028+0000","last_fullsized":"2024-09-21T09:59:22.747028+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:00.312924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843891+0000","last_change":"2024-09-21T09:48:54.125569+0000","last_active":"2024-09-21T09:59:21.843891+0000","last_peered":"2024-09-21T09:59:21.843891+0000","last_clean":"2024-09-21T09:59:21.843891+0000","last_became_active":"2024-09-21T09:48:54.125212+0000","last_became_peered":"2024-09-21T09:48:54.125212+0000","last_unstale":"2024-09-21T09:59:21.843891+0000","last_undegraded":"2024-09-21T09:59:21.843891+0000","last_fullsized":"2024-09-21T09:59:21.843891+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:01.434278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":144,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672095+0000","last_change":"2024-09-21T09:48:54.131373+0000","last_active":"2024-09-21T09:59:22.672095+0000","last_peered":"2024-09-21T09:59:22.672095+0000","last_clean":"2024-09-21T09:59:22.672095+0000","last_became_active":"2024-09-21T09:48:54.130391+0000","last_became_peered":"2024-09-21T09:48:54.130391+0000","last_unstale":"2024-09-21T09:59:22.672095+0000","last_undegraded":"2024-09-21T09:59:22.672095+0000","last_fullsized":"2024-09-21T09:59:22.672095+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:46.136344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":91,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.686169+0000","last_change":"2024-09-21T09:56:12.579726+0000","last_active":"2024-09-21T09:59:22.686169+0000","last_peered":"2024-09-21T09:59:22.686169+0000","last_clean":"2024-09-21T09:59:22.686169+0000","last_became_active":"2024-09-21T09:56:12.579304+0000","last_became_peered":"2024-09-21T09:56:12.579304+0000","last_unstale":"2024-09-21T09:59:22.686169+0000","last_undegraded":"2024-09-21T09:59:22.686169+0000","last_fullsized":"2024-09-21T09:59:22.686169+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:44.605640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":124,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.768137+0000","last_change":"2024-09-21T09:50:45.513017+0000","last_active":"2024-09-21T09:59:22.768137+0000","last_peered":"2024-09-21T09:59:22.768137+0000","last_clean":"2024-09-21T09:59:22.768137+0000","last_became_active":"2024-09-21T09:50:45.512629+0000","last_became_peered":"2024-09-21T09:50:45.512629+0000","last_unstale":"2024-09-21T09:59:22.768137+0000","last_undegraded":"2024-09-21T09:59:22.768137+0000","last_fullsized":"2024-09-21T09:59:22.768137+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:47:08.421575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":89,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.688449+0000","last_change":"2024-09-21T09:56:12.578501+0000","last_active":"2024-09-21T09:59:22.688449+0000","last_peered":"2024-09-21T09:59:22.688449+0000","last_clean":"2024-09-21T09:59:22.688449+0000","last_became_active":"2024-09-21T09:56:12.578060+0000","last_became_peered":"2024-09-21T09:56:12.578060+0000","last_unstale":"2024-09-21T09:59:22.688449+0000","last_undegraded":"2024-09-21T09:59:22.688449+0000","last_fullsized":"2024-09-21T09:59:22.688449+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:46:42.441923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":158,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.677411+0000","last_change":"2024-09-21T09:50:45.510812+0000","last_active":"2024-09-21T09:59:22.677411+0000","last_peered":"2024-09-21T09:59:22.677411+0000","last_clean":"2024-09-21T09:59:22.677411+0000","last_became_active":"2024-09-21T09:50:45.510480+0000","last_became_peered":"2024-09-21T09:50:45.510480+0000","last_unstale":"2024-09-21T09:59:22.677411+0000","last_undegraded":"2024-09-21T09:59:22.677411+0000","last_fullsized":"2024-09-21T09:59:22.677411+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:05.408194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.844139+0000","last_change":"2024-09-21T09:51:30.646536+0000","last_active":"2024-09-21T09:59:21.844139+0000","last_peered":"2024-09-21T09:59:21.844139+0000","last_clean":"2024-09-21T09:59:21.844139+0000","last_became_active":"2024-09-21T09:51:30.646204+0000","last_became_peered":"2024-09-21T09:51:30.646204+0000","last_unstale":"2024-09-21T09:59:21.844139+0000","last_undegraded":"2024-09-21T09:59:21.844139+0000","last_fullsized":"2024-09-21T09:59:21.844139+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:20.327494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":86,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665138+0000","last_change":"2024-09-21T09:40:49.190763+0000","last_active":"2024-09-21T09:59:21.665138+0000","last_peered":"2024-09-21T09:59:21.665138+0000","last_clean":"2024-09-21T09:59:21.665138+0000","last_became_active":"2024-09-21T09:40:49.190463+0000","last_became_peered":"2024-09-21T09:40:49.190463+0000","last_unstale":"2024-09-21T09:59:21.665138+0000","last_undegraded":"2024-09-21T09:59:21.665138+0000","last_fullsized":"2024-09-21T09:59:21.665138+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:15.669404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":113,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665996+0000","last_change":"2024-09-21T09:51:30.642074+0000","last_active":"2024-09-21T09:59:21.665996+0000","last_peered":"2024-09-21T09:59:21.665996+0000","last_clean":"2024-09-21T09:59:21.665996+0000","last_became_active":"2024-09-21T09:51:30.641176+0000","last_became_peered":"2024-09-21T09:51:30.641176+0000","last_unstale":"2024-09-21T09:59:21.665996+0000","last_undegraded":"2024-09-21T09:59:21.665996+0000","last_fullsized":"2024-09-21T09:59:21.665996+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:18.766865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":110,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665580+0000","last_change":"2024-09-21T09:40:49.096338+0000","last_active":"2024-09-21T09:59:21.665580+0000","last_peered":"2024-09-21T09:59:21.665580+0000","last_clean":"2024-09-21T09:59:21.665580+0000","last_became_active":"2024-09-21T09:40:49.096078+0000","last_became_peered":"2024-09-21T09:40:49.096078+0000","last_unstale":"2024-09-21T09:59:21.665580+0000","last_undegraded":"2024-09-21T09:59:21.665580+0000","last_fullsized":"2024-09-21T09:59:21.665580+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:56.500555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":125,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.746249+0000","last_change":"2024-09-21T09:51:30.642334+0000","last_active":"2024-09-21T09:59:22.746249+0000","last_peered":"2024-09-21T09:59:22.746249+0000","last_clean":"2024-09-21T09:59:22.746249+0000","last_became_active":"2024-09-21T09:51:30.641267+0000","last_became_peered":"2024-09-21T09:51:30.641267+0000","last_unstale":"2024-09-21T09:59:22.746249+0000","last_undegraded":"2024-09-21T09:59:22.746249+0000","last_fullsized":"2024-09-21T09:59:22.746249+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:24.201971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":107,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843634+0000","last_change":"2024-09-21T09:56:12.580601+0000","last_active":"2024-09-21T09:59:21.843634+0000","last_peered":"2024-09-21T09:59:21.843634+0000","last_clean":"2024-09-21T09:59:21.843634+0000","last_became_active":"2024-09-21T09:56:12.580162+0000","last_became_peered":"2024-09-21T09:56:12.580162+0000","last_unstale":"2024-09-21T09:59:21.843634+0000","last_undegraded":"2024-09-21T09:59:21.843634+0000","last_fullsized":"2024-09-21T09:59:21.843634+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:49:16.381068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":105,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660053+0000","last_change":"2024-09-21T09:50:45.510379+0000","last_active":"2024-09-21T09:59:21.660053+0000","last_peered":"2024-09-21T09:59:21.660053+0000","last_clean":"2024-09-21T09:59:21.660053+0000","last_became_active":"2024-09-21T09:50:45.510082+0000","last_became_peered":"2024-09-21T09:50:45.510082+0000","last_unstale":"2024-09-21T09:59:21.660053+0000","last_undegraded":"2024-09-21T09:59:21.660053+0000","last_fullsized":"2024-09-21T09:59:21.660053+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:50.140708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":93,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.684782+0000","last_change":"2024-09-21T09:56:12.579297+0000","last_active":"2024-09-21T09:59:22.684782+0000","last_peered":"2024-09-21T09:59:22.684782+0000","last_clean":"2024-09-21T09:59:22.684782+0000","last_became_active":"2024-09-21T09:56:12.578321+0000","last_became_peered":"2024-09-21T09:56:12.578321+0000","last_unstale":"2024-09-21T09:59:22.684782+0000","last_undegraded":"2024-09-21T09:59:22.684782+0000","last_fullsized":"2024-09-21T09:59:22.684782+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:27:33.794391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":153,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672638+0000","last_change":"2024-09-21T09:56:12.574472+0000","last_active":"2024-09-21T09:59:22.672638+0000","last_peered":"2024-09-21T09:59:22.672638+0000","last_clean":"2024-09-21T09:59:22.672638+0000","last_became_active":"2024-09-21T09:56:12.573966+0000","last_became_peered":"2024-09-21T09:56:12.573966+0000","last_unstale":"2024-09-21T09:59:22.672638+0000","last_undegraded":"2024-09-21T09:59:22.672638+0000","last_fullsized":"2024-09-21T09:59:22.672638+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:37:21.415933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":103,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.687919+0000","last_change":"2024-09-21T09:56:12.576756+0000","last_active":"2024-09-21T09:59:22.687919+0000","last_peered":"2024-09-21T09:59:22.687919+0000","last_clean":"2024-09-21T09:59:22.687919+0000","last_became_active":"2024-09-21T09:56:12.575752+0000","last_became_peered":"2024-09-21T09:56:12.575752+0000","last_unstale":"2024-09-21T09:59:22.687919+0000","last_undegraded":"2024-09-21T09:59:22.687919+0000","last_fullsized":"2024-09-21T09:59:22.687919+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:55.224839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.844056+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T09:59:21.844056+0000","last_peered":"2024-09-21T09:59:21.844056+0000","last_clean":"2024-09-21T09:59:21.844056+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T09:59:21.844056+0000","last_undegraded":"2024-09-21T09:59:21.844056+0000","last_fullsized":"2024-09-21T09:59:21.844056+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":121,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.688491+0000","last_change":"2024-09-21T09:56:12.571495+0000","last_active":"2024-09-21T09:59:22.688491+0000","last_peered":"2024-09-21T09:59:22.688491+0000","last_clean":"2024-09-21T09:59:22.688491+0000","last_became_active":"2024-09-21T09:56:12.571200+0000","last_became_peered":"2024-09-21T09:56:12.571200+0000","last_unstale":"2024-09-21T09:59:22.688491+0000","last_undegraded":"2024-09-21T09:59:22.688491+0000","last_fullsized":"2024-09-21T09:59:22.688491+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:53:58.851300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":155,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.677181+0000","last_change":"2024-09-21T09:56:12.574888+0000","last_active":"2024-09-21T09:59:22.677181+0000","last_peered":"2024-09-21T09:59:22.677181+0000","last_clean":"2024-09-21T09:59:22.677181+0000","last_became_active":"2024-09-21T09:56:12.574215+0000","last_became_peered":"2024-09-21T09:56:12.574215+0000","last_unstale":"2024-09-21T09:59:22.677181+0000","last_undegraded":"2024-09-21T09:59:22.677181+0000","last_fullsized":"2024-09-21T09:59:22.677181+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:48.599320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":123,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.747399+0000","last_change":"2024-09-21T09:56:12.580276+0000","last_active":"2024-09-21T09:59:22.747399+0000","last_peered":"2024-09-21T09:59:22.747399+0000","last_clean":"2024-09-21T09:59:22.747399+0000","last_became_active":"2024-09-21T09:56:12.579989+0000","last_became_peered":"2024-09-21T09:56:12.579989+0000","last_unstale":"2024-09-21T09:59:22.747399+0000","last_undegraded":"2024-09-21T09:59:22.747399+0000","last_fullsized":"2024-09-21T09:59:22.747399+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:14:56.361873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":103,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.688111+0000","last_change":"2024-09-21T09:56:12.582383+0000","last_active":"2024-09-21T09:59:22.688111+0000","last_peered":"2024-09-21T09:59:22.688111+0000","last_clean":"2024-09-21T09:59:22.688111+0000","last_became_active":"2024-09-21T09:56:12.581427+0000","last_became_peered":"2024-09-21T09:56:12.581427+0000","last_unstale":"2024-09-21T09:59:22.688111+0000","last_undegraded":"2024-09-21T09:59:22.688111+0000","last_fullsized":"2024-09-21T09:59:22.688111+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:24.275633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":122,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665267+0000","last_change":"2024-09-21T09:51:30.645644+0000","last_active":"2024-09-21T09:59:21.665267+0000","last_peered":"2024-09-21T09:59:21.665267+0000","last_clean":"2024-09-21T09:59:21.665267+0000","last_became_active":"2024-09-21T09:51:30.644883+0000","last_became_peered":"2024-09-21T09:51:30.644883+0000","last_unstale":"2024-09-21T09:59:21.665267+0000","last_undegraded":"2024-09-21T09:59:21.665267+0000","last_fullsized":"2024-09-21T09:59:21.665267+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:39.331936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":157,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.675764+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T09:59:22.675764+0000","last_peered":"2024-09-21T09:59:22.675764+0000","last_clean":"2024-09-21T09:59:22.675764+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T09:59:22.675764+0000","last_undegraded":"2024-09-21T09:59:22.675764+0000","last_fullsized":"2024-09-21T09:59:22.675764+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":146,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672236+0000","last_change":"2024-09-21T09:48:54.130327+0000","last_active":"2024-09-21T09:59:22.672236+0000","last_peered":"2024-09-21T09:59:22.672236+0000","last_clean":"2024-09-21T09:59:22.672236+0000","last_became_active":"2024-09-21T09:48:54.128350+0000","last_became_peered":"2024-09-21T09:48:54.128350+0000","last_unstale":"2024-09-21T09:59:22.672236+0000","last_undegraded":"2024-09-21T09:59:22.672236+0000","last_fullsized":"2024-09-21T09:59:22.672236+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:31:40.490861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":120,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665946+0000","last_change":"2024-09-21T09:56:12.576148+0000","last_active":"2024-09-21T09:59:21.665946+0000","last_peered":"2024-09-21T09:59:21.665946+0000","last_clean":"2024-09-21T09:59:21.665946+0000","last_became_active":"2024-09-21T09:56:12.575813+0000","last_became_peered":"2024-09-21T09:56:12.575813+0000","last_unstale":"2024-09-21T09:59:21.665946+0000","last_undegraded":"2024-09-21T09:59:21.665946+0000","last_fullsized":"2024-09-21T09:59:21.665946+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:53:01.367816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":107,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660030+0000","last_change":"2024-09-21T09:50:45.502447+0000","last_active":"2024-09-21T09:59:21.660030+0000","last_peered":"2024-09-21T09:59:21.660030+0000","last_clean":"2024-09-21T09:59:21.660030+0000","last_became_active":"2024-09-21T09:50:45.500800+0000","last_became_peered":"2024-09-21T09:50:45.500800+0000","last_unstale":"2024-09-21T09:59:21.660030+0000","last_undegraded":"2024-09-21T09:59:21.660030+0000","last_fullsized":"2024-09-21T09:59:21.660030+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:20:51.819410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":120,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665155+0000","last_change":"2024-09-21T09:56:12.576457+0000","last_active":"2024-09-21T09:59:21.665155+0000","last_peered":"2024-09-21T09:59:21.665155+0000","last_clean":"2024-09-21T09:59:21.665155+0000","last_became_active":"2024-09-21T09:56:12.576061+0000","last_became_peered":"2024-09-21T09:56:12.576061+0000","last_unstale":"2024-09-21T09:59:21.665155+0000","last_undegraded":"2024-09-21T09:59:21.665155+0000","last_fullsized":"2024-09-21T09:59:21.665155+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:51:46.676699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":90,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660124+0000","last_change":"2024-09-21T09:40:49.094916+0000","last_active":"2024-09-21T09:59:21.660124+0000","last_peered":"2024-09-21T09:59:21.660124+0000","last_clean":"2024-09-21T09:59:21.660124+0000","last_became_active":"2024-09-21T09:40:49.094264+0000","last_became_peered":"2024-09-21T09:40:49.094264+0000","last_unstale":"2024-09-21T09:59:21.660124+0000","last_undegraded":"2024-09-21T09:59:21.660124+0000","last_fullsized":"2024-09-21T09:59:21.660124+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:53.663226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":129,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.685081+0000","last_change":"2024-09-21T09:56:12.577329+0000","last_active":"2024-09-21T09:59:22.685081+0000","last_peered":"2024-09-21T09:59:22.685081+0000","last_clean":"2024-09-21T09:59:22.685081+0000","last_became_active":"2024-09-21T09:56:12.576941+0000","last_became_peered":"2024-09-21T09:56:12.576941+0000","last_unstale":"2024-09-21T09:59:22.685081+0000","last_undegraded":"2024-09-21T09:59:22.685081+0000","last_fullsized":"2024-09-21T09:59:22.685081+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:35.235768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":115,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.689026+0000","last_change":"2024-09-21T09:56:12.577137+0000","last_active":"2024-09-21T09:59:22.689026+0000","last_peered":"2024-09-21T09:59:22.689026+0000","last_clean":"2024-09-21T09:59:22.689026+0000","last_became_active":"2024-09-21T09:56:12.576668+0000","last_became_peered":"2024-09-21T09:56:12.576668+0000","last_unstale":"2024-09-21T09:59:22.689026+0000","last_undegraded":"2024-09-21T09:59:22.689026+0000","last_fullsized":"2024-09-21T09:59:22.689026+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:56.772427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.664974+0000","last_change":"2024-09-21T09:50:45.502819+0000","last_active":"2024-09-21T09:59:21.664974+0000","last_peered":"2024-09-21T09:59:21.664974+0000","last_clean":"2024-09-21T09:59:21.664974+0000","last_became_active":"2024-09-21T09:50:45.502483+0000","last_became_peered":"2024-09-21T09:50:45.502483+0000","last_unstale":"2024-09-21T09:59:21.664974+0000","last_undegraded":"2024-09-21T09:59:21.664974+0000","last_fullsized":"2024-09-21T09:59:21.664974+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:26.549958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":158,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672143+0000","last_change":"2024-09-21T09:51:30.650116+0000","last_active":"2024-09-21T09:59:22.672143+0000","last_peered":"2024-09-21T09:59:22.672143+0000","last_clean":"2024-09-21T09:59:22.672143+0000","last_became_active":"2024-09-21T09:51:30.649798+0000","last_became_peered":"2024-09-21T09:51:30.649798+0000","last_unstale":"2024-09-21T09:59:22.672143+0000","last_undegraded":"2024-09-21T09:59:22.672143+0000","last_fullsized":"2024-09-21T09:59:22.672143+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:40.222590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13748,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626695,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299524,"kb_used_data":2164,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93445628,"statfs":{"total":95995035648,"available":95688323072,"internally_reserved":0,"allocated":2215936,"data_stored":1518744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019153,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":587332,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93157820,"statfs":{"total":95995035648,"available":95393607680,"internally_reserved":0,"allocated":1626112,"data_stored":928433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":81,"seq":347892351114,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":406520,"kb_used_data":2152,"kb_used_omap":5,"kb_used_meta":404346,"kb_avail":93338632,"statfs":{"total":95995035648,"available":95578759168,"internally_reserved":0,"allocated":2203648,"data_stored":1514539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":414050364},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430467,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":596236,"kb_used_data":2172,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93148916,"statfs":{"total":95995035648,"available":95384489984,"internally_reserved":0,"allocated":2224128,"data_stored":1518887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822924,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":910832,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":909246,"kb_avail":92834320,"statfs":{"total":95995035648,"available":95062343680,"internally_reserved":0,"allocated":1605632,"data_stored":923580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931068248},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879347,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":222768,"kb_used_data":1568,"kb_used_omap":4,"kb_used_meta":221179,"kb_avail":93522384,"statfs":{"total":95995035648,"available":95766921216,"internally_reserved":0,"allocated":1605632,"data_stored":923653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":226487740},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566187,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":291252,"kb_used_data":1572,"kb_used_omap":6,"kb_used_meta":289657,"kb_avail":93453900,"statfs":{"total":95995035648,"available":95696793600,"internally_reserved":0,"allocated":1609728,"data_stored":924175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":296609191},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860532,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":329964,"kb_used_data":1564,"kb_used_omap":5,"kb_used_meta":328378,"kb_avail":93415188,"statfs":{"total":95995035648,"available":95657152512,"internally_reserved":0,"allocated":1601536,"data_stored":923267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":336259311},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:00:20.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:00:20.566 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:00:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:00:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:20.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:20 smithi107 ceph-mon[102221]: from='client.45692 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:00:20.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:20 smithi107 ceph-mon[105022]: from='client.45692 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:00:20.618 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:20 smithi129 ceph-mon[88471]: from='client.45692 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:00:21.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:21 smithi107 ceph-mon[102221]: pgmap v484: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:21.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:21 smithi107 ceph-mon[105022]: pgmap v484: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:21 smithi129 ceph-mon[88471]: pgmap v484: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:22.105 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:00:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:00:21] "GET /metrics HTTP/1.1" 200 34150 "" "Prometheus/2.51.0" 2024-09-21T10:00:22.495 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:00:22.495 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:00:23.158 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-21T10:00:22.131253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":213,"num_write_kb":1660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3644428,"kb_used_data":14348,"kb_used_omap":29,"kb_used_meta":3629922,"kb_avail":746316788,"statfs":{"total":767960285184,"available":764228390912,"internally_reserved":0,"allocated":14692352,"data_stored":9175278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30226,"internal_metadata":3717040622},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002969"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":103,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.685385+0000","last_change":"2024-09-21T09:56:12.582314+0000","last_active":"2024-09-21T09:59:22.685385+0000","last_peered":"2024-09-21T09:59:22.685385+0000","last_clean":"2024-09-21T09:59:22.685385+0000","last_became_active":"2024-09-21T09:56:12.581458+0000","last_became_peered":"2024-09-21T09:56:12.581458+0000","last_unstale":"2024-09-21T09:59:22.685385+0000","last_undegraded":"2024-09-21T09:59:22.685385+0000","last_fullsized":"2024-09-21T09:59:22.685385+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:17:21.624171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":106,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843561+0000","last_change":"2024-09-21T09:51:30.640091+0000","last_active":"2024-09-21T09:59:21.843561+0000","last_peered":"2024-09-21T09:59:21.843561+0000","last_clean":"2024-09-21T09:59:21.843561+0000","last_became_active":"2024-09-21T09:51:30.639739+0000","last_became_peered":"2024-09-21T09:51:30.639739+0000","last_unstale":"2024-09-21T09:59:21.843561+0000","last_undegraded":"2024-09-21T09:59:21.843561+0000","last_fullsized":"2024-09-21T09:59:21.843561+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:41.958415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660114+0000","last_change":"2024-09-21T09:56:12.583291+0000","last_active":"2024-09-21T09:59:21.660114+0000","last_peered":"2024-09-21T09:59:21.660114+0000","last_clean":"2024-09-21T09:59:21.660114+0000","last_became_active":"2024-09-21T09:56:12.582908+0000","last_became_peered":"2024-09-21T09:56:12.582908+0000","last_unstale":"2024-09-21T09:59:21.660114+0000","last_undegraded":"2024-09-21T09:59:21.660114+0000","last_fullsized":"2024-09-21T09:59:21.660114+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:04:07.985942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":153,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672057+0000","last_change":"2024-09-21T09:56:12.573112+0000","last_active":"2024-09-21T09:59:22.672057+0000","last_peered":"2024-09-21T09:59:22.672057+0000","last_clean":"2024-09-21T09:59:22.672057+0000","last_became_active":"2024-09-21T09:56:12.572835+0000","last_became_peered":"2024-09-21T09:56:12.572835+0000","last_unstale":"2024-09-21T09:59:22.672057+0000","last_undegraded":"2024-09-21T09:59:22.672057+0000","last_fullsized":"2024-09-21T09:59:22.672057+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:53.493530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":154,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.677271+0000","last_change":"2024-09-21T09:56:12.574922+0000","last_active":"2024-09-21T09:59:22.677271+0000","last_peered":"2024-09-21T09:59:22.677271+0000","last_clean":"2024-09-21T09:59:22.677271+0000","last_became_active":"2024-09-21T09:56:12.574450+0000","last_became_peered":"2024-09-21T09:56:12.574450+0000","last_unstale":"2024-09-21T09:59:22.677271+0000","last_undegraded":"2024-09-21T09:59:22.677271+0000","last_fullsized":"2024-09-21T09:59:22.677271+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:48.408302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":87,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843305+0000","last_change":"2024-09-21T09:40:49.101911+0000","last_active":"2024-09-21T09:59:21.843305+0000","last_peered":"2024-09-21T09:59:21.843305+0000","last_clean":"2024-09-21T09:59:21.843305+0000","last_became_active":"2024-09-21T09:40:49.101629+0000","last_became_peered":"2024-09-21T09:40:49.101629+0000","last_unstale":"2024-09-21T09:59:21.843305+0000","last_undegraded":"2024-09-21T09:59:21.843305+0000","last_fullsized":"2024-09-21T09:59:21.843305+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:07:19.215551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":103,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.686040+0000","last_change":"2024-09-21T09:56:12.601652+0000","last_active":"2024-09-21T09:59:22.686040+0000","last_peered":"2024-09-21T09:59:22.686040+0000","last_clean":"2024-09-21T09:59:22.686040+0000","last_became_active":"2024-09-21T09:56:12.583693+0000","last_became_peered":"2024-09-21T09:56:12.583693+0000","last_unstale":"2024-09-21T09:59:22.686040+0000","last_undegraded":"2024-09-21T09:59:22.686040+0000","last_fullsized":"2024-09-21T09:59:22.686040+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:24.992107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":112,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843253+0000","last_change":"2024-09-21T09:48:54.140948+0000","last_active":"2024-09-21T09:59:21.843253+0000","last_peered":"2024-09-21T09:59:21.843253+0000","last_clean":"2024-09-21T09:59:21.843253+0000","last_became_active":"2024-09-21T09:48:54.140641+0000","last_became_peered":"2024-09-21T09:48:54.140641+0000","last_unstale":"2024-09-21T09:59:21.843253+0000","last_undegraded":"2024-09-21T09:59:21.843253+0000","last_fullsized":"2024-09-21T09:59:21.843253+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:25.157599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":105,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843654+0000","last_change":"2024-09-21T09:48:54.127687+0000","last_active":"2024-09-21T09:59:21.843654+0000","last_peered":"2024-09-21T09:59:21.843654+0000","last_clean":"2024-09-21T09:59:21.843654+0000","last_became_active":"2024-09-21T09:48:54.127325+0000","last_became_peered":"2024-09-21T09:48:54.127325+0000","last_unstale":"2024-09-21T09:59:21.843654+0000","last_undegraded":"2024-09-21T09:59:21.843654+0000","last_fullsized":"2024-09-21T09:59:21.843654+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:30.004794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":87,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.659704+0000","last_change":"2024-09-21T09:40:49.397673+0000","last_active":"2024-09-21T09:59:21.659704+0000","last_peered":"2024-09-21T09:59:21.659704+0000","last_clean":"2024-09-21T09:59:21.659704+0000","last_became_active":"2024-09-21T09:40:49.397365+0000","last_became_peered":"2024-09-21T09:40:49.397365+0000","last_unstale":"2024-09-21T09:59:21.659704+0000","last_undegraded":"2024-09-21T09:59:21.659704+0000","last_fullsized":"2024-09-21T09:59:21.659704+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:48:55.191816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":106,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843931+0000","last_change":"2024-09-21T09:50:45.508896+0000","last_active":"2024-09-21T09:59:21.843931+0000","last_peered":"2024-09-21T09:59:21.843931+0000","last_clean":"2024-09-21T09:59:21.843931+0000","last_became_active":"2024-09-21T09:50:45.508504+0000","last_became_peered":"2024-09-21T09:50:45.508504+0000","last_unstale":"2024-09-21T09:59:21.843931+0000","last_undegraded":"2024-09-21T09:59:21.843931+0000","last_fullsized":"2024-09-21T09:59:21.843931+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:51:23.098434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":120,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665055+0000","last_change":"2024-09-21T09:56:12.576294+0000","last_active":"2024-09-21T09:59:21.665055+0000","last_peered":"2024-09-21T09:59:21.665055+0000","last_clean":"2024-09-21T09:59:21.665055+0000","last_became_active":"2024-09-21T09:56:12.575928+0000","last_became_peered":"2024-09-21T09:56:12.575928+0000","last_unstale":"2024-09-21T09:59:21.665055+0000","last_undegraded":"2024-09-21T09:59:21.665055+0000","last_fullsized":"2024-09-21T09:59:21.665055+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:23.227435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":88,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843761+0000","last_change":"2024-09-21T09:40:49.096812+0000","last_active":"2024-09-21T09:59:21.843761+0000","last_peered":"2024-09-21T09:59:21.843761+0000","last_clean":"2024-09-21T09:59:21.843761+0000","last_became_active":"2024-09-21T09:40:49.096549+0000","last_became_peered":"2024-09-21T09:40:49.096549+0000","last_unstale":"2024-09-21T09:59:21.843761+0000","last_undegraded":"2024-09-21T09:59:21.843761+0000","last_fullsized":"2024-09-21T09:59:21.843761+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:24.512887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843208+0000","last_change":"2024-09-21T09:50:45.500221+0000","last_active":"2024-09-21T09:59:21.843208+0000","last_peered":"2024-09-21T09:59:21.843208+0000","last_clean":"2024-09-21T09:59:21.843208+0000","last_became_active":"2024-09-21T09:50:45.499939+0000","last_became_peered":"2024-09-21T09:50:45.499939+0000","last_unstale":"2024-09-21T09:59:21.843208+0000","last_undegraded":"2024-09-21T09:59:21.843208+0000","last_fullsized":"2024-09-21T09:59:21.843208+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:08:12.782227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":105,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660058+0000","last_change":"2024-09-21T09:56:12.590219+0000","last_active":"2024-09-21T09:59:21.660058+0000","last_peered":"2024-09-21T09:59:21.660058+0000","last_clean":"2024-09-21T09:59:21.660058+0000","last_became_active":"2024-09-21T09:56:12.589946+0000","last_became_peered":"2024-09-21T09:56:12.589946+0000","last_unstale":"2024-09-21T09:59:21.660058+0000","last_undegraded":"2024-09-21T09:59:21.660058+0000","last_fullsized":"2024-09-21T09:59:21.660058+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:48.919929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":155,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672048+0000","last_change":"2024-09-21T09:50:45.503336+0000","last_active":"2024-09-21T09:59:22.672048+0000","last_peered":"2024-09-21T09:59:22.672048+0000","last_clean":"2024-09-21T09:59:22.672048+0000","last_became_active":"2024-09-21T09:50:45.503045+0000","last_became_peered":"2024-09-21T09:50:45.503045+0000","last_unstale":"2024-09-21T09:59:22.672048+0000","last_undegraded":"2024-09-21T09:59:22.672048+0000","last_fullsized":"2024-09-21T09:59:22.672048+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:46.728347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":91,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.685711+0000","last_change":"2024-09-21T09:56:12.575897+0000","last_active":"2024-09-21T09:59:22.685711+0000","last_peered":"2024-09-21T09:59:22.685711+0000","last_clean":"2024-09-21T09:59:22.685711+0000","last_became_active":"2024-09-21T09:56:12.575532+0000","last_became_peered":"2024-09-21T09:56:12.575532+0000","last_unstale":"2024-09-21T09:59:22.685711+0000","last_undegraded":"2024-09-21T09:59:22.685711+0000","last_fullsized":"2024-09-21T09:59:22.685711+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:01.302499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665473+0000","last_change":"2024-09-21T09:56:12.588824+0000","last_active":"2024-09-21T09:59:21.665473+0000","last_peered":"2024-09-21T09:59:21.665473+0000","last_clean":"2024-09-21T09:59:21.665473+0000","last_became_active":"2024-09-21T09:56:12.588561+0000","last_became_peered":"2024-09-21T09:56:12.588561+0000","last_unstale":"2024-09-21T09:59:21.665473+0000","last_undegraded":"2024-09-21T09:59:21.665473+0000","last_fullsized":"2024-09-21T09:59:21.665473+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:31:48.900623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843805+0000","last_change":"2024-09-21T09:50:45.501596+0000","last_active":"2024-09-21T09:59:21.843805+0000","last_peered":"2024-09-21T09:59:21.843805+0000","last_clean":"2024-09-21T09:59:21.843805+0000","last_became_active":"2024-09-21T09:50:45.501186+0000","last_became_peered":"2024-09-21T09:50:45.501186+0000","last_unstale":"2024-09-21T09:59:21.843805+0000","last_undegraded":"2024-09-21T09:59:21.843805+0000","last_fullsized":"2024-09-21T09:59:21.843805+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:43:16.449425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":113,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665698+0000","last_change":"2024-09-21T09:51:30.639545+0000","last_active":"2024-09-21T09:59:21.665698+0000","last_peered":"2024-09-21T09:59:21.665698+0000","last_clean":"2024-09-21T09:59:21.665698+0000","last_became_active":"2024-09-21T09:51:30.639106+0000","last_became_peered":"2024-09-21T09:51:30.639106+0000","last_unstale":"2024-09-21T09:59:21.665698+0000","last_undegraded":"2024-09-21T09:59:21.665698+0000","last_fullsized":"2024-09-21T09:59:21.665698+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:19:19.607327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665245+0000","last_change":"2024-09-21T09:51:30.639276+0000","last_active":"2024-09-21T09:59:21.665245+0000","last_peered":"2024-09-21T09:59:21.665245+0000","last_clean":"2024-09-21T09:59:21.665245+0000","last_became_active":"2024-09-21T09:51:30.638273+0000","last_became_peered":"2024-09-21T09:51:30.638273+0000","last_unstale":"2024-09-21T09:59:21.665245+0000","last_undegraded":"2024-09-21T09:59:21.665245+0000","last_fullsized":"2024-09-21T09:59:21.665245+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:19.093920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665822+0000","last_change":"2024-09-21T09:50:45.510764+0000","last_active":"2024-09-21T09:59:21.665822+0000","last_peered":"2024-09-21T09:59:21.665822+0000","last_clean":"2024-09-21T09:59:21.665822+0000","last_became_active":"2024-09-21T09:50:45.510226+0000","last_became_peered":"2024-09-21T09:50:45.510226+0000","last_unstale":"2024-09-21T09:59:21.665822+0000","last_undegraded":"2024-09-21T09:59:21.665822+0000","last_fullsized":"2024-09-21T09:59:21.665822+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:31:13.491427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":104,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.689113+0000","last_change":"2024-09-21T09:56:12.578433+0000","last_active":"2024-09-21T09:59:22.689113+0000","last_peered":"2024-09-21T09:59:22.689113+0000","last_clean":"2024-09-21T09:59:22.689113+0000","last_became_active":"2024-09-21T09:56:12.578012+0000","last_became_peered":"2024-09-21T09:56:12.578012+0000","last_unstale":"2024-09-21T09:59:22.689113+0000","last_undegraded":"2024-09-21T09:59:22.689113+0000","last_fullsized":"2024-09-21T09:59:22.689113+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:07:59.598935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":125,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.768359+0000","last_change":"2024-09-21T09:56:12.575396+0000","last_active":"2024-09-21T09:59:22.768359+0000","last_peered":"2024-09-21T09:59:22.768359+0000","last_clean":"2024-09-21T09:59:22.768359+0000","last_became_active":"2024-09-21T09:56:12.574913+0000","last_became_peered":"2024-09-21T09:56:12.574913+0000","last_unstale":"2024-09-21T09:59:22.768359+0000","last_undegraded":"2024-09-21T09:59:22.768359+0000","last_fullsized":"2024-09-21T09:59:22.768359+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:05.462880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":221,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843628+0000","last_change":"2024-09-21T09:48:54.128018+0000","last_active":"2024-09-21T09:59:21.843628+0000","last_peered":"2024-09-21T09:59:21.843628+0000","last_clean":"2024-09-21T09:59:21.843628+0000","last_became_active":"2024-09-21T09:48:54.127533+0000","last_became_peered":"2024-09-21T09:48:54.127533+0000","last_unstale":"2024-09-21T09:59:21.843628+0000","last_undegraded":"2024-09-21T09:59:21.843628+0000","last_fullsized":"2024-09-21T09:59:21.843628+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:42.410506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":118,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.659885+0000","last_change":"2024-09-21T09:51:30.636796+0000","last_active":"2024-09-21T09:59:21.659885+0000","last_peered":"2024-09-21T09:59:21.659885+0000","last_clean":"2024-09-21T09:59:21.659885+0000","last_became_active":"2024-09-21T09:51:30.636052+0000","last_became_peered":"2024-09-21T09:51:30.636052+0000","last_unstale":"2024-09-21T09:59:21.659885+0000","last_undegraded":"2024-09-21T09:59:21.659885+0000","last_fullsized":"2024-09-21T09:59:21.659885+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:21:36.694847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039581700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":93,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.685916+0000","last_change":"2024-09-21T09:56:12.580057+0000","last_active":"2024-09-21T09:59:22.685916+0000","last_peered":"2024-09-21T09:59:22.685916+0000","last_clean":"2024-09-21T09:59:22.685916+0000","last_became_active":"2024-09-21T09:56:12.579700+0000","last_became_peered":"2024-09-21T09:56:12.579700+0000","last_unstale":"2024-09-21T09:59:22.685916+0000","last_undegraded":"2024-09-21T09:59:22.685916+0000","last_fullsized":"2024-09-21T09:59:22.685916+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:49.768872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":86,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.844142+0000","last_change":"2024-09-21T09:40:50.115273+0000","last_active":"2024-09-21T09:59:21.844142+0000","last_peered":"2024-09-21T09:59:21.844142+0000","last_clean":"2024-09-21T09:59:21.844142+0000","last_became_active":"2024-09-21T09:40:50.114998+0000","last_became_peered":"2024-09-21T09:40:50.114998+0000","last_unstale":"2024-09-21T09:59:21.844142+0000","last_undegraded":"2024-09-21T09:59:21.844142+0000","last_fullsized":"2024-09-21T09:59:21.844142+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:42:39.208914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":133,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843492+0000","last_change":"2024-09-21T09:50:45.501687+0000","last_active":"2024-09-21T09:59:21.843492+0000","last_peered":"2024-09-21T09:59:21.843492+0000","last_clean":"2024-09-21T09:59:21.843492+0000","last_became_active":"2024-09-21T09:50:45.501368+0000","last_became_peered":"2024-09-21T09:50:45.501368+0000","last_unstale":"2024-09-21T09:59:21.843492+0000","last_undegraded":"2024-09-21T09:59:21.843492+0000","last_fullsized":"2024-09-21T09:59:21.843492+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:51.187511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":154,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672126+0000","last_change":"2024-09-21T09:56:12.573698+0000","last_active":"2024-09-21T09:59:22.672126+0000","last_peered":"2024-09-21T09:59:22.672126+0000","last_clean":"2024-09-21T09:59:22.672126+0000","last_became_active":"2024-09-21T09:56:12.573390+0000","last_became_peered":"2024-09-21T09:56:12.573390+0000","last_unstale":"2024-09-21T09:59:22.672126+0000","last_undegraded":"2024-09-21T09:59:22.672126+0000","last_fullsized":"2024-09-21T09:59:22.672126+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:55:12.937878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":166,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.677131+0000","last_change":"2024-09-21T09:51:30.666462+0000","last_active":"2024-09-21T09:59:22.677131+0000","last_peered":"2024-09-21T09:59:22.677131+0000","last_clean":"2024-09-21T09:59:22.677131+0000","last_became_active":"2024-09-21T09:51:30.666006+0000","last_became_peered":"2024-09-21T09:51:30.666006+0000","last_unstale":"2024-09-21T09:59:22.677131+0000","last_undegraded":"2024-09-21T09:59:22.677131+0000","last_fullsized":"2024-09-21T09:59:22.677131+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:51.481698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":156,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.674729+0000","last_change":"2024-09-21T09:56:12.580659+0000","last_active":"2024-09-21T09:59:22.674729+0000","last_peered":"2024-09-21T09:59:22.674729+0000","last_clean":"2024-09-21T09:59:22.674729+0000","last_became_active":"2024-09-21T09:56:12.580300+0000","last_became_peered":"2024-09-21T09:56:12.580300+0000","last_unstale":"2024-09-21T09:59:22.674729+0000","last_undegraded":"2024-09-21T09:59:22.674729+0000","last_fullsized":"2024-09-21T09:59:22.674729+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:04.945404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":124,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.746911+0000","last_change":"2024-09-21T09:51:30.646466+0000","last_active":"2024-09-21T09:59:22.746911+0000","last_peered":"2024-09-21T09:59:22.746911+0000","last_clean":"2024-09-21T09:59:22.746911+0000","last_became_active":"2024-09-21T09:51:30.646209+0000","last_became_peered":"2024-09-21T09:51:30.646209+0000","last_unstale":"2024-09-21T09:59:22.746911+0000","last_undegraded":"2024-09-21T09:59:22.746911+0000","last_fullsized":"2024-09-21T09:59:22.746911+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:44:41.465418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":129,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.675687+0000","last_change":"2024-09-21T09:51:30.649056+0000","last_active":"2024-09-21T09:59:22.675687+0000","last_peered":"2024-09-21T09:59:22.675687+0000","last_clean":"2024-09-21T09:59:22.675687+0000","last_became_active":"2024-09-21T09:51:30.647245+0000","last_became_peered":"2024-09-21T09:51:30.647245+0000","last_unstale":"2024-09-21T09:59:22.675687+0000","last_undegraded":"2024-09-21T09:59:22.675687+0000","last_fullsized":"2024-09-21T09:59:22.675687+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:32.661847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":119,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.659804+0000","last_change":"2024-09-21T09:50:45.502094+0000","last_active":"2024-09-21T09:59:21.659804+0000","last_peered":"2024-09-21T09:59:21.659804+0000","last_clean":"2024-09-21T09:59:21.659804+0000","last_became_active":"2024-09-21T09:50:45.501603+0000","last_became_peered":"2024-09-21T09:50:45.501603+0000","last_unstale":"2024-09-21T09:59:21.659804+0000","last_undegraded":"2024-09-21T09:59:21.659804+0000","last_fullsized":"2024-09-21T09:59:21.659804+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:01.747067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091939799999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":104,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.666081+0000","last_change":"2024-09-21T09:48:54.128617+0000","last_active":"2024-09-21T09:59:21.666081+0000","last_peered":"2024-09-21T09:59:21.666081+0000","last_clean":"2024-09-21T09:59:21.666081+0000","last_became_active":"2024-09-21T09:48:54.127878+0000","last_became_peered":"2024-09-21T09:48:54.127878+0000","last_unstale":"2024-09-21T09:59:21.666081+0000","last_undegraded":"2024-09-21T09:59:21.666081+0000","last_fullsized":"2024-09-21T09:59:21.666081+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:07.305638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":136,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.762089+0000","last_change":"2024-09-21T09:56:12.588992+0000","last_active":"2024-09-21T09:59:22.762089+0000","last_peered":"2024-09-21T09:59:22.762089+0000","last_clean":"2024-09-21T09:59:22.762089+0000","last_became_active":"2024-09-21T09:56:12.588665+0000","last_became_peered":"2024-09-21T09:56:12.588665+0000","last_unstale":"2024-09-21T09:59:22.762089+0000","last_undegraded":"2024-09-21T09:59:22.762089+0000","last_fullsized":"2024-09-21T09:59:22.762089+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:19:06.160456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":122,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660179+0000","last_change":"2024-09-21T09:56:12.583280+0000","last_active":"2024-09-21T09:59:21.660179+0000","last_peered":"2024-09-21T09:59:21.660179+0000","last_clean":"2024-09-21T09:59:21.660179+0000","last_became_active":"2024-09-21T09:56:12.583007+0000","last_became_peered":"2024-09-21T09:56:12.583007+0000","last_unstale":"2024-09-21T09:59:21.660179+0000","last_undegraded":"2024-09-21T09:59:21.660179+0000","last_fullsized":"2024-09-21T09:59:21.660179+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:26.991596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":163,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.762196+0000","last_change":"2024-09-21T09:50:45.509978+0000","last_active":"2024-09-21T09:59:22.762196+0000","last_peered":"2024-09-21T09:59:22.762196+0000","last_clean":"2024-09-21T09:59:22.762196+0000","last_became_active":"2024-09-21T09:50:45.509391+0000","last_became_peered":"2024-09-21T09:50:45.509391+0000","last_unstale":"2024-09-21T09:59:22.762196+0000","last_undegraded":"2024-09-21T09:59:22.762196+0000","last_fullsized":"2024-09-21T09:59:22.762196+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:50.360847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":114,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.751096+0000","last_change":"2024-09-21T09:50:45.501349+0000","last_active":"2024-09-21T09:59:22.751096+0000","last_peered":"2024-09-21T09:59:22.751096+0000","last_clean":"2024-09-21T09:59:22.751096+0000","last_became_active":"2024-09-21T09:50:45.501065+0000","last_became_peered":"2024-09-21T09:50:45.501065+0000","last_unstale":"2024-09-21T09:59:22.751096+0000","last_undegraded":"2024-09-21T09:59:22.751096+0000","last_fullsized":"2024-09-21T09:59:22.751096+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:28:58.570586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":124,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.747028+0000","last_change":"2024-09-21T09:51:30.646531+0000","last_active":"2024-09-21T09:59:22.747028+0000","last_peered":"2024-09-21T09:59:22.747028+0000","last_clean":"2024-09-21T09:59:22.747028+0000","last_became_active":"2024-09-21T09:51:30.646248+0000","last_became_peered":"2024-09-21T09:51:30.646248+0000","last_unstale":"2024-09-21T09:59:22.747028+0000","last_undegraded":"2024-09-21T09:59:22.747028+0000","last_fullsized":"2024-09-21T09:59:22.747028+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:00.312924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843891+0000","last_change":"2024-09-21T09:48:54.125569+0000","last_active":"2024-09-21T09:59:21.843891+0000","last_peered":"2024-09-21T09:59:21.843891+0000","last_clean":"2024-09-21T09:59:21.843891+0000","last_became_active":"2024-09-21T09:48:54.125212+0000","last_became_peered":"2024-09-21T09:48:54.125212+0000","last_unstale":"2024-09-21T09:59:21.843891+0000","last_undegraded":"2024-09-21T09:59:21.843891+0000","last_fullsized":"2024-09-21T09:59:21.843891+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:01.434278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":144,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672095+0000","last_change":"2024-09-21T09:48:54.131373+0000","last_active":"2024-09-21T09:59:22.672095+0000","last_peered":"2024-09-21T09:59:22.672095+0000","last_clean":"2024-09-21T09:59:22.672095+0000","last_became_active":"2024-09-21T09:48:54.130391+0000","last_became_peered":"2024-09-21T09:48:54.130391+0000","last_unstale":"2024-09-21T09:59:22.672095+0000","last_undegraded":"2024-09-21T09:59:22.672095+0000","last_fullsized":"2024-09-21T09:59:22.672095+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:46.136344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":91,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.686169+0000","last_change":"2024-09-21T09:56:12.579726+0000","last_active":"2024-09-21T09:59:22.686169+0000","last_peered":"2024-09-21T09:59:22.686169+0000","last_clean":"2024-09-21T09:59:22.686169+0000","last_became_active":"2024-09-21T09:56:12.579304+0000","last_became_peered":"2024-09-21T09:56:12.579304+0000","last_unstale":"2024-09-21T09:59:22.686169+0000","last_undegraded":"2024-09-21T09:59:22.686169+0000","last_fullsized":"2024-09-21T09:59:22.686169+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:44.605640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":124,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.768137+0000","last_change":"2024-09-21T09:50:45.513017+0000","last_active":"2024-09-21T09:59:22.768137+0000","last_peered":"2024-09-21T09:59:22.768137+0000","last_clean":"2024-09-21T09:59:22.768137+0000","last_became_active":"2024-09-21T09:50:45.512629+0000","last_became_peered":"2024-09-21T09:50:45.512629+0000","last_unstale":"2024-09-21T09:59:22.768137+0000","last_undegraded":"2024-09-21T09:59:22.768137+0000","last_fullsized":"2024-09-21T09:59:22.768137+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:47:08.421575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":89,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.688449+0000","last_change":"2024-09-21T09:56:12.578501+0000","last_active":"2024-09-21T09:59:22.688449+0000","last_peered":"2024-09-21T09:59:22.688449+0000","last_clean":"2024-09-21T09:59:22.688449+0000","last_became_active":"2024-09-21T09:56:12.578060+0000","last_became_peered":"2024-09-21T09:56:12.578060+0000","last_unstale":"2024-09-21T09:59:22.688449+0000","last_undegraded":"2024-09-21T09:59:22.688449+0000","last_fullsized":"2024-09-21T09:59:22.688449+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:46:42.441923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":158,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.677411+0000","last_change":"2024-09-21T09:50:45.510812+0000","last_active":"2024-09-21T09:59:22.677411+0000","last_peered":"2024-09-21T09:59:22.677411+0000","last_clean":"2024-09-21T09:59:22.677411+0000","last_became_active":"2024-09-21T09:50:45.510480+0000","last_became_peered":"2024-09-21T09:50:45.510480+0000","last_unstale":"2024-09-21T09:59:22.677411+0000","last_undegraded":"2024-09-21T09:59:22.677411+0000","last_fullsized":"2024-09-21T09:59:22.677411+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:05.408194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":116,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.844139+0000","last_change":"2024-09-21T09:51:30.646536+0000","last_active":"2024-09-21T09:59:21.844139+0000","last_peered":"2024-09-21T09:59:21.844139+0000","last_clean":"2024-09-21T09:59:21.844139+0000","last_became_active":"2024-09-21T09:51:30.646204+0000","last_became_peered":"2024-09-21T09:51:30.646204+0000","last_unstale":"2024-09-21T09:59:21.844139+0000","last_undegraded":"2024-09-21T09:59:21.844139+0000","last_fullsized":"2024-09-21T09:59:21.844139+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:20.327494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":86,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665138+0000","last_change":"2024-09-21T09:40:49.190763+0000","last_active":"2024-09-21T09:59:21.665138+0000","last_peered":"2024-09-21T09:59:21.665138+0000","last_clean":"2024-09-21T09:59:21.665138+0000","last_became_active":"2024-09-21T09:40:49.190463+0000","last_became_peered":"2024-09-21T09:40:49.190463+0000","last_unstale":"2024-09-21T09:59:21.665138+0000","last_undegraded":"2024-09-21T09:59:21.665138+0000","last_fullsized":"2024-09-21T09:59:21.665138+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:15.669404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":113,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665996+0000","last_change":"2024-09-21T09:51:30.642074+0000","last_active":"2024-09-21T09:59:21.665996+0000","last_peered":"2024-09-21T09:59:21.665996+0000","last_clean":"2024-09-21T09:59:21.665996+0000","last_became_active":"2024-09-21T09:51:30.641176+0000","last_became_peered":"2024-09-21T09:51:30.641176+0000","last_unstale":"2024-09-21T09:59:21.665996+0000","last_undegraded":"2024-09-21T09:59:21.665996+0000","last_fullsized":"2024-09-21T09:59:21.665996+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:18.766865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":110,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665580+0000","last_change":"2024-09-21T09:40:49.096338+0000","last_active":"2024-09-21T09:59:21.665580+0000","last_peered":"2024-09-21T09:59:21.665580+0000","last_clean":"2024-09-21T09:59:21.665580+0000","last_became_active":"2024-09-21T09:40:49.096078+0000","last_became_peered":"2024-09-21T09:40:49.096078+0000","last_unstale":"2024-09-21T09:59:21.665580+0000","last_undegraded":"2024-09-21T09:59:21.665580+0000","last_fullsized":"2024-09-21T09:59:21.665580+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:56.500555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":125,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.746249+0000","last_change":"2024-09-21T09:51:30.642334+0000","last_active":"2024-09-21T09:59:22.746249+0000","last_peered":"2024-09-21T09:59:22.746249+0000","last_clean":"2024-09-21T09:59:22.746249+0000","last_became_active":"2024-09-21T09:51:30.641267+0000","last_became_peered":"2024-09-21T09:51:30.641267+0000","last_unstale":"2024-09-21T09:59:22.746249+0000","last_undegraded":"2024-09-21T09:59:22.746249+0000","last_fullsized":"2024-09-21T09:59:22.746249+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:24.201971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":107,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.843634+0000","last_change":"2024-09-21T09:56:12.580601+0000","last_active":"2024-09-21T09:59:21.843634+0000","last_peered":"2024-09-21T09:59:21.843634+0000","last_clean":"2024-09-21T09:59:21.843634+0000","last_became_active":"2024-09-21T09:56:12.580162+0000","last_became_peered":"2024-09-21T09:56:12.580162+0000","last_unstale":"2024-09-21T09:59:21.843634+0000","last_undegraded":"2024-09-21T09:59:21.843634+0000","last_fullsized":"2024-09-21T09:59:21.843634+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:49:16.381068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":105,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660053+0000","last_change":"2024-09-21T09:50:45.510379+0000","last_active":"2024-09-21T09:59:21.660053+0000","last_peered":"2024-09-21T09:59:21.660053+0000","last_clean":"2024-09-21T09:59:21.660053+0000","last_became_active":"2024-09-21T09:50:45.510082+0000","last_became_peered":"2024-09-21T09:50:45.510082+0000","last_unstale":"2024-09-21T09:59:21.660053+0000","last_undegraded":"2024-09-21T09:59:21.660053+0000","last_fullsized":"2024-09-21T09:59:21.660053+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:50.140708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":93,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.684782+0000","last_change":"2024-09-21T09:56:12.579297+0000","last_active":"2024-09-21T09:59:22.684782+0000","last_peered":"2024-09-21T09:59:22.684782+0000","last_clean":"2024-09-21T09:59:22.684782+0000","last_became_active":"2024-09-21T09:56:12.578321+0000","last_became_peered":"2024-09-21T09:56:12.578321+0000","last_unstale":"2024-09-21T09:59:22.684782+0000","last_undegraded":"2024-09-21T09:59:22.684782+0000","last_fullsized":"2024-09-21T09:59:22.684782+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:27:33.794391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":153,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672638+0000","last_change":"2024-09-21T09:56:12.574472+0000","last_active":"2024-09-21T09:59:22.672638+0000","last_peered":"2024-09-21T09:59:22.672638+0000","last_clean":"2024-09-21T09:59:22.672638+0000","last_became_active":"2024-09-21T09:56:12.573966+0000","last_became_peered":"2024-09-21T09:56:12.573966+0000","last_unstale":"2024-09-21T09:59:22.672638+0000","last_undegraded":"2024-09-21T09:59:22.672638+0000","last_fullsized":"2024-09-21T09:59:22.672638+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:37:21.415933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":103,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.687919+0000","last_change":"2024-09-21T09:56:12.576756+0000","last_active":"2024-09-21T09:59:22.687919+0000","last_peered":"2024-09-21T09:59:22.687919+0000","last_clean":"2024-09-21T09:59:22.687919+0000","last_became_active":"2024-09-21T09:56:12.575752+0000","last_became_peered":"2024-09-21T09:56:12.575752+0000","last_unstale":"2024-09-21T09:59:22.687919+0000","last_undegraded":"2024-09-21T09:59:22.687919+0000","last_fullsized":"2024-09-21T09:59:22.687919+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:55.224839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.844056+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T09:59:21.844056+0000","last_peered":"2024-09-21T09:59:21.844056+0000","last_clean":"2024-09-21T09:59:21.844056+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T09:59:21.844056+0000","last_undegraded":"2024-09-21T09:59:21.844056+0000","last_fullsized":"2024-09-21T09:59:21.844056+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":121,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.688491+0000","last_change":"2024-09-21T09:56:12.571495+0000","last_active":"2024-09-21T09:59:22.688491+0000","last_peered":"2024-09-21T09:59:22.688491+0000","last_clean":"2024-09-21T09:59:22.688491+0000","last_became_active":"2024-09-21T09:56:12.571200+0000","last_became_peered":"2024-09-21T09:56:12.571200+0000","last_unstale":"2024-09-21T09:59:22.688491+0000","last_undegraded":"2024-09-21T09:59:22.688491+0000","last_fullsized":"2024-09-21T09:59:22.688491+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:53:58.851300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":155,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.677181+0000","last_change":"2024-09-21T09:56:12.574888+0000","last_active":"2024-09-21T09:59:22.677181+0000","last_peered":"2024-09-21T09:59:22.677181+0000","last_clean":"2024-09-21T09:59:22.677181+0000","last_became_active":"2024-09-21T09:56:12.574215+0000","last_became_peered":"2024-09-21T09:56:12.574215+0000","last_unstale":"2024-09-21T09:59:22.677181+0000","last_undegraded":"2024-09-21T09:59:22.677181+0000","last_fullsized":"2024-09-21T09:59:22.677181+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:48.599320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":123,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.747399+0000","last_change":"2024-09-21T09:56:12.580276+0000","last_active":"2024-09-21T09:59:22.747399+0000","last_peered":"2024-09-21T09:59:22.747399+0000","last_clean":"2024-09-21T09:59:22.747399+0000","last_became_active":"2024-09-21T09:56:12.579989+0000","last_became_peered":"2024-09-21T09:56:12.579989+0000","last_unstale":"2024-09-21T09:59:22.747399+0000","last_undegraded":"2024-09-21T09:59:22.747399+0000","last_fullsized":"2024-09-21T09:59:22.747399+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:14:56.361873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":103,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.688111+0000","last_change":"2024-09-21T09:56:12.582383+0000","last_active":"2024-09-21T09:59:22.688111+0000","last_peered":"2024-09-21T09:59:22.688111+0000","last_clean":"2024-09-21T09:59:22.688111+0000","last_became_active":"2024-09-21T09:56:12.581427+0000","last_became_peered":"2024-09-21T09:56:12.581427+0000","last_unstale":"2024-09-21T09:59:22.688111+0000","last_undegraded":"2024-09-21T09:59:22.688111+0000","last_fullsized":"2024-09-21T09:59:22.688111+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:24.275633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":122,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665267+0000","last_change":"2024-09-21T09:51:30.645644+0000","last_active":"2024-09-21T09:59:21.665267+0000","last_peered":"2024-09-21T09:59:21.665267+0000","last_clean":"2024-09-21T09:59:21.665267+0000","last_became_active":"2024-09-21T09:51:30.644883+0000","last_became_peered":"2024-09-21T09:51:30.644883+0000","last_unstale":"2024-09-21T09:59:21.665267+0000","last_undegraded":"2024-09-21T09:59:21.665267+0000","last_fullsized":"2024-09-21T09:59:21.665267+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:39.331936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":157,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.675764+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T09:59:22.675764+0000","last_peered":"2024-09-21T09:59:22.675764+0000","last_clean":"2024-09-21T09:59:22.675764+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T09:59:22.675764+0000","last_undegraded":"2024-09-21T09:59:22.675764+0000","last_fullsized":"2024-09-21T09:59:22.675764+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":146,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672236+0000","last_change":"2024-09-21T09:48:54.130327+0000","last_active":"2024-09-21T09:59:22.672236+0000","last_peered":"2024-09-21T09:59:22.672236+0000","last_clean":"2024-09-21T09:59:22.672236+0000","last_became_active":"2024-09-21T09:48:54.128350+0000","last_became_peered":"2024-09-21T09:48:54.128350+0000","last_unstale":"2024-09-21T09:59:22.672236+0000","last_undegraded":"2024-09-21T09:59:22.672236+0000","last_fullsized":"2024-09-21T09:59:22.672236+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:31:40.490861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":120,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665946+0000","last_change":"2024-09-21T09:56:12.576148+0000","last_active":"2024-09-21T09:59:21.665946+0000","last_peered":"2024-09-21T09:59:21.665946+0000","last_clean":"2024-09-21T09:59:21.665946+0000","last_became_active":"2024-09-21T09:56:12.575813+0000","last_became_peered":"2024-09-21T09:56:12.575813+0000","last_unstale":"2024-09-21T09:59:21.665946+0000","last_undegraded":"2024-09-21T09:59:21.665946+0000","last_fullsized":"2024-09-21T09:59:21.665946+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:53:01.367816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":107,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660030+0000","last_change":"2024-09-21T09:50:45.502447+0000","last_active":"2024-09-21T09:59:21.660030+0000","last_peered":"2024-09-21T09:59:21.660030+0000","last_clean":"2024-09-21T09:59:21.660030+0000","last_became_active":"2024-09-21T09:50:45.500800+0000","last_became_peered":"2024-09-21T09:50:45.500800+0000","last_unstale":"2024-09-21T09:59:21.660030+0000","last_undegraded":"2024-09-21T09:59:21.660030+0000","last_fullsized":"2024-09-21T09:59:21.660030+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:20:51.819410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":120,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.665155+0000","last_change":"2024-09-21T09:56:12.576457+0000","last_active":"2024-09-21T09:59:21.665155+0000","last_peered":"2024-09-21T09:59:21.665155+0000","last_clean":"2024-09-21T09:59:21.665155+0000","last_became_active":"2024-09-21T09:56:12.576061+0000","last_became_peered":"2024-09-21T09:56:12.576061+0000","last_unstale":"2024-09-21T09:59:21.665155+0000","last_undegraded":"2024-09-21T09:59:21.665155+0000","last_fullsized":"2024-09-21T09:59:21.665155+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:51:46.676699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":90,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.660124+0000","last_change":"2024-09-21T09:40:49.094916+0000","last_active":"2024-09-21T09:59:21.660124+0000","last_peered":"2024-09-21T09:59:21.660124+0000","last_clean":"2024-09-21T09:59:21.660124+0000","last_became_active":"2024-09-21T09:40:49.094264+0000","last_became_peered":"2024-09-21T09:40:49.094264+0000","last_unstale":"2024-09-21T09:59:21.660124+0000","last_undegraded":"2024-09-21T09:59:21.660124+0000","last_fullsized":"2024-09-21T09:59:21.660124+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:53.663226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":129,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.685081+0000","last_change":"2024-09-21T09:56:12.577329+0000","last_active":"2024-09-21T09:59:22.685081+0000","last_peered":"2024-09-21T09:59:22.685081+0000","last_clean":"2024-09-21T09:59:22.685081+0000","last_became_active":"2024-09-21T09:56:12.576941+0000","last_became_peered":"2024-09-21T09:56:12.576941+0000","last_unstale":"2024-09-21T09:59:22.685081+0000","last_undegraded":"2024-09-21T09:59:22.685081+0000","last_fullsized":"2024-09-21T09:59:22.685081+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:35.235768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":115,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.689026+0000","last_change":"2024-09-21T09:56:12.577137+0000","last_active":"2024-09-21T09:59:22.689026+0000","last_peered":"2024-09-21T09:59:22.689026+0000","last_clean":"2024-09-21T09:59:22.689026+0000","last_became_active":"2024-09-21T09:56:12.576668+0000","last_became_peered":"2024-09-21T09:56:12.576668+0000","last_unstale":"2024-09-21T09:59:22.689026+0000","last_undegraded":"2024-09-21T09:59:22.689026+0000","last_fullsized":"2024-09-21T09:59:22.689026+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:56.772427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":103,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-21T09:59:21.664974+0000","last_change":"2024-09-21T09:50:45.502819+0000","last_active":"2024-09-21T09:59:21.664974+0000","last_peered":"2024-09-21T09:59:21.664974+0000","last_clean":"2024-09-21T09:59:21.664974+0000","last_became_active":"2024-09-21T09:50:45.502483+0000","last_became_peered":"2024-09-21T09:50:45.502483+0000","last_unstale":"2024-09-21T09:59:21.664974+0000","last_undegraded":"2024-09-21T09:59:21.664974+0000","last_fullsized":"2024-09-21T09:59:21.664974+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:26.549958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":158,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-09-21T09:59:22.672143+0000","last_change":"2024-09-21T09:51:30.650116+0000","last_active":"2024-09-21T09:59:22.672143+0000","last_peered":"2024-09-21T09:59:22.672143+0000","last_clean":"2024-09-21T09:59:22.672143+0000","last_became_active":"2024-09-21T09:51:30.649798+0000","last_became_peered":"2024-09-21T09:51:30.649798+0000","last_unstale":"2024-09-21T09:59:22.672143+0000","last_undegraded":"2024-09-21T09:59:22.672143+0000","last_fullsized":"2024-09-21T09:59:22.672143+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:40.222590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13748,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626695,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299524,"kb_used_data":2164,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93445628,"statfs":{"total":95995035648,"available":95688323072,"internally_reserved":0,"allocated":2215936,"data_stored":1518744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019153,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":587332,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93157820,"statfs":{"total":95995035648,"available":95393607680,"internally_reserved":0,"allocated":1626112,"data_stored":928433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":81,"seq":347892351115,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":406520,"kb_used_data":2152,"kb_used_omap":5,"kb_used_meta":404346,"kb_avail":93338632,"statfs":{"total":95995035648,"available":95578759168,"internally_reserved":0,"allocated":2203648,"data_stored":1514539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":414050364},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430467,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":596236,"kb_used_data":2172,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":93148916,"statfs":{"total":95995035648,"available":95384489984,"internally_reserved":0,"allocated":2224128,"data_stored":1518887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822924,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":910832,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":909246,"kb_avail":92834320,"statfs":{"total":95995035648,"available":95062343680,"internally_reserved":0,"allocated":1605632,"data_stored":923580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931068248},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879348,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":222768,"kb_used_data":1568,"kb_used_omap":4,"kb_used_meta":221179,"kb_avail":93522384,"statfs":{"total":95995035648,"available":95766921216,"internally_reserved":0,"allocated":1605632,"data_stored":923653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":226487740},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566188,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":291252,"kb_used_data":1572,"kb_used_omap":6,"kb_used_meta":289657,"kb_avail":93453900,"statfs":{"total":95995035648,"available":95696793600,"internally_reserved":0,"allocated":1609728,"data_stored":924175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":296609191},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860533,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":329964,"kb_used_data":1564,"kb_used_omap":5,"kb_used_meta":328378,"kb_avail":93415188,"statfs":{"total":95995035648,"available":95657152512,"internally_reserved":0,"allocated":1601536,"data_stored":923267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":336259311},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:00:23.161 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T10:00:23.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd dump 2024-09-21T10:00:23.404 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:23.821 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:23 smithi107 ceph-mon[102221]: pgmap v485: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:23.821 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:23 smithi107 ceph-mon[102221]: from='client.35898 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:00:23.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:23 smithi107 ceph-mon[105022]: pgmap v485: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:23.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:23 smithi107 ceph-mon[105022]: from='client.35898 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:00:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:23 smithi129 ceph-mon[88471]: pgmap v485: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:23 smithi129 ceph-mon[88471]: from='client.35898 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:00:25.071 INFO:teuthology.orchestra.run.smithi107.stdout:epoch 140 2024-09-21T10:00:25.071 INFO:teuthology.orchestra.run.smithi107.stdout:fsid 86c49514-77fc-11ef-baf6-efdc52797490 2024-09-21T10:00:25.071 INFO:teuthology.orchestra.run.smithi107.stdout:created 2024-09-21T09:34:03.635940+0000 2024-09-21T10:00:25.071 INFO:teuthology.orchestra.run.smithi107.stdout:modified 2024-09-21T09:59:22.662912+0000 2024-09-21T10:00:25.071 INFO:teuthology.orchestra.run.smithi107.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-21T10:00:25.071 INFO:teuthology.orchestra.run.smithi107.stdout:crush_version 22 2024-09-21T10:00:25.071 INFO:teuthology.orchestra.run.smithi107.stdout:full_ratio 0.95 2024-09-21T10:00:25.071 INFO:teuthology.orchestra.run.smithi107.stdout:backfillfull_ratio 0.9 2024-09-21T10:00:25.071 INFO:teuthology.orchestra.run.smithi107.stdout:nearfull_ratio 0.85 2024-09-21T10:00:25.072 INFO:teuthology.orchestra.run.smithi107.stdout:require_min_compat_client luminous 2024-09-21T10:00:25.072 INFO:teuthology.orchestra.run.smithi107.stdout:min_compat_client luminous 2024-09-21T10:00:25.072 INFO:teuthology.orchestra.run.smithi107.stdout:require_osd_release quincy 2024-09-21T10:00:25.072 INFO:teuthology.orchestra.run.smithi107.stdout:stretch_mode_enabled false 2024-09-21T10:00:25.072 INFO:teuthology.orchestra.run.smithi107.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-21T10:00:25.072 INFO:teuthology.orchestra.run.smithi107.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 57 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.00 2024-09-21T10:00:25.072 INFO:teuthology.orchestra.run.smithi107.stdout:pool 3 'cephfs.foo.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 62 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.50 2024-09-21T10:00:25.072 INFO:teuthology.orchestra.run.smithi107.stdout:pool 4 'cephfs.foo.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 63 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 2.25 2024-09-21T10:00:25.072 INFO:teuthology.orchestra.run.smithi107.stdout:max_osd 8 2024-09-21T10:00:25.072 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 up in weight 1 up_from 81 up_thru 129 down_at 78 last_clean_interval [8,77) [v2:172.21.15.107:6802/1236110780,v1:172.21.15.107:6803/1236110780] [v2:172.21.15.107:6804/1236110780,v1:172.21.15.107:6805/1236110780] exists,up 635c1eb0-29b1-4fdd-bd47-881ef05e49bd 2024-09-21T10:00:25.072 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1 up in weight 1 up_from 96 up_thru 138 down_at 92 last_clean_interval [13,91) [v2:172.21.15.107:6810/2988029448,v1:172.21.15.107:6811/2988029448] [v2:172.21.15.107:6812/2988029448,v1:172.21.15.107:6813/2988029448] exists,up a39b6a18-939d-4999-89b2-1126c1b2d1f0 2024-09-21T10:00:25.072 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2 up in weight 1 up_from 105 up_thru 126 down_at 102 last_clean_interval [18,101) [v2:172.21.15.107:6818/1583228557,v1:172.21.15.107:6819/1583228557] [v2:172.21.15.107:6820/1583228557,v1:172.21.15.107:6821/1583228557] exists,up b01c5f1b-77e4-462f-80d0-fd1d3afb786b 2024-09-21T10:00:25.072 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3 up in weight 1 up_from 126 up_thru 139 down_at 123 last_clean_interval [25,122) [v2:172.21.15.107:6826/860985313,v1:172.21.15.107:6827/860985313] [v2:172.21.15.107:6828/860985313,v1:172.21.15.107:6829/860985313] exists,up 5135d269-b6c4-4a9d-bcf9-ffe19bc5c9ad 2024-09-21T10:00:25.072 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4 up in weight 1 up_from 30 up_thru 126 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6800/4101249335,v1:172.21.15.129:6801/4101249335] [v2:172.21.15.129:6802/4101249335,v1:172.21.15.129:6803/4101249335] exists,up 23242972-e341-4773-809f-299aabeacd56 2024-09-21T10:00:25.073 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5 up in weight 1 up_from 36 up_thru 133 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6808/2733195233,v1:172.21.15.129:6809/2733195233] [v2:172.21.15.129:6810/2733195233,v1:172.21.15.129:6811/2733195233] exists,up f9da25cd-1680-4d97-a9d1-98dded10766f 2024-09-21T10:00:25.073 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6 up in weight 1 up_from 42 up_thru 123 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6816/3400496608,v1:172.21.15.129:6817/3400496608] [v2:172.21.15.129:6818/3400496608,v1:172.21.15.129:6819/3400496608] exists,up d6d2bd99-0178-409c-a548-cbc30b8b76fb 2024-09-21T10:00:25.073 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7 up in weight 1 up_from 48 up_thru 126 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6824/101668996,v1:172.21.15.129:6825/101668996] [v2:172.21.15.129:6826/101668996,v1:172.21.15.129:6827/101668996] exists,up 2c0a9723-0328-4134-b5d4-cf307839018a 2024-09-21T10:00:25.073 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 4.7 [1,2] 2024-09-21T10:00:25.073 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 4.d [1,5] 2024-09-21T10:00:25.073 INFO:teuthology.orchestra.run.smithi107.stdout:pg_upmap_items 4.13 [1,2] 2024-09-21T10:00:25.073 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/3807929340 expires 2024-09-22T09:46:25.797234+0000 2024-09-21T10:00:25.073 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6800/1580746173 expires 2024-09-22T09:46:25.797234+0000 2024-09-21T10:00:25.073 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/3989069453 expires 2024-09-22T09:46:25.797234+0000 2024-09-21T10:00:25.073 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/4262440050 expires 2024-09-22T09:46:25.797234+0000 2024-09-21T10:00:25.073 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/1261754960 expires 2024-09-22T09:46:25.797234+0000 2024-09-21T10:00:25.073 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/1942117255 expires 2024-09-22T09:34:56.643317+0000 2024-09-21T10:00:25.073 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/627572584 expires 2024-09-22T09:46:25.797234+0000 2024-09-21T10:00:25.073 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/51270168 expires 2024-09-22T09:43:44.074024+0000 2024-09-21T10:00:25.074 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/3025710247 expires 2024-09-22T09:43:44.074024+0000 2024-09-21T10:00:25.074 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6801/4208383830 expires 2024-09-22T09:34:56.643317+0000 2024-09-21T10:00:25.074 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.129:0/306463122 expires 2024-09-22T09:45:31.607526+0000 2024-09-21T10:00:25.074 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6800/4208383830 expires 2024-09-22T09:34:56.643317+0000 2024-09-21T10:00:25.074 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6800/4055614859 expires 2024-09-22T09:43:44.074024+0000 2024-09-21T10:00:25.074 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/4103644615 expires 2024-09-22T09:39:30.975484+0000 2024-09-21T10:00:25.074 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6801/1580746173 expires 2024-09-22T09:46:25.797234+0000 2024-09-21T10:00:25.074 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6801/2716631102 expires 2024-09-22T09:39:30.975484+0000 2024-09-21T10:00:25.074 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6800/2716631102 expires 2024-09-22T09:39:30.975484+0000 2024-09-21T10:00:25.074 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/637981220 expires 2024-09-22T09:34:34.697433+0000 2024-09-21T10:00:25.074 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6801/1686228827 expires 2024-09-22T09:34:34.697433+0000 2024-09-21T10:00:25.074 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.129:6835/2445466552 expires 2024-09-22T09:45:31.607526+0000 2024-09-21T10:00:25.074 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.129:6834/2445466552 expires 2024-09-22T09:45:31.607526+0000 2024-09-21T10:00:25.074 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/2526044436 expires 2024-09-22T09:39:30.975484+0000 2024-09-21T10:00:25.075 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.129:0/3127898221 expires 2024-09-22T09:45:31.607526+0000 2024-09-21T10:00:25.075 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6800/1686228827 expires 2024-09-22T09:34:34.697433+0000 2024-09-21T10:00:25.075 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.129:0/4112718236 expires 2024-09-22T09:45:31.607526+0000 2024-09-21T10:00:25.075 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/2156530655 expires 2024-09-22T09:39:30.975484+0000 2024-09-21T10:00:25.075 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:6801/4055614859 expires 2024-09-22T09:43:44.074024+0000 2024-09-21T10:00:25.075 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.129:0/2239836304 expires 2024-09-22T09:45:31.607526+0000 2024-09-21T10:00:25.075 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/1526680749 expires 2024-09-22T09:43:44.074024+0000 2024-09-21T10:00:25.075 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/837896416 expires 2024-09-22T09:34:34.697433+0000 2024-09-21T10:00:25.075 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/3424783814 expires 2024-09-22T09:34:56.643317+0000 2024-09-21T10:00:25.075 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/2701254891 expires 2024-09-22T09:43:44.074024+0000 2024-09-21T10:00:25.075 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/782384657 expires 2024-09-22T09:34:56.643317+0000 2024-09-21T10:00:25.075 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/2492859118 expires 2024-09-22T09:34:34.697433+0000 2024-09-21T10:00:25.075 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/218684017 expires 2024-09-22T09:39:30.975484+0000 2024-09-21T10:00:25.075 INFO:teuthology.orchestra.run.smithi107.stdout:blocklist 172.21.15.107:0/2945318176 expires 2024-09-22T09:46:25.797234+0000 2024-09-21T10:00:25.723 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 81 up_thru 129 down_at 78 last_clean_interval [8,77) [v2:172.21.15.107:6802/1236110780,v1:172.21.15.107:6803/1236110780] [v2:172.21.15.107:6804/1236110780,v1:172.21.15.107:6805/1236110780] exists,up 635c1eb0-29b1-4fdd-bd47-881ef05e49bd', 'osd.1 up in weight 1 up_from 96 up_thru 138 down_at 92 last_clean_interval [13,91) [v2:172.21.15.107:6810/2988029448,v1:172.21.15.107:6811/2988029448] [v2:172.21.15.107:6812/2988029448,v1:172.21.15.107:6813/2988029448] exists,up a39b6a18-939d-4999-89b2-1126c1b2d1f0', 'osd.2 up in weight 1 up_from 105 up_thru 126 down_at 102 last_clean_interval [18,101) [v2:172.21.15.107:6818/1583228557,v1:172.21.15.107:6819/1583228557] [v2:172.21.15.107:6820/1583228557,v1:172.21.15.107:6821/1583228557] exists,up b01c5f1b-77e4-462f-80d0-fd1d3afb786b', 'osd.3 up in weight 1 up_from 126 up_thru 139 down_at 123 last_clean_interval [25,122) [v2:172.21.15.107:6826/860985313,v1:172.21.15.107:6827/860985313] [v2:172.21.15.107:6828/860985313,v1:172.21.15.107:6829/860985313] exists,up 5135d269-b6c4-4a9d-bcf9-ffe19bc5c9ad', 'osd.4 up in weight 1 up_from 30 up_thru 126 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6800/4101249335,v1:172.21.15.129:6801/4101249335] [v2:172.21.15.129:6802/4101249335,v1:172.21.15.129:6803/4101249335] exists,up 23242972-e341-4773-809f-299aabeacd56', 'osd.5 up in weight 1 up_from 36 up_thru 133 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6808/2733195233,v1:172.21.15.129:6809/2733195233] [v2:172.21.15.129:6810/2733195233,v1:172.21.15.129:6811/2733195233] exists,up f9da25cd-1680-4d97-a9d1-98dded10766f', 'osd.6 up in weight 1 up_from 42 up_thru 123 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6816/3400496608,v1:172.21.15.129:6817/3400496608] [v2:172.21.15.129:6818/3400496608,v1:172.21.15.129:6819/3400496608] exists,up d6d2bd99-0178-409c-a548-cbc30b8b76fb', 'osd.7 up in weight 1 up_from 48 up_thru 126 down_at 0 last_clean_interval [0,0) [v2:172.21.15.129:6824/101668996,v1:172.21.15.129:6825/101668996] [v2:172.21.15.129:6826/101668996,v1:172.21.15.129:6827/101668996] exists,up 2c0a9723-0328-4134-b5d4-cf307839018a'] 2024-09-21T10:00:25.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon mon.a config show 2024-09-21T10:00:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:25 smithi107 ceph-mon[102221]: pgmap v486: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:25 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4216919942' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T10:00:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:25 smithi107 ceph-mon[105022]: pgmap v486: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:25 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4216919942' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T10:00:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:25 smithi129 ceph-mon[88471]: pgmap v486: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:25 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4216919942' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T10:00:25.977 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:26.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:27.575 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:00:27.575 INFO:teuthology.orchestra.run.smithi107.stdout: "name": "mon.a", 2024-09-21T10:00:27.575 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster": "ceph", 2024-09-21T10:00:27.575 INFO:teuthology.orchestra.run.smithi107.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-21T10:00:27.575 INFO:teuthology.orchestra.run.smithi107.stdout: "admin_socket_mode": "", 2024-09-21T10:00:27.575 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-21T10:00:27.575 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_client_required": "cephx, none", 2024-09-21T10:00:27.575 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_cluster_required": "cephx", 2024-09-21T10:00:27.575 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_debug": "true", 2024-09-21T10:00:27.576 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-21T10:00:27.576 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-21T10:00:27.576 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_service_required": "cephx", 2024-09-21T10:00:27.576 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-21T10:00:27.576 INFO:teuthology.orchestra.run.smithi107.stdout: "auth_supported": "", 2024-09-21T10:00:27.576 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_aio": "true", 2024-09-21T10:00:27.576 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-21T10:00:27.576 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_aio_poll_ms": "250", 2024-09-21T10:00:27.576 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_aio_reap_max": "16", 2024-09-21T10:00:27.576 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_async_discard_threads": "0", 2024-09-21T10:00:27.576 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_block_size": "4096", 2024-09-21T10:00:27.576 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_debug_aio": "false", 2024-09-21T10:00:27.576 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-21T10:00:27.576 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-21T10:00:27.577 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_debug_inflight_ios": "false", 2024-09-21T10:00:27.577 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_enable_discard": "false", 2024-09-21T10:00:27.577 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_flock_retry": "3", 2024-09-21T10:00:27.577 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-21T10:00:27.577 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_inject_crash": "0", 2024-09-21T10:00:27.577 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-21T10:00:27.577 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_ioring": "false", 2024-09-21T10:00:27.577 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_ioring_hipri": "false", 2024-09-21T10:00:27.577 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-21T10:00:27.577 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-21T10:00:27.577 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-21T10:00:27.577 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-21T10:00:27.577 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-21T10:00:27.577 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-21T10:00:27.578 INFO:teuthology.orchestra.run.smithi107.stdout: "bdev_type": "", 2024-09-21T10:00:27.578 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_alloc_size": "1048576", 2024-09-21T10:00:27.578 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_allocator": "hybrid", 2024-09-21T10:00:27.578 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_buffered_io": "true", 2024-09-21T10:00:27.578 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_check_for_zeros": "false", 2024-09-21T10:00:27.578 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-21T10:00:27.578 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-21T10:00:27.578 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_compact_log_sync": "false", 2024-09-21T10:00:27.578 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-21T10:00:27.578 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-21T10:00:27.578 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-21T10:00:27.578 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-21T10:00:27.578 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_max_log_runway": "4194304", 2024-09-21T10:00:27.578 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_max_prefetch": "1048576", 2024-09-21T10:00:27.579 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_min_flush_size": "524288", 2024-09-21T10:00:27.579 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_min_log_runway": "1048576", 2024-09-21T10:00:27.579 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_replay_recovery": "false", 2024-09-21T10:00:27.579 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-21T10:00:27.579 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-21T10:00:27.579 INFO:teuthology.orchestra.run.smithi107.stdout: "bluefs_sync_write": "false", 2024-09-21T10:00:27.579 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-21T10:00:27.579 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-21T10:00:27.579 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-21T10:00:27.579 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_allocation_from_file": "true", 2024-09-21T10:00:27.579 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_allocator": "hybrid", 2024-09-21T10:00:27.579 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_async_db_compaction": "true", 2024-09-21T10:00:27.579 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-21T10:00:27.579 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-21T10:00:27.579 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-21T10:00:27.580 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-21T10:00:27.580 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bdev_label_multi": "true", 2024-09-21T10:00:27.580 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-21T10:00:27.580 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-21T10:00:27.580 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-21T10:00:27.580 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-21T10:00:27.580 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-21T10:00:27.580 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_create": "true", 2024-09-21T10:00:27.580 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_db_create": "false", 2024-09-21T10:00:27.580 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_db_path": "", 2024-09-21T10:00:27.580 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_db_size": "0", 2024-09-21T10:00:27.580 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_path": "", 2024-09-21T10:00:27.580 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_preallocate_file": "false", 2024-09-21T10:00:27.580 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_size": "107374182400", 2024-09-21T10:00:27.581 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_wal_create": "false", 2024-09-21T10:00:27.581 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_wal_path": "", 2024-09-21T10:00:27.581 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_block_wal_size": "100663296", 2024-09-21T10:00:27.581 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs": "true", 2024-09-21T10:00:27.581 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-21T10:00:27.581 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-21T10:00:27.581 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-21T10:00:27.581 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-21T10:00:27.581 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-21T10:00:27.581 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T10:00:27.581 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T10:00:27.581 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-21T10:00:27.581 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T10:00:27.581 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_autotune": "true", 2024-09-21T10:00:27.581 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-21T10:00:27.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-21T10:00:27.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-21T10:00:27.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-21T10:00:27.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_size": "0", 2024-09-21T10:00:27.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-21T10:00:27.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-21T10:00:27.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-21T10:00:27.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-21T10:00:27.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cache_type": "2q", 2024-09-21T10:00:27.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-21T10:00:27.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_clone_cow": "true", 2024-09-21T10:00:27.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-21T10:00:27.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-21T10:00:27.582 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-21T10:00:27.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-21T10:00:27.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-21T10:00:27.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-21T10:00:27.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-21T10:00:27.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_mode": "none", 2024-09-21T10:00:27.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-21T10:00:27.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_csum_type": "crc32c", 2024-09-21T10:00:27.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-21T10:00:27.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-21T10:00:27.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_freelist": "false", 2024-09-21T10:00:27.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-21T10:00:27.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-21T10:00:27.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-21T10:00:27.583 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-21T10:00:27.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-21T10:00:27.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-21T10:00:27.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-21T10:00:27.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-21T10:00:27.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-21T10:00:27.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-21T10:00:27.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-21T10:00:27.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-21T10:00:27.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_small_allocations": "0", 2024-09-21T10:00:27.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-21T10:00:27.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_default_buffered_read": "true", 2024-09-21T10:00:27.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_default_buffered_write": "false", 2024-09-21T10:00:27.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-21T10:00:27.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-21T10:00:27.584 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-21T10:00:27.585 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-21T10:00:27.585 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-21T10:00:27.585 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-21T10:00:27.585 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-21T10:00:27.585 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-21T10:00:27.585 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-21T10:00:27.585 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fail_eio": "false", 2024-09-21T10:00:27.585 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-21T10:00:27.585 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-21T10:00:27.585 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-21T10:00:27.585 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-21T10:00:27.585 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-21T10:00:27.585 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-21T10:00:27.586 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_mount": "false", 2024-09-21T10:00:27.586 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-21T10:00:27.586 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_umount": "false", 2024-09-21T10:00:27.586 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-21T10:00:27.586 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-21T10:00:27.586 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-21T10:00:27.586 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-21T10:00:27.586 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-21T10:00:27.586 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-21T10:00:27.586 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-21T10:00:27.586 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-21T10:00:27.586 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_ignore_data_csum": "false", 2024-09-21T10:00:27.586 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-21T10:00:27.586 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-21T10:00:27.587 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-21T10:00:27.587 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-21T10:00:27.587 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_log_op_age": "5.000000", 2024-09-21T10:00:27.587 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_alloc_size": "0", 2024-09-21T10:00:27.587 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_blob_size": "0", 2024-09-21T10:00:27.587 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-21T10:00:27.587 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-21T10:00:27.587 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-21T10:00:27.587 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_max_deferred_txc": "32", 2024-09-21T10:00:27.587 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_min_alloc_size": "0", 2024-09-21T10:00:27.587 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-21T10:00:27.587 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-21T10:00:27.587 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_nid_prealloc": "1024", 2024-09-21T10:00:27.587 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-21T10:00:27.588 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-21T10:00:27.588 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-21T10:00:27.588 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-21T10:00:27.588 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_retry_disk_reads": "3", 2024-09-21T10:00:27.588 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_rocksdb_cf": "true", 2024-09-21T10:00:27.588 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-21T10:00:27.589 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-21T10:00:27.589 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-21T10:00:27.589 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-21T10:00:27.589 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-21T10:00:27.589 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-21T10:00:27.589 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-21T10:00:27.589 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-21T10:00:27.589 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_spdk_mem": "512", 2024-09-21T10:00:27.589 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-21T10:00:27.589 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-21T10:00:27.589 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-21T10:00:27.589 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-21T10:00:27.589 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-21T10:00:27.589 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-21T10:00:27.590 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-21T10:00:27.590 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_tracing": "false", 2024-09-21T10:00:27.590 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-21T10:00:27.590 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-21T10:00:27.590 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-21T10:00:27.590 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-21T10:00:27.590 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-21T10:00:27.590 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-21T10:00:27.590 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-21T10:00:27.590 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-21T10:00:27.590 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-21T10:00:27.590 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_write_v2": "false", 2024-09-21T10:00:27.590 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_write_v2_random": "false", 2024-09-21T10:00:27.590 INFO:teuthology.orchestra.run.smithi107.stdout: "bluestore_zero_block_detection": "false", 2024-09-21T10:00:27.590 INFO:teuthology.orchestra.run.smithi107.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-21T10:00:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-21T10:00:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-21T10:00:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-21T10:00:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-21T10:00:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-21T10:00:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-21T10:00:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-21T10:00:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-21T10:00:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-21T10:00:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-21T10:00:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-21T10:00:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-21T10:00:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-21T10:00:27.591 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_cluster_require_signatures": "false", 2024-09-21T10:00:27.592 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_cluster_require_version": "2", 2024-09-21T10:00:27.592 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_require_signatures": "false", 2024-09-21T10:00:27.592 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_require_version": "2", 2024-09-21T10:00:27.592 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_service_require_signatures": "false", 2024-09-21T10:00:27.592 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_service_require_version": "2", 2024-09-21T10:00:27.592 INFO:teuthology.orchestra.run.smithi107.stdout: "cephx_sign_messages": "true", 2024-09-21T10:00:27.592 INFO:teuthology.orchestra.run.smithi107.stdout: "chdir": "", 2024-09-21T10:00:27.592 INFO:teuthology.orchestra.run.smithi107.stdout: "client_acl_type": "", 2024-09-21T10:00:27.592 INFO:teuthology.orchestra.run.smithi107.stdout: "client_asio_thread_count": "2", 2024-09-21T10:00:27.592 INFO:teuthology.orchestra.run.smithi107.stdout: "client_cache_mid": "0.750000", 2024-09-21T10:00:27.592 INFO:teuthology.orchestra.run.smithi107.stdout: "client_cache_size": "16384", 2024-09-21T10:00:27.592 INFO:teuthology.orchestra.run.smithi107.stdout: "client_caps_release_delay": "5", 2024-09-21T10:00:27.592 INFO:teuthology.orchestra.run.smithi107.stdout: "client_check_pool_perm": "true", 2024-09-21T10:00:27.592 INFO:teuthology.orchestra.run.smithi107.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-21T10:00:27.592 INFO:teuthology.orchestra.run.smithi107.stdout: "client_debug_force_sync_read": "false", 2024-09-21T10:00:27.593 INFO:teuthology.orchestra.run.smithi107.stdout: "client_debug_getattr_caps": "false", 2024-09-21T10:00:27.593 INFO:teuthology.orchestra.run.smithi107.stdout: "client_debug_inject_features": "", 2024-09-21T10:00:27.593 INFO:teuthology.orchestra.run.smithi107.stdout: "client_debug_inject_tick_delay": "0", 2024-09-21T10:00:27.593 INFO:teuthology.orchestra.run.smithi107.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-21T10:00:27.593 INFO:teuthology.orchestra.run.smithi107.stdout: "client_die_on_failed_remount": "false", 2024-09-21T10:00:27.593 INFO:teuthology.orchestra.run.smithi107.stdout: "client_dirsize_rbytes": "true", 2024-09-21T10:00:27.593 INFO:teuthology.orchestra.run.smithi107.stdout: "client_force_lazyio": "false", 2024-09-21T10:00:27.593 INFO:teuthology.orchestra.run.smithi107.stdout: "client_fs": "", 2024-09-21T10:00:27.593 INFO:teuthology.orchestra.run.smithi107.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-21T10:00:27.593 INFO:teuthology.orchestra.run.smithi107.stdout: "client_inject_release_failure": "false", 2024-09-21T10:00:27.593 INFO:teuthology.orchestra.run.smithi107.stdout: "client_max_inline_size": "4096", 2024-09-21T10:00:27.593 INFO:teuthology.orchestra.run.smithi107.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-21T10:00:27.593 INFO:teuthology.orchestra.run.smithi107.stdout: "client_mds_namespace": "", 2024-09-21T10:00:27.593 INFO:teuthology.orchestra.run.smithi107.stdout: "client_metadata": "", 2024-09-21T10:00:27.593 INFO:teuthology.orchestra.run.smithi107.stdout: "client_mount_gid": "-1", 2024-09-21T10:00:27.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client_mount_timeout": "300", 2024-09-21T10:00:27.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client_mount_uid": "-1", 2024-09-21T10:00:27.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client_mountpoint": "/", 2024-09-21T10:00:27.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client_notify_timeout": "10", 2024-09-21T10:00:27.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc": "true", 2024-09-21T10:00:27.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc_max_dirty": "104857600", 2024-09-21T10:00:27.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-21T10:00:27.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc_max_objects": "1000", 2024-09-21T10:00:27.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc_size": "209715200", 2024-09-21T10:00:27.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client_oc_target_dirty": "8388608", 2024-09-21T10:00:27.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client_permissions": "true", 2024-09-21T10:00:27.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client_quota": "true", 2024-09-21T10:00:27.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client_quota_df": "true", 2024-09-21T10:00:27.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client_readahead_max_bytes": "0", 2024-09-21T10:00:27.595 INFO:teuthology.orchestra.run.smithi107.stdout: "client_readahead_max_periods": "4", 2024-09-21T10:00:27.595 INFO:teuthology.orchestra.run.smithi107.stdout: "client_readahead_min": "131072", 2024-09-21T10:00:27.595 INFO:teuthology.orchestra.run.smithi107.stdout: "client_reconnect_stale": "false", 2024-09-21T10:00:27.595 INFO:teuthology.orchestra.run.smithi107.stdout: "client_shutdown_timeout": "30", 2024-09-21T10:00:27.595 INFO:teuthology.orchestra.run.smithi107.stdout: "client_snapdir": ".snap", 2024-09-21T10:00:27.595 INFO:teuthology.orchestra.run.smithi107.stdout: "client_tick_interval": "1", 2024-09-21T10:00:27.595 INFO:teuthology.orchestra.run.smithi107.stdout: "client_trace": "", 2024-09-21T10:00:27.595 INFO:teuthology.orchestra.run.smithi107.stdout: "client_try_dentry_invalidate": "false", 2024-09-21T10:00:27.595 INFO:teuthology.orchestra.run.smithi107.stdout: "client_use_faked_inos": "false", 2024-09-21T10:00:27.595 INFO:teuthology.orchestra.run.smithi107.stdout: "client_use_random_mds": "false", 2024-09-21T10:00:27.595 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_graylog": "false", 2024-09-21T10:00:27.595 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-21T10:00:27.595 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_graylog_port": "12201", 2024-09-21T10:00:27.595 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_monitors": "default=true", 2024-09-21T10:00:27.595 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_syslog": "false", 2024-09-21T10:00:27.596 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-21T10:00:27.596 INFO:teuthology.orchestra.run.smithi107.stdout: "clog_to_syslog_level": "info", 2024-09-21T10:00:27.596 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_addr": "-", 2024-09-21T10:00:27.596 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_network": "", 2024-09-21T10:00:27.596 INFO:teuthology.orchestra.run.smithi107.stdout: "cluster_network_interface": "", 2024-09-21T10:00:27.596 INFO:teuthology.orchestra.run.smithi107.stdout: "compressor_zlib_isal": "false", 2024-09-21T10:00:27.596 INFO:teuthology.orchestra.run.smithi107.stdout: "compressor_zlib_level": "5", 2024-09-21T10:00:27.596 INFO:teuthology.orchestra.run.smithi107.stdout: "compressor_zlib_winsize": "-15", 2024-09-21T10:00:27.596 INFO:teuthology.orchestra.run.smithi107.stdout: "compressor_zstd_level": "1", 2024-09-21T10:00:27.596 INFO:teuthology.orchestra.run.smithi107.stdout: "container_image": "quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47", 2024-09-21T10:00:27.596 INFO:teuthology.orchestra.run.smithi107.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-21T10:00:27.596 INFO:teuthology.orchestra.run.smithi107.stdout: "crimson_alien_op_num_threads": "6", 2024-09-21T10:00:27.596 INFO:teuthology.orchestra.run.smithi107.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-21T10:00:27.596 INFO:teuthology.orchestra.run.smithi107.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-21T10:00:27.597 INFO:teuthology.orchestra.run.smithi107.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-21T10:00:27.597 INFO:teuthology.orchestra.run.smithi107.stdout: "crimson_osd_stat_interval": "0", 2024-09-21T10:00:27.597 INFO:teuthology.orchestra.run.smithi107.stdout: "crimson_seastar_cpu_cores": "", 2024-09-21T10:00:27.597 INFO:teuthology.orchestra.run.smithi107.stdout: "crimson_seastar_num_threads": "0", 2024-09-21T10:00:27.597 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_location": "", 2024-09-21T10:00:27.597 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_location_hook": "", 2024-09-21T10:00:27.597 INFO:teuthology.orchestra.run.smithi107.stdout: "crush_location_hook_timeout": "10", 2024-09-21T10:00:27.597 INFO:teuthology.orchestra.run.smithi107.stdout: "daemonize": "false", 2024-09-21T10:00:27.597 INFO:teuthology.orchestra.run.smithi107.stdout: "daos_pool": "tank", 2024-09-21T10:00:27.597 INFO:teuthology.orchestra.run.smithi107.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-21T10:00:27.597 INFO:teuthology.orchestra.run.smithi107.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-21T10:00:27.597 INFO:teuthology.orchestra.run.smithi107.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-21T10:00:27.597 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_alienstore": "0/5", 2024-09-21T10:00:27.597 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_allow_any_pool_priority": "false", 2024-09-21T10:00:27.597 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_asok": "1/5", 2024-09-21T10:00:27.598 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_asok_assert_abort": "false", 2024-09-21T10:00:27.598 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_asserts_on_shutdown": "true", 2024-09-21T10:00:27.598 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_auth": "1/5", 2024-09-21T10:00:27.598 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_bdev": "1/3", 2024-09-21T10:00:27.598 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_bluefs": "1/5", 2024-09-21T10:00:27.598 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_bluestore": "1/5", 2024-09-21T10:00:27.598 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_buffer": "0/1", 2024-09-21T10:00:27.598 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_ceph_dedup": "0/5", 2024-09-21T10:00:27.598 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_ceph_exporter": "1/5", 2024-09-21T10:00:27.598 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_cephfs_mirror": "0/5", 2024-09-21T10:00:27.598 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_cephsqlite": "0/5", 2024-09-21T10:00:27.598 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_client": "0/5", 2024-09-21T10:00:27.598 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_compressor": "1/5", 2024-09-21T10:00:27.598 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_context": "0/1", 2024-09-21T10:00:27.598 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_crimson_interrupt": "0/5", 2024-09-21T10:00:27.599 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_crush": "1/1", 2024-09-21T10:00:27.599 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_crypto": "1/5", 2024-09-21T10:00:27.599 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_cyanstore": "0/5", 2024-09-21T10:00:27.599 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_deliberately_leak_memory": "false", 2024-09-21T10:00:27.599 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_disable_randomized_ping": "false", 2024-09-21T10:00:27.599 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_dpdk": "1/5", 2024-09-21T10:00:27.599 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_eventtrace": "1/5", 2024-09-21T10:00:27.599 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_filer": "0/1", 2024-09-21T10:00:27.599 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_filestore": "1/3", 2024-09-21T10:00:27.599 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_finisher": "1/1", 2024-09-21T10:00:27.599 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_fuse": "1/5", 2024-09-21T10:00:27.599 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_heartbeat_testing_span": "0", 2024-09-21T10:00:27.599 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_heartbeatmap": "1/5", 2024-09-21T10:00:27.599 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-21T10:00:27.599 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_javaclient": "1/5", 2024-09-21T10:00:27.600 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_journal": "1/3", 2024-09-21T10:00:27.600 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_journaler": "0/5", 2024-09-21T10:00:27.600 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_kstore": "1/5", 2024-09-21T10:00:27.600 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_lockdep": "0/1", 2024-09-21T10:00:27.600 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mclock": "1/5", 2024-09-21T10:00:27.600 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds": "1/5", 2024-09-21T10:00:27.600 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_balancer": "1/5", 2024-09-21T10:00:27.600 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_locker": "1/5", 2024-09-21T10:00:27.600 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_log": "1/5", 2024-09-21T10:00:27.600 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_log_expire": "1/5", 2024-09-21T10:00:27.600 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_migrator": "1/5", 2024-09-21T10:00:27.600 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mds_quiesce": "3/5", 2024-09-21T10:00:27.600 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_memstore": "1/5", 2024-09-21T10:00:27.600 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mgr": "2/5", 2024-09-21T10:00:27.600 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mgrc": "1/5", 2024-09-21T10:00:27.601 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_mon": "20/20", 2024-09-21T10:00:27.601 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_monc": "0/10", 2024-09-21T10:00:27.601 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_ms": "1/1", 2024-09-21T10:00:27.601 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_none": "0/5", 2024-09-21T10:00:27.601 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_objclass": "0/5", 2024-09-21T10:00:27.601 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_objectcacher": "0/5", 2024-09-21T10:00:27.601 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_objecter": "0/1", 2024-09-21T10:00:27.601 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_optracker": "0/5", 2024-09-21T10:00:27.601 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_osd": "1/5", 2024-09-21T10:00:27.601 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_paxos": "20/20", 2024-09-21T10:00:27.601 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_perfcounter": "1/5", 2024-09-21T10:00:27.601 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_prioritycache": "1/5", 2024-09-21T10:00:27.601 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rados": "0/5", 2024-09-21T10:00:27.601 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rbd": "0/5", 2024-09-21T10:00:27.601 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rbd_mirror": "0/5", 2024-09-21T10:00:27.602 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rbd_pwl": "0/5", 2024-09-21T10:00:27.602 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rbd_replay": "0/5", 2024-09-21T10:00:27.602 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_refs": "0/0", 2024-09-21T10:00:27.602 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_reserver": "1/1", 2024-09-21T10:00:27.602 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rgw": "1/5", 2024-09-21T10:00:27.602 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rgw_access": "1/5", 2024-09-21T10:00:27.602 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rgw_datacache": "1/5", 2024-09-21T10:00:27.602 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rgw_dbstore": "1/5", 2024-09-21T10:00:27.602 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rgw_flight": "1/5", 2024-09-21T10:00:27.602 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-21T10:00:27.602 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rgw_notification": "1/5", 2024-09-21T10:00:27.602 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rgw_sync": "1/5", 2024-09-21T10:00:27.602 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_rocksdb": "4/5", 2024-09-21T10:00:27.602 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore": "0/5", 2024-09-21T10:00:27.603 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_backref": "0/5", 2024-09-21T10:00:27.603 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_cache": "0/5", 2024-09-21T10:00:27.603 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_cleaner": "0/5", 2024-09-21T10:00:27.603 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_device": "0/5", 2024-09-21T10:00:27.603 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_epm": "0/5", 2024-09-21T10:00:27.603 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-21T10:00:27.603 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_journal": "0/5", 2024-09-21T10:00:27.603 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_lba": "0/5", 2024-09-21T10:00:27.603 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_odata": "0/5", 2024-09-21T10:00:27.603 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_omap": "0/5", 2024-09-21T10:00:27.603 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_onode": "0/5", 2024-09-21T10:00:27.603 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_t": "0/5", 2024-09-21T10:00:27.603 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_seastore_tm": "0/5", 2024-09-21T10:00:27.603 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_striper": "0/1", 2024-09-21T10:00:27.604 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_test": "0/5", 2024-09-21T10:00:27.604 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_throttle": "1/1", 2024-09-21T10:00:27.604 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_timer": "0/1", 2024-09-21T10:00:27.604 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_tp": "0/5", 2024-09-21T10:00:27.604 INFO:teuthology.orchestra.run.smithi107.stdout: "debug_trace": "1/5", 2024-09-21T10:00:27.604 INFO:teuthology.orchestra.run.smithi107.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-21T10:00:27.604 INFO:teuthology.orchestra.run.smithi107.stdout: "device_failure_prediction_mode": "none", 2024-09-21T10:00:27.604 INFO:teuthology.orchestra.run.smithi107.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-21T10:00:27.604 INFO:teuthology.orchestra.run.smithi107.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-21T10:00:27.604 INFO:teuthology.orchestra.run.smithi107.stdout: "err_to_graylog": "false", 2024-09-21T10:00:27.604 INFO:teuthology.orchestra.run.smithi107.stdout: "err_to_journald": "false", 2024-09-21T10:00:27.604 INFO:teuthology.orchestra.run.smithi107.stdout: "err_to_stderr": "true", 2024-09-21T10:00:27.604 INFO:teuthology.orchestra.run.smithi107.stdout: "err_to_syslog": "false", 2024-09-21T10:00:27.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event_tracing": "false", 2024-09-21T10:00:27.605 INFO:teuthology.orchestra.run.smithi107.stdout: "exporter_addr": "0.0.0.0", 2024-09-21T10:00:27.605 INFO:teuthology.orchestra.run.smithi107.stdout: "exporter_cert_file": "", 2024-09-21T10:00:27.605 INFO:teuthology.orchestra.run.smithi107.stdout: "exporter_http_port": "9926", 2024-09-21T10:00:27.605 INFO:teuthology.orchestra.run.smithi107.stdout: "exporter_key_file": "", 2024-09-21T10:00:27.605 INFO:teuthology.orchestra.run.smithi107.stdout: "exporter_prio_limit": "5", 2024-09-21T10:00:27.605 INFO:teuthology.orchestra.run.smithi107.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-21T10:00:27.605 INFO:teuthology.orchestra.run.smithi107.stdout: "exporter_sort_metrics": "true", 2024-09-21T10:00:27.605 INFO:teuthology.orchestra.run.smithi107.stdout: "exporter_stats_period": "5", 2024-09-21T10:00:27.605 INFO:teuthology.orchestra.run.smithi107.stdout: "fake_statfs_for_testing": "0", 2024-09-21T10:00:27.605 INFO:teuthology.orchestra.run.smithi107.stdout: "fatal_signal_handlers": "true", 2024-09-21T10:00:27.605 INFO:teuthology.orchestra.run.smithi107.stdout: "filer_max_purge_ops": "10", 2024-09-21T10:00:27.605 INFO:teuthology.orchestra.run.smithi107.stdout: "filer_max_truncate_ops": "128", 2024-09-21T10:00:27.605 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_apply_finisher_threads": "1", 2024-09-21T10:00:27.605 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_blackhole": "false", 2024-09-21T10:00:27.605 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_btrfs_clone_range": "true", 2024-09-21T10:00:27.606 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_btrfs_snap": "true", 2024-09-21T10:00:27.606 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_caller_concurrency": "10", 2024-09-21T10:00:27.606 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_collect_device_partition_information": "true", 2024-09-21T10:00:27.606 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_commit_timeout": "600.000000", 2024-09-21T10:00:27.606 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_debug_inject_read_err": "false", 2024-09-21T10:00:27.606 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_debug_omap_check": "false", 2024-09-21T10:00:27.606 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_debug_verify_split": "false", 2024-09-21T10:00:27.606 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_dump_file": "", 2024-09-21T10:00:27.606 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-21T10:00:27.606 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-21T10:00:27.606 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fadvise": "true", 2024-09-21T10:00:27.606 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fail_eio": "true", 2024-09-21T10:00:27.606 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fd_cache_shards": "16", 2024-09-21T10:00:27.606 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fd_cache_size": "128", 2024-09-21T10:00:27.607 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fiemap": "false", 2024-09-21T10:00:27.607 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fiemap_threshold": "4096", 2024-09-21T10:00:27.607 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-21T10:00:27.607 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-21T10:00:27.607 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_inject_stall": "0", 2024-09-21T10:00:27.607 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_journal_parallel": "false", 2024-09-21T10:00:27.607 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_journal_trailing": "false", 2024-09-21T10:00:27.607 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_journal_writeahead": "false", 2024-09-21T10:00:27.607 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_kill_at": "0", 2024-09-21T10:00:27.607 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-21T10:00:27.607 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-21T10:00:27.607 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-21T10:00:27.607 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-21T10:00:27.607 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-21T10:00:27.607 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattrs": "0", 2024-09-21T10:00:27.608 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-21T10:00:27.608 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-21T10:00:27.608 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-21T10:00:27.608 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-21T10:00:27.608 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_xattr_value_size": "0", 2024-09-21T10:00:27.608 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-21T10:00:27.608 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-21T10:00:27.608 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-21T10:00:27.608 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_merge_threshold": "-10", 2024-09-21T10:00:27.608 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-21T10:00:27.608 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_odsync_write": "false", 2024-09-21T10:00:27.608 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_omap_backend": "rocksdb", 2024-09-21T10:00:27.608 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_omap_backend_path": "", 2024-09-21T10:00:27.608 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-21T10:00:27.608 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-21T10:00:27.609 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-21T10:00:27.609 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_op_thread_timeout": "60", 2024-09-21T10:00:27.609 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_op_threads": "2", 2024-09-21T10:00:27.609 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_punch_hole": "false", 2024-09-21T10:00:27.609 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-21T10:00:27.609 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-21T10:00:27.609 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-21T10:00:27.609 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-21T10:00:27.609 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-21T10:00:27.609 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-21T10:00:27.609 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-21T10:00:27.609 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-21T10:00:27.609 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-21T10:00:27.609 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_queue_max_ops": "50", 2024-09-21T10:00:27.610 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-21T10:00:27.610 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_seek_data_hole": "false", 2024-09-21T10:00:27.610 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_sloppy_crc": "false", 2024-09-21T10:00:27.610 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-21T10:00:27.610 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_splice": "false", 2024-09-21T10:00:27.610 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_split_multiple": "2", 2024-09-21T10:00:27.610 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_split_rand_factor": "20", 2024-09-21T10:00:27.610 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_update_to": "1000", 2024-09-21T10:00:27.610 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-21T10:00:27.610 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-21T10:00:27.610 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-21T10:00:27.610 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-21T10:00:27.610 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-21T10:00:27.610 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-21T10:00:27.611 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_enable": "true", 2024-09-21T10:00:27.611 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-21T10:00:27.611 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-21T10:00:27.611 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-21T10:00:27.611 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-21T10:00:27.611 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-21T10:00:27.611 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-21T10:00:27.611 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_xfs_extsize": "false", 2024-09-21T10:00:27.611 INFO:teuthology.orchestra.run.smithi107.stdout: "filestore_zfs_snap": "false", 2024-09-21T10:00:27.611 INFO:teuthology.orchestra.run.smithi107.stdout: "fio_dir": "/tmp/fio", 2024-09-21T10:00:27.611 INFO:teuthology.orchestra.run.smithi107.stdout: "fsid": "86c49514-77fc-11ef-baf6-efdc52797490", 2024-09-21T10:00:27.611 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_allow_other": "true", 2024-09-21T10:00:27.611 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_atomic_o_trunc": "true", 2024-09-21T10:00:27.611 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_big_writes": "true", 2024-09-21T10:00:27.612 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_debug": "false", 2024-09-21T10:00:27.612 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_default_permissions": "false", 2024-09-21T10:00:27.612 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_disable_pagecache": "false", 2024-09-21T10:00:27.612 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_max_write": "0", 2024-09-21T10:00:27.612 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_multithreaded": "true", 2024-09-21T10:00:27.612 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_require_active_mds": "true", 2024-09-21T10:00:27.612 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_set_user_groups": "true", 2024-09-21T10:00:27.612 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_splice_move": "true", 2024-09-21T10:00:27.612 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_splice_read": "true", 2024-09-21T10:00:27.612 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_splice_write": "true", 2024-09-21T10:00:27.612 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-21T10:00:27.612 INFO:teuthology.orchestra.run.smithi107.stdout: "fuse_use_invalidate_cb": "true", 2024-09-21T10:00:27.612 INFO:teuthology.orchestra.run.smithi107.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-21T10:00:27.612 INFO:teuthology.orchestra.run.smithi107.stdout: "gss_target_name": "ceph", 2024-09-21T10:00:27.612 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_file": "", 2024-09-21T10:00:27.613 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_inject_failure": "0", 2024-09-21T10:00:27.613 INFO:teuthology.orchestra.run.smithi107.stdout: "heartbeat_interval": "5", 2024-09-21T10:00:27.613 INFO:teuthology.orchestra.run.smithi107.stdout: "host": "smithi107", 2024-09-21T10:00:27.613 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-21T10:00:27.613 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-21T10:00:27.613 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-21T10:00:27.613 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-21T10:00:27.613 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-21T10:00:27.613 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-21T10:00:27.613 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-21T10:00:27.613 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-21T10:00:27.613 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-21T10:00:27.613 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-21T10:00:27.613 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-21T10:00:27.613 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-21T10:00:27.614 INFO:teuthology.orchestra.run.smithi107.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-21T10:00:27.614 INFO:teuthology.orchestra.run.smithi107.stdout: "inject_early_sigterm": "false", 2024-09-21T10:00:27.614 INFO:teuthology.orchestra.run.smithi107.stdout: "jaeger_agent_port": "6799", 2024-09-21T10:00:27.614 INFO:teuthology.orchestra.run.smithi107.stdout: "jaeger_tracing_enable": "false", 2024-09-21T10:00:27.614 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_aio": "true", 2024-09-21T10:00:27.614 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_align_min_size": "65536", 2024-09-21T10:00:27.614 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_block_align": "true", 2024-09-21T10:00:27.614 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_block_size": "4096", 2024-09-21T10:00:27.614 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_dio": "true", 2024-09-21T10:00:27.614 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_discard": "false", 2024-09-21T10:00:27.614 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_force_aio": "false", 2024-09-21T10:00:27.614 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_ignore_corruption": "false", 2024-09-21T10:00:27.614 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_max_write_bytes": "10485760", 2024-09-21T10:00:27.614 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_max_write_entries": "100", 2024-09-21T10:00:27.614 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_replay_from": "0", 2024-09-21T10:00:27.615 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-21T10:00:27.615 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-21T10:00:27.615 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-21T10:00:27.615 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-21T10:00:27.615 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_write_header_frequency": "0", 2024-09-21T10:00:27.615 INFO:teuthology.orchestra.run.smithi107.stdout: "journal_zero_on_create": "false", 2024-09-21T10:00:27.615 INFO:teuthology.orchestra.run.smithi107.stdout: "journaler_prefetch_periods": "10", 2024-09-21T10:00:27.615 INFO:teuthology.orchestra.run.smithi107.stdout: "journaler_prezero_periods": "5", 2024-09-21T10:00:27.615 INFO:teuthology.orchestra.run.smithi107.stdout: "journaler_write_head_interval": "15", 2024-09-21T10:00:27.615 INFO:teuthology.orchestra.run.smithi107.stdout: "key": "", 2024-09-21T10:00:27.615 INFO:teuthology.orchestra.run.smithi107.stdout: "keyfile": "", 2024-09-21T10:00:27.615 INFO:teuthology.orchestra.run.smithi107.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-21T10:00:27.615 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_backend": "rocksdb", 2024-09-21T10:00:27.615 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_default_stripe_size": "65536", 2024-09-21T10:00:27.615 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_fsck_on_mount": "false", 2024-09-21T10:00:27.616 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-21T10:00:27.616 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_max_bytes": "67108864", 2024-09-21T10:00:27.616 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_max_ops": "512", 2024-09-21T10:00:27.616 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_nid_prealloc": "1024", 2024-09-21T10:00:27.616 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_onode_map_size": "1024", 2024-09-21T10:00:27.616 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-21T10:00:27.616 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_sync_submit_transaction": "false", 2024-09-21T10:00:27.616 INFO:teuthology.orchestra.run.smithi107.stdout: "kstore_sync_transaction": "false", 2024-09-21T10:00:27.616 INFO:teuthology.orchestra.run.smithi107.stdout: "librados_thread_count": "2", 2024-09-21T10:00:27.616 INFO:teuthology.orchestra.run.smithi107.stdout: "lockdep": "false", 2024-09-21T10:00:27.616 INFO:teuthology.orchestra.run.smithi107.stdout: "lockdep_force_backtrace": "false", 2024-09-21T10:00:27.616 INFO:teuthology.orchestra.run.smithi107.stdout: "log_coarse_timestamps": "true", 2024-09-21T10:00:27.616 INFO:teuthology.orchestra.run.smithi107.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-21T10:00:27.616 INFO:teuthology.orchestra.run.smithi107.stdout: "log_flush_on_exit": "false", 2024-09-21T10:00:27.617 INFO:teuthology.orchestra.run.smithi107.stdout: "log_graylog_host": "127.0.0.1", 2024-09-21T10:00:27.617 INFO:teuthology.orchestra.run.smithi107.stdout: "log_graylog_port": "12201", 2024-09-21T10:00:27.617 INFO:teuthology.orchestra.run.smithi107.stdout: "log_max_new": "1000", 2024-09-21T10:00:27.617 INFO:teuthology.orchestra.run.smithi107.stdout: "log_max_recent": "10000", 2024-09-21T10:00:27.617 INFO:teuthology.orchestra.run.smithi107.stdout: "log_stderr_prefix": "", 2024-09-21T10:00:27.617 INFO:teuthology.orchestra.run.smithi107.stdout: "log_stop_at_utilization": "0.970000", 2024-09-21T10:00:27.617 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_file": "true", 2024-09-21T10:00:27.617 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_graylog": "false", 2024-09-21T10:00:27.617 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_journald": "false", 2024-09-21T10:00:27.617 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_stderr": "false", 2024-09-21T10:00:27.617 INFO:teuthology.orchestra.run.smithi107.stdout: "log_to_syslog": "false", 2024-09-21T10:00:27.617 INFO:teuthology.orchestra.run.smithi107.stdout: "max_rotating_auth_attempts": "10", 2024-09-21T10:00:27.617 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-21T10:00:27.617 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_action_on_write_error": "1", 2024-09-21T10:00:27.618 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_alternate_name_max": "8192", 2024-09-21T10:00:27.618 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_asio_thread_count": "2", 2024-09-21T10:00:27.618 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_export_pin": "true", 2024-09-21T10:00:27.618 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_fragment_dirs": "true", 2024-09-21T10:00:27.618 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-21T10:00:27.618 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_fragment_interval": "5", 2024-09-21T10:00:27.618 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-21T10:00:27.618 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-21T10:00:27.618 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_interval": "10", 2024-09-21T10:00:27.618 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_max": "-1", 2024-09-21T10:00:27.618 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_max_until": "-1", 2024-09-21T10:00:27.618 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_merge_size": "50", 2024-09-21T10:00:27.618 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_midchunk": "0.300000", 2024-09-21T10:00:27.618 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-21T10:00:27.619 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_min_start": "0.200000", 2024-09-21T10:00:27.619 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_minchunk": "0.001000", 2024-09-21T10:00:27.619 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_mode": "0", 2024-09-21T10:00:27.619 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_need_max": "1.200000", 2024-09-21T10:00:27.619 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_need_min": "0.800000", 2024-09-21T10:00:27.619 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_overload_epochs": "2", 2024-09-21T10:00:27.619 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-21T10:00:27.619 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-21T10:00:27.619 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_split_bits": "3", 2024-09-21T10:00:27.619 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-21T10:00:27.619 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_split_size": "10000", 2024-09-21T10:00:27.619 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-21T10:00:27.619 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_target_decay": "10.000000", 2024-09-21T10:00:27.619 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-21T10:00:27.619 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_beacon_grace": "15.000000", 2024-09-21T10:00:27.620 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_beacon_interval": "4.000000", 2024-09-21T10:00:27.620 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-21T10:00:27.620 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-21T10:00:27.620 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_mid": "0.700000", 2024-09-21T10:00:27.620 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-21T10:00:27.620 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_quiesce_delay": "0", 2024-09-21T10:00:27.620 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-21T10:00:27.620 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-21T10:00:27.620 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-21T10:00:27.620 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_release_free_interval": "10", 2024-09-21T10:00:27.620 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_reservation": "0.050000", 2024-09-21T10:00:27.620 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-21T10:00:27.620 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_trim_interval": "1", 2024-09-21T10:00:27.620 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cache_trim_threshold": "262144", 2024-09-21T10:00:27.621 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-21T10:00:27.621 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-21T10:00:27.621 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-21T10:00:27.621 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_client_prealloc_inos": "1000", 2024-09-21T10:00:27.621 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-21T10:00:27.621 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_connect_bootstrapping": "false", 2024-09-21T10:00:27.621 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_damage_table_max_entries": "10000", 2024-09-21T10:00:27.621 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-21T10:00:27.621 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_debug_auth_pins": "false", 2024-09-21T10:00:27.621 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_debug_frag": "false", 2024-09-21T10:00:27.621 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_debug_scatterstat": "false", 2024-09-21T10:00:27.621 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_debug_subtrees": "false", 2024-09-21T10:00:27.621 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_decay_halflife": "5.000000", 2024-09-21T10:00:27.621 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_default_dir_hash": "2", 2024-09-21T10:00:27.621 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_defer_session_stale": "true", 2024-09-21T10:00:27.622 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_deny_all_reconnect": "false", 2024-09-21T10:00:27.622 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dir_keys_per_op": "16384", 2024-09-21T10:00:27.622 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dir_max_commit_size": "10", 2024-09-21T10:00:27.622 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dir_max_entries": "0", 2024-09-21T10:00:27.622 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dir_prefetch": "true", 2024-09-21T10:00:27.622 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-21T10:00:27.622 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-21T10:00:27.622 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dump_cache_on_map": "false", 2024-09-21T10:00:27.622 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-21T10:00:27.622 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-21T10:00:27.622 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_early_reply": "true", 2024-09-21T10:00:27.622 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_enable_op_tracker": "true", 2024-09-21T10:00:27.622 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_enforce_unique_name": "true", 2024-09-21T10:00:27.622 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-21T10:00:27.623 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-21T10:00:27.623 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_export_ephemeral_random": "true", 2024-09-21T10:00:27.623 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-21T10:00:27.623 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-21T10:00:27.623 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-21T10:00:27.623 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-21T10:00:27.623 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-21T10:00:27.623 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-21T10:00:27.623 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-21T10:00:27.623 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-21T10:00:27.623 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_health_summarize_threshold": "10", 2024-09-21T10:00:27.623 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-21T10:00:27.623 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-21T10:00:27.623 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_inject_health_dummy": "false", 2024-09-21T10:00:27.623 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-21T10:00:27.624 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_inject_migrator_session_race": "false", 2024-09-21T10:00:27.624 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-21T10:00:27.624 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-21T10:00:27.624 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-21T10:00:27.624 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_join_fs": "", 2024-09-21T10:00:27.624 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_journal_format": "1", 2024-09-21T10:00:27.624 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-21T10:00:27.624 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_create_at": "0", 2024-09-21T10:00:27.624 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_dirfrag_at": "0", 2024-09-21T10:00:27.624 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_export_at": "0", 2024-09-21T10:00:27.624 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_import_at": "0", 2024-09-21T10:00:27.624 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_journal_at": "0", 2024-09-21T10:00:27.624 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_journal_expire_at": "0", 2024-09-21T10:00:27.624 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_journal_replay_at": "0", 2024-09-21T10:00:27.624 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_link_at": "0", 2024-09-21T10:00:27.625 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_mdstable_at": "0", 2024-09-21T10:00:27.625 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_openc_at": "0", 2024-09-21T10:00:27.625 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_rename_at": "0", 2024-09-21T10:00:27.625 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_kill_shutdown_at": "0", 2024-09-21T10:00:27.625 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_event_large_threshold": "524288", 2024-09-21T10:00:27.625 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_events_per_segment": "1024", 2024-09-21T10:00:27.625 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-21T10:00:27.625 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_max_events": "-1", 2024-09-21T10:00:27.625 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_max_segments": "128", 2024-09-21T10:00:27.625 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_pause": "false", 2024-09-21T10:00:27.625 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_segment_size": "0", 2024-09-21T10:00:27.625 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-21T10:00:27.625 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-21T10:00:27.625 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-21T10:00:27.625 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_trim_threshold": "128", 2024-09-21T10:00:27.626 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-21T10:00:27.626 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_log_warn_factor": "2.000000", 2024-09-21T10:00:27.626 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_caps_per_client": "1048576", 2024-09-21T10:00:27.626 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_completed_flushes": "100000", 2024-09-21T10:00:27.626 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_completed_requests": "100000", 2024-09-21T10:00:27.626 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_export_size": "20971520", 2024-09-21T10:00:27.626 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_file_recover": "32", 2024-09-21T10:00:27.626 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_purge_files": "64", 2024-09-21T10:00:27.626 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_purge_ops": "8192", 2024-09-21T10:00:27.626 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-21T10:00:27.626 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-21T10:00:27.626 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_max_snaps_per_dir": "100", 2024-09-21T10:00:27.626 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_metrics_update_interval": "2", 2024-09-21T10:00:27.626 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_min_caps_per_client": "100", 2024-09-21T10:00:27.627 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_min_caps_working_set": "10000", 2024-09-21T10:00:27.627 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-21T10:00:27.627 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_numa_node": "-1", 2024-09-21T10:00:27.627 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-21T10:00:27.627 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_op_complaint_time": "30.000000", 2024-09-21T10:00:27.627 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_op_history_duration": "600", 2024-09-21T10:00:27.627 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_op_history_size": "20", 2024-09-21T10:00:27.627 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_op_history_slow_op_size": "20", 2024-09-21T10:00:27.627 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-21T10:00:27.627 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_op_log_threshold": "5", 2024-09-21T10:00:27.627 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_ping_grace": "15", 2024-09-21T10:00:27.627 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_ping_interval": "5", 2024-09-21T10:00:27.627 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-21T10:00:27.627 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-21T10:00:27.627 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_max_caps": "30000", 2024-09-21T10:00:27.628 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-21T10:00:27.628 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-21T10:00:27.628 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-21T10:00:27.628 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_recall_warning_threshold": "262144", 2024-09-21T10:00:27.628 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-21T10:00:27.628 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_replay_interval": "1.000000", 2024-09-21T10:00:27.628 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-21T10:00:27.628 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-21T10:00:27.628 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_root_ino_gid": "0", 2024-09-21T10:00:27.628 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_root_ino_uid": "0", 2024-09-21T10:00:27.628 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-21T10:00:27.628 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_scrub_stats_review_period": "1", 2024-09-21T10:00:27.628 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-21T10:00:27.628 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-21T10:00:27.628 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-21T10:00:27.629 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-21T10:00:27.629 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-21T10:00:27.629 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-21T10:00:27.629 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-21T10:00:27.629 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-21T10:00:27.629 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-21T10:00:27.629 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-21T10:00:27.629 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-21T10:00:27.629 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_shutdown_check": "0", 2024-09-21T10:00:27.629 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_skip_ino": "0", 2024-09-21T10:00:27.629 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-21T10:00:27.629 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_snap_max_uid": "4294967294", 2024-09-21T10:00:27.629 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_snap_min_uid": "0", 2024-09-21T10:00:27.629 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_snap_rstat": "false", 2024-09-21T10:00:27.631 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_standby_replay_damaged": "false", 2024-09-21T10:00:27.631 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_symlink_recovery": "true", 2024-09-21T10:00:27.631 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-21T10:00:27.631 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_thrash_exports": "0", 2024-09-21T10:00:27.631 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_thrash_fragments": "0", 2024-09-21T10:00:27.631 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_tick_interval": "5.000000", 2024-09-21T10:00:27.631 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_valgrind_exit": "false", 2024-09-21T10:00:27.631 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_verify_backtrace": "1", 2024-09-21T10:00:27.631 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_verify_scatter": "false", 2024-09-21T10:00:27.631 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-21T10:00:27.632 INFO:teuthology.orchestra.run.smithi107.stdout: "mds_wipe_sessions": "false", 2024-09-21T10:00:27.632 INFO:teuthology.orchestra.run.smithi107.stdout: "mempool_debug": "false", 2024-09-21T10:00:27.632 INFO:teuthology.orchestra.run.smithi107.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-21T10:00:27.632 INFO:teuthology.orchestra.run.smithi107.stdout: "memstore_device_bytes": "1073741824", 2024-09-21T10:00:27.632 INFO:teuthology.orchestra.run.smithi107.stdout: "memstore_page_set": "false", 2024-09-21T10:00:27.632 INFO:teuthology.orchestra.run.smithi107.stdout: "memstore_page_size": "65536", 2024-09-21T10:00:27.632 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_client_bytes": "134217728", 2024-09-21T10:00:27.632 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_client_messages": "512", 2024-09-21T10:00:27.632 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-21T10:00:27.632 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-21T10:00:27.632 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-21T10:00:27.632 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-21T10:00:27.632 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_disabled_modules": "", 2024-09-21T10:00:27.632 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_enable_op_tracker": "true", 2024-09-21T10:00:27.633 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-21T10:00:27.633 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_max_pg_creating": "1024", 2024-09-21T10:00:27.633 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_max_pg_num_change": "128", 2024-09-21T10:00:27.633 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_mds_bytes": "134217728", 2024-09-21T10:00:27.633 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_mds_messages": "128", 2024-09-21T10:00:27.633 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-21T10:00:27.633 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_mon_bytes": "134217728", 2024-09-21T10:00:27.633 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_mon_messages": "128", 2024-09-21T10:00:27.633 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-21T10:00:27.633 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-21T10:00:27.633 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_op_history_duration": "600", 2024-09-21T10:00:27.633 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_op_history_size": "20", 2024-09-21T10:00:27.633 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-21T10:00:27.633 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-21T10:00:27.634 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_op_log_threshold": "5", 2024-09-21T10:00:27.634 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_osd_bytes": "536870912", 2024-09-21T10:00:27.634 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_osd_messages": "8192", 2024-09-21T10:00:27.634 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_pool": "true", 2024-09-21T10:00:27.634 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-21T10:00:27.634 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_standby_modules": "true", 2024-09-21T10:00:27.634 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_stats_period": "5", 2024-09-21T10:00:27.634 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_stats_threshold": "5", 2024-09-21T10:00:27.634 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_tick_period": "2", 2024-09-21T10:00:27.634 INFO:teuthology.orchestra.run.smithi107.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-21T10:00:27.634 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-21T10:00:27.634 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_allow_pool_delete": "true", 2024-09-21T10:00:27.634 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_allow_pool_size_one": "false", 2024-09-21T10:00:27.634 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_auth_validate_all_caps": "true", 2024-09-21T10:00:27.635 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-21T10:00:27.635 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-21T10:00:27.635 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_bytes": "104857600", 2024-09-21T10:00:27.635 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_directed_command_retry": "2", 2024-09-21T10:00:27.635 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-21T10:00:27.635 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-21T10:00:27.635 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-21T10:00:27.635 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-21T10:00:27.635 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_hunt_parallel": "3", 2024-09-21T10:00:27.635 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_log_interval": "1.000000", 2024-09-21T10:00:27.635 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-21T10:00:27.635 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_ping_interval": "10.000000", 2024-09-21T10:00:27.635 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-21T10:00:27.635 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_client_target_rank": "-1", 2024-09-21T10:00:27.635 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-21T10:00:27.636 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-21T10:00:27.636 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-21T10:00:27.636 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_level": "debug", 2024-09-21T10:00:27.636 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_file": "true", 2024-09-21T10:00:27.636 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-21T10:00:27.636 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-21T10:00:27.636 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-21T10:00:27.636 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_journald": "true", 2024-09-21T10:00:27.636 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-21T10:00:27.636 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-21T10:00:27.636 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-21T10:00:27.636 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_compact_on_bootstrap": "false", 2024-09-21T10:00:27.636 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_compact_on_start": "false", 2024-09-21T10:00:27.636 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_compact_on_trim": "true", 2024-09-21T10:00:27.637 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-21T10:00:27.637 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-21T10:00:27.637 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-21T10:00:27.637 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_cpu_threads": "4", 2024-09-21T10:00:27.637 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_crush_min_required_version": "hammer", 2024-09-21T10:00:27.637 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_daemon_bytes": "419430400", 2024-09-21T10:00:27.637 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-21T10:00:27.637 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_data_avail_crit": "5", 2024-09-21T10:00:27.637 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_data_avail_warn": "5", 2024-09-21T10:00:27.637 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_data_size_warn": "16106127360", 2024-09-21T10:00:27.637 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-21T10:00:27.637 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-21T10:00:27.637 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_dump_json": "false", 2024-09-21T10:00:27.637 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-21T10:00:27.638 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_dump_transactions": "false", 2024-09-21T10:00:27.638 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_extra_checks": "false", 2024-09-21T10:00:27.638 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-21T10:00:27.638 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-21T10:00:27.638 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_no_require_reef": "false", 2024-09-21T10:00:27.638 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_no_require_squid": "false", 2024-09-21T10:00:27.638 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-21T10:00:27.638 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-21T10:00:27.638 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-21T10:00:27.638 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_down_mkfs_grace": "60", 2024-09-21T10:00:27.638 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_down_uptime_grace": "60", 2024-09-21T10:00:27.638 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_election_default_strategy": "3", 2024-09-21T10:00:27.638 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_election_timeout": "5.000000", 2024-09-21T10:00:27.638 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-21T10:00:27.639 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_elector_ping_divisor": "2", 2024-09-21T10:00:27.639 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-21T10:00:27.639 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_enable_op_tracker": "true", 2024-09-21T10:00:27.639 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_fake_pool_delete": "false", 2024-09-21T10:00:27.639 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_force_quorum_join": "false", 2024-09-21T10:00:27.639 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-21T10:00:27.639 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_globalid_prealloc": "10000", 2024-09-21T10:00:27.639 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_detail_to_clog": "true", 2024-09-21T10:00:27.639 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_log_update_period": "5", 2024-09-21T10:00:27.639 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_max_detail": "50", 2024-09-21T10:00:27.639 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_to_clog": "true", 2024-09-21T10:00:27.639 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_to_clog_interval": "600", 2024-09-21T10:00:27.639 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-21T10:00:27.639 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_host": "[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] [v2:172.21.15.129:3300/0,v1:172.21.15.129:6789/0] [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0]", 2024-09-21T10:00:27.639 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_host_override": "", 2024-09-21T10:00:27.640 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_initial_members": "", 2024-09-21T10:00:27.640 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-21T10:00:27.640 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-21T10:00:27.640 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-21T10:00:27.640 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-21T10:00:27.640 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-21T10:00:27.640 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_lease": "5.000000", 2024-09-21T10:00:27.640 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-21T10:00:27.640 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-21T10:00:27.640 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_log_full_interval": "50", 2024-09-21T10:00:27.640 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_log_max": "10000", 2024-09-21T10:00:27.640 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_log_max_summary": "50", 2024-09-21T10:00:27.640 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-21T10:00:27.640 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_log_epochs": "500", 2024-09-21T10:00:27.640 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-21T10:00:27.641 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-21T10:00:27.641 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-21T10:00:27.641 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_osd": "10000", 2024-09-21T10:00:27.641 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_pg_per_osd": "10000", 2024-09-21T10:00:27.641 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_pool_pg_num": "65536", 2024-09-21T10:00:27.641 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-21T10:00:27.641 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-21T10:00:27.641 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mds_force_trim_to": "0", 2024-09-21T10:00:27.641 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mds_skip_sanity": "true", 2024-09-21T10:00:27.641 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_memory_autotune": "true", 2024-09-21T10:00:27.641 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_memory_target": "2147483648", 2024-09-21T10:00:27.641 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_beacon_grace": "30", 2024-09-21T10:00:27.641 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-21T10:00:27.641 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_digest_period": "5", 2024-09-21T10:00:27.641 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_inactive_grace": "60", 2024-09-21T10:00:27.642 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-21T10:00:27.642 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-21T10:00:27.642 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_min_osdmap_epochs": "500", 2024-09-21T10:00:27.642 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-21T10:00:27.642 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-21T10:00:27.642 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_complaint_time": "30", 2024-09-21T10:00:27.642 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_history_duration": "600", 2024-09-21T10:00:27.642 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_history_size": "20", 2024-09-21T10:00:27.642 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_history_slow_op_size": "20", 2024-09-21T10:00:27.642 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-21T10:00:27.642 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_op_log_threshold": "5", 2024-09-21T10:00:27.642 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-21T10:00:27.642 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-21T10:00:27.642 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-21T10:00:27.643 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_auto_mark_in": "false", 2024-09-21T10:00:27.643 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-21T10:00:27.643 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-21T10:00:27.643 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-21T10:00:27.643 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_cache_size": "500", 2024-09-21T10:00:27.643 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-21T10:00:27.643 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-21T10:00:27.643 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-21T10:00:27.643 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_down_out_interval": "600", 2024-09-21T10:00:27.643 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-21T10:00:27.643 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-21T10:00:27.643 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_force_trim_to": "0", 2024-09-21T10:00:27.643 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-21T10:00:27.643 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-21T10:00:27.643 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-21T10:00:27.644 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-21T10:00:27.644 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-21T10:00:27.644 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-21T10:00:27.644 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-21T10:00:27.644 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_min_down_reporters": "2", 2024-09-21T10:00:27.644 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-21T10:00:27.644 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-21T10:00:27.644 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-21T10:00:27.644 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-21T10:00:27.644 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-21T10:00:27.644 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-21T10:00:27.644 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_report_timeout": "900", 2024-09-21T10:00:27.644 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-21T10:00:27.644 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-21T10:00:27.645 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-21T10:00:27.645 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-21T10:00:27.645 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-21T10:00:27.645 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-21T10:00:27.645 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-21T10:00:27.645 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-21T10:00:27.645 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-21T10:00:27.645 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_stuck_threshold": "60", 2024-09-21T10:00:27.645 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-21T10:00:27.645 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-21T10:00:27.645 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-21T10:00:27.645 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-21T10:00:27.645 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-21T10:00:27.645 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-21T10:00:27.646 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_probe_timeout": "2.000000", 2024-09-21T10:00:27.646 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_reweight_max_change": "0.050000", 2024-09-21T10:00:27.646 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_reweight_max_osds": "4", 2024-09-21T10:00:27.646 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-21T10:00:27.646 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-21T10:00:27.646 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-21T10:00:27.646 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-21T10:00:27.646 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-21T10:00:27.646 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_scrub_interval": "86400", 2024-09-21T10:00:27.646 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_scrub_max_keys": "100", 2024-09-21T10:00:27.646 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_scrub_timeout": "300", 2024-09-21T10:00:27.646 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_session_timeout": "300", 2024-09-21T10:00:27.646 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_smart_report_timeout": "5", 2024-09-21T10:00:27.646 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_stat_smooth_intervals": "6", 2024-09-21T10:00:27.646 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-21T10:00:27.647 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_stretch_pool_min_size": "2", 2024-09-21T10:00:27.647 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_stretch_pool_size": "4", 2024-09-21T10:00:27.647 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-21T10:00:27.647 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-21T10:00:27.647 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_debug": "false", 2024-09-21T10:00:27.647 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-21T10:00:27.647 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-21T10:00:27.647 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_provider_kill_at": "0", 2024-09-21T10:00:27.647 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_requester_kill_at": "0", 2024-09-21T10:00:27.647 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_sync_timeout": "60.000000", 2024-09-21T10:00:27.647 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_target_pg_per_osd": "100", 2024-09-21T10:00:27.647 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_tick_interval": "5", 2024-09-21T10:00:27.647 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_timecheck_interval": "300.000000", 2024-09-21T10:00:27.647 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-21T10:00:27.648 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_use_min_delay_socket": "false", 2024-09-21T10:00:27.648 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_older_version_delay": "604800", 2024-09-21T10:00:27.648 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-21T10:00:27.648 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-21T10:00:27.648 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-21T10:00:27.648 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-21T10:00:27.648 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-21T10:00:27.648 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-21T10:00:27.648 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-21T10:00:27.648 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_misplaced": "false", 2024-09-21T10:00:27.648 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-21T10:00:27.648 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_older_version": "true", 2024-09-21T10:00:27.648 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-21T10:00:27.648 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-21T10:00:27.648 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-21T10:00:27.649 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-21T10:00:27.649 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-21T10:00:27.649 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-21T10:00:27.649 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-21T10:00:27.649 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-21T10:00:27.649 INFO:teuthology.orchestra.run.smithi107.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-21T10:00:27.649 INFO:teuthology.orchestra.run.smithi107.stdout: "monmap": "", 2024-09-21T10:00:27.649 INFO:teuthology.orchestra.run.smithi107.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T10:00:27.649 INFO:teuthology.orchestra.run.smithi107.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-21T10:00:27.649 INFO:teuthology.orchestra.run.smithi107.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-21T10:00:27.649 INFO:teuthology.orchestra.run.smithi107.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T10:00:27.649 INFO:teuthology.orchestra.run.smithi107.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-21T10:00:27.649 INFO:teuthology.orchestra.run.smithi107.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-21T10:00:27.649 INFO:teuthology.orchestra.run.smithi107.stdout: "motr_tracing_enabled": "false", 2024-09-21T10:00:27.649 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_op_threads": "3", 2024-09-21T10:00:27.650 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-21T10:00:27.650 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_cm": "false", 2024-09-21T10:00:27.650 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_device_name": "", 2024-09-21T10:00:27.650 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_dscp": "96", 2024-09-21T10:00:27.650 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-21T10:00:27.650 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-21T10:00:27.650 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_local_gid": "", 2024-09-21T10:00:27.650 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-21T10:00:27.650 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_port_num": "1", 2024-09-21T10:00:27.650 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-21T10:00:27.650 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-21T10:00:27.650 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-21T10:00:27.650 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-21T10:00:27.650 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_sl": "3", 2024-09-21T10:00:27.651 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_support_srq": "true", 2024-09-21T10:00:27.651 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_rdma_type": "ib", 2024-09-21T10:00:27.651 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_async_reap_threshold": "5", 2024-09-21T10:00:27.651 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_before_connect": "false", 2024-09-21T10:00:27.651 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_ipv4": "true", 2024-09-21T10:00:27.651 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_ipv6": "false", 2024-09-21T10:00:27.651 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_msgr1": "true", 2024-09-21T10:00:27.651 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_msgr2": "true", 2024-09-21T10:00:27.651 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_port_max": "7568", 2024-09-21T10:00:27.651 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_port_min": "6800", 2024-09-21T10:00:27.651 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-21T10:00:27.651 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_retry_count": "3", 2024-09-21T10:00:27.651 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_bind_retry_delay": "5", 2024-09-21T10:00:27.651 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_blackhole_client": "false", 2024-09-21T10:00:27.651 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_blackhole_mds": "false", 2024-09-21T10:00:27.652 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_blackhole_mgr": "false", 2024-09-21T10:00:27.652 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_blackhole_mon": "false", 2024-09-21T10:00:27.652 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_blackhole_osd": "false", 2024-09-21T10:00:27.652 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_client_mode": "crc secure", 2024-09-21T10:00:27.652 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-21T10:00:27.652 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_cluster_mode": "crc secure", 2024-09-21T10:00:27.652 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_cluster_type": "", 2024-09-21T10:00:27.652 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_compress_secure": "false", 2024-09-21T10:00:27.652 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_connection_idle_timeout": "900", 2024-09-21T10:00:27.652 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_connection_ready_timeout": "10", 2024-09-21T10:00:27.652 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_crc_data": "true", 2024-09-21T10:00:27.652 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_crc_header": "true", 2024-09-21T10:00:27.652 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_die_on_bad_msg": "false", 2024-09-21T10:00:27.652 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_die_on_bug": "true", 2024-09-21T10:00:27.653 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_die_on_old_message": "true", 2024-09-21T10:00:27.653 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_die_on_skipped_message": "false", 2024-09-21T10:00:27.653 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-21T10:00:27.653 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-21T10:00:27.653 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_coremask": "0xF", 2024-09-21T10:00:27.653 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-21T10:00:27.653 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-21T10:00:27.653 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_enable_tso": "true", 2024-09-21T10:00:27.653 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-21T10:00:27.653 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-21T10:00:27.653 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_hugepages": "", 2024-09-21T10:00:27.653 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-21T10:00:27.653 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-21T10:00:27.653 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_lro": "true", 2024-09-21T10:00:27.653 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_memory_channel": "4", 2024-09-21T10:00:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-21T10:00:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_pmd": "", 2024-09-21T10:00:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_port_id": "0", 2024-09-21T10:00:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-21T10:00:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-21T10:00:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_dump_on_send": "false", 2024-09-21T10:00:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_initial_backoff": "0.200000", 2024-09-21T10:00:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_delay_max": "1.000000", 2024-09-21T10:00:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-21T10:00:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_delay_type": "", 2024-09-21T10:00:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-21T10:00:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_network_congestion": "0", 2024-09-21T10:00:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_inject_socket_failures": "0", 2024-09-21T10:00:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_learn_addr_from_peer": "true", 2024-09-21T10:00:27.654 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_max_accept_failures": "4", 2024-09-21T10:00:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_max_backoff": "15.000000", 2024-09-21T10:00:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_mon_client_mode": "secure crc", 2024-09-21T10:00:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-21T10:00:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_mon_service_mode": "secure crc", 2024-09-21T10:00:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_osd_compress_min_size": "1024", 2024-09-21T10:00:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_osd_compress_mode": "none", 2024-09-21T10:00:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-21T10:00:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-21T10:00:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_pq_min_cost": "65536", 2024-09-21T10:00:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_public_type": "", 2024-09-21T10:00:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_service_mode": "crc secure", 2024-09-21T10:00:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_tcp_listen_backlog": "512", 2024-09-21T10:00:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_tcp_nodelay": "true", 2024-09-21T10:00:27.655 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-21T10:00:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_tcp_rcvbuf": "0", 2024-09-21T10:00:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-21T10:00:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "ms_type": "async+posix", 2024-09-21T10:00:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "no_config_file": "false", 2024-09-21T10:00:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-21T10:00:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-21T10:00:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_completion_locks_per_session": "32", 2024-09-21T10:00:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-21T10:00:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-21T10:00:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_inflight_ops": "1024", 2024-09-21T10:00:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-21T10:00:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-21T10:00:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_tick_interval": "5.000000", 2024-09-21T10:00:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "objecter_timeout": "10.000000", 2024-09-21T10:00:27.656 INFO:teuthology.orchestra.run.smithi107.stdout: "objectstore_blackhole": "false", 2024-09-21T10:00:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-21T10:00:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "openssl_engine_opts": "", 2024-09-21T10:00:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_delay_time": "5.000000", 2024-09-21T10:00:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_hist_halflife": "1000", 2024-09-21T10:00:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_max_low_ops": "2", 2024-09-21T10:00:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_max_ops": "4", 2024-09-21T10:00:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-21T10:00:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-21T10:00:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_agent_slop": "0.020000", 2024-09-21T10:00:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-21T10:00:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-21T10:00:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_asio_thread_count": "2", 2024-09-21T10:00:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_async_recovery_min_cost": "100", 2024-09-21T10:00:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-21T10:00:27.657 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-21T10:00:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backfill_scan_max": "512", 2024-09-21T10:00:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backfill_scan_min": "64", 2024-09-21T10:00:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backoff_on_degraded": "false", 2024-09-21T10:00:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backoff_on_peering": "false", 2024-09-21T10:00:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_backoff_on_unfound": "true", 2024-09-21T10:00:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_beacon_report_interval": "300", 2024-09-21T10:00:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bench_duration": "30", 2024-09-21T10:00:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-21T10:00:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bench_max_block_size": "67108864", 2024-09-21T10:00:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-21T10:00:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_blkin_trace_all": "false", 2024-09-21T10:00:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-21T10:00:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-21T10:00:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-21T10:00:27.658 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-21T10:00:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_check_for_log_corruption": "false", 2024-09-21T10:00:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-21T10:00:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T10:00:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-21T10:00:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T10:00:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_class_update_on_start": "true", 2024-09-21T10:00:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_client_message_cap": "256", 2024-09-21T10:00:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_client_message_size_cap": "524288000", 2024-09-21T10:00:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_client_op_priority": "63", 2024-09-21T10:00:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_client_watch_timeout": "30", 2024-09-21T10:00:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_command_max_records": "256", 2024-09-21T10:00:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_compact_on_start": "false", 2024-09-21T10:00:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-21T10:00:27.659 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-21T10:00:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-21T10:00:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_crush_update_on_start": "true", 2024-09-21T10:00:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_crush_update_weight_set": "true", 2024-09-21T10:00:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-21T10:00:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-21T10:00:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-21T10:00:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-21T10:00:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-21T10:00:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_feed_pullee": "-1", 2024-09-21T10:00:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-21T10:00:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-21T10:00:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-21T10:00:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_misdirected_ops": "false", 2024-09-21T10:00:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_no_acting_change": "false", 2024-09-21T10:00:27.660 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_no_purge_strays": "false", 2024-09-21T10:00:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_op_order": "false", 2024-09-21T10:00:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-21T10:00:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-21T10:00:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-21T10:00:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-21T10:00:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_shutdown": "false", 2024-09-21T10:00:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-21T10:00:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-21T10:00:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_trim_objects": "false", 2024-09-21T10:00:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-21T10:00:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-21T10:00:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_verify_snaps": "false", 2024-09-21T10:00:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-21T10:00:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-21T10:00:27.661 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_keys": "1024", 2024-09-21T10:00:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-21T10:00:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-21T10:00:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-21T10:00:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_stride": "524288", 2024-09-21T10:00:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-21T10:00:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-21T10:00:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_default_notify_timeout": "30", 2024-09-21T10:00:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_delete_sleep": "0.000000", 2024-09-21T10:00:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-21T10:00:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-21T10:00:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-21T10:00:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_discard_disconnected_ops": "true", 2024-09-21T10:00:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_ec_partial_reads": "true", 2024-09-21T10:00:27.662 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_enable_op_tracker": "true", 2024-09-21T10:00:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-21T10:00:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-21T10:00:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-21T10:00:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-21T10:00:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_fast_info": "true", 2024-09-21T10:00:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_fast_shutdown": "true", 2024-09-21T10:00:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-21T10:00:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-21T10:00:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-21T10:00:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-21T10:00:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-21T10:00:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_function_tracing": "false", 2024-09-21T10:00:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_grace": "20", 2024-09-21T10:00:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_interval": "6", 2024-09-21T10:00:27.663 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-21T10:00:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_min_peers": "10", 2024-09-21T10:00:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_min_size": "2000", 2024-09-21T10:00:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_stale": "600", 2024-09-21T10:00:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-21T10:00:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_hit_set_max_size": "100000", 2024-09-21T10:00:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_hit_set_min_size": "1000", 2024-09-21T10:00:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-21T10:00:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-21T10:00:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-21T10:00:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-21T10:00:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-21T10:00:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-21T10:00:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_journal_size": "5120", 2024-09-21T10:00:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_kill_backfill_at": "0", 2024-09-21T10:00:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-21T10:00:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_map_cache_size": "50", 2024-09-21T10:00:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_map_dedup": "true", 2024-09-21T10:00:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_map_message_max": "40", 2024-09-21T10:00:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-21T10:00:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_map_share_max_epochs": "40", 2024-09-21T10:00:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_attr_name_len": "100", 2024-09-21T10:00:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_attr_size": "0", 2024-09-21T10:00:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_backfills": "1", 2024-09-21T10:00:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_markdown_count": "5", 2024-09-21T10:00:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_markdown_period": "600", 2024-09-21T10:00:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_object_name_len": "2048", 2024-09-21T10:00:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_object_namespace_len": "256", 2024-09-21T10:00:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_object_size": "134217728", 2024-09-21T10:00:27.665 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-21T10:00:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-21T10:00:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_pg_blocked_by": "16", 2024-09-21T10:00:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_pg_log_entries": "10000", 2024-09-21T10:00:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-21T10:00:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_pgls": "1024", 2024-09-21T10:00:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_push_cost": "8388608", 2024-09-21T10:00:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_push_objects": "10", 2024-09-21T10:00:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_scrubs": "3", 2024-09-21T10:00:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-21T10:00:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_trimming_pgs": "2", 2024-09-21T10:00:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_write_op_reply_len": "64", 2024-09-21T10:00:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_max_write_size": "90", 2024-09-21T10:00:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-21T10:00:27.666 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-21T10:00:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-21T10:00:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-21T10:00:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-21T10:00:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-21T10:00:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-21T10:00:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-21T10:00:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_profile": "balanced", 2024-09-21T10:00:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-21T10:00:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-21T10:00:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-21T10:00:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-21T10:00:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-21T10:00:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-21T10:00:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-21T10:00:27.667 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-21T10:00:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-21T10:00:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-21T10:00:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-21T10:00:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_base": "805306368", 2024-09-21T10:00:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_cache_min": "134217728", 2024-09-21T10:00:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-21T10:00:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-21T10:00:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_target": "4294967296", 2024-09-21T10:00:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_target_autotune": "false", 2024-09-21T10:00:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-21T10:00:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_min_pg_log_entries": "250", 2024-09-21T10:00:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_min_recovery_priority": "0", 2024-09-21T10:00:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-21T10:00:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-21T10:00:27.668 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_report_interval": "5", 2024-09-21T10:00:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-21T10:00:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-21T10:00:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_num_cache_shards": "32", 2024-09-21T10:00:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_num_op_tracker_shard": "32", 2024-09-21T10:00:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_numa_auto_affinity": "true", 2024-09-21T10:00:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_numa_node": "-1", 2024-09-21T10:00:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_numa_prefer_iface": "true", 2024-09-21T10:00:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-21T10:00:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_objecter_finishers": "1", 2024-09-21T10:00:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_objectstore": "bluestore", 2024-09-21T10:00:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_objectstore_fuse": "false", 2024-09-21T10:00:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-21T10:00:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_objectstore_tracing": "false", 2024-09-21T10:00:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_complaint_time": "30.000000", 2024-09-21T10:00:27.669 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_history_duration": "600", 2024-09-21T10:00:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_history_size": "20", 2024-09-21T10:00:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_history_slow_op_size": "20", 2024-09-21T10:00:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-21T10:00:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_log_threshold": "5", 2024-09-21T10:00:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_shards": "0", 2024-09-21T10:00:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_shards_hdd": "5", 2024-09-21T10:00:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_shards_ssd": "8", 2024-09-21T10:00:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-21T10:00:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-21T10:00:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-21T10:00:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-21T10:00:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_pq_min_cost": "65536", 2024-09-21T10:00:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-21T10:00:27.670 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_queue_cut_off": "high", 2024-09-21T10:00:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-21T10:00:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_op_thread_timeout": "15", 2024-09-21T10:00:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_open_classes_on_start": "true", 2024-09-21T10:00:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_os_flags": "0", 2024-09-21T10:00:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_peering_op_priority": "255", 2024-09-21T10:00:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_delete_cost": "1048576", 2024-09-21T10:00:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_delete_priority": "5", 2024-09-21T10:00:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-21T10:00:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-21T10:00:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-21T10:00:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_log_trim_max": "10000", 2024-09-21T10:00:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_log_trim_min": "100", 2024-09-21T10:00:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-21T10:00:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-21T10:00:27.671 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-21T10:00:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-21T10:00:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-21T10:00:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-21T10:00:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-21T10:00:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-21T10:00:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-21T10:00:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-21T10:00:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_crimson": "false", 2024-09-21T10:00:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-21T10:00:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-21T10:00:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-21T10:00:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-21T10:00:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-21T10:00:27.672 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-21T10:00:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-21T10:00:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-21T10:00:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_flags": "0", 2024-09-21T10:00:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-21T10:00:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_min_size": "0", 2024-09-21T10:00:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-21T10:00:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_pg_num": "32", 2024-09-21T10:00:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_pgp_num": "0", 2024-09-21T10:00:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-21T10:00:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_read_ratio": "70", 2024-09-21T10:00:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_size": "3", 2024-09-21T10:00:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_default_type": "replicated", 2024-09-21T10:00:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-21T10:00:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-21T10:00:27.673 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_push_per_object_cost": "1000", 2024-09-21T10:00:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-21T10:00:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recover_clone_overlap": "true", 2024-09-21T10:00:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-21T10:00:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_cost": "20971520", 2024-09-21T10:00:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-21T10:00:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_active": "0", 2024-09-21T10:00:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-21T10:00:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-21T10:00:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-21T10:00:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-21T10:00:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_max_single_start": "1", 2024-09-21T10:00:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_op_priority": "3", 2024-09-21T10:00:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-21T10:00:27.674 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_priority": "5", 2024-09-21T10:00:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-21T10:00:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_sleep": "0.000000", 2024-09-21T10:00:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-21T10:00:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-21T10:00:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-21T10:00:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_repair_during_recovery": "false", 2024-09-21T10:00:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_requested_scrub_priority": "5", 2024-09-21T10:00:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-21T10:00:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-21T10:00:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_auto_repair": "false", 2024-09-21T10:00:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-21T10:00:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-21T10:00:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_begin_hour": "0", 2024-09-21T10:00:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_begin_week_day": "0", 2024-09-21T10:00:27.675 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_chunk_max": "15", 2024-09-21T10:00:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_chunk_min": "5", 2024-09-21T10:00:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_cost": "52428800", 2024-09-21T10:00:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-21T10:00:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_during_recovery": "false", 2024-09-21T10:00:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_end_hour": "0", 2024-09-21T10:00:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_end_week_day": "0", 2024-09-21T10:00:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_event_cost": "4096", 2024-09-21T10:00:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-21T10:00:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-21T10:00:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_invalid_stats": "true", 2024-09-21T10:00:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-21T10:00:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-21T10:00:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_max_preemptions": "5", 2024-09-21T10:00:27.676 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-21T10:00:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_priority": "5", 2024-09-21T10:00:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_retry_after_noscrub": "60", 2024-09-21T10:00:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_retry_delay": "30", 2024-09-21T10:00:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_retry_new_interval": "10", 2024-09-21T10:00:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_retry_pg_state": "60", 2024-09-21T10:00:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_retry_trimming": "10", 2024-09-21T10:00:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_scrub_sleep": "0.000000", 2024-09-21T10:00:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-21T10:00:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-21T10:00:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-21T10:00:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-21T10:00:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_skip_data_digest": "false", 2024-09-21T10:00:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_smart_report_timeout": "5", 2024-09-21T10:00:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_cost": "1048576", 2024-09-21T10:00:27.677 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_priority": "5", 2024-09-21T10:00:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-21T10:00:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-21T10:00:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-21T10:00:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-21T10:00:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-21T10:00:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-21T10:00:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-21T10:00:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_target_transaction_size": "30", 2024-09-21T10:00:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-21T10:00:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-21T10:00:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-21T10:00:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-21T10:00:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-21T10:00:27.678 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-21T10:00:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-21T10:00:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-21T10:00:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-21T10:00:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-21T10:00:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_tracing": "false", 2024-09-21T10:00:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_use_stale_snap": "false", 2024-09-21T10:00:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-21T10:00:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "osdc_blkin_trace_all": "false", 2024-09-21T10:00:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_kill_at": "0", 2024-09-21T10:00:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_max_join_drift": "10", 2024-09-21T10:00:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_min": "500", 2024-09-21T10:00:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_min_wait": "0.050000", 2024-09-21T10:00:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_propose_interval": "1.000000", 2024-09-21T10:00:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_service_trim_max": "500", 2024-09-21T10:00:27.679 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-21T10:00:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_service_trim_min": "250", 2024-09-21T10:00:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_stash_full_interval": "25", 2024-09-21T10:00:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_trim_max": "500", 2024-09-21T10:00:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "paxos_trim_min": "250", 2024-09-21T10:00:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "perf": "true", 2024-09-21T10:00:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "pid_file": "", 2024-09-21T10:00:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-21T10:00:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-21T10:00:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addr": "-", 2024-09-21T10:00:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "public_addrv": "", 2024-09-21T10:00:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "public_bind_addr": "-", 2024-09-21T10:00:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "public_network": "172.21.0.0/20", 2024-09-21T10:00:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "public_network_interface": "", 2024-09-21T10:00:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "qat_compressor_busy_polling": "false", 2024-09-21T10:00:27.680 INFO:teuthology.orchestra.run.smithi107.stdout: "qat_compressor_enabled": "false", 2024-09-21T10:00:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "qat_compressor_session_max_number": "256", 2024-09-21T10:00:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "rados_mon_op_timeout": "0", 2024-09-21T10:00:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "rados_osd_op_timeout": "0", 2024-09-21T10:00:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "rados_tracing": "false", 2024-09-21T10:00:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_atime_update_interval": "60", 2024-09-21T10:00:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-21T10:00:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_balance_parent_reads": "false", 2024-09-21T10:00:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_balance_snap_reads": "false", 2024-09-21T10:00:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_blkin_trace_all": "false", 2024-09-21T10:00:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-21T10:00:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-21T10:00:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache": "true", 2024-09-21T10:00:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-21T10:00:27.681 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-21T10:00:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-21T10:00:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-21T10:00:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_policy": "writearound", 2024-09-21T10:00:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_size": "33554432", 2024-09-21T10:00:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-21T10:00:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-21T10:00:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_clone_copy_on_read": "false", 2024-09-21T10:00:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_compression_hint": "none", 2024-09-21T10:00:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_concurrent_management_ops": "10", 2024-09-21T10:00:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-21T10:00:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_clone_format": "auto", 2024-09-21T10:00:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_data_pool": "", 2024-09-21T10:00:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_features": "61", 2024-09-21T10:00:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_format": "2", 2024-09-21T10:00:27.682 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_map_options": "", 2024-09-21T10:00:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_order": "22", 2024-09-21T10:00:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_pool": "rbd", 2024-09-21T10:00:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-21T10:00:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_stripe_count": "0", 2024-09-21T10:00:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_default_stripe_unit": "0", 2024-09-21T10:00:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-21T10:00:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-21T10:00:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-21T10:00:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_enable_alloc_hint": "true", 2024-09-21T10:00:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-21T10:00:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_io_scheduler": "simple", 2024-09-21T10:00:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-21T10:00:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-21T10:00:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-21T10:00:27.683 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-21T10:00:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-21T10:00:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-21T10:00:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-21T10:00:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-21T10:00:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-21T10:00:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_order": "24", 2024-09-21T10:00:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_pool": "", 2024-09-21T10:00:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_journal_splay_width": "4", 2024-09-21T10:00:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_localize_parent_reads": "false", 2024-09-21T10:00:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_localize_snap_reads": "false", 2024-09-21T10:00:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-21T10:00:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-21T10:00:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-21T10:00:27.684 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-21T10:00:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-21T10:00:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-21T10:00:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-21T10:00:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-21T10:00:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-21T10:00:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-21T10:00:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-21T10:00:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-21T10:00:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-21T10:00:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-21T10:00:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-21T10:00:27.685 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-21T10:00:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-21T10:00:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-21T10:00:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-21T10:00:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-21T10:00:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-21T10:00:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-21T10:00:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-21T10:00:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-21T10:00:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-21T10:00:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-21T10:00:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-21T10:00:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-21T10:00:27.686 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-21T10:00:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-21T10:00:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-21T10:00:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_mtime_update_interval": "60", 2024-09-21T10:00:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_non_blocking_aio": "true", 2024-09-21T10:00:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_op_thread_timeout": "60", 2024-09-21T10:00:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_op_threads": "1", 2024-09-21T10:00:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_parent_cache_enabled": "false", 2024-09-21T10:00:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-21T10:00:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-21T10:00:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-21T10:00:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_plugins": "", 2024-09-21T10:00:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_bps_burst": "0", 2024-09-21T10:00:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-21T10:00:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_bps_limit": "0", 2024-09-21T10:00:27.687 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_exclude_ops": "0", 2024-09-21T10:00:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_iops_burst": "0", 2024-09-21T10:00:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-21T10:00:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_iops_limit": "0", 2024-09-21T10:00:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-21T10:00:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-21T10:00:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-21T10:00:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-21T10:00:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-21T10:00:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-21T10:00:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-21T10:00:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-21T10:00:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-21T10:00:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-21T10:00:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-21T10:00:27.688 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-21T10:00:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-21T10:00:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-21T10:00:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_read_from_replica_policy": "default", 2024-09-21T10:00:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-21T10:00:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-21T10:00:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-21T10:00:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-21T10:00:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_skip_partial_discard": "true", 2024-09-21T10:00:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-21T10:00:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_tracing": "false", 2024-09-21T10:00:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_validate_names": "true", 2024-09-21T10:00:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "rbd_validate_pool": "true", 2024-09-21T10:00:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "restapi_base_url": "", 2024-09-21T10:00:27.689 INFO:teuthology.orchestra.run.smithi107.stdout: "restapi_log_level": "", 2024-09-21T10:00:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-21T10:00:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-21T10:00:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_acl_grants_max_num": "100", 2024-09-21T10:00:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_admin_entry": "admin", 2024-09-21T10:00:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-21T10:00:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_asio_assert_yielding": "false", 2024-09-21T10:00:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_backend_store": "rados", 2024-09-21T10:00:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_barbican_url": "", 2024-09-21T10:00:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_beast_enable_async": "true", 2024-09-21T10:00:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_counters_cache": "false", 2024-09-21T10:00:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-21T10:00:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-21T10:00:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-21T10:00:27.690 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-21T10:00:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-21T10:00:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-21T10:00:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-21T10:00:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-21T10:00:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_cache_enabled": "true", 2024-09-21T10:00:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_cache_expiry_interval": "900", 2024-09-21T10:00:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_cache_lru_size": "25000", 2024-09-21T10:00:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_config_store": "rados", 2024-09-21T10:00:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_content_length_compat": "false", 2024-09-21T10:00:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_copy_obj_progress": "true", 2024-09-21T10:00:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-21T10:00:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_cors_rules_max_num": "100", 2024-09-21T10:00:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_cross_domain_policy": "", 2024-09-21T10:00:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-21T10:00:27.691 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_addr": "", 2024-09-21T10:00:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-21T10:00:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-21T10:00:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-21T10:00:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-21T10:00:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_password": "", 2024-09-21T10:00:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-21T10:00:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_kmip_username": "", 2024-09-21T10:00:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_require_ssl": "true", 2024-09-21T10:00:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-21T10:00:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-21T10:00:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-21T10:00:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-21T10:00:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-21T10:00:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-21T10:00:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-21T10:00:27.693 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-21T10:00:27.693 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-21T10:00:27.693 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-21T10:00:27.693 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-21T10:00:27.693 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-21T10:00:27.693 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-21T10:00:27.693 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-21T10:00:27.693 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-21T10:00:27.693 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_addr": "", 2024-09-21T10:00:27.693 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_auth": "token", 2024-09-21T10:00:27.693 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_namespace": "", 2024-09-21T10:00:27.693 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_prefix": "", 2024-09-21T10:00:27.693 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-21T10:00:27.693 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-21T10:00:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-21T10:00:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-21T10:00:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_token_file": "", 2024-09-21T10:00:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-21T10:00:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_curl_buffersize": "524288", 2024-09-21T10:00:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-21T10:00:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_curl_low_speed_time": "300", 2024-09-21T10:00:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-21T10:00:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-21T10:00:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-21T10:00:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-21T10:00:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-21T10:00:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-21T10:00:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-21T10:00:27.694 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-21T10:00:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-21T10:00:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-21T10:00:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-21T10:00:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-21T10:00:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-21T10:00:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-21T10:00:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-21T10:00:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-21T10:00:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-21T10:00:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-21T10:00:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-21T10:00:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_log_changes_size": "1000", 2024-09-21T10:00:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_log_num_shards": "128", 2024-09-21T10:00:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-21T10:00:27.695 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_log_window": "30", 2024-09-21T10:00:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-21T10:00:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-21T10:00:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-21T10:00:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-21T10:00:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-21T10:00:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-21T10:00:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-21T10:00:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-21T10:00:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-21T10:00:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-21T10:00:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-21T10:00:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-21T10:00:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-21T10:00:27.696 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_disable_s3select": "false", 2024-09-21T10:00:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-21T10:00:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-21T10:00:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-21T10:00:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-21T10:00:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-21T10:00:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-21T10:00:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-21T10:00:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-21T10:00:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-21T10:00:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-21T10:00:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-21T10:00:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-21T10:00:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dns_name": "", 2024-09-21T10:00:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dns_s3website_name": "", 2024-09-21T10:00:27.697 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dynamic_resharding": "true", 2024-09-21T10:00:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-21T10:00:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-21T10:00:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-21T10:00:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_gc_threads": "true", 2024-09-21T10:00:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_lc_threads": "true", 2024-09-21T10:00:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_mdsearch": "true", 2024-09-21T10:00:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_ops_log": "false", 2024-09-21T10:00:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_quota_threads": "true", 2024-09-21T10:00:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_static_website": "false", 2024-09-21T10:00:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enable_usage_log": "false", 2024-09-21T10:00:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_enforce_swift_acls": "true", 2024-09-21T10:00:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_exit_timeout_secs": "120", 2024-09-21T10:00:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_expose_bucket": "false", 2024-09-21T10:00:27.698 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_extended_http_attrs": "", 2024-09-21T10:00:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_filter": "none", 2024-09-21T10:00:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-21T10:00:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_frontends": "beast port=7480", 2024-09-21T10:00:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-21T10:00:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_deferred": "50", 2024-09-21T10:00:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-21T10:00:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_objs": "32", 2024-09-21T10:00:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-21T10:00:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-21T10:00:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-21T10:00:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-21T10:00:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_gc_processor_period": "3600", 2024-09-21T10:00:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-21T10:00:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-21T10:00:27.699 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-21T10:00:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-21T10:00:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_init_timeout": "300", 2024-09-21T10:00:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-21T10:00:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-21T10:00:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_kafka_connection_idle": "300", 2024-09-21T10:00:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-21T10:00:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-21T10:00:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-21T10:00:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-21T10:00:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-21T10:00:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_domain": "", 2024-09-21T10:00:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_password": "", 2024-09-21T10:00:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_password_path": "", 2024-09-21T10:00:27.700 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_project": "", 2024-09-21T10:00:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_tenant": "", 2024-09-21T10:00:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_token": "", 2024-09-21T10:00:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_token_path": "", 2024-09-21T10:00:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_admin_user": "", 2024-09-21T10:00:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_barbican_domain": "", 2024-09-21T10:00:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_barbican_password": "", 2024-09-21T10:00:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_barbican_project": "", 2024-09-21T10:00:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-21T10:00:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_barbican_user": "", 2024-09-21T10:00:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-21T10:00:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-21T10:00:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-21T10:00:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-21T10:00:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-21T10:00:27.701 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_url": "", 2024-09-21T10:00:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-21T10:00:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_debug_interval": "-1", 2024-09-21T10:00:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_lock_max_time": "90", 2024-09-21T10:00:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_max_objs": "32", 2024-09-21T10:00:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_max_rules": "1000", 2024-09-21T10:00:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_max_worker": "3", 2024-09-21T10:00:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-21T10:00:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lc_thread_delay": "0", 2024-09-21T10:00:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-21T10:00:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_dnattr": "uid", 2024-09-21T10:00:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-21T10:00:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_searchfilter": "", 2024-09-21T10:00:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-21T10:00:27.702 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ldap_uri": "", 2024-09-21T10:00:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-21T10:00:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-21T10:00:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-21T10:00:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-21T10:00:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_log_http_headers": "", 2024-09-21T10:00:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-21T10:00:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-21T10:00:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_log_object_name_utc": "false", 2024-09-21T10:00:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-21T10:00:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-21T10:00:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_attr_name_len": "0", 2024-09-21T10:00:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_attr_size": "0", 2024-09-21T10:00:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-21T10:00:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_chunk_size": "4194304", 2024-09-21T10:00:27.703 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-21T10:00:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-21T10:00:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-21T10:00:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_listing_results": "5000", 2024-09-21T10:00:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_notify_retries": "10", 2024-09-21T10:00:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-21T10:00:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_put_param_size": "1048576", 2024-09-21T10:00:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_put_size": "5368709120", 2024-09-21T10:00:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_max_slo_entries": "1000", 2024-09-21T10:00:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_md_log_max_shards": "64", 2024-09-21T10:00:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-21T10:00:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-21T10:00:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-21T10:00:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-21T10:00:27.704 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_mp_lock_max_time": "600", 2024-09-21T10:00:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-21T10:00:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-21T10:00:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-21T10:00:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-21T10:00:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-21T10:00:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-21T10:00:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-21T10:00:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-21T10:00:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_max_gc": "300", 2024-09-21T10:00:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-21T10:00:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-21T10:00:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-21T10:00:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-21T10:00:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-21T10:00:27.705 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-21T10:00:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-21T10:00:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_num_async_rados_threads": "32", 2024-09-21T10:00:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_num_control_oids": "8", 2024-09-21T10:00:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_numa_node": "-1", 2024-09-21T10:00:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-21T10:00:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-21T10:00:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_objexp_chunk_size": "100", 2024-09-21T10:00:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_objexp_gc_interval": "600", 2024-09-21T10:00:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-21T10:00:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-21T10:00:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-21T10:00:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_op_thread_timeout": "600", 2024-09-21T10:00:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_op_tracing": "false", 2024-09-21T10:00:27.706 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_opa_token": "", 2024-09-21T10:00:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_opa_url": "", 2024-09-21T10:00:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_opa_verify_ssl": "true", 2024-09-21T10:00:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-21T10:00:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-21T10:00:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ops_log_rados": "false", 2024-09-21T10:00:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_ops_log_socket_path": "", 2024-09-21T10:00:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-21T10:00:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-21T10:00:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-21T10:00:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_period_push_interval": "2.000000", 2024-09-21T10:00:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-21T10:00:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-21T10:00:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-21T10:00:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-21T10:00:27.707 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_posix_cache_lanes": "3", 2024-09-21T10:00:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-21T10:00:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-21T10:00:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_posix_cache_partitions": "3", 2024-09-21T10:00:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-21T10:00:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_print_continue": "true", 2024-09-21T10:00:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-21T10:00:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-21T10:00:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-21T10:00:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-21T10:00:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-21T10:00:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_rados_tracing": "false", 2024-09-21T10:00:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_realm": "", 2024-09-21T10:00:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_realm_id": "", 2024-09-21T10:00:27.708 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-21T10:00:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_region": "", 2024-09-21T10:00:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-21T10:00:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-21T10:00:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-21T10:00:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-21T10:00:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_request_uri": "", 2024-09-21T10:00:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_batch_size": "64", 2024-09-21T10:00:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-21T10:00:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-21T10:00:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_max_aio": "128", 2024-09-21T10:00:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_num_logs": "16", 2024-09-21T10:00:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_progress_judge_interval": "120", 2024-09-21T10:00:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2024-09-21T10:00:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshard_thread_interval": "600", 2024-09-21T10:00:27.709 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_reshardlog_threshold": "30000", 2024-09-21T10:00:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_resolve_cname": "false", 2024-09-21T10:00:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-21T10:00:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_run_sync_thread": "true", 2024-09-21T10:00:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-21T10:00:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-21T10:00:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-21T10:00:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-21T10:00:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-21T10:00:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-21T10:00:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-21T10:00:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-21T10:00:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-21T10:00:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_scheduler_type": "throttler", 2024-09-21T10:00:27.710 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_script_uri": "", 2024-09-21T10:00:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_service_provider_name": "", 2024-09-21T10:00:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-21T10:00:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_client_id": "", 2024-09-21T10:00:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_client_secret": "", 2024-09-21T10:00:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_entry": "sts", 2024-09-21T10:00:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_key": "sts", 2024-09-21T10:00:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-21T10:00:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_min_session_duration": "900", 2024-09-21T10:00:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sts_token_introspection_url": "", 2024-09-21T10:00:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_account_in_url": "false", 2024-09-21T10:00:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_auth_entry": "auth", 2024-09-21T10:00:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_auth_url": "", 2024-09-21T10:00:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_custom_header": "", 2024-09-21T10:00:27.711 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-21T10:00:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_need_stats": "true", 2024-09-21T10:00:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_tenant_name": "", 2024-09-21T10:00:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_token_expiration": "86400", 2024-09-21T10:00:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_url": "", 2024-09-21T10:00:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_url_prefix": "swift", 2024-09-21T10:00:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-21T10:00:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-21T10:00:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-21T10:00:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_lease_period": "120", 2024-09-21T10:00:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-21T10:00:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-21T10:00:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-21T10:00:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-21T10:00:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-21T10:00:27.712 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-21T10:00:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-21T10:00:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-21T10:00:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-21T10:00:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_thread_pool_size": "512", 2024-09-21T10:00:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-21T10:00:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-21T10:00:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-21T10:00:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-21T10:00:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_comment": "", 2024-09-21T10:00:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_createby": "", 2024-09-21T10:00:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_encoding": "", 2024-09-21T10:00:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_flag": "false", 2024-09-21T10:00:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-21T10:00:27.713 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_origin": "", 2024-09-21T10:00:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-21T10:00:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_torrent_tracker": "", 2024-09-21T10:00:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_trust_forwarded_https": "false", 2024-09-21T10:00:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-21T10:00:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-21T10:00:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_usage_max_shards": "32", 2024-09-21T10:00:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_usage_max_user_shards": "1", 2024-09-21T10:00:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_use_opa_authz": "false", 2024-09-21T10:00:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_counters_cache": "false", 2024-09-21T10:00:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-21T10:00:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-21T10:00:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-21T10:00:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_max_buckets": "1000", 2024-09-21T10:00:27.714 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_policies_max_num": "100", 2024-09-21T10:00:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-21T10:00:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-21T10:00:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-21T10:00:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-21T10:00:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_user_unique_email": "true", 2024-09-21T10:00:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_verify_ssl": "true", 2024-09-21T10:00:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-21T10:00:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_zone": "", 2024-09-21T10:00:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_zone_id": "", 2024-09-21T10:00:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-21T10:00:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_zonegroup": "", 2024-09-21T10:00:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_zonegroup_id": "", 2024-09-21T10:00:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-21T10:00:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "rgwlc_auto_session_clear": "true", 2024-09-21T10:00:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-21T10:00:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_block_size": "4096", 2024-09-21T10:00:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-21T10:00:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-21T10:00:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-21T10:00:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-21T10:00:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-21T10:00:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_size": "536870912", 2024-09-21T10:00:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-21T10:00:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-21T10:00:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-21T10:00:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-21T10:00:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-21T10:00:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-21T10:00:27.716 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-21T10:00:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-21T10:00:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_index_type": "binary_search", 2024-09-21T10:00:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-21T10:00:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-21T10:00:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_partition_filters": "false", 2024-09-21T10:00:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_perf": "false", 2024-09-21T10:00:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-21T10:00:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-21T10:00:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-21T10:00:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "run_dir": "/var/run/ceph", 2024-09-21T10:00:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_block_create": "true", 2024-09-21T10:00:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_cache_lru_size": "67108864", 2024-09-21T10:00:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-21T10:00:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-21T10:00:27.717 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_default_max_object_size": "16777216", 2024-09-21T10:00:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-21T10:00:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_device_size": "53687091200", 2024-09-21T10:00:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-21T10:00:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_full_integrity_check": "false", 2024-09-21T10:00:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_journal_batch_capacity": "16", 2024-09-21T10:00:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-21T10:00:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-21T10:00:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-21T10:00:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_main_device_type": "SSD", 2024-09-21T10:00:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-21T10:00:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-21T10:00:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-21T10:00:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-21T10:00:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-21T10:00:27.718 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-21T10:00:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "seastore_segment_size": "67108864", 2024-09-21T10:00:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "set_keepcaps": "false", 2024-09-21T10:00:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "setgroup": "ceph", 2024-09-21T10:00:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "setuser": "ceph", 2024-09-21T10:00:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "setuser_match_path": "", 2024-09-21T10:00:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-21T10:00:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "thp": "false", 2024-09-21T10:00:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "threadpool_default_timeout": "60", 2024-09-21T10:00:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-21T10:00:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "throttler_perf_counter": "true", 2024-09-21T10:00:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "tmp_dir": "/tmp", 2024-09-21T10:00:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-21T10:00:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "uadk_compressor_enabled": "false", 2024-09-21T10:00:27.719 INFO:teuthology.orchestra.run.smithi107.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-21T10:00:27.719 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:00:27.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:27 smithi107 ceph-mon[102221]: pgmap v487: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:27.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:27 smithi107 ceph-mon[105022]: pgmap v487: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:27 smithi129 ceph-mon[88471]: pgmap v487: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:28.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-21T10:00:28.733 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:29.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:29 smithi107 ceph-mon[105022]: pgmap v488: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:29.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:29 smithi107 ceph-mon[102221]: pgmap v488: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:29 smithi129 ceph-mon[88471]: pgmap v488: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:30.367 INFO:teuthology.orchestra.run.smithi107.stdout:mon.a: {} 2024-09-21T10:00:30.368 INFO:teuthology.orchestra.run.smithi107.stdout:mon.b: {} 2024-09-21T10:00:30.368 INFO:teuthology.orchestra.run.smithi107.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:00:30.368 INFO:teuthology.orchestra.run.smithi107.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:00:30.374 INFO:teuthology.orchestra.run.smithi107.stdout:mon.c: {} 2024-09-21T10:00:30.375 INFO:teuthology.orchestra.run.smithi107.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:00:30.586 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:00:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:00:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:30.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:30 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2112811415' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:00:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:30 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2112811415' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:00:30.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:30 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2112811415' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:00:30.971 INFO:tasks.thrashosds.thrasher:Unable to test ceph-objectstore-tool, not available on all OSD nodes 2024-09-21T10:00:30.971 INFO:teuthology.task.sequential:In sequential, running task full_sequential... 2024-09-21T10:00:30.971 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:00:30.972 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:00:30.972 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-21T10:00:30.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_4 16 2024-09-21T10:00:30.973 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-21T10:00:30.973 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:00:30.973 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:00:30.973 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:00:30.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd out 7 2024-09-21T10:00:30.973 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-21T10:00:30.974 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:30.974 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-21T10:00:30.974 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:00:30.975 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-21T10:00:30.975 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:00:30.975 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-21T10:00:30.975 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:00:31.004 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:31.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:31.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:31.106 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:31.134 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:31.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:31.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:31.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:31.269 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:31.271 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:31.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:31.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:31.273 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:31.275 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:31.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:31.395 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:31.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:31.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:31.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:00:31.518 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:31.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:31.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:00:31.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:31.641 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:31.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:31.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:31.659 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:31 smithi107 ceph-mon[102221]: pgmap v489: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:31.659 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:31 smithi107 ceph-mon[105022]: pgmap v489: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:31.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:31.765 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:31.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:31.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:31.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:31.887 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:31.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:31.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:31 smithi129 ceph-mon[88471]: pgmap v489: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:31.989 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:32.003 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:00:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:00:31] "GET /metrics HTTP/1.1" 200 34153 "" "Prometheus/2.51.0" 2024-09-21T10:00:32.010 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:32.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:32.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:32.112 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:32.138 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:32.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:32.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:32.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:00:32.261 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:32.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:32.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:00:32.363 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:32.383 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:32.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:32.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:32.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:32.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:32.507 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:32.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:32.608 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:32.634 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:32.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:32.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:32.736 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:32.762 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:32.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:32.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:32.864 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:32.888 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:32.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:32.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:32.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:33.011 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:33.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:33.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:33.113 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:33.139 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:33.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:33.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:33.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:33.263 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:33.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:33.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:33.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:33.385 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:33.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:33.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:33.487 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:33.513 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:33.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:33.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:33.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:33.636 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:33.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:33.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:33.737 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:33.758 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:33.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:33.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:33.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:33 smithi107 ceph-mon[102221]: pgmap v490: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:33.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:33 smithi107 ceph-mon[105022]: pgmap v490: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:33.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:33.880 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:33.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:33.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:33 smithi129 ceph-mon[88471]: pgmap v490: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:33.981 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:34.002 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:34.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:34.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:34.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:34.125 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:34.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:34.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:34.227 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:34.248 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:34.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:34.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:34.350 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:34.375 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:34.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:34.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:34.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:00:34.499 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:34.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:34.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:00:34.601 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:34.622 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:34.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:34.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:34.724 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:34.751 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:34.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:34.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:34.852 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:34.878 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:34.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:34.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:34.938 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:00:34.938 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:00:34.938 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-09-21T10:00:34.938 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:00:34.980 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:35.005 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:35.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:35.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:35.107 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:35.132 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:35.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:35.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:35.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:00:35.256 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:35.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:35.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:00:35.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:00:35.379 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:35.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:35.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:00:35.481 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:35.502 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:35.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:35.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:35.604 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:35.629 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:35.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:35.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:35.731 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:35.755 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:35.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:35.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:35.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:35 smithi107 ceph-mon[102221]: pgmap v491: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:35.768 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:35 smithi107 ceph-mon[105022]: pgmap v491: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:35.785 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:00:35.785 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:00:35.785 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:00:35.786 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:00:35.791 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:35.792 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:35.807 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:00:35.807 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:00:35.826 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:00:35.826 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:00:35.842 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:00:35.842 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:00:35.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:35.858 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:00:35.858 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:00:35.874 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:00:35.874 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:00:35.877 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:35.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:35.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:35.890 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:00:35.890 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:00:35.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:35 smithi129 ceph-mon[88471]: pgmap v491: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:35.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:36.001 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:36.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:36.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:36.103 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:36.129 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:36.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:36.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:36.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:36.253 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:36.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:36.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:36.258 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:00:36.355 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:36.380 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:36.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:36.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:36.482 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:36.509 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:36.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:36.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:36.610 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:36.631 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:36.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:36.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:36.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:36.755 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:36.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:36.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:36.856 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:36 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3130949689' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:00:36.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:36 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3130949689' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:00:36.883 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:36.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:36.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:36 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3130949689' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:00:36.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:37.006 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:37.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:37.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:37.107 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:37.133 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:37.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:37.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:37.234 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:37.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:37.261 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:37.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:37.363 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:00:37.384 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:37.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:37.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:00:37.486 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:37.511 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:37.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:37.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:37.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:37.635 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:37.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:37.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:37.736 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:37.763 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:37.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:37.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:37 smithi107 ceph-mon[102221]: pgmap v492: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:37.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:37 smithi107 ceph-mon[105022]: pgmap v492: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:37.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:37.886 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:37.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:37.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:37 smithi129 ceph-mon[88471]: pgmap v492: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:37.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:38.014 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:38.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:38.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:38.116 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:38.154 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:38.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:38.256 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:38.281 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:38.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:38.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:38.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:38.404 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:38.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:38.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:38.506 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:38.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:38.529 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:38.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:38.606 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_4' created 2024-09-21T10:00:38.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:38.651 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:38.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:38.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:38.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:38 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/682698931' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T10:00:38.747 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:38 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/682698931' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T10:00:38.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:38.775 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:38.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:38.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:38.877 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:38.898 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:38.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:38.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:38 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/682698931' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T10:00:38.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:39.021 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:39.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:39.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:39.123 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:39.149 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:39.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:39.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:39.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:39.273 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:39.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:39.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:39.374 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:39.398 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:39.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:39.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:39.500 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:39.523 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:39.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:39.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:39.625 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:39.646 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:39.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:39.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:39.748 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:39.774 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:39.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:39.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:39.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-21T10:00:39.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:39.897 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:39.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:39.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:39.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:39 smithi129 ceph-mon[88471]: pgmap v493: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:39.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:39 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/682698931' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T10:00:39.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:39 smithi129 ceph-mon[88471]: osdmap e141: 8 total, 8 up, 8 in 2024-09-21T10:00:39.951 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:39 smithi107 ceph-mon[102221]: pgmap v493: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:39.951 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:39 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/682698931' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T10:00:39.951 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:39 smithi107 ceph-mon[102221]: osdmap e141: 8 total, 8 up, 8 in 2024-09-21T10:00:39.951 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:39 smithi107 ceph-mon[105022]: pgmap v493: 73 pgs: 73 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:39.951 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:39 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/682698931' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T10:00:39.951 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:39 smithi107 ceph-mon[105022]: osdmap e141: 8 total, 8 up, 8 in 2024-09-21T10:00:39.999 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:40.020 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:40.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:40.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:40.122 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:40.143 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:40.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:40.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:40.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:40.266 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:40.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:40.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:40.368 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:40.389 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:40.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:40.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:40.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:00:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:00:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:40.492 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:40.513 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:40.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:40.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:40.615 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:40.638 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:40.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:40.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:40.740 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:40.762 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:40.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:40.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:40.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:00:40.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:40 smithi107 ceph-mon[105022]: osdmap e142: 8 total, 8 up, 8 in 2024-09-21T10:00:40.854 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:40 smithi107 ceph-mon[102221]: osdmap e142: 8 total, 8 up, 8 in 2024-09-21T10:00:40.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:00:40.885 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:40.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:40.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:00:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:40 smithi129 ceph-mon[88471]: osdmap e142: 8 total, 8 up, 8 in 2024-09-21T10:00:40.986 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:41.008 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:41.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:41.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:41.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:41.131 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:41.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:41.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:41.232 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:41.254 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:41.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:41.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:41.296 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:41.325 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:41.341 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:41.356 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:41.377 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:41.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:41.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:41.422 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a[102198]: 2024-09-21T10:00:41.275+0000 7f3189124640 -1 mon.a@0(leader).osd e143 definitely_dead 0 2024-09-21T10:00:41.479 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:41.501 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:41.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:41.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:41.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:41.625 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:41.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:41.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:41.726 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:41.732 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-mon[102221]: pgmap v496: 89 pgs: 1 creating+activating, 3 creating+peering, 11 unknown, 74 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:41.732 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-mon[102221]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:00:41.732 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-mon[102221]: osdmap e143: 8 total, 8 up, 8 in 2024-09-21T10:00:41.732 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:00:41.732 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:41.733 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/113794066' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T10:00:41.733 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T10:00:41.733 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-mon[102221]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-21T10:00:41.733 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4202093169' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:00:41.733 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:00:41.733 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-mon[105022]: pgmap v496: 89 pgs: 1 creating+activating, 3 creating+peering, 11 unknown, 74 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:41.733 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-mon[105022]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:00:41.733 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-mon[105022]: osdmap e143: 8 total, 8 up, 8 in 2024-09-21T10:00:41.733 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:00:41.733 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:41.733 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/113794066' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T10:00:41.733 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T10:00:41.733 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-mon[105022]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-21T10:00:41.734 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4202093169' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:00:41.734 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:00:41.749 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:41.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:41.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:41.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:41.872 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:41.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:41.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:41 smithi129 ceph-mon[88471]: pgmap v496: 89 pgs: 1 creating+activating, 3 creating+peering, 11 unknown, 74 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:41 smithi129 ceph-mon[88471]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:00:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:41 smithi129 ceph-mon[88471]: osdmap e143: 8 total, 8 up, 8 in 2024-09-21T10:00:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:00:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:41 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/113794066' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T10:00:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:41 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T10:00:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:41 smithi129 ceph-mon[88471]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-21T10:00:41.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:41 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4202093169' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:00:41.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:41 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:00:41.975 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:41.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:41.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:42.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:00:42.108 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:00:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:00:41] "GET /metrics HTTP/1.1" 200 36442 "" "Prometheus/2.51.0" 2024-09-21T10:00:42.121 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:42.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:42.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:00:42.146 INFO:teuthology.orchestra.run.smithi107.stderr:marked out osd.7. 2024-09-21T10:00:42.146 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:00:42.223 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:42.246 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:42.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:42.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:42.347 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:42.369 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:42.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:42.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:42.471 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:42.492 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:42.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:42.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:42.594 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:42.616 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:42.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:42.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:42.717 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:42.739 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:42.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:42.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:42.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:42 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:00:42.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:42 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-21T10:00:42.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:42 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:00:42.795 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:42 smithi107 ceph-mon[102221]: osdmap e144: 8 total, 8 up, 7 in 2024-09-21T10:00:42.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:42 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:00:42.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:42 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-21T10:00:42.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:42 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:00:42.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:42 smithi107 ceph-mon[105022]: osdmap e144: 8 total, 8 up, 7 in 2024-09-21T10:00:42.841 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:42.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:42.864 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:42.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:42 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:00:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:42 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-21T10:00:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:42 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:00:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:42 smithi129 ceph-mon[88471]: osdmap e144: 8 total, 8 up, 7 in 2024-09-21T10:00:42.964 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:42.986 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:42.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:42.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:43.088 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:43.109 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:43.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:43.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:43.211 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:43.232 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:43.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:43.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:43.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:43.355 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:43.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:43.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:43.457 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:43.479 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:43.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:43.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:43.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:43.602 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:43.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:43.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:43.704 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:43.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:43.727 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:43.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:43.828 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:43.850 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:43.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:43.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:43.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:43 smithi107 ceph-mon[102221]: pgmap v498: 89 pgs: 1 creating+activating, 3 creating+peering, 7 unknown, 78 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:43 smithi107 ceph-mon[102221]: osdmap e145: 8 total, 8 up, 7 in 2024-09-21T10:00:43.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:43 smithi107 ceph-mon[105022]: pgmap v498: 89 pgs: 1 creating+activating, 3 creating+peering, 7 unknown, 78 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:43.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:43 smithi107 ceph-mon[105022]: osdmap e145: 8 total, 8 up, 7 in 2024-09-21T10:00:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:43 smithi129 ceph-mon[88471]: pgmap v498: 89 pgs: 1 creating+activating, 3 creating+peering, 7 unknown, 78 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:43 smithi129 ceph-mon[88471]: osdmap e145: 8 total, 8 up, 7 in 2024-09-21T10:00:43.952 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:43.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:43.975 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:43.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:44.075 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:44.099 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:44.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:44.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:44.200 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:44.224 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:44.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:44.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:44.326 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:44.349 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:44.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:44.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:44.450 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:44.473 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:44.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:44.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:44.574 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:44.597 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:44.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:44.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:44.699 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:44.720 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:44.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:44.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:44.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:00:44.842 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:44.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:44.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:00:44.944 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:44.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:44.969 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:44.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:45.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:45.093 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:45.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:45.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:45.195 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:45.216 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:45.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:45.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:45.318 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:45.344 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:45.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:45.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:45.445 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:45.472 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:45.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:45.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:45.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:45.582 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:45 smithi107 ceph-mon[102221]: pgmap v501: 89 pgs: 17 peering, 1 creating+activating, 2 unknown, 69 active+clean; 582 KiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-09-21T10:00:45.582 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:45 smithi107 ceph-mon[102221]: Health check failed: Reduced data availability: 1 pg inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-21T10:00:45.582 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:45 smithi107 ceph-mon[105022]: pgmap v501: 89 pgs: 17 peering, 1 creating+activating, 2 unknown, 69 active+clean; 582 KiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-09-21T10:00:45.582 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:45 smithi107 ceph-mon[105022]: Health check failed: Reduced data availability: 1 pg inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-21T10:00:45.597 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:45.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:45.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:45 smithi129 ceph-mon[88471]: pgmap v501: 89 pgs: 17 peering, 1 creating+activating, 2 unknown, 69 active+clean; 582 KiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-09-21T10:00:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:45 smithi129 ceph-mon[88471]: Health check failed: Reduced data availability: 1 pg inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-21T10:00:45.699 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:45.725 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:45.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:45.827 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:45.851 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:45.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:45.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:45.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:45.974 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:45.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:45.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:46.076 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:46.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:46.103 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:46.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:46.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:46.229 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:46.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:46.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:46.331 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:46.357 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:46.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:46.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:46.459 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:46.467 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:46 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/743938330' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:00:46.467 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:46 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/743938330' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:00:46.482 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:46.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:46.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:46.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:46.605 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:46.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:46.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:46.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:46 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/743938330' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:00:46.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:00:46.728 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:46.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:46.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:00:46.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:00:46.830 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:00:46.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:46.890 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:46.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:00:46.991 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:47.017 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:47.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:47.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:47.119 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:47.141 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:47.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:47.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:47.170 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:00:47.170 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:00:47.170 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:00:47.170 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:00:47.170 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:00:47.170 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:00:47.170 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:00:47.171 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:00:47.171 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:00:47.171 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:00:47.171 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:00:47.171 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:00:47.171 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:00:47.171 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:00:47.171 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:00:47.171 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:00:47.242 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:47.264 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:47.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:47.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:47.320 INFO:teuthology.orchestra.run.smithi107.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-21T10:00:47.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:47 smithi107 ceph-mon[105022]: pgmap v502: 89 pgs: 17 peering, 72 active+clean; 582 KiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T10:00:47.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:47 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1351567080' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:00:47.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:47 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:00:47.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:47 smithi107 ceph-mon[102221]: pgmap v502: 89 pgs: 17 peering, 72 active+clean; 582 KiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T10:00:47.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:47 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1351567080' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:00:47.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:47 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:00:47.365 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:47.390 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:47.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:47.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:47.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:47.514 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:47.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:47.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:47.616 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:47.641 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:47.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:47.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:47 smithi129 ceph-mon[88471]: pgmap v502: 89 pgs: 17 peering, 72 active+clean; 582 KiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T10:00:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:47 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1351567080' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:00:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:47 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:00:47.743 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:47.769 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:47.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:47.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:47.871 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:47.896 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:47.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:47.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:47.997 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:48.022 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:48.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:48.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:48.124 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:48.174 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:48.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:48.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:48.276 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:48.301 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:48.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:48.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:48.403 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:48.428 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:48.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:48.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:48.509 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:00:48.510 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:00:48.510 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-09-21T10:00:48.510 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-09-21T10:00:48.510 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:00:48.510 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:48 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:00:48.510 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:48 smithi107 ceph-mon[102221]: osdmap e146: 8 total, 8 up, 7 in 2024-09-21T10:00:48.511 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:48 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:00:48.511 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:48 smithi107 ceph-mon[105022]: osdmap e146: 8 total, 8 up, 7 in 2024-09-21T10:00:48.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:48.551 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:48.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:48.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:48.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:48.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:48 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:00:48.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:48 smithi129 ceph-mon[88471]: osdmap e146: 8 total, 8 up, 7 in 2024-09-21T10:00:48.675 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:48.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:48.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:48.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:48.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:48.798 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:48.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:48.899 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:48.925 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:48.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:48.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:49.027 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:49.052 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:49.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:49.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:49.141 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:00:49.153 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:49.178 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:49.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:49.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:49.210 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:49.277 DEBUG:teuthology.orchestra.run.smithi107:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-21T10:00:49.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:49.281 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:00:49.281 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:00:49.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:49.302 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:49.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:49.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:49 smithi107 ceph-mon[102221]: pgmap v504: 89 pgs: 89 active+clean; 582 KiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 97 KiB/s, 1 keys/s, 2 objects/s recovering 2024-09-21T10:00:49.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:49 smithi107 ceph-mon[102221]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 5 pgs peering) 2024-09-21T10:00:49.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:49 smithi107 ceph-mon[102221]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:00:49.377 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:49 smithi107 ceph-mon[105022]: pgmap v504: 89 pgs: 89 active+clean; 582 KiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 97 KiB/s, 1 keys/s, 2 objects/s recovering 2024-09-21T10:00:49.377 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:49 smithi107 ceph-mon[105022]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 5 pgs peering) 2024-09-21T10:00:49.377 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:49 smithi107 ceph-mon[105022]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:00:49.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:00:49.407 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:00:49.407 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:00:49.407 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-21T10:00:49.407 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:00:49.407 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:00:49.407 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-21T10:00:49.424 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:49.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:49.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:00:49.464 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:hints = 1 2024-09-21T10:00:49.464 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:00:49.464 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object prefix: benchmark_data_smithi107_152010 2024-09-21T10:00:49.464 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:00:49.464 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:00:49.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:49.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:49.574 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:49.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:49 smithi129 ceph-mon[88471]: pgmap v504: 89 pgs: 89 active+clean; 582 KiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 97 KiB/s, 1 keys/s, 2 objects/s recovering 2024-09-21T10:00:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:49 smithi129 ceph-mon[88471]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 5 pgs peering) 2024-09-21T10:00:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:49 smithi129 ceph-mon[88471]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:00:49.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:49.716 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:49.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:49.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:49.757 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:00:49.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:49.854 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:49.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:49.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:49.918 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:00:49.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:50.003 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:50.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:50.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:50.085 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:50.105 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:50.136 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:50.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:50.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:50.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:50.282 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:50.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:50.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:50.384 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:50.415 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:50.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:50.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:50.465 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1 16 3306 3290 205.613 205.625 0.00567749 0.00484753 2024-09-21T10:00:50.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:50.555 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:50.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:50.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:50.585 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:00:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:00:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:50.656 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:50.683 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:50.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:50.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:50.785 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:50.814 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:50.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:50.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:00:50.961 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:50.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:50.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:00:51.063 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:51.091 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:51.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:51.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:51.193 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:51.221 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:51.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:51.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:51.323 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:51.351 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:51.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:51.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:51.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:51 smithi129 ceph-mon[88471]: pgmap v505: 89 pgs: 89 active+clean; 9.0 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 1.1 MiB/s wr, 16 op/s; 73 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:00:51.452 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:51.465 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2 16 6570 6554 204.801 204 0.00323353 0.00487334 2024-09-21T10:00:51.493 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:51.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:51.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:51.595 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:51.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:51 smithi107 ceph-mon[102221]: pgmap v505: 89 pgs: 89 active+clean; 9.0 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 1.1 MiB/s wr, 16 op/s; 73 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:00:51.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:51 smithi107 ceph-mon[105022]: pgmap v505: 89 pgs: 89 active+clean; 9.0 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 1.1 MiB/s wr, 16 op/s; 73 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:00:51.628 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:51.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:51.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:51.730 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:51.745 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:00:51.745 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:00:51.745 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:00:51.761 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:51.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:51.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:51.863 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:51.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:51.889 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:51.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:51.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:52.030 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:52.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:52.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:52.050 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:00:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:00:51] "GET /metrics HTTP/1.1" 200 36442 "" "Prometheus/2.51.0" 2024-09-21T10:00:52.119 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:00:52.133 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:52.157 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:52.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:52.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:52.258 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:52.286 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:52.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:52.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:52.388 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:52.399 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:00:52.416 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:52.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:52.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:52.465 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3 16 9410 9394 195.695 177.5 0.00529107 0.00509986 2024-09-21T10:00:52.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:52.555 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:52.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:52.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:52.657 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:52.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:52 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/352577624' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:00:52.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:52 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:00:52.688 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:52.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:52.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:52.789 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:00:52.828 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:52.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:52.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:00:52.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:52 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/352577624' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:00:52.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:52 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:00:52.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:52 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/352577624' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:00:52.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:52 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:00:52.930 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:52.959 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:52.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:52.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:53.061 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:53.088 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:53.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:53.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:53.190 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:53.227 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:53.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:53.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:53.329 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:53.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:53.360 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:53.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:53.461 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:53.465 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 4 16 12426 12410 193.891 188.5 0.00667163 0.00515004 2024-09-21T10:00:53.491 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:53.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:53.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:53.593 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:53.621 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:53.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:53.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:53 smithi129 ceph-mon[88471]: pgmap v506: 89 pgs: 89 active+clean; 76 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 8.3 MiB/s wr, 133 op/s; 65 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:00:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:53 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:00:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:53 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:00:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:53 smithi129 ceph-mon[88471]: osdmap e147: 8 total, 8 up, 7 in 2024-09-21T10:00:53.723 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:53.750 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:53.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:53.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:53.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:53 smithi107 ceph-mon[102221]: pgmap v506: 89 pgs: 89 active+clean; 76 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 8.3 MiB/s wr, 133 op/s; 65 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:00:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:53 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:00:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:53 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:00:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:53 smithi107 ceph-mon[102221]: osdmap e147: 8 total, 8 up, 7 in 2024-09-21T10:00:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:53 smithi107 ceph-mon[105022]: pgmap v506: 89 pgs: 89 active+clean; 76 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 8.3 MiB/s wr, 133 op/s; 65 KiB/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:00:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:53 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:00:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:53 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:00:53.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:53 smithi107 ceph-mon[105022]: osdmap e147: 8 total, 8 up, 7 in 2024-09-21T10:00:53.896 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:53.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:53.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:53.997 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:54.025 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:54.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:54.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:54.127 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:54.152 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:54.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:54.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:54.254 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:00:54.281 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:54.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:54.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:00:54.382 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:54.426 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:54.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:54.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:54.465 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 5 16 14710 14694 183.66 142.75 0.0094771 0.00543662 2024-09-21T10:00:54.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:54.565 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:54.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:54.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:54.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:54.690 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:54.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:54.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:54.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:00:54.831 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:54.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:54.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:00:54.933 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:54.958 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:54.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:54.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:55.060 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:55.087 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:55.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:55.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:55.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:55.233 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:55.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:55.334 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:55.361 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:55.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:55.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:55.463 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:55.465 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 6 16 16786 16770 174.673 129.75 0.00853204 0.00571849 2024-09-21T10:00:55.489 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:55.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:55.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:55.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:55.618 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:55.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:55.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:55 smithi129 ceph-mon[88471]: pgmap v508: 89 pgs: 89 active+clean; 412 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 51 MiB/s wr, 823 op/s; 608 B/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:00:55.719 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:55.745 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:55.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:55.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:55.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:55.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:55 smithi107 ceph-mon[105022]: pgmap v508: 89 pgs: 89 active+clean; 412 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 51 MiB/s wr, 823 op/s; 608 B/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:00:55.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:55 smithi107 ceph-mon[102221]: pgmap v508: 89 pgs: 89 active+clean; 412 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 51 MiB/s wr, 823 op/s; 608 B/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:00:55.884 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:55.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:55.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:55.986 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:56.015 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:56.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:56.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:56.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:00:56.117 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:56.145 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:56.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:56.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:56.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:56.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:56.287 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:56.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:56.388 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:56.443 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:56.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:56.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:56.465 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 7 15 18782 18767 167.549 124.812 0.00961856 0.00596133 2024-09-21T10:00:56.544 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:56.566 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:56.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:56.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:56.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:56.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:56.702 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:56.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:56.803 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:56.831 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:56.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:56.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:56.933 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:56.959 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:56.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:56.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:57.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:57.099 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:57.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:57.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:57.200 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:00:57.237 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:57.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:57.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:00:57.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:00:57.376 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:57.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:57.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:00:57.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:57 smithi129 ceph-mon[88471]: pgmap v509: 89 pgs: 89 active+clean; 848 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 96 MiB/s wr, 1.53k op/s; 7 B/s, 0 objects/s recovering 2024-09-21T10:00:57.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:57 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:00:57.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:57 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:57.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:57 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:00:57.465 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 8 16 20645 20629 161.151 116.375 0.00469747 0.00619949 2024-09-21T10:00:57.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:00:57.513 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:57.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:57.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:00:57.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:57 smithi107 ceph-mon[102221]: pgmap v509: 89 pgs: 89 active+clean; 848 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 96 MiB/s wr, 1.53k op/s; 7 B/s, 0 objects/s recovering 2024-09-21T10:00:57.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:57 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:00:57.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:57 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:57.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:57 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:00:57.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:57 smithi107 ceph-mon[105022]: pgmap v509: 89 pgs: 89 active+clean; 848 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 96 MiB/s wr, 1.53k op/s; 7 B/s, 0 objects/s recovering 2024-09-21T10:00:57.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:57 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:00:57.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:57 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:00:57.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:57 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:00:57.615 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:57.642 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:57.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:57.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:57.744 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:57.746 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:00:57.746 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:00:57.746 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-09-21T10:00:57.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd in 7 2024-09-21T10:00:57.770 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:57.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:57.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:57.872 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:00:57.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:57.895 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:57.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:00:57.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:58.041 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:58.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:58.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:58.144 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:58.171 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:58.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:58.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:58.273 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:58.302 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:58.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:58.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:58.403 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:58.431 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:58.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:58.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:58.465 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 9 16 22620 22604 156.958 123.438 0.00252872 0.00634603 2024-09-21T10:00:58.533 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:00:58.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:58.555 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:58.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:00:58.656 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:58.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:58.684 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:58.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:58.785 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:58.813 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:58.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:58.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:58.915 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:58.943 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:58.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:58.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:59.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:59.080 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:59.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:59.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:59.181 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:59.207 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:59.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:59.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:00:59.309 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:59.336 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:59.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:59.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:59.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:00:59 smithi129 ceph-mon[88471]: pgmap v510: 89 pgs: 89 active+clean; 1.0 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-21T10:00:59.438 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:00:59.465 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:59.466 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 10 16 24561 24545 153.392 121.312 0.00913095 0.00651348 2024-09-21T10:00:59.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:59.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:00:59.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:59.590 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:59.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:59.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:59.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:00:59 smithi107 ceph-mon[102221]: pgmap v510: 89 pgs: 89 active+clean; 1.0 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-21T10:00:59.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:00:59 smithi107 ceph-mon[105022]: pgmap v510: 89 pgs: 89 active+clean; 1.0 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-21T10:00:59.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:00:59.727 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:00:59.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:59.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:00:59.829 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:00:59.860 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:59.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:59.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:00:59.962 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:00:59.991 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:00:59.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:00:59.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:00.093 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:00.122 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:00.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:00.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:00.224 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:00.259 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:00.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:00.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:00.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:00.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:00.388 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:00.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:00.465 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 11 16 26512 26496 150.532 121.938 0.00724083 0.00663752 2024-09-21T10:01:00.489 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:00.523 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:00.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:00.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:00.625 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:00.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:00.664 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:00.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:01:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:01:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:00.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:00.804 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:00.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:00.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:00.906 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:00.922 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:01:00.922 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-09-21T10:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-09-21T10:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.44375.0:73109 7.e 7:7a424784:::benchmark_data_smithi107_123142_object73108:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e106)", 2024-09-21T10:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:51:30.636317+0000", 2024-09-21T10:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 570.27983836199996, 2024-09-21T10:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1168578870000001, 2024-09-21T10:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:00.923 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.44375", 2024-09-21T10:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/926669403", 2024-09-21T10:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 73109 2024-09-21T10:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:51:30.636317+0000", 2024-09-21T10:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:51:30.636317+0000", 2024-09-21T10:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:51:30.636318+0000", 2024-09-21T10:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.344e-06 2024-09-21T10:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:51:30.636346+0000", 2024-09-21T10:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.747e-05 2024-09-21T10:01:00.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.926 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:00.926 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:51:30.636358+0000", 2024-09-21T10:01:00.926 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1883e-05 2024-09-21T10:01:00.926 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.926 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:00.926 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:51:30.636370+0000", 2024-09-21T10:01:00.926 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2632e-05 2024-09-21T10:01:00.926 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.926 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.926 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:00.926 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:51:30.636688+0000", 2024-09-21T10:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00031777500000000001 2024-09-21T10:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-09-21T10:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:51:30.636697+0000", 2024-09-21T10:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8899999999999996e-06 2024-09-21T10:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:51:30.639559+0000", 2024-09-21T10:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028618189999999998 2024-09-21T10:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.927 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:51:30.639571+0000", 2024-09-21T10:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1755e-05 2024-09-21T10:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:51:31.742571+0000", 2024-09-21T10:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1030001279999999 2024-09-21T10:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:51:31.743939+0000", 2024-09-21T10:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001367809 2024-09-21T10:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.928 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:51:31.744061+0000", 2024-09-21T10:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000122059 2024-09-21T10:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:51:31.747168+0000", 2024-09-21T10:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031069840000000001 2024-09-21T10:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:51:31.747307+0000", 2024-09-21T10:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013936800000000001 2024-09-21T10:01:00.929 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:51:31.752154+0000", 2024-09-21T10:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048467609999999998 2024-09-21T10:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:51:31.752225+0000", 2024-09-21T10:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1463000000000003e-05 2024-09-21T10:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:51:31.753175+0000", 2024-09-21T10:01:00.930 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00094974700000000003 2024-09-21T10:01:00.931 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:00.931 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:00.931 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:00.931 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.931 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.931 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:60903 8.c 8:3521a52a:::benchmark_data_smithi107_135256_object60902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T10:01:00.931 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:55:49.440786+0000", 2024-09-21T10:01:00.931 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 311.47536987299998, 2024-09-21T10:01:00.931 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.98623753599999997, 2024-09-21T10:01:00.931 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:00.931 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:00.931 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 60903 2024-09-21T10:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.440786+0000", 2024-09-21T10:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:00.932 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.440786+0000", 2024-09-21T10:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.440787+0000", 2024-09-21T10:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6780000000000001e-06 2024-09-21T10:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.440810+0000", 2024-09-21T10:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2419e-05 2024-09-21T10:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.933 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.440831+0000", 2024-09-21T10:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1409000000000001e-05 2024-09-21T10:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.934 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.440840+0000", 2024-09-21T10:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3739999999999998e-06 2024-09-21T10:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:00.935 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.440894+0000", 2024-09-21T10:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4148999999999999e-05 2024-09-21T10:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.440901+0000", 2024-09-21T10:01:00.936 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4449999999999999e-06 2024-09-21T10:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.422672+0000", 2024-09-21T10:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.98177129799999996 2024-09-21T10:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.937 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.422813+0000", 2024-09-21T10:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014044099999999999 2024-09-21T10:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 7", 2024-09-21T10:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.422953+0000", 2024-09-21T10:01:00.938 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014047500000000001 2024-09-21T10:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.425307+0000", 2024-09-21T10:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023539419999999999 2024-09-21T10:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.939 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.426879+0000", 2024-09-21T10:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0015720809999999999 2024-09-21T10:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:00.940 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.426943+0000", 2024-09-21T10:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3342000000000004e-05 2024-09-21T10:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.427023+0000", 2024-09-21T10:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0482999999999995e-05 2024-09-21T10:01:00.941 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:91040 8.c 8:385e0a70:::benchmark_data_smithi107_135256_object91039:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T10:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:11.570105+0000", 2024-09-21T10:01:00.942 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 289.34605050499999, 2024-09-21T10:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2792078780000002, 2024-09-21T10:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91040 2024-09-21T10:01:00.943 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.570105+0000", 2024-09-21T10:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:00.944 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.570105+0000", 2024-09-21T10:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.570107+0000", 2024-09-21T10:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0480000000000001e-06 2024-09-21T10:01:00.945 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.570129+0000", 2024-09-21T10:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1902000000000001e-05 2024-09-21T10:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.946 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.570143+0000", 2024-09-21T10:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.413e-05 2024-09-21T10:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:00.947 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.570152+0000", 2024-09-21T10:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5699999999999993e-06 2024-09-21T10:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.574728+0000", 2024-09-21T10:01:00.948 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045761559999999996 2024-09-21T10:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.574740+0000", 2024-09-21T10:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1965e-05 2024-09-21T10:01:00.949 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.565090+0000", 2024-09-21T10:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.99035070000000003 2024-09-21T10:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:00.950 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.565103+0000", 2024-09-21T10:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2675e-05 2024-09-21T10:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.833118+0000", 2024-09-21T10:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2680145729999999 2024-09-21T10:01:00.951 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.833305+0000", 2024-09-21T10:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018731800000000001 2024-09-21T10:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.952 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.833500+0000", 2024-09-21T10:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019540599999999999 2024-09-21T10:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.840924+0000", 2024-09-21T10:01:00.953 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0074235300000000002 2024-09-21T10:01:00.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.954 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.954 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:00.954 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.841020+0000", 2024-09-21T10:01:00.954 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5838e-05 2024-09-21T10:01:00.954 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.847877+0000", 2024-09-21T10:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0068567639999999996 2024-09-21T10:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.955 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.847952+0000", 2024-09-21T10:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5427999999999997e-05 2024-09-21T10:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:00.956 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.849313+0000", 2024-09-21T10:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0013608750000000001 2024-09-21T10:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.957 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:91061 8.c 8:3da63729:::benchmark_data_smithi107_135256_object91060:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T10:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:11.587362+0000", 2024-09-21T10:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 289.32879380999998, 2024-09-21T10:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2647233089999999, 2024-09-21T10:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91061 2024-09-21T10:01:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.587362+0000", 2024-09-21T10:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.587362+0000", 2024-09-21T10:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:00.959 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.587363+0000", 2024-09-21T10:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3260000000000001e-06 2024-09-21T10:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.587386+0000", 2024-09-21T10:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3260000000000001e-05 2024-09-21T10:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.587397+0000", 2024-09-21T10:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0740000000000001e-05 2024-09-21T10:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.960 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.587405+0000", 2024-09-21T10:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.481e-06 2024-09-21T10:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.587469+0000", 2024-09-21T10:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4486000000000006e-05 2024-09-21T10:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.587478+0000", 2024-09-21T10:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1619999999999997e-06 2024-09-21T10:01:00.961 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.565129+0000", 2024-09-21T10:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.97765050799999997 2024-09-21T10:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.565137+0000", 2024-09-21T10:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0770000000000008e-06 2024-09-21T10:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.962 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:00.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.837501+0000", 2024-09-21T10:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2723637750000001 2024-09-21T10:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.837668+0000", 2024-09-21T10:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016706100000000001 2024-09-21T10:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.964 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.837860+0000", 2024-09-21T10:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019234699999999999 2024-09-21T10:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.841028+0000", 2024-09-21T10:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031678869999999999 2024-09-21T10:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.841065+0000", 2024-09-21T10:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7001e-05 2024-09-21T10:01:00.965 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.850576+0000", 2024-09-21T10:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0095107420000000008 2024-09-21T10:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.850648+0000", 2024-09-21T10:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2016999999999998e-05 2024-09-21T10:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.852085+0000", 2024-09-21T10:01:00.966 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001437439 2024-09-21T10:01:00.967 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:00.967 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:00.967 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:00.967 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.967 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.967 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:123764 8.c 8:386542af:::benchmark_data_smithi107_135256_object123763:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-21T10:01:00.967 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:48.220309+0000", 2024-09-21T10:01:00.967 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 252.69584636799999, 2024-09-21T10:01:00.967 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.047124956, 2024-09-21T10:01:00.967 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:00.967 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:00.967 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:01:00.967 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:00.967 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:00.968 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:00.968 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 123764 2024-09-21T10:01:00.968 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.968 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:00.968 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.968 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:00.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:48.220309+0000", 2024-09-21T10:01:00.969 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:00.969 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.969 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.969 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:00.969 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:48.220309+0000", 2024-09-21T10:01:00.970 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:00.970 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.970 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.970 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:00.970 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:48.220310+0000", 2024-09-21T10:01:00.970 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0699999999999999e-06 2024-09-21T10:01:00.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.971 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.971 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:00.971 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:48.220329+0000", 2024-09-21T10:01:00.971 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8913999999999999e-05 2024-09-21T10:01:00.971 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.972 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:00.972 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:48.220345+0000", 2024-09-21T10:01:00.972 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6231000000000001e-05 2024-09-21T10:01:00.972 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.972 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.972 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:00.973 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:48.220353+0000", 2024-09-21T10:01:00.973 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5290000000000003e-06 2024-09-21T10:01:00.973 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.973 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.973 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:00.973 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:48.220438+0000", 2024-09-21T10:01:00.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.53e-05 2024-09-21T10:01:00.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.974 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.974 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:00.974 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:48.220951+0000", 2024-09-21T10:01:00.974 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051272899999999996 2024-09-21T10:01:00.974 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:01:00.975 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:48.221094+0000", 2024-09-21T10:01:00.975 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014285099999999999 2024-09-21T10:01:00.975 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.975 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.975 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:00.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:48.233425+0000", 2024-09-21T10:01:00.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012331474 2024-09-21T10:01:00.976 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.976 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.976 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:00.976 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:48.233438+0000", 2024-09-21T10:01:00.976 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2465e-05 2024-09-21T10:01:00.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.977 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.977 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:00.977 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:49.254899+0000", 2024-09-21T10:01:00.977 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0214611010000001 2024-09-21T10:01:00.977 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.977 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for degraded object", 2024-09-21T10:01:00.978 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:49.255552+0000", 2024-09-21T10:01:00.978 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00065344399999999999 2024-09-21T10:01:00.978 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.978 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.978 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:00.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:49.267369+0000", 2024-09-21T10:01:00.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01181634 2024-09-21T10:01:00.979 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.979 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.979 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:00.979 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:49.267434+0000", 2024-09-21T10:01:00.979 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5507999999999997e-05 2024-09-21T10:01:00.980 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:00.980 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:00.980 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:00.980 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.980 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.980 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:123775 8.c 8:375ece25:::benchmark_data_smithi107_135256_object123774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-21T10:01:00.980 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:48.238479+0000", 2024-09-21T10:01:00.981 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 252.67767644899999, 2024-09-21T10:01:00.981 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0292252019999999, 2024-09-21T10:01:00.981 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:00.981 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:00.981 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:00.981 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:00.981 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:00.981 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:00.981 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 123775 2024-09-21T10:01:00.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.982 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:00.982 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.982 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:00.982 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:48.238479+0000", 2024-09-21T10:01:00.982 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:00.982 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.983 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:00.983 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:48.238479+0000", 2024-09-21T10:01:00.983 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:00.983 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.983 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.983 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:00.984 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:48.238481+0000", 2024-09-21T10:01:00.984 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.485e-06 2024-09-21T10:01:00.984 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.984 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.984 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:00.984 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:48.238502+0000", 2024-09-21T10:01:00.985 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1178e-05 2024-09-21T10:01:00.985 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.985 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.985 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:00.985 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:48.238517+0000", 2024-09-21T10:01:00.985 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4831000000000001e-05 2024-09-21T10:01:00.986 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.986 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.986 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:00.986 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:48.238526+0000", 2024-09-21T10:01:00.986 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5719999999999992e-06 2024-09-21T10:01:00.986 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.987 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.987 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:00.987 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:48.238590+0000", 2024-09-21T10:01:00.987 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3843999999999994e-05 2024-09-21T10:01:00.987 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.987 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.988 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:00.988 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:48.238599+0000", 2024-09-21T10:01:00.988 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1919999999999994e-06 2024-09-21T10:01:00.988 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.988 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.989 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:00.989 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:49.255864+0000", 2024-09-21T10:01:00.989 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0172650409999999 2024-09-21T10:01:00.989 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.989 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.989 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:00.989 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:49.255998+0000", 2024-09-21T10:01:00.989 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013423600000000001 2024-09-21T10:01:00.989 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.990 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.990 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:01:00.990 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:49.256163+0000", 2024-09-21T10:01:00.990 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016465100000000001 2024-09-21T10:01:00.990 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.990 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.990 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:00.990 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:49.267004+0000", 2024-09-21T10:01:00.990 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010840862999999999 2024-09-21T10:01:00.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.991 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:00.991 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:49.267017+0000", 2024-09-21T10:01:00.991 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2969000000000001e-05 2024-09-21T10:01:00.991 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.991 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.991 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:00.991 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:49.267544+0000", 2024-09-21T10:01:00.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052755900000000004 2024-09-21T10:01:00.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.992 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.992 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:00.992 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:49.267608+0000", 2024-09-21T10:01:00.992 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3571999999999994e-05 2024-09-21T10:01:00.992 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.993 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.993 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:00.993 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:49.267704+0000", 2024-09-21T10:01:00.993 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6209e-05 2024-09-21T10:01:00.993 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:00.993 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:00.993 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:00.994 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.994 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.994 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45761.0:15005 9.5 9:ac20faa1:::benchmark_data_smithi107_152010_object15004:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:01:00.994 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:00:54.584433+0000", 2024-09-21T10:01:00.994 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 6.3317224559999996, 2024-09-21T10:01:00.994 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.097972090999999997, 2024-09-21T10:01:00.995 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:00.995 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:00.995 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:00.995 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:00.995 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45761", 2024-09-21T10:01:00.995 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:01:00.995 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15005 2024-09-21T10:01:00.995 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.995 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:00.995 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.995 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:00.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.584433+0000", 2024-09-21T10:01:00.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:00.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.996 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.996 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:00.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.584433+0000", 2024-09-21T10:01:00.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:00.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.996 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.996 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:00.996 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.584441+0000", 2024-09-21T10:01:00.996 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1040000000000007e-06 2024-09-21T10:01:00.996 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.996 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.997 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:00.997 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.584461+0000", 2024-09-21T10:01:00.997 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0313999999999999e-05 2024-09-21T10:01:00.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.997 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:00.997 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.584473+0000", 2024-09-21T10:01:00.997 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1618e-05 2024-09-21T10:01:00.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.997 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.997 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:00.997 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.584481+0000", 2024-09-21T10:01:00.997 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6109999999999996e-06 2024-09-21T10:01:00.997 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.998 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.998 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:00.998 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.587595+0000", 2024-09-21T10:01:00.998 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031146979999999999 2024-09-21T10:01:00.998 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.998 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.998 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:00.998 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.587790+0000", 2024-09-21T10:01:00.998 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019467400000000001 2024-09-21T10:01:00.998 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.998 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.998 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:01:00.998 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.587928+0000", 2024-09-21T10:01:00.999 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000137625 2024-09-21T10:01:00.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.999 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.999 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:00.999 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.589647+0000", 2024-09-21T10:01:00.999 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017197619999999999 2024-09-21T10:01:00.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.999 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.999 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:00.999 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590554+0000", 2024-09-21T10:01:00.999 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00090651399999999997 2024-09-21T10:01:00.999 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:00.999 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:00.999 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:01.000 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.682213+0000", 2024-09-21T10:01:01.000 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091658785000000007 2024-09-21T10:01:01.000 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.000 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.000 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:01.001 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.682361+0000", 2024-09-21T10:01:01.001 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000148627 2024-09-21T10:01:01.001 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.001 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.001 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:01.002 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.682405+0000", 2024-09-21T10:01:01.002 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3758999999999999e-05 2024-09-21T10:01:01.002 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.002 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:01.002 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.002 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.002 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.002 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45761.0:15011 9.9 9:92eeb3a4:::benchmark_data_smithi107_152010_object15010:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:01:01.002 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:00:54.586472+0000", 2024-09-21T10:01:01.002 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 6.3296834539999995, 2024-09-21T10:01:01.002 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.094112760000000004, 2024-09-21T10:01:01.002 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:01.002 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:01.003 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:01.003 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:01.003 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45761", 2024-09-21T10:01:01.003 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:01:01.003 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15011 2024-09-21T10:01:01.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.003 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:01.003 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.003 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:01.003 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.586472+0000", 2024-09-21T10:01:01.003 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.003 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.003 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.003 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:01.004 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.586472+0000", 2024-09-21T10:01:01.004 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.004 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.004 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:01.004 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.586473+0000", 2024-09-21T10:01:01.004 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5700000000000002e-07 2024-09-21T10:01:01.004 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.004 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:01.004 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.586493+0000", 2024-09-21T10:01:01.004 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0361e-05 2024-09-21T10:01:01.004 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.004 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.004 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:01.005 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.586500+0000", 2024-09-21T10:01:01.005 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3810000000000001e-06 2024-09-21T10:01:01.005 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.005 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.005 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:01.005 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.586506+0000", 2024-09-21T10:01:01.005 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5500000000000002e-06 2024-09-21T10:01:01.005 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.005 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.005 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:01.005 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.588885+0000", 2024-09-21T10:01:01.006 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00237944 2024-09-21T10:01:01.006 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.006 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.006 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:01.006 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.589006+0000", 2024-09-21T10:01:01.006 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012108999999999999 2024-09-21T10:01:01.006 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.006 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.006 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:01:01.006 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.589129+0000", 2024-09-21T10:01:01.007 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012266800000000001 2024-09-21T10:01:01.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.007 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:01.007 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590690+0000", 2024-09-21T10:01:01.007 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001560536 2024-09-21T10:01:01.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.007 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:01.007 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590729+0000", 2024-09-21T10:01:01.007 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9161e-05 2024-09-21T10:01:01.007 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.007 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.008 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:01.008 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.679607+0000", 2024-09-21T10:01:01.008 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.088878175000000004 2024-09-21T10:01:01.008 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.008 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.008 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:01.008 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.680544+0000", 2024-09-21T10:01:01.008 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093715200000000002 2024-09-21T10:01:01.008 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.008 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.008 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:01.008 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.680585+0000", 2024-09-21T10:01:01.008 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0689000000000003e-05 2024-09-21T10:01:01.008 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.009 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:01.009 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.009 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.009 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.009 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45761.0:15017 9.9 9:9ea14eb3:::benchmark_data_smithi107_152010_object15016:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:01:01.009 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:00:54.589617+0000", 2024-09-21T10:01:01.009 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 6.3265382920000004, 2024-09-21T10:01:01.009 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091144216, 2024-09-21T10:01:01.009 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:01.009 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:01.009 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:01.009 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:01.009 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45761", 2024-09-21T10:01:01.009 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:01:01.010 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15017 2024-09-21T10:01:01.010 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.010 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:01.010 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.010 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:01.010 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.589617+0000", 2024-09-21T10:01:01.010 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.010 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.010 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.010 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:01.010 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.589617+0000", 2024-09-21T10:01:01.010 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.010 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:01.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.589619+0000", 2024-09-21T10:01:01.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.381e-06 2024-09-21T10:01:01.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:01.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.589667+0000", 2024-09-21T10:01:01.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8226000000000002e-05 2024-09-21T10:01:01.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.011 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.011 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:01.011 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.589678+0000", 2024-09-21T10:01:01.011 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1661000000000001e-05 2024-09-21T10:01:01.011 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:01.012 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.589685+0000", 2024-09-21T10:01:01.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.861e-06 2024-09-21T10:01:01.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:01.012 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590757+0000", 2024-09-21T10:01:01.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010713560000000001 2024-09-21T10:01:01.012 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.012 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.012 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:01.012 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590882+0000", 2024-09-21T10:01:01.012 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012501499999999999 2024-09-21T10:01:01.013 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.013 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:01:01.013 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.591004+0000", 2024-09-21T10:01:01.013 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012271100000000001 2024-09-21T10:01:01.013 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.013 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:01.013 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.679595+0000", 2024-09-21T10:01:01.013 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.088590304999999994 2024-09-21T10:01:01.013 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.013 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.013 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:01.013 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.680596+0000", 2024-09-21T10:01:01.014 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010014990000000001 2024-09-21T10:01:01.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.014 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.014 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:01.014 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.680681+0000", 2024-09-21T10:01:01.014 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4336e-05 2024-09-21T10:01:01.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.014 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.014 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:01.014 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.680727+0000", 2024-09-21T10:01:01.014 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6451e-05 2024-09-21T10:01:01.014 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.014 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.015 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:01.015 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.680761+0000", 2024-09-21T10:01:01.015 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4413999999999999e-05 2024-09-21T10:01:01.015 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.015 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:01.015 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.015 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.015 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.015 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45761.0:15019 9.5 9:adffd20a:::benchmark_data_smithi107_152010_object15018:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:01:01.015 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:00:54.589945+0000", 2024-09-21T10:01:01.015 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 6.3262102599999999, 2024-09-21T10:01:01.015 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.097978265999999994, 2024-09-21T10:01:01.015 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:01.016 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:01.016 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:01.016 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:01.016 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45761", 2024-09-21T10:01:01.016 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:01:01.016 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15019 2024-09-21T10:01:01.016 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.016 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:01.016 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.017 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:01.017 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.589945+0000", 2024-09-21T10:01:01.017 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.017 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.017 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.017 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:01.017 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.589945+0000", 2024-09-21T10:01:01.018 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.018 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.018 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:01.018 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.589946+0000", 2024-09-21T10:01:01.018 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8000000000000004e-07 2024-09-21T10:01:01.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.018 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.018 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:01.018 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590026+0000", 2024-09-21T10:01:01.018 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0140000000000002e-05 2024-09-21T10:01:01.018 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.018 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.019 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:01.019 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590083+0000", 2024-09-21T10:01:01.019 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6499999999999998e-05 2024-09-21T10:01:01.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.019 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:01.019 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590089+0000", 2024-09-21T10:01:01.019 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.63e-06 2024-09-21T10:01:01.019 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.019 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:01.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590581+0000", 2024-09-21T10:01:01.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049187700000000005 2024-09-21T10:01:01.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:01.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590957+0000", 2024-09-21T10:01:01.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037534899999999997 2024-09-21T10:01:01.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.020 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:01:01.020 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.591112+0000", 2024-09-21T10:01:01.020 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015519899999999999 2024-09-21T10:01:01.020 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.020 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.021 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:01.021 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.682443+0000", 2024-09-21T10:01:01.021 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091331530999999994 2024-09-21T10:01:01.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.021 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:01.021 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.682479+0000", 2024-09-21T10:01:01.021 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5250000000000003e-05 2024-09-21T10:01:01.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.021 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.021 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:01.021 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.687595+0000", 2024-09-21T10:01:01.021 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0051165530000000002 2024-09-21T10:01:01.021 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.022 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:01.022 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.687885+0000", 2024-09-21T10:01:01.022 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00029007700000000002 2024-09-21T10:01:01.022 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.022 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:01.022 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.687924+0000", 2024-09-21T10:01:01.022 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8279999999999999e-05 2024-09-21T10:01:01.022 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.022 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:01.022 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.022 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.022 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.023 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.45761.0:15018 9.e e147/144 9:70f0954b:::benchmark_data_smithi107_152010_object15017:head v 147'949, mlcod=147'947)", 2024-09-21T10:01:01.023 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:00:54.590278+0000", 2024-09-21T10:01:01.023 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 6.3258777999999998, 2024-09-21T10:01:01.023 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.092867039999999998, 2024-09-21T10:01:01.023 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:01.023 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:01.023 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:01.023 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:01.023 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.023 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:01.023 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590278+0000", 2024-09-21T10:01:01.023 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.023 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.023 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.024 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:01.024 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590278+0000", 2024-09-21T10:01:01.024 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.024 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:01.024 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590278+0000", 2024-09-21T10:01:01.024 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4899999999999995e-07 2024-09-21T10:01:01.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.024 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.024 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:01.024 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.591162+0000", 2024-09-21T10:01:01.024 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00088335699999999998 2024-09-21T10:01:01.024 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.025 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:01.025 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.591173+0000", 2024-09-21T10:01:01.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1042e-05 2024-09-21T10:01:01.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.025 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:01.025 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.591179+0000", 2024-09-21T10:01:01.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3879999999999997e-06 2024-09-21T10:01:01.025 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.025 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.025 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:01.025 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.681393+0000", 2024-09-21T10:01:01.025 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.090213843000000002 2024-09-21T10:01:01.026 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.026 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.026 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:01.026 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.681417+0000", 2024-09-21T10:01:01.026 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4332e-05 2024-09-21T10:01:01.026 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.026 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.026 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:01.026 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.683099+0000", 2024-09-21T10:01:01.026 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0016819129999999999 2024-09-21T10:01:01.026 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.026 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.026 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:01.026 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.683145+0000", 2024-09-21T10:01:01.027 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5615999999999998e-05 2024-09-21T10:01:01.027 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.027 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:01.027 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.027 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.027 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.027 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45761.0:15022 9.b 9:dded4e04:::benchmark_data_smithi107_152010_object15021:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:01:01.027 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:00:54.590743+0000", 2024-09-21T10:01:01.027 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 6.3254126509999997, 2024-09-21T10:01:01.027 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.092743300000000001, 2024-09-21T10:01:01.027 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:01.027 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:01.027 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:01.028 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:01.028 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45761", 2024-09-21T10:01:01.028 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:01:01.028 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 15022 2024-09-21T10:01:01.028 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.028 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:01.028 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:01.029 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590743+0000", 2024-09-21T10:01:01.029 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.029 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.029 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.029 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:01.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590743+0000", 2024-09-21T10:01:01.030 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.030 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.030 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.030 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:01.030 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590744+0000", 2024-09-21T10:01:01.030 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9100000000000002e-07 2024-09-21T10:01:01.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.031 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:01.031 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590765+0000", 2024-09-21T10:01:01.031 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1274e-05 2024-09-21T10:01:01.031 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.031 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.032 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:01.032 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590774+0000", 2024-09-21T10:01:01.032 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4129999999999995e-06 2024-09-21T10:01:01.032 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.032 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.032 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:01.032 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590782+0000", 2024-09-21T10:01:01.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1149999999999996e-06 2024-09-21T10:01:01.033 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.033 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:01.033 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590848+0000", 2024-09-21T10:01:01.033 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6104000000000003e-05 2024-09-21T10:01:01.033 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.033 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:01.034 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.590956+0000", 2024-09-21T10:01:01.034 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108541 2024-09-21T10:01:01.034 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.034 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.034 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:01:01.034 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.591118+0000", 2024-09-21T10:01:01.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016140600000000001 2024-09-21T10:01:01.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.035 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.035 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:01.035 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.681595+0000", 2024-09-21T10:01:01.035 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.090476931999999996 2024-09-21T10:01:01.035 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.036 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.036 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:01.036 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.681605+0000", 2024-09-21T10:01:01.036 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.012e-05 2024-09-21T10:01:01.036 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.036 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.036 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:01.037 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.681637+0000", 2024-09-21T10:01:01.037 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2784000000000003e-05 2024-09-21T10:01:01.037 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.037 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.037 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:01.037 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.683446+0000", 2024-09-21T10:01:01.037 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001808891 2024-09-21T10:01:01.038 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.038 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.038 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:01.038 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.683486+0000", 2024-09-21T10:01:01.038 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9829000000000001e-05 2024-09-21T10:01:01.038 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.038 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:01.039 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.039 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.039 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.45761.0:15005 9.5 e147/144 ondisk, result = 0)", 2024-09-21T10:01:01.039 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:00:54.592285+0000", 2024-09-21T10:01:01.039 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 6.3238709179999999, 2024-09-21T10:01:01.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.090126215999999995, 2024-09-21T10:01:01.040 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:01.040 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:01.040 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:01:01.040 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:01.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.040 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:01.041 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.592285+0000", 2024-09-21T10:01:01.041 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.041 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.041 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.041 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:01.041 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.592285+0000", 2024-09-21T10:01:01.041 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.042 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:01.042 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.592285+0000", 2024-09-21T10:01:01.042 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2999999999999999e-07 2024-09-21T10:01:01.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.043 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:01.043 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.592295+0000", 2024-09-21T10:01:01.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4290000000000007e-06 2024-09-21T10:01:01.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.044 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:01.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:01.046 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:01.046 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.679097+0000", 2024-09-21T10:01:01.046 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.086802351 2024-09-21T10:01:01.046 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.046 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:01.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.679118+0000", 2024-09-21T10:01:01.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0661000000000001e-05 2024-09-21T10:01:01.047 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:01.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.682201+0000", 2024-09-21T10:01:01.048 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003083227 2024-09-21T10:01:01.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.048 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:01.048 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.682208+0000", 2024-09-21T10:01:01.048 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7870000000000004e-06 2024-09-21T10:01:01.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.049 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.049 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:01.049 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.682411+0000", 2024-09-21T10:01:01.049 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000202931 2024-09-21T10:01:01.049 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.049 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:01.049 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.049 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.050 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.050 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.45761.0:15022 9.b e147/141 ondisk, result = 0)", 2024-09-21T10:01:01.050 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:00:54.592331+0000", 2024-09-21T10:01:01.050 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 6.32382417, 2024-09-21T10:01:01.050 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091167279000000004, 2024-09-21T10:01:01.050 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:01.050 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:01.051 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:01:01.051 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:01.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:01.051 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.592331+0000", 2024-09-21T10:01:01.051 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.052 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:01.052 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.592331+0000", 2024-09-21T10:01:01.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.052 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.052 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.052 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:01.052 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.592332+0000", 2024-09-21T10:01:01.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3799999999999996e-07 2024-09-21T10:01:01.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.053 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:01.053 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.592344+0000", 2024-09-21T10:01:01.053 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1918e-05 2024-09-21T10:01:01.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.054 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.054 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:01.054 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.679434+0000", 2024-09-21T10:01:01.054 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.087089942000000004 2024-09-21T10:01:01.054 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.054 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.055 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:01.055 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.679441+0000", 2024-09-21T10:01:01.055 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.391e-06 2024-09-21T10:01:01.055 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.055 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.055 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:01.056 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.681628+0000", 2024-09-21T10:01:01.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002186927 2024-09-21T10:01:01.056 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.056 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.056 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:01.056 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.681633+0000", 2024-09-21T10:01:01.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.135e-06 2024-09-21T10:01:01.056 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.057 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:01.057 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:54.683499+0000", 2024-09-21T10:01:01.057 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001865228 2024-09-21T10:01:01.057 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.057 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:01.057 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.058 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45761.0:19604 9.9 9:950c32f6:::benchmark_data_smithi107_152010_object19603:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:01:01.058 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:00:56.946219+0000", 2024-09-21T10:01:01.058 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.9699368010000002, 2024-09-21T10:01:01.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.114790594, 2024-09-21T10:01:01.058 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:01.058 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:01.058 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:01.058 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:01.058 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45761", 2024-09-21T10:01:01.059 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:01:01.059 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 19604 2024-09-21T10:01:01.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.059 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:01.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:01.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.946219+0000", 2024-09-21T10:01:01.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.060 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:01.060 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.946219+0000", 2024-09-21T10:01:01.060 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:01.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.946220+0000", 2024-09-21T10:01:01.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1829999999999999e-06 2024-09-21T10:01:01.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:01.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.946241+0000", 2024-09-21T10:01:01.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0814000000000001e-05 2024-09-21T10:01:01.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:01.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.946251+0000", 2024-09-21T10:01:01.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0078000000000001e-05 2024-09-21T10:01:01.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:01.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.946258+0000", 2024-09-21T10:01:01.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6690000000000006e-06 2024-09-21T10:01:01.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:01.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.946382+0000", 2024-09-21T10:01:01.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012360100000000001 2024-09-21T10:01:01.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:01.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.946497+0000", 2024-09-21T10:01:01.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114836 2024-09-21T10:01:01.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:01:01.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.946616+0000", 2024-09-21T10:01:01.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011904 2024-09-21T10:01:01.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:01.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.947591+0000", 2024-09-21T10:01:01.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00097480500000000003 2024-09-21T10:01:01.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:01.065 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.948134+0000", 2024-09-21T10:01:01.065 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054373799999999999 2024-09-21T10:01:01.065 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.065 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.065 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:01.065 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:57.060757+0000", 2024-09-21T10:01:01.065 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.11262226 2024-09-21T10:01:01.065 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.065 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.065 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:01.065 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:57.060934+0000", 2024-09-21T10:01:01.065 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017742300000000001 2024-09-21T10:01:01.065 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.065 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.066 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:01.066 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:57.061009+0000", 2024-09-21T10:01:01.066 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5147000000000004e-05 2024-09-21T10:01:01.066 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.066 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:01.066 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.066 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.066 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.066 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45761.0:19609 9.9 9:9b4ea8c8:::benchmark_data_smithi107_152010_object19608:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:01:01.066 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:00:56.948786+0000", 2024-09-21T10:01:01.066 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 3.9673691760000001, 2024-09-21T10:01:01.066 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.11235871, 2024-09-21T10:01:01.066 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:01.067 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:01.067 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:01.067 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:01.067 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45761", 2024-09-21T10:01:01.067 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:01:01.067 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 19609 2024-09-21T10:01:01.067 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.067 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:01.067 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.067 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:01.067 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.948786+0000", 2024-09-21T10:01:01.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.067 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.067 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.067 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:01.068 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.948786+0000", 2024-09-21T10:01:01.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.068 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:01.068 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.948787+0000", 2024-09-21T10:01:01.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2900000000000003e-07 2024-09-21T10:01:01.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.068 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:01.068 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.948806+0000", 2024-09-21T10:01:01.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8799000000000001e-05 2024-09-21T10:01:01.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.069 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:01.069 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.948816+0000", 2024-09-21T10:01:01.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0074999999999999e-05 2024-09-21T10:01:01.069 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.069 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.069 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:01.069 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.948831+0000", 2024-09-21T10:01:01.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.525e-05 2024-09-21T10:01:01.069 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.069 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.069 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:01.069 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.948885+0000", 2024-09-21T10:01:01.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4293999999999999e-05 2024-09-21T10:01:01.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:01.070 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.948971+0000", 2024-09-21T10:01:01.070 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5062000000000007e-05 2024-09-21T10:01:01.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:01:01.070 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.949057+0000", 2024-09-21T10:01:01.070 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6316000000000005e-05 2024-09-21T10:01:01.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:01.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.949757+0000", 2024-09-21T10:01:01.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00069972299999999997 2024-09-21T10:01:01.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:01.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:56.950323+0000", 2024-09-21T10:01:01.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00056636999999999996 2024-09-21T10:01:01.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:01.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:57.061057+0000", 2024-09-21T10:01:01.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.110733793 2024-09-21T10:01:01.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:01.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:57.061109+0000", 2024-09-21T10:01:01.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2299999999999997e-05 2024-09-21T10:01:01.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:01.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:57.061145+0000", 2024-09-21T10:01:01.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5998999999999999e-05 2024-09-21T10:01:01.072 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.072 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:01.072 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.072 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.45761.0:22612 9.8 e147/141 9:1a0601a7:::benchmark_data_smithi107_152010_object22611:head v 147'1413, mlcod=147'1410)", 2024-09-21T10:01:01.073 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:00:58.435547+0000", 2024-09-21T10:01:01.073 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.4806082140000001, 2024-09-21T10:01:01.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.128022051, 2024-09-21T10:01:01.073 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:01.073 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:01.073 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:01.073 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:01.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.074 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:01.074 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:58.435547+0000", 2024-09-21T10:01:01.074 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.074 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.074 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.074 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:01.074 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:58.435547+0000", 2024-09-21T10:01:01.074 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.074 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.074 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.074 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:01.074 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:58.435548+0000", 2024-09-21T10:01:01.074 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4000000000000001e-07 2024-09-21T10:01:01.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:01.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:58.435569+0000", 2024-09-21T10:01:01.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1396e-05 2024-09-21T10:01:01.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:01.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:58.435580+0000", 2024-09-21T10:01:01.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0509999999999999e-05 2024-09-21T10:01:01.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:01.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:58.435586+0000", 2024-09-21T10:01:01.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1160000000000004e-06 2024-09-21T10:01:01.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:01.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:58.435650+0000", 2024-09-21T10:01:01.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4145000000000003e-05 2024-09-21T10:01:01.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:01.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:58.435668+0000", 2024-09-21T10:01:01.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7320999999999999e-05 2024-09-21T10:01:01.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:01.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:58.436702+0000", 2024-09-21T10:01:01.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010340919999999999 2024-09-21T10:01:01.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.077 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:01.077 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:58.563569+0000", 2024-09-21T10:01:01.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.12686773100000001 2024-09-21T10:01:01.077 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.077 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:01.077 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.077 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45761.0:24604 9.9 9:9076b24b:::benchmark_data_smithi107_152010_object24603:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:01:01.077 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:00:59.481828+0000", 2024-09-21T10:01:01.077 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.4343273810000001, 2024-09-21T10:01:01.078 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.097479664999999993, 2024-09-21T10:01:01.078 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:01.078 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:01.078 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:01.078 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:01.078 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45761", 2024-09-21T10:01:01.078 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:01:01.078 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 24604 2024-09-21T10:01:01.078 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.078 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:01.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.078 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:01.078 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.481828+0000", 2024-09-21T10:01:01.078 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:01.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.481828+0000", 2024-09-21T10:01:01.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:01.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.481830+0000", 2024-09-21T10:01:01.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4729999999999999e-06 2024-09-21T10:01:01.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:01.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.481853+0000", 2024-09-21T10:01:01.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3115000000000001e-05 2024-09-21T10:01:01.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:01.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.481865+0000", 2024-09-21T10:01:01.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.272e-05 2024-09-21T10:01:01.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:01.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.481874+0000", 2024-09-21T10:01:01.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.585e-06 2024-09-21T10:01:01.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:01.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.483501+0000", 2024-09-21T10:01:01.081 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001627273 2024-09-21T10:01:01.081 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.081 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.081 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:01.081 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.484054+0000", 2024-09-21T10:01:01.081 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055281999999999998 2024-09-21T10:01:01.081 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.081 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.081 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:01:01.081 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.484180+0000", 2024-09-21T10:01:01.081 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000125728 2024-09-21T10:01:01.081 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.081 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.081 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:01.081 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.487772+0000", 2024-09-21T10:01:01.082 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003592203 2024-09-21T10:01:01.082 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.082 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.082 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:01.082 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.487836+0000", 2024-09-21T10:01:01.082 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4215999999999996e-05 2024-09-21T10:01:01.082 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.082 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.082 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:01.082 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.579136+0000", 2024-09-21T10:01:01.082 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.091299569999999997 2024-09-21T10:01:01.082 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:01.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.579250+0000", 2024-09-21T10:01:01.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114589 2024-09-21T10:01:01.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:01.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.579308+0000", 2024-09-21T10:01:01.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7373000000000001e-05 2024-09-21T10:01:01.083 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.083 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:01.083 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.083 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45761.0:24610 9.5 9:a5b55614:::benchmark_data_smithi107_152010_object24609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:01:01.084 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:00:59.485185+0000", 2024-09-21T10:01:01.084 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.4309701910000001, 2024-09-21T10:01:01.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.12438764099999999, 2024-09-21T10:01:01.084 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:01.084 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:01.084 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:01.085 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:01.085 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45761", 2024-09-21T10:01:01.085 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:01:01.085 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 24610 2024-09-21T10:01:01.085 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.085 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:01.085 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.085 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:01.085 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.485185+0000", 2024-09-21T10:01:01.085 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.086 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.086 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.086 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:01.086 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.485185+0000", 2024-09-21T10:01:01.086 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.086 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.086 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.086 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:01.086 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.485187+0000", 2024-09-21T10:01:01.086 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3340000000000001e-06 2024-09-21T10:01:01.086 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.086 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.086 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:01.087 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.485218+0000", 2024-09-21T10:01:01.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1016000000000003e-05 2024-09-21T10:01:01.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.087 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.087 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:01.087 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.485227+0000", 2024-09-21T10:01:01.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8289999999999997e-06 2024-09-21T10:01:01.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.087 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.087 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:01.087 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.485232+0000", 2024-09-21T10:01:01.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1599999999999997e-06 2024-09-21T10:01:01.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.087 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:01.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.485422+0000", 2024-09-21T10:01:01.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019006000000000001 2024-09-21T10:01:01.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:01.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.485538+0000", 2024-09-21T10:01:01.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000116579 2024-09-21T10:01:01.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:01:01.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.485665+0000", 2024-09-21T10:01:01.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126737 2024-09-21T10:01:01.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:01.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.486745+0000", 2024-09-21T10:01:01.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001079899 2024-09-21T10:01:01.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:01.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.487412+0000", 2024-09-21T10:01:01.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00066691000000000005 2024-09-21T10:01:01.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:01.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.609463+0000", 2024-09-21T10:01:01.090 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.122051257 2024-09-21T10:01:01.090 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.090 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.090 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:01.090 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.609518+0000", 2024-09-21T10:01:01.090 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5389000000000001e-05 2024-09-21T10:01:01.090 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.090 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.090 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:01.090 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.609573+0000", 2024-09-21T10:01:01.090 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4471000000000001e-05 2024-09-21T10:01:01.090 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.090 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:01.091 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.091 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.091 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.091 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45761.0:24612 9.9 9:9e813913:::benchmark_data_smithi107_152010_object24611:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:01:01.091 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:00:59.485337+0000", 2024-09-21T10:01:01.091 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 1.4308180619999999, 2024-09-21T10:01:01.091 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.095816555999999997, 2024-09-21T10:01:01.091 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:01.091 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:01.091 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:01.091 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:01.091 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45761", 2024-09-21T10:01:01.091 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:01:01.091 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 24612 2024-09-21T10:01:01.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.092 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:01.092 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:01.092 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.485337+0000", 2024-09-21T10:01:01.092 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.092 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:01.092 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.485337+0000", 2024-09-21T10:01:01.092 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:01.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.092 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:01.093 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.485338+0000", 2024-09-21T10:01:01.093 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.92e-07 2024-09-21T10:01:01.093 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.093 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.093 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:01.093 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.485356+0000", 2024-09-21T10:01:01.093 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7961999999999998e-05 2024-09-21T10:01:01.093 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.093 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.093 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:01.093 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.485365+0000", 2024-09-21T10:01:01.093 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0210000000000005e-06 2024-09-21T10:01:01.093 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.093 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.094 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:01.094 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.485372+0000", 2024-09-21T10:01:01.094 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6460000000000004e-06 2024-09-21T10:01:01.094 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.094 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.094 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:01.094 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.486050+0000", 2024-09-21T10:01:01.094 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00067868400000000004 2024-09-21T10:01:01.094 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.094 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.094 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:01.094 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.486158+0000", 2024-09-21T10:01:01.094 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010720799999999999 2024-09-21T10:01:01.095 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.095 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.095 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:01:01.095 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.486282+0000", 2024-09-21T10:01:01.095 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000123857 2024-09-21T10:01:01.095 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.095 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.095 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:01.095 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.487874+0000", 2024-09-21T10:01:01.095 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001592046 2024-09-21T10:01:01.096 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.096 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.096 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:01.096 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.487885+0000", 2024-09-21T10:01:01.096 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1221e-05 2024-09-21T10:01:01.096 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.096 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.097 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:01.097 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.581026+0000", 2024-09-21T10:01:01.097 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.093140730000000005 2024-09-21T10:01:01.097 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.097 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.097 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:01.098 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.581113+0000", 2024-09-21T10:01:01.098 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7727999999999998e-05 2024-09-21T10:01:01.098 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:01.098 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:01.098 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:01.098 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:00:59.581154+0000", 2024-09-21T10:01:01.098 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0760999999999998e-05 2024-09-21T10:01:01.099 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.099 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:01.099 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.099 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:01.099 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:01.099 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:01:01.145 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:01.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:01.175 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:01.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:01.276 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:01.314 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:01.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:01.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:01.416 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:01.438 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:01.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:01.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:01.465 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 12 16 28524 28508 148.466 125.75 0.00353083 0.00672955 2024-09-21T10:01:01.539 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:01.561 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:01.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:01.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:01.588 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:01 smithi129 ceph-mon[88471]: pgmap v511: 89 pgs: 89 active+clean; 1.3 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-21T10:01:01.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:01 smithi107 ceph-mon[102221]: pgmap v511: 89 pgs: 89 active+clean; 1.3 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-21T10:01:01.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:01 smithi107 ceph-mon[105022]: pgmap v511: 89 pgs: 89 active+clean; 1.3 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 129 MiB/s wr, 2.06k op/s 2024-09-21T10:01:01.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:01.704 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:01.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:01.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:01.806 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:01:01.833 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:01.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:01.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:01:01.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:01.968 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:01.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:02.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:02.106 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:02.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:02.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:02.108 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:01:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:01:01] "GET /metrics HTTP/1.1" 200 36565 "" "Prometheus/2.51.0" 2024-09-21T10:01:02.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:02.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:02.247 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:02.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:02.348 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:02.377 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:02.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:02.466 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 13 16 30375 30359 145.943 115.688 0.00441412 0.00684677 2024-09-21T10:01:02.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:02.502 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:02.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:02.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:02.603 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:01:02.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:02.634 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:02.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:01:02.735 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:02.768 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:02.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:02.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:02.870 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:02.898 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:02.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:02.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:03.000 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:03.029 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:03.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:03.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:03.131 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:03.152 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:03.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:03.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:03.254 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:03.279 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:03.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:03.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:03.381 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:03.409 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:03.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:03.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:03.466 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 14 16 32126 32110 143.335 109.438 0.00517664 0.00697114 2024-09-21T10:01:03.510 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:03.544 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:03.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:03.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:03 smithi107 ceph-mon[102221]: pgmap v512: 89 pgs: 89 active+clean; 1.5 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-09-21T10:01:03.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:03 smithi107 ceph-mon[105022]: pgmap v512: 89 pgs: 89 active+clean; 1.5 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-09-21T10:01:03.646 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:03.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:03 smithi129 ceph-mon[88471]: pgmap v512: 89 pgs: 89 active+clean; 1.5 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-09-21T10:01:03.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:03.675 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:03.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:03.776 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:03.803 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:03.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:03.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:03.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:03.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:03.947 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:03.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:04.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:04.085 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:04.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:04.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:04.187 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:01:04.214 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:04.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:04.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:01:04.316 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:04.342 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:04.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:04.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:04.444 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:01:04.466 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 15 16 33622 33606 140.012 93.5 0.00463942 0.00713572 2024-09-21T10:01:04.471 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:04.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:04.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:01:04.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:04.610 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:04.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:04.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:04.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:04.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:04.763 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:04.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:04.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:01:04.865 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:04.893 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:04.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:04.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:04.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:05.038 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:05.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:05.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:05.140 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:05.177 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:05.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:05.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:05.278 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:05.292 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:01:05.297 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:01:05.301 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:05.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:05.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:05.402 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:01:05.428 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:05.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:05.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:01:05.466 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 16 16 35413 35397 138.251 111.938 0.0122477 0.00722732 2024-09-21T10:01:05.475 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:05 smithi107 ceph-mon[102221]: pgmap v513: 89 pgs: 89 active+clean; 1.7 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-09-21T10:01:05.475 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:05 smithi107 ceph-mon[105022]: pgmap v513: 89 pgs: 89 active+clean; 1.7 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-09-21T10:01:05.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:05.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:05.568 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:05.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:05.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:05 smithi129 ceph-mon[88471]: pgmap v513: 89 pgs: 89 active+clean; 1.7 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-09-21T10:01:05.669 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:05.694 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:05.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:05.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:05.796 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:05.823 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:05.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:05.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:05.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:05.965 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:05.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:05.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:06.067 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:06.095 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:06.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:06.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:06.197 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:06.233 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:06.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:06.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:06.250 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:01:06.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:06.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:06.375 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:06.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:06.467 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 17 16 37236 37220 136.821 113.938 0.00636079 0.00730335 2024-09-21T10:01:06.475 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:06.501 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:06.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:06.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:06.603 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:06.632 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:06.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:06.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:06.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:06.773 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:06.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:06.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:06.875 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:06.903 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:06.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:06.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:07.005 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:07.033 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:07.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:07.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:07.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:07.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:07.177 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:07.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:07.278 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:07.304 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:07.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:07.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:07.405 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:07.432 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:07.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:07.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:07.467 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 18 16 38961 38945 135.209 107.812 0.00956465 0.00738952 2024-09-21T10:01:07.485 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:01:07.486 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:01:07.529 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:01:07.530 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:01:07.533 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:07.572 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:07.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:07.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:07.575 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:01:07.575 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:01:07.586 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:07 smithi107 ceph-mon[102221]: pgmap v514: 89 pgs: 89 active+clean; 2.0 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 138 MiB/s wr, 2.20k op/s 2024-09-21T10:01:07.586 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:07 smithi107 ceph-mon[105022]: pgmap v514: 89 pgs: 89 active+clean; 2.0 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 138 MiB/s wr, 2.20k op/s 2024-09-21T10:01:07.621 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:01:07.621 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:01:07.659 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:01:07.659 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:01:07.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:07 smithi129 ceph-mon[88471]: pgmap v514: 89 pgs: 89 active+clean; 2.0 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 138 MiB/s wr, 2.20k op/s 2024-09-21T10:01:07.674 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:07.693 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:01:07.693 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:01:07.700 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:07.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:07.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:07.727 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:01:07.727 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:01:07.772 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:01:07.772 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:01:07.801 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:07.824 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:07.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:07.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:07.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:07.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:07.967 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:07.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:08.068 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:08.095 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:08.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:08.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:08.197 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:08.225 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:07 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a[102198]: 2024-09-21T10:01:07.938+0000 7f3189124640 -1 mon.a@0(leader).osd e147 definitely_dead 0 2024-09-21T10:01:08.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:08.235 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:08.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:08.337 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:08.358 INFO:teuthology.orchestra.run.smithi107.stderr:marked in osd.7. 2024-09-21T10:01:08.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:08.367 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:08.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:08.467 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 19 16 40401 40385 132.828 90 0.00228358 0.0074845 2024-09-21T10:01:08.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:08.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:08.491 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:08.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:08.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:08.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:08 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/985795657' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:01:08.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:08 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/813202300' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T10:01:08.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:08 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T10:01:08.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:08 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/985795657' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:01:08.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:08 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/813202300' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T10:01:08.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:08 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T10:01:08.614 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:08.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:08.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:08.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:08 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/985795657' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:01:08.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:08 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/813202300' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T10:01:08.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:08 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T10:01:08.716 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:08.742 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:08.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:08.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:08.843 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:08.869 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:08.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:08.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:08.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:08.992 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:08.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:08.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:09.094 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:09.115 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:09.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:09.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:09.217 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:09.229 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-09-21T10:01:09.242 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:09.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:09.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:09.344 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:09.368 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:09.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:09.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:09.467 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:01:09.466241+0000 min lat: 0.0012462 max lat: 0.240652 avg lat: 0.0074845 2024-09-21T10:01:09.467 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:01:09.467 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 20 16 40401 40385 126.187 0 - 0.0074845 2024-09-21T10:01:09.469 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:09.490 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:09.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:09.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:09.591 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:09 smithi107 ceph-mon[102221]: pgmap v515: 89 pgs: 89 active+clean; 2.2 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-09-21T10:01:09.591 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:09 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-21T10:01:09.591 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:09 smithi107 ceph-mon[102221]: osdmap e148: 8 total, 8 up, 8 in 2024-09-21T10:01:09.591 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:09 smithi107 ceph-mon[102221]: osdmap e149: 8 total, 8 up, 8 in 2024-09-21T10:01:09.591 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:09 smithi107 ceph-mon[105022]: pgmap v515: 89 pgs: 89 active+clean; 2.2 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-09-21T10:01:09.591 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:09 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-21T10:01:09.591 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:09 smithi107 ceph-mon[105022]: osdmap e148: 8 total, 8 up, 8 in 2024-09-21T10:01:09.591 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:09 smithi107 ceph-mon[105022]: osdmap e149: 8 total, 8 up, 8 in 2024-09-21T10:01:09.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:09.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:09.613 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:09.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:09.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:09 smithi129 ceph-mon[88471]: pgmap v515: 89 pgs: 89 active+clean; 2.2 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-09-21T10:01:09.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:09 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-21T10:01:09.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:09 smithi129 ceph-mon[88471]: osdmap e148: 8 total, 8 up, 8 in 2024-09-21T10:01:09.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:09 smithi129 ceph-mon[88471]: osdmap e149: 8 total, 8 up, 8 in 2024-09-21T10:01:09.714 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:09.736 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:09.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:09.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:09.838 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:09.859 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:09.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:09.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:09.961 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:09.983 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:09.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:09.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:10.085 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:01:10.106 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:10.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:10.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:01:10.208 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:10.230 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:10.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:10.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:10.332 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:10.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:10.368 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:10.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:10.467 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 21 16 40401 40385 120.178 0 - 0.0074845 2024-09-21T10:01:10.469 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:01:10.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:10.497 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:10.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:01:10.598 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:10.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:10.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:10.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:01:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:01:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:10.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:10.756 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:10.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:10.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:10.858 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:10.895 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:10.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:10.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:10.997 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:11.035 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:11.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:11.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:11.137 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:01:11.173 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:11.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:11.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:01:11.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:11.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:11.318 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:11.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:11.421 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:11.459 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:11.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:11.467 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 22 16 41507 41491 117.857 23.0417 0.00350477 0.00847838 2024-09-21T10:01:11.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:11 smithi107 ceph-mon[102221]: pgmap v518: 89 pgs: 2 remapped+peering, 7 peering, 80 active+clean; 2.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-09-21T10:01:11.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:11 smithi107 ceph-mon[102221]: osdmap e150: 8 total, 8 up, 8 in 2024-09-21T10:01:11.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:11.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:11 smithi107 ceph-mon[105022]: pgmap v518: 89 pgs: 2 remapped+peering, 7 peering, 80 active+clean; 2.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-09-21T10:01:11.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:11 smithi107 ceph-mon[105022]: osdmap e150: 8 total, 8 up, 8 in 2024-09-21T10:01:11.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:11.561 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:11.598 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:11.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:11.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:11.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:11 smithi129 ceph-mon[88471]: pgmap v518: 89 pgs: 2 remapped+peering, 7 peering, 80 active+clean; 2.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-09-21T10:01:11.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:11 smithi129 ceph-mon[88471]: osdmap e150: 8 total, 8 up, 8 in 2024-09-21T10:01:11.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:11.700 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:11.735 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:11.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:11.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:11.794 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:01:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:01:11] "GET /metrics HTTP/1.1" 200 36574 "" "Prometheus/2.51.0" 2024-09-21T10:01:11.837 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:01:11.876 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:11.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:11.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:01:11.977 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:12.012 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:01:12.012 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:01:12.018 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:12.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:12.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:12.120 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:12.166 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:12.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:12.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:12.268 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:01:12.305 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:12.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:12.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:01:12.334 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:01:12.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:12.446 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:12.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:12.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:12.467 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 23 16 42724 42708 116.04 76.0625 0.00743917 0.00861029 2024-09-21T10:01:12.548 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:12.592 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:12.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:12.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:12.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:12 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1505293541' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:01:12.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:12 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:01:12.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:12.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:12 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1505293541' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:01:12.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:12 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:01:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:12 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1505293541' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:01:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:12 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:01:12.694 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:01:12.731 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:12.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:12.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:01:12.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:01:12.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:12.889 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:12.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:12.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:12.990 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:13.027 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:13.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:13.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:13.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:01:13.130 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:13.166 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:13.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:13.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:13.268 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:13.316 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:13.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:13.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:13.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:13 smithi107 ceph-mon[102221]: pgmap v520: 89 pgs: 2 activating+undersized+degraded+remapped, 2 remapped+peering, 7 peering, 78 active+clean; 2.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 72 MiB/s wr, 1.16k op/s; 5005/119979 objects degraded (4.172%); 0 B/s, 0 objects/s recovering 2024-09-21T10:01:13.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:13 smithi107 ceph-mon[102221]: Health check failed: Degraded data redundancy: 5005/119979 objects degraded (4.172%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:13.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:13 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:01:13.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:13 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:01:13.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:13 smithi107 ceph-mon[102221]: osdmap e151: 8 total, 8 up, 8 in 2024-09-21T10:01:13.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:13 smithi107 ceph-mon[105022]: pgmap v520: 89 pgs: 2 activating+undersized+degraded+remapped, 2 remapped+peering, 7 peering, 78 active+clean; 2.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 72 MiB/s wr, 1.16k op/s; 5005/119979 objects degraded (4.172%); 0 B/s, 0 objects/s recovering 2024-09-21T10:01:13.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:13 smithi107 ceph-mon[105022]: Health check failed: Degraded data redundancy: 5005/119979 objects degraded (4.172%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:13.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:13 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:01:13.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:13 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:01:13.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:13 smithi107 ceph-mon[105022]: osdmap e151: 8 total, 8 up, 8 in 2024-09-21T10:01:13.418 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:13.462 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:13.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:13.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:13.470 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 24 16 43989 43973 114.499 79.0625 0.043044 0.00872581 2024-09-21T10:01:13.564 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:13.591 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:01:13.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:13.603 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:13.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:13.631 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:01:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:13 smithi129 ceph-mon[88471]: pgmap v520: 89 pgs: 2 activating+undersized+degraded+remapped, 2 remapped+peering, 7 peering, 78 active+clean; 2.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 72 MiB/s wr, 1.16k op/s; 5005/119979 objects degraded (4.172%); 0 B/s, 0 objects/s recovering 2024-09-21T10:01:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:13 smithi129 ceph-mon[88471]: Health check failed: Degraded data redundancy: 5005/119979 objects degraded (4.172%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:13 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:01:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:13 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:01:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:13 smithi129 ceph-mon[88471]: osdmap e151: 8 total, 8 up, 8 in 2024-09-21T10:01:13.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:13.746 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:13.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:13.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:13.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:13.881 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:13.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:13.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:13.983 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:01:14.023 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:14.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:14.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:01:14.125 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:01:14.160 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:14.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:14.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:01:14.229 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:01:14.230 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:01:14.230 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:01:14.230 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-21T10:01:14.230 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.7 2024-09-21T10:01:14.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:14.300 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:14.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:14.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:14.402 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:14.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:14.453 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:14.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:14.467 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 25 16 45197 45181 112.939 75.5 0.0193948 0.00884764 2024-09-21T10:01:14.554 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:14.598 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:14.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:14.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:14.669 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:01:14 smithi129 systemd[1]: Stopping Ceph osd.7 for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:01:14.700 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:14.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:14.741 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:14.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:14.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:14.874 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:14.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:14.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:14.977 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:15.002 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:15.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:15.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:15.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:15.138 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:15.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:15.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:15.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:15.273 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:15.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:15.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:15.375 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:15.412 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:15.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:15.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:15.467 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 26 16 46337 46321 111.335 71.25 0.0172415 0.00897577 2024-09-21T10:01:15.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:01:15.514 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:15.589 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:15.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:15.607 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:15 smithi129 ceph-mon[88471]: pgmap v522: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 activating+undersized+degraded+remapped, 2 remapped+peering, 80 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 12570/126153 objects degraded (9.964%); 4957/126153 objects misplaced (3.929%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:01:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:15 smithi107 ceph-mon[102221]: pgmap v522: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 activating+undersized+degraded+remapped, 2 remapped+peering, 80 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 12570/126153 objects degraded (9.964%); 4957/126153 objects misplaced (3.929%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:01:15.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:15 smithi107 ceph-mon[105022]: pgmap v522: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 activating+undersized+degraded+remapped, 2 remapped+peering, 80 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 12570/126153 objects degraded (9.964%); 4957/126153 objects misplaced (3.929%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:01:15.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:15.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:15.727 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:15.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:15.828 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:15.856 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:15.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:15.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:15.919 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:01:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[56850]: 2024-09-21T10:01:15.604+0000 7fcad778a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:01:15.919 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:01:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[56850]: 2024-09-21T10:01:15.604+0000 7fcad778a640 -1 osd.7 151 *** Got signal Terminated *** 2024-09-21T10:01:15.919 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:01:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[56850]: 2024-09-21T10:01:15.604+0000 7fcad778a640 -1 osd.7 151 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:01:15.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:15.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:15.994 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:15.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:16.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:16.131 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:16.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:16.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:16.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:16.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:16.270 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:16.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:16.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:16.400 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:16.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:16.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:16.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:16 smithi129 ceph-mon[88471]: osd.7 marked itself down and dead 2024-09-21T10:01:16.468 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 27 16 47414 47398 109.704 67.3125 0.0467302 0.00907173 2024-09-21T10:01:16.502 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:16.524 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:16.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:16.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:16.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:16 smithi107 ceph-mon[102221]: osd.7 marked itself down and dead 2024-09-21T10:01:16.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:16 smithi107 ceph-mon[105022]: osd.7 marked itself down and dead 2024-09-21T10:01:16.625 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:16.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:16.647 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:16.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:16.750 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:16.774 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:16.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:16.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:16.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:16.902 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:16.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:16.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:17.003 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:17.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:17.029 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:17.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:17.130 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:17.152 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:17.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:17.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:17.254 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:17.274 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:17.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:17.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:17.343 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:01:16 smithi129 podman[102392]: 2024-09-21 10:01:16.99208269 +0000 UTC m=+2.372343467 container died a32c9bd56124d929b9dd9d923f8c8d75217fca2cc31b3d7586559b399f339411 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=quincy-fbd1a6a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-21T10:01:17.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:17 smithi107 ceph-mon[102221]: pgmap v523: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 2.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 17663/134373 objects degraded (13.145%); 4957/134373 objects misplaced (3.689%); 890 KiB/s, 13 objects/s recovering 2024-09-21T10:01:17.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:17 smithi107 ceph-mon[102221]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:01:17.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:17 smithi107 ceph-mon[102221]: osdmap e152: 8 total, 7 up, 8 in 2024-09-21T10:01:17.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:17 smithi107 ceph-mon[105022]: pgmap v523: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 2.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 17663/134373 objects degraded (13.145%); 4957/134373 objects misplaced (3.689%); 890 KiB/s, 13 objects/s recovering 2024-09-21T10:01:17.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:17 smithi107 ceph-mon[105022]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:01:17.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:17 smithi107 ceph-mon[105022]: osdmap e152: 8 total, 7 up, 8 in 2024-09-21T10:01:17.376 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:17.397 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:17.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:17.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:17.468 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 28 16 47414 47398 105.786 0 - 0.00907173 2024-09-21T10:01:17.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:17.520 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:17.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:17.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:17.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:17.643 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:17.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:17.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:17.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:17 smithi129 ceph-mon[88471]: pgmap v523: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 2.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 17663/134373 objects degraded (13.145%); 4957/134373 objects misplaced (3.689%); 890 KiB/s, 13 objects/s recovering 2024-09-21T10:01:17.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:17 smithi129 ceph-mon[88471]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:01:17.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:17 smithi129 ceph-mon[88471]: osdmap e152: 8 total, 7 up, 8 in 2024-09-21T10:01:17.646 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:01:17 smithi129 podman[102392]: 2024-09-21 10:01:17.451323154 +0000 UTC m=+2.831583922 container cleanup a32c9bd56124d929b9dd9d923f8c8d75217fca2cc31b3d7586559b399f339411 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=quincy-fbd1a6a, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-21T10:01:17.646 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:01:17 smithi129 bash[102392]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7 2024-09-21T10:01:17.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:17.766 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:17.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:17.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:17.868 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:17.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:17.890 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:17.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:17.906 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:01:17 smithi129 podman[102631]: 2024-09-21 10:01:17.644655046 +0000 UTC m=+0.646576082 container remove a32c9bd56124d929b9dd9d923f8c8d75217fca2cc31b3d7586559b399f339411 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7, RELEASE=quincy-fbd1a6a, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-21T10:01:17.991 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:18.012 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:18.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:18.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:18.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:18.136 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:18.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:18.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:18.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:18.260 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:18.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:18.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:18.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:18.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:18.395 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:18.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:18.468 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 29 16 47487 47471 102.295 2.28125 0.0207791 0.00976946 2024-09-21T10:01:18.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:18.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:18.556 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:18.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:18.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:18 smithi107 ceph-mon[102221]: osdmap e153: 8 total, 7 up, 8 in 2024-09-21T10:01:18.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:18 smithi107 ceph-mon[105022]: osdmap e153: 8 total, 7 up, 8 in 2024-09-21T10:01:18.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:18 smithi129 ceph-mon[88471]: osdmap e153: 8 total, 7 up, 8 in 2024-09-21T10:01:18.702 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:18.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:18.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:18.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:18.846 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:01:18.874 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:18.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:18.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:18.884 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:01:18.884 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:01:18.977 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:19.001 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:19.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:19.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:19.032 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:01:19.032 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:01:19.070 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:01:19.070 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:01:19.104 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:19.106 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:01:19.106 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:01:19.132 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:19.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:19.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:19.138 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:01:19.139 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:01:19.172 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:01:19.173 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:01:19.204 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:01:19.205 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:01:19.206 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T10:01:19.234 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:19.273 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:19.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:19.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:19.375 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:19.401 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:01:19 smithi129 podman[102991]: 2024-09-21 10:01:19.02198432 +0000 UTC m=+0.020395035 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:01:19.401 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:01:19 smithi129 podman[102991]: 2024-09-21 10:01:19.136850575 +0000 UTC m=+0.135261284 container create 9cf2a5c18020cb16322d0a3166a51efb094934063949dc284431e8f022bdbe3c (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:01:19.401 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:19 smithi129 ceph-mon[88471]: pgmap v526: 89 pgs: 2 remapped+peering, 3 active+undersized, 9 peering, 1 active+recovery_wait+remapped, 4 stale+active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded, 62 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 62 MiB/s wr, 985 op/s; 15156/137721 objects degraded (11.005%); 2501/137721 objects misplaced (1.816%); 2.6 MiB/s, 40 objects/s recovering 2024-09-21T10:01:19.401 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:19 smithi129 ceph-mon[88471]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:01:19.401 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:19 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15156/137721 objects degraded (11.005%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:19.401 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:19 smithi129 ceph-mon[88471]: osdmap e154: 8 total, 7 up, 8 in 2024-09-21T10:01:19.401 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:19 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4258946718' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:01:19.401 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:19 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:19.402 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:19 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:19.408 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:19.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:19.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:19.468 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 30 16 48564 48548 101.129 67.3125 0.0109384 0.00988294 2024-09-21T10:01:19.510 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:19.534 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:19.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:19.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:19.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:19.669 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:01:19 smithi129 podman[102991]: 2024-09-21 10:01:19.399167446 +0000 UTC m=+0.397578149 container init 9cf2a5c18020cb16322d0a3166a51efb094934063949dc284431e8f022bdbe3c (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-21T10:01:19.669 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:01:19 smithi129 podman[102991]: 2024-09-21 10:01:19.407626125 +0000 UTC m=+0.406036837 container start 9cf2a5c18020cb16322d0a3166a51efb094934063949dc284431e8f022bdbe3c (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=quincy-fbd1a6a, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-21T10:01:19.669 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:01:19 smithi129 podman[102991]: 2024-09-21 10:01:19.523672241 +0000 UTC m=+0.522082947 container attach 9cf2a5c18020cb16322d0a3166a51efb094934063949dc284431e8f022bdbe3c (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-21T10:01:19.673 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:19.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:19.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:19.775 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:19.803 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:19.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:19.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:19 smithi107 ceph-mon[102221]: pgmap v526: 89 pgs: 2 remapped+peering, 3 active+undersized, 9 peering, 1 active+recovery_wait+remapped, 4 stale+active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded, 62 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 62 MiB/s wr, 985 op/s; 15156/137721 objects degraded (11.005%); 2501/137721 objects misplaced (1.816%); 2.6 MiB/s, 40 objects/s recovering 2024-09-21T10:01:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:19 smithi107 ceph-mon[102221]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:01:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:19 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15156/137721 objects degraded (11.005%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:19 smithi107 ceph-mon[102221]: osdmap e154: 8 total, 7 up, 8 in 2024-09-21T10:01:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:19 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4258946718' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:01:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:19 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:19 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:19.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:19 smithi107 ceph-mon[105022]: pgmap v526: 89 pgs: 2 remapped+peering, 3 active+undersized, 9 peering, 1 active+recovery_wait+remapped, 4 stale+active+clean, 6 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+degraded, 62 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 62 MiB/s wr, 985 op/s; 15156/137721 objects degraded (11.005%); 2501/137721 objects misplaced (1.816%); 2.6 MiB/s, 40 objects/s recovering 2024-09-21T10:01:19.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:19 smithi107 ceph-mon[105022]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:01:19.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:19 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15156/137721 objects degraded (11.005%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:19.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:19 smithi107 ceph-mon[105022]: osdmap e154: 8 total, 7 up, 8 in 2024-09-21T10:01:19.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:19 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4258946718' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:01:19.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:19 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:19.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:19 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:19.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:19.945 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:19.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:19.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:20.047 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:20.066 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:01:19 smithi129 conmon[103053]: conmon 9cf2a5c18020cb16322d : Failed to open cgroups file: /sys/fs/cgroup/machine.slice/libpod-9cf2a5c18020cb16322d0a3166a51efb094934063949dc284431e8f022bdbe3c.scope/memory.events 2024-09-21T10:01:20.066 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:01:19 smithi129 podman[102991]: 2024-09-21 10:01:19.678510243 +0000 UTC m=+0.676920955 container died 9cf2a5c18020cb16322d0a3166a51efb094934063949dc284431e8f022bdbe3c (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS) 2024-09-21T10:01:20.084 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:20.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:20.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:20.186 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:20.214 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:20.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:20.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:20.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:20.358 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:20.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:20.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:20.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:01:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:01:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:20.460 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:20.469 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 31 16 49894 49878 100.545 83.125 0.011173 0.00994065 2024-09-21T10:01:20.500 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:20.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:20.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:20.592 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.7.service' 2024-09-21T10:01:20.603 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:20.656 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:01:20.656 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-21T10:01:20.666 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:20.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:20.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:20.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:20.809 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:20.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:20.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:20.911 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:20.938 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:20.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:20.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:21.040 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:21.070 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:21.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:21.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:21.171 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:21.198 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:21.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:21.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:21.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:21.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:21.346 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:21.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:21.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:21 smithi129 ceph-mon[88471]: pgmap v528: 89 pgs: 2 active+recovery_wait+degraded+remapped, 2 remapped+peering, 11 active+undersized, 9 peering, 1 active+recovery_wait+remapped, 1 stale+active+clean, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 56 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 54 MiB/s wr, 870 op/s; 11374/141819 objects degraded (8.020%); 7746/141819 objects misplaced (5.462%); 1.7 MiB/s, 28 objects/s recovering 2024-09-21T10:01:21.447 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:21.469 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 32 16 51103 51087 99.7641 75.5625 0.00979119 0.0100176 2024-09-21T10:01:21.487 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:21.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:21.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:21.588 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:21.614 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:21.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:21.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:21.716 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:21.755 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:21.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:21.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:21.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:21 smithi107 ceph-mon[102221]: pgmap v528: 89 pgs: 2 active+recovery_wait+degraded+remapped, 2 remapped+peering, 11 active+undersized, 9 peering, 1 active+recovery_wait+remapped, 1 stale+active+clean, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 56 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 54 MiB/s wr, 870 op/s; 11374/141819 objects degraded (8.020%); 7746/141819 objects misplaced (5.462%); 1.7 MiB/s, 28 objects/s recovering 2024-09-21T10:01:21.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:21 smithi107 ceph-mon[105022]: pgmap v528: 89 pgs: 2 active+recovery_wait+degraded+remapped, 2 remapped+peering, 11 active+undersized, 9 peering, 1 active+recovery_wait+remapped, 1 stale+active+clean, 1 active+recovering+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+undersized+degraded, 56 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 54 MiB/s wr, 870 op/s; 11374/141819 objects degraded (8.020%); 7746/141819 objects misplaced (5.462%); 1.7 MiB/s, 28 objects/s recovering 2024-09-21T10:01:21.858 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:21.886 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:21.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:21.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:21.988 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:22.014 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:22.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:22.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:22.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:01:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:01:21] "GET /metrics HTTP/1.1" 200 36574 "" "Prometheus/2.51.0" 2024-09-21T10:01:22.116 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:22.143 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:22.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:22.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:22.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:22.282 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:22.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:22.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:22.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:22.420 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:22.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:22.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:22.470 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 33 16 52546 52530 99.4737 90.1875 0.00339309 0.010047 2024-09-21T10:01:22.523 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:22.564 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:22.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:22.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:22.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:22.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:22.706 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:22.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:22.807 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:22.835 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:22.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:22.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:22.936 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:22.969 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:22.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:22.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:23.070 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:23.098 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:23.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:23.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:23.200 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:23.227 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:23.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:23.328 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:23.355 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:23.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:23.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:23.456 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:23.469 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 34 16 53846 53830 98.9375 81.25 0.00861051 0.0101019 2024-09-21T10:01:23.482 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:23.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:23.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:23.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:23.629 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:23.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:23.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:23.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:23 smithi129 ceph-mon[88471]: pgmap v529: 89 pgs: 4 active+recovery_wait+degraded+remapped, 2 remapped+peering, 16 active+undersized, 9 peering, 1 active+recovery_wait+remapped, 2 active+recovering+degraded+remapped, 5 active+undersized+degraded, 50 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 38 MiB/s wr, 604 op/s; 7789/145260 objects degraded (5.362%); 13073/145260 objects misplaced (9.000%); 3.9 MiB/s, 64 objects/s recovering 2024-09-21T10:01:23.677 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:23 smithi107 ceph-mon[102221]: pgmap v529: 89 pgs: 4 active+recovery_wait+degraded+remapped, 2 remapped+peering, 16 active+undersized, 9 peering, 1 active+recovery_wait+remapped, 2 active+recovering+degraded+remapped, 5 active+undersized+degraded, 50 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 38 MiB/s wr, 604 op/s; 7789/145260 objects degraded (5.362%); 13073/145260 objects misplaced (9.000%); 3.9 MiB/s, 64 objects/s recovering 2024-09-21T10:01:23.677 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:23 smithi107 ceph-mon[105022]: pgmap v529: 89 pgs: 4 active+recovery_wait+degraded+remapped, 2 remapped+peering, 16 active+undersized, 9 peering, 1 active+recovery_wait+remapped, 2 active+recovering+degraded+remapped, 5 active+undersized+degraded, 50 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 38 MiB/s wr, 604 op/s; 7789/145260 objects degraded (5.362%); 13073/145260 objects misplaced (9.000%); 3.9 MiB/s, 64 objects/s recovering 2024-09-21T10:01:23.730 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:23.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:23.772 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:23.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:23.873 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:23.898 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:23.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:23.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:24.000 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:24.024 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:24.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:24.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:24.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:24.190 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:24.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:24.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:24.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:24.335 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:24.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:24.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:24.437 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:24.464 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:24.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:24.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:24.469 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 35 16 54971 54955 98.1196 70.3125 0.00438407 0.0101862 2024-09-21T10:01:24.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:24.602 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:24.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:24.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:24.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:24 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 7789/145260 objects degraded (5.362%), 11 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:24.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:24.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:24.747 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:24.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:24.765 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:24 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 7789/145260 objects degraded (5.362%), 11 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:24.765 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:24 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 7789/145260 objects degraded (5.362%), 11 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:24.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:24.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:24.884 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:24.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:24.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:25.019 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:25.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:25.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:25.120 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:25.162 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:25.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:25.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:25.263 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:25.289 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:25.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:25.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:25.391 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:25.416 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:25.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:25.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:25.470 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 36 16 56064 56048 97.2915 68.3125 0.00336841 0.0102605 2024-09-21T10:01:25.517 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:25.556 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:25.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:25.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:25.657 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-09-21T10:01:25.658 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:01:25.658 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-21T10:01:25.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:01:25.659 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:25.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:25 smithi129 ceph-mon[88471]: pgmap v530: 89 pgs: 2 active+clean+remapped, 5 active+recovery_wait+degraded+remapped, 21 active+undersized, 2 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 53 MiB/s wr, 844 op/s; 8619/154122 objects degraded (5.592%); 20319/154122 objects misplaced (13.184%); 9.3 MiB/s, 150 objects/s recovering 2024-09-21T10:01:25.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:25 smithi129 ceph-mon[88471]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:01:25.669 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:25 smithi107 ceph-mon[102221]: pgmap v530: 89 pgs: 2 active+clean+remapped, 5 active+recovery_wait+degraded+remapped, 21 active+undersized, 2 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 53 MiB/s wr, 844 op/s; 8619/154122 objects degraded (5.592%); 20319/154122 objects misplaced (13.184%); 9.3 MiB/s, 150 objects/s recovering 2024-09-21T10:01:25.669 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:25 smithi107 ceph-mon[102221]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:01:25.669 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:25 smithi107 ceph-mon[105022]: pgmap v530: 89 pgs: 2 active+clean+remapped, 5 active+recovery_wait+degraded+remapped, 21 active+undersized, 2 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 53 MiB/s wr, 844 op/s; 8619/154122 objects degraded (5.592%); 20319/154122 objects misplaced (13.184%); 9.3 MiB/s, 150 objects/s recovering 2024-09-21T10:01:25.669 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:25 smithi107 ceph-mon[105022]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:01:25.683 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:25.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:25.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:25.785 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:25.809 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:25.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:25.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:25.911 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:25.936 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:25.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:25.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:26.038 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:26.094 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:26.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:26.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:26.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:26.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:26.242 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:26.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:26.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:26.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:26.385 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:26.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:26.470 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 37 16 57218 57202 96.6113 72.125 0.0127025 0.0103455 2024-09-21T10:01:26.486 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:26.516 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:26.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:26.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:26.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:26.655 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:26.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:26.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:26.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:26.799 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:26.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:26.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:26.901 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:26.929 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:26.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:26.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:26.998 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:01:26.998 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:01:26.998 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-09-21T10:01:26.998 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-09-21T10:01:26.998 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:01:27.031 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:27.056 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:27.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:27.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:27.158 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:27.183 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:27.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:27.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:27.285 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:27.313 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:27.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:27.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:27 smithi107 ceph-mon[102221]: pgmap v531: 89 pgs: 3 active+clean+remapped, 5 active+recovery_wait+degraded+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 5546/163602 objects degraded (3.390%); 25029/163602 objects misplaced (15.299%); 10 MiB/s, 160 objects/s recovering 2024-09-21T10:01:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:27 smithi107 ceph-mon[105022]: pgmap v531: 89 pgs: 3 active+clean+remapped, 5 active+recovery_wait+degraded+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 5546/163602 objects degraded (3.390%); 25029/163602 objects misplaced (15.299%); 10 MiB/s, 160 objects/s recovering 2024-09-21T10:01:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:27.415 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:27.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:27 smithi129 ceph-mon[88471]: pgmap v531: 89 pgs: 3 active+clean+remapped, 5 active+recovery_wait+degraded+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 5546/163602 objects degraded (3.390%); 25029/163602 objects misplaced (15.299%); 10 MiB/s, 160 objects/s recovering 2024-09-21T10:01:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:27.439 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:27.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:27.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:27.470 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 38 16 58521 58505 96.2118 81.4375 0.00864544 0.0103875 2024-09-21T10:01:27.540 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:27.562 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:27.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:27.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:27.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:27.702 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:27.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:27.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:27.803 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:27.843 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:27.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:27.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:27.945 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:27.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:27.968 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:27.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:28.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:28.104 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:28.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:28.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:28.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:28.227 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:28.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:28.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:28.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:28.351 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:28.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:28.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:28.452 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:28.470 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 39 15 59541 59526 95.3809 63.8125 0.00906867 0.0104792 2024-09-21T10:01:28.479 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:28.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:28.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:28.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:28.620 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:28.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:28.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:28.722 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:28.750 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:28.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:28.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:28.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:28.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:28.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:28.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:29.022 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:29.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:29.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:29.109 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:01:29.303 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:01:29.304 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:29.346 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:29.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:29.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:29.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:29 smithi107 ceph-mon[102221]: pgmap v532: 89 pgs: 3 active+clean+remapped, 4 active+recovery_wait+degraded+remapped, 21 active+undersized, 2 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 65 MiB/s wr, 1.05k op/s; 8699/169119 objects degraded (5.144%); 22954/169119 objects misplaced (13.573%); 9.0 MiB/s, 144 objects/s recovering 2024-09-21T10:01:29.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:29 smithi107 ceph-mon[105022]: pgmap v532: 89 pgs: 3 active+clean+remapped, 4 active+recovery_wait+degraded+remapped, 21 active+undersized, 2 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 65 MiB/s wr, 1.05k op/s; 8699/169119 objects degraded (5.144%); 22954/169119 objects misplaced (13.573%); 9.0 MiB/s, 144 objects/s recovering 2024-09-21T10:01:29.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:29 smithi129 ceph-mon[88471]: pgmap v532: 89 pgs: 3 active+clean+remapped, 4 active+recovery_wait+degraded+remapped, 21 active+undersized, 2 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 65 MiB/s wr, 1.05k op/s; 8699/169119 objects degraded (5.144%); 22954/169119 objects misplaced (13.573%); 9.0 MiB/s, 144 objects/s recovering 2024-09-21T10:01:29.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:29.470 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:01:29.469424+0000 min lat: 0.0012462 max lat: 2.31158 avg lat: 0.0105375 2024-09-21T10:01:29.471 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:01:29.471 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 40 16 60730 60714 94.8521 74.25 0.0222146 0.0105375 2024-09-21T10:01:29.500 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:29.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:29.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:29.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:29.649 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:29.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:29.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:29.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:29.791 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:29.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:29.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:29.893 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:29.899 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:01:29.917 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:29.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:29.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:29.952 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:01:30.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:30.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:30.091 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:30.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:30.192 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:30.198 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:01:30.217 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:30.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:30.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:30.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:30.365 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:30.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:30.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:30.467 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:30.470 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 41 16 61816 61800 94.194 67.875 0.00762626 0.0106112 2024-09-21T10:01:30.492 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:30.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:30.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:30.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:30.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:30 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 8699/169119 objects degraded (5.144%), 15 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:30.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:30 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/60410070' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:01:30.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:30 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 8699/169119 objects degraded (5.144%), 15 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:30.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:30 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/60410070' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:01:30.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:30.638 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:30.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:30 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 8699/169119 objects degraded (5.144%), 15 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:30 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/60410070' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:01:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:01:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:01:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:30.739 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:30.765 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:30.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:30.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:30.867 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:30.893 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:30.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:30.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:30.995 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:31.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:31.038 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:31.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:31.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:31.173 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:31.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:31.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:31.225 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:31 smithi107 ceph-mon[105022]: pgmap v533: 89 pgs: 5 active+clean+remapped, 3 active+recovery_wait+degraded+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 4862/175719 objects degraded (2.767%); 27889/175719 objects misplaced (15.871%); 10 MiB/s, 165 objects/s recovering 2024-09-21T10:01:31.225 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:31 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:01:31.226 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:31 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/60410070' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:01:31.226 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:31 smithi107 ceph-mon[105022]: osdmap e155: 8 total, 7 up, 8 in 2024-09-21T10:01:31.226 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:31 smithi107 ceph-mon[102221]: pgmap v533: 89 pgs: 5 active+clean+remapped, 3 active+recovery_wait+degraded+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 4862/175719 objects degraded (2.767%); 27889/175719 objects misplaced (15.871%); 10 MiB/s, 165 objects/s recovering 2024-09-21T10:01:31.226 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:31 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:01:31.226 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:31 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/60410070' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:01:31.226 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:31 smithi107 ceph-mon[102221]: osdmap e155: 8 total, 7 up, 8 in 2024-09-21T10:01:31.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:31.304 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:01:31.311 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:31.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:31.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:31.414 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:31.440 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:31.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:31.471 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 42 16 63032 63016 93.758 76 0.00996132 0.0106588 2024-09-21T10:01:31.542 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:31.566 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:31.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:31.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:31.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:31 smithi129 ceph-mon[88471]: pgmap v533: 89 pgs: 5 active+clean+remapped, 3 active+recovery_wait+degraded+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 4862/175719 objects degraded (2.767%); 27889/175719 objects misplaced (15.871%); 10 MiB/s, 165 objects/s recovering 2024-09-21T10:01:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:31 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:01:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:31 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/60410070' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:01:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:31 smithi129 ceph-mon[88471]: osdmap e155: 8 total, 7 up, 8 in 2024-09-21T10:01:31.694 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:31.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:31.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:31.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:31.860 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:31.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:31.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:31.962 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:31.986 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:31.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:31.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:32.088 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:01:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:01:31] "GET /metrics HTTP/1.1" 200 36600 "" "Prometheus/2.51.0" 2024-09-21T10:01:32.114 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:32.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:32.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:32.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:32.255 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:32.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:32.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:32.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:32.400 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:32.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:32.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:32.472 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 43 16 64222 64206 93.3053 74.375 0.00779097 0.0107116 2024-09-21T10:01:32.502 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:32.552 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:32.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:32.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:32.653 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:32.658 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:01:32.677 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:32.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:32.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:32.727 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:01:32.779 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:32.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:32.830 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:32.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:32.930 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:32.958 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:32.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:32.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:33.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:01:33.059 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:33.084 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:33.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:33.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:33.186 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:33.234 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:33.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:33.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:33.320 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:33 smithi107 ceph-mon[102221]: pgmap v535: 89 pgs: 5 active+clean+remapped, 2 active+recovery_wait+degraded+remapped, 21 active+undersized, 2 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 75 MiB/s wr, 1.19k op/s; 8308/181050 objects degraded (4.589%); 25472/181050 objects misplaced (14.069%); 11 MiB/s, 179 objects/s recovering 2024-09-21T10:01:33.320 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:33 smithi107 ceph-mon[105022]: pgmap v535: 89 pgs: 5 active+clean+remapped, 2 active+recovery_wait+degraded+remapped, 21 active+undersized, 2 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 75 MiB/s wr, 1.19k op/s; 8308/181050 objects degraded (4.589%); 25472/181050 objects misplaced (14.069%); 11 MiB/s, 179 objects/s recovering 2024-09-21T10:01:33.336 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:33.391 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:33.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:33.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:33.473 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 44 16 65341 65325 92.7741 69.9375 0.0189524 0.0107734 2024-09-21T10:01:33.493 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:33.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:33.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:33.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:33.658 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:33.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:33 smithi129 ceph-mon[88471]: pgmap v535: 89 pgs: 5 active+clean+remapped, 2 active+recovery_wait+degraded+remapped, 21 active+undersized, 2 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 75 MiB/s wr, 1.19k op/s; 8308/181050 objects degraded (4.589%); 25472/181050 objects misplaced (14.069%); 11 MiB/s, 179 objects/s recovering 2024-09-21T10:01:33.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:33.803 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:33.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:33.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:33.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:33.944 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:33.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:33.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:34.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:34.083 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:34.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:34.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:34.184 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:34.206 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:34.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:34.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:34.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:34.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:34.346 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:34.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:34.447 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:34.473 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 45 16 66361 66345 92.1291 63.75 0.0105323 0.0108489 2024-09-21T10:01:34.474 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:34.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:34.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:34.576 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:34.602 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:34.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:34.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:34.704 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:34.725 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:34.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:34.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:34.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:34.857 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:34.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:34.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:34.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:34.997 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:34.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:34.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:35.099 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:35.125 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:35.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:35.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:35.227 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:35.269 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:35.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:35.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:35.371 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:35.397 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:35.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:35.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:35.473 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 46 16 67459 67443 91.618 68.625 0.00442423 0.0109096 2024-09-21T10:01:35.499 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:35.538 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:35.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:35.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:35.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:35 smithi107 ceph-mon[102221]: pgmap v536: 89 pgs: 6 active+clean+remapped, 1 active+recovery_wait+degraded+remapped, 21 active+undersized, 2 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s; 8386/189762 objects degraded (4.419%); 27142/189762 objects misplaced (14.303%); 9.4 MiB/s, 149 objects/s recovering 2024-09-21T10:01:35.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:35 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 8386/189762 objects degraded (4.419%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:35.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:35 smithi107 ceph-mon[105022]: pgmap v536: 89 pgs: 6 active+clean+remapped, 1 active+recovery_wait+degraded+remapped, 21 active+undersized, 2 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s; 8386/189762 objects degraded (4.419%); 27142/189762 objects misplaced (14.303%); 9.4 MiB/s, 149 objects/s recovering 2024-09-21T10:01:35.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:35 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 8386/189762 objects degraded (4.419%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:35.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:35 smithi129 ceph-mon[88471]: pgmap v536: 89 pgs: 6 active+clean+remapped, 1 active+recovery_wait+degraded+remapped, 21 active+undersized, 2 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s; 8386/189762 objects degraded (4.419%); 27142/189762 objects misplaced (14.303%); 9.4 MiB/s, 149 objects/s recovering 2024-09-21T10:01:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:35 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 8386/189762 objects degraded (4.419%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:35.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:35.680 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:35.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:35.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:35.826 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:35.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:35.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:35.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:35.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:35.964 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:35.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:36.065 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:36.090 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:36.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:36.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:36.151 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:01:36.153 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:01:36.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:36.231 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:36.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:36.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:36.333 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:36.358 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:36.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:36.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:36.460 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:36.473 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 47 16 68563 68547 91.1367 69 0.0121072 0.0109669 2024-09-21T10:01:36.486 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:36.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:36.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:36.588 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:36.630 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:36.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:36.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:36.732 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:36.768 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:36.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:36.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:36.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:36.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:36.906 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:36.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:36.972 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":536,"stamp":"2024-09-21T10:01:34.153533+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4144306125,"num_objects":63254,"num_object_clones":0,"num_object_copies":189762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8386,"num_objects_misplaced":27142,"num_objects_unfound":0,"num_objects_dirty":63254,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":63441,"num_write_kb":4048252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2781,"num_bytes_recovered":181938728,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63387,"ondisk_log_size":63387,"up":228,"acting":237,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":276,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15816764,"kb_used_data":11994544,"kb_used_omap":29,"kb_used_meta":3822114,"kb_avail":734144452,"statfs":{"total":767960285184,"available":751763918848,"internally_reserved":0,"allocated":12282413056,"data_stored":12276317749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30226,"internal_metadata":3913845230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,4,13,11,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":778567680,"num_objects":11880,"num_object_clones":0,"num_object_copies":35640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-233,"num_objects_misplaced":6823,"num_objects_unfound":0,"num_objects_dirty":11880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11880,"num_write_kb":760320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1499,"num_bytes_recovered":98238464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.006305"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":152,"reported_epoch":155,"state":"active+undersized","last_fresh":"2024-09-21T10:01:30.201540+0000","last_change":"2024-09-21T10:01:17.360671+0000","last_active":"2024-09-21T10:01:30.201540+0000","last_peered":"2024-09-21T10:01:30.201540+0000","last_clean":"2024-09-21T10:01:12.339813+0000","last_became_active":"2024-09-21T10:01:17.360671+0000","last_became_peered":"2024-09-21T10:01:17.360671+0000","last_unstale":"2024-09-21T10:01:30.201540+0000","last_undegraded":"2024-09-21T10:01:30.201540+0000","last_fullsized":"2024-09-21T10:01:17.352087+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:38.696084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":43,"reported_epoch":154,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:01:18.361929+0000","last_change":"2024-09-21T10:01:17.355722+0000","last_active":"2024-09-21T10:01:18.361929+0000","last_peered":"2024-09-21T10:01:18.361929+0000","last_clean":"2024-09-21T10:00:52.395654+0000","last_became_active":"2024-09-21T10:01:17.355722+0000","last_became_peered":"2024-09-21T10:01:17.355722+0000","last_unstale":"2024-09-21T10:01:18.361929+0000","last_undegraded":"2024-09-21T10:01:17.351745+0000","last_fullsized":"2024-09-21T10:01:17.351642+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:08:49.082950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":146,"reported_epoch":154,"state":"active+undersized","last_fresh":"2024-09-21T10:01:18.359357+0000","last_change":"2024-09-21T10:01:17.360872+0000","last_active":"2024-09-21T10:01:18.359357+0000","last_peered":"2024-09-21T10:01:18.359357+0000","last_clean":"2024-09-21T10:01:12.334087+0000","last_became_active":"2024-09-21T10:01:17.360872+0000","last_became_peered":"2024-09-21T10:01:17.360872+0000","last_unstale":"2024-09-21T10:01:18.359357+0000","last_undegraded":"2024-09-21T10:01:18.359357+0000","last_fullsized":"2024-09-21T10:01:17.356555+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:03.409955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":202,"reported_epoch":155,"state":"active+undersized","last_fresh":"2024-09-21T10:01:30.202017+0000","last_change":"2024-09-21T10:01:17.362044+0000","last_active":"2024-09-21T10:01:30.202017+0000","last_peered":"2024-09-21T10:01:30.202017+0000","last_clean":"2024-09-21T10:01:12.346595+0000","last_became_active":"2024-09-21T10:01:17.362044+0000","last_became_peered":"2024-09-21T10:01:17.362044+0000","last_unstale":"2024-09-21T10:01:30.202017+0000","last_undegraded":"2024-09-21T10:01:30.202017+0000","last_fullsized":"2024-09-21T10:01:17.354667+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:41.227793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":203,"reported_epoch":155,"state":"active+undersized","last_fresh":"2024-09-21T10:01:30.203261+0000","last_change":"2024-09-21T10:01:17.358516+0000","last_active":"2024-09-21T10:01:30.203261+0000","last_peered":"2024-09-21T10:01:30.203261+0000","last_clean":"2024-09-21T10:01:12.350555+0000","last_became_active":"2024-09-21T10:01:17.358516+0000","last_became_peered":"2024-09-21T10:01:17.358516+0000","last_unstale":"2024-09-21T10:01:30.203261+0000","last_undegraded":"2024-09-21T10:01:30.203261+0000","last_fullsized":"2024-09-21T10:01:17.354339+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:23.544961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":26,"reported_epoch":154,"state":"active+undersized","last_fresh":"2024-09-21T10:01:18.361357+0000","last_change":"2024-09-21T10:01:17.362650+0000","last_active":"2024-09-21T10:01:18.361357+0000","last_peered":"2024-09-21T10:01:18.361357+0000","last_clean":"2024-09-21T09:40:48.067317+0000","last_became_active":"2024-09-21T10:01:17.362650+0000","last_became_peered":"2024-09-21T10:01:17.362650+0000","last_unstale":"2024-09-21T10:01:18.361357+0000","last_undegraded":"2024-09-21T10:01:18.361357+0000","last_fullsized":"2024-09-21T10:01:17.351981+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:14.466960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":152,"reported_epoch":155,"state":"active+undersized","last_fresh":"2024-09-21T10:01:30.204721+0000","last_change":"2024-09-21T10:01:17.361530+0000","last_active":"2024-09-21T10:01:30.204721+0000","last_peered":"2024-09-21T10:01:30.204721+0000","last_clean":"2024-09-21T10:01:12.338661+0000","last_became_active":"2024-09-21T10:01:17.361530+0000","last_became_peered":"2024-09-21T10:01:17.361530+0000","last_unstale":"2024-09-21T10:01:30.204721+0000","last_undegraded":"2024-09-21T10:01:30.204721+0000","last_fullsized":"2024-09-21T10:01:17.354825+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:10.575192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":127,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.361323+0000","last_change":"2024-09-21T09:48:54.140948+0000","last_active":"2024-09-21T10:01:18.361323+0000","last_peered":"2024-09-21T10:01:18.361323+0000","last_clean":"2024-09-21T10:01:18.361323+0000","last_became_active":"2024-09-21T09:48:54.140641+0000","last_became_peered":"2024-09-21T09:48:54.140641+0000","last_unstale":"2024-09-21T10:01:18.361323+0000","last_undegraded":"2024-09-21T10:01:18.361323+0000","last_fullsized":"2024-09-21T10:01:18.361323+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:25.157599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"155'4124","reported_seq":5098,"reported_epoch":155,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:01:33.901243+0000","last_change":"2024-09-21T10:01:25.317266+0000","last_active":"2024-09-21T10:01:33.901243+0000","last_peered":"2024-09-21T10:01:33.901243+0000","last_clean":"2024-09-21T10:01:33.901243+0000","last_became_active":"2024-09-21T10:01:18.394618+0000","last_became_peered":"2024-09-21T10:01:18.394618+0000","last_unstale":"2024-09-21T10:01:33.901243+0000","last_undegraded":"2024-09-21T10:01:33.901243+0000","last_fullsized":"2024-09-21T10:01:33.901243+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":4124,"log_dups_size":0,"ondisk_log_size":4124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:20:01.567428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270270464,"num_objects":4124,"num_object_clones":0,"num_object_copies":12372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4124,"num_objects_unfound":0,"num_objects_dirty":4124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4124,"num_write_kb":263936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":458,"num_bytes_recovered":30015488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,1],"avail_no_missing":["0","1","5"],"object_location_counts":[{"shards":"0,1,5","objects":4124}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":41,"reported_epoch":154,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:01:18.361647+0000","last_change":"2024-09-21T10:01:17.362126+0000","last_active":"2024-09-21T10:01:18.361647+0000","last_peered":"2024-09-21T10:01:18.361647+0000","last_clean":"2024-09-21T10:00:52.396178+0000","last_became_active":"2024-09-21T10:01:17.362126+0000","last_became_peered":"2024-09-21T10:01:17.362126+0000","last_unstale":"2024-09-21T10:01:18.361647+0000","last_undegraded":"2024-09-21T10:01:17.352720+0000","last_fullsized":"2024-09-21T10:01:17.352607+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:15.152821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":129,"reported_epoch":154,"state":"active+undersized","last_fresh":"2024-09-21T10:01:18.358839+0000","last_change":"2024-09-21T10:01:17.360241+0000","last_active":"2024-09-21T10:01:18.358839+0000","last_peered":"2024-09-21T10:01:18.358839+0000","last_clean":"2024-09-21T10:01:12.331943+0000","last_became_active":"2024-09-21T10:01:17.360241+0000","last_became_peered":"2024-09-21T10:01:17.360241+0000","last_unstale":"2024-09-21T10:01:18.358839+0000","last_undegraded":"2024-09-21T10:01:18.358839+0000","last_fullsized":"2024-09-21T10:01:17.354797+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:56.749660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"154'3804","reported_seq":3841,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:29.721739+0000","last_change":"2024-09-21T10:00:39.617105+0000","last_active":"2024-09-21T10:01:29.721739+0000","last_peered":"2024-09-21T10:01:29.721739+0000","last_clean":"2024-09-21T10:01:29.721739+0000","last_became_active":"2024-09-21T10:00:39.616814+0000","last_became_peered":"2024-09-21T10:00:39.616814+0000","last_unstale":"2024-09-21T10:01:29.721739+0000","last_undegraded":"2024-09-21T10:01:29.721739+0000","last_fullsized":"2024-09-21T10:01:29.721739+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":3804,"log_dups_size":0,"ondisk_log_size":3804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:56.849964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249298944,"num_objects":3804,"num_object_clones":0,"num_object_copies":11412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3804,"num_write_kb":243456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":41,"reported_epoch":154,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:01:18.362185+0000","last_change":"2024-09-21T10:01:17.365156+0000","last_active":"2024-09-21T10:01:18.362185+0000","last_peered":"2024-09-21T10:01:18.362185+0000","last_clean":"2024-09-21T10:00:52.396279+0000","last_became_active":"2024-09-21T10:01:17.365156+0000","last_became_peered":"2024-09-21T10:01:17.365156+0000","last_unstale":"2024-09-21T10:01:18.362185+0000","last_undegraded":"2024-09-21T10:01:17.352823+0000","last_fullsized":"2024-09-21T10:01:17.352726+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:25.514616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":135,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.361261+0000","last_change":"2024-09-21T09:56:12.576294+0000","last_active":"2024-09-21T10:01:18.361261+0000","last_peered":"2024-09-21T10:01:18.361261+0000","last_clean":"2024-09-21T10:01:18.361261+0000","last_became_active":"2024-09-21T09:56:12.575928+0000","last_became_peered":"2024-09-21T09:56:12.575928+0000","last_unstale":"2024-09-21T10:01:18.361261+0000","last_undegraded":"2024-09-21T10:01:18.361261+0000","last_fullsized":"2024-09-21T10:01:18.361261+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:23.227435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"155'4015","reported_seq":5415,"reported_epoch":155,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:01:32.562152+0000","last_change":"2024-09-21T10:01:21.071741+0000","last_active":"2024-09-21T10:01:32.562152+0000","last_peered":"2024-09-21T10:01:32.562152+0000","last_clean":"2024-09-21T10:01:32.562152+0000","last_became_active":"2024-09-21T10:01:18.387241+0000","last_became_peered":"2024-09-21T10:01:18.387241+0000","last_unstale":"2024-09-21T10:01:32.562152+0000","last_undegraded":"2024-09-21T10:01:32.562152+0000","last_fullsized":"2024-09-21T10:01:32.562152+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":4015,"log_dups_size":0,"ondisk_log_size":4015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:02:42.497261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263127040,"num_objects":4015,"num_object_clones":0,"num_object_copies":12045,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4015,"num_objects_unfound":0,"num_objects_dirty":4015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4015,"num_write_kb":256960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":671,"num_bytes_recovered":43974656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3,2],"avail_no_missing":["5","2","3"],"object_location_counts":[{"shards":"2,3,5","objects":4015}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":41,"reported_epoch":154,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:01:18.359223+0000","last_change":"2024-09-21T10:01:17.360223+0000","last_active":"2024-09-21T10:01:18.359223+0000","last_peered":"2024-09-21T10:01:18.359223+0000","last_clean":"2024-09-21T10:00:52.397515+0000","last_became_active":"2024-09-21T10:01:17.360223+0000","last_became_peered":"2024-09-21T10:01:17.360223+0000","last_unstale":"2024-09-21T10:01:18.359223+0000","last_undegraded":"2024-09-21T10:01:17.357081+0000","last_fullsized":"2024-09-21T10:01:17.356981+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:14:39.079600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":145,"reported_epoch":154,"state":"active+undersized","last_fresh":"2024-09-21T10:01:18.361867+0000","last_change":"2024-09-21T10:01:17.364112+0000","last_active":"2024-09-21T10:01:18.361867+0000","last_peered":"2024-09-21T10:01:18.361867+0000","last_clean":"2024-09-21T10:01:12.345222+0000","last_became_active":"2024-09-21T10:01:17.364112+0000","last_became_peered":"2024-09-21T10:01:17.364112+0000","last_unstale":"2024-09-21T10:01:18.361867+0000","last_undegraded":"2024-09-21T10:01:18.361867+0000","last_fullsized":"2024-09-21T10:01:17.355079+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:43:09.114787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"155'3972","reported_seq":3994,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T10:01:32.566693+0000","last_change":"2024-09-21T10:00:39.614448+0000","last_active":"2024-09-21T10:01:32.566693+0000","last_peered":"2024-09-21T10:01:32.566693+0000","last_clean":"2024-09-21T10:01:32.566693+0000","last_became_active":"2024-09-21T10:00:39.614071+0000","last_became_peered":"2024-09-21T10:00:39.614071+0000","last_unstale":"2024-09-21T10:01:32.566693+0000","last_undegraded":"2024-09-21T10:01:32.566693+0000","last_fullsized":"2024-09-21T10:01:32.566693+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":3972,"log_dups_size":0,"ondisk_log_size":3972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:57.436342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260308992,"num_objects":3972,"num_object_clones":0,"num_object_copies":11916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3972,"num_write_kb":254208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":120,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.359065+0000","last_change":"2024-09-21T09:56:12.590219+0000","last_active":"2024-09-21T10:01:18.359065+0000","last_peered":"2024-09-21T10:01:18.359065+0000","last_clean":"2024-09-21T10:01:18.359065+0000","last_became_active":"2024-09-21T09:56:12.589946+0000","last_became_peered":"2024-09-21T09:56:12.589946+0000","last_unstale":"2024-09-21T10:01:18.359065+0000","last_undegraded":"2024-09-21T10:01:18.359065+0000","last_fullsized":"2024-09-21T10:01:18.359065+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:48.919929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":185,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T10:01:30.204870+0000","last_change":"2024-09-21T09:50:45.503336+0000","last_active":"2024-09-21T10:01:30.204870+0000","last_peered":"2024-09-21T10:01:30.204870+0000","last_clean":"2024-09-21T10:01:30.204870+0000","last_became_active":"2024-09-21T09:50:45.503045+0000","last_became_peered":"2024-09-21T09:50:45.503045+0000","last_unstale":"2024-09-21T10:01:30.204870+0000","last_undegraded":"2024-09-21T10:01:30.204870+0000","last_fullsized":"2024-09-21T10:01:30.204870+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:46.728347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"155'3792","reported_seq":4727,"reported_epoch":155,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:01:30.478935+0000","last_change":"2024-09-21T10:01:22.281434+0000","last_active":"2024-09-21T10:01:30.478935+0000","last_peered":"2024-09-21T10:01:30.478935+0000","last_clean":"2024-09-21T10:01:30.478935+0000","last_became_active":"2024-09-21T10:01:18.410689+0000","last_became_peered":"2024-09-21T10:01:18.410689+0000","last_unstale":"2024-09-21T10:01:30.478935+0000","last_undegraded":"2024-09-21T10:01:30.478935+0000","last_fullsized":"2024-09-21T10:01:30.478935+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":3792,"log_dups_size":0,"ondisk_log_size":3792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:48.612460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248512512,"num_objects":3792,"num_object_clones":0,"num_object_copies":11376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3792,"num_objects_unfound":0,"num_objects_dirty":3792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3792,"num_write_kb":242688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":439,"num_bytes_recovered":28770304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":3792}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":122,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T10:01:30.202124+0000","last_change":"2024-09-21T09:56:12.575897+0000","last_active":"2024-09-21T10:01:30.202124+0000","last_peered":"2024-09-21T10:01:30.202124+0000","last_clean":"2024-09-21T10:01:30.202124+0000","last_became_active":"2024-09-21T09:56:12.575532+0000","last_became_peered":"2024-09-21T09:56:12.575532+0000","last_unstale":"2024-09-21T10:01:30.202124+0000","last_undegraded":"2024-09-21T10:01:30.202124+0000","last_fullsized":"2024-09-21T10:01:30.202124+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:01.302499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":118,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.362122+0000","last_change":"2024-09-21T09:56:12.588824+0000","last_active":"2024-09-21T10:01:18.362122+0000","last_peered":"2024-09-21T10:01:18.362122+0000","last_clean":"2024-09-21T10:01:18.362122+0000","last_became_active":"2024-09-21T09:56:12.588561+0000","last_became_peered":"2024-09-21T09:56:12.588561+0000","last_unstale":"2024-09-21T10:01:18.362122+0000","last_undegraded":"2024-09-21T10:01:18.362122+0000","last_fullsized":"2024-09-21T10:01:18.362122+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:31:48.900623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"154'3903","reported_seq":3940,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:29.713324+0000","last_change":"2024-09-21T10:00:40.152470+0000","last_active":"2024-09-21T10:01:29.713324+0000","last_peered":"2024-09-21T10:01:29.713324+0000","last_clean":"2024-09-21T10:01:29.713324+0000","last_became_active":"2024-09-21T10:00:40.152054+0000","last_became_peered":"2024-09-21T10:00:40.152054+0000","last_unstale":"2024-09-21T10:01:29.713324+0000","last_undegraded":"2024-09-21T10:01:29.713324+0000","last_fullsized":"2024-09-21T10:01:29.713324+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":3903,"log_dups_size":0,"ondisk_log_size":3903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:07.993580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255787008,"num_objects":3903,"num_object_clones":0,"num_object_copies":11709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3903,"num_write_kb":249792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":119,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.362003+0000","last_change":"2024-09-21T09:50:45.501596+0000","last_active":"2024-09-21T10:01:18.362003+0000","last_peered":"2024-09-21T10:01:18.362003+0000","last_clean":"2024-09-21T10:01:18.362003+0000","last_became_active":"2024-09-21T09:50:45.501186+0000","last_became_peered":"2024-09-21T09:50:45.501186+0000","last_unstale":"2024-09-21T10:01:18.362003+0000","last_undegraded":"2024-09-21T10:01:18.362003+0000","last_fullsized":"2024-09-21T10:01:18.362003+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:43:16.449425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":128,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.361611+0000","last_change":"2024-09-21T09:51:30.639545+0000","last_active":"2024-09-21T10:01:18.361611+0000","last_peered":"2024-09-21T10:01:18.361611+0000","last_clean":"2024-09-21T10:01:18.361611+0000","last_became_active":"2024-09-21T09:51:30.639106+0000","last_became_peered":"2024-09-21T09:51:30.639106+0000","last_unstale":"2024-09-21T10:01:18.361611+0000","last_undegraded":"2024-09-21T10:01:18.361611+0000","last_fullsized":"2024-09-21T10:01:18.361611+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:19:19.607327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"155'3802","reported_seq":3854,"reported_epoch":155,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2024-09-21T10:01:30.504112+0000","last_change":"2024-09-21T10:01:18.418027+0000","last_active":"2024-09-21T10:01:30.504112+0000","last_peered":"2024-09-21T10:01:30.504112+0000","last_clean":"2024-09-21T10:01:08.321238+0000","last_became_active":"2024-09-21T10:01:18.386759+0000","last_became_peered":"2024-09-21T10:01:18.386759+0000","last_unstale":"2024-09-21T10:01:30.504112+0000","last_undegraded":"2024-09-21T10:01:18.370317+0000","last_fullsized":"2024-09-21T10:01:30.504112+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":3802,"log_dups_size":0,"ondisk_log_size":3802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249167872,"num_objects":3802,"num_object_clones":0,"num_object_copies":11406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":407,"num_objects_misplaced":3395,"num_objects_unfound":0,"num_objects_dirty":3802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3802,"num_write_kb":243328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":393216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,5],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":407},{"shards":"1,3,5","objects":3395}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":146,"reported_epoch":154,"state":"active+undersized","last_fresh":"2024-09-21T10:01:18.361915+0000","last_change":"2024-09-21T10:01:17.356446+0000","last_active":"2024-09-21T10:01:18.361915+0000","last_peered":"2024-09-21T10:01:18.361915+0000","last_clean":"2024-09-21T10:01:12.333151+0000","last_became_active":"2024-09-21T10:01:17.356446+0000","last_became_peered":"2024-09-21T10:01:17.356446+0000","last_unstale":"2024-09-21T10:01:18.361915+0000","last_undegraded":"2024-09-21T10:01:18.361915+0000","last_fullsized":"2024-09-21T10:01:17.352337+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:01:29.779281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":118,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.362278+0000","last_change":"2024-09-21T09:50:45.510764+0000","last_active":"2024-09-21T10:01:18.362278+0000","last_peered":"2024-09-21T10:01:18.362278+0000","last_clean":"2024-09-21T10:01:18.362278+0000","last_became_active":"2024-09-21T09:50:45.510226+0000","last_became_peered":"2024-09-21T09:50:45.510226+0000","last_unstale":"2024-09-21T10:01:18.362278+0000","last_undegraded":"2024-09-21T10:01:18.362278+0000","last_fullsized":"2024-09-21T10:01:18.362278+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:31:13.491427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"155'4004","reported_seq":4041,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T10:01:30.520157+0000","last_change":"2024-09-21T10:00:39.616847+0000","last_active":"2024-09-21T10:01:30.520157+0000","last_peered":"2024-09-21T10:01:30.520157+0000","last_clean":"2024-09-21T10:01:30.520157+0000","last_became_active":"2024-09-21T10:00:39.616555+0000","last_became_peered":"2024-09-21T10:00:39.616555+0000","last_unstale":"2024-09-21T10:01:30.520157+0000","last_undegraded":"2024-09-21T10:01:30.520157+0000","last_fullsized":"2024-09-21T10:01:30.520157+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":4004,"log_dups_size":0,"ondisk_log_size":4004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:13.417964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262406144,"num_objects":4004,"num_object_clones":0,"num_object_copies":12012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4004,"num_write_kb":256256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":153,"reported_epoch":155,"state":"active+undersized","last_fresh":"2024-09-21T10:01:30.203904+0000","last_change":"2024-09-21T10:01:17.365027+0000","last_active":"2024-09-21T10:01:30.203904+0000","last_peered":"2024-09-21T10:01:30.203904+0000","last_clean":"2024-09-21T10:01:12.337728+0000","last_became_active":"2024-09-21T10:01:17.365027+0000","last_became_peered":"2024-09-21T10:01:17.365027+0000","last_unstale":"2024-09-21T10:01:30.203904+0000","last_undegraded":"2024-09-21T10:01:30.203904+0000","last_fullsized":"2024-09-21T10:01:17.352121+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:14:52.580064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":153,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.360778+0000","last_change":"2024-09-21T09:56:12.575396+0000","last_active":"2024-09-21T10:01:18.360778+0000","last_peered":"2024-09-21T10:01:18.360778+0000","last_clean":"2024-09-21T10:01:18.360778+0000","last_became_active":"2024-09-21T09:56:12.574913+0000","last_became_peered":"2024-09-21T09:56:12.574913+0000","last_unstale":"2024-09-21T10:01:18.360778+0000","last_undegraded":"2024-09-21T10:01:18.360778+0000","last_fullsized":"2024-09-21T10:01:18.360778+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:05.462880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"155'3861","reported_seq":3900,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T10:01:30.492652+0000","last_change":"2024-09-21T10:00:39.615333+0000","last_active":"2024-09-21T10:01:30.492652+0000","last_peered":"2024-09-21T10:01:30.492652+0000","last_clean":"2024-09-21T10:01:30.492652+0000","last_became_active":"2024-09-21T10:00:39.614910+0000","last_became_peered":"2024-09-21T10:00:39.614910+0000","last_unstale":"2024-09-21T10:01:30.492652+0000","last_undegraded":"2024-09-21T10:01:30.492652+0000","last_fullsized":"2024-09-21T10:01:30.492652+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":3861,"log_dups_size":0,"ondisk_log_size":3861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:17.632326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253034496,"num_objects":3861,"num_object_clones":0,"num_object_copies":11583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3861,"num_write_kb":247104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":181,"reported_epoch":155,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:01:30.204333+0000","last_change":"2024-09-21T10:01:17.363096+0000","last_active":"2024-09-21T10:01:30.204333+0000","last_peered":"2024-09-21T10:01:30.204333+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:17.363096+0000","last_became_peered":"2024-09-21T10:01:17.363096+0000","last_unstale":"2024-09-21T10:01:30.204333+0000","last_undegraded":"2024-09-21T10:01:17.358421+0000","last_fullsized":"2024-09-21T10:01:17.358317+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:51:01.208485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":168,"reported_epoch":154,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:01:18.359012+0000","last_change":"2024-09-21T10:01:17.359511+0000","last_active":"2024-09-21T10:01:18.359012+0000","last_peered":"2024-09-21T10:01:18.359012+0000","last_clean":"2024-09-21T10:01:12.332227+0000","last_became_active":"2024-09-21T10:01:17.359511+0000","last_became_peered":"2024-09-21T10:01:17.359511+0000","last_unstale":"2024-09-21T10:01:18.359012+0000","last_undegraded":"2024-09-21T10:01:17.357161+0000","last_fullsized":"2024-09-21T10:01:17.356996+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:39:03.218322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039581700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":123,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T10:01:30.200686+0000","last_change":"2024-09-21T09:56:12.580057+0000","last_active":"2024-09-21T10:01:30.200686+0000","last_peered":"2024-09-21T10:01:30.200686+0000","last_clean":"2024-09-21T10:01:30.200686+0000","last_became_active":"2024-09-21T09:56:12.579700+0000","last_became_peered":"2024-09-21T09:56:12.579700+0000","last_unstale":"2024-09-21T10:01:30.200686+0000","last_undegraded":"2024-09-21T10:01:30.200686+0000","last_fullsized":"2024-09-21T10:01:30.200686+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:49.768872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":128,"reported_epoch":154,"state":"active+undersized","last_fresh":"2024-09-21T10:01:18.361713+0000","last_change":"2024-09-21T10:01:17.359297+0000","last_active":"2024-09-21T10:01:18.361713+0000","last_peered":"2024-09-21T10:01:18.361713+0000","last_clean":"2024-09-21T10:01:12.345523+0000","last_became_active":"2024-09-21T10:01:17.359297+0000","last_became_peered":"2024-09-21T10:01:17.359297+0000","last_unstale":"2024-09-21T10:01:18.361713+0000","last_undegraded":"2024-09-21T10:01:18.361713+0000","last_fullsized":"2024-09-21T10:01:17.354792+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:54:20.381939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"155'4199","reported_seq":4238,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T10:01:33.910562+0000","last_change":"2024-09-21T10:00:40.152326+0000","last_active":"2024-09-21T10:01:33.910562+0000","last_peered":"2024-09-21T10:01:33.910562+0000","last_clean":"2024-09-21T10:01:33.910562+0000","last_became_active":"2024-09-21T10:00:40.152002+0000","last_became_peered":"2024-09-21T10:00:40.152002+0000","last_unstale":"2024-09-21T10:01:33.910562+0000","last_undegraded":"2024-09-21T10:01:33.910562+0000","last_fullsized":"2024-09-21T10:01:33.910562+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":4199,"log_dups_size":0,"ondisk_log_size":4199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:31.868823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275185664,"num_objects":4199,"num_object_clones":0,"num_object_copies":12597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4199,"num_write_kb":268736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":108,"reported_epoch":154,"state":"active+undersized","last_fresh":"2024-09-21T10:01:18.360721+0000","last_change":"2024-09-21T10:01:17.367117+0000","last_active":"2024-09-21T10:01:18.360721+0000","last_peered":"2024-09-21T10:01:18.360721+0000","last_clean":"2024-09-21T09:50:10.064980+0000","last_became_active":"2024-09-21T10:01:17.367117+0000","last_became_peered":"2024-09-21T10:01:17.367117+0000","last_unstale":"2024-09-21T10:01:18.360721+0000","last_undegraded":"2024-09-21T10:01:18.360721+0000","last_fullsized":"2024-09-21T10:01:17.359433+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:44.782398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":185,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T10:01:30.219454+0000","last_change":"2024-09-21T09:56:12.573698+0000","last_active":"2024-09-21T10:01:30.219454+0000","last_peered":"2024-09-21T10:01:30.219454+0000","last_clean":"2024-09-21T10:01:30.219454+0000","last_became_active":"2024-09-21T09:56:12.573390+0000","last_became_peered":"2024-09-21T09:56:12.573390+0000","last_unstale":"2024-09-21T10:01:30.219454+0000","last_undegraded":"2024-09-21T10:01:30.219454+0000","last_fullsized":"2024-09-21T10:01:30.219454+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:55:12.937878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":196,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T10:01:30.202950+0000","last_change":"2024-09-21T09:51:30.666462+0000","last_active":"2024-09-21T10:01:30.202950+0000","last_peered":"2024-09-21T10:01:30.202950+0000","last_clean":"2024-09-21T10:01:30.202950+0000","last_became_active":"2024-09-21T09:51:30.666006+0000","last_became_peered":"2024-09-21T09:51:30.666006+0000","last_unstale":"2024-09-21T10:01:30.202950+0000","last_undegraded":"2024-09-21T10:01:30.202950+0000","last_fullsized":"2024-09-21T10:01:30.202950+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:51.481698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"155'3919","reported_seq":4836,"reported_epoch":155,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:01:32.549449+0000","last_change":"2024-09-21T10:01:29.735004+0000","last_active":"2024-09-21T10:01:32.549449+0000","last_peered":"2024-09-21T10:01:32.549449+0000","last_clean":"2024-09-21T10:01:32.549449+0000","last_became_active":"2024-09-21T10:01:18.386142+0000","last_became_peered":"2024-09-21T10:01:18.386142+0000","last_unstale":"2024-09-21T10:01:32.549449+0000","last_undegraded":"2024-09-21T10:01:32.549449+0000","last_fullsized":"2024-09-21T10:01:32.549449+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":3919,"log_dups_size":0,"ondisk_log_size":3919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:41.716874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256835584,"num_objects":3919,"num_object_clones":0,"num_object_copies":11757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3919,"num_objects_unfound":0,"num_objects_dirty":3919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3919,"num_write_kb":250816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":431,"num_bytes_recovered":28246016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1,3],"avail_no_missing":["5","1","3"],"object_location_counts":[{"shards":"1,3,5","objects":3919}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":184,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.357576+0000","last_change":"2024-09-21T09:56:12.580659+0000","last_active":"2024-09-21T10:01:18.357576+0000","last_peered":"2024-09-21T10:01:18.357576+0000","last_clean":"2024-09-21T10:01:18.357576+0000","last_became_active":"2024-09-21T09:56:12.580300+0000","last_became_peered":"2024-09-21T09:56:12.580300+0000","last_unstale":"2024-09-21T10:01:18.357576+0000","last_undegraded":"2024-09-21T10:01:18.357576+0000","last_fullsized":"2024-09-21T10:01:18.357576+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:04.945404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":153,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.361011+0000","last_change":"2024-09-21T09:51:30.646466+0000","last_active":"2024-09-21T10:01:18.361011+0000","last_peered":"2024-09-21T10:01:18.361011+0000","last_clean":"2024-09-21T10:01:18.361011+0000","last_became_active":"2024-09-21T09:51:30.646209+0000","last_became_peered":"2024-09-21T09:51:30.646209+0000","last_unstale":"2024-09-21T10:01:18.361011+0000","last_undegraded":"2024-09-21T10:01:18.361011+0000","last_fullsized":"2024-09-21T10:01:18.361011+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:44:41.465418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":157,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.357842+0000","last_change":"2024-09-21T09:51:30.649056+0000","last_active":"2024-09-21T10:01:18.357842+0000","last_peered":"2024-09-21T10:01:18.357842+0000","last_clean":"2024-09-21T10:01:18.357842+0000","last_became_active":"2024-09-21T09:51:30.647245+0000","last_became_peered":"2024-09-21T09:51:30.647245+0000","last_unstale":"2024-09-21T10:01:18.357842+0000","last_undegraded":"2024-09-21T10:01:18.357842+0000","last_fullsized":"2024-09-21T10:01:18.357842+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:32.661847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.9","version":"155'4086","reported_seq":4626,"reported_epoch":155,"state":"active+recovering+degraded+remapped","last_fresh":"2024-09-21T10:01:33.910311+0000","last_change":"2024-09-21T10:01:29.739772+0000","last_active":"2024-09-21T10:01:33.910311+0000","last_peered":"2024-09-21T10:01:33.910311+0000","last_clean":"2024-09-21T10:01:08.308117+0000","last_became_active":"2024-09-21T10:01:18.418674+0000","last_became_peered":"2024-09-21T10:01:18.418674+0000","last_unstale":"2024-09-21T10:01:33.910311+0000","last_undegraded":"2024-09-21T10:01:18.371800+0000","last_fullsized":"2024-09-21T10:01:33.910311+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":4086,"log_dups_size":0,"ondisk_log_size":4086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267780096,"num_objects":4086,"num_object_clones":0,"num_object_copies":12258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4086,"num_write_kb":261504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":15925248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,3],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,3,5","objects":3902},{"shards":"0,5","objects":184}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":134,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.365426+0000","last_change":"2024-09-21T09:50:45.502094+0000","last_active":"2024-09-21T10:01:18.365426+0000","last_peered":"2024-09-21T10:01:18.365426+0000","last_clean":"2024-09-21T10:01:18.365426+0000","last_became_active":"2024-09-21T09:50:45.501603+0000","last_became_peered":"2024-09-21T09:50:45.501603+0000","last_unstale":"2024-09-21T10:01:18.365426+0000","last_undegraded":"2024-09-21T10:01:18.365426+0000","last_fullsized":"2024-09-21T10:01:18.365426+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:01.747067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091939799999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":119,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.361823+0000","last_change":"2024-09-21T09:48:54.128617+0000","last_active":"2024-09-21T10:01:18.361823+0000","last_peered":"2024-09-21T10:01:18.361823+0000","last_clean":"2024-09-21T10:01:18.361823+0000","last_became_active":"2024-09-21T09:48:54.127878+0000","last_became_peered":"2024-09-21T09:48:54.127878+0000","last_unstale":"2024-09-21T10:01:18.361823+0000","last_undegraded":"2024-09-21T10:01:18.361823+0000","last_fullsized":"2024-09-21T10:01:18.361823+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:07.305638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":165,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.360505+0000","last_change":"2024-09-21T09:56:12.588992+0000","last_active":"2024-09-21T10:01:18.360505+0000","last_peered":"2024-09-21T10:01:18.360505+0000","last_clean":"2024-09-21T10:01:18.360505+0000","last_became_active":"2024-09-21T09:56:12.588665+0000","last_became_peered":"2024-09-21T09:56:12.588665+0000","last_unstale":"2024-09-21T10:01:18.360505+0000","last_undegraded":"2024-09-21T10:01:18.360505+0000","last_fullsized":"2024-09-21T10:01:18.360505+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:19:06.160456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"155'3972","reported_seq":3994,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T10:01:32.549871+0000","last_change":"2024-09-21T10:00:39.617307+0000","last_active":"2024-09-21T10:01:32.549871+0000","last_peered":"2024-09-21T10:01:32.549871+0000","last_clean":"2024-09-21T10:01:32.549871+0000","last_became_active":"2024-09-21T10:00:39.617020+0000","last_became_peered":"2024-09-21T10:00:39.617020+0000","last_unstale":"2024-09-21T10:01:32.549871+0000","last_undegraded":"2024-09-21T10:01:32.549871+0000","last_fullsized":"2024-09-21T10:01:32.549871+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":3972,"log_dups_size":0,"ondisk_log_size":3972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:21:32.333188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260308992,"num_objects":3972,"num_object_clones":0,"num_object_copies":11916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3972,"num_write_kb":254208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":137,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.359476+0000","last_change":"2024-09-21T09:56:12.583280+0000","last_active":"2024-09-21T10:01:18.359476+0000","last_peered":"2024-09-21T10:01:18.359476+0000","last_clean":"2024-09-21T10:01:18.359476+0000","last_became_active":"2024-09-21T09:56:12.583007+0000","last_became_peered":"2024-09-21T09:56:12.583007+0000","last_unstale":"2024-09-21T10:01:18.359476+0000","last_undegraded":"2024-09-21T10:01:18.359476+0000","last_fullsized":"2024-09-21T10:01:18.359476+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:26.991596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":210,"reported_epoch":154,"state":"active+undersized","last_fresh":"2024-09-21T10:01:18.361070+0000","last_change":"2024-09-21T10:01:17.367142+0000","last_active":"2024-09-21T10:01:18.361070+0000","last_peered":"2024-09-21T10:01:18.361070+0000","last_clean":"2024-09-21T10:01:12.331437+0000","last_became_active":"2024-09-21T10:01:17.367142+0000","last_became_peered":"2024-09-21T10:01:17.367142+0000","last_unstale":"2024-09-21T10:01:18.361070+0000","last_undegraded":"2024-09-21T10:01:18.361070+0000","last_fullsized":"2024-09-21T10:01:17.359249+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:46.792787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":143,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.360968+0000","last_change":"2024-09-21T09:50:45.501349+0000","last_active":"2024-09-21T10:01:18.360968+0000","last_peered":"2024-09-21T10:01:18.360968+0000","last_clean":"2024-09-21T10:01:18.360968+0000","last_became_active":"2024-09-21T09:50:45.501065+0000","last_became_peered":"2024-09-21T09:50:45.501065+0000","last_unstale":"2024-09-21T10:01:18.360968+0000","last_undegraded":"2024-09-21T10:01:18.360968+0000","last_fullsized":"2024-09-21T10:01:18.360968+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:28:58.570586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"155'3969","reported_seq":4006,"reported_epoch":155,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:01:32.225715+0000","last_change":"2024-09-21T10:01:18.378023+0000","last_active":"2024-09-21T10:01:32.225715+0000","last_peered":"2024-09-21T10:01:32.225715+0000","last_clean":"2024-09-21T10:01:32.225715+0000","last_became_active":"2024-09-21T10:01:18.375695+0000","last_became_peered":"2024-09-21T10:01:18.375695+0000","last_unstale":"2024-09-21T10:01:32.225715+0000","last_undegraded":"2024-09-21T10:01:32.225715+0000","last_fullsized":"2024-09-21T10:01:32.225715+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":3969,"log_dups_size":0,"ondisk_log_size":3969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:37:12.806595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260112384,"num_objects":3969,"num_object_clones":0,"num_object_copies":11907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3969,"num_objects_unfound":0,"num_objects_dirty":3969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3969,"num_write_kb":254016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,0],"avail_no_missing":["4","0","1"],"object_location_counts":[{"shards":"0,1,4","objects":3969}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":153,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.360986+0000","last_change":"2024-09-21T09:51:30.646531+0000","last_active":"2024-09-21T10:01:18.360986+0000","last_peered":"2024-09-21T10:01:18.360986+0000","last_clean":"2024-09-21T10:01:18.360986+0000","last_became_active":"2024-09-21T09:51:30.646248+0000","last_became_peered":"2024-09-21T09:51:30.646248+0000","last_unstale":"2024-09-21T10:01:18.360986+0000","last_undegraded":"2024-09-21T10:01:18.360986+0000","last_fullsized":"2024-09-21T10:01:18.360986+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:00.312924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":79,"reported_epoch":155,"state":"active+undersized","last_fresh":"2024-09-21T10:01:30.205464+0000","last_change":"2024-09-21T10:01:17.364179+0000","last_active":"2024-09-21T10:01:30.205464+0000","last_peered":"2024-09-21T10:01:30.205464+0000","last_clean":"2024-09-21T10:00:52.397528+0000","last_became_active":"2024-09-21T10:01:17.364179+0000","last_became_peered":"2024-09-21T10:01:17.364179+0000","last_unstale":"2024-09-21T10:01:30.205464+0000","last_undegraded":"2024-09-21T10:01:30.205464+0000","last_fullsized":"2024-09-21T10:01:17.360621+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:26:17.268991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":193,"reported_epoch":155,"state":"active+undersized","last_fresh":"2024-09-21T10:01:30.204964+0000","last_change":"2024-09-21T10:01:17.371359+0000","last_active":"2024-09-21T10:01:30.204964+0000","last_peered":"2024-09-21T10:01:30.204964+0000","last_clean":"2024-09-21T10:01:12.347295+0000","last_became_active":"2024-09-21T10:01:17.371359+0000","last_became_peered":"2024-09-21T10:01:17.371359+0000","last_unstale":"2024-09-21T10:01:30.204964+0000","last_undegraded":"2024-09-21T10:01:30.204964+0000","last_fullsized":"2024-09-21T10:01:17.366525+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:38.963323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"154'3928","reported_seq":4809,"reported_epoch":154,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:01:29.696343+0000","last_change":"2024-09-21T10:01:29.079779+0000","last_active":"2024-09-21T10:01:29.696343+0000","last_peered":"2024-09-21T10:01:29.696343+0000","last_clean":"2024-09-21T10:01:29.696343+0000","last_became_active":"2024-09-21T10:01:18.393943+0000","last_became_peered":"2024-09-21T10:01:18.393943+0000","last_unstale":"2024-09-21T10:01:29.696343+0000","last_undegraded":"2024-09-21T10:01:29.696343+0000","last_fullsized":"2024-09-21T10:01:29.696343+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":3928,"log_dups_size":0,"ondisk_log_size":3928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:37:39.568085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257425408,"num_objects":3928,"num_object_clones":0,"num_object_copies":11784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3928,"num_objects_unfound":0,"num_objects_dirty":3928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3928,"num_write_kb":251392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":418,"num_bytes_recovered":27394048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5,4],"avail_no_missing":["1","4","5"],"object_location_counts":[{"shards":"1,4,5","objects":3928}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":148,"reported_epoch":155,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:01:30.204978+0000","last_change":"2024-09-21T10:01:17.363383+0000","last_active":"2024-09-21T10:01:30.204978+0000","last_peered":"2024-09-21T10:01:30.204978+0000","last_clean":"2024-09-21T10:01:12.338791+0000","last_became_active":"2024-09-21T10:01:17.363383+0000","last_became_peered":"2024-09-21T10:01:17.363383+0000","last_unstale":"2024-09-21T10:01:30.204978+0000","last_undegraded":"2024-09-21T10:01:17.354694+0000","last_fullsized":"2024-09-21T10:01:17.354594+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:24:08.763448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":152,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.361806+0000","last_change":"2024-09-21T09:50:45.513017+0000","last_active":"2024-09-21T10:01:18.361806+0000","last_peered":"2024-09-21T10:01:18.361806+0000","last_clean":"2024-09-21T10:01:18.361806+0000","last_became_active":"2024-09-21T09:50:45.512629+0000","last_became_peered":"2024-09-21T09:50:45.512629+0000","last_unstale":"2024-09-21T10:01:18.361806+0000","last_undegraded":"2024-09-21T10:01:18.361806+0000","last_fullsized":"2024-09-21T10:01:18.361806+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:47:08.421575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":137,"reported_epoch":155,"state":"active+undersized","last_fresh":"2024-09-21T10:01:30.203546+0000","last_change":"2024-09-21T10:01:17.357067+0000","last_active":"2024-09-21T10:01:30.203546+0000","last_peered":"2024-09-21T10:01:30.203546+0000","last_clean":"2024-09-21T10:01:12.337480+0000","last_became_active":"2024-09-21T10:01:17.357067+0000","last_became_peered":"2024-09-21T10:01:17.357067+0000","last_unstale":"2024-09-21T10:01:30.203546+0000","last_undegraded":"2024-09-21T10:01:30.203546+0000","last_fullsized":"2024-09-21T10:01:17.351626+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:21.390162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"155'3878","reported_seq":4106,"reported_epoch":155,"state":"active+recovering+degraded+remapped","last_fresh":"2024-09-21T10:01:30.517460+0000","last_change":"2024-09-21T10:01:29.081177+0000","last_active":"2024-09-21T10:01:30.517460+0000","last_peered":"2024-09-21T10:01:30.517460+0000","last_clean":"2024-09-21T10:01:08.336950+0000","last_became_active":"2024-09-21T10:01:18.395798+0000","last_became_peered":"2024-09-21T10:01:18.395798+0000","last_unstale":"2024-09-21T10:01:30.517460+0000","last_undegraded":"2024-09-21T10:01:18.374118+0000","last_fullsized":"2024-09-21T10:01:30.517460+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":145,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":3878,"log_dups_size":0,"ondisk_log_size":3878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254148608,"num_objects":3878,"num_object_clones":0,"num_object_copies":11634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3878,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3878,"num_write_kb":248192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":6029312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,0],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"0,3,4","objects":3528},{"shards":"3,4","objects":350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":188,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T10:01:30.203441+0000","last_change":"2024-09-21T09:50:45.510812+0000","last_active":"2024-09-21T10:01:30.203441+0000","last_peered":"2024-09-21T10:01:30.203441+0000","last_clean":"2024-09-21T10:01:30.203441+0000","last_became_active":"2024-09-21T09:50:45.510480+0000","last_became_peered":"2024-09-21T09:50:45.510480+0000","last_unstale":"2024-09-21T10:01:30.203441+0000","last_undegraded":"2024-09-21T10:01:30.203441+0000","last_fullsized":"2024-09-21T10:01:30.203441+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:05.408194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":158,"reported_epoch":154,"state":"active+undersized","last_fresh":"2024-09-21T10:01:18.361381+0000","last_change":"2024-09-21T10:01:17.357896+0000","last_active":"2024-09-21T10:01:18.361381+0000","last_peered":"2024-09-21T10:01:18.361381+0000","last_clean":"2024-09-21T10:01:12.345631+0000","last_became_active":"2024-09-21T10:01:17.357896+0000","last_became_peered":"2024-09-21T10:01:17.357896+0000","last_unstale":"2024-09-21T10:01:18.361381+0000","last_undegraded":"2024-09-21T10:01:18.361381+0000","last_fullsized":"2024-09-21T10:01:17.354502+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:45.683879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":101,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.361471+0000","last_change":"2024-09-21T09:40:49.190763+0000","last_active":"2024-09-21T10:01:18.361471+0000","last_peered":"2024-09-21T10:01:18.361471+0000","last_clean":"2024-09-21T10:01:18.361471+0000","last_became_active":"2024-09-21T09:40:49.190463+0000","last_became_peered":"2024-09-21T09:40:49.190463+0000","last_unstale":"2024-09-21T10:01:18.361471+0000","last_undegraded":"2024-09-21T10:01:18.361471+0000","last_fullsized":"2024-09-21T10:01:18.361471+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:15.669404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":128,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.361692+0000","last_change":"2024-09-21T09:51:30.642074+0000","last_active":"2024-09-21T10:01:18.361692+0000","last_peered":"2024-09-21T10:01:18.361692+0000","last_clean":"2024-09-21T10:01:18.361692+0000","last_became_active":"2024-09-21T09:51:30.641176+0000","last_became_peered":"2024-09-21T09:51:30.641176+0000","last_unstale":"2024-09-21T10:01:18.361692+0000","last_undegraded":"2024-09-21T10:01:18.361692+0000","last_fullsized":"2024-09-21T10:01:18.361692+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:18.766865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":172,"reported_epoch":154,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:01:18.362162+0000","last_change":"2024-09-21T10:01:17.359126+0000","last_active":"2024-09-21T10:01:18.362162+0000","last_peered":"2024-09-21T10:01:18.362162+0000","last_clean":"2024-09-21T10:01:12.333760+0000","last_became_active":"2024-09-21T10:01:17.359126+0000","last_became_peered":"2024-09-21T10:01:17.359126+0000","last_unstale":"2024-09-21T10:01:18.362162+0000","last_undegraded":"2024-09-21T10:01:17.353723+0000","last_fullsized":"2024-09-21T10:01:17.353628+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:19.570827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":153,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.360815+0000","last_change":"2024-09-21T09:51:30.642334+0000","last_active":"2024-09-21T10:01:18.360815+0000","last_peered":"2024-09-21T10:01:18.360815+0000","last_clean":"2024-09-21T10:01:18.360815+0000","last_became_active":"2024-09-21T09:51:30.641267+0000","last_became_peered":"2024-09-21T09:51:30.641267+0000","last_unstale":"2024-09-21T10:01:18.360815+0000","last_undegraded":"2024-09-21T10:01:18.360815+0000","last_fullsized":"2024-09-21T10:01:18.360815+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:24.201971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":46,"reported_epoch":155,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:01:30.202523+0000","last_change":"2024-09-21T10:01:17.357199+0000","last_active":"2024-09-21T10:01:30.202523+0000","last_peered":"2024-09-21T10:01:30.202523+0000","last_clean":"2024-09-21T10:00:52.399787+0000","last_became_active":"2024-09-21T10:01:17.357199+0000","last_became_peered":"2024-09-21T10:01:17.357199+0000","last_unstale":"2024-09-21T10:01:30.202523+0000","last_undegraded":"2024-09-21T10:01:17.352030+0000","last_fullsized":"2024-09-21T10:01:17.351935+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:34.172159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":147,"reported_epoch":154,"state":"active+undersized","last_fresh":"2024-09-21T10:01:18.359163+0000","last_change":"2024-09-21T10:01:17.372819+0000","last_active":"2024-09-21T10:01:18.359163+0000","last_peered":"2024-09-21T10:01:18.359163+0000","last_clean":"2024-09-21T10:01:12.334093+0000","last_became_active":"2024-09-21T10:01:17.372819+0000","last_became_peered":"2024-09-21T10:01:17.372819+0000","last_unstale":"2024-09-21T10:01:18.359163+0000","last_undegraded":"2024-09-21T10:01:18.359163+0000","last_fullsized":"2024-09-21T10:01:17.355655+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:03:49.178062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":142,"reported_epoch":155,"state":"active+undersized","last_fresh":"2024-09-21T10:01:30.203879+0000","last_change":"2024-09-21T10:01:17.368801+0000","last_active":"2024-09-21T10:01:30.203879+0000","last_peered":"2024-09-21T10:01:30.203879+0000","last_clean":"2024-09-21T10:01:12.340026+0000","last_became_active":"2024-09-21T10:01:17.368801+0000","last_became_peered":"2024-09-21T10:01:17.368801+0000","last_unstale":"2024-09-21T10:01:30.203879+0000","last_undegraded":"2024-09-21T10:01:30.203879+0000","last_fullsized":"2024-09-21T10:01:17.353280+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:23.625415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":184,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T10:01:30.202518+0000","last_change":"2024-09-21T09:56:12.574472+0000","last_active":"2024-09-21T10:01:30.202518+0000","last_peered":"2024-09-21T10:01:30.202518+0000","last_clean":"2024-09-21T10:01:30.202518+0000","last_became_active":"2024-09-21T09:56:12.573966+0000","last_became_peered":"2024-09-21T09:56:12.573966+0000","last_unstale":"2024-09-21T10:01:30.202518+0000","last_undegraded":"2024-09-21T10:01:30.202518+0000","last_fullsized":"2024-09-21T10:01:30.202518+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:37:21.415933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":134,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T10:01:30.203463+0000","last_change":"2024-09-21T09:56:12.576756+0000","last_active":"2024-09-21T10:01:30.203463+0000","last_peered":"2024-09-21T10:01:30.203463+0000","last_clean":"2024-09-21T10:01:30.203463+0000","last_became_active":"2024-09-21T09:56:12.575752+0000","last_became_peered":"2024-09-21T09:56:12.575752+0000","last_unstale":"2024-09-21T10:01:30.203463+0000","last_undegraded":"2024-09-21T10:01:30.203463+0000","last_fullsized":"2024-09-21T10:01:30.203463+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:55.224839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":118,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.361296+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T10:01:18.361296+0000","last_peered":"2024-09-21T10:01:18.361296+0000","last_clean":"2024-09-21T10:01:18.361296+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T10:01:18.361296+0000","last_undegraded":"2024-09-21T10:01:18.361296+0000","last_fullsized":"2024-09-21T10:01:18.361296+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":152,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T10:01:30.203807+0000","last_change":"2024-09-21T09:56:12.571495+0000","last_active":"2024-09-21T10:01:30.203807+0000","last_peered":"2024-09-21T10:01:30.203807+0000","last_clean":"2024-09-21T10:01:30.203807+0000","last_became_active":"2024-09-21T09:56:12.571200+0000","last_became_peered":"2024-09-21T09:56:12.571200+0000","last_unstale":"2024-09-21T10:01:30.203807+0000","last_undegraded":"2024-09-21T10:01:30.203807+0000","last_fullsized":"2024-09-21T10:01:30.203807+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:53:58.851300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":204,"reported_epoch":155,"state":"active+undersized","last_fresh":"2024-09-21T10:01:30.203042+0000","last_change":"2024-09-21T10:01:17.363986+0000","last_active":"2024-09-21T10:01:30.203042+0000","last_peered":"2024-09-21T10:01:30.203042+0000","last_clean":"2024-09-21T10:01:12.350477+0000","last_became_active":"2024-09-21T10:01:17.363986+0000","last_became_peered":"2024-09-21T10:01:17.363986+0000","last_unstale":"2024-09-21T10:01:30.203042+0000","last_undegraded":"2024-09-21T10:01:30.203042+0000","last_fullsized":"2024-09-21T10:01:17.354948+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:29:32.179855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":152,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.360496+0000","last_change":"2024-09-21T09:56:12.580276+0000","last_active":"2024-09-21T10:01:18.360496+0000","last_peered":"2024-09-21T10:01:18.360496+0000","last_clean":"2024-09-21T10:01:18.360496+0000","last_became_active":"2024-09-21T09:56:12.579989+0000","last_became_peered":"2024-09-21T09:56:12.579989+0000","last_unstale":"2024-09-21T10:01:18.360496+0000","last_undegraded":"2024-09-21T10:01:18.360496+0000","last_fullsized":"2024-09-21T10:01:18.360496+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:14:56.361873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":134,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T10:01:30.200717+0000","last_change":"2024-09-21T09:56:12.582383+0000","last_active":"2024-09-21T10:01:30.200717+0000","last_peered":"2024-09-21T10:01:30.200717+0000","last_clean":"2024-09-21T10:01:30.200717+0000","last_became_active":"2024-09-21T09:56:12.581427+0000","last_became_peered":"2024-09-21T09:56:12.581427+0000","last_unstale":"2024-09-21T10:01:30.200717+0000","last_undegraded":"2024-09-21T10:01:30.200717+0000","last_fullsized":"2024-09-21T10:01:30.200717+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:24.275633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":137,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.361219+0000","last_change":"2024-09-21T09:51:30.645644+0000","last_active":"2024-09-21T10:01:18.361219+0000","last_peered":"2024-09-21T10:01:18.361219+0000","last_clean":"2024-09-21T10:01:18.361219+0000","last_became_active":"2024-09-21T09:51:30.644883+0000","last_became_peered":"2024-09-21T09:51:30.644883+0000","last_unstale":"2024-09-21T10:01:18.361219+0000","last_undegraded":"2024-09-21T10:01:18.361219+0000","last_fullsized":"2024-09-21T10:01:18.361219+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:39.331936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":186,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.357638+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:01:18.357638+0000","last_peered":"2024-09-21T10:01:18.357638+0000","last_clean":"2024-09-21T10:01:18.357638+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:01:18.357638+0000","last_undegraded":"2024-09-21T10:01:18.357638+0000","last_fullsized":"2024-09-21T10:01:18.357638+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":195,"reported_epoch":155,"state":"active+undersized","last_fresh":"2024-09-21T10:01:30.205140+0000","last_change":"2024-09-21T10:01:17.367707+0000","last_active":"2024-09-21T10:01:30.205140+0000","last_peered":"2024-09-21T10:01:30.205140+0000","last_clean":"2024-09-21T10:01:12.346791+0000","last_became_active":"2024-09-21T10:01:17.367707+0000","last_became_peered":"2024-09-21T10:01:17.367707+0000","last_unstale":"2024-09-21T10:01:30.205140+0000","last_undegraded":"2024-09-21T10:01:30.205140+0000","last_fullsized":"2024-09-21T10:01:17.354751+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:58:00.301582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":135,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.361621+0000","last_change":"2024-09-21T09:56:12.576148+0000","last_active":"2024-09-21T10:01:18.361621+0000","last_peered":"2024-09-21T10:01:18.361621+0000","last_clean":"2024-09-21T10:01:18.361621+0000","last_became_active":"2024-09-21T09:56:12.575813+0000","last_became_peered":"2024-09-21T09:56:12.575813+0000","last_unstale":"2024-09-21T10:01:18.361621+0000","last_undegraded":"2024-09-21T10:01:18.361621+0000","last_fullsized":"2024-09-21T10:01:18.361621+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:53:01.367816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":122,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.359119+0000","last_change":"2024-09-21T09:50:45.502447+0000","last_active":"2024-09-21T10:01:18.359119+0000","last_peered":"2024-09-21T10:01:18.359119+0000","last_clean":"2024-09-21T10:01:18.359119+0000","last_became_active":"2024-09-21T09:50:45.500800+0000","last_became_peered":"2024-09-21T09:50:45.500800+0000","last_unstale":"2024-09-21T10:01:18.359119+0000","last_undegraded":"2024-09-21T10:01:18.359119+0000","last_fullsized":"2024-09-21T10:01:18.359119+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:20:51.819410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":135,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.361130+0000","last_change":"2024-09-21T09:56:12.576457+0000","last_active":"2024-09-21T10:01:18.361130+0000","last_peered":"2024-09-21T10:01:18.361130+0000","last_clean":"2024-09-21T10:01:18.361130+0000","last_became_active":"2024-09-21T09:56:12.576061+0000","last_became_peered":"2024-09-21T09:56:12.576061+0000","last_unstale":"2024-09-21T10:01:18.361130+0000","last_undegraded":"2024-09-21T10:01:18.361130+0000","last_fullsized":"2024-09-21T10:01:18.361130+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:51:46.676699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":105,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.359387+0000","last_change":"2024-09-21T09:40:49.094916+0000","last_active":"2024-09-21T10:01:18.359387+0000","last_peered":"2024-09-21T10:01:18.359387+0000","last_clean":"2024-09-21T10:01:18.359387+0000","last_became_active":"2024-09-21T09:40:49.094264+0000","last_became_peered":"2024-09-21T09:40:49.094264+0000","last_unstale":"2024-09-21T10:01:18.359387+0000","last_undegraded":"2024-09-21T10:01:18.359387+0000","last_fullsized":"2024-09-21T10:01:18.359387+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:53.663226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":160,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T10:01:30.204296+0000","last_change":"2024-09-21T09:56:12.577329+0000","last_active":"2024-09-21T10:01:30.204296+0000","last_peered":"2024-09-21T10:01:30.204296+0000","last_clean":"2024-09-21T10:01:30.204296+0000","last_became_active":"2024-09-21T09:56:12.576941+0000","last_became_peered":"2024-09-21T09:56:12.576941+0000","last_unstale":"2024-09-21T10:01:30.204296+0000","last_undegraded":"2024-09-21T10:01:30.204296+0000","last_fullsized":"2024-09-21T10:01:30.204296+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:35.235768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":145,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T10:01:30.202418+0000","last_change":"2024-09-21T09:56:12.577137+0000","last_active":"2024-09-21T10:01:30.202418+0000","last_peered":"2024-09-21T10:01:30.202418+0000","last_clean":"2024-09-21T10:01:30.202418+0000","last_became_active":"2024-09-21T09:56:12.576668+0000","last_became_peered":"2024-09-21T09:56:12.576668+0000","last_unstale":"2024-09-21T10:01:30.202418+0000","last_undegraded":"2024-09-21T10:01:30.202418+0000","last_fullsized":"2024-09-21T10:01:30.202418+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:56.772427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":118,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-21T10:01:18.360848+0000","last_change":"2024-09-21T09:50:45.502819+0000","last_active":"2024-09-21T10:01:18.360848+0000","last_peered":"2024-09-21T10:01:18.360848+0000","last_clean":"2024-09-21T10:01:18.360848+0000","last_became_active":"2024-09-21T09:50:45.502483+0000","last_became_peered":"2024-09-21T09:50:45.502483+0000","last_unstale":"2024-09-21T10:01:18.360848+0000","last_undegraded":"2024-09-21T10:01:18.360848+0000","last_fullsized":"2024-09-21T10:01:18.360848+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:26.549958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":189,"reported_epoch":155,"state":"active+clean","last_fresh":"2024-09-21T10:01:30.202245+0000","last_change":"2024-09-21T09:51:30.650116+0000","last_active":"2024-09-21T10:01:30.202245+0000","last_peered":"2024-09-21T10:01:30.202245+0000","last_clean":"2024-09-21T10:01:30.202245+0000","last_became_active":"2024-09-21T09:51:30.649798+0000","last_became_peered":"2024-09-21T09:51:30.649798+0000","last_unstale":"2024-09-21T10:01:30.202245+0000","last_undegraded":"2024-09-21T10:01:30.202245+0000","last_fullsized":"2024-09-21T10:01:30.202245+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:40.222590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4143710208,"num_objects":63228,"num_object_clones":0,"num_object_copies":189684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8371,"num_objects_misplaced":27142,"num_objects_unfound":0,"num_objects_dirty":63228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63228,"num_write_kb":4046592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2758,"num_bytes_recovered":180748288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12400197632,"data_stored":12400197632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63228,"ondisk_log_size":63228,"up":39,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":85,"acting":85,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":9704,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13748,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":83,"acting":83,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":19,"acting":19,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626710,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1074636,"kb_used_data":760640,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":92670516,"statfs":{"total":95995035648,"available":94894608384,"internally_reserved":0,"allocated":778895360,"data_stored":778118649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019168,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1860108,"kb_used_data":1257728,"kb_used_omap":1,"kb_used_meta":602366,"kb_avail":91885044,"statfs":{"total":95995035648,"available":94090285056,"internally_reserved":0,"allocated":1287913472,"data_stored":1287136274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":616823128},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":81,"seq":347892351130,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2017084,"kb_used_data":1572468,"kb_used_omap":5,"kb_used_meta":444602,"kb_avail":91728068,"statfs":{"total":95995035648,"available":93929541632,"internally_reserved":0,"allocated":1610207232,"data_stored":1609438604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":455272508},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430478,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":673476,"kb_used_data":75260,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93071676,"statfs":{"total":95995035648,"available":95305396224,"internally_reserved":0,"allocated":77066240,"data_stored":76315786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822939,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3413816,"kb_used_data":2504556,"kb_used_omap":1,"kb_used_meta":909246,"kb_avail":90331336,"statfs":{"total":95995035648,"available":92499288064,"internally_reserved":0,"allocated":2564665344,"data_stored":2563903773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931068248},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,5,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879362,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2641252,"kb_used_data":2405532,"kb_used_omap":4,"kb_used_meta":235707,"kb_avail":91103900,"statfs":{"total":95995035648,"available":93290393600,"internally_reserved":0,"allocated":2463264768,"data_stored":2462511141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":241364412},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566202,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1312488,"kb_used_data":976992,"kb_used_omap":6,"kb_used_meta":335481,"kb_avail":92432664,"statfs":{"total":95995035648,"available":94651047936,"internally_reserved":0,"allocated":1000439808,"data_stored":999682607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":343532967},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860547,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2823904,"kb_used_data":2441368,"kb_used_omap":5,"kb_used_meta":382522,"kb_avail":90921248,"statfs":{"total":95995035648,"available":93103357952,"internally_reserved":0,"allocated":2499960832,"data_stored":2499210915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":391702767},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,4,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1614479360,"data_stored":1614479360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2500919296,"data_stored":2500919296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1006698496,"data_stored":1006698496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2497314816,"data_stored":2497314816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1308753920,"data_stored":1308753920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2603089920,"data_stored":2603089920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":791609344,"data_stored":791609344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":77332480,"data_stored":77332480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:01:36.976 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T10:01:36.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pg_num 11 2024-09-21T10:01:37.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:37.067 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:37.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:37.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:37.168 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:37.194 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:37.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:37.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:37.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:37.355 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:37.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:37.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:37.457 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:37.475 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 48 16 69702 69686 90.7169 71.1875 0.0109661 0.0110176 2024-09-21T10:01:37.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:37.502 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:37.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:37.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:37 smithi107 ceph-mon[102221]: from='client.36000 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:01:37.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:37 smithi107 ceph-mon[102221]: pgmap v537: 89 pgs: 1 active+recovery_wait+degraded+remapped, 6 active+clean+remapped, 21 active+undersized, 2 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 8744/198096 objects degraded (4.414%); 28189/198096 objects misplaced (14.230%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:01:37.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:37 smithi107 ceph-mon[105022]: from='client.36000 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:01:37.510 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:37 smithi107 ceph-mon[105022]: pgmap v537: 89 pgs: 1 active+recovery_wait+degraded+remapped, 6 active+clean+remapped, 21 active+undersized, 2 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 8744/198096 objects degraded (4.414%); 28189/198096 objects misplaced (14.230%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:01:37.564 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:01:37.564 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-09-21T10:01:37.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-09-21T10:01:37.564 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:01:37.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.564 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:60883 8.7 8:e456cd96:::benchmark_data_smithi107_135256_object60882:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-21T10:01:37.564 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:55:49.416349+0000", 2024-09-21T10:01:37.564 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 348.14284046400002, 2024-09-21T10:01:37.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0164208669999999, 2024-09-21T10:01:37.564 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:37.564 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:37.565 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:37.565 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:37.565 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:37.565 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:37.565 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 60883 2024-09-21T10:01:37.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.565 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:37.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:37.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.416349+0000", 2024-09-21T10:01:37.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:37.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.416349+0000", 2024-09-21T10:01:37.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:37.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.416351+0000", 2024-09-21T10:01:37.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4640000000000001e-06 2024-09-21T10:01:37.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:37.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.416371+0000", 2024-09-21T10:01:37.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.037e-05 2024-09-21T10:01:37.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:37.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.416411+0000", 2024-09-21T10:01:37.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0126999999999997e-05 2024-09-21T10:01:37.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:37.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.416420+0000", 2024-09-21T10:01:37.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3580000000000003e-06 2024-09-21T10:01:37.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.436748+0000", 2024-09-21T10:01:37.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020328828 2024-09-21T10:01:37.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.436757+0000", 2024-09-21T10:01:37.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7320000000000004e-06 2024-09-21T10:01:37.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.425114+0000", 2024-09-21T10:01:37.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.98835659099999995 2024-09-21T10:01:37.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:37.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.425222+0000", 2024-09-21T10:01:37.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010793000000000001 2024-09-21T10:01:37.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 6", 2024-09-21T10:01:37.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.425346+0000", 2024-09-21T10:01:37.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012422999999999999 2024-09-21T10:01:37.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:37.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.430297+0000", 2024-09-21T10:01:37.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004951118 2024-09-21T10:01:37.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.432613+0000", 2024-09-21T10:01:37.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023161589999999999 2024-09-21T10:01:37.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:37.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.432682+0000", 2024-09-21T10:01:37.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8922000000000002e-05 2024-09-21T10:01:37.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:37.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.432770+0000", 2024-09-21T10:01:37.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8038e-05 2024-09-21T10:01:37.570 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.570 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.570 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.571 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:60884 8.7 8:e0a705eb:::benchmark_data_smithi107_135256_object60883:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-21T10:01:37.571 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:55:49.416472+0000", 2024-09-21T10:01:37.571 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 348.14271778699998, 2024-09-21T10:01:37.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.016476605, 2024-09-21T10:01:37.571 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:37.571 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:37.571 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:37.571 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:37.571 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:37.571 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:37.571 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 60884 2024-09-21T10:01:37.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.572 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:37.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:37.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.416472+0000", 2024-09-21T10:01:37.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:37.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.416472+0000", 2024-09-21T10:01:37.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:37.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.416473+0000", 2024-09-21T10:01:37.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6599999999999995e-07 2024-09-21T10:01:37.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:37.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.416494+0000", 2024-09-21T10:01:37.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1549e-05 2024-09-21T10:01:37.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:37.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.416503+0000", 2024-09-21T10:01:37.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5350000000000004e-06 2024-09-21T10:01:37.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:37.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.416509+0000", 2024-09-21T10:01:37.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0750000000000003e-06 2024-09-21T10:01:37.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.574 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.436832+0000", 2024-09-21T10:01:37.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020323279 2024-09-21T10:01:37.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.574 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.436844+0000", 2024-09-21T10:01:37.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1909999999999999e-05 2024-09-21T10:01:37.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.574 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.426839+0000", 2024-09-21T10:01:37.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.98999456799999996 2024-09-21T10:01:37.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:37.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.426918+0000", 2024-09-21T10:01:37.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9348e-05 2024-09-21T10:01:37.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 6", 2024-09-21T10:01:37.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.427021+0000", 2024-09-21T10:01:37.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010267 2024-09-21T10:01:37.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:37.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.430304+0000", 2024-09-21T10:01:37.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003282872 2024-09-21T10:01:37.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.432807+0000", 2024-09-21T10:01:37.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002503178 2024-09-21T10:01:37.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:37.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.432869+0000", 2024-09-21T10:01:37.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2524e-05 2024-09-21T10:01:37.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:37.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.432949+0000", 2024-09-21T10:01:37.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9331000000000003e-05 2024-09-21T10:01:37.577 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.577 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.577 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.577 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:60891 8.7 8:e37fb757:::benchmark_data_smithi107_135256_object60890:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T10:01:37.577 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:55:49.429940+0000", 2024-09-21T10:01:37.578 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 348.12924985699999, 2024-09-21T10:01:37.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.004946777, 2024-09-21T10:01:37.578 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:37.578 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:37.578 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:37.578 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:37.578 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:37.578 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:37.578 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 60891 2024-09-21T10:01:37.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.578 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:37.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:37.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.429940+0000", 2024-09-21T10:01:37.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:37.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.429940+0000", 2024-09-21T10:01:37.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:37.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.429941+0000", 2024-09-21T10:01:37.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5069999999999999e-06 2024-09-21T10:01:37.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:37.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.429967+0000", 2024-09-21T10:01:37.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5366000000000001e-05 2024-09-21T10:01:37.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:37.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.429978+0000", 2024-09-21T10:01:37.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1474999999999999e-05 2024-09-21T10:01:37.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:37.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.429987+0000", 2024-09-21T10:01:37.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9789999999999999e-06 2024-09-21T10:01:37.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.436875+0000", 2024-09-21T10:01:37.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0068874219999999998 2024-09-21T10:01:37.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.436880+0000", 2024-09-21T10:01:37.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7289999999999997e-06 2024-09-21T10:01:37.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.428147+0000", 2024-09-21T10:01:37.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.99126615200000001 2024-09-21T10:01:37.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:37.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.428218+0000", 2024-09-21T10:01:37.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1406e-05 2024-09-21T10:01:37.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 6", 2024-09-21T10:01:37.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.428316+0000", 2024-09-21T10:01:37.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8397e-05 2024-09-21T10:01:37.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:37.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.430308+0000", 2024-09-21T10:01:37.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019917910000000001 2024-09-21T10:01:37.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.434726+0000", 2024-09-21T10:01:37.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044173349999999997 2024-09-21T10:01:37.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:37.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.434781+0000", 2024-09-21T10:01:37.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5448999999999997e-05 2024-09-21T10:01:37.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:37.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.434887+0000", 2024-09-21T10:01:37.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105769 2024-09-21T10:01:37.584 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.584 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.584 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.584 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:60899 8.7 8:e6b1bfe5:::benchmark_data_smithi107_135256_object60898:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T10:01:37.584 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:55:49.435790+0000", 2024-09-21T10:01:37.584 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 348.12340016399997, 2024-09-21T10:01:37.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.99925730899999998, 2024-09-21T10:01:37.584 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:37.584 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:37.584 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:37.584 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:37.585 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:37.585 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:37.585 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 60899 2024-09-21T10:01:37.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.585 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:37.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:37.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.435790+0000", 2024-09-21T10:01:37.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:37.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.435790+0000", 2024-09-21T10:01:37.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:37.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.435792+0000", 2024-09-21T10:01:37.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.962e-06 2024-09-21T10:01:37.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:37.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.435852+0000", 2024-09-21T10:01:37.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0093000000000002e-05 2024-09-21T10:01:37.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:37.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.435868+0000", 2024-09-21T10:01:37.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6489000000000001e-05 2024-09-21T10:01:37.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:37.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.435878+0000", 2024-09-21T10:01:37.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8679999999999994e-06 2024-09-21T10:01:37.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.436897+0000", 2024-09-21T10:01:37.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010189559999999999 2024-09-21T10:01:37.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:49.436900+0000", 2024-09-21T10:01:37.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4510000000000001e-06 2024-09-21T10:01:37.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.429150+0000", 2024-09-21T10:01:37.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.992249296 2024-09-21T10:01:37.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:37.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.429254+0000", 2024-09-21T10:01:37.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010401800000000001 2024-09-21T10:01:37.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 6", 2024-09-21T10:01:37.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.429371+0000", 2024-09-21T10:01:37.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000117191 2024-09-21T10:01:37.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:37.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.430442+0000", 2024-09-21T10:01:37.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010706400000000001 2024-09-21T10:01:37.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.434940+0000", 2024-09-21T10:01:37.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00449805 2024-09-21T10:01:37.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:37.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.434976+0000", 2024-09-21T10:01:37.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6677999999999997e-05 2024-09-21T10:01:37.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:37.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:55:50.435047+0000", 2024-09-21T10:01:37.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0617000000000005e-05 2024-09-21T10:01:37.590 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.590 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.590 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.590 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:91022 8.7 8:e95b95fb:::benchmark_data_smithi107_135256_object91021:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-21T10:01:37.590 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:11.547095+0000", 2024-09-21T10:01:37.590 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 326.01209496299998, 2024-09-21T10:01:37.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2963853809999999, 2024-09-21T10:01:37.591 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:37.591 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:37.591 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:01:37.591 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:37.591 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:37.591 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:37.591 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91022 2024-09-21T10:01:37.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.591 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:37.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:37.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.547095+0000", 2024-09-21T10:01:37.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:37.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.547095+0000", 2024-09-21T10:01:37.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:37.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.547096+0000", 2024-09-21T10:01:37.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8899999999999998e-07 2024-09-21T10:01:37.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:37.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.547115+0000", 2024-09-21T10:01:37.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8791999999999999e-05 2024-09-21T10:01:37.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:37.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.547121+0000", 2024-09-21T10:01:37.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3099999999999997e-06 2024-09-21T10:01:37.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:37.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.547125+0000", 2024-09-21T10:01:37.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9400000000000004e-06 2024-09-21T10:01:37.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.547167+0000", 2024-09-21T10:01:37.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2379999999999997e-05 2024-09-21T10:01:37.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:37.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.547300+0000", 2024-09-21T10:01:37.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013263000000000001 2024-09-21T10:01:37.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 6", 2024-09-21T10:01:37.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.547441+0000", 2024-09-21T10:01:37.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000141342 2024-09-21T10:01:37.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:37.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.551278+0000", 2024-09-21T10:01:37.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038370320000000002 2024-09-21T10:01:37.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.565455+0000", 2024-09-21T10:01:37.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014177283000000001 2024-09-21T10:01:37.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.565466+0000", 2024-09-21T10:01:37.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0617e-05 2024-09-21T10:01:37.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.566352+0000", 2024-09-21T10:01:37.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0008862039999999 2024-09-21T10:01:37.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.566367+0000", 2024-09-21T10:01:37.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.502e-05 2024-09-21T10:01:37.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.843376+0000", 2024-09-21T10:01:37.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2770090409999999 2024-09-21T10:01:37.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:37.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.843480+0000", 2024-09-21T10:01:37.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010390099999999999 2024-09-21T10:01:37.597 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.597 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.597 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.597 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:91052 8.7 8:eb99ea61:::benchmark_data_smithi107_135256_object91051:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T10:01:37.597 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:11.578752+0000", 2024-09-21T10:01:37.597 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 325.98043819499998, 2024-09-21T10:01:37.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2844650670000002, 2024-09-21T10:01:37.597 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:37.597 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:37.597 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:37.598 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:37.598 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:37.598 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:37.598 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91052 2024-09-21T10:01:37.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.598 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:37.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:37.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.578752+0000", 2024-09-21T10:01:37.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:37.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.578752+0000", 2024-09-21T10:01:37.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:37.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.578753+0000", 2024-09-21T10:01:37.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.037e-06 2024-09-21T10:01:37.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:37.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.578774+0000", 2024-09-21T10:01:37.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0979999999999999e-05 2024-09-21T10:01:37.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:37.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.578785+0000", 2024-09-21T10:01:37.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1409e-05 2024-09-21T10:01:37.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:37.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.578792+0000", 2024-09-21T10:01:37.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.99e-06 2024-09-21T10:01:37.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.578875+0000", 2024-09-21T10:01:37.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3114000000000004e-05 2024-09-21T10:01:37.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.578884+0000", 2024-09-21T10:01:37.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5399999999999996e-06 2024-09-21T10:01:37.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.566395+0000", 2024-09-21T10:01:37.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.98751118199999999 2024-09-21T10:01:37.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.566402+0000", 2024-09-21T10:01:37.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3420000000000002e-06 2024-09-21T10:01:37.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.844680+0000", 2024-09-21T10:01:37.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.278277871 2024-09-21T10:01:37.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:37.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.844843+0000", 2024-09-21T10:01:37.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016314899999999999 2024-09-21T10:01:37.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:01:37.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.845030+0000", 2024-09-21T10:01:37.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018692700000000001 2024-09-21T10:01:37.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:37.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.851854+0000", 2024-09-21T10:01:37.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0068234919999999996 2024-09-21T10:01:37.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.851906+0000", 2024-09-21T10:01:37.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2169999999999997e-05 2024-09-21T10:01:37.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.858617+0000", 2024-09-21T10:01:37.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0067108599999999999 2024-09-21T10:01:37.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:37.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.858675+0000", 2024-09-21T10:01:37.604 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:37.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8257999999999997e-05 2024-09-21T10:01:37.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:37.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.863217+0000", 2024-09-21T10:01:37.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045417460000000002 2024-09-21T10:01:37.605 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.605 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.605 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.606 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:91059 8.7 8:e62bec10:::benchmark_data_smithi107_135256_object91058:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T10:01:37.606 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:11.587248+0000", 2024-09-21T10:01:37.606 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 325.97194181600003, 2024-09-21T10:01:37.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2815936059999999, 2024-09-21T10:01:37.606 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:37.606 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:37.606 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:37.606 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:37.606 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:37.606 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:37.606 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91059 2024-09-21T10:01:37.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.587248+0000", 2024-09-21T10:01:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.587248+0000", 2024-09-21T10:01:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.587250+0000", 2024-09-21T10:01:37.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5570000000000001e-06 2024-09-21T10:01:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.587316+0000", 2024-09-21T10:01:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6921999999999994e-05 2024-09-21T10:01:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.587337+0000", 2024-09-21T10:01:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0288999999999998e-05 2024-09-21T10:01:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.587347+0000", 2024-09-21T10:01:37.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7559999999999994e-06 2024-09-21T10:01:37.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.587413+0000", 2024-09-21T10:01:37.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6934999999999995e-05 2024-09-21T10:01:37.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.587424+0000", 2024-09-21T10:01:37.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0593e-05 2024-09-21T10:01:37.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.566425+0000", 2024-09-21T10:01:37.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.97900133899999997 2024-09-21T10:01:37.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.566432+0000", 2024-09-21T10:01:37.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1060000000000002e-06 2024-09-21T10:01:37.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.847936+0000", 2024-09-21T10:01:37.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.281503499 2024-09-21T10:01:37.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:37.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.848070+0000", 2024-09-21T10:01:37.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013390799999999999 2024-09-21T10:01:37.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:01:37.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.848229+0000", 2024-09-21T10:01:37.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000159419 2024-09-21T10:01:37.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:37.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.851921+0000", 2024-09-21T10:01:37.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003691639 2024-09-21T10:01:37.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.851943+0000", 2024-09-21T10:01:37.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1597999999999999e-05 2024-09-21T10:01:37.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.866156+0000", 2024-09-21T10:01:37.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014213198999999999 2024-09-21T10:01:37.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:37.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.866211+0000", 2024-09-21T10:01:37.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5352999999999997e-05 2024-09-21T10:01:37.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:37.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.868842+0000", 2024-09-21T10:01:37.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0026304940000000001 2024-09-21T10:01:37.613 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.613 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.613 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.613 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:91039 8.0 8:04bf9daa:::benchmark_data_smithi107_135256_object91038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e127)", 2024-09-21T10:01:37.613 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:12.567048+0000", 2024-09-21T10:01:37.613 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 324.992142194, 2024-09-21T10:01:37.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3101543950000001, 2024-09-21T10:01:37.613 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:37.613 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:37.614 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:37.614 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:37.614 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:37.614 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:37.614 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91039 2024-09-21T10:01:37.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.614 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:37.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:37.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567048+0000", 2024-09-21T10:01:37.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:37.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567048+0000", 2024-09-21T10:01:37.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:37.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567049+0000", 2024-09-21T10:01:37.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.813e-06 2024-09-21T10:01:37.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:37.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567104+0000", 2024-09-21T10:01:37.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4528999999999999e-05 2024-09-21T10:01:37.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:37.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567114+0000", 2024-09-21T10:01:37.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0237999999999999e-05 2024-09-21T10:01:37.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:37.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567124+0000", 2024-09-21T10:01:37.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4499999999999993e-06 2024-09-21T10:01:37.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567260+0000", 2024-09-21T10:01:37.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013585299999999999 2024-09-21T10:01:37.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567269+0000", 2024-09-21T10:01:37.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0000000000000002e-06 2024-09-21T10:01:37.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.851895+0000", 2024-09-21T10:01:37.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.284626754 2024-09-21T10:01:37.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:37.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.852023+0000", 2024-09-21T10:01:37.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012801300000000001 2024-09-21T10:01:37.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:01:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.852195+0000", 2024-09-21T10:01:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000171736 2024-09-21T10:01:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.869927+0000", 2024-09-21T10:01:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017731982 2024-09-21T10:01:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.869965+0000", 2024-09-21T10:01:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8244999999999997e-05 2024-09-21T10:01:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:37.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.876449+0000", 2024-09-21T10:01:37.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0064835769999999999 2024-09-21T10:01:37.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:37.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.876528+0000", 2024-09-21T10:01:37.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9468000000000006e-05 2024-09-21T10:01:37.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:37.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.877202+0000", 2024-09-21T10:01:37.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00067373699999999999 2024-09-21T10:01:37.619 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:91044 8.0 8:09420d4d:::benchmark_data_smithi107_135256_object91043:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e127)", 2024-09-21T10:01:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:12.567135+0000", 2024-09-21T10:01:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 324.99205450900001, 2024-09-21T10:01:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.326400939, 2024-09-21T10:01:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:37.620 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91044 2024-09-21T10:01:37.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.621 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:37.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:37.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567135+0000", 2024-09-21T10:01:37.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:37.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567135+0000", 2024-09-21T10:01:37.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:37.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567136+0000", 2024-09-21T10:01:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3000000000000001e-07 2024-09-21T10:01:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567181+0000", 2024-09-21T10:01:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5405000000000003e-05 2024-09-21T10:01:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567190+0000", 2024-09-21T10:01:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0839999999999998e-06 2024-09-21T10:01:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:37.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567196+0000", 2024-09-21T10:01:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2559999999999999e-06 2024-09-21T10:01:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567292+0000", 2024-09-21T10:01:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5352000000000005e-05 2024-09-21T10:01:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567298+0000", 2024-09-21T10:01:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5209999999999999e-06 2024-09-21T10:01:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.863636+0000", 2024-09-21T10:01:37.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2963372289999999 2024-09-21T10:01:37.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:37.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.863767+0000", 2024-09-21T10:01:37.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131643 2024-09-21T10:01:37.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:01:37.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.863935+0000", 2024-09-21T10:01:37.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016803900000000001 2024-09-21T10:01:37.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.877254+0000", 2024-09-21T10:01:37.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013318773000000001 2024-09-21T10:01:37.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:37.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.877267+0000", 2024-09-21T10:01:37.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2876e-05 2024-09-21T10:01:37.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.892781+0000", 2024-09-21T10:01:37.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.015513880000000001 2024-09-21T10:01:37.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:37.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.892838+0000", 2024-09-21T10:01:37.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7114999999999998e-05 2024-09-21T10:01:37.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.626 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:37.626 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.893536+0000", 2024-09-21T10:01:37.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00069833600000000003 2024-09-21T10:01:37.626 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.626 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.626 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.626 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:91045 8.f 8:fa4eface:::benchmark_data_smithi107_135256_object91044:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e127)", 2024-09-21T10:01:37.626 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:12.567217+0000", 2024-09-21T10:01:37.626 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 324.99197251099997, 2024-09-21T10:01:37.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.30111807, 2024-09-21T10:01:37.627 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:37.627 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:37.627 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:37.627 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:37.627 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:37.627 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:37.627 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91045 2024-09-21T10:01:37.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.627 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:37.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:37.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567217+0000", 2024-09-21T10:01:37.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:37.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567217+0000", 2024-09-21T10:01:37.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:37.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567218+0000", 2024-09-21T10:01:37.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7e-07 2024-09-21T10:01:37.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:37.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567262+0000", 2024-09-21T10:01:37.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4214000000000001e-05 2024-09-21T10:01:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567270+0000", 2024-09-21T10:01:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1820000000000006e-06 2024-09-21T10:01:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567276+0000", 2024-09-21T10:01:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.263e-06 2024-09-21T10:01:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567332+0000", 2024-09-21T10:01:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6054000000000002e-05 2024-09-21T10:01:37.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567340+0000", 2024-09-21T10:01:37.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9999999999999996e-06 2024-09-21T10:01:37.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.851876+0000", 2024-09-21T10:01:37.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2845351629999999 2024-09-21T10:01:37.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:37.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.852023+0000", 2024-09-21T10:01:37.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014735 2024-09-21T10:01:37.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:01:37.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.852195+0000", 2024-09-21T10:01:37.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017199399999999999 2024-09-21T10:01:37.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.862212+0000", 2024-09-21T10:01:37.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010017096 2024-09-21T10:01:37.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.863232+0000", 2024-09-21T10:01:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010196370000000001 2024-09-21T10:01:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.867265+0000", 2024-09-21T10:01:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0040331330000000004 2024-09-21T10:01:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.867375+0000", 2024-09-21T10:01:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010990800000000001 2024-09-21T10:01:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.868335+0000", 2024-09-21T10:01:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00096060600000000002 2024-09-21T10:01:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:91054 8.0 8:076f8259:::benchmark_data_smithi107_135256_object91053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e127)", 2024-09-21T10:01:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:12.567291+0000", 2024-09-21T10:01:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 324.99189863800001, 2024-09-21T10:01:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3280276689999999, 2024-09-21T10:01:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:37.633 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:37.634 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:37.634 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91054 2024-09-21T10:01:37.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.634 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:37.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:37.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567291+0000", 2024-09-21T10:01:37.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:37.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567291+0000", 2024-09-21T10:01:37.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:37.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567292+0000", 2024-09-21T10:01:37.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4700000000000001e-07 2024-09-21T10:01:37.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:37.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567337+0000", 2024-09-21T10:01:37.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4798e-05 2024-09-21T10:01:37.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:37.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567345+0000", 2024-09-21T10:01:37.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9869999999999999e-06 2024-09-21T10:01:37.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:37.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567352+0000", 2024-09-21T10:01:37.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6799999999999993e-06 2024-09-21T10:01:37.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567401+0000", 2024-09-21T10:01:37.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8832000000000003e-05 2024-09-21T10:01:37.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567410+0000", 2024-09-21T10:01:37.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.636e-06 2024-09-21T10:01:37.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.866830+0000", 2024-09-21T10:01:37.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.299420209 2024-09-21T10:01:37.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:37.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.866943+0000", 2024-09-21T10:01:37.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011357999999999999 2024-09-21T10:01:37.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:01:37.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.867103+0000", 2024-09-21T10:01:37.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000159829 2024-09-21T10:01:37.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:37.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.877273+0000", 2024-09-21T10:01:37.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010169327000000001 2024-09-21T10:01:37.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.880618+0000", 2024-09-21T10:01:37.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0033455780000000001 2024-09-21T10:01:37.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.894150+0000", 2024-09-21T10:01:37.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013532216 2024-09-21T10:01:37.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:37.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.894675+0000", 2024-09-21T10:01:37.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052420799999999997 2024-09-21T10:01:37.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:37.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.895319+0000", 2024-09-21T10:01:37.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00064424199999999995 2024-09-21T10:01:37.639 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.639 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.639 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:91056 8.f 8:f474519c:::benchmark_data_smithi107_135256_object91055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e127)", 2024-09-21T10:01:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:12.567361+0000", 2024-09-21T10:01:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 324.99182876200001, 2024-09-21T10:01:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3048748960000001, 2024-09-21T10:01:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91056 2024-09-21T10:01:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.640 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:37.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:37.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567361+0000", 2024-09-21T10:01:37.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:37.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567361+0000", 2024-09-21T10:01:37.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:37.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567362+0000", 2024-09-21T10:01:37.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0900000000000002e-07 2024-09-21T10:01:37.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:37.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567457+0000", 2024-09-21T10:01:37.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5488000000000005e-05 2024-09-21T10:01:37.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:37.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567466+0000", 2024-09-21T10:01:37.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4360000000000008e-06 2024-09-21T10:01:37.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:37.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567473+0000", 2024-09-21T10:01:37.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2700000000000001e-06 2024-09-21T10:01:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567552+0000", 2024-09-21T10:01:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9141999999999995e-05 2024-09-21T10:01:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567558+0000", 2024-09-21T10:01:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3169999999999998e-06 2024-09-21T10:01:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.855375+0000", 2024-09-21T10:01:37.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2878165529999999 2024-09-21T10:01:37.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:37.646 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.855513+0000", 2024-09-21T10:01:37.646 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013819200000000001 2024-09-21T10:01:37.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:01:37.646 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.855670+0000", 2024-09-21T10:01:37.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015720299999999999 2024-09-21T10:01:37.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.862258+0000", 2024-09-21T10:01:37.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006588216 2024-09-21T10:01:37.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:37.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.863241+0000", 2024-09-21T10:01:37.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00098270499999999995 2024-09-21T10:01:37.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.871355+0000", 2024-09-21T10:01:37.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0081137669999999992 2024-09-21T10:01:37.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:37.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.871402+0000", 2024-09-21T10:01:37.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7445999999999999e-05 2024-09-21T10:01:37.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:37.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.872236+0000", 2024-09-21T10:01:37.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00083365200000000005 2024-09-21T10:01:37.649 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.649 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.649 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.649 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:91064 8.f 8:f16b1a22:::benchmark_data_smithi107_135256_object91063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e127)", 2024-09-21T10:01:37.649 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:12.567489+0000", 2024-09-21T10:01:37.649 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 324.99170100600003, 2024-09-21T10:01:37.650 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3055731769999999, 2024-09-21T10:01:37.650 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:37.650 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:37.650 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:37.650 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:37.650 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:37.650 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:37.650 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91064 2024-09-21T10:01:37.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.650 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:37.650 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:37.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567489+0000", 2024-09-21T10:01:37.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:37.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567489+0000", 2024-09-21T10:01:37.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:37.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567489+0000", 2024-09-21T10:01:37.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4199999999999996e-07 2024-09-21T10:01:37.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:37.652 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567538+0000", 2024-09-21T10:01:37.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8976e-05 2024-09-21T10:01:37.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:37.652 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567546+0000", 2024-09-21T10:01:37.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0409999999999998e-06 2024-09-21T10:01:37.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:37.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567553+0000", 2024-09-21T10:01:37.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2319999999999998e-06 2024-09-21T10:01:37.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567573+0000", 2024-09-21T10:01:37.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0188000000000001e-05 2024-09-21T10:01:37.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567577+0000", 2024-09-21T10:01:37.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8990000000000003e-06 2024-09-21T10:01:37.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.858979+0000", 2024-09-21T10:01:37.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.291402318 2024-09-21T10:01:37.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:37.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.859114+0000", 2024-09-21T10:01:37.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013478000000000001 2024-09-21T10:01:37.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:01:37.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.859260+0000", 2024-09-21T10:01:37.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014609900000000001 2024-09-21T10:01:37.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.866005+0000", 2024-09-21T10:01:37.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0067447929999999998 2024-09-21T10:01:37.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.872293+0000", 2024-09-21T10:01:37.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0062881669999999999 2024-09-21T10:01:37.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:37.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.872310+0000", 2024-09-21T10:01:37.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6891000000000001e-05 2024-09-21T10:01:37.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:37.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.872361+0000", 2024-09-21T10:01:37.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0850999999999998e-05 2024-09-21T10:01:37.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:37.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.873062+0000", 2024-09-21T10:01:37.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00070140000000000003 2024-09-21T10:01:37.657 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.657 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.658 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.658 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:91066 8.b 8:d1e34193:::benchmark_data_smithi107_135256_object91065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e127)", 2024-09-21T10:01:37.658 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:12.567561+0000", 2024-09-21T10:01:37.658 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 324.99162921200002, 2024-09-21T10:01:37.658 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3137006979999999, 2024-09-21T10:01:37.658 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:37.658 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:37.658 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:37.658 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:37.659 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:37.659 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:37.659 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91066 2024-09-21T10:01:37.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.659 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:37.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.659 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:37.659 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567561+0000", 2024-09-21T10:01:37.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.660 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:37.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567561+0000", 2024-09-21T10:01:37.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.660 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:37.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567561+0000", 2024-09-21T10:01:37.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7100000000000002e-07 2024-09-21T10:01:37.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.660 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:37.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567604+0000", 2024-09-21T10:01:37.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3112000000000003e-05 2024-09-21T10:01:37.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:37.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567623+0000", 2024-09-21T10:01:37.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8766999999999998e-05 2024-09-21T10:01:37.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:37.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567629+0000", 2024-09-21T10:01:37.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1430000000000003e-06 2024-09-21T10:01:37.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.662 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.662 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567692+0000", 2024-09-21T10:01:37.662 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2710000000000001e-05 2024-09-21T10:01:37.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.662 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.662 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567702+0000", 2024-09-21T10:01:37.662 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0025e-05 2024-09-21T10:01:37.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.663 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.854131+0000", 2024-09-21T10:01:37.663 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.286429574 2024-09-21T10:01:37.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.663 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:37.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.857686+0000", 2024-09-21T10:01:37.663 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035544629999999999 2024-09-21T10:01:37.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.663 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:01:37.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.857865+0000", 2024-09-21T10:01:37.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017937500000000001 2024-09-21T10:01:37.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.664 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.664 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.870894+0000", 2024-09-21T10:01:37.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013028902 2024-09-21T10:01:37.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.664 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:37.664 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.871005+0000", 2024-09-21T10:01:37.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011037800000000001 2024-09-21T10:01:37.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.872631+0000", 2024-09-21T10:01:37.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0016260479999999999 2024-09-21T10:01:37.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:37.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.872695+0000", 2024-09-21T10:01:37.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4153999999999996e-05 2024-09-21T10:01:37.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:37.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.881261+0000", 2024-09-21T10:01:37.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085665759999999994 2024-09-21T10:01:37.666 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.666 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.666 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.666 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:91068 8.0 8:00f8d7c1:::benchmark_data_smithi107_135256_object91067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e127)", 2024-09-21T10:01:37.666 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:12.567711+0000", 2024-09-21T10:01:37.666 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 324.99147846400001, 2024-09-21T10:01:37.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.329420407, 2024-09-21T10:01:37.666 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:37.667 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:37.667 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:37.667 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:37.667 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:37.667 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:37.667 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91068 2024-09-21T10:01:37.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.667 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:37.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.667 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:37.667 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567711+0000", 2024-09-21T10:01:37.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.668 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:37.668 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567711+0000", 2024-09-21T10:01:37.668 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.668 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:37.668 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567712+0000", 2024-09-21T10:01:37.668 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6199999999999998e-07 2024-09-21T10:01:37.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.668 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:37.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567759+0000", 2024-09-21T10:01:37.669 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6678000000000003e-05 2024-09-21T10:01:37.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:37.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567768+0000", 2024-09-21T10:01:37.669 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0899999999999994e-06 2024-09-21T10:01:37.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:37.670 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567774+0000", 2024-09-21T10:01:37.670 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3929999999999996e-06 2024-09-21T10:01:37.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.670 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.670 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567811+0000", 2024-09-21T10:01:37.670 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6467000000000002e-05 2024-09-21T10:01:37.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.670 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.670 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.567837+0000", 2024-09-21T10:01:37.670 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6718999999999999e-05 2024-09-21T10:01:37.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.869995+0000", 2024-09-21T10:01:37.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.302157244 2024-09-21T10:01:37.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:37.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.870124+0000", 2024-09-21T10:01:37.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012925400000000001 2024-09-21T10:01:37.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.672 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:01:37.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.870271+0000", 2024-09-21T10:01:37.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014681199999999999 2024-09-21T10:01:37.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.672 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:37.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.877279+0000", 2024-09-21T10:01:37.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0070079340000000004 2024-09-21T10:01:37.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.672 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.880664+0000", 2024-09-21T10:01:37.673 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0033851210000000001 2024-09-21T10:01:37.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.673 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.673 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.895364+0000", 2024-09-21T10:01:37.673 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014700492000000001 2024-09-21T10:01:37.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.673 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:37.673 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.896265+0000", 2024-09-21T10:01:37.673 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00090135300000000003 2024-09-21T10:01:37.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.674 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:37.674 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:13.897132+0000", 2024-09-21T10:01:37.674 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000866288 2024-09-21T10:01:37.674 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.674 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.674 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.674 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.674 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:158339 8.7 8:e7073c80:::benchmark_data_smithi107_135256_object32455:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-21T10:01:37.674 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:57:27.428303+0000", 2024-09-21T10:01:37.674 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 250.13088706600001, 2024-09-21T10:01:37.674 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.98175775799999998, 2024-09-21T10:01:37.675 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:37.675 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:37.675 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:37.675 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:37.675 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:37.675 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:37.675 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 158339 2024-09-21T10:01:37.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.675 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:37.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.675 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:37.675 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.428303+0000", 2024-09-21T10:01:37.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:37.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.428303+0000", 2024-09-21T10:01:37.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:37.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.428304+0000", 2024-09-21T10:01:37.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.508e-06 2024-09-21T10:01:37.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.677 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:37.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.428309+0000", 2024-09-21T10:01:37.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6720000000000003e-06 2024-09-21T10:01:37.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.677 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:37.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.428321+0000", 2024-09-21T10:01:37.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.252e-05 2024-09-21T10:01:37.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.677 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:37.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.428328+0000", 2024-09-21T10:01:37.678 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0330000000000003e-06 2024-09-21T10:01:37.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.678 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.678 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.428400+0000", 2024-09-21T10:01:37.678 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1438000000000003e-05 2024-09-21T10:01:37.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.678 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.678 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.428411+0000", 2024-09-21T10:01:37.679 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0906e-05 2024-09-21T10:01:37.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.679 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.679 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.679 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.399624+0000", 2024-09-21T10:01:37.679 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.97121332000000005 2024-09-21T10:01:37.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.679 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.679 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:37.679 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.399730+0000", 2024-09-21T10:01:37.679 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010598399999999999 2024-09-21T10:01:37.679 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.680 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:01:37.680 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.399850+0000", 2024-09-21T10:01:37.680 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119973 2024-09-21T10:01:37.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.680 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.680 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.403185+0000", 2024-09-21T10:01:37.680 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0033346769999999999 2024-09-21T10:01:37.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.680 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:37.681 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.406980+0000", 2024-09-21T10:01:37.681 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0037956869999999998 2024-09-21T10:01:37.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.681 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.681 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.410001+0000", 2024-09-21T10:01:37.681 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003020226 2024-09-21T10:01:37.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.681 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:37.681 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.410034+0000", 2024-09-21T10:01:37.681 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3491999999999997e-05 2024-09-21T10:01:37.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.682 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:37.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.410061+0000", 2024-09-21T10:01:37.682 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6322e-05 2024-09-21T10:01:37.682 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.682 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.682 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.682 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:158349 8.7 8:ebfe815c:::benchmark_data_smithi107_135256_object32465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-21T10:01:37.682 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:57:27.441590+0000", 2024-09-21T10:01:37.682 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 250.11759944600001, 2024-09-21T10:01:37.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.968624712, 2024-09-21T10:01:37.683 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:37.683 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:37.683 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:37.683 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:37.683 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:37.683 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:37.683 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 158349 2024-09-21T10:01:37.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.683 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:37.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:37.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.441590+0000", 2024-09-21T10:01:37.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.684 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:37.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.441590+0000", 2024-09-21T10:01:37.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.684 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:37.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.441591+0000", 2024-09-21T10:01:37.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3700000000000001e-07 2024-09-21T10:01:37.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.685 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:37.685 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.441593+0000", 2024-09-21T10:01:37.685 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.869e-06 2024-09-21T10:01:37.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.685 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:37.685 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.441599+0000", 2024-09-21T10:01:37.685 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9499999999999998e-06 2024-09-21T10:01:37.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.685 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:37.686 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.441603+0000", 2024-09-21T10:01:37.686 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6460000000000001e-06 2024-09-21T10:01:37.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.686 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.686 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.441835+0000", 2024-09-21T10:01:37.686 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023167500000000001 2024-09-21T10:01:37.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.686 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.686 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.441844+0000", 2024-09-21T10:01:37.686 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5270000000000004e-06 2024-09-21T10:01:37.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.687 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.403238+0000", 2024-09-21T10:01:37.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.96139332200000005 2024-09-21T10:01:37.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.687 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:37.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.403354+0000", 2024-09-21T10:01:37.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000116173 2024-09-21T10:01:37.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.688 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:01:37.688 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.403476+0000", 2024-09-21T10:01:37.688 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000122244 2024-09-21T10:01:37.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.688 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.688 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.409942+0000", 2024-09-21T10:01:37.688 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0064655980000000004 2024-09-21T10:01:37.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.688 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:37.688 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.410118+0000", 2024-09-21T10:01:37.689 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000175761 2024-09-21T10:01:37.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.689 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.689 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.689 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.410160+0000", 2024-09-21T10:01:37.689 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2011999999999997e-05 2024-09-21T10:01:37.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.689 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.689 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:37.689 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.410191+0000", 2024-09-21T10:01:37.689 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1652000000000001e-05 2024-09-21T10:01:37.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.690 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:37.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.410215+0000", 2024-09-21T10:01:37.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3845999999999998e-05 2024-09-21T10:01:37.690 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.690 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.690 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.690 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:158357 8.7 8:e3d25f0c:::benchmark_data_smithi107_135256_object32473:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-21T10:01:37.690 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:57:27.450362+0000", 2024-09-21T10:01:37.690 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 250.10882760600001, 2024-09-21T10:01:37.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.95995327600000002, 2024-09-21T10:01:37.691 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:37.691 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:37.691 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:37.691 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:37.691 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:37.691 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:37.691 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 158357 2024-09-21T10:01:37.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.691 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:37.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:37.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.450362+0000", 2024-09-21T10:01:37.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:37.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.450362+0000", 2024-09-21T10:01:37.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:37.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.450363+0000", 2024-09-21T10:01:37.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0500000000000004e-07 2024-09-21T10:01:37.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.693 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:37.693 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.450365+0000", 2024-09-21T10:01:37.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1799999999999999e-06 2024-09-21T10:01:37.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.693 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:37.693 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.450371+0000", 2024-09-21T10:01:37.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5989999999999998e-06 2024-09-21T10:01:37.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.693 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:37.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:37.694 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:37.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:37.695 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:37 smithi129 ceph-mon[88471]: from='client.36000 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:01:37.695 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:37 smithi129 ceph-mon[88471]: pgmap v537: 89 pgs: 1 active+recovery_wait+degraded+remapped, 6 active+clean+remapped, 21 active+undersized, 2 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 8744/198096 objects degraded (4.414%); 28189/198096 objects misplaced (14.230%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:01:37.695 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.450377+0000", 2024-09-21T10:01:37.695 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.215e-06 2024-09-21T10:01:37.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.696 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.696 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.696 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.450972+0000", 2024-09-21T10:01:37.696 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00059530300000000004 2024-09-21T10:01:37.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.696 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.696 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.696 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.450981+0000", 2024-09-21T10:01:37.696 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6119999999999999e-06 2024-09-21T10:01:37.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.696 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.696 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.407015+0000", 2024-09-21T10:01:37.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.95603417499999999 2024-09-21T10:01:37.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.697 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:37.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.407084+0000", 2024-09-21T10:01:37.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9148999999999998e-05 2024-09-21T10:01:37.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.697 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:01:37.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.407151+0000", 2024-09-21T10:01:37.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7219999999999997e-05 2024-09-21T10:01:37.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.410104+0000", 2024-09-21T10:01:37.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0029532460000000001 2024-09-21T10:01:37.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:37.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.410124+0000", 2024-09-21T10:01:37.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9673e-05 2024-09-21T10:01:37.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.410256+0000", 2024-09-21T10:01:37.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013185900000000001 2024-09-21T10:01:37.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:37.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.410281+0000", 2024-09-21T10:01:37.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5418e-05 2024-09-21T10:01:37.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:37.700 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.410315+0000", 2024-09-21T10:01:37.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4122000000000003e-05 2024-09-21T10:01:37.700 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.700 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.700 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.700 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:158363 8.7 8:ebff2044:::benchmark_data_smithi107_135256_object32479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-21T10:01:37.700 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:57:27.456852+0000", 2024-09-21T10:01:37.700 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 250.10233762499999, 2024-09-21T10:01:37.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.96498841599999996, 2024-09-21T10:01:37.701 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:37.701 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:37.701 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:37.701 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:37.701 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:37.701 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:37.701 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 158363 2024-09-21T10:01:37.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.701 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:37.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:37.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.456852+0000", 2024-09-21T10:01:37.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:37.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.456852+0000", 2024-09-21T10:01:37.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:37.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.456854+0000", 2024-09-21T10:01:37.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5409999999999999e-06 2024-09-21T10:01:37.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:37.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.456859+0000", 2024-09-21T10:01:37.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9239999999999998e-06 2024-09-21T10:01:37.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:37.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.456873+0000", 2024-09-21T10:01:37.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4796e-05 2024-09-21T10:01:37.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:37.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.456881+0000", 2024-09-21T10:01:37.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2880000000000004e-06 2024-09-21T10:01:37.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.467565+0000", 2024-09-21T10:01:37.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010684315999999999 2024-09-21T10:01:37.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.704 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.467573+0000", 2024-09-21T10:01:37.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3960000000000006e-06 2024-09-21T10:01:37.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.410357+0000", 2024-09-21T10:01:37.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.94278382199999999 2024-09-21T10:01:37.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:37.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.410451+0000", 2024-09-21T10:01:37.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3484000000000001e-05 2024-09-21T10:01:37.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:01:37.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.410554+0000", 2024-09-21T10:01:37.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010292400000000001 2024-09-21T10:01:37.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.421608+0000", 2024-09-21T10:01:37.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011053947 2024-09-21T10:01:37.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.421650+0000", 2024-09-21T10:01:37.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2438999999999998e-05 2024-09-21T10:01:37.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:37.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.421739+0000", 2024-09-21T10:01:37.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9166999999999995e-05 2024-09-21T10:01:37.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:37.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.421788+0000", 2024-09-21T10:01:37.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8401e-05 2024-09-21T10:01:37.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:37.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.421841+0000", 2024-09-21T10:01:37.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2970999999999998e-05 2024-09-21T10:01:37.708 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.708 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.708 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.708 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:158368 8.7 8:e066f4d1:::benchmark_data_smithi107_135256_object32484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-21T10:01:37.708 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:57:27.456937+0000", 2024-09-21T10:01:37.708 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 250.1022524, 2024-09-21T10:01:37.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.96927246, 2024-09-21T10:01:37.709 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:01:37.709 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:01:37.709 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:01:37.709 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:01:37.709 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:01:37.709 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:01:37.709 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 158368 2024-09-21T10:01:37.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.709 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:01:37.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:01:37.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.456937+0000", 2024-09-21T10:01:37.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:01:37.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.456937+0000", 2024-09-21T10:01:37.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:01:37.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:01:37.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.456938+0000", 2024-09-21T10:01:37.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3200000000000001e-07 2024-09-21T10:01:37.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:01:37.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.456939+0000", 2024-09-21T10:01:37.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.68e-06 2024-09-21T10:01:37.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:01:37.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.456945+0000", 2024-09-21T10:01:37.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2024-09-21T10:01:37.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:01:37.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.456949+0000", 2024-09-21T10:01:37.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0910000000000003e-06 2024-09-21T10:01:37.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.467596+0000", 2024-09-21T10:01:37.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010647408000000001 2024-09-21T10:01:37.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:01:37.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:27.467603+0000", 2024-09-21T10:01:37.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5420000000000002e-06 2024-09-21T10:01:37.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:01:37.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.414115+0000", 2024-09-21T10:01:37.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.94651182 2024-09-21T10:01:37.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:01:37.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.414245+0000", 2024-09-21T10:01:37.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013061400000000001 2024-09-21T10:01:37.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:01:37.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.414376+0000", 2024-09-21T10:01:37.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013125800000000001 2024-09-21T10:01:37.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.421691+0000", 2024-09-21T10:01:37.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0073150389999999997 2024-09-21T10:01:37.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:01:37.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.421728+0000", 2024-09-21T10:01:37.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6429e-05 2024-09-21T10:01:37.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:01:37.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.426124+0000", 2024-09-21T10:01:37.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043961950000000003 2024-09-21T10:01:37.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:01:37.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.426167+0000", 2024-09-21T10:01:37.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3234e-05 2024-09-21T10:01:37.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:01:37.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:01:37.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:01:37.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:57:28.426210+0000", 2024-09-21T10:01:37.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2577999999999998e-05 2024-09-21T10:01:37.716 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.716 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.716 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.716 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:01:37.716 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:01:37.716 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:01:37.795 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:37.838 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:37.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:37.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:37.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:37.980 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:37.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:37.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:38.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:38.121 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:38.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:38.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:38.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:38.268 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:38.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:38.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:38.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:38.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:38.412 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:38.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:38.475 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 49 16 70869 70853 90.3539 72.9375 0.0129567 0.011062 2024-09-21T10:01:38.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:38.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:38.552 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:38.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:38.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:38.688 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:38.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:38.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:38.790 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:38.816 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:38.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:38.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:38.917 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:38.944 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:38.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:38.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:39.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:39.083 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:39.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:39.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:39.185 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:39.226 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:39.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:39.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:39.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:39.349 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:39.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:39.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:39.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:39.470 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:39 smithi129 ceph-mon[88471]: pgmap v538: 89 pgs: 1 active+recovery_wait+degraded+remapped, 6 active+clean+remapped, 21 active+undersized, 2 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s; 8744/203115 objects degraded (4.305%); 29201/203115 objects misplaced (14.377%); 8.8 MiB/s, 140 objects/s recovering 2024-09-21T10:01:39.475 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 50 16 71890 71874 89.8224 63.8125 0.0164639 0.0111273 2024-09-21T10:01:39.488 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:39.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:39.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:39.589 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:39.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:39 smithi107 ceph-mon[102221]: pgmap v538: 89 pgs: 1 active+recovery_wait+degraded+remapped, 6 active+clean+remapped, 21 active+undersized, 2 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s; 8744/203115 objects degraded (4.305%); 29201/203115 objects misplaced (14.377%); 8.8 MiB/s, 140 objects/s recovering 2024-09-21T10:01:39.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:39 smithi107 ceph-mon[105022]: pgmap v538: 89 pgs: 1 active+recovery_wait+degraded+remapped, 6 active+clean+remapped, 21 active+undersized, 2 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s; 8744/203115 objects degraded (4.305%); 29201/203115 objects misplaced (14.377%); 8.8 MiB/s, 140 objects/s recovering 2024-09-21T10:01:39.617 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:39.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:39.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:39.719 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:39.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:39.748 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:39.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:39.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:39.891 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:39.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:39.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:39.992 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:40.021 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:40.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:40.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:40.123 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:40.152 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:40.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:40.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:40.254 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:40.300 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:40.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:40.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:40.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:40.444 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:40.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:40.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:40.459 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:40 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 8744/203115 objects degraded (4.305%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:40.459 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:40 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 8744/203115 objects degraded (4.305%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:40.475 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 51 16 72966 72950 89.3797 67.25 0.0102668 0.0111831 2024-09-21T10:01:40.546 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:40.574 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:40.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:40.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:40.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:40 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 8744/203115 objects degraded (4.305%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:40.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:01:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:01:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:40.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:40.705 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:40.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:40.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:40.806 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:40.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:40.838 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:40.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:40.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:40.959 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:40.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:40.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:41.061 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:41.088 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:41.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:41.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:41.190 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:41.218 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:41.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:41.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:41.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:41.361 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:41.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:41.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:41.463 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:41.476 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 52 16 74059 74043 88.9745 68.3125 0.0149568 0.0112336 2024-09-21T10:01:41.497 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:41.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:41.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:41.598 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:41 smithi107 ceph-mon[102221]: pgmap v539: 89 pgs: 1 active+recovery_wait+degraded+remapped, 7 active+clean+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 4658/209334 objects degraded (2.225%); 34325/209334 objects misplaced (16.397%); 6.8 MiB/s, 108 objects/s recovering 2024-09-21T10:01:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:41.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:41 smithi107 ceph-mon[105022]: pgmap v539: 89 pgs: 1 active+recovery_wait+degraded+remapped, 7 active+clean+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 4658/209334 objects degraded (2.225%); 34325/209334 objects misplaced (16.397%); 6.8 MiB/s, 108 objects/s recovering 2024-09-21T10:01:41.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:41.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:41.633 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:41.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:41.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:41 smithi129 ceph-mon[88471]: pgmap v539: 89 pgs: 1 active+recovery_wait+degraded+remapped, 7 active+clean+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 4658/209334 objects degraded (2.225%); 34325/209334 objects misplaced (16.397%); 6.8 MiB/s, 108 objects/s recovering 2024-09-21T10:01:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:41.735 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:41.761 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:41.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:41.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:41.863 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:41.886 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:41.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:41.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:41.988 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:42.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:42.010 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:42.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:42.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:01:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:01:41] "GET /metrics HTTP/1.1" 200 36602 "" "Prometheus/2.51.0" 2024-09-21T10:01:42.111 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:42.135 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:42.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:42.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:42.237 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:42.271 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:42.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:42.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:42.375 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:42.403 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:42.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:42.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:42.476 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 53 16 74896 74880 88.2828 52.3125 0.0100998 0.0113215 2024-09-21T10:01:42.504 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:42.532 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:42.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:42.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:42.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:42.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:42.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:42.776 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:42.803 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:42.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:42.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:42.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:42.942 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:42.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:42.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:43.044 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:43.070 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:43.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:43.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:43.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:43.215 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:43.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:43.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:43.317 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:43.346 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:43.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:43.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:43.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:43.468 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:43 smithi107 ceph-mon[102221]: pgmap v540: 89 pgs: 8 active+clean+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 4.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 4496/214725 objects degraded (2.094%); 35567/214725 objects misplaced (16.564%); 7.6 MiB/s, 122 objects/s recovering 2024-09-21T10:01:43.468 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:43 smithi107 ceph-mon[105022]: pgmap v540: 89 pgs: 8 active+clean+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 4.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 4496/214725 objects degraded (2.094%); 35567/214725 objects misplaced (16.564%); 7.6 MiB/s, 122 objects/s recovering 2024-09-21T10:01:43.476 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 54 16 76094 76078 88.0344 74.875 0.00744127 0.0113532 2024-09-21T10:01:43.497 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:43.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:43.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:43.599 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:43.629 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:43.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:43.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:43 smithi129 ceph-mon[88471]: pgmap v540: 89 pgs: 8 active+clean+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 4.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 4496/214725 objects degraded (2.094%); 35567/214725 objects misplaced (16.564%); 7.6 MiB/s, 122 objects/s recovering 2024-09-21T10:01:43.731 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:43.754 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:43.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:43.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:43.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:43.890 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:43.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:43.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:43.991 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:44.019 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:44.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:44.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:44.120 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:44.160 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:44.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:44.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:44.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:44.312 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:44.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:44.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:44.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:44.449 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:44.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:44.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:44.476 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 55 16 77116 77100 87.5951 63.875 0.00719259 0.0114107 2024-09-21T10:01:44.550 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:44.576 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:44.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:44.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:44.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:44.721 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:44.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:44.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:44.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:44.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:44.870 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:44.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:44.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:44.975 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:01:44.975 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:01:45.017 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:01:45.017 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:01:45.048 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:45.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:45.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:45.050 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:01:45.050 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:01:45.092 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:01:45.093 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:01:45.128 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:01:45.128 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:01:45.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:45.164 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:01:45.164 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:01:45.195 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:01:45.196 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:01:45.196 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T10:01:45.204 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:45.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:45.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:45.306 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:45.335 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:45.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:45.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:45 smithi107 ceph-mon[102221]: pgmap v541: 89 pgs: 8 active+clean+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 4496/222408 objects degraded (2.022%); 37179/222408 objects misplaced (16.717%); 7.2 MiB/s, 114 objects/s recovering 2024-09-21T10:01:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:45 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 4496/222408 objects degraded (2.022%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:45 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3451368943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:01:45.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:45 smithi107 ceph-mon[105022]: pgmap v541: 89 pgs: 8 active+clean+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 4496/222408 objects degraded (2.022%); 37179/222408 objects misplaced (16.717%); 7.2 MiB/s, 114 objects/s recovering 2024-09-21T10:01:45.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:45 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 4496/222408 objects degraded (2.022%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:45.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:45 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3451368943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:01:45.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:01:45.437 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:45.462 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:45.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:45.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:45.476 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 56 16 78202 78186 87.2429 67.875 0.00856451 0.011456 2024-09-21T10:01:45.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:45.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:45.600 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:45.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:45 smithi129 ceph-mon[88471]: pgmap v541: 89 pgs: 8 active+clean+remapped, 21 active+undersized, 1 active+recovering+degraded+remapped, 9 active+undersized+degraded, 50 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 4496/222408 objects degraded (2.022%); 37179/222408 objects misplaced (16.717%); 7.2 MiB/s, 114 objects/s recovering 2024-09-21T10:01:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:45 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 4496/222408 objects degraded (2.022%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:45 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3451368943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:01:45.700 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:45.724 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:45.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:45.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:45.826 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:45.854 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:45.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:45.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:45.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:45.999 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:46.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:46.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:46.101 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:46.129 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:46.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:46.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:46.151 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:01:46.151 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:01:46.184 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:01:46.186 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:01:46.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:46.276 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:46.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:46.377 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:46.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:46.428 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:46.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:46.476 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 57 16 79309 79293 86.9259 69.1875 0.0100122 0.011498 2024-09-21T10:01:46.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:46.571 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:46.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:46.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:46.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:46.727 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:46.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:46.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:46.829 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:46.854 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:46.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:46.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:46.955 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:46.981 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:46.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:46.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:47.082 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:47.108 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:47.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:47.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:47.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:47.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:47.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:47.350 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:47.394 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:47.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:47.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:47.476 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 58 15 80377 80362 86.5791 66.8125 0.00472282 0.0115444 2024-09-21T10:01:47.496 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:47.522 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:47.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:47.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:47.536 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:47 smithi107 ceph-mon[102221]: pgmap v542: 89 pgs: 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 4.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 15/230238 objects degraded (0.007%); 42993/230238 objects misplaced (18.673%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T10:01:47.536 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:47 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:47.536 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:47 smithi107 ceph-mon[105022]: pgmap v542: 89 pgs: 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 4.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 15/230238 objects degraded (0.007%); 42993/230238 objects misplaced (18.673%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T10:01:47.536 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:47 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:47.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:47.667 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:47.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:47.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:47.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:47 smithi129 ceph-mon[88471]: pgmap v542: 89 pgs: 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 4.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 15/230238 objects degraded (0.007%); 42993/230238 objects misplaced (18.673%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T10:01:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:47 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:47.769 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:47.795 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:47.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:47.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:47.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:47.934 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:47.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:47.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:48.036 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:48.065 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:48.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:48.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:48.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:01:48.167 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:48.189 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:48.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:48.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:48.274 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:01:48.291 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:48.314 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:48.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:48.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:48.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:48 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:48.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:48 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:48.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:48 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:01:48.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:48 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:48.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:48 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:48.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:48 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:48.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:48 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:01:48.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:48 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:48.416 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:48.439 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:48.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:48.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:48.477 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 59 16 81353 81337 86.1444 60.9375 0.0252136 0.0116032 2024-09-21T10:01:48.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:48.587 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:48.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:48.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:48.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:48 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:48.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:48 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:48.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:48 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:01:48.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:48 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:48.689 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:48.725 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:48.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:48.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:48.826 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:48.855 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:48.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:48.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:48.957 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:48.991 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:48.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:48.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:49.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:49.130 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:49.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:49.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:49.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:49.270 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:49.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:49.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:49.372 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:49.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:49.403 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:49.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:49.477 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:01:49.475926+0000 min lat: 0.0012462 max lat: 2.31158 avg lat: 0.0116448 2024-09-21T10:01:49.477 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:01:49.477 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 60 16 82437 82421 85.8378 67.75 0.0149647 0.0116448 2024-09-21T10:01:49.504 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:49.530 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:49.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:49.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:49 smithi107 ceph-mon[102221]: pgmap v543: 89 pgs: 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 15/235332 objects degraded (0.006%); 44000/235332 objects misplaced (18.697%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T10:01:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:49 smithi107 ceph-mon[102221]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T10:01:49.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:49 smithi107 ceph-mon[105022]: pgmap v543: 89 pgs: 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 15/235332 objects degraded (0.006%); 44000/235332 objects misplaced (18.697%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T10:01:49.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:49 smithi107 ceph-mon[105022]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T10:01:49.632 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:49.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:49.657 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:49.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:49 smithi129 ceph-mon[88471]: pgmap v543: 89 pgs: 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 15/235332 objects degraded (0.006%); 44000/235332 objects misplaced (18.697%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T10:01:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:49 smithi129 ceph-mon[88471]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T10:01:49.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:49.779 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:49.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:49.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:49.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:49.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:49.911 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:49.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:50.012 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:50.041 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:50.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:50.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:50.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:50.177 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:50.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:50.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:50.280 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:50.304 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:50.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:50.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:50.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:50.447 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:50.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:50.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:50.477 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 61 16 83339 83323 85.3547 56.375 0.00526257 0.0117103 2024-09-21T10:01:50.549 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:50.561 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:50 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/235332 objects degraded (0.006%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:50.562 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:01:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:01:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:50.582 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:50.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:50.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:50.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:50 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/235332 objects degraded (0.006%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:50.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:50 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/235332 objects degraded (0.006%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:50.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:50.724 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:50.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:50.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:50.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:50.846 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:50.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:50.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:50.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:50.975 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:50.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:50.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:51.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:51.098 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:51.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:51.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:51.200 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:51.246 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:51.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:51.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:51.348 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:51.375 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:51.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:51.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:51.477 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 62 16 84141 84125 84.7865 50.125 0.00909148 0.0117892 2024-09-21T10:01:51.477 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:51.488 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:51 smithi107 ceph-mon[102221]: pgmap v544: 89 pgs: 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 4.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 15/241329 objects degraded (0.006%); 44987/241329 objects misplaced (18.641%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T10:01:51.488 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:51 smithi107 ceph-mon[105022]: pgmap v544: 89 pgs: 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 4.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 15/241329 objects degraded (0.006%); 44987/241329 objects misplaced (18.641%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T10:01:51.504 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:51.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:51.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:51.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:51.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:51.650 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:51.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:51.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:51 smithi129 ceph-mon[88471]: pgmap v544: 89 pgs: 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 4.9 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 15/241329 objects degraded (0.006%); 44987/241329 objects misplaced (18.641%); 3.2 MiB/s, 51 objects/s recovering 2024-09-21T10:01:51.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:51.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:51.797 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:51.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:51.898 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:51.924 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:51.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:51.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:52.025 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:52.028 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:01:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:01:51] "GET /metrics HTTP/1.1" 200 36602 "" "Prometheus/2.51.0" 2024-09-21T10:01:52.047 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:52.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:52.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:52.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:52.206 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:52.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:52.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:52.308 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:52.333 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:52.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:52.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:52.435 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:52.459 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:52.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:52.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:52.477 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 63 16 85228 85212 84.519 67.9375 0.018682 0.0118264 2024-09-21T10:01:52.561 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:52.585 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:52.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:52.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:52.687 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:52.712 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:52.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:52.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:52.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:52.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:52.859 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:52.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:52.960 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:52.985 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:52.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:52.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:53.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:53.128 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:53.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:53.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:53.230 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:53.256 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:53.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:53.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:53.266 INFO:teuthology.orchestra.run.smithi107.stderr:set pool 1 pg_num to 11 2024-09-21T10:01:53.357 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:53.381 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:53.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:53.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:53.480 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 64 16 86251 86235 84.194 63.9375 0.0148756 0.0118724 2024-09-21T10:01:53.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:53.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:53.530 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:53.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:53.551 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:01:53.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:53 smithi107 ceph-mon[102221]: pgmap v545: 89 pgs: 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.05k op/s; 15/246156 objects degraded (0.006%); 45932/246156 objects misplaced (18.660%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:01:53.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:53 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3410789540' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T10:01:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:53 smithi107 ceph-mon[105022]: pgmap v545: 89 pgs: 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.05k op/s; 15/246156 objects degraded (0.006%); 45932/246156 objects misplaced (18.660%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:01:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:53 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3410789540' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T10:01:53.631 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:53.655 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:53.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:53.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:53 smithi129 ceph-mon[88471]: pgmap v545: 89 pgs: 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 5.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 65 MiB/s wr, 1.05k op/s; 15/246156 objects degraded (0.006%); 45932/246156 objects misplaced (18.660%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:01:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:53 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3410789540' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T10:01:53.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:53.794 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:53.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:53.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:53.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:53.936 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:53.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:53.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:54.038 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:54.062 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:54.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:54.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:54.163 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:54.187 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:54.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:54.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:54.290 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:54.317 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:54.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:54.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:54.417 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:54 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3410789540' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T10:01:54.417 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:54 smithi107 ceph-mon[105022]: osdmap e156: 8 total, 7 up, 8 in 2024-09-21T10:01:54.417 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:54 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T10:01:54.417 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:54 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3410789540' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T10:01:54.418 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:54 smithi107 ceph-mon[102221]: osdmap e156: 8 total, 7 up, 8 in 2024-09-21T10:01:54.418 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:54 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T10:01:54.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:54.468 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:54.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:54.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:54.479 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 65 16 87295 87279 83.9025 65.25 0.0130424 0.0119135 2024-09-21T10:01:54.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:54.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:54.622 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:54.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:54 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3410789540' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T10:01:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:54 smithi129 ceph-mon[88471]: osdmap e156: 8 total, 7 up, 8 in 2024-09-21T10:01:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:54 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T10:01:54.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:54.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:54.769 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:54.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:54.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:54.908 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:54.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:54.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:55.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:55.059 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:55.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:55.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:55.096 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:01:55.097 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:01:55.162 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:55.189 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:55.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:55.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:55.290 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:55.318 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:55.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:55.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:55.349 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:01:55.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:55.466 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:55.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:55.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:55.480 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 66 16 88109 88093 83.4021 50.875 0.0287648 0.0119846 2024-09-21T10:01:55.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:55 smithi107 ceph-mon[102221]: pgmap v547: 89 pgs: 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 61 MiB/s wr, 968 op/s; 15/250641 objects degraded (0.006%); 46818/250641 objects misplaced (18.679%); 323 KiB/s, 5 objects/s recovering 2024-09-21T10:01:55.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:55 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T10:01:55.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:55 smithi107 ceph-mon[102221]: osdmap e157: 8 total, 7 up, 8 in 2024-09-21T10:01:55.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:55 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/250641 objects degraded (0.006%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:55.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:55 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1425206558' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:01:55.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:55 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:01:55.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:55 smithi107 ceph-mon[105022]: pgmap v547: 89 pgs: 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 61 MiB/s wr, 968 op/s; 15/250641 objects degraded (0.006%); 46818/250641 objects misplaced (18.679%); 323 KiB/s, 5 objects/s recovering 2024-09-21T10:01:55.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:55 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T10:01:55.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:55 smithi107 ceph-mon[105022]: osdmap e157: 8 total, 7 up, 8 in 2024-09-21T10:01:55.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:55 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/250641 objects degraded (0.006%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:55.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:55 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1425206558' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:01:55.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:55 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:01:55.568 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:55.608 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:55.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:55.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:55 smithi129 ceph-mon[88471]: pgmap v547: 89 pgs: 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 61 MiB/s wr, 968 op/s; 15/250641 objects degraded (0.006%); 46818/250641 objects misplaced (18.679%); 323 KiB/s, 5 objects/s recovering 2024-09-21T10:01:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:55 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T10:01:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:55 smithi129 ceph-mon[88471]: osdmap e157: 8 total, 7 up, 8 in 2024-09-21T10:01:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:55 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/250641 objects degraded (0.006%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:55 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1425206558' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:01:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:55 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:01:55.710 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:55.739 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:55.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:55.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:55.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:55.885 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:55.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:55.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:55.987 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:56.011 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:56.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:56.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:56.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:56.151 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:56.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:56.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:56.254 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:56.279 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:56.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:56.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:56.380 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:56.407 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:56.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:56.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:56.480 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 67 16 89080 89064 83.063 60.6875 0.00519698 0.0120337 2024-09-21T10:01:56.509 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:56.540 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:56.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:56.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:56 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:01:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:56 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:01:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:56 smithi107 ceph-mon[102221]: osdmap e158: 8 total, 7 up, 8 in 2024-09-21T10:01:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:56 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:01:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:56 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:01:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:56 smithi107 ceph-mon[105022]: osdmap e158: 8 total, 7 up, 8 in 2024-09-21T10:01:56.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:56.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:56.641 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:56 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:01:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:56 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:01:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:56 smithi129 ceph-mon[88471]: osdmap e158: 8 total, 7 up, 8 in 2024-09-21T10:01:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:01:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:56.683 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:56.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:56.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:56.785 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:56.812 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:56.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:56.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:56.914 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:56.952 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:56.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:56.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:57.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:57.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:01:57.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:57.105 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:57.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:57.205 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:57.229 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:57.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:57.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:57.330 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:57.356 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:57.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:57.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:57.458 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:57.481 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 68 16 90145 90129 82.8193 66.5625 0.0221198 0.01207 2024-09-21T10:01:57.483 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:57.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:57.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:57.585 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:57.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:57 smithi107 ceph-mon[102221]: pgmap v550: 99 pgs: 10 unknown, 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 15/260670 objects degraded (0.006%); 48647/260670 objects misplaced (18.662%) 2024-09-21T10:01:57.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:57 smithi107 ceph-mon[105022]: pgmap v550: 99 pgs: 10 unknown, 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 15/260670 objects degraded (0.006%); 48647/260670 objects misplaced (18.662%) 2024-09-21T10:01:57.611 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:57.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:57.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:57 smithi129 ceph-mon[88471]: pgmap v550: 99 pgs: 10 unknown, 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 15/260670 objects degraded (0.006%); 48647/260670 objects misplaced (18.662%) 2024-09-21T10:01:57.713 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:57.738 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:57.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:57.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:57.840 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:57.867 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:57.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:57.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:57.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:58.010 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:58.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:58.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:58.051 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:01:58.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:01:58.163 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:58.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:58.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:01:58.265 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:58.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:58.295 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:58.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:58.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:01:58.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:58.443 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:58.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:58.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:58.481 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 69 16 91157 91141 82.5356 63.25 0.0168554 0.01211 2024-09-21T10:01:58.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:58.593 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:58.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:58.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:58.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:58.738 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:58.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:58.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:01:58.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:58.880 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:58.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:58.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:58.982 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:01:59.007 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:59.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:59.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:01:59.109 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:01:59.133 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:59.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:59.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:01:59.205 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-09-21T10:01:59.205 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:01:59.205 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:01:59.205 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-09-21T10:01:59.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 1 1 2024-09-21T10:01:59.234 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:59.306 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:59.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:59.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:59.407 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:59.433 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:59.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:59.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:59.482 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 70 16 92093 92077 82.1915 58.5 0.0211958 0.0121618 2024-09-21T10:01:59.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:01:59.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:59.584 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:59.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:01:59.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:59 smithi107 ceph-mon[102221]: pgmap v551: 99 pgs: 10 unknown, 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 15/265293 objects degraded (0.006%); 49585/265293 objects misplaced (18.691%) 2024-09-21T10:01:59.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:01:59 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/265293 objects degraded (0.006%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:59.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:59 smithi107 ceph-mon[105022]: pgmap v551: 99 pgs: 10 unknown, 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 15/265293 objects degraded (0.006%); 49585/265293 objects misplaced (18.691%) 2024-09-21T10:01:59.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:01:59 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/265293 objects degraded (0.006%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:59 smithi129 ceph-mon[88471]: pgmap v551: 99 pgs: 10 unknown, 9 active+clean+remapped, 21 active+undersized, 9 active+undersized+degraded, 50 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 15/265293 objects degraded (0.006%); 49585/265293 objects misplaced (18.691%) 2024-09-21T10:01:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:01:59 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/265293 objects degraded (0.006%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:01:59.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:01:59.736 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:01:59.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:59.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:01:59.840 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:01:59.866 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:01:59.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:01:59.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:01:59.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:01:59.978 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:01:59.978 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:02:00.011 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:00.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:00.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:00.024 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:02:00.024 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:02:00.057 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:02:00.108 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:02:00.109 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:02:00.109 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:02:00.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:00.133 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:02:00.134 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:02:00.149 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:00.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:00.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:00.160 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:02:00.160 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:02:00.192 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:02:00.192 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:02:00.193 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T10:02:00.251 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:00.288 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:00.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:00.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:00.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:00.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:00.433 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:00.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:00.482 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 71 16 93045 93029 81.8714 59.5 0.0240816 0.0122092 2024-09-21T10:02:00.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:00.579 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:00.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:00.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:00 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1816310833' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:02:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:02:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:02:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:00.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:00.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:00 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1816310833' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:02:00.681 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:00 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1816310833' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:02:00.719 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:00.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:00.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:00.821 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:00.848 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:00.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:00.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:00.949 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:00.990 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:00.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:00.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:01.051 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:02:01.051 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:02:01.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:01.133 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:01.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:01.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:01.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:01.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:01.272 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:01.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:01.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:01.416 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:01.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:01.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:01.482 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 72 16 94070 94054 81.624 64.0625 0.009534 0.0122455 2024-09-21T10:02:01.518 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:01.543 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:01.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:01.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:01.555 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:01 smithi107 ceph-mon[102221]: pgmap v552: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 15/270726 objects degraded (0.006%); 50504/270726 objects misplaced (18.655%) 2024-09-21T10:02:01.556 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:01 smithi107 ceph-mon[105022]: pgmap v552: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 15/270726 objects degraded (0.006%); 50504/270726 objects misplaced (18.655%) 2024-09-21T10:02:01.568 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:01.610 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:02:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:02:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-09-21T10:02:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-09-21T10:02:01.611 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:02:01.639 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:01.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:01 smithi129 ceph-mon[88471]: pgmap v552: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 15/270726 objects degraded (0.006%); 50504/270726 objects misplaced (18.655%) 2024-09-21T10:02:01.698 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:01.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:01.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:01.800 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:01.825 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:01.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:01.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:01.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:02:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:02:01] "GET /metrics HTTP/1.1" 200 36654 "" "Prometheus/2.51.0" 2024-09-21T10:02:01.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:01.969 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:01.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:01.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:02.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:02.108 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:02.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:02.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:02.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:02.248 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:02.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:02.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:02.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:02:02.350 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:02.400 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:02.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:02.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:02.483 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 73 16 95058 95042 81.3514 61.75 0.0215222 0.0122874 2024-09-21T10:02:02.502 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:02.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:02.565 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:02.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:02.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:02.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:02.716 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:02.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:02.817 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:02.841 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:02.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:02.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:02.943 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:02.970 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:02.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:02.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:03.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:02:03.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:03.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:03.167 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:03.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:03.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:03.324 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:03.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:03.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:03.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:03.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:03.468 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:03.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:03.483 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 74 16 95922 95906 80.9818 54 0.0228278 0.0123434 2024-09-21T10:02:03.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:03.606 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:03.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:03.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:03 smithi129 ceph-mon[88471]: pgmap v553: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s; 15/275496 objects degraded (0.005%); 51469/275496 objects misplaced (18.682%) 2024-09-21T10:02:03.708 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:03.734 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:03.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:03.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:03.836 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:03.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:03 smithi107 ceph-mon[102221]: pgmap v553: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s; 15/275496 objects degraded (0.005%); 51469/275496 objects misplaced (18.682%) 2024-09-21T10:02:03.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:03 smithi107 ceph-mon[105022]: pgmap v553: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s; 15/275496 objects degraded (0.005%); 51469/275496 objects misplaced (18.682%) 2024-09-21T10:02:03.861 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:03.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:03.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:03.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:03.995 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:03.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:03.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:04.097 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:04.134 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:04.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:04.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:04.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:04.279 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:04.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:04.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:04.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:04.410 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:04.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:04.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:04.483 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 75 16 96839 96823 80.6662 57.3125 0.0107114 0.0123912 2024-09-21T10:02:04.512 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:04.538 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:04.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:04.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:04.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:04 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/275496 objects degraded (0.005%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:02:04.686 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:04.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:04.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:04.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:04.827 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:04.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:04.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:04 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/275496 objects degraded (0.005%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:02:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:04 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/275496 objects degraded (0.005%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:02:04.929 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:04.968 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:04.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:04.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:05.070 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:05.105 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:05.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:05.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:05.206 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:05.233 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:05.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:05.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:05.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:05.384 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:05.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:05.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:05.483 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 76 16 97774 97758 80.3737 58.4375 0.0222061 0.0124373 2024-09-21T10:02:05.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:05.528 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:05.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:05.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:05.630 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:05.654 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:05.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:05.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:05.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:05 smithi129 ceph-mon[88471]: pgmap v554: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 58 MiB/s wr, 926 op/s; 15/280008 objects degraded (0.005%); 52349/280008 objects misplaced (18.696%) 2024-09-21T10:02:05.756 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:05.782 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:05.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:05.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:05.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:05 smithi107 ceph-mon[102221]: pgmap v554: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 58 MiB/s wr, 926 op/s; 15/280008 objects degraded (0.005%); 52349/280008 objects misplaced (18.696%) 2024-09-21T10:02:05.793 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:05 smithi107 ceph-mon[105022]: pgmap v554: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 58 MiB/s wr, 926 op/s; 15/280008 objects degraded (0.005%); 52349/280008 objects misplaced (18.696%) 2024-09-21T10:02:05.884 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:05.910 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:05.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:05.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:06.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:06.045 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:06.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:06.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:06.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:06.168 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:06.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:06.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:06.270 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:06.296 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:06.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:06.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:06.398 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:06.423 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:06.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:06.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:06.483 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 77 16 98564 98548 79.9711 49.375 0.0298068 0.0124988 2024-09-21T10:02:06.524 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:06.550 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:06.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:06.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:06.652 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:06.692 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:06.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:06.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:06.794 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:06.819 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:06.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:06.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:06.921 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:06.957 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:06.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:06.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:07.059 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:07.084 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:07.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:07.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:07.186 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:07.211 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:07.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:07.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:07.312 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:07.340 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:07.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:07.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:07.441 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:07.467 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:07.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:07.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:07.483 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 78 16 99559 99543 79.7431 62.1875 0.00832099 0.0125349 2024-09-21T10:02:07.499 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:07 smithi107 ceph-mon[105022]: pgmap v555: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 5.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 15/289638 objects degraded (0.005%); 54143/289638 objects misplaced (18.693%) 2024-09-21T10:02:07.499 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:07 smithi107 ceph-mon[102221]: pgmap v555: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 5.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 15/289638 objects degraded (0.005%); 54143/289638 objects misplaced (18.693%) 2024-09-21T10:02:07.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:07.614 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:07.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:07.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:07.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:07 smithi129 ceph-mon[88471]: pgmap v555: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 5.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 15/289638 objects degraded (0.005%); 54143/289638 objects misplaced (18.693%) 2024-09-21T10:02:07.716 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:07.758 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:07.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:07.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:07.860 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:07.884 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:07.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:07.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:07.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:07.987 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:08.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:08.052 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:08.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:08.072 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:08.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:08.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:08.201 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:08.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:08.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:08.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:08.346 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:08.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:08.447 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:08.472 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:08.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:08.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:08.483 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 79 16 100493 100477 79.4725 58.375 0.00839245 0.0125777 2024-09-21T10:02:08.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:08.615 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:08.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:08.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:08.717 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:08.738 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:08.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:08.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:08.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:08.876 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:08.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:08.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:08.978 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:09.018 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:09.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:09.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:09.119 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:09.139 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:09.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:09.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:09.241 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:09.267 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:09.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:09.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:09.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:09.410 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:09.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:09.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:09.483 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:02:09.482457+0000 min lat: 0.0012462 max lat: 2.31158 avg lat: 0.012644 2024-09-21T10:02:09.483 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:02:09.484 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 80 16 101228 101212 79.0534 45.9375 0.0111838 0.012644 2024-09-21T10:02:09.512 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:09.533 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:09.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:09.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:09.634 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:09.656 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:09.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:09.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:09.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:09 smithi129 ceph-mon[88471]: pgmap v556: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 58 MiB/s wr, 924 op/s; 15/293982 objects degraded (0.005%); 55015/293982 objects misplaced (18.714%) 2024-09-21T10:02:09.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:09 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/293982 objects degraded (0.005%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:02:09.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:09.799 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:09.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:09.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:09.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:09 smithi107 ceph-mon[102221]: pgmap v556: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 58 MiB/s wr, 924 op/s; 15/293982 objects degraded (0.005%); 55015/293982 objects misplaced (18.714%) 2024-09-21T10:02:09.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:09 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/293982 objects degraded (0.005%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:02:09.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:09 smithi107 ceph-mon[105022]: pgmap v556: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 58 MiB/s wr, 924 op/s; 15/293982 objects degraded (0.005%); 55015/293982 objects misplaced (18.714%) 2024-09-21T10:02:09.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:09 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/293982 objects degraded (0.005%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:02:09.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:09.938 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:09.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:10.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:10.074 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:10.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:10.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:10.176 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:10.201 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:10.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:10.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:10.303 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:10.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:10.334 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:10.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:10.436 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:10.476 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:10.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:10.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:10.485 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 81 16 101895 101879 78.5921 41.6875 0.0130347 0.0127182 2024-09-21T10:02:10.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:10.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:10.620 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:10.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:10.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:02:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:02:10.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:10.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:10.766 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:10.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:10.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:10.867 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:10.893 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:10.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:10.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:10.995 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:11.038 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:11.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:11.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:11.140 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:11.183 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:11.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:11.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:11.285 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:11.308 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:11.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:11.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:11.410 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:11.415 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-21T10:02:11.452 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:11.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:11.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:11.483 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 82 16 102823 102807 78.3409 58 0.0441014 0.0127596 2024-09-21T10:02:11.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:11.596 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:11.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:11.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:11.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:11.744 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:11.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:11.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:11.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:11 smithi107 ceph-mon[102221]: pgmap v557: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 58 MiB/s wr, 932 op/s; 15/298869 objects degraded (0.005%); 55836/298869 objects misplaced (18.682%) 2024-09-21T10:02:11.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:11 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2157570971' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:02:11.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:11 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:02:11.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:11.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:11 smithi107 ceph-mon[105022]: pgmap v557: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 58 MiB/s wr, 932 op/s; 15/298869 objects degraded (0.005%); 55836/298869 objects misplaced (18.682%) 2024-09-21T10:02:11.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:11 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2157570971' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:02:11.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:11 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:02:11.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:11.845 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:11.870 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:11.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:11.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:11 smithi129 ceph-mon[88471]: pgmap v557: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 58 MiB/s wr, 932 op/s; 15/298869 objects degraded (0.005%); 55836/298869 objects misplaced (18.682%) 2024-09-21T10:02:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:11 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2157570971' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:02:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:11 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:02:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:11.973 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:12.000 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:12.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:12.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:12.101 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:12.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:02:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:02:11] "GET /metrics HTTP/1.1" 200 36662 "" "Prometheus/2.51.0" 2024-09-21T10:02:12.127 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:12.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:12.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:12.228 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:12.256 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:12.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:12.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:12.358 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:12.383 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:12.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:12.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:12.484 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 83 16 103810 103794 78.1402 61.6875 0.0104491 0.012792 2024-09-21T10:02:12.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:12.524 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:12.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:12.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:12.625 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:12.652 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:12.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:12.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:12.724 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:12 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:02:12.724 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:12 smithi107 ceph-mon[105022]: osdmap e159: 8 total, 7 up, 8 in 2024-09-21T10:02:12.724 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:12 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:02:12.724 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:12 smithi107 ceph-mon[102221]: osdmap e159: 8 total, 7 up, 8 in 2024-09-21T10:02:12.754 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:12.781 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:12.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:12.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:12.883 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:12.906 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:12.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:12.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:12.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:12 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:02:12.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:12 smithi129 ceph-mon[88471]: osdmap e159: 8 total, 7 up, 8 in 2024-09-21T10:02:13.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:13.049 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:13.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:13.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:13.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:13.191 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:13.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:13.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:13.293 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:13.318 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:13.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:13.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:13.420 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:13.446 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:02:13.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:13.446 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:13.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:13.486 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 84 16 104654 104638 77.8353 52.75 0.00840207 0.0128411 2024-09-21T10:02:13.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:13.588 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:13.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:13.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:13.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:13 smithi107 ceph-mon[102221]: pgmap v559: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 911 op/s; 15/302850 objects degraded (0.005%); 56654/302850 objects misplaced (18.707%) 2024-09-21T10:02:13.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:13 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2315765664' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:02:13.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:13 smithi107 ceph-mon[105022]: pgmap v559: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 911 op/s; 15/302850 objects degraded (0.005%); 56654/302850 objects misplaced (18.707%) 2024-09-21T10:02:13.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:13 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2315765664' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:02:13.689 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:13.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:13.734 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:13.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:13.837 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:13.863 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:13.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:13.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:13 smithi129 ceph-mon[88471]: pgmap v559: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 57 MiB/s wr, 911 op/s; 15/302850 objects degraded (0.005%); 56654/302850 objects misplaced (18.707%) 2024-09-21T10:02:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:13 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2315765664' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:02:13.966 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:13.992 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:13.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:13.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:14.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:14.137 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:14.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:14.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:14.237 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:02:14.237 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:02:14.239 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:14.264 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:14.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:14.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:14.277 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:02:14.277 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:02:14.310 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:02:14.310 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:02:14.350 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:02:14.351 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:02:14.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:14.378 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:02:14.378 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:02:14.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:14.406 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:14.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:14.412 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:02:14.412 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:02:14.444 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:02:14.445 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:02:14.450 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T10:02:14.487 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 85 16 105559 105543 77.585 56.5625 0.0216246 0.012884 2024-09-21T10:02:14.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:14.546 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:14.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:14.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:14 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:02:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:14 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2315765664' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:02:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:14 smithi107 ceph-mon[102221]: osdmap e160: 8 total, 7 up, 8 in 2024-09-21T10:02:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:14 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2427330269' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:02:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:14 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/302850 objects degraded (0.005%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:02:14.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:14 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:02:14.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:14 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2315765664' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:02:14.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:14 smithi107 ceph-mon[105022]: osdmap e160: 8 total, 7 up, 8 in 2024-09-21T10:02:14.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:14 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2427330269' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:02:14.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:14 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/302850 objects degraded (0.005%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:02:14.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:14.685 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:14.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:14.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:14.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:14.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:14.825 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:14.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:14 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:02:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:14 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2315765664' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:02:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:14 smithi129 ceph-mon[88471]: osdmap e160: 8 total, 7 up, 8 in 2024-09-21T10:02:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:14 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2427330269' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:02:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:14 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/302850 objects degraded (0.005%), 10 pgs degraded (PG_DEGRADED) 2024-09-21T10:02:14.926 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:14.951 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:14.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:14.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:15.053 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:15.077 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:15.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:15.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:15.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:15.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:15.218 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:15.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:15.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:15.360 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:15.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:15.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:15.462 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:15.487 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 86 16 106469 106453 77.3442 56.875 0.0126162 0.012924 2024-09-21T10:02:15.488 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:15.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:15.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:15.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:15.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:15.638 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:15.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:15.651 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:02:15.651 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:02:15.739 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:15.764 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:15.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:15.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:15.770 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:02:15.770 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:02:15.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:15 smithi107 ceph-mon[102221]: pgmap v561: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 45 MiB/s wr, 713 op/s; 15/306777 objects degraded (0.005%); 57431/306777 objects misplaced (18.721%) 2024-09-21T10:02:15.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:15 smithi107 ceph-mon[105022]: pgmap v561: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 45 MiB/s wr, 713 op/s; 15/306777 objects degraded (0.005%); 57431/306777 objects misplaced (18.721%) 2024-09-21T10:02:15.865 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:15.889 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:15.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:15.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:15 smithi129 ceph-mon[88471]: pgmap v561: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 45 MiB/s wr, 713 op/s; 15/306777 objects degraded (0.005%); 57431/306777 objects misplaced (18.721%) 2024-09-21T10:02:15.992 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:16.017 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:16.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:16.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:16.119 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:16.162 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:16.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:16.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:16.264 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:16.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:16.291 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:16.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:16.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:16.428 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:16.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:16.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:16.487 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 87 16 107427 107411 77.1433 59.875 0.00991043 0.0129573 2024-09-21T10:02:16.530 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:16.556 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:16.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:16.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:16.658 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:16.684 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:16.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:16.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:16.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:02:16.785 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:16.810 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:16.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:16.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:16.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:16.914 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:02:16.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:16.971 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:16.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:17.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:17.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:17.160 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:17.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:17.229 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:17.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:17.329 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:17.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:17.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:17.431 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:17.469 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:17.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:17.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:17.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 88 16 108251 108235 76.8515 51.5 0.0181995 0.0130067 2024-09-21T10:02:17.571 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:17.599 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:17.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:17.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:17.652 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:02:17.685 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:17 smithi107 ceph-mon[102221]: pgmap v562: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 57 MiB/s wr, 918 op/s; 15/316035 objects degraded (0.005%); 59083/316035 objects misplaced (18.695%) 2024-09-21T10:02:17.685 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:17 smithi107 ceph-mon[105022]: pgmap v562: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 57 MiB/s wr, 918 op/s; 15/316035 objects degraded (0.005%); 59083/316035 objects misplaced (18.695%) 2024-09-21T10:02:17.700 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:17.725 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:17.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:17.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:17.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:17.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:17.886 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:17.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:17 smithi129 ceph-mon[88471]: pgmap v562: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 57 MiB/s wr, 918 op/s; 15/316035 objects degraded (0.005%); 59083/316035 objects misplaced (18.695%) 2024-09-21T10:02:17.987 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:18.044 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:18.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:18.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:18.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:18.192 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:18.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:18.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:18.294 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:18.334 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:18.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:18.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:18.436 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:18.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:18.477 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:18.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:18.487 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 89 16 109117 109101 76.5961 54.125 0.0101869 0.0130497 2024-09-21T10:02:18.546 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:18.578 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:18.641 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:18.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:18.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:18.740 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-09-21T10:02:18.740 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:02:18.740 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_4 2024-09-21T10:02:18.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:02:18.743 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:18.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:18.826 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:18.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:18.927 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:18.948 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:18.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:18.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:19.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:19.095 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:19.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:19.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:19.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:19.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:19.238 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:19.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:19.339 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:19.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:19.366 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:19.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:19.467 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:19.487 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total time run: 90.0146 2024-09-21T10:02:19.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total writes made: 109934 2024-09-21T10:02:19.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Write size: 65536 2024-09-21T10:02:19.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object size: 65536 2024-09-21T10:02:19.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Bandwidth (MB/sec): 76.3306 2024-09-21T10:02:19.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Bandwidth: 37.3148 2024-09-21T10:02:19.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max bandwidth (MB/sec): 205.625 2024-09-21T10:02:19.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min bandwidth (MB/sec): 0 2024-09-21T10:02:19.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average IOPS: 1221 2024-09-21T10:02:19.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev IOPS: 597.058 2024-09-21T10:02:19.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max IOPS: 3290 2024-09-21T10:02:19.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min IOPS: 0 2024-09-21T10:02:19.488 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average Latency(s): 0.0130964 2024-09-21T10:02:19.489 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Latency(s): 0.0386807 2024-09-21T10:02:19.489 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max latency(s): 2.31158 2024-09-21T10:02:19.489 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min latency(s): 0.0012462 2024-09-21T10:02:19.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:19.490 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:19.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:19.495 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:02:19.570 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:19.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:19.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:19.638 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:19.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:19.739 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:19.764 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:19.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:19.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:19.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:19 smithi107 ceph-mon[102221]: pgmap v563: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 55 MiB/s wr, 884 op/s; 15/320112 objects degraded (0.005%); 59909/320112 objects misplaced (18.715%) 2024-09-21T10:02:19.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:19 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/320112 objects degraded (0.005%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:19.791 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:19 smithi107 ceph-mon[105022]: pgmap v563: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 55 MiB/s wr, 884 op/s; 15/320112 objects degraded (0.005%); 59909/320112 objects misplaced (18.715%) 2024-09-21T10:02:19.791 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:19 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/320112 objects degraded (0.005%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:19.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:19.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:19.914 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:19.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:19 smithi129 ceph-mon[88471]: pgmap v563: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 55 MiB/s wr, 884 op/s; 15/320112 objects degraded (0.005%); 59909/320112 objects misplaced (18.715%) 2024-09-21T10:02:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:19 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/320112 objects degraded (0.005%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:20.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:20.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:20.062 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:20.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:20.163 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:20.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:20.190 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:20.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:20.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:20.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:20.339 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:20.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:20.434 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:02:20.434 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:02:20.440 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:20.478 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:20.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:20.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:20.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:20.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:20.625 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:20.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:02:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:02:20.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:20.726 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:20.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:20.760 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:20.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:20.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:20.899 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:20.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:20.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:21.001 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:21.029 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:21.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:21.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:21.131 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:21.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:21.158 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:21.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:21.260 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:21.286 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:21.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:21.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:21.388 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:21.435 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:21.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:21.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:21.537 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:21.561 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:21.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:21.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:21.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:21.697 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:21.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:21.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:21 smithi107 ceph-mon[102221]: pgmap v564: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 120 B/s rd, 54 MiB/s wr, 875 op/s; 15/324804 objects degraded (0.005%); 60718/324804 objects misplaced (18.694%) 2024-09-21T10:02:21.705 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:21 smithi107 ceph-mon[105022]: pgmap v564: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 120 B/s rd, 54 MiB/s wr, 875 op/s; 15/324804 objects degraded (0.005%); 60718/324804 objects misplaced (18.694%) 2024-09-21T10:02:21.799 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:21.825 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:21.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:21.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:21 smithi129 ceph-mon[88471]: pgmap v564: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 120 B/s rd, 54 MiB/s wr, 875 op/s; 15/324804 objects degraded (0.005%); 60718/324804 objects misplaced (18.694%) 2024-09-21T10:02:21.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:21.961 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:21.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:22.040 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:22.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:22.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:02:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:02:21] "GET /metrics HTTP/1.1" 200 36662 "" "Prometheus/2.51.0" 2024-09-21T10:02:22.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:22.126 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:22.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:22.226 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:22.252 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:22.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:22.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:22.354 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:22.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:22.381 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:22.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:22.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:22.519 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:22.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:22.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:22.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:02:22.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:22.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:22.679 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:22.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:22.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:22.816 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:22.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:22.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:22.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:22.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:22.973 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:22.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:23.074 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:23.100 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:23.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:23.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:23.202 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:23.225 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:23.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:23.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:23.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:23.369 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:23.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:23.470 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:23.494 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:23.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:23.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:23.596 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:23.622 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:23.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:23.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:23.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:23.765 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:23.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:23.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:23.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:23 smithi107 ceph-mon[102221]: pgmap v565: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 53 MiB/s wr, 879 op/s; 15/327030 objects degraded (0.005%); 61151/327030 objects misplaced (18.699%) 2024-09-21T10:02:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:23 smithi107 ceph-mon[105022]: pgmap v565: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 53 MiB/s wr, 879 op/s; 15/327030 objects degraded (0.005%); 61151/327030 objects misplaced (18.699%) 2024-09-21T10:02:23.867 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:23.913 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:23.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:23.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:23 smithi129 ceph-mon[88471]: pgmap v565: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 53 MiB/s wr, 879 op/s; 15/327030 objects degraded (0.005%); 61151/327030 objects misplaced (18.699%) 2024-09-21T10:02:24.015 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:24.040 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:24.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:24.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:24.142 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:24.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:24.166 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:24.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:24.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:24.315 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:24.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:24.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:24.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:24.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:24.467 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:24.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:24.568 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:24.590 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:24.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:24.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:24.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:24 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/327030 objects degraded (0.005%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:24.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:24 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/327030 objects degraded (0.005%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:24.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:24.738 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:24.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:24.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:24.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:24.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:24.887 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:24.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:24.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:24 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/327030 objects degraded (0.005%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:24.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:25.035 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:25.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:25.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:25.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:25.167 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:25.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:25.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:25.269 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:25.302 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:25.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:25.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:25.404 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:25.445 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:25.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:25.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:25.539 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:02:25.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:25.591 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:25.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:25.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:25.716 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:25.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:25.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:25.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:25 smithi107 ceph-mon[102221]: pgmap v566: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 98 B/s rd, 46 MiB/s wr, 847 op/s; 15/325938 objects degraded (0.005%); 60978/325938 objects misplaced (18.708%) 2024-09-21T10:02:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:25 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1327447112' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:02:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:25 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:02:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:25 smithi107 ceph-mon[105022]: pgmap v566: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 98 B/s rd, 46 MiB/s wr, 847 op/s; 15/325938 objects degraded (0.005%); 60978/325938 objects misplaced (18.708%) 2024-09-21T10:02:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:25 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1327447112' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:02:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:25 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:02:25.863 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:25.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:25.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:25 smithi129 ceph-mon[88471]: pgmap v566: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 98 B/s rd, 46 MiB/s wr, 847 op/s; 15/325938 objects degraded (0.005%); 60978/325938 objects misplaced (18.708%) 2024-09-21T10:02:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:25 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1327447112' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:02:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:25 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:02:25.964 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:25.987 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:25.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:26.088 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:26.110 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:26.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:26.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:26.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:26.232 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:26.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:26.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:26.334 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:26.357 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:26.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:26.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:26.460 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:26.484 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:26.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:26.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:26.586 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:26.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:26.621 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:26.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:26.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:26.759 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:26.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:26.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:26 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:02:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:26 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:02:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:26 smithi107 ceph-mon[102221]: osdmap e161: 8 total, 7 up, 8 in 2024-09-21T10:02:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:26 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:02:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:26 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:02:26.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:26 smithi107 ceph-mon[105022]: osdmap e161: 8 total, 7 up, 8 in 2024-09-21T10:02:26.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:26.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:26.917 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:26.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:26.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:26 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:02:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:26 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:02:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:26 smithi129 ceph-mon[88471]: osdmap e161: 8 total, 7 up, 8 in 2024-09-21T10:02:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:27.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:27.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:27.066 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:27.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:27.167 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:27.191 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:27.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:27.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:27.293 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:27.317 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:27.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:27.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:27.419 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:27.445 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:27.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:27.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:27.547 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:27.571 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:27.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:27.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:27.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:27.714 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:27.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:27.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:27.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:27.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:27 smithi107 ceph-mon[105022]: pgmap v568: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 29 MiB/s wr, 874 op/s; 15/317481 objects degraded (0.005%); 59443/317481 objects misplaced (18.723%) 2024-09-21T10:02:27.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:27 smithi107 ceph-mon[102221]: pgmap v568: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 29 MiB/s wr, 874 op/s; 15/317481 objects degraded (0.005%); 59443/317481 objects misplaced (18.723%) 2024-09-21T10:02:27.860 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:27.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:27.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:27 smithi129 ceph-mon[88471]: pgmap v568: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 29 MiB/s wr, 874 op/s; 15/317481 objects degraded (0.005%); 59443/317481 objects misplaced (18.723%) 2024-09-21T10:02:27.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:28.003 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:28.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:28.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:28.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:28.139 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:28.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:28.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:28.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:28.281 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:28.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:28.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:28.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:28.394 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:28.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:28.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:28.452 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:02:28.452 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:02:28.489 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:02:28.489 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:02:28.522 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:02:28.523 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:02:28.527 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:28.552 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:28.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:28.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:28.556 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:02:28.556 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:02:28.604 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:02:28.604 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:02:28.642 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:02:28.642 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:02:28.654 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:28.677 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:28.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:28.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:28.680 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:02:28.680 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:02:28.681 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T10:02:28.779 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:28.797 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:28 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/363809011' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:02:28.797 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:28 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/363809011' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:02:28.805 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:28.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:28.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:28.908 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:28.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:28 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/363809011' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:02:28.932 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:28.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:28.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:29.035 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:02:29.036 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:02:29.037 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:29.060 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:29.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:29.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:29.162 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:29.187 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:29.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:29.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:29.289 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:29.315 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:29.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:29.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:29.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:29.455 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:29.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:29.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:29.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:29.590 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:29.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:29.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:29.692 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:29.716 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:29.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:29.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:29.785 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:29 smithi107 ceph-mon[105022]: pgmap v569: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 102 B/s rd, 20 MiB/s wr, 856 op/s; 15/313938 objects degraded (0.005%); 58709/313938 objects misplaced (18.701%) 2024-09-21T10:02:29.785 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:29 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/313938 objects degraded (0.005%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:29.786 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:29 smithi107 ceph-mon[102221]: pgmap v569: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 102 B/s rd, 20 MiB/s wr, 856 op/s; 15/313938 objects degraded (0.005%); 58709/313938 objects misplaced (18.701%) 2024-09-21T10:02:29.786 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:29 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/313938 objects degraded (0.005%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:29.817 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:02:29.821 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:02:29.852 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:29.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:29.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:29 smithi129 ceph-mon[88471]: pgmap v569: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 102 B/s rd, 20 MiB/s wr, 856 op/s; 15/313938 objects degraded (0.005%); 58709/313938 objects misplaced (18.701%) 2024-09-21T10:02:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:29 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/313938 objects degraded (0.005%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:29.954 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:29.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:29.995 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:29.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:30.031 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:02:30.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:30.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:30.151 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:30.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:30.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:30.300 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:30.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:30.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:30.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:30.455 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:30.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:30.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:30.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:30.562 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:02:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:02:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:30.602 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:30.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:30.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:30.700 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:30 smithi107 ceph-mon[102221]: from='client.36048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:02:30.701 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:30 smithi107 ceph-mon[105022]: from='client.36048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:02:30.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:30.743 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":569,"stamp":"2024-09-21T10:02:27.890386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6856906725,"num_objects":104646,"num_object_clones":0,"num_object_copies":313938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":58709,"num_objects_unfound":0,"num_objects_dirty":104646,"num_whiteouts":0,"num_read":293,"num_read_kb":748,"num_write":115463,"num_write_kb":7037437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3722,"num_bytes_recovered":243608104,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116309,"ondisk_log_size":116309,"up":248,"acting":257,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":296,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":24596176,"kb_used_data":20249036,"kb_used_omap":29,"kb_used_meta":4346978,"kb_avail":725365040,"statfs":{"total":767960285184,"available":742773800960,"internally_reserved":0,"allocated":20735012864,"data_stored":20728588522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30226,"internal_metadata":4451305966},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-134938600,"num_objects":-2058,"num_object_clones":0,"num_object_copies":-6174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1200,"num_objects_unfound":0,"num_objects_dirty":-2058,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8572,"num_write_kb":208385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004124"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":164,"reported_epoch":161,"state":"active+undersized","last_fresh":"2024-09-21T10:02:25.539996+0000","last_change":"2024-09-21T10:01:17.360671+0000","last_active":"2024-09-21T10:02:25.539996+0000","last_peered":"2024-09-21T10:02:25.539996+0000","last_clean":"2024-09-21T10:01:12.339813+0000","last_became_active":"2024-09-21T10:01:17.360671+0000","last_became_peered":"2024-09-21T10:01:17.360671+0000","last_unstale":"2024-09-21T10:02:25.539996+0000","last_undegraded":"2024-09-21T10:02:25.539996+0000","last_fullsized":"2024-09-21T10:01:17.352087+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:38.696084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":49,"reported_epoch":160,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:02:13.452915+0000","last_change":"2024-09-21T10:01:17.355722+0000","last_active":"2024-09-21T10:02:13.452915+0000","last_peered":"2024-09-21T10:02:13.452915+0000","last_clean":"2024-09-21T10:00:52.395654+0000","last_became_active":"2024-09-21T10:01:17.355722+0000","last_became_peered":"2024-09-21T10:01:17.355722+0000","last_unstale":"2024-09-21T10:02:13.452915+0000","last_undegraded":"2024-09-21T10:01:17.351745+0000","last_fullsized":"2024-09-21T10:01:17.351642+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:08:49.082950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":152,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.443430+0000","last_change":"2024-09-21T10:01:17.360872+0000","last_active":"2024-09-21T10:02:13.443430+0000","last_peered":"2024-09-21T10:02:13.443430+0000","last_clean":"2024-09-21T10:01:12.334087+0000","last_became_active":"2024-09-21T10:01:17.360872+0000","last_became_peered":"2024-09-21T10:01:17.360872+0000","last_unstale":"2024-09-21T10:02:13.443430+0000","last_undegraded":"2024-09-21T10:02:13.443430+0000","last_fullsized":"2024-09-21T10:01:17.356555+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:03.409955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":212,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.452313+0000","last_change":"2024-09-21T10:01:17.362044+0000","last_active":"2024-09-21T10:02:13.452313+0000","last_peered":"2024-09-21T10:02:13.452313+0000","last_clean":"2024-09-21T10:01:12.346595+0000","last_became_active":"2024-09-21T10:01:17.362044+0000","last_became_peered":"2024-09-21T10:01:17.362044+0000","last_unstale":"2024-09-21T10:02:13.452313+0000","last_undegraded":"2024-09-21T10:02:13.452313+0000","last_fullsized":"2024-09-21T10:01:17.354667+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:41.227793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.454097+0000","last_change":"2024-09-21T10:01:17.358516+0000","last_active":"2024-09-21T10:02:13.454097+0000","last_peered":"2024-09-21T10:02:13.454097+0000","last_clean":"2024-09-21T10:01:12.350555+0000","last_became_active":"2024-09-21T10:01:17.358516+0000","last_became_peered":"2024-09-21T10:01:17.358516+0000","last_unstale":"2024-09-21T10:02:13.454097+0000","last_undegraded":"2024-09-21T10:02:13.454097+0000","last_fullsized":"2024-09-21T10:01:17.354339+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:23.544961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":32,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.453532+0000","last_change":"2024-09-21T10:01:17.362650+0000","last_active":"2024-09-21T10:02:13.453532+0000","last_peered":"2024-09-21T10:02:13.453532+0000","last_clean":"2024-09-21T09:40:48.067317+0000","last_became_active":"2024-09-21T10:01:17.362650+0000","last_became_peered":"2024-09-21T10:01:17.362650+0000","last_unstale":"2024-09-21T10:02:13.453532+0000","last_undegraded":"2024-09-21T10:02:13.453532+0000","last_fullsized":"2024-09-21T10:01:17.351981+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:14.466960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":164,"reported_epoch":161,"state":"active+undersized","last_fresh":"2024-09-21T10:02:25.535582+0000","last_change":"2024-09-21T10:01:17.361530+0000","last_active":"2024-09-21T10:02:25.535582+0000","last_peered":"2024-09-21T10:02:25.535582+0000","last_clean":"2024-09-21T10:01:12.338661+0000","last_became_active":"2024-09-21T10:01:17.361530+0000","last_became_peered":"2024-09-21T10:01:17.361530+0000","last_unstale":"2024-09-21T10:02:25.535582+0000","last_undegraded":"2024-09-21T10:02:25.535582+0000","last_fullsized":"2024-09-21T10:01:17.354825+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:10.575192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":133,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.468848+0000","last_change":"2024-09-21T09:48:54.140948+0000","last_active":"2024-09-21T10:02:13.468848+0000","last_peered":"2024-09-21T10:02:13.468848+0000","last_clean":"2024-09-21T10:02:13.468848+0000","last_became_active":"2024-09-21T09:48:54.140641+0000","last_became_peered":"2024-09-21T09:48:54.140641+0000","last_unstale":"2024-09-21T10:02:13.468848+0000","last_undegraded":"2024-09-21T10:02:13.468848+0000","last_fullsized":"2024-09-21T10:02:13.468848+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:25.157599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"160'7129","reported_seq":8113,"reported_epoch":160,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:23.920171+0000","last_change":"2024-09-21T10:01:25.317266+0000","last_active":"2024-09-21T10:02:23.920171+0000","last_peered":"2024-09-21T10:02:23.920171+0000","last_clean":"2024-09-21T10:02:23.920171+0000","last_became_active":"2024-09-21T10:01:18.394618+0000","last_became_peered":"2024-09-21T10:01:18.394618+0000","last_unstale":"2024-09-21T10:02:23.920171+0000","last_undegraded":"2024-09-21T10:02:23.920171+0000","last_fullsized":"2024-09-21T10:02:23.920171+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7129,"log_dups_size":0,"ondisk_log_size":7129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:20:01.567428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436404224,"num_objects":6659,"num_object_clones":0,"num_object_copies":19977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6659,"num_objects_unfound":0,"num_objects_dirty":6659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7129,"num_write_kb":441216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":458,"num_bytes_recovered":30015488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,1],"avail_no_missing":["0","1","5"],"object_location_counts":[{"shards":"0,1,5","objects":6659}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":47,"reported_epoch":160,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:02:13.453210+0000","last_change":"2024-09-21T10:01:17.362126+0000","last_active":"2024-09-21T10:02:13.453210+0000","last_peered":"2024-09-21T10:02:13.453210+0000","last_clean":"2024-09-21T10:00:52.396178+0000","last_became_active":"2024-09-21T10:01:17.362126+0000","last_became_peered":"2024-09-21T10:01:17.362126+0000","last_unstale":"2024-09-21T10:02:13.453210+0000","last_undegraded":"2024-09-21T10:01:17.352720+0000","last_fullsized":"2024-09-21T10:01:17.352607+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:15.152821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":135,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.444438+0000","last_change":"2024-09-21T10:01:17.360241+0000","last_active":"2024-09-21T10:02:13.444438+0000","last_peered":"2024-09-21T10:02:13.444438+0000","last_clean":"2024-09-21T10:01:12.331943+0000","last_became_active":"2024-09-21T10:01:17.360241+0000","last_became_peered":"2024-09-21T10:01:17.360241+0000","last_unstale":"2024-09-21T10:02:13.444438+0000","last_undegraded":"2024-09-21T10:02:13.444438+0000","last_fullsized":"2024-09-21T10:01:17.354797+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:56.749660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"160'7156","reported_seq":7205,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:24.746540+0000","last_change":"2024-09-21T10:00:39.617105+0000","last_active":"2024-09-21T10:02:24.746540+0000","last_peered":"2024-09-21T10:02:24.746540+0000","last_clean":"2024-09-21T10:02:24.746540+0000","last_became_active":"2024-09-21T10:00:39.616814+0000","last_became_peered":"2024-09-21T10:00:39.616814+0000","last_unstale":"2024-09-21T10:02:24.746540+0000","last_undegraded":"2024-09-21T10:02:24.746540+0000","last_fullsized":"2024-09-21T10:02:24.746540+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7156,"log_dups_size":0,"ondisk_log_size":7156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:56.849964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430178304,"num_objects":6564,"num_object_clones":0,"num_object_copies":19692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7156,"num_write_kb":439040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":47,"reported_epoch":160,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:02:13.453080+0000","last_change":"2024-09-21T10:01:17.365156+0000","last_active":"2024-09-21T10:02:13.453080+0000","last_peered":"2024-09-21T10:02:13.453080+0000","last_clean":"2024-09-21T10:00:52.396279+0000","last_became_active":"2024-09-21T10:01:17.365156+0000","last_became_peered":"2024-09-21T10:01:17.365156+0000","last_unstale":"2024-09-21T10:02:13.453080+0000","last_undegraded":"2024-09-21T10:01:17.352823+0000","last_fullsized":"2024-09-21T10:01:17.352726+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:25.514616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":141,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.452791+0000","last_change":"2024-09-21T09:56:12.576294+0000","last_active":"2024-09-21T10:02:13.452791+0000","last_peered":"2024-09-21T10:02:13.452791+0000","last_clean":"2024-09-21T10:02:13.452791+0000","last_became_active":"2024-09-21T09:56:12.575928+0000","last_became_peered":"2024-09-21T09:56:12.575928+0000","last_unstale":"2024-09-21T10:02:13.452791+0000","last_undegraded":"2024-09-21T10:02:13.452791+0000","last_fullsized":"2024-09-21T10:02:13.452791+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:23.227435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"161'7297","reported_seq":8703,"reported_epoch":161,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:27.578138+0000","last_change":"2024-09-21T10:01:21.071741+0000","last_active":"2024-09-21T10:02:27.578138+0000","last_peered":"2024-09-21T10:02:27.578138+0000","last_clean":"2024-09-21T10:02:27.578138+0000","last_became_active":"2024-09-21T10:01:18.387241+0000","last_became_peered":"2024-09-21T10:01:18.387241+0000","last_unstale":"2024-09-21T10:02:27.578138+0000","last_undegraded":"2024-09-21T10:02:27.578138+0000","last_fullsized":"2024-09-21T10:02:27.578138+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7297,"log_dups_size":0,"ondisk_log_size":7297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:02:42.497261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421593088,"num_objects":6433,"num_object_clones":0,"num_object_copies":19299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6433,"num_objects_unfound":0,"num_objects_dirty":6433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7297,"num_write_kb":439360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":671,"num_bytes_recovered":43974656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3,2],"avail_no_missing":["5","2","3"],"object_location_counts":[{"shards":"2,3,5","objects":6433}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":47,"reported_epoch":160,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:02:13.443342+0000","last_change":"2024-09-21T10:01:17.360223+0000","last_active":"2024-09-21T10:02:13.443342+0000","last_peered":"2024-09-21T10:02:13.443342+0000","last_clean":"2024-09-21T10:00:52.397515+0000","last_became_active":"2024-09-21T10:01:17.360223+0000","last_became_peered":"2024-09-21T10:01:17.360223+0000","last_unstale":"2024-09-21T10:02:13.443342+0000","last_undegraded":"2024-09-21T10:01:17.357081+0000","last_fullsized":"2024-09-21T10:01:17.356981+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:14:39.079600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":151,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.468816+0000","last_change":"2024-09-21T10:01:17.364112+0000","last_active":"2024-09-21T10:02:13.468816+0000","last_peered":"2024-09-21T10:02:13.468816+0000","last_clean":"2024-09-21T10:01:12.345222+0000","last_became_active":"2024-09-21T10:01:17.364112+0000","last_became_peered":"2024-09-21T10:01:17.364112+0000","last_unstale":"2024-09-21T10:02:13.468816+0000","last_undegraded":"2024-09-21T10:02:13.468816+0000","last_fullsized":"2024-09-21T10:01:17.355079+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:43:09.114787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"161'7235","reported_seq":7262,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-21T10:02:27.584438+0000","last_change":"2024-09-21T10:00:39.614448+0000","last_active":"2024-09-21T10:02:27.584438+0000","last_peered":"2024-09-21T10:02:27.584438+0000","last_clean":"2024-09-21T10:02:27.584438+0000","last_became_active":"2024-09-21T10:00:39.614071+0000","last_became_peered":"2024-09-21T10:00:39.614071+0000","last_unstale":"2024-09-21T10:02:27.584438+0000","last_undegraded":"2024-09-21T10:02:27.584438+0000","last_fullsized":"2024-09-21T10:02:27.584438+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7235,"log_dups_size":0,"ondisk_log_size":7235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:57.436342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422903808,"num_objects":6453,"num_object_clones":0,"num_object_copies":19359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7235,"num_write_kb":438016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":126,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.443219+0000","last_change":"2024-09-21T09:56:12.590219+0000","last_active":"2024-09-21T10:02:13.443219+0000","last_peered":"2024-09-21T10:02:13.443219+0000","last_clean":"2024-09-21T10:02:13.443219+0000","last_became_active":"2024-09-21T09:56:12.589946+0000","last_became_peered":"2024-09-21T09:56:12.589946+0000","last_unstale":"2024-09-21T10:02:13.443219+0000","last_undegraded":"2024-09-21T10:02:13.443219+0000","last_fullsized":"2024-09-21T10:02:13.443219+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:48.919929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":196,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.453266+0000","last_change":"2024-09-21T09:50:45.503336+0000","last_active":"2024-09-21T10:02:13.453266+0000","last_peered":"2024-09-21T10:02:13.453266+0000","last_clean":"2024-09-21T10:02:13.453266+0000","last_became_active":"2024-09-21T09:50:45.503045+0000","last_became_peered":"2024-09-21T09:50:45.503045+0000","last_unstale":"2024-09-21T10:02:13.453266+0000","last_undegraded":"2024-09-21T10:02:13.453266+0000","last_fullsized":"2024-09-21T10:02:13.453266+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:46.728347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":197,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.450555+0000","last_change":"2024-09-21T10:01:55.372679+0000","last_active":"2024-09-21T10:02:13.450555+0000","last_peered":"2024-09-21T10:02:13.450555+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.372679+0000","last_became_peered":"2024-09-21T10:01:55.372679+0000","last_unstale":"2024-09-21T10:02:13.450555+0000","last_undegraded":"2024-09-21T10:02:13.450555+0000","last_fullsized":"2024-09-21T10:01:55.355636+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:03:12.985869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"161'7178","reported_seq":8125,"reported_epoch":161,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:25.650292+0000","last_change":"2024-09-21T10:01:22.281434+0000","last_active":"2024-09-21T10:02:25.650292+0000","last_peered":"2024-09-21T10:02:25.650292+0000","last_clean":"2024-09-21T10:02:25.650292+0000","last_became_active":"2024-09-21T10:01:18.410689+0000","last_became_peered":"2024-09-21T10:01:18.410689+0000","last_unstale":"2024-09-21T10:02:25.650292+0000","last_undegraded":"2024-09-21T10:02:25.650292+0000","last_fullsized":"2024-09-21T10:02:25.650292+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7178,"log_dups_size":0,"ondisk_log_size":7178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:48.612460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427819008,"num_objects":6528,"num_object_clones":0,"num_object_copies":19584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6528,"num_objects_unfound":0,"num_objects_dirty":6528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7178,"num_write_kb":438592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":439,"num_bytes_recovered":28770304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":6528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":134,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-21T10:02:25.535911+0000","last_change":"2024-09-21T09:56:12.575897+0000","last_active":"2024-09-21T10:02:25.535911+0000","last_peered":"2024-09-21T10:02:25.535911+0000","last_clean":"2024-09-21T10:02:25.535911+0000","last_became_active":"2024-09-21T09:56:12.575532+0000","last_became_peered":"2024-09-21T09:56:12.575532+0000","last_unstale":"2024-09-21T10:02:25.535911+0000","last_undegraded":"2024-09-21T10:02:25.535911+0000","last_fullsized":"2024-09-21T10:02:25.535911+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:01.302499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":124,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.454952+0000","last_change":"2024-09-21T09:56:12.588824+0000","last_active":"2024-09-21T10:02:13.454952+0000","last_peered":"2024-09-21T10:02:13.454952+0000","last_clean":"2024-09-21T10:02:13.454952+0000","last_became_active":"2024-09-21T09:56:12.588561+0000","last_became_peered":"2024-09-21T09:56:12.588561+0000","last_unstale":"2024-09-21T10:02:13.454952+0000","last_undegraded":"2024-09-21T10:02:13.454952+0000","last_fullsized":"2024-09-21T10:02:13.454952+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:31:48.900623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":197,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.450614+0000","last_change":"2024-09-21T10:01:55.386193+0000","last_active":"2024-09-21T10:02:13.450614+0000","last_peered":"2024-09-21T10:02:13.450614+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.386193+0000","last_became_peered":"2024-09-21T10:01:55.386193+0000","last_unstale":"2024-09-21T10:02:13.450614+0000","last_undegraded":"2024-09-21T10:02:13.450614+0000","last_fullsized":"2024-09-21T10:01:55.366920+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"160'7240","reported_seq":7289,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:24.628499+0000","last_change":"2024-09-21T10:00:40.152470+0000","last_active":"2024-09-21T10:02:24.628499+0000","last_peered":"2024-09-21T10:02:24.628499+0000","last_clean":"2024-09-21T10:02:24.628499+0000","last_became_active":"2024-09-21T10:00:40.152054+0000","last_became_peered":"2024-09-21T10:00:40.152054+0000","last_unstale":"2024-09-21T10:02:24.628499+0000","last_undegraded":"2024-09-21T10:02:24.628499+0000","last_fullsized":"2024-09-21T10:02:24.628499+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7240,"log_dups_size":0,"ondisk_log_size":7240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:07.993580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437780480,"num_objects":6680,"num_object_clones":0,"num_object_copies":20040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7240,"num_write_kb":445440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":125,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.469019+0000","last_change":"2024-09-21T09:50:45.501596+0000","last_active":"2024-09-21T10:02:13.469019+0000","last_peered":"2024-09-21T10:02:13.469019+0000","last_clean":"2024-09-21T10:02:13.469019+0000","last_became_active":"2024-09-21T09:50:45.501186+0000","last_became_peered":"2024-09-21T09:50:45.501186+0000","last_unstale":"2024-09-21T10:02:13.469019+0000","last_undegraded":"2024-09-21T10:02:13.469019+0000","last_fullsized":"2024-09-21T10:02:13.469019+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:43:16.449425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":134,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.453134+0000","last_change":"2024-09-21T09:51:30.639545+0000","last_active":"2024-09-21T10:02:13.453134+0000","last_peered":"2024-09-21T10:02:13.453134+0000","last_clean":"2024-09-21T10:02:13.453134+0000","last_became_active":"2024-09-21T09:51:30.639106+0000","last_became_peered":"2024-09-21T09:51:30.639106+0000","last_unstale":"2024-09-21T10:02:13.453134+0000","last_undegraded":"2024-09-21T10:02:13.453134+0000","last_fullsized":"2024-09-21T10:02:13.453134+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:19:19.607327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"161'7121","reported_seq":8004,"reported_epoch":161,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:25.682499+0000","last_change":"2024-09-21T10:01:41.114121+0000","last_active":"2024-09-21T10:02:25.682499+0000","last_peered":"2024-09-21T10:02:25.682499+0000","last_clean":"2024-09-21T10:02:25.682499+0000","last_became_active":"2024-09-21T10:01:18.386759+0000","last_became_peered":"2024-09-21T10:01:18.386759+0000","last_unstale":"2024-09-21T10:02:25.682499+0000","last_undegraded":"2024-09-21T10:02:25.682499+0000","last_fullsized":"2024-09-21T10:02:25.682499+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7121,"log_dups_size":0,"ondisk_log_size":7121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:14:37.831906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420937728,"num_objects":6423,"num_object_clones":0,"num_object_copies":19269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6423,"num_objects_unfound":0,"num_objects_dirty":6423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7121,"num_write_kb":433408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":413,"num_bytes_recovered":27066368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,5],"avail_no_missing":["3","1","5"],"object_location_counts":[{"shards":"1,3,5","objects":6423}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":152,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.453384+0000","last_change":"2024-09-21T10:01:17.356446+0000","last_active":"2024-09-21T10:02:13.453384+0000","last_peered":"2024-09-21T10:02:13.453384+0000","last_clean":"2024-09-21T10:01:12.333151+0000","last_became_active":"2024-09-21T10:01:17.356446+0000","last_became_peered":"2024-09-21T10:01:17.356446+0000","last_unstale":"2024-09-21T10:02:13.453384+0000","last_undegraded":"2024-09-21T10:02:13.453384+0000","last_fullsized":"2024-09-21T10:01:17.352337+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:01:29.779281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":124,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.453217+0000","last_change":"2024-09-21T09:50:45.510764+0000","last_active":"2024-09-21T10:02:13.453217+0000","last_peered":"2024-09-21T10:02:13.453217+0000","last_clean":"2024-09-21T10:02:13.453217+0000","last_became_active":"2024-09-21T09:50:45.510226+0000","last_became_peered":"2024-09-21T09:50:45.510226+0000","last_unstale":"2024-09-21T10:02:13.453217+0000","last_undegraded":"2024-09-21T10:02:13.453217+0000","last_fullsized":"2024-09-21T10:02:13.453217+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:31:13.491427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":197,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.450689+0000","last_change":"2024-09-21T10:01:55.369453+0000","last_active":"2024-09-21T10:02:13.450689+0000","last_peered":"2024-09-21T10:02:13.450689+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.369453+0000","last_became_peered":"2024-09-21T10:01:55.369453+0000","last_unstale":"2024-09-21T10:02:13.450689+0000","last_undegraded":"2024-09-21T10:02:13.450689+0000","last_fullsized":"2024-09-21T10:01:55.356778+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:02:56.964734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"161'7397","reported_seq":7448,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-21T10:02:25.606574+0000","last_change":"2024-09-21T10:00:39.616847+0000","last_active":"2024-09-21T10:02:25.606574+0000","last_peered":"2024-09-21T10:02:25.606574+0000","last_clean":"2024-09-21T10:02:25.606574+0000","last_became_active":"2024-09-21T10:00:39.616555+0000","last_became_peered":"2024-09-21T10:00:39.616555+0000","last_unstale":"2024-09-21T10:02:25.606574+0000","last_undegraded":"2024-09-21T10:02:25.606574+0000","last_fullsized":"2024-09-21T10:02:25.606574+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7397,"log_dups_size":0,"ondisk_log_size":7397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:13.417964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443613184,"num_objects":6769,"num_object_clones":0,"num_object_copies":20307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7397,"num_write_kb":453312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":165,"reported_epoch":161,"state":"active+undersized","last_fresh":"2024-09-21T10:02:25.536570+0000","last_change":"2024-09-21T10:01:17.365027+0000","last_active":"2024-09-21T10:02:25.536570+0000","last_peered":"2024-09-21T10:02:25.536570+0000","last_clean":"2024-09-21T10:01:12.337728+0000","last_became_active":"2024-09-21T10:01:17.365027+0000","last_became_peered":"2024-09-21T10:01:17.365027+0000","last_unstale":"2024-09-21T10:02:25.536570+0000","last_undegraded":"2024-09-21T10:02:25.536570+0000","last_fullsized":"2024-09-21T10:01:17.352121+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:14:52.580064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":166,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.457169+0000","last_change":"2024-09-21T09:56:12.575396+0000","last_active":"2024-09-21T10:02:13.457169+0000","last_peered":"2024-09-21T10:02:13.457169+0000","last_clean":"2024-09-21T10:02:13.457169+0000","last_became_active":"2024-09-21T09:56:12.574913+0000","last_became_peered":"2024-09-21T09:56:12.574913+0000","last_unstale":"2024-09-21T10:02:13.457169+0000","last_undegraded":"2024-09-21T10:02:13.457169+0000","last_fullsized":"2024-09-21T10:02:13.457169+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:05.462880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"161'7130","reported_seq":7181,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-21T10:02:25.680367+0000","last_change":"2024-09-21T10:00:39.615333+0000","last_active":"2024-09-21T10:02:25.680367+0000","last_peered":"2024-09-21T10:02:25.680367+0000","last_clean":"2024-09-21T10:02:25.680367+0000","last_became_active":"2024-09-21T10:00:39.614910+0000","last_became_peered":"2024-09-21T10:00:39.614910+0000","last_unstale":"2024-09-21T10:02:25.680367+0000","last_undegraded":"2024-09-21T10:02:25.680367+0000","last_fullsized":"2024-09-21T10:02:25.680367+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7130,"log_dups_size":0,"ondisk_log_size":7130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:17.632326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422707200,"num_objects":6450,"num_object_clones":0,"num_object_copies":19350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7130,"num_write_kb":434560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":197,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.450924+0000","last_change":"2024-09-21T10:01:55.375667+0000","last_active":"2024-09-21T10:02:13.450924+0000","last_peered":"2024-09-21T10:02:13.450924+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.375667+0000","last_became_peered":"2024-09-21T10:01:55.375667+0000","last_unstale":"2024-09-21T10:02:13.450924+0000","last_undegraded":"2024-09-21T10:02:13.450924+0000","last_fullsized":"2024-09-21T10:01:55.353939+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":67,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":174,"reported_epoch":160,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:02:13.443992+0000","last_change":"2024-09-21T10:01:17.359511+0000","last_active":"2024-09-21T10:02:13.443992+0000","last_peered":"2024-09-21T10:02:13.443992+0000","last_clean":"2024-09-21T10:01:12.332227+0000","last_became_active":"2024-09-21T10:01:17.359511+0000","last_became_peered":"2024-09-21T10:01:17.359511+0000","last_unstale":"2024-09-21T10:02:13.443992+0000","last_undegraded":"2024-09-21T10:01:17.357161+0000","last_fullsized":"2024-09-21T10:01:17.356996+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:39:03.218322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039581700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":136,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-21T10:02:25.545269+0000","last_change":"2024-09-21T09:56:12.580057+0000","last_active":"2024-09-21T10:02:25.545269+0000","last_peered":"2024-09-21T10:02:25.545269+0000","last_clean":"2024-09-21T10:02:25.545269+0000","last_became_active":"2024-09-21T09:56:12.579700+0000","last_became_peered":"2024-09-21T09:56:12.579700+0000","last_unstale":"2024-09-21T10:02:25.545269+0000","last_undegraded":"2024-09-21T10:02:25.545269+0000","last_fullsized":"2024-09-21T10:02:25.545269+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:49.768872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":134,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.469228+0000","last_change":"2024-09-21T10:01:17.359297+0000","last_active":"2024-09-21T10:02:13.469228+0000","last_peered":"2024-09-21T10:02:13.469228+0000","last_clean":"2024-09-21T10:01:12.345523+0000","last_became_active":"2024-09-21T10:01:17.359297+0000","last_became_peered":"2024-09-21T10:01:17.359297+0000","last_unstale":"2024-09-21T10:02:13.469228+0000","last_undegraded":"2024-09-21T10:02:13.469228+0000","last_fullsized":"2024-09-21T10:01:17.354792+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:54:20.381939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":197,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.452947+0000","last_change":"2024-09-21T10:01:55.378211+0000","last_active":"2024-09-21T10:02:13.452947+0000","last_peered":"2024-09-21T10:02:13.452947+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.378211+0000","last_became_peered":"2024-09-21T10:01:55.378211+0000","last_unstale":"2024-09-21T10:02:13.452947+0000","last_undegraded":"2024-09-21T10:02:13.452947+0000","last_fullsized":"2024-09-21T10:01:55.368219+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":107340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"160'7155","reported_seq":7204,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:23.949861+0000","last_change":"2024-09-21T10:00:40.152326+0000","last_active":"2024-09-21T10:02:23.949861+0000","last_peered":"2024-09-21T10:02:23.949861+0000","last_clean":"2024-09-21T10:02:23.949861+0000","last_became_active":"2024-09-21T10:00:40.152002+0000","last_became_peered":"2024-09-21T10:00:40.152002+0000","last_unstale":"2024-09-21T10:02:23.949861+0000","last_undegraded":"2024-09-21T10:02:23.949861+0000","last_fullsized":"2024-09-21T10:02:23.949861+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7155,"log_dups_size":0,"ondisk_log_size":7155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:31.868823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436666368,"num_objects":6663,"num_object_clones":0,"num_object_copies":19989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7155,"num_write_kb":442176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":120,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.457637+0000","last_change":"2024-09-21T10:01:17.367117+0000","last_active":"2024-09-21T10:02:13.457637+0000","last_peered":"2024-09-21T10:02:13.457637+0000","last_clean":"2024-09-21T09:50:10.064980+0000","last_became_active":"2024-09-21T10:01:17.367117+0000","last_became_peered":"2024-09-21T10:01:17.367117+0000","last_unstale":"2024-09-21T10:02:13.457637+0000","last_undegraded":"2024-09-21T10:02:13.457637+0000","last_fullsized":"2024-09-21T10:01:17.359433+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:44.782398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":195,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.451262+0000","last_change":"2024-09-21T09:56:12.573698+0000","last_active":"2024-09-21T10:02:13.451262+0000","last_peered":"2024-09-21T10:02:13.451262+0000","last_clean":"2024-09-21T10:02:13.451262+0000","last_became_active":"2024-09-21T09:56:12.573390+0000","last_became_peered":"2024-09-21T09:56:12.573390+0000","last_unstale":"2024-09-21T10:02:13.451262+0000","last_undegraded":"2024-09-21T10:02:13.451262+0000","last_fullsized":"2024-09-21T10:02:13.451262+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:55:12.937878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":207,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.451240+0000","last_change":"2024-09-21T09:51:30.666462+0000","last_active":"2024-09-21T10:02:13.451240+0000","last_peered":"2024-09-21T10:02:13.451240+0000","last_clean":"2024-09-21T10:02:13.451240+0000","last_became_active":"2024-09-21T09:51:30.666006+0000","last_became_peered":"2024-09-21T09:51:30.666006+0000","last_unstale":"2024-09-21T10:02:13.451240+0000","last_undegraded":"2024-09-21T10:02:13.451240+0000","last_fullsized":"2024-09-21T10:02:13.451240+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:51.481698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":197,"reported_epoch":160,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:02:13.450812+0000","last_change":"2024-09-21T10:01:55.369767+0000","last_active":"2024-09-21T10:02:13.450812+0000","last_peered":"2024-09-21T10:02:13.450812+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.369767+0000","last_became_peered":"2024-09-21T10:01:55.369767+0000","last_unstale":"2024-09-21T10:02:13.450812+0000","last_undegraded":"2024-09-21T10:01:55.367783+0000","last_fullsized":"2024-09-21T10:01:55.358839+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:02:59.949481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":107340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"161'7155","reported_seq":8079,"reported_epoch":161,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:27.562446+0000","last_change":"2024-09-21T10:01:29.735004+0000","last_active":"2024-09-21T10:02:27.562446+0000","last_peered":"2024-09-21T10:02:27.562446+0000","last_clean":"2024-09-21T10:02:27.562446+0000","last_became_active":"2024-09-21T10:01:18.386142+0000","last_became_peered":"2024-09-21T10:01:18.386142+0000","last_unstale":"2024-09-21T10:02:27.562446+0000","last_undegraded":"2024-09-21T10:02:27.562446+0000","last_fullsized":"2024-09-21T10:02:27.562446+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7155,"log_dups_size":0,"ondisk_log_size":7155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:41.716874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414384128,"num_objects":6323,"num_object_clones":0,"num_object_copies":18969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6323,"num_objects_unfound":0,"num_objects_dirty":6323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7155,"num_write_kb":431296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":431,"num_bytes_recovered":28246016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1,3],"avail_no_missing":["5","1","3"],"object_location_counts":[{"shards":"1,3,5","objects":6323}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":197,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.460597+0000","last_change":"2024-09-21T09:56:12.580659+0000","last_active":"2024-09-21T10:02:13.460597+0000","last_peered":"2024-09-21T10:02:13.460597+0000","last_clean":"2024-09-21T10:02:13.460597+0000","last_became_active":"2024-09-21T09:56:12.580300+0000","last_became_peered":"2024-09-21T09:56:12.580300+0000","last_unstale":"2024-09-21T10:02:13.460597+0000","last_undegraded":"2024-09-21T10:02:13.460597+0000","last_fullsized":"2024-09-21T10:02:13.460597+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:04.945404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":165,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.457556+0000","last_change":"2024-09-21T09:51:30.646466+0000","last_active":"2024-09-21T10:02:13.457556+0000","last_peered":"2024-09-21T10:02:13.457556+0000","last_clean":"2024-09-21T10:02:13.457556+0000","last_became_active":"2024-09-21T09:51:30.646209+0000","last_became_peered":"2024-09-21T09:51:30.646209+0000","last_unstale":"2024-09-21T10:02:13.457556+0000","last_undegraded":"2024-09-21T10:02:13.457556+0000","last_fullsized":"2024-09-21T10:02:13.457556+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:44:41.465418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":170,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.465464+0000","last_change":"2024-09-21T09:51:30.649056+0000","last_active":"2024-09-21T10:02:13.465464+0000","last_peered":"2024-09-21T10:02:13.465464+0000","last_clean":"2024-09-21T10:02:13.465464+0000","last_became_active":"2024-09-21T09:51:30.647245+0000","last_became_peered":"2024-09-21T09:51:30.647245+0000","last_unstale":"2024-09-21T10:02:13.465464+0000","last_undegraded":"2024-09-21T10:02:13.465464+0000","last_fullsized":"2024-09-21T10:02:13.465464+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:32.661847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":196,"reported_epoch":160,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:02:13.450762+0000","last_change":"2024-09-21T10:01:55.369315+0000","last_active":"2024-09-21T10:02:13.450762+0000","last_peered":"2024-09-21T10:02:13.450762+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.369315+0000","last_became_peered":"2024-09-21T10:01:55.369315+0000","last_unstale":"2024-09-21T10:02:13.450762+0000","last_undegraded":"2024-09-21T10:01:55.367566+0000","last_fullsized":"2024-09-21T10:01:55.354220+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:03:10.982992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":107340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"160'7041","reported_seq":7962,"reported_epoch":160,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:23.991064+0000","last_change":"2024-09-21T10:01:36.934043+0000","last_active":"2024-09-21T10:02:23.991064+0000","last_peered":"2024-09-21T10:02:23.991064+0000","last_clean":"2024-09-21T10:02:23.991064+0000","last_became_active":"2024-09-21T10:01:18.418674+0000","last_became_peered":"2024-09-21T10:01:18.418674+0000","last_unstale":"2024-09-21T10:02:23.991064+0000","last_undegraded":"2024-09-21T10:02:23.991064+0000","last_fullsized":"2024-09-21T10:02:23.991064+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7041,"log_dups_size":0,"ondisk_log_size":7041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:32:50.784561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428670976,"num_objects":6541,"num_object_clones":0,"num_object_copies":19623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6541,"num_objects_unfound":0,"num_objects_dirty":6541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7041,"num_write_kb":434624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":427,"num_bytes_recovered":27983872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":6541}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":140,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.443618+0000","last_change":"2024-09-21T09:50:45.502094+0000","last_active":"2024-09-21T10:02:13.443618+0000","last_peered":"2024-09-21T10:02:13.443618+0000","last_clean":"2024-09-21T10:02:13.443618+0000","last_became_active":"2024-09-21T09:50:45.501603+0000","last_became_peered":"2024-09-21T09:50:45.501603+0000","last_unstale":"2024-09-21T10:02:13.443618+0000","last_undegraded":"2024-09-21T10:02:13.443618+0000","last_fullsized":"2024-09-21T10:02:13.443618+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:01.747067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091939799999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":125,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.453226+0000","last_change":"2024-09-21T09:48:54.128617+0000","last_active":"2024-09-21T10:02:13.453226+0000","last_peered":"2024-09-21T10:02:13.453226+0000","last_clean":"2024-09-21T10:02:13.453226+0000","last_became_active":"2024-09-21T09:48:54.127878+0000","last_became_peered":"2024-09-21T09:48:54.127878+0000","last_unstale":"2024-09-21T10:02:13.453226+0000","last_undegraded":"2024-09-21T10:02:13.453226+0000","last_fullsized":"2024-09-21T10:02:13.453226+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:07.305638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":177,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.455485+0000","last_change":"2024-09-21T09:56:12.588992+0000","last_active":"2024-09-21T10:02:13.455485+0000","last_peered":"2024-09-21T10:02:13.455485+0000","last_clean":"2024-09-21T10:02:13.455485+0000","last_became_active":"2024-09-21T09:56:12.588665+0000","last_became_peered":"2024-09-21T09:56:12.588665+0000","last_unstale":"2024-09-21T10:02:13.455485+0000","last_undegraded":"2024-09-21T10:02:13.455485+0000","last_fullsized":"2024-09-21T10:02:13.455485+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:19:06.160456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":197,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.451006+0000","last_change":"2024-09-21T10:01:55.368329+0000","last_active":"2024-09-21T10:02:13.451006+0000","last_peered":"2024-09-21T10:02:13.451006+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.368329+0000","last_became_peered":"2024-09-21T10:01:55.368329+0000","last_unstale":"2024-09-21T10:02:13.451006+0000","last_undegraded":"2024-09-21T10:02:13.451006+0000","last_fullsized":"2024-09-21T10:01:55.358639+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:02:55.947353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.f","version":"161'7156","reported_seq":7184,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-21T10:02:27.525193+0000","last_change":"2024-09-21T10:00:39.617307+0000","last_active":"2024-09-21T10:02:27.525193+0000","last_peered":"2024-09-21T10:02:27.525193+0000","last_clean":"2024-09-21T10:02:27.525193+0000","last_became_active":"2024-09-21T10:00:39.617020+0000","last_became_peered":"2024-09-21T10:00:39.617020+0000","last_unstale":"2024-09-21T10:02:27.525193+0000","last_undegraded":"2024-09-21T10:02:27.525193+0000","last_fullsized":"2024-09-21T10:02:27.525193+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7156,"log_dups_size":0,"ondisk_log_size":7156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:21:32.333188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414973952,"num_objects":6332,"num_object_clones":0,"num_object_copies":18996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7156,"num_write_kb":431616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":143,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.443538+0000","last_change":"2024-09-21T09:56:12.583280+0000","last_active":"2024-09-21T10:02:13.443538+0000","last_peered":"2024-09-21T10:02:13.443538+0000","last_clean":"2024-09-21T10:02:13.443538+0000","last_became_active":"2024-09-21T09:56:12.583007+0000","last_became_peered":"2024-09-21T09:56:12.583007+0000","last_unstale":"2024-09-21T10:02:13.443538+0000","last_undegraded":"2024-09-21T10:02:13.443538+0000","last_fullsized":"2024-09-21T10:02:13.443538+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:26.991596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":222,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.455631+0000","last_change":"2024-09-21T10:01:17.367142+0000","last_active":"2024-09-21T10:02:13.455631+0000","last_peered":"2024-09-21T10:02:13.455631+0000","last_clean":"2024-09-21T10:01:12.331437+0000","last_became_active":"2024-09-21T10:01:17.367142+0000","last_became_peered":"2024-09-21T10:01:17.367142+0000","last_unstale":"2024-09-21T10:02:13.455631+0000","last_undegraded":"2024-09-21T10:02:13.455631+0000","last_fullsized":"2024-09-21T10:01:17.359249+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:46.792787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":155,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.457421+0000","last_change":"2024-09-21T09:50:45.501349+0000","last_active":"2024-09-21T10:02:13.457421+0000","last_peered":"2024-09-21T10:02:13.457421+0000","last_clean":"2024-09-21T10:02:13.457421+0000","last_became_active":"2024-09-21T09:50:45.501065+0000","last_became_peered":"2024-09-21T09:50:45.501065+0000","last_unstale":"2024-09-21T10:02:13.457421+0000","last_undegraded":"2024-09-21T10:02:13.457421+0000","last_fullsized":"2024-09-21T10:02:13.457421+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:28:58.570586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":197,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.450923+0000","last_change":"2024-09-21T10:01:55.370970+0000","last_active":"2024-09-21T10:02:13.450923+0000","last_peered":"2024-09-21T10:02:13.450923+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.370970+0000","last_became_peered":"2024-09-21T10:01:55.370970+0000","last_unstale":"2024-09-21T10:02:13.450923+0000","last_undegraded":"2024-09-21T10:02:13.450923+0000","last_fullsized":"2024-09-21T10:01:55.359846+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:03:11.964683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"161'7213","reported_seq":7256,"reported_epoch":161,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:27.237736+0000","last_change":"2024-09-21T10:01:18.378023+0000","last_active":"2024-09-21T10:02:27.237736+0000","last_peered":"2024-09-21T10:02:27.237736+0000","last_clean":"2024-09-21T10:02:27.237736+0000","last_became_active":"2024-09-21T10:01:18.375695+0000","last_became_peered":"2024-09-21T10:01:18.375695+0000","last_unstale":"2024-09-21T10:02:27.237736+0000","last_undegraded":"2024-09-21T10:02:27.237736+0000","last_fullsized":"2024-09-21T10:02:27.237736+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7213,"log_dups_size":0,"ondisk_log_size":7213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:37:12.806595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421330944,"num_objects":6429,"num_object_clones":0,"num_object_copies":19287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6429,"num_objects_unfound":0,"num_objects_dirty":6429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7213,"num_write_kb":436544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,0],"avail_no_missing":["4","0","1"],"object_location_counts":[{"shards":"0,1,4","objects":6429}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":165,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.455569+0000","last_change":"2024-09-21T09:51:30.646531+0000","last_active":"2024-09-21T10:02:13.455569+0000","last_peered":"2024-09-21T10:02:13.455569+0000","last_clean":"2024-09-21T10:02:13.455569+0000","last_became_active":"2024-09-21T09:51:30.646248+0000","last_became_peered":"2024-09-21T09:51:30.646248+0000","last_unstale":"2024-09-21T10:02:13.455569+0000","last_undegraded":"2024-09-21T10:02:13.455569+0000","last_fullsized":"2024-09-21T10:02:13.455569+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:00.312924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":89,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.454545+0000","last_change":"2024-09-21T10:01:17.364179+0000","last_active":"2024-09-21T10:02:13.454545+0000","last_peered":"2024-09-21T10:02:13.454545+0000","last_clean":"2024-09-21T10:00:52.397528+0000","last_became_active":"2024-09-21T10:01:17.364179+0000","last_became_peered":"2024-09-21T10:01:17.364179+0000","last_unstale":"2024-09-21T10:02:13.454545+0000","last_undegraded":"2024-09-21T10:02:13.454545+0000","last_fullsized":"2024-09-21T10:01:17.360621+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:26:17.268991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":203,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.453939+0000","last_change":"2024-09-21T10:01:17.371359+0000","last_active":"2024-09-21T10:02:13.453939+0000","last_peered":"2024-09-21T10:02:13.453939+0000","last_clean":"2024-09-21T10:01:12.347295+0000","last_became_active":"2024-09-21T10:01:17.371359+0000","last_became_peered":"2024-09-21T10:01:17.371359+0000","last_unstale":"2024-09-21T10:02:13.453939+0000","last_undegraded":"2024-09-21T10:02:13.453939+0000","last_fullsized":"2024-09-21T10:01:17.366525+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:38.963323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":197,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.454237+0000","last_change":"2024-09-21T10:01:55.373699+0000","last_active":"2024-09-21T10:02:13.454237+0000","last_peered":"2024-09-21T10:02:13.454237+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.373699+0000","last_became_peered":"2024-09-21T10:01:55.373699+0000","last_unstale":"2024-09-21T10:02:13.454237+0000","last_undegraded":"2024-09-21T10:02:13.454237+0000","last_fullsized":"2024-09-21T10:01:55.366377+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"160'7284","reported_seq":8179,"reported_epoch":160,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:24.776595+0000","last_change":"2024-09-21T10:01:29.079779+0000","last_active":"2024-09-21T10:02:24.776595+0000","last_peered":"2024-09-21T10:02:24.776595+0000","last_clean":"2024-09-21T10:02:24.776595+0000","last_became_active":"2024-09-21T10:01:18.393943+0000","last_became_peered":"2024-09-21T10:01:18.393943+0000","last_unstale":"2024-09-21T10:02:24.776595+0000","last_undegraded":"2024-09-21T10:02:24.776595+0000","last_fullsized":"2024-09-21T10:02:24.776595+0000","mapping_epoch":153,"log_start":"146'100","ondisk_log_start":"146'100","created":141,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7184,"log_dups_size":0,"ondisk_log_size":7184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:37:39.568085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441516056,"num_objects":6738,"num_object_clones":0,"num_object_copies":20214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6738,"num_objects_unfound":0,"num_objects_dirty":6738,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7284,"num_write_kb":448641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":418,"num_bytes_recovered":27394048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5,4],"avail_no_missing":["1","4","5"],"object_location_counts":[{"shards":"1,4,5","objects":6738}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":160,"reported_epoch":161,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:02:25.535851+0000","last_change":"2024-09-21T10:01:17.363383+0000","last_active":"2024-09-21T10:02:25.535851+0000","last_peered":"2024-09-21T10:02:25.535851+0000","last_clean":"2024-09-21T10:01:12.338791+0000","last_became_active":"2024-09-21T10:01:17.363383+0000","last_became_peered":"2024-09-21T10:01:17.363383+0000","last_unstale":"2024-09-21T10:02:25.535851+0000","last_undegraded":"2024-09-21T10:01:17.354694+0000","last_fullsized":"2024-09-21T10:01:17.354594+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:24:08.763448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":165,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.455955+0000","last_change":"2024-09-21T09:50:45.513017+0000","last_active":"2024-09-21T10:02:13.455955+0000","last_peered":"2024-09-21T10:02:13.455955+0000","last_clean":"2024-09-21T10:02:13.455955+0000","last_became_active":"2024-09-21T09:50:45.512629+0000","last_became_peered":"2024-09-21T09:50:45.512629+0000","last_unstale":"2024-09-21T10:02:13.455955+0000","last_undegraded":"2024-09-21T10:02:13.455955+0000","last_fullsized":"2024-09-21T10:02:13.455955+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:47:08.421575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":150,"reported_epoch":161,"state":"active+undersized","last_fresh":"2024-09-21T10:02:25.535689+0000","last_change":"2024-09-21T10:01:17.357067+0000","last_active":"2024-09-21T10:02:25.535689+0000","last_peered":"2024-09-21T10:02:25.535689+0000","last_clean":"2024-09-21T10:01:12.337480+0000","last_became_active":"2024-09-21T10:01:17.357067+0000","last_became_peered":"2024-09-21T10:01:17.357067+0000","last_unstale":"2024-09-21T10:02:25.535689+0000","last_undegraded":"2024-09-21T10:02:25.535689+0000","last_fullsized":"2024-09-21T10:01:17.351626+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:21.390162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":197,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.451051+0000","last_change":"2024-09-21T10:01:55.374579+0000","last_active":"2024-09-21T10:02:13.451051+0000","last_peered":"2024-09-21T10:02:13.451051+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.374579+0000","last_became_peered":"2024-09-21T10:01:55.374579+0000","last_unstale":"2024-09-21T10:02:13.451051+0000","last_undegraded":"2024-09-21T10:02:13.451051+0000","last_fullsized":"2024-09-21T10:01:55.359516+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":107340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"161'7363","reported_seq":8307,"reported_epoch":161,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:25.662223+0000","last_change":"2024-09-21T10:01:35.790770+0000","last_active":"2024-09-21T10:02:25.662223+0000","last_peered":"2024-09-21T10:02:25.662223+0000","last_clean":"2024-09-21T10:02:25.662223+0000","last_became_active":"2024-09-21T10:01:18.395798+0000","last_became_peered":"2024-09-21T10:01:18.395798+0000","last_unstale":"2024-09-21T10:02:25.662223+0000","last_undegraded":"2024-09-21T10:02:25.662223+0000","last_fullsized":"2024-09-21T10:02:25.662223+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7363,"log_dups_size":0,"ondisk_log_size":7363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:00:51.556626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434831360,"num_objects":6635,"num_object_clones":0,"num_object_copies":19905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6635,"num_objects_unfound":0,"num_objects_dirty":6635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7363,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":442,"num_bytes_recovered":28966912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,0],"avail_no_missing":["3","0","4"],"object_location_counts":[{"shards":"0,3,4","objects":6635}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":199,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.454250+0000","last_change":"2024-09-21T09:50:45.510812+0000","last_active":"2024-09-21T10:02:13.454250+0000","last_peered":"2024-09-21T10:02:13.454250+0000","last_clean":"2024-09-21T10:02:13.454250+0000","last_became_active":"2024-09-21T09:50:45.510480+0000","last_became_peered":"2024-09-21T09:50:45.510480+0000","last_unstale":"2024-09-21T10:02:13.454250+0000","last_undegraded":"2024-09-21T10:02:13.454250+0000","last_fullsized":"2024-09-21T10:02:13.454250+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:05.408194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":164,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.469549+0000","last_change":"2024-09-21T10:01:17.357896+0000","last_active":"2024-09-21T10:02:13.469549+0000","last_peered":"2024-09-21T10:02:13.469549+0000","last_clean":"2024-09-21T10:01:12.345631+0000","last_became_active":"2024-09-21T10:01:17.357896+0000","last_became_peered":"2024-09-21T10:01:17.357896+0000","last_unstale":"2024-09-21T10:02:13.469549+0000","last_undegraded":"2024-09-21T10:02:13.469549+0000","last_fullsized":"2024-09-21T10:01:17.354502+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:45.683879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":107,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.453303+0000","last_change":"2024-09-21T09:40:49.190763+0000","last_active":"2024-09-21T10:02:13.453303+0000","last_peered":"2024-09-21T10:02:13.453303+0000","last_clean":"2024-09-21T10:02:13.453303+0000","last_became_active":"2024-09-21T09:40:49.190463+0000","last_became_peered":"2024-09-21T09:40:49.190463+0000","last_unstale":"2024-09-21T10:02:13.453303+0000","last_undegraded":"2024-09-21T10:02:13.453303+0000","last_fullsized":"2024-09-21T10:02:13.453303+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:15.669404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":134,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.453068+0000","last_change":"2024-09-21T09:51:30.642074+0000","last_active":"2024-09-21T10:02:13.453068+0000","last_peered":"2024-09-21T10:02:13.453068+0000","last_clean":"2024-09-21T10:02:13.453068+0000","last_became_active":"2024-09-21T09:51:30.641176+0000","last_became_peered":"2024-09-21T09:51:30.641176+0000","last_unstale":"2024-09-21T10:02:13.453068+0000","last_undegraded":"2024-09-21T10:02:13.453068+0000","last_fullsized":"2024-09-21T10:02:13.453068+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:18.766865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":178,"reported_epoch":160,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:02:13.454962+0000","last_change":"2024-09-21T10:01:17.359126+0000","last_active":"2024-09-21T10:02:13.454962+0000","last_peered":"2024-09-21T10:02:13.454962+0000","last_clean":"2024-09-21T10:01:12.333760+0000","last_became_active":"2024-09-21T10:01:17.359126+0000","last_became_peered":"2024-09-21T10:01:17.359126+0000","last_unstale":"2024-09-21T10:02:13.454962+0000","last_undegraded":"2024-09-21T10:01:17.353723+0000","last_fullsized":"2024-09-21T10:01:17.353628+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:19.570827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":166,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.457324+0000","last_change":"2024-09-21T09:51:30.642334+0000","last_active":"2024-09-21T10:02:13.457324+0000","last_peered":"2024-09-21T10:02:13.457324+0000","last_clean":"2024-09-21T10:02:13.457324+0000","last_became_active":"2024-09-21T09:51:30.641267+0000","last_became_peered":"2024-09-21T09:51:30.641267+0000","last_unstale":"2024-09-21T10:02:13.457324+0000","last_undegraded":"2024-09-21T10:02:13.457324+0000","last_fullsized":"2024-09-21T10:02:13.457324+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:24.201971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":58,"reported_epoch":161,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:02:25.530199+0000","last_change":"2024-09-21T10:01:17.357199+0000","last_active":"2024-09-21T10:02:25.530199+0000","last_peered":"2024-09-21T10:02:25.530199+0000","last_clean":"2024-09-21T10:00:52.399787+0000","last_became_active":"2024-09-21T10:01:17.357199+0000","last_became_peered":"2024-09-21T10:01:17.357199+0000","last_unstale":"2024-09-21T10:02:25.530199+0000","last_undegraded":"2024-09-21T10:01:17.352030+0000","last_fullsized":"2024-09-21T10:01:17.351935+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:34.172159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":153,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.446348+0000","last_change":"2024-09-21T10:01:17.372819+0000","last_active":"2024-09-21T10:02:13.446348+0000","last_peered":"2024-09-21T10:02:13.446348+0000","last_clean":"2024-09-21T10:01:12.334093+0000","last_became_active":"2024-09-21T10:01:17.372819+0000","last_became_peered":"2024-09-21T10:01:17.372819+0000","last_unstale":"2024-09-21T10:02:13.446348+0000","last_undegraded":"2024-09-21T10:02:13.446348+0000","last_fullsized":"2024-09-21T10:01:17.355655+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:03:49.178062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":154,"reported_epoch":161,"state":"active+undersized","last_fresh":"2024-09-21T10:02:25.538680+0000","last_change":"2024-09-21T10:01:17.368801+0000","last_active":"2024-09-21T10:02:25.538680+0000","last_peered":"2024-09-21T10:02:25.538680+0000","last_clean":"2024-09-21T10:01:12.340026+0000","last_became_active":"2024-09-21T10:01:17.368801+0000","last_became_peered":"2024-09-21T10:01:17.368801+0000","last_unstale":"2024-09-21T10:02:25.538680+0000","last_undegraded":"2024-09-21T10:02:25.538680+0000","last_fullsized":"2024-09-21T10:01:17.353280+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:23.625415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":194,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.451370+0000","last_change":"2024-09-21T09:56:12.574472+0000","last_active":"2024-09-21T10:02:13.451370+0000","last_peered":"2024-09-21T10:02:13.451370+0000","last_clean":"2024-09-21T10:02:13.451370+0000","last_became_active":"2024-09-21T09:56:12.573966+0000","last_became_peered":"2024-09-21T09:56:12.573966+0000","last_unstale":"2024-09-21T10:02:13.451370+0000","last_undegraded":"2024-09-21T10:02:13.451370+0000","last_fullsized":"2024-09-21T10:02:13.451370+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:37:21.415933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":146,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-21T10:02:25.535599+0000","last_change":"2024-09-21T09:56:12.576756+0000","last_active":"2024-09-21T10:02:25.535599+0000","last_peered":"2024-09-21T10:02:25.535599+0000","last_clean":"2024-09-21T10:02:25.535599+0000","last_became_active":"2024-09-21T09:56:12.575752+0000","last_became_peered":"2024-09-21T09:56:12.575752+0000","last_unstale":"2024-09-21T10:02:25.535599+0000","last_undegraded":"2024-09-21T10:02:25.535599+0000","last_fullsized":"2024-09-21T10:02:25.535599+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:55.224839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":124,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.469504+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T10:02:13.469504+0000","last_peered":"2024-09-21T10:02:13.469504+0000","last_clean":"2024-09-21T10:02:13.469504+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T10:02:13.469504+0000","last_undegraded":"2024-09-21T10:02:13.469504+0000","last_fullsized":"2024-09-21T10:02:13.469504+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":164,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-21T10:02:25.536481+0000","last_change":"2024-09-21T09:56:12.571495+0000","last_active":"2024-09-21T10:02:25.536481+0000","last_peered":"2024-09-21T10:02:25.536481+0000","last_clean":"2024-09-21T10:02:25.536481+0000","last_became_active":"2024-09-21T09:56:12.571200+0000","last_became_peered":"2024-09-21T09:56:12.571200+0000","last_unstale":"2024-09-21T10:02:25.536481+0000","last_undegraded":"2024-09-21T10:02:25.536481+0000","last_fullsized":"2024-09-21T10:02:25.536481+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:53:58.851300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":214,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.451428+0000","last_change":"2024-09-21T10:01:17.363986+0000","last_active":"2024-09-21T10:02:13.451428+0000","last_peered":"2024-09-21T10:02:13.451428+0000","last_clean":"2024-09-21T10:01:12.350477+0000","last_became_active":"2024-09-21T10:01:17.363986+0000","last_became_peered":"2024-09-21T10:01:17.363986+0000","last_unstale":"2024-09-21T10:02:13.451428+0000","last_undegraded":"2024-09-21T10:02:13.451428+0000","last_fullsized":"2024-09-21T10:01:17.354948+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:29:32.179855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":164,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.455179+0000","last_change":"2024-09-21T09:56:12.580276+0000","last_active":"2024-09-21T10:02:13.455179+0000","last_peered":"2024-09-21T10:02:13.455179+0000","last_clean":"2024-09-21T10:02:13.455179+0000","last_became_active":"2024-09-21T09:56:12.579989+0000","last_became_peered":"2024-09-21T09:56:12.579989+0000","last_unstale":"2024-09-21T10:02:13.455179+0000","last_undegraded":"2024-09-21T10:02:13.455179+0000","last_fullsized":"2024-09-21T10:02:13.455179+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:14:56.361873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":146,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-21T10:02:25.530068+0000","last_change":"2024-09-21T09:56:12.582383+0000","last_active":"2024-09-21T10:02:25.530068+0000","last_peered":"2024-09-21T10:02:25.530068+0000","last_clean":"2024-09-21T10:02:25.530068+0000","last_became_active":"2024-09-21T09:56:12.581427+0000","last_became_peered":"2024-09-21T09:56:12.581427+0000","last_unstale":"2024-09-21T10:02:25.530068+0000","last_undegraded":"2024-09-21T10:02:25.530068+0000","last_fullsized":"2024-09-21T10:02:25.530068+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:24.275633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":143,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.452889+0000","last_change":"2024-09-21T09:51:30.645644+0000","last_active":"2024-09-21T10:02:13.452889+0000","last_peered":"2024-09-21T10:02:13.452889+0000","last_clean":"2024-09-21T10:02:13.452889+0000","last_became_active":"2024-09-21T09:51:30.644883+0000","last_became_peered":"2024-09-21T09:51:30.644883+0000","last_unstale":"2024-09-21T10:02:13.452889+0000","last_undegraded":"2024-09-21T10:02:13.452889+0000","last_fullsized":"2024-09-21T10:02:13.452889+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:39.331936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":198,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.460672+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:02:13.460672+0000","last_peered":"2024-09-21T10:02:13.460672+0000","last_clean":"2024-09-21T10:02:13.460672+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:02:13.460672+0000","last_undegraded":"2024-09-21T10:02:13.460672+0000","last_fullsized":"2024-09-21T10:02:13.460672+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":205,"reported_epoch":160,"state":"active+undersized","last_fresh":"2024-09-21T10:02:13.453496+0000","last_change":"2024-09-21T10:01:17.367707+0000","last_active":"2024-09-21T10:02:13.453496+0000","last_peered":"2024-09-21T10:02:13.453496+0000","last_clean":"2024-09-21T10:01:12.346791+0000","last_became_active":"2024-09-21T10:01:17.367707+0000","last_became_peered":"2024-09-21T10:01:17.367707+0000","last_unstale":"2024-09-21T10:02:13.453496+0000","last_undegraded":"2024-09-21T10:02:13.453496+0000","last_fullsized":"2024-09-21T10:01:17.354751+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:58:00.301582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":141,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.452951+0000","last_change":"2024-09-21T09:56:12.576148+0000","last_active":"2024-09-21T10:02:13.452951+0000","last_peered":"2024-09-21T10:02:13.452951+0000","last_clean":"2024-09-21T10:02:13.452951+0000","last_became_active":"2024-09-21T09:56:12.575813+0000","last_became_peered":"2024-09-21T09:56:12.575813+0000","last_unstale":"2024-09-21T10:02:13.452951+0000","last_undegraded":"2024-09-21T10:02:13.452951+0000","last_fullsized":"2024-09-21T10:02:13.452951+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:53:01.367816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":128,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.443173+0000","last_change":"2024-09-21T09:50:45.502447+0000","last_active":"2024-09-21T10:02:13.443173+0000","last_peered":"2024-09-21T10:02:13.443173+0000","last_clean":"2024-09-21T10:02:13.443173+0000","last_became_active":"2024-09-21T09:50:45.500800+0000","last_became_peered":"2024-09-21T09:50:45.500800+0000","last_unstale":"2024-09-21T10:02:13.443173+0000","last_undegraded":"2024-09-21T10:02:13.443173+0000","last_fullsized":"2024-09-21T10:02:13.443173+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:20:51.819410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":141,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.452837+0000","last_change":"2024-09-21T09:56:12.576457+0000","last_active":"2024-09-21T10:02:13.452837+0000","last_peered":"2024-09-21T10:02:13.452837+0000","last_clean":"2024-09-21T10:02:13.452837+0000","last_became_active":"2024-09-21T09:56:12.576061+0000","last_became_peered":"2024-09-21T09:56:12.576061+0000","last_unstale":"2024-09-21T10:02:13.452837+0000","last_undegraded":"2024-09-21T10:02:13.452837+0000","last_fullsized":"2024-09-21T10:02:13.452837+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:51:46.676699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":111,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.443482+0000","last_change":"2024-09-21T09:40:49.094916+0000","last_active":"2024-09-21T10:02:13.443482+0000","last_peered":"2024-09-21T10:02:13.443482+0000","last_clean":"2024-09-21T10:02:13.443482+0000","last_became_active":"2024-09-21T09:40:49.094264+0000","last_became_peered":"2024-09-21T09:40:49.094264+0000","last_unstale":"2024-09-21T10:02:13.443482+0000","last_undegraded":"2024-09-21T10:02:13.443482+0000","last_fullsized":"2024-09-21T10:02:13.443482+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:53.663226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":172,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-21T10:02:25.539063+0000","last_change":"2024-09-21T09:56:12.577329+0000","last_active":"2024-09-21T10:02:25.539063+0000","last_peered":"2024-09-21T10:02:25.539063+0000","last_clean":"2024-09-21T10:02:25.539063+0000","last_became_active":"2024-09-21T09:56:12.576941+0000","last_became_peered":"2024-09-21T09:56:12.576941+0000","last_unstale":"2024-09-21T10:02:25.539063+0000","last_undegraded":"2024-09-21T10:02:25.539063+0000","last_fullsized":"2024-09-21T10:02:25.539063+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:35.235768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":158,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-21T10:02:25.540239+0000","last_change":"2024-09-21T09:56:12.577137+0000","last_active":"2024-09-21T10:02:25.540239+0000","last_peered":"2024-09-21T10:02:25.540239+0000","last_clean":"2024-09-21T10:02:25.540239+0000","last_became_active":"2024-09-21T09:56:12.576668+0000","last_became_peered":"2024-09-21T09:56:12.576668+0000","last_unstale":"2024-09-21T10:02:25.540239+0000","last_undegraded":"2024-09-21T10:02:25.540239+0000","last_fullsized":"2024-09-21T10:02:25.540239+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:56.772427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":124,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.452704+0000","last_change":"2024-09-21T09:50:45.502819+0000","last_active":"2024-09-21T10:02:13.452704+0000","last_peered":"2024-09-21T10:02:13.452704+0000","last_clean":"2024-09-21T10:02:13.452704+0000","last_became_active":"2024-09-21T09:50:45.502483+0000","last_became_peered":"2024-09-21T09:50:45.502483+0000","last_unstale":"2024-09-21T10:02:13.452704+0000","last_undegraded":"2024-09-21T10:02:13.452704+0000","last_fullsized":"2024-09-21T10:02:13.452704+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:26.549958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":199,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-21T10:02:13.451123+0000","last_change":"2024-09-21T09:51:30.650116+0000","last_active":"2024-09-21T10:02:13.451123+0000","last_peered":"2024-09-21T10:02:13.451123+0000","last_clean":"2024-09-21T10:02:13.451123+0000","last_became_active":"2024-09-21T09:51:30.649798+0000","last_became_peered":"2024-09-21T09:51:30.649798+0000","last_unstale":"2024-09-21T10:02:13.451123+0000","last_undegraded":"2024-09-21T10:02:13.451123+0000","last_fullsized":"2024-09-21T10:02:13.451123+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:40.222590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6856310808,"num_objects":104620,"num_object_clones":0,"num_object_copies":313860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":58709,"num_objects_unfound":0,"num_objects_dirty":104620,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115250,"num_write_kb":7035777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3699,"num_bytes_recovered":242417664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20628123648,"data_stored":20628111432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115150,"ondisk_log_size":115150,"up":39,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":85,"acting":85,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":9704,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13748,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":83,"acting":83,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":19,"acting":19,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":22,"acting":22,"num_store_stats":4}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626721,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1563884,"kb_used_data":1262368,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92181268,"statfs":{"total":95995035648,"available":94393618432,"internally_reserved":0,"allocated":1292664832,"data_stored":1291841755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019179,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2978732,"kb_used_data":2091940,"kb_used_omap":1,"kb_used_meta":886782,"kb_avail":90766420,"statfs":{"total":95995035648,"available":92944814080,"internally_reserved":0,"allocated":2142146560,"data_stored":2141318924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":908065112},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":81,"seq":347892351140,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3041228,"kb_used_data":2560196,"kb_used_omap":5,"kb_used_meta":480954,"kb_avail":90703924,"statfs":{"total":95995035648,"available":92880818176,"internally_reserved":0,"allocated":2621640704,"data_stored":2620833549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":492496956},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430478,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":673476,"kb_used_data":75260,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93071676,"statfs":{"total":95995035648,"available":95305396224,"internally_reserved":0,"allocated":77066240,"data_stored":76315786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822950,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5031768,"kb_used_data":4122512,"kb_used_omap":1,"kb_used_meta":909246,"kb_avail":88713384,"statfs":{"total":95995035648,"available":90842505216,"internally_reserved":0,"allocated":4221452288,"data_stored":4220640279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931068248},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879373,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4547524,"kb_used_data":4197948,"kb_used_omap":4,"kb_used_meta":349563,"kb_avail":89197628,"statfs":{"total":95995035648,"available":91338371072,"internally_reserved":0,"allocated":4298698752,"data_stored":4297898727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":357952956},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566213,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2092552,"kb_used_data":1716800,"kb_used_omap":6,"kb_used_meta":375737,"kb_avail":91652600,"statfs":{"total":95995035648,"available":93852262400,"internally_reserved":0,"allocated":1758003200,"data_stored":1757199601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":384755111},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860558,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4667012,"kb_used_data":4222012,"kb_used_omap":5,"kb_used_meta":444986,"kb_avail":89078140,"statfs":{"total":95995035648,"available":91216015360,"internally_reserved":0,"allocated":4323340288,"data_stored":4322539901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":455665903},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2600927232,"data_stored":2600927232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4318892032,"data_stored":4318887960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1746862080,"data_stored":1746862080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4267442176,"data_stored":4267442176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2131103744,"data_stored":2131099672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4205645824,"data_stored":4205641752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1279918080,"data_stored":1279918080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":77332480,"data_stored":77332480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:02:30.747 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T10:02:30.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_4 pg_num 26 2024-09-21T10:02:30.750 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:30.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:30.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:30.854 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:31.021 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:30 smithi129 ceph-mon[88471]: from='client.36048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:02:31.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:31.050 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:31.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:31.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:31.204 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:31.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:31.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:31.284 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:31.298 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:31.306 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:31.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:31.359 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:31.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:31.460 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:31.508 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:31.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:31.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:31.609 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:31.634 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:31.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:31.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:31.692 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:31 smithi107 ceph-mon[102221]: pgmap v570: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 10 MiB/s wr, 842 op/s; 15/309375 objects degraded (0.005%); 57952/309375 objects misplaced (18.732%) 2024-09-21T10:02:31.692 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:31 smithi107 ceph-mon[105022]: pgmap v570: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 10 MiB/s wr, 842 op/s; 15/309375 objects degraded (0.005%); 57952/309375 objects misplaced (18.732%) 2024-09-21T10:02:31.736 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:31.761 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:31.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:31.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:31.822 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:02:31.863 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:31.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:31.917 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:31.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:31.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:31 smithi129 ceph-mon[88471]: pgmap v570: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 10 MiB/s wr, 842 op/s; 15/309375 objects degraded (0.005%); 57952/309375 objects misplaced (18.732%) 2024-09-21T10:02:31.953 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:02:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:02:31] "GET /metrics HTTP/1.1" 200 36665 "" "Prometheus/2.51.0" 2024-09-21T10:02:31.987 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:02:31.987 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:02:31.987 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-09-21T10:02:31.987 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-09-21T10:02:31.987 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:02:32.018 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:32.042 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:32.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:32.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:32.143 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:32.167 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:32.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:32.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:32.269 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:32.312 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:32.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:32.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:32.414 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:32.437 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:32.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:32.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:32.539 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:32.580 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:32.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:32.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:32.682 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:32.707 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:32.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:32.809 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:32.834 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:32.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:32.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:32.936 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:32.975 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:32.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:32.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:33.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:33.118 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:33.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:33.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:33.220 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:33.246 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:33.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:33.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:33.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:33.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:33.387 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:33.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:33.488 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:33.515 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:33.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:33.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:33.539 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:02:33.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:33.658 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:33.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:33.761 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:33.784 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:33.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:33.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:33.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:33 smithi107 ceph-mon[102221]: pgmap v571: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 3.6 MiB/s wr, 837 op/s; 15/305373 objects degraded (0.005%); 57180/305373 objects misplaced (18.725%) 2024-09-21T10:02:33.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:33 smithi107 ceph-mon[105022]: pgmap v571: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 3.6 MiB/s wr, 837 op/s; 15/305373 objects degraded (0.005%); 57180/305373 objects misplaced (18.725%) 2024-09-21T10:02:33.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:33 smithi129 ceph-mon[88471]: pgmap v571: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 3.6 MiB/s wr, 837 op/s; 15/305373 objects degraded (0.005%); 57180/305373 objects misplaced (18.725%) 2024-09-21T10:02:33.934 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:33.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:33.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:34.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:34.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:34.089 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:34.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:34.190 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:34.229 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:34.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:34.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:34.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:34.367 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:34.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:34.469 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:34.493 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:34.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:34.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:34.595 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:34.619 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:34.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:34.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:34.722 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:34.750 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:34.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:34.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:34.803 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:34 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/305373 objects degraded (0.005%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:34.804 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:34 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/305373 objects degraded (0.005%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:34.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:34.867 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:34.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:34.911 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:34.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:34 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/305373 objects degraded (0.005%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:35.013 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:35.056 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:35.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:35.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:35.158 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:35.181 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:35.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:35.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:35.284 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:35.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:35.328 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:35.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:35.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:35.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:35.470 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:35.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:35.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:35.614 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:35.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:35.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:35.716 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:35.739 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:35.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:35.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:35.811 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:35 smithi107 ceph-mon[102221]: pgmap v572: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 537 KiB/s wr, 833 op/s; 15/301419 objects degraded (0.005%); 56386/301419 objects misplaced (18.707%) 2024-09-21T10:02:35.812 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:35 smithi107 ceph-mon[105022]: pgmap v572: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 537 KiB/s wr, 833 op/s; 15/301419 objects degraded (0.005%); 56386/301419 objects misplaced (18.707%) 2024-09-21T10:02:35.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:35.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:35.890 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:35.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:35.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:35 smithi129 ceph-mon[88471]: pgmap v572: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 537 KiB/s wr, 833 op/s; 15/301419 objects degraded (0.005%); 56386/301419 objects misplaced (18.707%) 2024-09-21T10:02:35.991 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:36.029 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:36.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:36.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:36.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:36.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:36.183 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:36.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:36.246 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:36.285 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:36.316 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:36.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:36.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:36.418 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:36.457 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:36.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:36.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:36.559 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:36.582 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:36.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:36.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:36.615 INFO:teuthology.orchestra.run.smithi107.stderr:set pool 9 pg_num to 26 2024-09-21T10:02:36.615 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:02:36.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:36.730 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:36.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:36.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:36.832 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:36.858 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:36.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:36.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:36 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2069925882' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:02:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:36 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:02:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:36 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3015072984' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:02:36.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:36 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:02:36.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:36 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2069925882' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:02:36.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:36 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:02:36.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:36 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3015072984' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:02:36.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:36 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:02:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:36 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2069925882' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:02:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:36 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:02:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:36 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3015072984' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:02:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:36 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:02:36.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:36.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:36.997 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:36.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:37.098 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:37.135 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:37.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:37.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:37.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:37.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:37.280 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:37.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:37.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:37.420 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:37.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:37.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:37.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:37.556 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:37.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:37.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:37.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:37.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:37.695 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:37.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:37.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:37.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:37 smithi107 ceph-mon[102221]: pgmap v573: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 931 op/s; 15/292638 objects degraded (0.005%); 54829/292638 objects misplaced (18.736%) 2024-09-21T10:02:37.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:37 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:02:37.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:37 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:02:37.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:37 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:02:37.829 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:37 smithi107 ceph-mon[102221]: osdmap e162: 8 total, 7 up, 8 in 2024-09-21T10:02:37.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:37 smithi107 ceph-mon[105022]: pgmap v573: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 931 op/s; 15/292638 objects degraded (0.005%); 54829/292638 objects misplaced (18.736%) 2024-09-21T10:02:37.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:37 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:02:37.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:37 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:02:37.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:37 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:02:37.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:37 smithi107 ceph-mon[105022]: osdmap e162: 8 total, 7 up, 8 in 2024-09-21T10:02:37.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:37.836 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:37.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:37 smithi129 ceph-mon[88471]: pgmap v573: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 931 op/s; 15/292638 objects degraded (0.005%); 54829/292638 objects misplaced (18.736%) 2024-09-21T10:02:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:37 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:02:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:37 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:02:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:37 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:02:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:37 smithi129 ceph-mon[88471]: osdmap e162: 8 total, 7 up, 8 in 2024-09-21T10:02:37.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:37.986 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:37.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:37.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:38.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:38.133 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:38.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:38.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:38.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:38.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:38.276 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:38.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:38.377 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:38.405 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:38.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:38.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:38.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:38.539 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:38.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:38.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:38.640 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:38.681 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:38.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:38.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:38.783 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:38.840 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:38.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:38.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:38.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:02:38.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:02:38.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:02:38.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:38.975 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:38.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:38.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:39.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:39.097 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:39.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:39.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:39.198 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:39.220 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:39.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:39.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:39.322 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:39.343 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:39.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:39.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:39.445 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:39.469 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:39.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:39.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:39.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:39.591 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:39.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:39.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:39.693 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:39.718 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:39.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:39.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:39.819 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:39.842 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:39.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:39.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:39 smithi107 ceph-mon[102221]: pgmap v575: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 837 op/s; 15/288795 objects degraded (0.005%); 54028/288795 objects misplaced (18.708%) 2024-09-21T10:02:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:39 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:02:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:39 smithi107 ceph-mon[102221]: osdmap e163: 8 total, 7 up, 8 in 2024-09-21T10:02:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:39 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/288795 objects degraded (0.005%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:39.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:39 smithi107 ceph-mon[105022]: pgmap v575: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 837 op/s; 15/288795 objects degraded (0.005%); 54028/288795 objects misplaced (18.708%) 2024-09-21T10:02:39.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:39 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:02:39.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:39 smithi107 ceph-mon[105022]: osdmap e163: 8 total, 7 up, 8 in 2024-09-21T10:02:39.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:39 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/288795 objects degraded (0.005%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:39.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:39 smithi129 ceph-mon[88471]: pgmap v575: 99 pgs: 9 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 50 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 837 op/s; 15/288795 objects degraded (0.005%); 54028/288795 objects misplaced (18.708%) 2024-09-21T10:02:39.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:39 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:02:39.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:39 smithi129 ceph-mon[88471]: osdmap e163: 8 total, 7 up, 8 in 2024-09-21T10:02:39.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:39 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/288795 objects degraded (0.005%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:39.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:39.966 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:39.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:39.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:40.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:40.088 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:40.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:40.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:40.190 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:40.216 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:40.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:40.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:40.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:40.340 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:40.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:40.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:40.441 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:40.463 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:40.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:40.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:40.564 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:40.585 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:40.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:40.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:40.629 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:02:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:02:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:40.687 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:40.709 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:40.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:40.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:40.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:40.832 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:40.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:40.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:40 smithi129 ceph-mon[88471]: osdmap e164: 8 total, 7 up, 8 in 2024-09-21T10:02:40.933 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:40.941 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:40 smithi107 ceph-mon[102221]: osdmap e164: 8 total, 7 up, 8 in 2024-09-21T10:02:40.941 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:40 smithi107 ceph-mon[105022]: osdmap e164: 8 total, 7 up, 8 in 2024-09-21T10:02:40.955 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:40.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:40.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:41.039 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:02:41.039 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-09-21T10:02:41.039 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-09-21T10:02:41.039 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:02:41.039 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.039 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:91039 8.0 8.55b9fd20 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T10:02:41.039 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:11.572408+0000", 2024-09-21T10:02:41.040 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 389.46347070000002, 2024-09-21T10:02:41.040 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.99420088299999998, 2024-09-21T10:02:41.040 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:02:41.040 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:02:41.040 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:02:41.040 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:02:41.040 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:02:41.040 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:02:41.040 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91039 2024-09-21T10:02:41.040 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.040 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:02:41.040 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.040 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:02:41.040 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.572408+0000", 2024-09-21T10:02:41.041 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.041 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.041 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.041 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:02:41.041 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.572408+0000", 2024-09-21T10:02:41.041 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.041 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.041 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.041 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:02:41.041 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.572410+0000", 2024-09-21T10:02:41.041 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.466e-06 2024-09-21T10:02:41.041 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.041 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.041 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:02:41.041 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.572439+0000", 2024-09-21T10:02:41.042 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8966999999999999e-05 2024-09-21T10:02:41.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.042 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:02:41.042 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.572447+0000", 2024-09-21T10:02:41.042 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0569999999999993e-06 2024-09-21T10:02:41.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.042 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:02:41.042 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.572455+0000", 2024-09-21T10:02:41.042 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2470000000000008e-06 2024-09-21T10:02:41.042 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.042 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.042 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.043 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.582688+0000", 2024-09-21T10:02:41.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010232751999999999 2024-09-21T10:02:41.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.043 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:02:41.043 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.582697+0000", 2024-09-21T10:02:41.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6049999999999998e-06 2024-09-21T10:02:41.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.043 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.043 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.566597+0000", 2024-09-21T10:02:41.043 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.98390088899999995 2024-09-21T10:02:41.043 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.043 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.044 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:02:41.044 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.566609+0000", 2024-09-21T10:02:41.044 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.19e-05 2024-09-21T10:02:41.044 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.044 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.044 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.044 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.044 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.044 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:91044 8.0 8.b2b04290 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T10:02:41.044 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:11.572468+0000", 2024-09-21T10:02:41.044 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 389.46341142099999, 2024-09-21T10:02:41.044 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.99417569299999997, 2024-09-21T10:02:41.044 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:02:41.044 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:02:41.045 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:02:41.045 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:02:41.045 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:02:41.045 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:02:41.045 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91044 2024-09-21T10:02:41.045 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.045 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:02:41.045 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.045 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:02:41.045 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.572468+0000", 2024-09-21T10:02:41.045 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.045 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.045 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.045 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:02:41.046 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.572468+0000", 2024-09-21T10:02:41.046 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.046 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.046 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.046 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:02:41.046 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.572468+0000", 2024-09-21T10:02:41.046 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0999999999999999e-07 2024-09-21T10:02:41.046 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.046 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.046 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:02:41.046 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.572521+0000", 2024-09-21T10:02:41.046 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2321999999999998e-05 2024-09-21T10:02:41.046 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.046 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:02:41.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.572529+0000", 2024-09-21T10:02:41.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3180000000000007e-06 2024-09-21T10:02:41.047 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:02:41.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.572535+0000", 2024-09-21T10:02:41.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4910000000000002e-06 2024-09-21T10:02:41.047 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.047 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.047 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.047 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.582722+0000", 2024-09-21T10:02:41.047 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010186758000000001 2024-09-21T10:02:41.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.048 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:02:41.048 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.582729+0000", 2024-09-21T10:02:41.048 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1640000000000004e-06 2024-09-21T10:02:41.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.048 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.048 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.566635+0000", 2024-09-21T10:02:41.048 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.98390534100000004 2024-09-21T10:02:41.048 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.048 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.048 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:02:41.048 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.566643+0000", 2024-09-21T10:02:41.048 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.789e-06 2024-09-21T10:02:41.049 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.049 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.049 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.049 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.049 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.049 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:91045 8.f 8.735f725f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T10:02:41.049 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:11.572710+0000", 2024-09-21T10:02:41.049 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 389.46316958400001, 2024-09-21T10:02:41.049 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.99713176299999995, 2024-09-21T10:02:41.049 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:02:41.049 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:02:41.049 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:02:41.049 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:02:41.049 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:02:41.050 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:02:41.050 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91045 2024-09-21T10:02:41.050 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.050 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:02:41.050 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.050 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:02:41.050 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.572710+0000", 2024-09-21T10:02:41.050 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.050 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.050 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.050 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:02:41.050 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.572710+0000", 2024-09-21T10:02:41.050 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.050 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.050 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:02:41.051 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.572710+0000", 2024-09-21T10:02:41.051 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2500000000000005e-07 2024-09-21T10:02:41.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:02:41.051 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.572752+0000", 2024-09-21T10:02:41.051 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1486999999999998e-05 2024-09-21T10:02:41.051 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.051 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.051 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:02:41.051 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.572761+0000", 2024-09-21T10:02:41.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7110000000000007e-06 2024-09-21T10:02:41.052 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.052 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.052 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:02:41.052 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.572768+0000", 2024-09-21T10:02:41.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2829999999999998e-06 2024-09-21T10:02:41.052 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.052 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.052 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.052 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.583874+0000", 2024-09-21T10:02:41.052 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011106306 2024-09-21T10:02:41.052 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.052 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.052 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:02:41.052 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.583882+0000", 2024-09-21T10:02:41.053 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4610000000000008e-06 2024-09-21T10:02:41.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.053 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.053 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.569830+0000", 2024-09-21T10:02:41.053 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.98594726600000004 2024-09-21T10:02:41.053 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.053 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.054 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:02:41.054 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.569841+0000", 2024-09-21T10:02:41.054 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1623999999999999e-05 2024-09-21T10:02:41.054 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.054 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.054 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.055 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.055 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.055 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:91054 8.0 8.9a41f6e0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T10:02:41.055 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:11.581072+0000", 2024-09-21T10:02:41.055 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 389.45480668300002, 2024-09-21T10:02:41.055 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.98560122999999999, 2024-09-21T10:02:41.055 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:02:41.055 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:02:41.055 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:02:41.055 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:02:41.055 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:02:41.055 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:02:41.055 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91054 2024-09-21T10:02:41.055 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.056 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:02:41.056 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.056 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:02:41.056 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.581072+0000", 2024-09-21T10:02:41.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.056 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.056 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.056 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:02:41.056 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.581072+0000", 2024-09-21T10:02:41.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.056 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.056 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.056 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:02:41.056 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.581075+0000", 2024-09-21T10:02:41.056 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.029e-06 2024-09-21T10:02:41.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.057 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:02:41.057 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.581128+0000", 2024-09-21T10:02:41.057 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3715000000000003e-05 2024-09-21T10:02:41.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.057 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:02:41.057 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.581141+0000", 2024-09-21T10:02:41.057 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.307e-05 2024-09-21T10:02:41.057 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.057 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:02:41.057 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.581152+0000", 2024-09-21T10:02:41.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0859000000000001e-05 2024-09-21T10:02:41.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.058 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.058 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.058 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.582751+0000", 2024-09-21T10:02:41.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001599329 2024-09-21T10:02:41.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.058 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.058 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:02:41.058 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.582758+0000", 2024-09-21T10:02:41.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2890000000000003e-06 2024-09-21T10:02:41.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.058 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.566665+0000", 2024-09-21T10:02:41.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.98390751600000004 2024-09-21T10:02:41.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:02:41.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.566674+0000", 2024-09-21T10:02:41.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4230000000000005e-06 2024-09-21T10:02:41.059 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.059 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.059 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.059 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:91056 8.f 8.398a2e2f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T10:02:41.060 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:11.582559+0000", 2024-09-21T10:02:41.060 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 389.45332042000001, 2024-09-21T10:02:41.060 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.98732057600000001, 2024-09-21T10:02:41.060 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:02:41.060 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:02:41.060 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:02:41.060 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:02:41.060 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:02:41.060 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:02:41.060 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91056 2024-09-21T10:02:41.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.060 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:02:41.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.060 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:02:41.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.582559+0000", 2024-09-21T10:02:41.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:02:41.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.582559+0000", 2024-09-21T10:02:41.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:02:41.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.582560+0000", 2024-09-21T10:02:41.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2610000000000001e-06 2024-09-21T10:02:41.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:02:41.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.582610+0000", 2024-09-21T10:02:41.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9667999999999999e-05 2024-09-21T10:02:41.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:02:41.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.582619+0000", 2024-09-21T10:02:41.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8209999999999997e-06 2024-09-21T10:02:41.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:02:41.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.582627+0000", 2024-09-21T10:02:41.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2930000000000003e-06 2024-09-21T10:02:41.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.583904+0000", 2024-09-21T10:02:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001277165 2024-09-21T10:02:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:02:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.583910+0000", 2024-09-21T10:02:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7679999999999997e-06 2024-09-21T10:02:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.569870+0000", 2024-09-21T10:02:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.98596038900000005 2024-09-21T10:02:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:02:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.569879+0000", 2024-09-21T10:02:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2110000000000004e-06 2024-09-21T10:02:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:91064 8.f 8.4458d68f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T10:02:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:11.591459+0000", 2024-09-21T10:02:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 389.44441987300002, 2024-09-21T10:02:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.97845398800000005, 2024-09-21T10:02:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:02:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:02:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:02:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:02:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:02:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:02:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91064 2024-09-21T10:02:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:02:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:02:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.591459+0000", 2024-09-21T10:02:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:02:41.066 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.591459+0000", 2024-09-21T10:02:41.066 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.066 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.066 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.066 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:02:41.066 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.591461+0000", 2024-09-21T10:02:41.066 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.435e-06 2024-09-21T10:02:41.066 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.066 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.066 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:02:41.066 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.591521+0000", 2024-09-21T10:02:41.066 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0727000000000003e-05 2024-09-21T10:02:41.066 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.066 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.067 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:02:41.067 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.591535+0000", 2024-09-21T10:02:41.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3081999999999999e-05 2024-09-21T10:02:41.067 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.067 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.067 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:02:41.067 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.591542+0000", 2024-09-21T10:02:41.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6399999999999997e-06 2024-09-21T10:02:41.067 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.067 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.067 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.067 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.591595+0000", 2024-09-21T10:02:41.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3128999999999999e-05 2024-09-21T10:02:41.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.068 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:02:41.068 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.591687+0000", 2024-09-21T10:02:41.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1710000000000001e-05 2024-09-21T10:02:41.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.068 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.068 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.569904+0000", 2024-09-21T10:02:41.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.97821709099999998 2024-09-21T10:02:41.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.068 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:02:41.069 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.569913+0000", 2024-09-21T10:02:41.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1740000000000006e-06 2024-09-21T10:02:41.069 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.069 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.069 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.069 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.069 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.069 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:91066 8.b 8.c982c78b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T10:02:41.069 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:56:11.591676+0000", 2024-09-21T10:02:41.069 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 389.44420346499999, 2024-09-21T10:02:41.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.97876237499999996, 2024-09-21T10:02:41.069 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:02:41.069 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:02:41.069 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:02:41.070 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:02:41.070 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:02:41.070 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:02:41.070 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91066 2024-09-21T10:02:41.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.070 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:02:41.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:02:41.070 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.591676+0000", 2024-09-21T10:02:41.070 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:02:41.070 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.591676+0000", 2024-09-21T10:02:41.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:02:41.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.591677+0000", 2024-09-21T10:02:41.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4e-07 2024-09-21T10:02:41.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:02:41.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.591714+0000", 2024-09-21T10:02:41.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7351000000000002e-05 2024-09-21T10:02:41.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:02:41.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.591727+0000", 2024-09-21T10:02:41.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2838e-05 2024-09-21T10:02:41.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:02:41.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.591732+0000", 2024-09-21T10:02:41.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6899999999999997e-06 2024-09-21T10:02:41.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.591793+0000", 2024-09-21T10:02:41.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0699999999999998e-05 2024-09-21T10:02:41.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:02:41.073 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:11.591799+0000", 2024-09-21T10:02:41.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9290000000000003e-06 2024-09-21T10:02:41.073 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.073 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.073 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.570427+0000", 2024-09-21T10:02:41.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.97862839700000004 2024-09-21T10:02:41.073 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.073 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:02:41.073 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:56:12.570438+0000", 2024-09-21T10:02:41.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.063e-05 2024-09-21T10:02:41.073 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.073 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.074 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.074 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.074 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.074 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:221535 8.f 8:fab7d42d:::benchmark_data_smithi107_135256_object95651:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e135)", 2024-09-21T10:02:41.074 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:58:40.510690+0000", 2024-09-21T10:02:41.074 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 240.52518873099999, 2024-09-21T10:02:41.074 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.025393335, 2024-09-21T10:02:41.074 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:02:41.074 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:02:41.074 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:02:41.074 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:02:41.074 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:02:41.074 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:02:41.074 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 221535 2024-09-21T10:02:41.074 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.075 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:02:41.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:02:41.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510690+0000", 2024-09-21T10:02:41.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:02:41.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510690+0000", 2024-09-21T10:02:41.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:02:41.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510692+0000", 2024-09-21T10:02:41.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7010000000000001e-06 2024-09-21T10:02:41.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:02:41.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510699+0000", 2024-09-21T10:02:41.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6730000000000003e-06 2024-09-21T10:02:41.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:02:41.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510712+0000", 2024-09-21T10:02:41.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2756e-05 2024-09-21T10:02:41.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:02:41.077 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510719+0000", 2024-09-21T10:02:41.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1779999999999997e-06 2024-09-21T10:02:41.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.077 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.077 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510796+0000", 2024-09-21T10:02:41.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7311999999999994e-05 2024-09-21T10:02:41.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.077 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:02:41.077 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510804+0000", 2024-09-21T10:02:41.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6490000000000008e-06 2024-09-21T10:02:41.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.078 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.078 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.536027+0000", 2024-09-21T10:02:41.078 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0252233879999999 2024-09-21T10:02:41.078 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.078 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:02:41.078 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.536084+0000", 2024-09-21T10:02:41.078 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6678000000000002e-05 2024-09-21T10:02:41.078 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.078 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.078 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.078 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.079 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:221543 8.f 8:f4aad323:::benchmark_data_smithi107_135256_object95659:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e135)", 2024-09-21T10:02:41.079 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:58:40.510733+0000", 2024-09-21T10:02:41.079 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 240.52514582200001, 2024-09-21T10:02:41.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.02596865, 2024-09-21T10:02:41.079 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:02:41.079 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:02:41.079 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:02:41.079 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:02:41.079 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:02:41.079 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:02:41.079 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 221543 2024-09-21T10:02:41.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.079 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:02:41.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:02:41.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510733+0000", 2024-09-21T10:02:41.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:02:41.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510733+0000", 2024-09-21T10:02:41.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:02:41.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510734+0000", 2024-09-21T10:02:41.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4000000000000002e-07 2024-09-21T10:02:41.081 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.081 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.081 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:02:41.081 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510736+0000", 2024-09-21T10:02:41.081 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.821e-06 2024-09-21T10:02:41.081 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.081 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.081 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:02:41.081 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510741+0000", 2024-09-21T10:02:41.081 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7790000000000001e-06 2024-09-21T10:02:41.081 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.081 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.081 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:02:41.081 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510746+0000", 2024-09-21T10:02:41.082 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5889999999999996e-06 2024-09-21T10:02:41.082 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.082 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.082 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.082 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510830+0000", 2024-09-21T10:02:41.082 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3891e-05 2024-09-21T10:02:41.082 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.082 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.082 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:02:41.082 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510835+0000", 2024-09-21T10:02:41.082 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4569999999999996e-06 2024-09-21T10:02:41.082 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.082 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.082 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.536632+0000", 2024-09-21T10:02:41.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.025796573 2024-09-21T10:02:41.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:02:41.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.536702+0000", 2024-09-21T10:02:41.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0099999999999996e-05 2024-09-21T10:02:41.083 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.083 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.083 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.083 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:221548 8.f 8:f8b642c3:::benchmark_data_smithi107_135256_object95664:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e135)", 2024-09-21T10:02:41.083 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:58:40.510752+0000", 2024-09-21T10:02:41.084 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 240.525127121, 2024-09-21T10:02:41.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0380984449999999, 2024-09-21T10:02:41.084 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:02:41.084 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:02:41.084 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:02:41.084 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:02:41.084 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:02:41.084 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:02:41.084 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 221548 2024-09-21T10:02:41.084 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.084 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:02:41.084 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.084 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:02:41.084 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510752+0000", 2024-09-21T10:02:41.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:02:41.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510752+0000", 2024-09-21T10:02:41.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:02:41.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510752+0000", 2024-09-21T10:02:41.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8500000000000002e-07 2024-09-21T10:02:41.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.090 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:02:41.090 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510754+0000", 2024-09-21T10:02:41.090 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.393e-06 2024-09-21T10:02:41.090 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.090 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.090 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:02:41.090 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510759+0000", 2024-09-21T10:02:41.090 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9099999999999996e-06 2024-09-21T10:02:41.090 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.090 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.090 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:02:41.090 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.510763+0000", 2024-09-21T10:02:41.090 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0759999999999996e-06 2024-09-21T10:02:41.091 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.091 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.091 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.091 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.511539+0000", 2024-09-21T10:02:41.091 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00077623199999999996 2024-09-21T10:02:41.091 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.091 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.091 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:02:41.092 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.511559+0000", 2024-09-21T10:02:41.092 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0208e-05 2024-09-21T10:02:41.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.092 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.092 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.536865+0000", 2024-09-21T10:02:41.092 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0253054399999999 2024-09-21T10:02:41.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.092 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:02:41.093 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.536969+0000", 2024-09-21T10:02:41.093 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103939 2024-09-21T10:02:41.093 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.093 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.093 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:02:41.093 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.537088+0000", 2024-09-21T10:02:41.093 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011981699999999999 2024-09-21T10:02:41.093 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.093 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.093 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:02:41.093 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.548677+0000", 2024-09-21T10:02:41.093 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011588675 2024-09-21T10:02:41.093 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.093 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.094 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:02:41.094 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.548714+0000", 2024-09-21T10:02:41.094 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6822000000000001e-05 2024-09-21T10:02:41.094 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.094 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.094 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:02:41.094 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.548755+0000", 2024-09-21T10:02:41.094 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0676000000000002e-05 2024-09-21T10:02:41.094 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.094 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.094 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:02:41.094 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.548804+0000", 2024-09-21T10:02:41.094 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9400999999999997e-05 2024-09-21T10:02:41.094 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.095 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.095 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:02:41.095 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.548850+0000", 2024-09-21T10:02:41.095 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6570999999999998e-05 2024-09-21T10:02:41.095 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.095 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.095 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.095 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.095 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.095 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:221553 8.f 8:f5eeb017:::benchmark_data_smithi107_135256_object95669:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-21T10:02:41.095 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:58:40.511385+0000", 2024-09-21T10:02:41.095 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 240.524493812, 2024-09-21T10:02:41.095 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0456087119999999, 2024-09-21T10:02:41.096 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:02:41.096 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:02:41.096 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:02:41.096 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:02:41.096 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:02:41.096 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:02:41.096 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 221553 2024-09-21T10:02:41.096 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.096 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:02:41.096 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.096 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:02:41.096 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.511385+0000", 2024-09-21T10:02:41.096 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.096 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.097 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.097 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:02:41.097 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.511385+0000", 2024-09-21T10:02:41.097 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.097 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.097 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.097 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:02:41.097 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.511386+0000", 2024-09-21T10:02:41.097 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7999999999999995e-07 2024-09-21T10:02:41.097 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.097 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.098 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:02:41.098 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.511389+0000", 2024-09-21T10:02:41.098 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2270000000000002e-06 2024-09-21T10:02:41.098 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.098 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.098 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:02:41.098 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.511403+0000", 2024-09-21T10:02:41.098 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3371e-05 2024-09-21T10:02:41.098 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.098 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.098 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:02:41.098 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.511409+0000", 2024-09-21T10:02:41.098 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6440000000000003e-06 2024-09-21T10:02:41.098 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.099 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.099 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.099 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.511679+0000", 2024-09-21T10:02:41.099 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00026932600000000002 2024-09-21T10:02:41.099 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.099 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.099 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:02:41.099 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.511696+0000", 2024-09-21T10:02:41.099 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7805e-05 2024-09-21T10:02:41.099 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.099 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.099 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.100 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.540891+0000", 2024-09-21T10:02:41.100 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.029194806 2024-09-21T10:02:41.100 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.100 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.100 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:02:41.100 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.541009+0000", 2024-09-21T10:02:41.100 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000117626 2024-09-21T10:02:41.100 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.100 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.100 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:02:41.100 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.541128+0000", 2024-09-21T10:02:41.100 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011870399999999999 2024-09-21T10:02:41.100 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.100 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.101 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:02:41.101 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.548725+0000", 2024-09-21T10:02:41.101 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0075970889999999996 2024-09-21T10:02:41.101 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.101 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.101 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:02:41.101 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.556802+0000", 2024-09-21T10:02:41.101 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0080777009999999996 2024-09-21T10:02:41.101 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.101 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.101 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:02:41.101 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.556900+0000", 2024-09-21T10:02:41.101 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7510999999999995e-05 2024-09-21T10:02:41.102 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.102 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.102 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:02:41.102 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.556953+0000", 2024-09-21T10:02:41.102 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2961999999999999e-05 2024-09-21T10:02:41.102 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.102 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.102 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:02:41.102 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.556994+0000", 2024-09-21T10:02:41.102 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1260000000000001e-05 2024-09-21T10:02:41.102 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.102 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.102 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.102 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.103 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.103 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:221564 8.f 8:f344f0ed:::benchmark_data_smithi107_135256_object95680:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-21T10:02:41.103 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:58:40.528861+0000", 2024-09-21T10:02:41.103 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 240.50701833100001, 2024-09-21T10:02:41.103 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0362351059999999, 2024-09-21T10:02:41.103 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:02:41.103 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:02:41.103 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:02:41.103 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:02:41.103 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:02:41.103 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:02:41.103 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 221564 2024-09-21T10:02:41.103 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.103 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:02:41.104 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.104 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:02:41.104 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.528861+0000", 2024-09-21T10:02:41.104 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.104 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.104 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.104 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:02:41.104 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.528861+0000", 2024-09-21T10:02:41.104 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.104 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.104 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.104 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:02:41.104 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.528862+0000", 2024-09-21T10:02:41.104 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.609e-06 2024-09-21T10:02:41.104 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.105 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.105 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:02:41.105 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.528868+0000", 2024-09-21T10:02:41.105 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1059999999999999e-06 2024-09-21T10:02:41.105 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.105 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.105 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:02:41.105 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.528893+0000", 2024-09-21T10:02:41.105 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5635000000000001e-05 2024-09-21T10:02:41.105 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.105 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.105 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:02:41.105 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.528901+0000", 2024-09-21T10:02:41.105 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5809999999999999e-06 2024-09-21T10:02:41.106 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.106 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.106 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.106 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.530379+0000", 2024-09-21T10:02:41.106 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014778230000000001 2024-09-21T10:02:41.106 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.106 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.106 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:02:41.106 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.530388+0000", 2024-09-21T10:02:41.106 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8830000000000001e-06 2024-09-21T10:02:41.106 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.106 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.106 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.106 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.544879+0000", 2024-09-21T10:02:41.107 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0144904320000001 2024-09-21T10:02:41.107 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.107 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.107 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:02:41.107 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.544978+0000", 2024-09-21T10:02:41.107 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9191999999999995e-05 2024-09-21T10:02:41.107 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.107 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.107 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:02:41.107 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.545073+0000", 2024-09-21T10:02:41.107 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5271000000000003e-05 2024-09-21T10:02:41.107 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.107 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.107 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:02:41.108 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.552684+0000", 2024-09-21T10:02:41.108 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0076109890000000003 2024-09-21T10:02:41.108 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.108 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.108 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:02:41.108 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.556854+0000", 2024-09-21T10:02:41.108 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041701330000000003 2024-09-21T10:02:41.108 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.108 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.108 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:02:41.109 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.564934+0000", 2024-09-21T10:02:41.109 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0080795099999999998 2024-09-21T10:02:41.109 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.109 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.109 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:02:41.109 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.565046+0000", 2024-09-21T10:02:41.109 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011199 2024-09-21T10:02:41.109 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.109 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.109 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:02:41.109 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.565096+0000", 2024-09-21T10:02:41.109 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9951999999999999e-05 2024-09-21T10:02:41.109 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.109 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.110 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.110 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.110 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.110 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:221580 8.f 8:f1e973df:::benchmark_data_smithi107_135256_object95696:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-21T10:02:41.110 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:58:40.548464+0000", 2024-09-21T10:02:41.110 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 240.487414965, 2024-09-21T10:02:41.110 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0168804259999999, 2024-09-21T10:02:41.110 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:02:41.110 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:02:41.110 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:02:41.110 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:02:41.110 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:02:41.110 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:02:41.110 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 221580 2024-09-21T10:02:41.111 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.111 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:02:41.111 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.111 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:02:41.111 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.548464+0000", 2024-09-21T10:02:41.111 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.111 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.111 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.111 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:02:41.111 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.548464+0000", 2024-09-21T10:02:41.111 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.111 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.111 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.111 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:02:41.112 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.548466+0000", 2024-09-21T10:02:41.112 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6559999999999999e-06 2024-09-21T10:02:41.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.112 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.112 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:02:41.112 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.548474+0000", 2024-09-21T10:02:41.112 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9999999999999996e-06 2024-09-21T10:02:41.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.112 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.112 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:02:41.112 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.548601+0000", 2024-09-21T10:02:41.112 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012704599999999999 2024-09-21T10:02:41.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.112 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.112 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:02:41.113 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.548609+0000", 2024-09-21T10:02:41.113 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6990000000000003e-06 2024-09-21T10:02:41.113 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.113 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.113 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.113 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.548802+0000", 2024-09-21T10:02:41.113 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019362000000000001 2024-09-21T10:02:41.113 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.113 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.113 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:02:41.113 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.548832+0000", 2024-09-21T10:02:41.113 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9346e-05 2024-09-21T10:02:41.113 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.113 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.114 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.114 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.548878+0000", 2024-09-21T10:02:41.114 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0000468410000001 2024-09-21T10:02:41.114 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.114 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.114 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:02:41.114 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.548968+0000", 2024-09-21T10:02:41.114 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9899000000000001e-05 2024-09-21T10:02:41.114 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.114 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.114 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:02:41.114 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.549059+0000", 2024-09-21T10:02:41.114 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0642000000000004e-05 2024-09-21T10:02:41.115 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.115 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.115 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:02:41.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:41.116 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.556865+0000", 2024-09-21T10:02:41.116 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0078057769999999999 2024-09-21T10:02:41.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.116 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.116 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:02:41.116 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.565229+0000", 2024-09-21T10:02:41.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0083639209999999999 2024-09-21T10:02:41.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.117 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:02:41.117 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.565263+0000", 2024-09-21T10:02:41.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4081000000000001e-05 2024-09-21T10:02:41.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.117 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:02:41.117 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.565310+0000", 2024-09-21T10:02:41.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7178000000000001e-05 2024-09-21T10:02:41.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.117 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:02:41.118 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.565345+0000", 2024-09-21T10:02:41.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.472e-05 2024-09-21T10:02:41.118 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.118 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.118 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.118 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.118 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.118 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:221593 8.f 8:fa117e1f:::benchmark_data_smithi107_135256_object95709:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-09-21T10:02:41.118 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:58:40.561944+0000", 2024-09-21T10:02:41.118 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 240.47393518499999, 2024-09-21T10:02:41.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.007378705, 2024-09-21T10:02:41.118 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:02:41.118 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:02:41.119 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:02:41.119 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:02:41.119 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:02:41.119 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:02:41.119 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 221593 2024-09-21T10:02:41.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.119 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:02:41.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.119 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:02:41.119 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.561944+0000", 2024-09-21T10:02:41.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.120 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:02:41.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.561944+0000", 2024-09-21T10:02:41.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.120 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:02:41.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.561946+0000", 2024-09-21T10:02:41.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5880000000000001e-06 2024-09-21T10:02:41.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.120 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:02:41.121 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.561951+0000", 2024-09-21T10:02:41.121 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5539999999999997e-06 2024-09-21T10:02:41.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.121 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:02:41.121 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.561988+0000", 2024-09-21T10:02:41.121 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6939e-05 2024-09-21T10:02:41.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.121 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:02:41.121 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.561996+0000", 2024-09-21T10:02:41.121 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9959999999999993e-06 2024-09-21T10:02:41.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.122 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.122 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.562148+0000", 2024-09-21T10:02:41.122 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015162300000000001 2024-09-21T10:02:41.122 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.122 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.122 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:02:41.122 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:40.562171+0000", 2024-09-21T10:02:41.122 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3226000000000001e-05 2024-09-21T10:02:41.122 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.122 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.122 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.122 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.552745+0000", 2024-09-21T10:02:41.122 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.99057452499999998 2024-09-21T10:02:41.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.123 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.123 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:02:41.123 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.552868+0000", 2024-09-21T10:02:41.123 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012269399999999999 2024-09-21T10:02:41.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.123 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.123 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:02:41.123 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.552973+0000", 2024-09-21T10:02:41.123 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000104913 2024-09-21T10:02:41.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.123 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.123 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:02:41.123 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.560908+0000", 2024-09-21T10:02:41.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0079354390000000007 2024-09-21T10:02:41.124 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.124 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.124 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:02:41.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.565379+0000", 2024-09-21T10:02:41.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044704360000000004 2024-09-21T10:02:41.124 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.124 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.124 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:02:41.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.569220+0000", 2024-09-21T10:02:41.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038406450000000002 2024-09-21T10:02:41.124 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.124 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.124 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:02:41.125 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.569283+0000", 2024-09-21T10:02:41.125 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3497000000000005e-05 2024-09-21T10:02:41.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.125 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.125 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:02:41.125 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:58:41.569323+0000", 2024-09-21T10:02:41.125 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9629999999999998e-05 2024-09-21T10:02:41.125 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.125 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.125 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.125 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.125 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:250389 8.0 8:006938c7:::benchmark_data_smithi107_135256_object124505:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-21T10:02:41.125 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:59:13.641318+0000", 2024-09-21T10:02:41.126 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 207.39456094400001, 2024-09-21T10:02:41.126 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.05030712, 2024-09-21T10:02:41.126 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:02:41.126 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:02:41.126 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:02:41.126 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:02:41.126 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:02:41.126 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:02:41.126 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 250389 2024-09-21T10:02:41.126 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.126 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:02:41.126 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.126 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:02:41.126 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.641318+0000", 2024-09-21T10:02:41.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.127 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:02:41.127 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.641318+0000", 2024-09-21T10:02:41.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.127 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:02:41.127 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.641319+0000", 2024-09-21T10:02:41.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2419999999999999e-06 2024-09-21T10:02:41.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.127 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:02:41.127 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.641324+0000", 2024-09-21T10:02:41.128 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4569999999999998e-06 2024-09-21T10:02:41.128 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.128 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.128 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:02:41.128 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.641335+0000", 2024-09-21T10:02:41.128 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1056000000000001e-05 2024-09-21T10:02:41.128 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.128 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.128 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:02:41.128 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.641344+0000", 2024-09-21T10:02:41.128 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0729999999999994e-06 2024-09-21T10:02:41.128 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.128 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.128 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.653302+0000", 2024-09-21T10:02:41.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011958158999999999 2024-09-21T10:02:41.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:02:41.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.653311+0000", 2024-09-21T10:02:41.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0729999999999994e-06 2024-09-21T10:02:41.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:14.691551+0000", 2024-09-21T10:02:41.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0382399680000001 2024-09-21T10:02:41.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:02:41.130 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:14.691625+0000", 2024-09-21T10:02:41.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4091999999999995e-05 2024-09-21T10:02:41.130 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.130 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.130 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.130 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.130 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:250392 8.0 8:0d038bfb:::benchmark_data_smithi107_135256_object124508:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-09-21T10:02:41.130 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:59:13.641363+0000", 2024-09-21T10:02:41.130 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 207.394515826, 2024-09-21T10:02:41.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.050756604, 2024-09-21T10:02:41.130 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:02:41.130 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:02:41.131 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:02:41.131 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:02:41.131 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:02:41.131 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:02:41.131 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 250392 2024-09-21T10:02:41.131 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.131 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:02:41.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.131 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:02:41.131 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.641363+0000", 2024-09-21T10:02:41.131 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.131 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.131 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:02:41.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.641363+0000", 2024-09-21T10:02:41.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:02:41.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.641364+0000", 2024-09-21T10:02:41.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6400000000000003e-07 2024-09-21T10:02:41.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:02:41.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.641366+0000", 2024-09-21T10:02:41.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.933e-06 2024-09-21T10:02:41.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:02:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.641372+0000", 2024-09-21T10:02:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.477e-06 2024-09-21T10:02:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:02:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.641377+0000", 2024-09-21T10:02:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2399999999999998e-06 2024-09-21T10:02:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.653335+0000", 2024-09-21T10:02:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011957855 2024-09-21T10:02:41.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:02:41.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.653343+0000", 2024-09-21T10:02:41.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4660000000000002e-06 2024-09-21T10:02:41.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:14.692045+0000", 2024-09-21T10:02:41.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0387022589999999 2024-09-21T10:02:41.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:02:41.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:14.692120+0000", 2024-09-21T10:02:41.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4909999999999999e-05 2024-09-21T10:02:41.135 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.135 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.135 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.135 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.135 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45320.0:250432 8.0 8:0edec2fa:::benchmark_data_smithi107_135256_object124548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-21T10:02:41.135 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T09:59:13.678329+0000", 2024-09-21T10:02:41.135 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 207.35755065699999, 2024-09-21T10:02:41.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.024690925, 2024-09-21T10:02:41.135 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:02:41.135 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:02:41.135 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:02:41.136 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:02:41.136 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45320", 2024-09-21T10:02:41.136 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:02:41.136 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 250432 2024-09-21T10:02:41.136 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.136 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:02:41.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.136 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:02:41.136 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.678329+0000", 2024-09-21T10:02:41.136 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.136 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.136 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:02:41.136 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.678329+0000", 2024-09-21T10:02:41.136 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:02:41.137 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.678330+0000", 2024-09-21T10:02:41.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.269e-06 2024-09-21T10:02:41.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:02:41.137 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.678335+0000", 2024-09-21T10:02:41.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8339999999999998e-06 2024-09-21T10:02:41.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:02:41.137 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.678345+0000", 2024-09-21T10:02:41.138 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0777e-05 2024-09-21T10:02:41.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.138 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.138 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:02:41.138 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.678353+0000", 2024-09-21T10:02:41.138 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2219999999999996e-06 2024-09-21T10:02:41.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.138 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.138 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.138 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.682318+0000", 2024-09-21T10:02:41.138 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0039649389999999998 2024-09-21T10:02:41.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.139 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:02:41.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:13.682330+0000", 2024-09-21T10:02:41.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2947e-05 2024-09-21T10:02:41.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.139 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:14.692159+0000", 2024-09-21T10:02:41.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.009828347 2024-09-21T10:02:41.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.139 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:02:41.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:14.692245+0000", 2024-09-21T10:02:41.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6384000000000005e-05 2024-09-21T10:02:41.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.140 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:02:41.140 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:14.692346+0000", 2024-09-21T10:02:41.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000100961 2024-09-21T10:02:41.140 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.140 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:02:41.140 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:14.702798+0000", 2024-09-21T10:02:41.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01045216 2024-09-21T10:02:41.140 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.140 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:02:41.140 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:14.702863+0000", 2024-09-21T10:02:41.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4494999999999998e-05 2024-09-21T10:02:41.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:02:41.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:14.702925+0000", 2024-09-21T10:02:41.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1713999999999999e-05 2024-09-21T10:02:41.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:02:41.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:14.702981+0000", 2024-09-21T10:02:41.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6814999999999997e-05 2024-09-21T10:02:41.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:02:41.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T09:59:14.703019+0000", 2024-09-21T10:02:41.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8061e-05 2024-09-21T10:02:41.142 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.142 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.142 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.142 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45761.0:126096 9.2 9:44494c7e:::benchmark_data_smithi107_152010_object16159:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163)", 2024-09-21T10:02:41.142 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:02:38.645018+0000", 2024-09-21T10:02:41.142 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.3908615100000001, 2024-09-21T10:02:41.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0044847509999999, 2024-09-21T10:02:41.142 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:02:41.143 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:02:41.143 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:02:41.143 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:02:41.143 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45761", 2024-09-21T10:02:41.143 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:02:41.143 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 126096 2024-09-21T10:02:41.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.143 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:02:41.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.143 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:02:41.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:38.645018+0000", 2024-09-21T10:02:41.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:02:41.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:38.645018+0000", 2024-09-21T10:02:41.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:02:41.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:38.645020+0000", 2024-09-21T10:02:41.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.249e-06 2024-09-21T10:02:41.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:02:41.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:38.645029+0000", 2024-09-21T10:02:41.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4700000000000008e-06 2024-09-21T10:02:41.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:02:41.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:38.645046+0000", 2024-09-21T10:02:41.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6977e-05 2024-09-21T10:02:41.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:02:41.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:38.645058+0000", 2024-09-21T10:02:41.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1584e-05 2024-09-21T10:02:41.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:39.025300+0000", 2024-09-21T10:02:41.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.38024224299999998 2024-09-21T10:02:41.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:02:41.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:39.025307+0000", 2024-09-21T10:02:41.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1570000000000003e-06 2024-09-21T10:02:41.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:39.649423+0000", 2024-09-21T10:02:41.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.62411523099999999 2024-09-21T10:02:41.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.147 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:02:41.147 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:39.649502+0000", 2024-09-21T10:02:41.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9839999999999995e-05 2024-09-21T10:02:41.147 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.147 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.147 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.147 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45761.0:126105 9.3 9:c4fe1679:::benchmark_data_smithi107_152010_object16168:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163)", 2024-09-21T10:02:41.147 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:02:38.645120+0000", 2024-09-21T10:02:41.147 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.3907593710000001, 2024-09-21T10:02:41.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.03659575, 2024-09-21T10:02:41.147 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:02:41.147 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:02:41.148 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:02:41.148 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:02:41.148 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45761", 2024-09-21T10:02:41.148 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:02:41.148 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 126105 2024-09-21T10:02:41.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.148 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:02:41.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.148 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:02:41.148 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:38.645120+0000", 2024-09-21T10:02:41.148 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:02:41.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:38.645120+0000", 2024-09-21T10:02:41.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:02:41.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:38.645120+0000", 2024-09-21T10:02:41.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5199999999999998e-07 2024-09-21T10:02:41.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:02:41.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:38.645122+0000", 2024-09-21T10:02:41.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8390000000000001e-06 2024-09-21T10:02:41.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:02:41.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:38.645127+0000", 2024-09-21T10:02:41.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8969999999999999e-06 2024-09-21T10:02:41.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:02:41.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:38.645131+0000", 2024-09-21T10:02:41.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2610000000000004e-06 2024-09-21T10:02:41.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:39.024119+0000", 2024-09-21T10:02:41.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.37898793600000003 2024-09-21T10:02:41.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:02:41.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:39.024128+0000", 2024-09-21T10:02:41.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8470000000000002e-06 2024-09-21T10:02:41.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:39.681581+0000", 2024-09-21T10:02:41.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.65745327200000003 2024-09-21T10:02:41.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:02:41.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:39.681716+0000", 2024-09-21T10:02:41.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000134346 2024-09-21T10:02:41.152 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.152 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.152 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.152 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45761.0:126110 9.12 9:493e7512:::benchmark_data_smithi107_152010_object16173:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163)", 2024-09-21T10:02:41.152 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:02:38.645139+0000", 2024-09-21T10:02:41.152 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.3907405129999999, 2024-09-21T10:02:41.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.036276703, 2024-09-21T10:02:41.152 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:02:41.152 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:02:41.152 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:02:41.152 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:02:41.153 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45761", 2024-09-21T10:02:41.153 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:02:41.153 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 126110 2024-09-21T10:02:41.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.153 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:02:41.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.153 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:02:41.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:38.645139+0000", 2024-09-21T10:02:41.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.153 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:02:41.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:38.645139+0000", 2024-09-21T10:02:41.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:02:41.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:02:41.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:38.645139+0000", 2024-09-21T10:02:41.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8299999999999998e-07 2024-09-21T10:02:41.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:02:41.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:38.645141+0000", 2024-09-21T10:02:41.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6309999999999999e-06 2024-09-21T10:02:41.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:02:41.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:38.645145+0000", 2024-09-21T10:02:41.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.775e-06 2024-09-21T10:02:41.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:02:41.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:38.645150+0000", 2024-09-21T10:02:41.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2889999999999999e-06 2024-09-21T10:02:41.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:39.025332+0000", 2024-09-21T10:02:41.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.38018187199999998 2024-09-21T10:02:41.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:02:41.156 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:39.025338+0000", 2024-09-21T10:02:41.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1720000000000004e-06 2024-09-21T10:02:41.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:02:41.156 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:39.672203+0000", 2024-09-21T10:02:41.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.64686534399999995 2024-09-21T10:02:41.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:02:41.156 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:39.672318+0000", 2024-09-21T10:02:41.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000115186 2024-09-21T10:02:41.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.157 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:02:41.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:39.672534+0000", 2024-09-21T10:02:41.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000215434 2024-09-21T10:02:41.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.157 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:02:41.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:39.674092+0000", 2024-09-21T10:02:41.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001558713 2024-09-21T10:02:41.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.157 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:02:41.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:39.679467+0000", 2024-09-21T10:02:41.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0053750559999999996 2024-09-21T10:02:41.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:02:41.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:39.681293+0000", 2024-09-21T10:02:41.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0018255179999999999 2024-09-21T10:02:41.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:02:41.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:39.681365+0000", 2024-09-21T10:02:41.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2210000000000002e-05 2024-09-21T10:02:41.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:02:41.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:02:41.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:02:41.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:02:39.681415+0000", 2024-09-21T10:02:41.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0219999999999997e-05 2024-09-21T10:02:41.159 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.159 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.159 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.159 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:02:41.159 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:02:41.159 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:02:41.185 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:41.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:41.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:41.277 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:02:41.277 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:02:41.277 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:02:41.277 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:02:41.282 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:02:41.282 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:02:41.287 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:41.300 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:02:41.301 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:02:41.309 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:41.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:41.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:41.316 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:02:41.317 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:02:41.333 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:02:41.333 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:02:41.349 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:02:41.349 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:02:41.350 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T10:02:41.411 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:41.434 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:41.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:41.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:41.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:02:41.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:41.558 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:41.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:41.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:41.710 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:41.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:41.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:41.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:41 smithi107 ceph-mon[102221]: pgmap v578: 109 pgs: 2 remapped+peering, 8 unknown, 7 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 52 active+clean; 5.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 931 op/s; 15/284643 objects degraded (0.005%); 41610/284643 objects misplaced (14.618%); 0 B/s, 0 objects/s recovering 2024-09-21T10:02:41.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:41 smithi107 ceph-mon[102221]: 9.6 scrub starts 2024-09-21T10:02:41.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:41 smithi107 ceph-mon[102221]: 9.6 scrub ok 2024-09-21T10:02:41.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:41 smithi107 ceph-mon[102221]: osdmap e165: 8 total, 7 up, 8 in 2024-09-21T10:02:41.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:02:41.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:41.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:41 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1533517230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:02:41.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:41 smithi107 ceph-mon[105022]: pgmap v578: 109 pgs: 2 remapped+peering, 8 unknown, 7 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 52 active+clean; 5.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 931 op/s; 15/284643 objects degraded (0.005%); 41610/284643 objects misplaced (14.618%); 0 B/s, 0 objects/s recovering 2024-09-21T10:02:41.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:41 smithi107 ceph-mon[105022]: 9.6 scrub starts 2024-09-21T10:02:41.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:41 smithi107 ceph-mon[105022]: 9.6 scrub ok 2024-09-21T10:02:41.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:41 smithi107 ceph-mon[105022]: osdmap e165: 8 total, 7 up, 8 in 2024-09-21T10:02:41.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:02:41.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:41.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:41 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1533517230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:02:41.812 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:41.848 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:41.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:41.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:41 smithi129 ceph-mon[88471]: pgmap v578: 109 pgs: 2 remapped+peering, 8 unknown, 7 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 52 active+clean; 5.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 931 op/s; 15/284643 objects degraded (0.005%); 41610/284643 objects misplaced (14.618%); 0 B/s, 0 objects/s recovering 2024-09-21T10:02:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:41 smithi129 ceph-mon[88471]: 9.6 scrub starts 2024-09-21T10:02:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:41 smithi129 ceph-mon[88471]: 9.6 scrub ok 2024-09-21T10:02:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:41 smithi129 ceph-mon[88471]: osdmap e165: 8 total, 7 up, 8 in 2024-09-21T10:02:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:02:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:41 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1533517230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:02:41.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:41.984 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:41.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:41.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:42.086 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:42.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:02:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:02:41] "GET /metrics HTTP/1.1" 200 36668 "" "Prometheus/2.51.0" 2024-09-21T10:02:42.109 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:42.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:42.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:42.211 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:42.247 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:42.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:42.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:42.349 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:42.376 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:42.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:42.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:42.478 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:42.482 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:02:42.482 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:02:42.503 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:42.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:42.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:42.605 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:42.630 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:42.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:42.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:42.732 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:42.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:42.769 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:42.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:42.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:42 smithi107 ceph-mon[102221]: 9.5 scrub starts 2024-09-21T10:02:42.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:42 smithi107 ceph-mon[102221]: osdmap e166: 8 total, 7 up, 8 in 2024-09-21T10:02:42.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:42 smithi107 ceph-mon[102221]: 9.5 scrub ok 2024-09-21T10:02:42.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:42 smithi107 ceph-mon[105022]: 9.5 scrub starts 2024-09-21T10:02:42.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:42 smithi107 ceph-mon[105022]: osdmap e166: 8 total, 7 up, 8 in 2024-09-21T10:02:42.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:42 smithi107 ceph-mon[105022]: 9.5 scrub ok 2024-09-21T10:02:42.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:42.905 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:42.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:42.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:42.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:42 smithi129 ceph-mon[88471]: 9.5 scrub starts 2024-09-21T10:02:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:42 smithi129 ceph-mon[88471]: osdmap e166: 8 total, 7 up, 8 in 2024-09-21T10:02:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:42 smithi129 ceph-mon[88471]: 9.5 scrub ok 2024-09-21T10:02:42.956 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:02:43.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:43.008 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:43.035 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:43.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:43.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:43.136 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:43.162 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:43.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:43.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:43.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:43.303 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:43.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:43.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:43.313 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-09-21T10:02:43.314 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:02:43.314 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-21T10:02:43.314 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:02:43.314 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:02:43.314 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-21T10:02:43.321 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:02:43.405 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:43.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:43.445 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:43.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:43.546 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:43.577 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:43.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:43.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:43.678 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:43.704 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:43.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:43.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:43.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:43.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:43.841 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:43.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:43.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:43 smithi107 ceph-mon[105022]: pgmap v581: 109 pgs: 4 remapped+peering, 4 unknown, 7 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 54 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 549 op/s; 15/282204 objects degraded (0.005%); 35300/282204 objects misplaced (12.509%); 0 B/s, 0 objects/s recovering 2024-09-21T10:02:43.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:43 smithi107 ceph-mon[105022]: 9.9 scrub starts 2024-09-21T10:02:43.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:43 smithi107 ceph-mon[105022]: 9.8 scrub starts 2024-09-21T10:02:43.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:43 smithi107 ceph-mon[105022]: 9.9 scrub ok 2024-09-21T10:02:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:43 smithi107 ceph-mon[102221]: pgmap v581: 109 pgs: 4 remapped+peering, 4 unknown, 7 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 54 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 549 op/s; 15/282204 objects degraded (0.005%); 35300/282204 objects misplaced (12.509%); 0 B/s, 0 objects/s recovering 2024-09-21T10:02:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:43 smithi107 ceph-mon[102221]: 9.9 scrub starts 2024-09-21T10:02:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:43 smithi107 ceph-mon[102221]: 9.8 scrub starts 2024-09-21T10:02:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:43 smithi107 ceph-mon[102221]: 9.9 scrub ok 2024-09-21T10:02:43.879 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:43 smithi129 ceph-mon[88471]: pgmap v581: 109 pgs: 4 remapped+peering, 4 unknown, 7 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 54 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 549 op/s; 15/282204 objects degraded (0.005%); 35300/282204 objects misplaced (12.509%); 0 B/s, 0 objects/s recovering 2024-09-21T10:02:43.879 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:43 smithi129 ceph-mon[88471]: 9.9 scrub starts 2024-09-21T10:02:43.879 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:43 smithi129 ceph-mon[88471]: 9.8 scrub starts 2024-09-21T10:02:43.879 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:43 smithi129 ceph-mon[88471]: 9.9 scrub ok 2024-09-21T10:02:43.942 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:43.968 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:43.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:43.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:44.070 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:44.107 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:44.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:44.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:44.154 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:02:44.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:44.245 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:44.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:44.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:44.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:44.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:44.387 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:44.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:44.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:02:44.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:44.551 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:44.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:44.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:44.652 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:44.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:44.674 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:44.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:44.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:44.803 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:44.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:44.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:44.867 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:44 smithi129 ceph-mon[88471]: 9.8 scrub ok 2024-09-21T10:02:44.868 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:44 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/282204 objects degraded (0.005%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:44.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:44.927 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:44.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:44.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:45.028 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:45.050 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:45.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:45.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:45.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:44 smithi107 ceph-mon[102221]: 9.8 scrub ok 2024-09-21T10:02:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:44 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/282204 objects degraded (0.005%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:45.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:44 smithi107 ceph-mon[105022]: 9.8 scrub ok 2024-09-21T10:02:45.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:44 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/282204 objects degraded (0.005%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:45.151 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:45.176 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:45.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:45.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:45.278 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:45.308 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:45.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:45.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:45.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:45.430 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:45.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:45.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:45.531 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:45.568 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:45.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:45.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:45.670 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:45.695 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:45.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:45.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:45.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:45.817 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:45.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:45.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:45.819 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:02:45.820 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:02:45.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:45.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:45.955 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:45.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:46.046 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:45 smithi107 ceph-mon[102221]: pgmap v582: 109 pgs: 4 remapped+peering, 2 unknown, 8 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 15/280233 objects degraded (0.005%); 34926/280233 objects misplaced (12.463%); 0 B/s, 0 objects/s recovering 2024-09-21T10:02:46.047 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:45 smithi107 ceph-mon[102221]: 9.7 scrub starts 2024-09-21T10:02:46.047 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:45 smithi107 ceph-mon[102221]: 9.19 scrub starts 2024-09-21T10:02:46.047 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:45 smithi107 ceph-mon[102221]: 9.19 scrub ok 2024-09-21T10:02:46.047 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:45 smithi107 ceph-mon[105022]: pgmap v582: 109 pgs: 4 remapped+peering, 2 unknown, 8 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 15/280233 objects degraded (0.005%); 34926/280233 objects misplaced (12.463%); 0 B/s, 0 objects/s recovering 2024-09-21T10:02:46.047 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:45 smithi107 ceph-mon[105022]: 9.7 scrub starts 2024-09-21T10:02:46.047 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:45 smithi107 ceph-mon[105022]: 9.19 scrub starts 2024-09-21T10:02:46.047 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:45 smithi107 ceph-mon[105022]: 9.19 scrub ok 2024-09-21T10:02:46.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:46.091 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:46.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:46.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:46.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:45 smithi129 ceph-mon[88471]: pgmap v582: 109 pgs: 4 remapped+peering, 2 unknown, 8 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 15/280233 objects degraded (0.005%); 34926/280233 objects misplaced (12.463%); 0 B/s, 0 objects/s recovering 2024-09-21T10:02:46.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:45 smithi129 ceph-mon[88471]: 9.7 scrub starts 2024-09-21T10:02:46.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:45 smithi129 ceph-mon[88471]: 9.19 scrub starts 2024-09-21T10:02:46.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:45 smithi129 ceph-mon[88471]: 9.19 scrub ok 2024-09-21T10:02:46.192 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:46.219 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:46.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:46.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:46.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:46.357 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:46.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:46.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:46.455 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:02:46.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:46.493 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:46.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:46.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:46.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:46.630 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:46.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:46.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:46.702 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:46 smithi129 ceph-mon[88471]: 9.7 scrub ok 2024-09-21T10:02:46.731 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:46.758 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:46.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:46.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:46.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:46.896 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:46.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:46.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:47.033 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:02:47.038 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:47.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:47.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:47.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:46 smithi107 ceph-mon[102221]: 9.7 scrub ok 2024-09-21T10:02:47.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:46 smithi107 ceph-mon[105022]: 9.7 scrub ok 2024-09-21T10:02:47.140 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:47.161 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:47.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:47.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:47.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:47.296 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:47.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:47.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:47.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:47.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:47.438 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:47.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:47.539 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:47.568 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:47.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:47.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:47.670 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:47.697 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:47.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:47.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:47.791 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:02:47.791 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:02:47.791 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:02:47.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:47.838 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:47.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:47.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:47.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:47 smithi107 ceph-mon[102221]: pgmap v583: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 15/272055 objects degraded (0.006%); 50941/272055 objects misplaced (18.725%) 2024-09-21T10:02:47.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:47 smithi107 ceph-mon[102221]: 9.2 scrub starts 2024-09-21T10:02:47.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:47 smithi107 ceph-mon[105022]: pgmap v583: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 15/272055 objects degraded (0.006%); 50941/272055 objects misplaced (18.725%) 2024-09-21T10:02:47.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:47 smithi107 ceph-mon[105022]: 9.2 scrub starts 2024-09-21T10:02:47.941 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:47.962 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:47.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:47.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:47.980 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:47 smithi129 ceph-mon[88471]: pgmap v583: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 15/272055 objects degraded (0.006%); 50941/272055 objects misplaced (18.725%) 2024-09-21T10:02:47.980 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:47 smithi129 ceph-mon[88471]: 9.2 scrub starts 2024-09-21T10:02:48.064 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:48.096 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:48.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:48.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:48.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:48.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:48.240 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:48.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:48.341 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:48.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:48.382 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:48.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:48.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:48.504 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:48.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:48.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:48.606 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:48.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:48.651 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:48.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:48.752 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:48.780 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:48.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:48.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:48.846 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:48 smithi129 ceph-mon[88471]: 9.16 scrub starts 2024-09-21T10:02:48.846 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:48 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:48.846 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:48 smithi129 ceph-mon[88471]: 9.2 scrub ok 2024-09-21T10:02:48.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:48.924 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:48.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:48.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:49.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:49.059 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:49.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:49.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:49.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:48 smithi107 ceph-mon[102221]: 9.16 scrub starts 2024-09-21T10:02:49.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:48 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:49.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:48 smithi107 ceph-mon[102221]: 9.2 scrub ok 2024-09-21T10:02:49.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:48 smithi107 ceph-mon[105022]: 9.16 scrub starts 2024-09-21T10:02:49.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:48 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:49.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:48 smithi107 ceph-mon[105022]: 9.2 scrub ok 2024-09-21T10:02:49.162 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:49.188 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:49.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:49.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:49.290 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:49.317 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:49.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:49.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:49.418 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:49.439 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:49.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:49.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:49.540 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:49.566 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:49.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:49.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:49.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:49.695 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:49.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:49.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:49.797 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:49.837 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:49.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:49.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:49.850 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:02:49.850 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:02:49.850 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:49 smithi129 ceph-mon[88471]: pgmap v584: 109 pgs: 1 active+clean+scrubbing, 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 54 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 15/267162 objects degraded (0.006%); 49929/267162 objects misplaced (18.689%) 2024-09-21T10:02:49.850 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:49 smithi129 ceph-mon[88471]: 9.16 scrub ok 2024-09-21T10:02:49.850 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:49 smithi129 ceph-mon[88471]: 9.0 scrub starts 2024-09-21T10:02:49.850 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:49 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/794043323' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:02:49.850 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:49 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:02:49.850 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:49 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/267162 objects degraded (0.006%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:49.939 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:49.962 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:49.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:49.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:49.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:49 smithi107 ceph-mon[102221]: pgmap v584: 109 pgs: 1 active+clean+scrubbing, 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 54 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 15/267162 objects degraded (0.006%); 49929/267162 objects misplaced (18.689%) 2024-09-21T10:02:49.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:49 smithi107 ceph-mon[102221]: 9.16 scrub ok 2024-09-21T10:02:49.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:49 smithi107 ceph-mon[102221]: 9.0 scrub starts 2024-09-21T10:02:49.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:49 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/794043323' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:02:49.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:49 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:02:49.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:49 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/267162 objects degraded (0.006%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:49.972 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:49 smithi107 ceph-mon[105022]: pgmap v584: 109 pgs: 1 active+clean+scrubbing, 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 54 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 15/267162 objects degraded (0.006%); 49929/267162 objects misplaced (18.689%) 2024-09-21T10:02:49.972 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:49 smithi107 ceph-mon[105022]: 9.16 scrub ok 2024-09-21T10:02:49.972 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:49 smithi107 ceph-mon[105022]: 9.0 scrub starts 2024-09-21T10:02:49.972 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:49 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/794043323' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:02:49.972 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:49 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:02:49.972 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:49 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/267162 objects degraded (0.006%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:49.972 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:02:50.064 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:50.101 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:50.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:50.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:50.203 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:50.236 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:50.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:50.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:50.338 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:50.373 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:50.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:50.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:50.474 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:50.477 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:02:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:02:50.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:50.505 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:50.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:50.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:50.512 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:02:50.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:50.638 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:50.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:50.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:50.740 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:50.767 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:50.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:50.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:50.832 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:50 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:02:50.832 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:50 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:02:50.832 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:50 smithi129 ceph-mon[88471]: osdmap e167: 8 total, 7 up, 8 in 2024-09-21T10:02:50.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:50.889 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:50.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:50.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:50.991 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:51.012 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:51.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:51.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:51.047 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:02:51.108 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:50 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:02:51.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:50 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:02:51.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:50 smithi107 ceph-mon[105022]: osdmap e167: 8 total, 7 up, 8 in 2024-09-21T10:02:51.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:50 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:02:51.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:50 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:02:51.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:50 smithi107 ceph-mon[102221]: osdmap e167: 8 total, 7 up, 8 in 2024-09-21T10:02:51.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:51.147 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:51.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:51.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:51.248 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:51.290 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:51.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:51.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:51.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:51.430 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:51.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:51.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:51.532 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:51.558 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:51.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:51.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:51.660 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:51.696 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:51.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:51.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:51.798 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:51.830 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:51.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:51.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:51.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:51 smithi129 ceph-mon[88471]: pgmap v586: 109 pgs: 1 active+clean+scrubbing, 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 54 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 15/261327 objects degraded (0.006%); 48989/261327 objects misplaced (18.746%) 2024-09-21T10:02:51.934 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:51.964 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:51.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:51.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:52.058 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:51 smithi107 ceph-mon[102221]: pgmap v586: 109 pgs: 1 active+clean+scrubbing, 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 54 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 15/261327 objects degraded (0.006%); 48989/261327 objects misplaced (18.746%) 2024-09-21T10:02:52.058 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:51 smithi107 ceph-mon[105022]: pgmap v586: 109 pgs: 1 active+clean+scrubbing, 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 54 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 15/261327 objects degraded (0.006%); 48989/261327 objects misplaced (18.746%) 2024-09-21T10:02:52.059 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:02:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:02:51] "GET /metrics HTTP/1.1" 200 36668 "" "Prometheus/2.51.0" 2024-09-21T10:02:52.066 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:52.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:52.107 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:52.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:52.208 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:52.232 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:52.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:52.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:52.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:52.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:52.369 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:52.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:52.419 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:52.470 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:52.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:52.528 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:52.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:52.629 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:52.656 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:52.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:52.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:52.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:52.804 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:52.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:52.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:52.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:52.941 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:52.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:53.042 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:53.069 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:53.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:53.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:53.171 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:53.210 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:53.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:53.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:53.312 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:53.342 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:53.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:53.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:53.444 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:53.468 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:53.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:53.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:53.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:53.608 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:53.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:53.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:53.710 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:53.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:53.749 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:53.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:53.833 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:02:53.833 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:02:53.850 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:53.880 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:53.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:53.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:53.968 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:53 smithi107 ceph-mon[102221]: pgmap v587: 109 pgs: 1 active+clean+scrubbing, 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 54 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 15/256305 objects degraded (0.006%); 48041/256305 objects misplaced (18.744%) 2024-09-21T10:02:53.968 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:53 smithi107 ceph-mon[105022]: pgmap v587: 109 pgs: 1 active+clean+scrubbing, 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 54 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 15/256305 objects degraded (0.006%); 48041/256305 objects misplaced (18.744%) 2024-09-21T10:02:53.982 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:54.005 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:54.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:54.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:54.106 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:54.137 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:54.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:54.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:54.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:53 smithi129 ceph-mon[88471]: pgmap v587: 109 pgs: 1 active+clean+scrubbing, 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 54 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 15/256305 objects degraded (0.006%); 48041/256305 objects misplaced (18.744%) 2024-09-21T10:02:54.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:54.288 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:54.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:54.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:54.339 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-09-21T10:02:54.339 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:02:54.339 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-21T10:02:54.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:02:54.391 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:54.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:02:54.413 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:54.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:54.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:54.515 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:54.559 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:54.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:54.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:54.571 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:02:54.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:54.722 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:54.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:54.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:54.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:54.870 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:54.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:54.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:54.909 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:02:54.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:54 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/256305 objects degraded (0.006%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:54.936 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:54 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/256305 objects degraded (0.006%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:54.936 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:54 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/256305 objects degraded (0.006%), 10 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-09-21T10:02:54.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:55.019 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:55.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:55.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:55.120 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:55.142 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:02:55.142 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:02:55.144 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:55.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:55.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:55.187 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:02:55.187 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:02:55.187 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:55.208 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:55.226 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:02:55.227 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:02:55.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:55.272 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:02:55.272 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:02:55.305 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:55.307 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:02:55.307 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:02:55.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:55.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:55.339 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:02:55.339 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:02:55.369 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:02:55.369 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:02:55.370 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T10:02:55.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:55.448 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:55.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:55.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:55.550 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:55.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:55.580 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:55.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:55.681 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:55.707 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:55.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:55.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:55.809 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:55.835 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:55.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:55.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:55.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:55 smithi129 ceph-mon[88471]: pgmap v588: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 15/251715 objects degraded (0.006%); 47069/251715 objects misplaced (18.699%) 2024-09-21T10:02:55.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:55 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1874543761' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:02:55.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:55.976 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:55.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:55.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:56.077 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:56.104 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:56.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:56.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:56.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:55 smithi107 ceph-mon[102221]: pgmap v588: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 15/251715 objects degraded (0.006%); 47069/251715 objects misplaced (18.699%) 2024-09-21T10:02:56.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:55 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1874543761' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:02:56.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:55 smithi107 ceph-mon[105022]: pgmap v588: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 15/251715 objects degraded (0.006%); 47069/251715 objects misplaced (18.699%) 2024-09-21T10:02:56.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:55 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1874543761' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:02:56.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:56.236 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:56.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:56.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:56.338 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:56.372 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:56.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:56.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:56.473 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:56.500 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:56.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:56.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:56.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:56.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:56.640 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:56.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:56.711 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:02:56.711 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:02:56.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:56.782 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:56.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:56.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:56.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:56.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:56.919 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:56.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:57.020 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:57.053 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:57.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:57.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:57.155 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:57.189 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:57.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:57.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:57.257 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:57 smithi107 ceph-mon[102221]: pgmap v589: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 15/240567 objects degraded (0.006%); 45027/240567 objects misplaced (18.717%) 2024-09-21T10:02:57.257 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:57 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:02:57.257 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:57 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:57.258 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:57 smithi107 ceph-mon[105022]: pgmap v589: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 15/240567 objects degraded (0.006%); 45027/240567 objects misplaced (18.717%) 2024-09-21T10:02:57.258 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:57 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:02:57.258 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:57 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:57.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:57.334 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:57.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:57.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:57.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:57 smithi129 ceph-mon[88471]: pgmap v589: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 15/240567 objects degraded (0.006%); 45027/240567 objects misplaced (18.717%) 2024-09-21T10:02:57.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:57 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:02:57.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:57 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:57.436 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:57.468 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:57.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:57.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:57.569 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:57.603 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:57.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:57.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:57.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:57.725 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:02:57.725 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:02:57.725 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.726 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:143389 9.6 9:62166f58:::benchmark_data_smithi107_152010_object33452:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-09-21T10:02:57.726 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:57.643167+0000", 2024-09-21T10:02:57.726 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.0095246480000000001, 2024-09-21T10:02:57.726 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0095523529999999995, 2024-09-21T10:02:57.726 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:02:57.726 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:02:57.726 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:02:57.726 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:02:57.726 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:02:57.726 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 143389 2024-09-21T10:02:57.726 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.726 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:02:57.726 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.727 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:02:57.727 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.643167+0000", 2024-09-21T10:02:57.727 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:02:57.727 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.727 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.727 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:02:57.727 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.643167+0000", 2024-09-21T10:02:57.727 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:02:57.727 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.727 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.727 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:02:57.727 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.643166+0000", 2024-09-21T10:02:57.727 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:02:57.727 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.728 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.728 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:02:57.728 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.643172+0000", 2024-09-21T10:02:57.728 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.319e-06 2024-09-21T10:02:57.728 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.728 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.728 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:02:57.728 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.643185+0000", 2024-09-21T10:02:57.728 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2831e-05 2024-09-21T10:02:57.728 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.728 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.728 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:02:57.728 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.643192+0000", 2024-09-21T10:02:57.729 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3860000000000004e-06 2024-09-21T10:02:57.729 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.729 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.729 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:02:57.729 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.643235+0000", 2024-09-21T10:02:57.729 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3090000000000002e-05 2024-09-21T10:02:57.729 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.729 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.729 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:02:57.729 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.643360+0000", 2024-09-21T10:02:57.729 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012536399999999999 2024-09-21T10:02:57.729 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.729 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.729 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:02:57.730 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.643763+0000", 2024-09-21T10:02:57.730 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00040287899999999998 2024-09-21T10:02:57.730 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.730 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.730 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:02:57.730 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.645935+0000", 2024-09-21T10:02:57.730 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0021723430000000002 2024-09-21T10:02:57.730 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.730 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.730 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:02:57.730 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.649663+0000", 2024-09-21T10:02:57.730 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0037275400000000001 2024-09-21T10:02:57.730 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:02:57.731 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:02:57.731 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:02:57.731 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.731 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.731 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:143396 9.7 9:e33138dc:::benchmark_data_smithi107_152010_object33459:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-09-21T10:02:57.731 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:57.649371+0000", 2024-09-21T10:02:57.731 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.0033207039999999998, 2024-09-21T10:02:57.731 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003434224, 2024-09-21T10:02:57.731 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:02:57.731 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:02:57.731 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:02:57.731 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:02:57.731 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:02:57.732 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 143396 2024-09-21T10:02:57.732 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.732 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:02:57.732 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.732 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:02:57.732 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.649371+0000", 2024-09-21T10:02:57.732 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:02:57.732 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.732 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.732 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:02:57.732 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.649371+0000", 2024-09-21T10:02:57.732 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:02:57.732 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.733 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.733 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:02:57.733 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.649369+0000", 2024-09-21T10:02:57.733 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:02:57.733 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.733 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.733 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:02:57.733 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.649377+0000", 2024-09-21T10:02:57.733 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.7230000000000004e-06 2024-09-21T10:02:57.733 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.733 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.733 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:02:57.733 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.649395+0000", 2024-09-21T10:02:57.734 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.8241e-05 2024-09-21T10:02:57.734 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.734 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.734 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:02:57.734 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.649402+0000", 2024-09-21T10:02:57.734 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4999999999999996e-06 2024-09-21T10:02:57.734 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.734 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.734 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:02:57.734 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.649491+0000", 2024-09-21T10:02:57.734 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.954e-05 2024-09-21T10:02:57.734 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.734 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.735 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:02:57.735 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.649711+0000", 2024-09-21T10:02:57.735 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000219633 2024-09-21T10:02:57.735 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.735 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.735 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:02:57.735 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.650155+0000", 2024-09-21T10:02:57.735 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00044427399999999998 2024-09-21T10:02:57.735 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.735 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.735 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:02:57.735 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.652423+0000", 2024-09-21T10:02:57.735 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0022678020000000002 2024-09-21T10:02:57.735 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:02:57.735 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:02:57.735 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:02:57.736 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.736 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.736 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:143397 9.a 9:54defd5e:::benchmark_data_smithi107_152010_object33460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-09-21T10:02:57.736 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:57.649424+0000", 2024-09-21T10:02:57.736 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.0032674750000000002, 2024-09-21T10:02:57.736 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003451431, 2024-09-21T10:02:57.736 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:02:57.736 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:02:57.736 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:02:57.736 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:02:57.736 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:02:57.736 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 143397 2024-09-21T10:02:57.737 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.737 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:02:57.737 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.737 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:02:57.737 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.649424+0000", 2024-09-21T10:02:57.737 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:02:57.737 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.737 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.737 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:02:57.737 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.649424+0000", 2024-09-21T10:02:57.737 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:02:57.737 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.737 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.737 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:02:57.737 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.649424+0000", 2024-09-21T10:02:57.738 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:02:57.738 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.738 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.738 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:02:57.738 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.649427+0000", 2024-09-21T10:02:57.738 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.343e-06 2024-09-21T10:02:57.738 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.738 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.738 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:02:57.738 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.649434+0000", 2024-09-21T10:02:57.738 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2840000000000001e-06 2024-09-21T10:02:57.738 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.738 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.738 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:02:57.738 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.649439+0000", 2024-09-21T10:02:57.739 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7330000000000003e-06 2024-09-21T10:02:57.739 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.739 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.739 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:02:57.739 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.649484+0000", 2024-09-21T10:02:57.739 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5127999999999999e-05 2024-09-21T10:02:57.739 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.739 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.739 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:02:57.739 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.649613+0000", 2024-09-21T10:02:57.739 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000128975 2024-09-21T10:02:57.739 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.739 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.739 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:02:57.740 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.649742+0000", 2024-09-21T10:02:57.740 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012925 2024-09-21T10:02:57.740 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:02:57.740 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:02:57.740 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:02:57.740 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.740 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.740 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.45761.0:143393 9.b e167/141 9:da62d5d1:::benchmark_data_smithi107_152010_object33456:head v 167'9031, mlcod=167'9029)", 2024-09-21T10:02:57.740 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:57.650065+0000", 2024-09-21T10:02:57.740 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.0026267560000000001, 2024-09-21T10:02:57.740 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0028757710000000001, 2024-09-21T10:02:57.740 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:02:57.740 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "started", 2024-09-21T10:02:57.741 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:02:57.741 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.741 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:02:57.741 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.650065+0000", 2024-09-21T10:02:57.741 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:02:57.741 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.741 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.741 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:02:57.741 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.650065+0000", 2024-09-21T10:02:57.741 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:02:57.741 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.741 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.741 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:02:57.741 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.650064+0000", 2024-09-21T10:02:57.741 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:02:57.742 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.742 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.742 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:02:57.742 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.650068+0000", 2024-09-21T10:02:57.742 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8020000000000002e-06 2024-09-21T10:02:57.742 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.742 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.742 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:02:57.742 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.650073+0000", 2024-09-21T10:02:57.742 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1970000000000004e-06 2024-09-21T10:02:57.742 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.742 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.742 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:02:57.743 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.650080+0000", 2024-09-21T10:02:57.743 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5590000000000002e-06 2024-09-21T10:02:57.743 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.743 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.743 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:02:57.743 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.650113+0000", 2024-09-21T10:02:57.743 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3395999999999997e-05 2024-09-21T10:02:57.743 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.743 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.743 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:02:57.743 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.650134+0000", 2024-09-21T10:02:57.743 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.0166999999999999e-05 2024-09-21T10:02:57.743 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:02:57.743 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:02:57.744 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:02:57.744 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.744 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.744 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:143382 9.a 9:5a910ef5:::benchmark_data_smithi107_152010_object33445:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-09-21T10:02:57.744 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:57.636420+0000", 2024-09-21T10:02:57.744 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.016271526000000001, 2024-09-21T10:02:57.744 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.016575189000000001, 2024-09-21T10:02:57.744 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:02:57.744 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:02:57.744 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:02:57.744 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:02:57.744 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:02:57.744 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 143382 2024-09-21T10:02:57.744 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.745 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:02:57.745 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.745 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:02:57.745 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.636420+0000", 2024-09-21T10:02:57.745 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:02:57.745 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.745 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.745 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:02:57.745 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.636420+0000", 2024-09-21T10:02:57.745 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:02:57.745 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.745 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.745 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:02:57.746 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.636419+0000", 2024-09-21T10:02:57.746 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:02:57.746 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.746 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.746 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:02:57.746 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.636425+0000", 2024-09-21T10:02:57.746 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0079999999999999e-06 2024-09-21T10:02:57.746 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.746 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.746 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:02:57.746 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.636436+0000", 2024-09-21T10:02:57.746 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1303e-05 2024-09-21T10:02:57.746 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.747 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.747 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:02:57.747 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.636442+0000", 2024-09-21T10:02:57.747 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.412e-06 2024-09-21T10:02:57.747 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.747 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.747 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:02:57.747 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.636554+0000", 2024-09-21T10:02:57.747 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011208499999999999 2024-09-21T10:02:57.747 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.747 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.747 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:02:57.747 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.636681+0000", 2024-09-21T10:02:57.748 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012697899999999999 2024-09-21T10:02:57.748 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.748 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.748 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:02:57.748 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.636799+0000", 2024-09-21T10:02:57.748 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011796800000000001 2024-09-21T10:02:57.748 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.748 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.748 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:02:57.748 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.642235+0000", 2024-09-21T10:02:57.748 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0054359580000000003 2024-09-21T10:02:57.748 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.748 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.748 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:02:57.749 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.647342+0000", 2024-09-21T10:02:57.749 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0051073749999999999 2024-09-21T10:02:57.749 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:02:57.749 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:02:57.749 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:02:57.749 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.749 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.749 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:143386 9.a 9:5cb446e2:::benchmark_data_smithi107_152010_object33449:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-09-21T10:02:57.749 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:57.642269+0000", 2024-09-21T10:02:57.749 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.010422463999999999, 2024-09-21T10:02:57.749 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.010801956, 2024-09-21T10:02:57.749 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:02:57.749 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:02:57.749 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:02:57.750 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:02:57.750 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:02:57.750 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 143386 2024-09-21T10:02:57.750 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.750 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:02:57.750 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.750 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:02:57.750 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.642269+0000", 2024-09-21T10:02:57.750 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:02:57.750 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.750 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.750 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:02:57.750 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.642269+0000", 2024-09-21T10:02:57.750 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:02:57.751 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.751 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.751 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:02:57.751 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.642267+0000", 2024-09-21T10:02:57.751 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:02:57.751 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.751 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.751 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:02:57.751 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.642275+0000", 2024-09-21T10:02:57.751 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.362e-06 2024-09-21T10:02:57.751 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.751 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.751 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:02:57.751 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.642291+0000", 2024-09-21T10:02:57.752 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.6555999999999999e-05 2024-09-21T10:02:57.752 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.752 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.752 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:02:57.752 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.642297+0000", 2024-09-21T10:02:57.752 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3010000000000003e-06 2024-09-21T10:02:57.752 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.752 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.752 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:02:57.752 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.642345+0000", 2024-09-21T10:02:57.752 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7691999999999997e-05 2024-09-21T10:02:57.752 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.752 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.753 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:02:57.753 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.642505+0000", 2024-09-21T10:02:57.753 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000159994 2024-09-21T10:02:57.753 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.753 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.753 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:02:57.753 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.642640+0000", 2024-09-21T10:02:57.753 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000134728 2024-09-21T10:02:57.753 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:02:57.753 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:02:57.753 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:02:57.753 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:57.647466+0000", 2024-09-21T10:02:57.754 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0048264520000000002 2024-09-21T10:02:57.754 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:02:57.754 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:02:57.754 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:02:57.754 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:02:57.754 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-21T10:02:57.754 INFO:teuthology.orchestra.run.smithi129.stdout: "num_ops": 6 2024-09-21T10:02:57.754 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:02:57.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:57.755 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:57.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:57.856 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:02:57.883 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:57.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:57.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:02:57.985 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:58.011 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:58.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:58.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:58.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:58.155 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:58.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:58.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:58.256 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:58.283 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:58.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:58.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:58.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:58.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:58.428 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:58.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:58.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:58.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:58.584 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:58.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:58.685 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:58.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:02:58.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:58.713 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:58.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:58.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:02:58.818 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:02:58.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:58.856 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:58.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:02:58.957 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:02:58.985 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:58.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:58.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:02:59.087 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:59.123 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:59.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:59.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:59.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:02:59.280 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:59.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:59.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:02:59.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:02:59 smithi107 ceph-mon[102221]: pgmap v590: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 15/235338 objects degraded (0.006%); 43994/235338 objects misplaced (18.694%) 2024-09-21T10:02:59.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:02:59 smithi107 ceph-mon[105022]: pgmap v590: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 15/235338 objects degraded (0.006%); 43994/235338 objects misplaced (18.694%) 2024-09-21T10:02:59.382 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:02:59.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:02:59 smithi129 ceph-mon[88471]: pgmap v590: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 15/235338 objects degraded (0.006%); 43994/235338 objects misplaced (18.694%) 2024-09-21T10:02:59.421 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:59.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:59.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:02:59.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:59.558 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:59.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:59.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:59.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:59.702 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:59.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:59.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:02:59.804 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:02:59.805 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:02:59.836 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:02:59.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:59.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:02:59.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:02:59.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:02:59.979 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:02:59.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:00.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:00.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:00.129 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:00.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:00.191 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:03:00.216 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:00 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/235338 objects degraded (0.006%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:00.216 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:00 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2827781881' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:03:00.216 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:00 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:03:00.231 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:00.255 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:00.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:00.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:00.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:00.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:00 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/235338 objects degraded (0.006%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:00.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:00 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2827781881' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:03:00.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:00 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:03:00.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:00 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/235338 objects degraded (0.006%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:00.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:00 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2827781881' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:03:00.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:00 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:03:00.396 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:00.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:00.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:00.488 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:03:00.489 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:03:00.498 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:00.524 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:00.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:00.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:00.599 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:03:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:03:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:00.626 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:00.650 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:00.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:00.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:00.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:00.794 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:00.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:00.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:00.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:00.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:00.943 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:00.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:01.043 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:01.070 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:01.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:01.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:01.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:01.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:01.211 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:01.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:01.312 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:01.338 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:01.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:01.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:01.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:01 smithi107 ceph-mon[102221]: pgmap v591: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 327 op/s; 15/229401 objects degraded (0.007%); 43000/229401 objects misplaced (18.744%) 2024-09-21T10:03:01.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:01 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:03:01.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:01 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:03:01.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:01 smithi107 ceph-mon[102221]: osdmap e168: 8 total, 7 up, 8 in 2024-09-21T10:03:01.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:01 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:01.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:01 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:01.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:01 smithi107 ceph-mon[102221]: from='client.55627 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:03:01.358 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:01 smithi107 ceph-mon[105022]: pgmap v591: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 327 op/s; 15/229401 objects degraded (0.007%); 43000/229401 objects misplaced (18.744%) 2024-09-21T10:03:01.358 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:01 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:03:01.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:01 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:03:01.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:01 smithi107 ceph-mon[105022]: osdmap e168: 8 total, 7 up, 8 in 2024-09-21T10:03:01.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:01 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:01.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:01 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:01.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:01 smithi107 ceph-mon[105022]: from='client.55627 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:03:01.439 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:01.467 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:01.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:01.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:01.569 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:01.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:01 smithi129 ceph-mon[88471]: pgmap v591: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 327 op/s; 15/229401 objects degraded (0.007%); 43000/229401 objects misplaced (18.744%) 2024-09-21T10:03:01.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:01 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:03:01.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:01 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:03:01.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:01 smithi129 ceph-mon[88471]: osdmap e168: 8 total, 7 up, 8 in 2024-09-21T10:03:01.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:01 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:01.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:01 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:01.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:01 smithi129 ceph-mon[88471]: from='client.55627 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:03:01.601 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:01.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:01.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:01.739 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:01.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:01.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:01.832 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:03:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:03:01] "GET /metrics HTTP/1.1" 200 36661 "" "Prometheus/2.51.0" 2024-09-21T10:03:01.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:01.881 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:01.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:01.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:01.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:02.024 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:02.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:02.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:02.167 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:03:02.167 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:03:02.167 INFO:teuthology.orchestra.run.smithi129.stdout: "complaint_time": 30, 2024-09-21T10:03:02.167 INFO:teuthology.orchestra.run.smithi129.stdout: "num_blocked_ops": 0 2024-09-21T10:03:02.167 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:03:02.255 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:02.285 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:02.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:02.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:02.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:02.431 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:02.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:02.494 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:02.517 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":592,"stamp":"2024-09-21T10:03:00.175101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5010102245,"num_objects":76467,"num_object_clones":0,"num_object_copies":229401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":43000,"num_objects_unfound":0,"num_objects_dirty":76467,"num_whiteouts":0,"num_read":293,"num_read_kb":748,"num_write":111401,"num_write_kb":5280573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2734,"num_bytes_recovered":178629160,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":203062,"ondisk_log_size":203062,"up":273,"acting":287,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":326,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":19632236,"kb_used_data":14788268,"kb_used_omap":29,"kb_used_meta":4843810,"kb_avail":730328980,"statfs":{"total":767960285184,"available":747856875520,"internally_reserved":0,"allocated":15143186432,"data_stored":15136411492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30226,"internal_metadata":4960061934},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,6,8,10],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-697434112,"num_objects":-10642,"num_object_clones":0,"num_object_copies":-31926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-5989,"num_objects_unfound":0,"num_objects_dirty":-10642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2280,"num_write_kb":-445664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.275299"},"pg_stats":[{"pgid":"9.19","version":"167'8405","reported_seq":9383,"reported_epoch":167,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:58.929446+0000","last_change":"2024-09-21T10:02:44.964039+0000","last_active":"2024-09-21T10:02:58.929446+0000","last_peered":"2024-09-21T10:02:58.929446+0000","last_clean":"2024-09-21T10:02:58.929446+0000","last_became_active":"2024-09-21T10:02:41.648521+0000","last_became_peered":"2024-09-21T10:02:41.648521+0000","last_unstale":"2024-09-21T10:02:58.929446+0000","last_undegraded":"2024-09-21T10:02:58.929446+0000","last_fullsized":"2024-09-21T10:02:58.929446+0000","mapping_epoch":165,"log_start":"147'600","ondisk_log_start":"147'600","created":163,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":5,"last_scrub":"166'7909","last_scrub_stamp":"2024-09-21T10:02:44.963922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:02:44.963922+0000","objects_scrubbed":2861,"log_size":7805,"log_dups_size":0,"ondisk_log_size":7805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:48.667074+0000","scrub_duration":512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154992640,"num_objects":2365,"num_object_clones":0,"num_object_copies":7095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2365,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":496,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":2365}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":176,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.739801+0000","last_change":"2024-09-21T10:01:17.360671+0000","last_active":"2024-09-21T10:02:49.739801+0000","last_peered":"2024-09-21T10:02:49.739801+0000","last_clean":"2024-09-21T10:01:12.339813+0000","last_became_active":"2024-09-21T10:01:17.360671+0000","last_became_peered":"2024-09-21T10:01:17.360671+0000","last_unstale":"2024-09-21T10:02:49.739801+0000","last_undegraded":"2024-09-21T10:02:49.739801+0000","last_fullsized":"2024-09-21T10:01:17.352087+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:28:38.696084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":55,"reported_epoch":166,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:02:41.635006+0000","last_change":"2024-09-21T10:01:17.355722+0000","last_active":"2024-09-21T10:02:41.635006+0000","last_peered":"2024-09-21T10:02:41.635006+0000","last_clean":"2024-09-21T10:00:52.395654+0000","last_became_active":"2024-09-21T10:01:17.355722+0000","last_became_peered":"2024-09-21T10:01:17.355722+0000","last_unstale":"2024-09-21T10:02:41.635006+0000","last_undegraded":"2024-09-21T10:01:17.351745+0000","last_fullsized":"2024-09-21T10:01:17.351642+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:08:49.082950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.18","version":"167'8301","reported_seq":8373,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:55.799524+0000","last_change":"2024-09-21T10:02:39.684341+0000","last_active":"2024-09-21T10:02:55.799524+0000","last_peered":"2024-09-21T10:02:55.799524+0000","last_clean":"2024-09-21T10:02:55.799524+0000","last_became_active":"2024-09-21T10:02:39.677756+0000","last_became_peered":"2024-09-21T10:02:39.677756+0000","last_unstale":"2024-09-21T10:02:55.799524+0000","last_undegraded":"2024-09-21T10:02:55.799524+0000","last_fullsized":"2024-09-21T10:02:55.799524+0000","mapping_epoch":163,"log_start":"146'400","ondisk_log_start":"146'400","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7901,"log_dups_size":0,"ondisk_log_size":7901,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156565504,"num_objects":2389,"num_object_clones":0,"num_object_copies":7167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4401,"num_write_kb":217280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":158,"reported_epoch":166,"state":"active+undersized","last_fresh":"2024-09-21T10:02:41.638064+0000","last_change":"2024-09-21T10:01:17.360872+0000","last_active":"2024-09-21T10:02:41.638064+0000","last_peered":"2024-09-21T10:02:41.638064+0000","last_clean":"2024-09-21T10:01:12.334087+0000","last_became_active":"2024-09-21T10:01:17.360872+0000","last_became_peered":"2024-09-21T10:01:17.360872+0000","last_unstale":"2024-09-21T10:02:41.638064+0000","last_undegraded":"2024-09-21T10:02:41.638064+0000","last_fullsized":"2024-09-21T10:01:17.356555+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:03.409955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":227,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.745080+0000","last_change":"2024-09-21T10:01:17.362044+0000","last_active":"2024-09-21T10:02:49.745080+0000","last_peered":"2024-09-21T10:02:49.745080+0000","last_clean":"2024-09-21T10:01:12.346595+0000","last_became_active":"2024-09-21T10:01:17.362044+0000","last_became_peered":"2024-09-21T10:01:17.362044+0000","last_unstale":"2024-09-21T10:02:49.745080+0000","last_undegraded":"2024-09-21T10:02:49.745080+0000","last_fullsized":"2024-09-21T10:01:17.354667+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:41.227793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":227,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.747339+0000","last_change":"2024-09-21T10:01:17.358516+0000","last_active":"2024-09-21T10:02:49.747339+0000","last_peered":"2024-09-21T10:02:49.747339+0000","last_clean":"2024-09-21T10:01:12.350555+0000","last_became_active":"2024-09-21T10:01:17.358516+0000","last_became_peered":"2024-09-21T10:01:17.358516+0000","last_unstale":"2024-09-21T10:02:49.747339+0000","last_undegraded":"2024-09-21T10:02:49.747339+0000","last_fullsized":"2024-09-21T10:01:17.354339+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:23.544961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":38,"reported_epoch":166,"state":"active+undersized","last_fresh":"2024-09-21T10:02:41.635380+0000","last_change":"2024-09-21T10:01:17.362650+0000","last_active":"2024-09-21T10:02:41.635380+0000","last_peered":"2024-09-21T10:02:41.635380+0000","last_clean":"2024-09-21T09:40:48.067317+0000","last_became_active":"2024-09-21T10:01:17.362650+0000","last_became_peered":"2024-09-21T10:01:17.362650+0000","last_unstale":"2024-09-21T10:02:41.635380+0000","last_undegraded":"2024-09-21T10:02:41.635380+0000","last_fullsized":"2024-09-21T10:01:17.351981+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:14.466960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":176,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.742463+0000","last_change":"2024-09-21T10:01:17.361530+0000","last_active":"2024-09-21T10:02:49.742463+0000","last_peered":"2024-09-21T10:02:49.742463+0000","last_clean":"2024-09-21T10:01:12.338661+0000","last_became_active":"2024-09-21T10:01:17.361530+0000","last_became_peered":"2024-09-21T10:01:17.361530+0000","last_unstale":"2024-09-21T10:02:49.742463+0000","last_undegraded":"2024-09-21T10:02:49.742463+0000","last_fullsized":"2024-09-21T10:01:17.354825+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:10.575192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":139,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T10:02:41.647141+0000","last_change":"2024-09-21T09:48:54.140948+0000","last_active":"2024-09-21T10:02:41.647141+0000","last_peered":"2024-09-21T10:02:41.647141+0000","last_clean":"2024-09-21T10:02:41.647141+0000","last_became_active":"2024-09-21T09:48:54.140641+0000","last_became_peered":"2024-09-21T09:48:54.140641+0000","last_unstale":"2024-09-21T10:02:41.647141+0000","last_undegraded":"2024-09-21T10:02:41.647141+0000","last_fullsized":"2024-09-21T10:02:41.647141+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:25.157599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"167'8455","reported_seq":9500,"reported_epoch":167,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:58.893702+0000","last_change":"2024-09-21T10:02:42.221067+0000","last_active":"2024-09-21T10:02:58.893702+0000","last_peered":"2024-09-21T10:02:58.893702+0000","last_clean":"2024-09-21T10:02:58.893702+0000","last_became_active":"2024-09-21T10:02:39.629264+0000","last_became_peered":"2024-09-21T10:02:39.629264+0000","last_unstale":"2024-09-21T10:02:58.893702+0000","last_undegraded":"2024-09-21T10:02:58.893702+0000","last_fullsized":"2024-09-21T10:02:58.893702+0000","mapping_epoch":153,"log_start":"147'700","ondisk_log_start":"147'700","created":141,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'7885","last_scrub_stamp":"2024-09-21T10:02:42.220944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:02:42.220944+0000","objects_scrubbed":2950,"log_size":7755,"log_dups_size":0,"ondisk_log_size":7755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:30.789546+0000","scrub_duration":721,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154730496,"num_objects":2361,"num_object_clones":0,"num_object_copies":7083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2361,"num_objects_unfound":0,"num_objects_dirty":2361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":589,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,1],"avail_no_missing":["0","1","5"],"object_location_counts":[{"shards":"0,1,5","objects":2361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":53,"reported_epoch":166,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:02:41.635227+0000","last_change":"2024-09-21T10:01:17.362126+0000","last_active":"2024-09-21T10:02:41.635227+0000","last_peered":"2024-09-21T10:02:41.635227+0000","last_clean":"2024-09-21T10:00:52.396178+0000","last_became_active":"2024-09-21T10:01:17.362126+0000","last_became_peered":"2024-09-21T10:01:17.362126+0000","last_unstale":"2024-09-21T10:02:41.635227+0000","last_undegraded":"2024-09-21T10:01:17.352720+0000","last_fullsized":"2024-09-21T10:01:17.352607+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:15.152821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":141,"reported_epoch":166,"state":"active+undersized","last_fresh":"2024-09-21T10:02:41.637551+0000","last_change":"2024-09-21T10:01:17.360241+0000","last_active":"2024-09-21T10:02:41.637551+0000","last_peered":"2024-09-21T10:02:41.637551+0000","last_clean":"2024-09-21T10:01:12.331943+0000","last_became_active":"2024-09-21T10:01:17.360241+0000","last_became_peered":"2024-09-21T10:01:17.360241+0000","last_unstale":"2024-09-21T10:02:41.637551+0000","last_undegraded":"2024-09-21T10:02:41.637551+0000","last_fullsized":"2024-09-21T10:01:17.354797+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:56.749660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"167'8522","reported_seq":8595,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:59.766738+0000","last_change":"2024-09-21T10:02:39.666157+0000","last_active":"2024-09-21T10:02:59.766738+0000","last_peered":"2024-09-21T10:02:59.766738+0000","last_clean":"2024-09-21T10:02:59.766738+0000","last_became_active":"2024-09-21T10:02:39.659997+0000","last_became_peered":"2024-09-21T10:02:39.659997+0000","last_unstale":"2024-09-21T10:02:59.766738+0000","last_undegraded":"2024-09-21T10:02:59.766738+0000","last_fullsized":"2024-09-21T10:02:59.766738+0000","mapping_epoch":141,"log_start":"147'700","ondisk_log_start":"147'700","created":141,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7822,"log_dups_size":0,"ondisk_log_size":7822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:03:51.817516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148799488,"num_objects":2270,"num_object_clones":0,"num_object_copies":6810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4589,"num_write_kb":219520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":53,"reported_epoch":166,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:02:41.635402+0000","last_change":"2024-09-21T10:01:17.365156+0000","last_active":"2024-09-21T10:02:41.635402+0000","last_peered":"2024-09-21T10:02:41.635402+0000","last_clean":"2024-09-21T10:00:52.396279+0000","last_became_active":"2024-09-21T10:01:17.365156+0000","last_became_peered":"2024-09-21T10:01:17.365156+0000","last_unstale":"2024-09-21T10:02:41.635402+0000","last_undegraded":"2024-09-21T10:01:17.352823+0000","last_fullsized":"2024-09-21T10:01:17.352726+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:25.514616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":147,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T10:02:41.634993+0000","last_change":"2024-09-21T09:56:12.576294+0000","last_active":"2024-09-21T10:02:41.634993+0000","last_peered":"2024-09-21T10:02:41.634993+0000","last_clean":"2024-09-21T10:02:41.634993+0000","last_became_active":"2024-09-21T09:56:12.575928+0000","last_became_peered":"2024-09-21T09:56:12.575928+0000","last_unstale":"2024-09-21T10:02:41.634993+0000","last_undegraded":"2024-09-21T10:02:41.634993+0000","last_fullsized":"2024-09-21T10:02:41.634993+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:23.227435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"167'8444","reported_seq":9987,"reported_epoch":167,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:57.602387+0000","last_change":"2024-09-21T10:02:45.731362+0000","last_active":"2024-09-21T10:02:57.602387+0000","last_peered":"2024-09-21T10:02:57.602387+0000","last_clean":"2024-09-21T10:02:57.602387+0000","last_became_active":"2024-09-21T10:02:39.660444+0000","last_became_peered":"2024-09-21T10:02:39.660444+0000","last_unstale":"2024-09-21T10:02:57.602387+0000","last_undegraded":"2024-09-21T10:02:57.602387+0000","last_fullsized":"2024-09-21T10:02:57.602387+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'7995","last_scrub_stamp":"2024-09-21T10:02:45.731210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:02:45.731210+0000","objects_scrubbed":2798,"log_size":8444,"log_dups_size":0,"ondisk_log_size":8444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:24:33.225118+0000","scrub_duration":1.443131521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152829952,"num_objects":2332,"num_object_clones":0,"num_object_copies":6996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2332,"num_objects_unfound":0,"num_objects_dirty":2332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":466,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3,2],"avail_no_missing":["5","2","3"],"object_location_counts":[{"shards":"2,3,5","objects":2332}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":53,"reported_epoch":166,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:02:41.638044+0000","last_change":"2024-09-21T10:01:17.360223+0000","last_active":"2024-09-21T10:02:41.638044+0000","last_peered":"2024-09-21T10:02:41.638044+0000","last_clean":"2024-09-21T10:00:52.397515+0000","last_became_active":"2024-09-21T10:01:17.360223+0000","last_became_peered":"2024-09-21T10:01:17.360223+0000","last_unstale":"2024-09-21T10:02:41.638044+0000","last_undegraded":"2024-09-21T10:01:17.357081+0000","last_fullsized":"2024-09-21T10:01:17.356981+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:14:39.079600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":157,"reported_epoch":166,"state":"active+undersized","last_fresh":"2024-09-21T10:02:41.647199+0000","last_change":"2024-09-21T10:01:17.364112+0000","last_active":"2024-09-21T10:02:41.647199+0000","last_peered":"2024-09-21T10:02:41.647199+0000","last_clean":"2024-09-21T10:01:12.345222+0000","last_became_active":"2024-09-21T10:01:17.364112+0000","last_became_peered":"2024-09-21T10:01:17.364112+0000","last_unstale":"2024-09-21T10:02:41.647199+0000","last_undegraded":"2024-09-21T10:02:41.647199+0000","last_fullsized":"2024-09-21T10:01:17.355079+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:43:09.114787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"167'8347","reported_seq":8517,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:57.563692+0000","last_change":"2024-09-21T10:02:40.516038+0000","last_active":"2024-09-21T10:02:57.563692+0000","last_peered":"2024-09-21T10:02:57.563692+0000","last_clean":"2024-09-21T10:02:57.563692+0000","last_became_active":"2024-09-21T10:02:39.667977+0000","last_became_peered":"2024-09-21T10:02:39.667977+0000","last_unstale":"2024-09-21T10:02:57.563692+0000","last_undegraded":"2024-09-21T10:02:57.563692+0000","last_fullsized":"2024-09-21T10:02:57.563692+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":141,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'7805","last_scrub_stamp":"2024-09-21T10:02:40.515975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:02:40.515975+0000","objects_scrubbed":2906,"log_size":8347,"log_dups_size":0,"ondisk_log_size":8347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:11.555893+0000","scrub_duration":0.28421902799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154927104,"num_objects":2364,"num_object_clones":0,"num_object_copies":7092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":542,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":132,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T10:02:41.637826+0000","last_change":"2024-09-21T09:56:12.590219+0000","last_active":"2024-09-21T10:02:41.637826+0000","last_peered":"2024-09-21T10:02:41.637826+0000","last_clean":"2024-09-21T10:02:41.637826+0000","last_became_active":"2024-09-21T09:56:12.589946+0000","last_became_peered":"2024-09-21T09:56:12.589946+0000","last_unstale":"2024-09-21T10:02:41.637826+0000","last_undegraded":"2024-09-21T10:02:41.637826+0000","last_fullsized":"2024-09-21T10:02:41.637826+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:48.919929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":210,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.745082+0000","last_change":"2024-09-21T09:50:45.503336+0000","last_active":"2024-09-21T10:02:49.745082+0000","last_peered":"2024-09-21T10:02:49.745082+0000","last_clean":"2024-09-21T10:02:49.745082+0000","last_became_active":"2024-09-21T09:50:45.503045+0000","last_became_peered":"2024-09-21T09:50:45.503045+0000","last_unstale":"2024-09-21T10:02:49.745082+0000","last_undegraded":"2024-09-21T10:02:49.745082+0000","last_fullsized":"2024-09-21T10:02:49.745082+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:46.728347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":211,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.744431+0000","last_change":"2024-09-21T10:01:55.372679+0000","last_active":"2024-09-21T10:02:49.744431+0000","last_peered":"2024-09-21T10:02:49.744431+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.372679+0000","last_became_peered":"2024-09-21T10:01:55.372679+0000","last_unstale":"2024-09-21T10:02:49.744431+0000","last_undegraded":"2024-09-21T10:02:49.744431+0000","last_fullsized":"2024-09-21T10:01:55.355636+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:03:12.985869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"167'8293","reported_seq":9261,"reported_epoch":167,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:55.791313+0000","last_change":"2024-09-21T10:02:39.654875+0000","last_active":"2024-09-21T10:02:55.791313+0000","last_peered":"2024-09-21T10:02:55.791313+0000","last_clean":"2024-09-21T10:02:55.791313+0000","last_became_active":"2024-09-21T10:02:39.640720+0000","last_became_peered":"2024-09-21T10:02:39.640720+0000","last_unstale":"2024-09-21T10:02:55.791313+0000","last_undegraded":"2024-09-21T10:02:55.791313+0000","last_fullsized":"2024-09-21T10:02:55.791313+0000","mapping_epoch":153,"log_start":"146'500","ondisk_log_start":"146'500","created":141,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7793,"log_dups_size":0,"ondisk_log_size":7793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:03:52.097226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162693120,"num_objects":2482,"num_object_clones":0,"num_object_copies":7446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2482,"num_objects_unfound":0,"num_objects_dirty":2482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4370,"num_write_kb":219296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":14385152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":2482}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":147,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.737350+0000","last_change":"2024-09-21T09:56:12.575897+0000","last_active":"2024-09-21T10:02:49.737350+0000","last_peered":"2024-09-21T10:02:49.737350+0000","last_clean":"2024-09-21T10:02:49.737350+0000","last_became_active":"2024-09-21T09:56:12.575532+0000","last_became_peered":"2024-09-21T09:56:12.575532+0000","last_unstale":"2024-09-21T10:02:49.737350+0000","last_undegraded":"2024-09-21T10:02:49.737350+0000","last_fullsized":"2024-09-21T10:02:49.737350+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:01.302499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":130,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T10:02:41.635299+0000","last_change":"2024-09-21T09:56:12.588824+0000","last_active":"2024-09-21T10:02:41.635299+0000","last_peered":"2024-09-21T10:02:41.635299+0000","last_clean":"2024-09-21T10:02:41.635299+0000","last_became_active":"2024-09-21T09:56:12.588561+0000","last_became_peered":"2024-09-21T09:56:12.588561+0000","last_unstale":"2024-09-21T10:02:41.635299+0000","last_undegraded":"2024-09-21T10:02:41.635299+0000","last_fullsized":"2024-09-21T10:02:41.635299+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:31:48.900623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":211,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.744433+0000","last_change":"2024-09-21T10:01:55.386193+0000","last_active":"2024-09-21T10:02:49.744433+0000","last_peered":"2024-09-21T10:02:49.744433+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.386193+0000","last_became_peered":"2024-09-21T10:01:55.386193+0000","last_unstale":"2024-09-21T10:02:49.744433+0000","last_undegraded":"2024-09-21T10:02:49.744433+0000","last_fullsized":"2024-09-21T10:01:55.366920+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:03:28.321050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"167'8656","reported_seq":8757,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:59.833602+0000","last_change":"2024-09-21T10:02:49.777569+0000","last_active":"2024-09-21T10:02:59.833602+0000","last_peered":"2024-09-21T10:02:59.833602+0000","last_clean":"2024-09-21T10:02:59.833602+0000","last_became_active":"2024-09-21T10:02:39.648843+0000","last_became_peered":"2024-09-21T10:02:39.648843+0000","last_unstale":"2024-09-21T10:02:59.833602+0000","last_undegraded":"2024-09-21T10:02:59.833602+0000","last_fullsized":"2024-09-21T10:02:59.833602+0000","mapping_epoch":141,"log_start":"147'800","ondisk_log_start":"147'800","created":141,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":2599,"log_size":7856,"log_dups_size":0,"ondisk_log_size":7856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:03:19.775507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150765568,"num_objects":2300,"num_object_clones":0,"num_object_copies":6900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4659,"num_write_kb":222720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":131,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T10:02:41.647385+0000","last_change":"2024-09-21T09:50:45.501596+0000","last_active":"2024-09-21T10:02:41.647385+0000","last_peered":"2024-09-21T10:02:41.647385+0000","last_clean":"2024-09-21T10:02:41.647385+0000","last_became_active":"2024-09-21T09:50:45.501186+0000","last_became_peered":"2024-09-21T09:50:45.501186+0000","last_unstale":"2024-09-21T10:02:41.647385+0000","last_undegraded":"2024-09-21T10:02:41.647385+0000","last_fullsized":"2024-09-21T10:02:41.647385+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:43:16.449425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":140,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T10:02:41.635313+0000","last_change":"2024-09-21T09:51:30.639545+0000","last_active":"2024-09-21T10:02:41.635313+0000","last_peered":"2024-09-21T10:02:41.635313+0000","last_clean":"2024-09-21T10:02:41.635313+0000","last_became_active":"2024-09-21T09:51:30.639106+0000","last_became_peered":"2024-09-21T09:51:30.639106+0000","last_unstale":"2024-09-21T10:02:41.635313+0000","last_undegraded":"2024-09-21T10:02:41.635313+0000","last_fullsized":"2024-09-21T10:02:41.635313+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:19:19.607327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"167'8282","reported_seq":9185,"reported_epoch":167,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:55.749772+0000","last_change":"2024-09-21T10:02:39.682062+0000","last_active":"2024-09-21T10:02:55.749772+0000","last_peered":"2024-09-21T10:02:55.749772+0000","last_clean":"2024-09-21T10:02:55.749772+0000","last_became_active":"2024-09-21T10:02:39.674946+0000","last_became_peered":"2024-09-21T10:02:39.674946+0000","last_unstale":"2024-09-21T10:02:55.749772+0000","last_undegraded":"2024-09-21T10:02:55.749772+0000","last_fullsized":"2024-09-21T10:02:55.749772+0000","mapping_epoch":153,"log_start":"146'400","ondisk_log_start":"146'400","created":141,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7882,"log_dups_size":0,"ondisk_log_size":7882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156991488,"num_objects":2395,"num_object_clones":0,"num_object_copies":7185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2395,"num_objects_unfound":0,"num_objects_dirty":2395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4376,"num_write_kb":216704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":13533184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,5],"avail_no_missing":["3","1","5"],"object_location_counts":[{"shards":"1,3,5","objects":2395}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":158,"reported_epoch":166,"state":"active+undersized","last_fresh":"2024-09-21T10:02:41.635157+0000","last_change":"2024-09-21T10:01:17.356446+0000","last_active":"2024-09-21T10:02:41.635157+0000","last_peered":"2024-09-21T10:02:41.635157+0000","last_clean":"2024-09-21T10:01:12.333151+0000","last_became_active":"2024-09-21T10:01:17.356446+0000","last_became_peered":"2024-09-21T10:01:17.356446+0000","last_unstale":"2024-09-21T10:02:41.635157+0000","last_undegraded":"2024-09-21T10:02:41.635157+0000","last_fullsized":"2024-09-21T10:01:17.352337+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:01:29.779281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":130,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T10:02:41.635443+0000","last_change":"2024-09-21T09:50:45.510764+0000","last_active":"2024-09-21T10:02:41.635443+0000","last_peered":"2024-09-21T10:02:41.635443+0000","last_clean":"2024-09-21T10:02:41.635443+0000","last_became_active":"2024-09-21T09:50:45.510226+0000","last_became_peered":"2024-09-21T09:50:45.510226+0000","last_unstale":"2024-09-21T10:02:41.635443+0000","last_undegraded":"2024-09-21T10:02:41.635443+0000","last_fullsized":"2024-09-21T10:02:41.635443+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:31:13.491427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":211,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.744482+0000","last_change":"2024-09-21T10:01:55.369453+0000","last_active":"2024-09-21T10:02:49.744482+0000","last_peered":"2024-09-21T10:02:49.744482+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.369453+0000","last_became_peered":"2024-09-21T10:01:55.369453+0000","last_unstale":"2024-09-21T10:02:49.744482+0000","last_undegraded":"2024-09-21T10:02:49.744482+0000","last_fullsized":"2024-09-21T10:01:55.356778+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:02:56.964734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2","version":"167'8614","reported_seq":8723,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:55.748998+0000","last_change":"2024-09-21T10:02:48.349939+0000","last_active":"2024-09-21T10:02:55.748998+0000","last_peered":"2024-09-21T10:02:55.748998+0000","last_clean":"2024-09-21T10:02:55.748998+0000","last_became_active":"2024-09-21T10:02:39.641495+0000","last_became_peered":"2024-09-21T10:02:39.641495+0000","last_unstale":"2024-09-21T10:02:55.748998+0000","last_undegraded":"2024-09-21T10:02:55.748998+0000","last_fullsized":"2024-09-21T10:02:55.748998+0000","mapping_epoch":141,"log_start":"147'800","ondisk_log_start":"147'800","created":141,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'8358","last_scrub_stamp":"2024-09-21T10:02:48.349773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:02:48.349773+0000","objects_scrubbed":2782,"log_size":7814,"log_dups_size":0,"ondisk_log_size":7814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:35.450135+0000","scrub_duration":1173,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164429824,"num_objects":2509,"num_object_clones":0,"num_object_copies":7527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":273,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":178,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.736050+0000","last_change":"2024-09-21T10:01:17.365027+0000","last_active":"2024-09-21T10:02:49.736050+0000","last_peered":"2024-09-21T10:02:49.736050+0000","last_clean":"2024-09-21T10:01:12.337728+0000","last_became_active":"2024-09-21T10:01:17.365027+0000","last_became_peered":"2024-09-21T10:01:17.365027+0000","last_unstale":"2024-09-21T10:02:49.736050+0000","last_undegraded":"2024-09-21T10:02:49.736050+0000","last_fullsized":"2024-09-21T10:01:17.352121+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:14:52.580064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":180,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.743460+0000","last_change":"2024-09-21T09:56:12.575396+0000","last_active":"2024-09-21T10:02:49.743460+0000","last_peered":"2024-09-21T10:02:49.743460+0000","last_clean":"2024-09-21T10:02:49.743460+0000","last_became_active":"2024-09-21T09:56:12.574913+0000","last_became_peered":"2024-09-21T09:56:12.574913+0000","last_unstale":"2024-09-21T10:02:49.743460+0000","last_undegraded":"2024-09-21T10:02:49.743460+0000","last_fullsized":"2024-09-21T10:02:49.743460+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:05.462880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"167'8246","reported_seq":8355,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:55.804510+0000","last_change":"2024-09-21T10:02:43.803389+0000","last_active":"2024-09-21T10:02:55.804510+0000","last_peered":"2024-09-21T10:02:55.804510+0000","last_clean":"2024-09-21T10:02:55.804510+0000","last_became_active":"2024-09-21T10:02:39.638710+0000","last_became_peered":"2024-09-21T10:02:39.638710+0000","last_unstale":"2024-09-21T10:02:55.804510+0000","last_undegraded":"2024-09-21T10:02:55.804510+0000","last_fullsized":"2024-09-21T10:02:55.804510+0000","mapping_epoch":141,"log_start":"146'400","ondisk_log_start":"146'400","created":141,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'7872","last_scrub_stamp":"2024-09-21T10:02:43.803106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:02:43.803106+0000","objects_scrubbed":2823,"log_size":7846,"log_dups_size":0,"ondisk_log_size":7846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:17:35.167144+0000","scrub_duration":744,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159842304,"num_objects":2439,"num_object_clones":0,"num_object_copies":7317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":384,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":212,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.744786+0000","last_change":"2024-09-21T10:01:55.375667+0000","last_active":"2024-09-21T10:02:49.744786+0000","last_peered":"2024-09-21T10:02:49.744786+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.375667+0000","last_became_peered":"2024-09-21T10:01:55.375667+0000","last_unstale":"2024-09-21T10:02:49.744786+0000","last_undegraded":"2024-09-21T10:02:49.744786+0000","last_fullsized":"2024-09-21T10:01:55.353939+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:03:25.092803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":67,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":180,"reported_epoch":166,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:02:41.637887+0000","last_change":"2024-09-21T10:01:17.359511+0000","last_active":"2024-09-21T10:02:41.637887+0000","last_peered":"2024-09-21T10:02:41.637887+0000","last_clean":"2024-09-21T10:01:12.332227+0000","last_became_active":"2024-09-21T10:01:17.359511+0000","last_became_peered":"2024-09-21T10:01:17.359511+0000","last_unstale":"2024-09-21T10:02:41.637887+0000","last_undegraded":"2024-09-21T10:01:17.357161+0000","last_fullsized":"2024-09-21T10:01:17.356996+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:39:03.218322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039581700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":148,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.738701+0000","last_change":"2024-09-21T09:56:12.580057+0000","last_active":"2024-09-21T10:02:49.738701+0000","last_peered":"2024-09-21T10:02:49.738701+0000","last_clean":"2024-09-21T10:02:49.738701+0000","last_became_active":"2024-09-21T09:56:12.579700+0000","last_became_peered":"2024-09-21T09:56:12.579700+0000","last_unstale":"2024-09-21T10:02:49.738701+0000","last_undegraded":"2024-09-21T10:02:49.738701+0000","last_fullsized":"2024-09-21T10:02:49.738701+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:49.768872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":140,"reported_epoch":166,"state":"active+undersized","last_fresh":"2024-09-21T10:02:41.647981+0000","last_change":"2024-09-21T10:01:17.359297+0000","last_active":"2024-09-21T10:02:41.647981+0000","last_peered":"2024-09-21T10:02:41.647981+0000","last_clean":"2024-09-21T10:01:12.345523+0000","last_became_active":"2024-09-21T10:01:17.359297+0000","last_became_peered":"2024-09-21T10:01:17.359297+0000","last_unstale":"2024-09-21T10:02:41.647981+0000","last_undegraded":"2024-09-21T10:02:41.647981+0000","last_fullsized":"2024-09-21T10:01:17.354792+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:54:20.381939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":211,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.744846+0000","last_change":"2024-09-21T10:01:55.378211+0000","last_active":"2024-09-21T10:02:49.744846+0000","last_peered":"2024-09-21T10:02:49.744846+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.378211+0000","last_became_peered":"2024-09-21T10:01:55.378211+0000","last_unstale":"2024-09-21T10:02:49.744846+0000","last_undegraded":"2024-09-21T10:02:49.744846+0000","last_fullsized":"2024-09-21T10:01:55.368219+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:03:26.247046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":107340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"167'9106","reported_seq":9172,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:59.001152+0000","last_change":"2024-09-21T10:00:40.152326+0000","last_active":"2024-09-21T10:02:59.001152+0000","last_peered":"2024-09-21T10:02:59.001152+0000","last_clean":"2024-09-21T10:02:59.001152+0000","last_became_active":"2024-09-21T10:00:40.152002+0000","last_became_peered":"2024-09-21T10:00:40.152002+0000","last_unstale":"2024-09-21T10:02:59.001152+0000","last_undegraded":"2024-09-21T10:02:59.001152+0000","last_fullsized":"2024-09-21T10:02:59.001152+0000","mapping_epoch":141,"log_start":"147'2402","ondisk_log_start":"147'2402","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":6704,"log_dups_size":0,"ondisk_log_size":6704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:37.560496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308805632,"num_objects":4712,"num_object_clones":0,"num_object_copies":14136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9106,"num_write_kb":442176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":135,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.771624+0000","last_change":"2024-09-21T10:01:17.367117+0000","last_active":"2024-09-21T10:02:49.771624+0000","last_peered":"2024-09-21T10:02:49.771624+0000","last_clean":"2024-09-21T09:50:10.064980+0000","last_became_active":"2024-09-21T10:01:17.367117+0000","last_became_peered":"2024-09-21T10:01:17.367117+0000","last_unstale":"2024-09-21T10:02:49.771624+0000","last_undegraded":"2024-09-21T10:02:49.771624+0000","last_fullsized":"2024-09-21T10:01:17.359433+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:55:44.782398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":209,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.747429+0000","last_change":"2024-09-21T09:56:12.573698+0000","last_active":"2024-09-21T10:02:49.747429+0000","last_peered":"2024-09-21T10:02:49.747429+0000","last_clean":"2024-09-21T10:02:49.747429+0000","last_became_active":"2024-09-21T09:56:12.573390+0000","last_became_peered":"2024-09-21T09:56:12.573390+0000","last_unstale":"2024-09-21T10:02:49.747429+0000","last_undegraded":"2024-09-21T10:02:49.747429+0000","last_fullsized":"2024-09-21T10:02:49.747429+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:55:12.937878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":221,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.747156+0000","last_change":"2024-09-21T09:51:30.666462+0000","last_active":"2024-09-21T10:02:49.747156+0000","last_peered":"2024-09-21T10:02:49.747156+0000","last_clean":"2024-09-21T10:02:49.747156+0000","last_became_active":"2024-09-21T09:51:30.666006+0000","last_became_peered":"2024-09-21T09:51:30.666006+0000","last_unstale":"2024-09-21T10:02:49.747156+0000","last_undegraded":"2024-09-21T10:02:49.747156+0000","last_fullsized":"2024-09-21T10:02:49.747156+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:51.481698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":211,"reported_epoch":167,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:02:49.744872+0000","last_change":"2024-09-21T10:01:55.369767+0000","last_active":"2024-09-21T10:02:49.744872+0000","last_peered":"2024-09-21T10:02:49.744872+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.369767+0000","last_became_peered":"2024-09-21T10:01:55.369767+0000","last_unstale":"2024-09-21T10:02:49.744872+0000","last_undegraded":"2024-09-21T10:01:55.367783+0000","last_fullsized":"2024-09-21T10:01:55.358839+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:02:59.949481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":107340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"167'8802","reported_seq":9732,"reported_epoch":167,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:57.567376+0000","last_change":"2024-09-21T10:01:29.735004+0000","last_active":"2024-09-21T10:02:57.567376+0000","last_peered":"2024-09-21T10:02:57.567376+0000","last_clean":"2024-09-21T10:02:57.567376+0000","last_became_active":"2024-09-21T10:01:18.386142+0000","last_became_peered":"2024-09-21T10:01:18.386142+0000","last_unstale":"2024-09-21T10:02:57.567376+0000","last_undegraded":"2024-09-21T10:02:57.567376+0000","last_fullsized":"2024-09-21T10:02:57.567376+0000","mapping_epoch":153,"log_start":"147'600","ondisk_log_start":"147'600","created":141,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":8202,"log_dups_size":0,"ondisk_log_size":8202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:41.716874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306446336,"num_objects":4676,"num_object_clones":0,"num_object_copies":14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4676,"num_objects_unfound":0,"num_objects_dirty":4676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8802,"num_write_kb":431296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":431,"num_bytes_recovered":28246016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1,3],"avail_no_missing":["5","1","3"],"object_location_counts":[{"shards":"1,3,5","objects":4676}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":211,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.735968+0000","last_change":"2024-09-21T09:56:12.580659+0000","last_active":"2024-09-21T10:02:49.735968+0000","last_peered":"2024-09-21T10:02:49.735968+0000","last_clean":"2024-09-21T10:02:49.735968+0000","last_became_active":"2024-09-21T09:56:12.580300+0000","last_became_peered":"2024-09-21T09:56:12.580300+0000","last_unstale":"2024-09-21T10:02:49.735968+0000","last_undegraded":"2024-09-21T10:02:49.735968+0000","last_fullsized":"2024-09-21T10:02:49.735968+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:04.945404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":180,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.771896+0000","last_change":"2024-09-21T09:51:30.646466+0000","last_active":"2024-09-21T10:02:49.771896+0000","last_peered":"2024-09-21T10:02:49.771896+0000","last_clean":"2024-09-21T10:02:49.771896+0000","last_became_active":"2024-09-21T09:51:30.646209+0000","last_became_peered":"2024-09-21T09:51:30.646209+0000","last_unstale":"2024-09-21T10:02:49.771896+0000","last_undegraded":"2024-09-21T10:02:49.771896+0000","last_fullsized":"2024-09-21T10:02:49.771896+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:44:41.465418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":184,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.738348+0000","last_change":"2024-09-21T09:51:30.649056+0000","last_active":"2024-09-21T10:02:49.738348+0000","last_peered":"2024-09-21T10:02:49.738348+0000","last_clean":"2024-09-21T10:02:49.738348+0000","last_became_active":"2024-09-21T09:51:30.647245+0000","last_became_peered":"2024-09-21T09:51:30.647245+0000","last_unstale":"2024-09-21T10:02:49.738348+0000","last_undegraded":"2024-09-21T10:02:49.738348+0000","last_fullsized":"2024-09-21T10:02:49.738348+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:32.661847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":210,"reported_epoch":167,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:02:49.744520+0000","last_change":"2024-09-21T10:01:55.369315+0000","last_active":"2024-09-21T10:02:49.744520+0000","last_peered":"2024-09-21T10:02:49.744520+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.369315+0000","last_became_peered":"2024-09-21T10:01:55.369315+0000","last_unstale":"2024-09-21T10:02:49.744520+0000","last_undegraded":"2024-09-21T10:01:55.367566+0000","last_fullsized":"2024-09-21T10:01:55.354220+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:03:10.982992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":107340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"167'8374","reported_seq":9355,"reported_epoch":167,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:58.993026+0000","last_change":"2024-09-21T10:02:43.241195+0000","last_active":"2024-09-21T10:02:58.993026+0000","last_peered":"2024-09-21T10:02:58.993026+0000","last_clean":"2024-09-21T10:02:58.993026+0000","last_became_active":"2024-09-21T10:02:39.635334+0000","last_became_peered":"2024-09-21T10:02:39.635334+0000","last_unstale":"2024-09-21T10:02:58.993026+0000","last_undegraded":"2024-09-21T10:02:58.993026+0000","last_fullsized":"2024-09-21T10:02:58.993026+0000","mapping_epoch":153,"log_start":"147'600","ondisk_log_start":"147'600","created":141,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'7865","last_scrub_stamp":"2024-09-21T10:02:43.240710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:02:43.240710+0000","objects_scrubbed":2756,"log_size":7774,"log_dups_size":0,"ondisk_log_size":7774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:50:00.416196+0000","scrub_duration":775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146407424,"num_objects":2234,"num_object_clones":0,"num_object_copies":6702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2234,"num_objects_unfound":0,"num_objects_dirty":2234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":522,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":2234}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":146,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T10:02:41.637703+0000","last_change":"2024-09-21T09:50:45.502094+0000","last_active":"2024-09-21T10:02:41.637703+0000","last_peered":"2024-09-21T10:02:41.637703+0000","last_clean":"2024-09-21T10:02:41.637703+0000","last_became_active":"2024-09-21T09:50:45.501603+0000","last_became_peered":"2024-09-21T09:50:45.501603+0000","last_unstale":"2024-09-21T10:02:41.637703+0000","last_undegraded":"2024-09-21T10:02:41.637703+0000","last_fullsized":"2024-09-21T10:02:41.637703+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:01.747067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091939799999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":131,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T10:02:41.635251+0000","last_change":"2024-09-21T09:48:54.128617+0000","last_active":"2024-09-21T10:02:41.635251+0000","last_peered":"2024-09-21T10:02:41.635251+0000","last_clean":"2024-09-21T10:02:41.635251+0000","last_became_active":"2024-09-21T09:48:54.127878+0000","last_became_peered":"2024-09-21T09:48:54.127878+0000","last_unstale":"2024-09-21T10:02:41.635251+0000","last_undegraded":"2024-09-21T10:02:41.635251+0000","last_fullsized":"2024-09-21T10:02:41.635251+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:07.305638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":191,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.758339+0000","last_change":"2024-09-21T09:56:12.588992+0000","last_active":"2024-09-21T10:02:49.758339+0000","last_peered":"2024-09-21T10:02:49.758339+0000","last_clean":"2024-09-21T10:02:49.758339+0000","last_became_active":"2024-09-21T09:56:12.588665+0000","last_became_peered":"2024-09-21T09:56:12.588665+0000","last_unstale":"2024-09-21T10:02:49.758339+0000","last_undegraded":"2024-09-21T10:02:49.758339+0000","last_fullsized":"2024-09-21T10:02:49.758339+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:19:06.160456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":211,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.744880+0000","last_change":"2024-09-21T10:01:55.368329+0000","last_active":"2024-09-21T10:02:49.744880+0000","last_peered":"2024-09-21T10:02:49.744880+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.368329+0000","last_became_peered":"2024-09-21T10:01:55.368329+0000","last_unstale":"2024-09-21T10:02:49.744880+0000","last_undegraded":"2024-09-21T10:02:49.744880+0000","last_fullsized":"2024-09-21T10:01:55.358639+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:02:55.947353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.f","version":"167'8700","reported_seq":8734,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:57.551404+0000","last_change":"2024-09-21T10:00:39.617307+0000","last_active":"2024-09-21T10:02:57.551404+0000","last_peered":"2024-09-21T10:02:57.551404+0000","last_clean":"2024-09-21T10:02:57.551404+0000","last_became_active":"2024-09-21T10:00:39.617020+0000","last_became_peered":"2024-09-21T10:00:39.617020+0000","last_unstale":"2024-09-21T10:02:57.551404+0000","last_undegraded":"2024-09-21T10:02:57.551404+0000","last_fullsized":"2024-09-21T10:02:57.551404+0000","mapping_epoch":141,"log_start":"146'500","ondisk_log_start":"146'500","created":141,"last_epoch_clean":142,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":8200,"log_dups_size":0,"ondisk_log_size":8200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:21:32.333188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313786368,"num_objects":4788,"num_object_clones":0,"num_object_copies":14364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8700,"num_write_kb":431616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":149,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T10:02:41.638170+0000","last_change":"2024-09-21T09:56:12.583280+0000","last_active":"2024-09-21T10:02:41.638170+0000","last_peered":"2024-09-21T10:02:41.638170+0000","last_clean":"2024-09-21T10:02:41.638170+0000","last_became_active":"2024-09-21T09:56:12.583007+0000","last_became_peered":"2024-09-21T09:56:12.583007+0000","last_unstale":"2024-09-21T10:02:41.638170+0000","last_undegraded":"2024-09-21T10:02:41.638170+0000","last_fullsized":"2024-09-21T10:02:41.638170+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:26.991596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":236,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.758505+0000","last_change":"2024-09-21T10:01:17.367142+0000","last_active":"2024-09-21T10:02:49.758505+0000","last_peered":"2024-09-21T10:02:49.758505+0000","last_clean":"2024-09-21T10:01:12.331437+0000","last_became_active":"2024-09-21T10:01:17.367142+0000","last_became_peered":"2024-09-21T10:01:17.367142+0000","last_unstale":"2024-09-21T10:02:49.758505+0000","last_undegraded":"2024-09-21T10:02:49.758505+0000","last_fullsized":"2024-09-21T10:01:17.359249+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:46.792787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":169,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.759961+0000","last_change":"2024-09-21T09:50:45.501349+0000","last_active":"2024-09-21T10:02:49.759961+0000","last_peered":"2024-09-21T10:02:49.759961+0000","last_clean":"2024-09-21T10:02:49.759961+0000","last_became_active":"2024-09-21T09:50:45.501065+0000","last_became_peered":"2024-09-21T09:50:45.501065+0000","last_unstale":"2024-09-21T10:02:49.759961+0000","last_undegraded":"2024-09-21T10:02:49.759961+0000","last_fullsized":"2024-09-21T10:02:49.759961+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:28:58.570586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":211,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.746960+0000","last_change":"2024-09-21T10:01:55.370970+0000","last_active":"2024-09-21T10:02:49.746960+0000","last_peered":"2024-09-21T10:02:49.746960+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.370970+0000","last_became_peered":"2024-09-21T10:01:55.370970+0000","last_unstale":"2024-09-21T10:02:49.746960+0000","last_undegraded":"2024-09-21T10:02:49.746960+0000","last_fullsized":"2024-09-21T10:01:55.359846+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:03:11.964683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"167'8854","reported_seq":8903,"reported_epoch":167,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:57.240184+0000","last_change":"2024-09-21T10:01:18.378023+0000","last_active":"2024-09-21T10:02:57.240184+0000","last_peered":"2024-09-21T10:02:57.240184+0000","last_clean":"2024-09-21T10:02:57.240184+0000","last_became_active":"2024-09-21T10:01:18.375695+0000","last_became_peered":"2024-09-21T10:01:18.375695+0000","last_unstale":"2024-09-21T10:02:57.240184+0000","last_undegraded":"2024-09-21T10:02:57.240184+0000","last_fullsized":"2024-09-21T10:02:57.240184+0000","mapping_epoch":153,"log_start":"147'1332","ondisk_log_start":"147'1332","created":141,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7522,"log_dups_size":0,"ondisk_log_size":7522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:37:12.806595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313786368,"num_objects":4788,"num_object_clones":0,"num_object_copies":14364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4788,"num_objects_unfound":0,"num_objects_dirty":4788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8854,"num_write_kb":436544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,0],"avail_no_missing":["4","0","1"],"object_location_counts":[{"shards":"0,1,4","objects":4788}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":179,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.758419+0000","last_change":"2024-09-21T09:51:30.646531+0000","last_active":"2024-09-21T10:02:49.758419+0000","last_peered":"2024-09-21T10:02:49.758419+0000","last_clean":"2024-09-21T10:02:49.758419+0000","last_became_active":"2024-09-21T09:51:30.646248+0000","last_became_peered":"2024-09-21T09:51:30.646248+0000","last_unstale":"2024-09-21T10:02:49.758419+0000","last_undegraded":"2024-09-21T10:02:49.758419+0000","last_fullsized":"2024-09-21T10:02:49.758419+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:00.312924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":104,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.747165+0000","last_change":"2024-09-21T10:01:17.364179+0000","last_active":"2024-09-21T10:02:49.747165+0000","last_peered":"2024-09-21T10:02:49.747165+0000","last_clean":"2024-09-21T10:00:52.397528+0000","last_became_active":"2024-09-21T10:01:17.364179+0000","last_became_peered":"2024-09-21T10:01:17.364179+0000","last_unstale":"2024-09-21T10:02:49.747165+0000","last_undegraded":"2024-09-21T10:02:49.747165+0000","last_fullsized":"2024-09-21T10:01:17.360621+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:26:17.268991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":217,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.747106+0000","last_change":"2024-09-21T10:01:17.371359+0000","last_active":"2024-09-21T10:02:49.747106+0000","last_peered":"2024-09-21T10:02:49.747106+0000","last_clean":"2024-09-21T10:01:12.347295+0000","last_became_active":"2024-09-21T10:01:17.371359+0000","last_became_peered":"2024-09-21T10:01:17.371359+0000","last_unstale":"2024-09-21T10:02:49.747106+0000","last_undegraded":"2024-09-21T10:02:49.747106+0000","last_fullsized":"2024-09-21T10:01:17.366525+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:38.963323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":211,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.746965+0000","last_change":"2024-09-21T10:01:55.373699+0000","last_active":"2024-09-21T10:02:49.746965+0000","last_peered":"2024-09-21T10:02:49.746965+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.373699+0000","last_became_peered":"2024-09-21T10:01:55.373699+0000","last_unstale":"2024-09-21T10:02:49.746965+0000","last_undegraded":"2024-09-21T10:02:49.746965+0000","last_fullsized":"2024-09-21T10:01:55.366377+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:03:18.060759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"167'9332","reported_seq":10244,"reported_epoch":167,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:59.777607+0000","last_change":"2024-09-21T10:01:29.079779+0000","last_active":"2024-09-21T10:02:59.777607+0000","last_peered":"2024-09-21T10:02:59.777607+0000","last_clean":"2024-09-21T10:02:59.777607+0000","last_became_active":"2024-09-21T10:01:18.393943+0000","last_became_peered":"2024-09-21T10:01:18.393943+0000","last_unstale":"2024-09-21T10:02:59.777607+0000","last_undegraded":"2024-09-21T10:02:59.777607+0000","last_fullsized":"2024-09-21T10:02:59.777607+0000","mapping_epoch":153,"log_start":"151'2985","ondisk_log_start":"151'2985","created":141,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":6347,"log_dups_size":0,"ondisk_log_size":6347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:16.177065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307298328,"num_objects":4690,"num_object_clones":0,"num_object_copies":14070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4690,"num_objects_unfound":0,"num_objects_dirty":4690,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9332,"num_write_kb":448641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":418,"num_bytes_recovered":27394048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5,4],"avail_no_missing":["1","4","5"],"object_location_counts":[{"shards":"1,4,5","objects":4690}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":173,"reported_epoch":167,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:02:49.758398+0000","last_change":"2024-09-21T10:01:17.363383+0000","last_active":"2024-09-21T10:02:49.758398+0000","last_peered":"2024-09-21T10:02:49.758398+0000","last_clean":"2024-09-21T10:01:12.338791+0000","last_became_active":"2024-09-21T10:01:17.363383+0000","last_became_peered":"2024-09-21T10:01:17.363383+0000","last_unstale":"2024-09-21T10:02:49.758398+0000","last_undegraded":"2024-09-21T10:01:17.354694+0000","last_fullsized":"2024-09-21T10:01:17.354594+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:24:08.763448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":179,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.761092+0000","last_change":"2024-09-21T09:50:45.513017+0000","last_active":"2024-09-21T10:02:49.761092+0000","last_peered":"2024-09-21T10:02:49.761092+0000","last_clean":"2024-09-21T10:02:49.761092+0000","last_became_active":"2024-09-21T09:50:45.512629+0000","last_became_peered":"2024-09-21T09:50:45.512629+0000","last_unstale":"2024-09-21T10:02:49.761092+0000","last_undegraded":"2024-09-21T10:02:49.761092+0000","last_fullsized":"2024-09-21T10:02:49.761092+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:47:08.421575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":162,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.735696+0000","last_change":"2024-09-21T10:01:17.357067+0000","last_active":"2024-09-21T10:02:49.735696+0000","last_peered":"2024-09-21T10:02:49.735696+0000","last_clean":"2024-09-21T10:01:12.337480+0000","last_became_active":"2024-09-21T10:01:17.357067+0000","last_became_peered":"2024-09-21T10:01:17.357067+0000","last_unstale":"2024-09-21T10:02:49.735696+0000","last_undegraded":"2024-09-21T10:02:49.735696+0000","last_fullsized":"2024-09-21T10:01:17.351626+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:21.390162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":211,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.746829+0000","last_change":"2024-09-21T10:01:55.374579+0000","last_active":"2024-09-21T10:02:49.746829+0000","last_peered":"2024-09-21T10:02:49.746829+0000","last_clean":"2024-09-21T10:00:52.397657+0000","last_became_active":"2024-09-21T10:01:55.374579+0000","last_became_peered":"2024-09-21T10:01:55.374579+0000","last_unstale":"2024-09-21T10:02:49.746829+0000","last_undegraded":"2024-09-21T10:02:49.746829+0000","last_fullsized":"2024-09-21T10:01:55.359516+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T09:37:09.076441+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:03:24.128110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":107340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"167'8962","reported_seq":9919,"reported_epoch":167,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:55.804116+0000","last_change":"2024-09-21T10:01:35.790770+0000","last_active":"2024-09-21T10:02:55.804116+0000","last_peered":"2024-09-21T10:02:55.804116+0000","last_clean":"2024-09-21T10:02:55.804116+0000","last_became_active":"2024-09-21T10:01:18.395798+0000","last_became_peered":"2024-09-21T10:01:18.395798+0000","last_unstale":"2024-09-21T10:02:55.804116+0000","last_undegraded":"2024-09-21T10:02:55.804116+0000","last_fullsized":"2024-09-21T10:02:55.804116+0000","mapping_epoch":153,"log_start":"150'2615","ondisk_log_start":"150'2615","created":141,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":6347,"log_dups_size":0,"ondisk_log_size":6347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:43:47.890038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330039296,"num_objects":5036,"num_object_clones":0,"num_object_copies":15108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5036,"num_objects_unfound":0,"num_objects_dirty":5036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8962,"num_write_kb":447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":442,"num_bytes_recovered":28966912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,0],"avail_no_missing":["3","0","4"],"object_location_counts":[{"shards":"0,3,4","objects":5036}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":213,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.747379+0000","last_change":"2024-09-21T09:50:45.510812+0000","last_active":"2024-09-21T10:02:49.747379+0000","last_peered":"2024-09-21T10:02:49.747379+0000","last_clean":"2024-09-21T10:02:49.747379+0000","last_became_active":"2024-09-21T09:50:45.510480+0000","last_became_peered":"2024-09-21T09:50:45.510480+0000","last_unstale":"2024-09-21T10:02:49.747379+0000","last_undegraded":"2024-09-21T10:02:49.747379+0000","last_fullsized":"2024-09-21T10:02:49.747379+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:05.408194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":170,"reported_epoch":166,"state":"active+undersized","last_fresh":"2024-09-21T10:02:41.647595+0000","last_change":"2024-09-21T10:01:17.357896+0000","last_active":"2024-09-21T10:02:41.647595+0000","last_peered":"2024-09-21T10:02:41.647595+0000","last_clean":"2024-09-21T10:01:12.345631+0000","last_became_active":"2024-09-21T10:01:17.357896+0000","last_became_peered":"2024-09-21T10:01:17.357896+0000","last_unstale":"2024-09-21T10:02:41.647595+0000","last_undegraded":"2024-09-21T10:02:41.647595+0000","last_fullsized":"2024-09-21T10:01:17.354502+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:45.683879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":113,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T10:02:41.635067+0000","last_change":"2024-09-21T09:40:49.190763+0000","last_active":"2024-09-21T10:02:41.635067+0000","last_peered":"2024-09-21T10:02:41.635067+0000","last_clean":"2024-09-21T10:02:41.635067+0000","last_became_active":"2024-09-21T09:40:49.190463+0000","last_became_peered":"2024-09-21T09:40:49.190463+0000","last_unstale":"2024-09-21T10:02:41.635067+0000","last_undegraded":"2024-09-21T10:02:41.635067+0000","last_fullsized":"2024-09-21T10:02:41.635067+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:15.669404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":140,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T10:02:41.635144+0000","last_change":"2024-09-21T09:51:30.642074+0000","last_active":"2024-09-21T10:02:41.635144+0000","last_peered":"2024-09-21T10:02:41.635144+0000","last_clean":"2024-09-21T10:02:41.635144+0000","last_became_active":"2024-09-21T09:51:30.641176+0000","last_became_peered":"2024-09-21T09:51:30.641176+0000","last_unstale":"2024-09-21T10:02:41.635144+0000","last_undegraded":"2024-09-21T10:02:41.635144+0000","last_fullsized":"2024-09-21T10:02:41.635144+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:18.766865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":184,"reported_epoch":166,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:02:41.635391+0000","last_change":"2024-09-21T10:01:17.359126+0000","last_active":"2024-09-21T10:02:41.635391+0000","last_peered":"2024-09-21T10:02:41.635391+0000","last_clean":"2024-09-21T10:01:12.333760+0000","last_became_active":"2024-09-21T10:01:17.359126+0000","last_became_peered":"2024-09-21T10:01:17.359126+0000","last_unstale":"2024-09-21T10:02:41.635391+0000","last_undegraded":"2024-09-21T10:01:17.353723+0000","last_fullsized":"2024-09-21T10:01:17.353628+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:19.570827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":180,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.759805+0000","last_change":"2024-09-21T09:51:30.642334+0000","last_active":"2024-09-21T10:02:49.759805+0000","last_peered":"2024-09-21T10:02:49.759805+0000","last_clean":"2024-09-21T10:02:49.759805+0000","last_became_active":"2024-09-21T09:51:30.641267+0000","last_became_peered":"2024-09-21T09:51:30.641267+0000","last_unstale":"2024-09-21T10:02:49.759805+0000","last_undegraded":"2024-09-21T10:02:49.759805+0000","last_fullsized":"2024-09-21T10:02:49.759805+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:24.201971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":70,"reported_epoch":167,"state":"active+undersized+degraded","last_fresh":"2024-09-21T10:02:49.738839+0000","last_change":"2024-09-21T10:01:17.357199+0000","last_active":"2024-09-21T10:02:49.738839+0000","last_peered":"2024-09-21T10:02:49.738839+0000","last_clean":"2024-09-21T10:00:52.399787+0000","last_became_active":"2024-09-21T10:01:17.357199+0000","last_became_peered":"2024-09-21T10:01:17.357199+0000","last_unstale":"2024-09-21T10:02:49.738839+0000","last_undegraded":"2024-09-21T10:01:17.352030+0000","last_fullsized":"2024-09-21T10:01:17.351935+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:34.172159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":159,"reported_epoch":166,"state":"active+undersized","last_fresh":"2024-09-21T10:02:41.638015+0000","last_change":"2024-09-21T10:01:17.372819+0000","last_active":"2024-09-21T10:02:41.638015+0000","last_peered":"2024-09-21T10:02:41.638015+0000","last_clean":"2024-09-21T10:01:12.334093+0000","last_became_active":"2024-09-21T10:01:17.372819+0000","last_became_peered":"2024-09-21T10:01:17.372819+0000","last_unstale":"2024-09-21T10:02:41.638015+0000","last_undegraded":"2024-09-21T10:02:41.638015+0000","last_fullsized":"2024-09-21T10:01:17.355655+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:03:49.178062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":166,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.737845+0000","last_change":"2024-09-21T10:01:17.368801+0000","last_active":"2024-09-21T10:02:49.737845+0000","last_peered":"2024-09-21T10:02:49.737845+0000","last_clean":"2024-09-21T10:01:12.340026+0000","last_became_active":"2024-09-21T10:01:17.368801+0000","last_became_peered":"2024-09-21T10:01:17.368801+0000","last_unstale":"2024-09-21T10:02:49.737845+0000","last_undegraded":"2024-09-21T10:02:49.737845+0000","last_fullsized":"2024-09-21T10:01:17.353280+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:23.625415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":208,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.747016+0000","last_change":"2024-09-21T09:56:12.574472+0000","last_active":"2024-09-21T10:02:49.747016+0000","last_peered":"2024-09-21T10:02:49.747016+0000","last_clean":"2024-09-21T10:02:49.747016+0000","last_became_active":"2024-09-21T09:56:12.573966+0000","last_became_peered":"2024-09-21T09:56:12.573966+0000","last_unstale":"2024-09-21T10:02:49.747016+0000","last_undegraded":"2024-09-21T10:02:49.747016+0000","last_fullsized":"2024-09-21T10:02:49.747016+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:37:21.415933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":158,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.735476+0000","last_change":"2024-09-21T09:56:12.576756+0000","last_active":"2024-09-21T10:02:49.735476+0000","last_peered":"2024-09-21T10:02:49.735476+0000","last_clean":"2024-09-21T10:02:49.735476+0000","last_became_active":"2024-09-21T09:56:12.575752+0000","last_became_peered":"2024-09-21T09:56:12.575752+0000","last_unstale":"2024-09-21T10:02:49.735476+0000","last_undegraded":"2024-09-21T10:02:49.735476+0000","last_fullsized":"2024-09-21T10:02:49.735476+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:55.224839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.12","version":"167'8625","reported_seq":8697,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:55.760667+0000","last_change":"2024-09-21T10:02:39.673885+0000","last_active":"2024-09-21T10:02:55.760667+0000","last_peered":"2024-09-21T10:02:55.760667+0000","last_clean":"2024-09-21T10:02:55.760667+0000","last_became_active":"2024-09-21T10:02:39.665024+0000","last_became_peered":"2024-09-21T10:02:39.665024+0000","last_unstale":"2024-09-21T10:02:55.760667+0000","last_undegraded":"2024-09-21T10:02:55.760667+0000","last_fullsized":"2024-09-21T10:02:55.760667+0000","mapping_epoch":163,"log_start":"147'800","ondisk_log_start":"147'800","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7825,"log_dups_size":0,"ondisk_log_size":7825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165150720,"num_objects":2520,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4563,"num_write_kb":226656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":130,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T10:02:41.647525+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T10:02:41.647525+0000","last_peered":"2024-09-21T10:02:41.647525+0000","last_clean":"2024-09-21T10:02:41.647525+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T10:02:41.647525+0000","last_undegraded":"2024-09-21T10:02:41.647525+0000","last_fullsized":"2024-09-21T10:02:41.647525+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":176,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.735960+0000","last_change":"2024-09-21T09:56:12.571495+0000","last_active":"2024-09-21T10:02:49.735960+0000","last_peered":"2024-09-21T10:02:49.735960+0000","last_clean":"2024-09-21T10:02:49.735960+0000","last_became_active":"2024-09-21T09:56:12.571200+0000","last_became_peered":"2024-09-21T09:56:12.571200+0000","last_unstale":"2024-09-21T10:02:49.735960+0000","last_undegraded":"2024-09-21T10:02:49.735960+0000","last_fullsized":"2024-09-21T10:02:49.735960+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:53:58.851300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"167'8235","reported_seq":9140,"reported_epoch":167,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:55.784002+0000","last_change":"2024-09-21T10:02:41.677527+0000","last_active":"2024-09-21T10:02:55.784002+0000","last_peered":"2024-09-21T10:02:55.784002+0000","last_clean":"2024-09-21T10:02:55.784002+0000","last_became_active":"2024-09-21T10:02:41.652189+0000","last_became_peered":"2024-09-21T10:02:41.652189+0000","last_unstale":"2024-09-21T10:02:55.784002+0000","last_undegraded":"2024-09-21T10:02:55.784002+0000","last_fullsized":"2024-09-21T10:02:55.784002+0000","mapping_epoch":165,"log_start":"146'400","ondisk_log_start":"146'400","created":163,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7835,"log_dups_size":0,"ondisk_log_size":7835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160071680,"num_objects":2443,"num_object_clones":0,"num_object_copies":7329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2443,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4330,"num_write_kb":216704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":13533184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,5],"avail_no_missing":["3","1","5"],"object_location_counts":[{"shards":"1,3,5","objects":2443}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":229,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.747310+0000","last_change":"2024-09-21T10:01:17.363986+0000","last_active":"2024-09-21T10:02:49.747310+0000","last_peered":"2024-09-21T10:02:49.747310+0000","last_clean":"2024-09-21T10:01:12.350477+0000","last_became_active":"2024-09-21T10:01:17.363986+0000","last_became_peered":"2024-09-21T10:01:17.363986+0000","last_unstale":"2024-09-21T10:02:49.747310+0000","last_undegraded":"2024-09-21T10:02:49.747310+0000","last_fullsized":"2024-09-21T10:01:17.354948+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:29:32.179855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":179,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.742686+0000","last_change":"2024-09-21T09:56:12.580276+0000","last_active":"2024-09-21T10:02:49.742686+0000","last_peered":"2024-09-21T10:02:49.742686+0000","last_clean":"2024-09-21T10:02:49.742686+0000","last_became_active":"2024-09-21T09:56:12.579989+0000","last_became_peered":"2024-09-21T09:56:12.579989+0000","last_unstale":"2024-09-21T10:02:49.742686+0000","last_undegraded":"2024-09-21T10:02:49.742686+0000","last_fullsized":"2024-09-21T10:02:49.742686+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:14:56.361873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"167'8627","reported_seq":8702,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:59.736652+0000","last_change":"2024-09-21T10:02:39.682108+0000","last_active":"2024-09-21T10:02:59.736652+0000","last_peered":"2024-09-21T10:02:59.736652+0000","last_clean":"2024-09-21T10:02:59.736652+0000","last_became_active":"2024-09-21T10:02:39.670320+0000","last_became_peered":"2024-09-21T10:02:39.670320+0000","last_unstale":"2024-09-21T10:02:59.736652+0000","last_undegraded":"2024-09-21T10:02:59.736652+0000","last_fullsized":"2024-09-21T10:02:59.736652+0000","mapping_epoch":163,"log_start":"147'800","ondisk_log_start":"147'800","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7827,"log_dups_size":0,"ondisk_log_size":7827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:03:55.853691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152666112,"num_objects":2330,"num_object_clones":0,"num_object_copies":6990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4631,"num_write_kb":222720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":158,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.737430+0000","last_change":"2024-09-21T09:56:12.582383+0000","last_active":"2024-09-21T10:02:49.737430+0000","last_peered":"2024-09-21T10:02:49.737430+0000","last_clean":"2024-09-21T10:02:49.737430+0000","last_became_active":"2024-09-21T09:56:12.581427+0000","last_became_peered":"2024-09-21T09:56:12.581427+0000","last_unstale":"2024-09-21T10:02:49.737430+0000","last_undegraded":"2024-09-21T10:02:49.737430+0000","last_fullsized":"2024-09-21T10:02:49.737430+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:24.275633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":149,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T10:02:41.634987+0000","last_change":"2024-09-21T09:51:30.645644+0000","last_active":"2024-09-21T10:02:41.634987+0000","last_peered":"2024-09-21T10:02:41.634987+0000","last_clean":"2024-09-21T10:02:41.634987+0000","last_became_active":"2024-09-21T09:51:30.644883+0000","last_became_peered":"2024-09-21T09:51:30.644883+0000","last_unstale":"2024-09-21T10:02:41.634987+0000","last_undegraded":"2024-09-21T10:02:41.634987+0000","last_fullsized":"2024-09-21T10:02:41.634987+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:39.331936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"167'8330","reported_seq":9299,"reported_epoch":167,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:55.767508+0000","last_change":"2024-09-21T10:02:41.682769+0000","last_active":"2024-09-21T10:02:55.767508+0000","last_peered":"2024-09-21T10:02:55.767508+0000","last_clean":"2024-09-21T10:02:55.767508+0000","last_became_active":"2024-09-21T10:02:41.650214+0000","last_became_peered":"2024-09-21T10:02:41.650214+0000","last_unstale":"2024-09-21T10:02:55.767508+0000","last_undegraded":"2024-09-21T10:02:55.767508+0000","last_fullsized":"2024-09-21T10:02:55.767508+0000","mapping_epoch":165,"log_start":"146'500","ondisk_log_start":"146'500","created":163,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7830,"log_dups_size":0,"ondisk_log_size":7830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160268288,"num_objects":2446,"num_object_clones":0,"num_object_copies":7338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2446,"num_objects_unfound":0,"num_objects_dirty":2446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4408,"num_write_kb":219296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":14385152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":2446}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":212,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.736272+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:02:49.736272+0000","last_peered":"2024-09-21T10:02:49.736272+0000","last_clean":"2024-09-21T10:02:49.736272+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:02:49.736272+0000","last_undegraded":"2024-09-21T10:02:49.736272+0000","last_fullsized":"2024-09-21T10:02:49.736272+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":219,"reported_epoch":167,"state":"active+undersized","last_fresh":"2024-09-21T10:02:49.747248+0000","last_change":"2024-09-21T10:01:17.367707+0000","last_active":"2024-09-21T10:02:49.747248+0000","last_peered":"2024-09-21T10:02:49.747248+0000","last_clean":"2024-09-21T10:01:12.346791+0000","last_became_active":"2024-09-21T10:01:17.367707+0000","last_became_peered":"2024-09-21T10:01:17.367707+0000","last_unstale":"2024-09-21T10:02:49.747248+0000","last_undegraded":"2024-09-21T10:02:49.747248+0000","last_fullsized":"2024-09-21T10:01:17.354751+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:58:00.301582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.16","version":"167'8320","reported_seq":8484,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:57.602511+0000","last_change":"2024-09-21T10:02:48.229282+0000","last_active":"2024-09-21T10:02:57.602511+0000","last_peered":"2024-09-21T10:02:57.602511+0000","last_clean":"2024-09-21T10:02:57.602511+0000","last_became_active":"2024-09-21T10:02:39.675302+0000","last_became_peered":"2024-09-21T10:02:39.675302+0000","last_unstale":"2024-09-21T10:02:57.602511+0000","last_undegraded":"2024-09-21T10:02:57.602511+0000","last_fullsized":"2024-09-21T10:02:57.602511+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":5,"last_scrub":"166'8003","last_scrub_stamp":"2024-09-21T10:02:48.229225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:02:48.229225+0000","objects_scrubbed":2783,"log_size":8320,"log_dups_size":0,"ondisk_log_size":8320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:00:22.419359+0000","scrub_duration":0.88147588499999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161153024,"num_objects":2459,"num_object_clones":0,"num_object_copies":7377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":324,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":147,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T10:02:41.635017+0000","last_change":"2024-09-21T09:56:12.576148+0000","last_active":"2024-09-21T10:02:41.635017+0000","last_peered":"2024-09-21T10:02:41.635017+0000","last_clean":"2024-09-21T10:02:41.635017+0000","last_became_active":"2024-09-21T09:56:12.575813+0000","last_became_peered":"2024-09-21T09:56:12.575813+0000","last_unstale":"2024-09-21T10:02:41.635017+0000","last_undegraded":"2024-09-21T10:02:41.635017+0000","last_fullsized":"2024-09-21T10:02:41.635017+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:53:01.367816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":134,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T10:02:41.637775+0000","last_change":"2024-09-21T09:50:45.502447+0000","last_active":"2024-09-21T10:02:41.637775+0000","last_peered":"2024-09-21T10:02:41.637775+0000","last_clean":"2024-09-21T10:02:41.637775+0000","last_became_active":"2024-09-21T09:50:45.500800+0000","last_became_peered":"2024-09-21T09:50:45.500800+0000","last_unstale":"2024-09-21T10:02:41.637775+0000","last_undegraded":"2024-09-21T10:02:41.637775+0000","last_fullsized":"2024-09-21T10:02:41.637775+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:20:51.819410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.17","version":"167'8398","reported_seq":9822,"reported_epoch":167,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:57.575113+0000","last_change":"2024-09-21T10:02:41.649949+0000","last_active":"2024-09-21T10:02:57.575113+0000","last_peered":"2024-09-21T10:02:57.575113+0000","last_clean":"2024-09-21T10:02:57.575113+0000","last_became_active":"2024-09-21T10:02:41.642622+0000","last_became_peered":"2024-09-21T10:02:41.642622+0000","last_unstale":"2024-09-21T10:02:57.575113+0000","last_undegraded":"2024-09-21T10:02:57.575113+0000","last_fullsized":"2024-09-21T10:02:57.575113+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":163,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":8398,"log_dups_size":0,"ondisk_log_size":8398,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157450240,"num_objects":2403,"num_object_clones":0,"num_object_copies":7209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2403,"num_objects_unfound":0,"num_objects_dirty":2403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4463,"num_write_kb":219680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":336,"num_bytes_recovered":21987328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3,2],"avail_no_missing":["5","2","3"],"object_location_counts":[{"shards":"2,3,5","objects":2403}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":147,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T10:02:41.634959+0000","last_change":"2024-09-21T09:56:12.576457+0000","last_active":"2024-09-21T10:02:41.634959+0000","last_peered":"2024-09-21T10:02:41.634959+0000","last_clean":"2024-09-21T10:02:41.634959+0000","last_became_active":"2024-09-21T09:56:12.576061+0000","last_became_peered":"2024-09-21T09:56:12.576061+0000","last_unstale":"2024-09-21T10:02:41.634959+0000","last_undegraded":"2024-09-21T10:02:41.634959+0000","last_fullsized":"2024-09-21T10:02:41.634959+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:51:46.676699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":117,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T10:02:41.638163+0000","last_change":"2024-09-21T09:40:49.094916+0000","last_active":"2024-09-21T10:02:41.638163+0000","last_peered":"2024-09-21T10:02:41.638163+0000","last_clean":"2024-09-21T10:02:41.638163+0000","last_became_active":"2024-09-21T09:40:49.094264+0000","last_became_peered":"2024-09-21T09:40:49.094264+0000","last_unstale":"2024-09-21T10:02:41.638163+0000","last_undegraded":"2024-09-21T10:02:41.638163+0000","last_fullsized":"2024-09-21T10:02:41.638163+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:53.663226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.14","version":"167'8432","reported_seq":8504,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:59.781713+0000","last_change":"2024-09-21T10:02:39.675915+0000","last_active":"2024-09-21T10:02:59.781713+0000","last_peered":"2024-09-21T10:02:59.781713+0000","last_clean":"2024-09-21T10:02:59.781713+0000","last_became_active":"2024-09-21T10:02:39.666254+0000","last_became_peered":"2024-09-21T10:02:39.666254+0000","last_unstale":"2024-09-21T10:02:59.781713+0000","last_undegraded":"2024-09-21T10:02:59.781713+0000","last_fullsized":"2024-09-21T10:02:59.781713+0000","mapping_epoch":163,"log_start":"147'700","ondisk_log_start":"147'700","created":163,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7732,"log_dups_size":0,"ondisk_log_size":7732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:03:53.782451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154697728,"num_objects":2361,"num_object_clones":0,"num_object_copies":7083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4500,"num_write_kb":219520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":184,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.738320+0000","last_change":"2024-09-21T09:56:12.577329+0000","last_active":"2024-09-21T10:02:49.738320+0000","last_peered":"2024-09-21T10:02:49.738320+0000","last_clean":"2024-09-21T10:02:49.738320+0000","last_became_active":"2024-09-21T09:56:12.576941+0000","last_became_peered":"2024-09-21T09:56:12.576941+0000","last_unstale":"2024-09-21T10:02:49.738320+0000","last_undegraded":"2024-09-21T10:02:49.738320+0000","last_fullsized":"2024-09-21T10:02:49.738320+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:35.235768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":170,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.738679+0000","last_change":"2024-09-21T09:56:12.577137+0000","last_active":"2024-09-21T10:02:49.738679+0000","last_peered":"2024-09-21T10:02:49.738679+0000","last_clean":"2024-09-21T10:02:49.738679+0000","last_became_active":"2024-09-21T09:56:12.576668+0000","last_became_peered":"2024-09-21T09:56:12.576668+0000","last_unstale":"2024-09-21T10:02:49.738679+0000","last_undegraded":"2024-09-21T10:02:49.738679+0000","last_fullsized":"2024-09-21T10:02:49.738679+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:56.772427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.15","version":"167'8475","reported_seq":9479,"reported_epoch":167,"state":"active+clean+remapped","last_fresh":"2024-09-21T10:02:58.977785+0000","last_change":"2024-09-21T10:02:41.670264+0000","last_active":"2024-09-21T10:02:58.977785+0000","last_peered":"2024-09-21T10:02:58.977785+0000","last_clean":"2024-09-21T10:02:58.977785+0000","last_became_active":"2024-09-21T10:02:41.657859+0000","last_became_peered":"2024-09-21T10:02:41.657859+0000","last_unstale":"2024-09-21T10:02:58.977785+0000","last_undegraded":"2024-09-21T10:02:58.977785+0000","last_fullsized":"2024-09-21T10:02:58.977785+0000","mapping_epoch":165,"log_start":"147'700","ondisk_log_start":"147'700","created":163,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:00:38.597842+0000","last_clean_scrub_stamp":"2024-09-21T10:00:38.597842+0000","objects_scrubbed":0,"log_size":7775,"log_dups_size":0,"ondisk_log_size":7775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:03:51.455345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153911296,"num_objects":2349,"num_object_clones":0,"num_object_copies":7047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2349,"num_objects_unfound":0,"num_objects_dirty":2349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4546,"num_write_kb":220608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":229,"num_bytes_recovered":15007744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,1],"avail_no_missing":["0","1","5"],"object_location_counts":[{"shards":"0,1,5","objects":2349}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":130,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T10:02:41.634802+0000","last_change":"2024-09-21T09:50:45.502819+0000","last_active":"2024-09-21T10:02:41.634802+0000","last_peered":"2024-09-21T10:02:41.634802+0000","last_clean":"2024-09-21T10:02:41.634802+0000","last_became_active":"2024-09-21T09:50:45.502483+0000","last_became_peered":"2024-09-21T09:50:45.502483+0000","last_unstale":"2024-09-21T10:02:41.634802+0000","last_undegraded":"2024-09-21T10:02:41.634802+0000","last_fullsized":"2024-09-21T10:02:41.634802+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:26.549958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":213,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T10:02:49.746699+0000","last_change":"2024-09-21T09:51:30.650116+0000","last_active":"2024-09-21T10:02:49.746699+0000","last_peered":"2024-09-21T10:02:49.746699+0000","last_clean":"2024-09-21T10:02:49.746699+0000","last_became_active":"2024-09-21T09:51:30.649798+0000","last_became_peered":"2024-09-21T09:51:30.649798+0000","last_unstale":"2024-09-21T10:02:49.746699+0000","last_undegraded":"2024-09-21T10:02:49.746699+0000","last_fullsized":"2024-09-21T10:02:49.746699+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:40.222590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":26,"stat_sum":{"num_bytes":5009506328,"num_objects":76441,"num_object_clones":0,"num_object_copies":229323,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":43000,"num_objects_unfound":0,"num_objects_dirty":76441,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111188,"num_write_kb":5278913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2711,"num_bytes_recovered":177438720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15041179648,"data_stored":15041167432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":201903,"ondisk_log_size":201903,"up":64,"acting":78,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":85,"acting":85,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":9704,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13748,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":83,"acting":83,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":19,"acting":19,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":292,"num_read_kb":747,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":22,"acting":22,"num_store_stats":4}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626727,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1521864,"kb_used_data":927808,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":92223288,"statfs":{"total":95995035648,"available":94436646912,"internally_reserved":0,"allocated":950075392,"data_stored":949206709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019185,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2447884,"kb_used_data":1561092,"kb_used_omap":1,"kb_used_meta":886782,"kb_avail":91297268,"statfs":{"total":95995035648,"available":93488402432,"internally_reserved":0,"allocated":1598558208,"data_stored":1597684966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":908065112},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":81,"seq":347892351147,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2340476,"kb_used_data":1815028,"kb_used_omap":5,"kb_used_meta":525370,"kb_avail":91404676,"statfs":{"total":95995035648,"available":93598388224,"internally_reserved":0,"allocated":1858588672,"data_stored":1857728072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":537978940},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430478,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":673476,"kb_used_data":75260,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":93071676,"statfs":{"total":95995035648,"available":95305396224,"internally_reserved":0,"allocated":77066240,"data_stored":76315786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822956,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3968248,"kb_used_data":3058992,"kb_used_omap":1,"kb_used_meta":909246,"kb_avail":89776904,"statfs":{"total":95995035648,"available":91931549696,"internally_reserved":0,"allocated":3132407808,"data_stored":3131550193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931068248},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879379,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3576372,"kb_used_data":3168492,"kb_used_omap":4,"kb_used_meta":407867,"kb_avail":90168780,"statfs":{"total":95995035648,"available":92332830720,"internally_reserved":0,"allocated":3244535808,"data_stored":3243682338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":417656252},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566220,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1605624,"kb_used_data":1215984,"kb_used_omap":6,"kb_used_meta":389625,"kb_avail":92139528,"statfs":{"total":95995035648,"available":94350876672,"internally_reserved":0,"allocated":1245167616,"data_stored":1244310572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":398976423},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860565,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3498292,"kb_used_data":2965612,"kb_used_omap":5,"kb_used_meta":532666,"kb_avail":90246860,"statfs":{"total":95995035648,"available":92412784640,"internally_reserved":0,"allocated":3036786688,"data_stored":3035932856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":545450223},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1838678016,"data_stored":1838678016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3033796608,"data_stored":3033792536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1226964992,"data_stored":1226964992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3213754368,"data_stored":3213754368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1585254400,"data_stored":1585250328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3118206976,"data_stored":3118202904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":947191808,"data_stored":947191808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":77332480,"data_stored":77332480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:03:02.522 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T10:03:02.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pg_num 21 2024-09-21T10:03:02.533 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:02.630 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:02.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:02.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:02.732 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:02.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:02.781 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:02.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:02.834 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:03:02.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:02.929 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:02.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:02.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:03.031 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:03.082 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:03.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:03.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:03.154 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:03:03.185 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:03.229 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:03.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:03.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:03.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:03.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:03 smithi107 ceph-mon[102221]: pgmap v593: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 15/224127 objects degraded (0.007%); 41952/224127 objects misplaced (18.718%) 2024-09-21T10:03:03.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:03 smithi107 ceph-mon[105022]: pgmap v593: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 15/224127 objects degraded (0.007%); 41952/224127 objects misplaced (18.718%) 2024-09-21T10:03:03.363 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:03.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:03.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:03.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:03:03.432 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:03 smithi129 ceph-mon[88471]: pgmap v593: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 666 op/s; 15/224127 objects degraded (0.007%); 41952/224127 objects misplaced (18.718%) 2024-09-21T10:03:03.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:03.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:03.514 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:03.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:03.615 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:03.661 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:03.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:03.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:03.764 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:03.807 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:03.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:03.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:03.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:03.947 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:03.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:03.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:04.049 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:04.071 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:04.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:04.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:04.173 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:04.193 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:04.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:04.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:04.295 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:04.320 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:04.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:04.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:04.422 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:04.447 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:04.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:04.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:04.549 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:04.576 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:04.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:04.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:04.677 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:04.711 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:04.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:04.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:04.812 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:04.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:04.851 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:04.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:04.937 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:03:04.937 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 20, 2024-09-21T10:03:04.937 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 600, 2024-09-21T10:03:04.937 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:03:04.937 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.937 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45320.0:91048 8.8 8:14c97e9c:::benchmark_data_smithi107_135256_object91047:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T10:03:04.937 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T09:56:11.577425+0000", 2024-09-21T10:03:04.937 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 413.35421974899998, 2024-09-21T10:03:04.937 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.2869700239999999, 2024-09-21T10:03:04.937 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:03:04.937 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:03:04.937 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:03:04.938 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45320", 2024-09-21T10:03:04.938 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:03:04.938 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 91048 2024-09-21T10:03:04.938 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.938 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:03:04.938 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.938 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:03:04.938 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.577425+0000", 2024-09-21T10:03:04.938 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:04.938 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.938 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.938 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:03:04.938 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.577425+0000", 2024-09-21T10:03:04.938 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:04.939 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.939 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.939 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:03:04.939 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.577424+0000", 2024-09-21T10:03:04.939 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:03:04.939 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.939 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.939 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:03:04.939 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.577487+0000", 2024-09-21T10:03:04.939 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2701999999999997e-05 2024-09-21T10:03:04.939 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.939 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.939 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:03:04.939 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.577501+0000", 2024-09-21T10:03:04.939 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4409e-05 2024-09-21T10:03:04.940 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.940 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.940 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:03:04.940 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.577507+0000", 2024-09-21T10:03:04.940 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5289999999999999e-06 2024-09-21T10:03:04.940 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.940 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.940 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:04.940 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.577562+0000", 2024-09-21T10:03:04.940 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5169e-05 2024-09-21T10:03:04.940 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.940 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.940 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:03:04.940 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.577570+0000", 2024-09-21T10:03:04.940 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.2919999999999996e-06 2024-09-21T10:03:04.941 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.941 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.941 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:04.941 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:12.576498+0000", 2024-09-21T10:03:04.941 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.99892736999999998 2024-09-21T10:03:04.941 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.941 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.941 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:03:04.941 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:12.576511+0000", 2024-09-21T10:03:04.941 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3871e-05 2024-09-21T10:03:04.941 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.941 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.941 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:04.941 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.852568+0000", 2024-09-21T10:03:04.941 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.27605678 2024-09-21T10:03:04.941 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.942 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.942 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:03:04.942 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.852719+0000", 2024-09-21T10:03:04.942 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015117099999999999 2024-09-21T10:03:04.942 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.942 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.942 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:03:04.942 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.852845+0000", 2024-09-21T10:03:04.942 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012582199999999999 2024-09-21T10:03:04.942 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.942 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.942 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:04.942 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.856876+0000", 2024-09-21T10:03:04.942 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0040305300000000001 2024-09-21T10:03:04.942 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.943 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.943 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:03:04.943 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.862820+0000", 2024-09-21T10:03:04.943 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0059439280000000002 2024-09-21T10:03:04.943 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.943 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.943 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:04.943 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.862871+0000", 2024-09-21T10:03:04.943 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1084000000000001e-05 2024-09-21T10:03:04.943 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.943 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.943 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:03:04.943 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.862930+0000", 2024-09-21T10:03:04.944 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9196e-05 2024-09-21T10:03:04.944 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.944 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.944 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:03:04.944 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.864395+0000", 2024-09-21T10:03:04.944 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0014652910000000001 2024-09-21T10:03:04.944 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:04.944 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:04.944 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:04.944 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.944 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.944 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45320.0:91049 8.8 8:1ce3c140:::benchmark_data_smithi107_135256_object91048:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T10:03:04.944 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T09:56:11.577871+0000", 2024-09-21T10:03:04.944 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 413.35377391499998, 2024-09-21T10:03:04.944 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.287433273, 2024-09-21T10:03:04.945 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:03:04.945 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:03:04.945 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:03:04.945 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45320", 2024-09-21T10:03:04.945 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:03:04.945 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 91049 2024-09-21T10:03:04.945 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.945 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:03:04.945 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.945 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:03:04.945 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.577871+0000", 2024-09-21T10:03:04.945 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:04.945 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.945 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.946 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:03:04.946 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.577871+0000", 2024-09-21T10:03:04.946 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:04.946 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.946 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.946 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:03:04.946 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.577870+0000", 2024-09-21T10:03:04.946 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:03:04.946 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.946 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.946 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:03:04.946 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.578145+0000", 2024-09-21T10:03:04.946 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000274493 2024-09-21T10:03:04.946 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.947 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.947 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:03:04.947 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.578159+0000", 2024-09-21T10:03:04.947 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4598e-05 2024-09-21T10:03:04.947 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.947 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.947 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:03:04.947 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.578165+0000", 2024-09-21T10:03:04.947 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3730000000000001e-06 2024-09-21T10:03:04.947 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.947 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.947 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:04.947 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.578211+0000", 2024-09-21T10:03:04.947 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6156999999999999e-05 2024-09-21T10:03:04.947 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.948 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.948 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:03:04.948 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.578218+0000", 2024-09-21T10:03:04.948 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3170000000000004e-06 2024-09-21T10:03:04.948 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.948 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.948 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:04.948 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:12.576541+0000", 2024-09-21T10:03:04.948 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.99832285399999998 2024-09-21T10:03:04.948 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.948 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.948 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:03:04.948 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:12.576548+0000", 2024-09-21T10:03:04.949 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9489999999999999e-06 2024-09-21T10:03:04.949 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.949 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.949 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:04.949 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.856938+0000", 2024-09-21T10:03:04.949 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.280390278 2024-09-21T10:03:04.949 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.949 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.949 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:03:04.949 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.857042+0000", 2024-09-21T10:03:04.949 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000103652 2024-09-21T10:03:04.949 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.949 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.949 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:03:04.950 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.857148+0000", 2024-09-21T10:03:04.950 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010624900000000001 2024-09-21T10:03:04.950 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.950 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.950 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:04.950 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.864453+0000", 2024-09-21T10:03:04.950 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0073053170000000004 2024-09-21T10:03:04.950 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.950 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.950 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:04.950 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.864486+0000", 2024-09-21T10:03:04.950 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2941000000000002e-05 2024-09-21T10:03:04.950 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.950 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.950 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:03:04.951 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.864524+0000", 2024-09-21T10:03:04.951 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7441000000000003e-05 2024-09-21T10:03:04.951 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.951 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.951 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:03:04.951 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.864559+0000", 2024-09-21T10:03:04.951 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4959999999999997e-05 2024-09-21T10:03:04.951 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.951 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.951 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:03:04.951 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.865304+0000", 2024-09-21T10:03:04.951 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00074546200000000001 2024-09-21T10:03:04.951 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:04.951 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:04.951 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:04.952 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.952 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.952 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45320.0:91073 8.8 8:12ba745f:::benchmark_data_smithi107_135256_object91072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T10:03:04.952 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T09:56:11.608544+0000", 2024-09-21T10:03:04.952 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 413.32310089999999, 2024-09-21T10:03:04.952 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.2816048840000001, 2024-09-21T10:03:04.952 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:03:04.952 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:03:04.952 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:03:04.952 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45320", 2024-09-21T10:03:04.952 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:03:04.952 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 91073 2024-09-21T10:03:04.952 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.952 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:03:04.952 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.953 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:03:04.953 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.608544+0000", 2024-09-21T10:03:04.953 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:04.953 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.953 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.953 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:03:04.953 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.608544+0000", 2024-09-21T10:03:04.953 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:04.953 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.953 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.953 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:03:04.953 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.608543+0000", 2024-09-21T10:03:04.953 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:03:04.953 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.953 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.954 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:03:04.954 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.608603+0000", 2024-09-21T10:03:04.954 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9265999999999998e-05 2024-09-21T10:03:04.954 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.954 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.954 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:03:04.954 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.608611+0000", 2024-09-21T10:03:04.954 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.5050000000000007e-06 2024-09-21T10:03:04.954 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.954 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.954 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:03:04.954 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.608615+0000", 2024-09-21T10:03:04.954 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4369999999999999e-06 2024-09-21T10:03:04.954 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.954 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.955 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:04.955 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.608651+0000", 2024-09-21T10:03:04.955 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6477000000000003e-05 2024-09-21T10:03:04.955 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.955 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.955 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:03:04.955 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:11.608657+0000", 2024-09-21T10:03:04.955 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9529999999999996e-06 2024-09-21T10:03:04.955 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.955 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.955 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:04.955 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:12.576574+0000", 2024-09-21T10:03:04.955 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.96791733199999996 2024-09-21T10:03:04.955 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.956 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.956 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:03:04.956 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:12.576581+0000", 2024-09-21T10:03:04.956 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6290000000000004e-06 2024-09-21T10:03:04.956 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.956 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.956 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:04.956 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.859834+0000", 2024-09-21T10:03:04.956 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.283252627 2024-09-21T10:03:04.956 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.956 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.956 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:03:04.956 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.859959+0000", 2024-09-21T10:03:04.956 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012550199999999999 2024-09-21T10:03:04.956 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.957 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.957 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:03:04.957 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.860127+0000", 2024-09-21T10:03:04.957 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000168266 2024-09-21T10:03:04.957 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.957 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.957 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:04.957 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.864512+0000", 2024-09-21T10:03:04.957 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.004384822 2024-09-21T10:03:04.957 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.957 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.957 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:03:04.957 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.865314+0000", 2024-09-21T10:03:04.957 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00080157099999999995 2024-09-21T10:03:04.957 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.958 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.958 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:04.958 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.888567+0000", 2024-09-21T10:03:04.958 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.023252919 2024-09-21T10:03:04.958 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.958 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.958 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:03:04.958 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.888631+0000", 2024-09-21T10:03:04.958 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3969999999999999e-05 2024-09-21T10:03:04.958 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.958 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.958 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:03:04.958 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:56:13.890149+0000", 2024-09-21T10:03:04.958 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0015182500000000001 2024-09-21T10:03:04.958 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:04.959 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:04.959 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:04.959 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.959 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.959 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32231 9.7 9:ecfb968d:::benchmark_data_smithi107_152010_object32230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:03:04.959 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.559414+0000", 2024-09-21T10:03:04.959 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 121.372231089, 2024-09-21T10:03:04.959 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23909994900000001, 2024-09-21T10:03:04.959 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:03:04.959 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:03:04.959 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:03:04.959 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:03:04.959 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:03:04.959 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32231 2024-09-21T10:03:04.959 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.959 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:03:04.960 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.960 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:03:04.960 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.559414+0000", 2024-09-21T10:03:04.960 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:04.960 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.960 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.960 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:03:04.960 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.559414+0000", 2024-09-21T10:03:04.960 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:04.960 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.960 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.960 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:03:04.960 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.559412+0000", 2024-09-21T10:03:04.960 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:03:04.960 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.960 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.961 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:03:04.961 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.560092+0000", 2024-09-21T10:03:04.961 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00067992900000000002 2024-09-21T10:03:04.961 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.961 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.961 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:03:04.961 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.560104+0000", 2024-09-21T10:03:04.961 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1887e-05 2024-09-21T10:03:04.961 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.961 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.961 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:03:04.961 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.560110+0000", 2024-09-21T10:03:04.961 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5740000000000003e-06 2024-09-21T10:03:04.961 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.961 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.961 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:04.962 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.560155+0000", 2024-09-21T10:03:04.962 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4913000000000002e-05 2024-09-21T10:03:04.962 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.962 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.962 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:03:04.962 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.560278+0000", 2024-09-21T10:03:04.962 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012316399999999999 2024-09-21T10:03:04.962 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.962 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.962 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:04.963 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:03:04.963 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.560401+0000", 2024-09-21T10:03:04.964 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012349299999999999 2024-09-21T10:03:04.964 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.964 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.964 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:03:04.964 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.562243+0000", 2024-09-21T10:03:04.964 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0018417520000000001 2024-09-21T10:03:04.964 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.964 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.964 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:04.964 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.563134+0000", 2024-09-21T10:03:04.965 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00089091400000000003 2024-09-21T10:03:04.965 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.965 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.965 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:04.965 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.798444+0000", 2024-09-21T10:03:04.965 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23531032700000001 2024-09-21T10:03:04.965 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.965 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.965 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:03:04.965 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.798486+0000", 2024-09-21T10:03:04.965 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1326999999999999e-05 2024-09-21T10:03:04.965 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.965 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.965 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:03:04.966 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.798514+0000", 2024-09-21T10:03:04.966 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.809e-05 2024-09-21T10:03:04.966 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:04.966 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:04.966 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:04.966 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.966 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.966 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32237 9.6 9:67022ea0:::benchmark_data_smithi107_152010_object32236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:03:04.966 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.561242+0000", 2024-09-21T10:03:04.966 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 121.37040329200001, 2024-09-21T10:03:04.966 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23891710999999999, 2024-09-21T10:03:04.966 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:03:04.966 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:03:04.966 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:03:04.967 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:03:04.967 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:03:04.967 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32237 2024-09-21T10:03:04.967 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.967 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:03:04.967 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.967 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:03:04.967 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561242+0000", 2024-09-21T10:03:04.967 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:04.967 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.967 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.967 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:03:04.967 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561242+0000", 2024-09-21T10:03:04.968 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:04.968 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.968 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.968 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:03:04.968 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561233+0000", 2024-09-21T10:03:04.968 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999914 2024-09-21T10:03:04.968 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.968 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.968 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:03:04.968 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561365+0000", 2024-09-21T10:03:04.968 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013210300000000001 2024-09-21T10:03:04.968 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.968 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.968 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:03:04.969 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561377+0000", 2024-09-21T10:03:04.969 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1426e-05 2024-09-21T10:03:04.969 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.969 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.969 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:03:04.969 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561382+0000", 2024-09-21T10:03:04.969 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4480000000000002e-06 2024-09-21T10:03:04.969 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.969 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.969 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:04.969 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561429+0000", 2024-09-21T10:03:04.969 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6761000000000002e-05 2024-09-21T10:03:04.969 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.969 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.969 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:03:04.970 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561536+0000", 2024-09-21T10:03:04.970 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000107021 2024-09-21T10:03:04.970 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.970 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.970 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:03:04.970 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561674+0000", 2024-09-21T10:03:04.970 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013777700000000001 2024-09-21T10:03:04.970 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.970 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.970 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:03:04.970 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.562828+0000", 2024-09-21T10:03:04.970 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0011543860000000001 2024-09-21T10:03:04.970 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.970 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.970 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:04.971 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564200+0000", 2024-09-21T10:03:04.971 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001372465 2024-09-21T10:03:04.971 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.971 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.971 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:04.971 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800061+0000", 2024-09-21T10:03:04.971 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.2358606 2024-09-21T10:03:04.971 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.971 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.971 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:03:04.971 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800127+0000", 2024-09-21T10:03:04.971 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6154000000000004e-05 2024-09-21T10:03:04.971 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.971 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.971 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:03:04.972 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800159+0000", 2024-09-21T10:03:04.972 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1534e-05 2024-09-21T10:03:04.972 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:04.972 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:04.972 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:04.972 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.972 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.972 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32243 9.6 9:68ed822c:::benchmark_data_smithi107_152010_object32242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:03:04.972 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.564332+0000", 2024-09-21T10:03:04.972 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 121.367313209, 2024-09-21T10:03:04.972 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23596514499999999, 2024-09-21T10:03:04.972 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:03:04.972 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:03:04.972 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:03:04.972 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:03:04.973 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:03:04.973 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32243 2024-09-21T10:03:04.973 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.973 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:03:04.973 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.973 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:03:04.973 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564332+0000", 2024-09-21T10:03:04.973 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:04.973 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.973 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.973 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:03:04.973 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564332+0000", 2024-09-21T10:03:04.973 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:04.973 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.973 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.973 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:03:04.974 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564331+0000", 2024-09-21T10:03:04.974 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:03:04.974 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.974 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.974 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:03:04.974 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564391+0000", 2024-09-21T10:03:04.974 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0105999999999997e-05 2024-09-21T10:03:04.974 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.974 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.974 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:03:04.974 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564401+0000", 2024-09-21T10:03:04.974 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.8099999999999992e-06 2024-09-21T10:03:04.974 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.974 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.975 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:03:04.975 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564413+0000", 2024-09-21T10:03:04.975 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2163e-05 2024-09-21T10:03:04.975 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.975 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.975 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:04.975 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564449+0000", 2024-09-21T10:03:04.975 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5775999999999998e-05 2024-09-21T10:03:04.975 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.975 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.975 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:03:04.975 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564557+0000", 2024-09-21T10:03:04.975 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010838199999999999 2024-09-21T10:03:04.975 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.975 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.975 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:03:04.976 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564694+0000", 2024-09-21T10:03:04.976 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013716299999999999 2024-09-21T10:03:04.976 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.976 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.976 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:03:04.976 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.565898+0000", 2024-09-21T10:03:04.976 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0012039570000000001 2024-09-21T10:03:04.976 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.976 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.976 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:04.976 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.567502+0000", 2024-09-21T10:03:04.976 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001604149 2024-09-21T10:03:04.976 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.976 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.976 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:04.976 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800209+0000", 2024-09-21T10:03:04.977 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.232706624 2024-09-21T10:03:04.977 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.977 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.977 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:03:04.977 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800255+0000", 2024-09-21T10:03:04.977 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6452000000000002e-05 2024-09-21T10:03:04.977 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.977 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.977 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:03:04.977 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800297+0000", 2024-09-21T10:03:04.977 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1508999999999998e-05 2024-09-21T10:03:04.977 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:04.977 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:04.977 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:04.977 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.977 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.978 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32262 9.6 9:64cc6811:::benchmark_data_smithi107_152010_object32261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:03:04.978 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.575084+0000", 2024-09-21T10:03:04.978 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 121.356561046, 2024-09-21T10:03:04.978 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.22529529000000001, 2024-09-21T10:03:04.978 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:03:04.978 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:03:04.978 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:03:04.978 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:03:04.978 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:03:04.978 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32262 2024-09-21T10:03:04.978 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.978 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:03:04.978 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.978 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:03:04.979 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575084+0000", 2024-09-21T10:03:04.979 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:04.979 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.979 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.979 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:03:04.979 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575084+0000", 2024-09-21T10:03:04.979 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:04.979 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.979 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.979 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:03:04.979 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575083+0000", 2024-09-21T10:03:04.979 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:03:04.979 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.979 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.979 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:03:04.979 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575130+0000", 2024-09-21T10:03:04.980 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7709e-05 2024-09-21T10:03:04.980 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.980 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.980 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:03:04.980 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575141+0000", 2024-09-21T10:03:04.980 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0971e-05 2024-09-21T10:03:04.980 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.980 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.980 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:03:04.980 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575146+0000", 2024-09-21T10:03:04.980 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.262e-06 2024-09-21T10:03:04.980 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.980 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.980 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:04.980 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575189+0000", 2024-09-21T10:03:04.981 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3216000000000001e-05 2024-09-21T10:03:04.981 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.981 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.981 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:03:04.981 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575313+0000", 2024-09-21T10:03:04.981 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012379099999999999 2024-09-21T10:03:04.981 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.981 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.981 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:03:04.981 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575477+0000", 2024-09-21T10:03:04.981 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00016417499999999999 2024-09-21T10:03:04.981 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.981 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.981 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:03:04.981 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.577444+0000", 2024-09-21T10:03:04.982 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0019669790000000002 2024-09-21T10:03:04.982 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.982 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.982 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:04.982 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.577574+0000", 2024-09-21T10:03:04.982 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012998900000000001 2024-09-21T10:03:04.982 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.982 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.982 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:04.982 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800326+0000", 2024-09-21T10:03:04.982 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.22275213099999999 2024-09-21T10:03:04.982 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.982 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.982 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:03:04.982 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800353+0000", 2024-09-21T10:03:04.983 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7056e-05 2024-09-21T10:03:04.983 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.983 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.983 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:03:04.983 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800379+0000", 2024-09-21T10:03:04.983 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.6203e-05 2024-09-21T10:03:04.983 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:04.983 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:04.983 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:04.983 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.983 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.983 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32272 9.6 9:68a66308:::benchmark_data_smithi107_152010_object32271:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:03:04.983 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.579482+0000", 2024-09-21T10:03:04.983 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 121.352162815, 2024-09-21T10:03:04.983 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.22308472700000001, 2024-09-21T10:03:04.984 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:03:04.984 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:03:04.984 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:03:04.984 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:03:04.984 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:03:04.984 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32272 2024-09-21T10:03:04.984 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.984 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:03:04.984 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.984 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:03:04.984 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579482+0000", 2024-09-21T10:03:04.984 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:04.984 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.984 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.984 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:03:04.985 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579482+0000", 2024-09-21T10:03:04.985 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:04.985 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.985 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.985 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:03:04.985 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579481+0000", 2024-09-21T10:03:04.985 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:03:04.985 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.985 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.985 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:03:04.985 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579545+0000", 2024-09-21T10:03:04.985 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4506999999999998e-05 2024-09-21T10:03:04.985 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.985 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.985 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:03:04.986 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579557+0000", 2024-09-21T10:03:04.986 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1610000000000001e-05 2024-09-21T10:03:04.986 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.986 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.986 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:03:04.986 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579562+0000", 2024-09-21T10:03:04.986 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4970000000000001e-06 2024-09-21T10:03:04.986 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.986 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.986 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:04.986 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579606+0000", 2024-09-21T10:03:04.986 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3180000000000003e-05 2024-09-21T10:03:04.986 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.986 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.986 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:03:04.987 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579720+0000", 2024-09-21T10:03:04.987 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000114689 2024-09-21T10:03:04.987 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.987 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.987 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:03:04.987 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579854+0000", 2024-09-21T10:03:04.987 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000134144 2024-09-21T10:03:04.987 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.987 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.987 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:03:04.987 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.580922+0000", 2024-09-21T10:03:04.987 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001067744 2024-09-21T10:03:04.987 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.987 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.987 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:04.988 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581827+0000", 2024-09-21T10:03:04.988 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00090439700000000004 2024-09-21T10:03:04.988 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.988 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.988 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:04.988 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.802500+0000", 2024-09-21T10:03:04.988 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.22067335699999999 2024-09-21T10:03:04.988 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.988 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.988 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:03:04.988 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.802531+0000", 2024-09-21T10:03:04.988 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1469e-05 2024-09-21T10:03:04.988 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.988 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.989 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:03:04.989 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.802567+0000", 2024-09-21T10:03:04.989 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5410000000000001e-05 2024-09-21T10:03:04.989 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:04.989 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:04.989 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:04.989 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.989 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.989 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32275 9.7 9:ef23b11d:::benchmark_data_smithi107_152010_object32274:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:03:04.989 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.581115+0000", 2024-09-21T10:03:04.989 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 121.350529444, 2024-09-21T10:03:04.989 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21812805800000001, 2024-09-21T10:03:04.989 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:03:04.989 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:03:04.989 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:03:04.990 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:03:04.990 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:03:04.990 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32275 2024-09-21T10:03:04.990 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.990 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:03:04.990 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.990 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:03:04.990 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581115+0000", 2024-09-21T10:03:04.990 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:04.990 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.990 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.990 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:03:04.990 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581115+0000", 2024-09-21T10:03:04.990 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:04.990 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.991 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.991 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:03:04.991 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581115+0000", 2024-09-21T10:03:04.991 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:03:04.991 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.991 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.991 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:03:04.991 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581168+0000", 2024-09-21T10:03:04.991 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3013000000000002e-05 2024-09-21T10:03:04.991 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.991 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.991 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:03:04.991 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581179+0000", 2024-09-21T10:03:04.991 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1970000000000001e-05 2024-09-21T10:03:04.991 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.992 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.992 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:03:04.992 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581184+0000", 2024-09-21T10:03:04.992 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7949999999999998e-06 2024-09-21T10:03:04.992 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.992 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.992 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:04.992 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581209+0000", 2024-09-21T10:03:04.992 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.4389999999999999e-05 2024-09-21T10:03:04.992 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.992 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.992 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:03:04.992 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581282+0000", 2024-09-21T10:03:04.992 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3590000000000005e-05 2024-09-21T10:03:04.993 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.993 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.993 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:03:04.993 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581399+0000", 2024-09-21T10:03:04.993 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000117137 2024-09-21T10:03:04.993 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.993 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.993 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:03:04.993 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.582724+0000", 2024-09-21T10:03:04.993 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0013241220000000001 2024-09-21T10:03:04.993 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.993 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.993 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:04.993 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.582977+0000", 2024-09-21T10:03:04.993 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00025303400000000002 2024-09-21T10:03:04.994 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.994 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.994 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:04.994 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.799174+0000", 2024-09-21T10:03:04.994 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21619723299999999 2024-09-21T10:03:04.994 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.994 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.994 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:03:04.994 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.799206+0000", 2024-09-21T10:03:04.994 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2026999999999997e-05 2024-09-21T10:03:04.994 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.994 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.994 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:03:04.994 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.799244+0000", 2024-09-21T10:03:04.994 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7721e-05 2024-09-21T10:03:04.995 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:04.995 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:04.995 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:04.995 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.995 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.995 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32283 9.7 9:ee5c92df:::benchmark_data_smithi107_152010_object32282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:03:04.995 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.589070+0000", 2024-09-21T10:03:04.995 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 121.342575364, 2024-09-21T10:03:04.995 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21034788500000001, 2024-09-21T10:03:04.995 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:03:04.995 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:03:04.995 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:03:04.995 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:03:04.995 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:03:04.996 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32283 2024-09-21T10:03:04.996 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.996 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:03:04.996 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.996 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:03:04.996 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589070+0000", 2024-09-21T10:03:04.996 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:04.996 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.996 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.996 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:03:04.996 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589070+0000", 2024-09-21T10:03:04.996 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:04.996 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.996 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.996 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:03:04.996 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589068+0000", 2024-09-21T10:03:04.997 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:03:04.997 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.997 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.997 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:03:04.997 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589113+0000", 2024-09-21T10:03:04.997 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4545000000000001e-05 2024-09-21T10:03:04.997 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.997 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.997 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:03:04.997 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589126+0000", 2024-09-21T10:03:04.997 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2738999999999999e-05 2024-09-21T10:03:04.997 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.997 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.997 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:03:04.998 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589130+0000", 2024-09-21T10:03:04.998 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3510000000000004e-06 2024-09-21T10:03:04.998 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.998 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.998 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:04.998 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589162+0000", 2024-09-21T10:03:04.998 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1612999999999997e-05 2024-09-21T10:03:04.998 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.998 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.998 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:03:04.998 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589244+0000", 2024-09-21T10:03:04.998 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.2725000000000005e-05 2024-09-21T10:03:04.998 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.998 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.998 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:03:04.999 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589327+0000", 2024-09-21T10:03:04.999 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.3182000000000004e-05 2024-09-21T10:03:04.999 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.999 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.999 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:03:04.999 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.590365+0000", 2024-09-21T10:03:04.999 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0010372059999999999 2024-09-21T10:03:04.999 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.999 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.999 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:04.999 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.590700+0000", 2024-09-21T10:03:04.999 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00033515699999999999 2024-09-21T10:03:04.999 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:04.999 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:04.999 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:05.000 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.799339+0000", 2024-09-21T10:03:05.000 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.20863885800000001 2024-09-21T10:03:05.000 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.000 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.000 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:03:05.000 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.799377+0000", 2024-09-21T10:03:05.000 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8281000000000002e-05 2024-09-21T10:03:05.000 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.000 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.000 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:03:05.000 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.799417+0000", 2024-09-21T10:03:05.000 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0466999999999997e-05 2024-09-21T10:03:05.000 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.000 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:05.000 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.001 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.001 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.001 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:101219 9.f 9:fbbb13dd:::benchmark_data_smithi107_152010_object101218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-21T10:03:05.001 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:09.472644+0000", 2024-09-21T10:03:05.001 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 55.459000635000002, 2024-09-21T10:03:05.001 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.27593879999999998, 2024-09-21T10:03:05.001 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:03:05.001 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:03:05.001 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:03:05.001 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:03:05.001 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:03:05.001 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 101219 2024-09-21T10:03:05.001 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.001 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:03:05.002 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.003 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:03:05.003 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472644+0000", 2024-09-21T10:03:05.003 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:05.003 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.003 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.003 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:03:05.003 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472644+0000", 2024-09-21T10:03:05.003 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:05.003 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.003 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.003 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:03:05.003 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472643+0000", 2024-09-21T10:03:05.003 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:03:05.003 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.004 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.004 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:03:05.004 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472693+0000", 2024-09-21T10:03:05.004 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9314999999999998e-05 2024-09-21T10:03:05.004 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.004 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.004 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:03:05.004 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472707+0000", 2024-09-21T10:03:05.004 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4048e-05 2024-09-21T10:03:05.004 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.004 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.004 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:03:05.004 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472712+0000", 2024-09-21T10:03:05.004 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.524e-06 2024-09-21T10:03:05.004 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.005 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.005 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:05.005 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472747+0000", 2024-09-21T10:03:05.005 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5207999999999999e-05 2024-09-21T10:03:05.005 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.005 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.005 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:03:05.005 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472830+0000", 2024-09-21T10:03:05.005 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.3107000000000002e-05 2024-09-21T10:03:05.005 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.005 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.005 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:03:05.005 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472921+0000", 2024-09-21T10:03:05.005 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.0845000000000002e-05 2024-09-21T10:03:05.006 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.006 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.006 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:03:05.006 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.475574+0000", 2024-09-21T10:03:05.006 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002653121 2024-09-21T10:03:05.006 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.006 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.006 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:05.006 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.484356+0000", 2024-09-21T10:03:05.006 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0087821459999999994 2024-09-21T10:03:05.006 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.006 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.006 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:05.006 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.748351+0000", 2024-09-21T10:03:05.007 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.26399478999999998 2024-09-21T10:03:05.007 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.007 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.007 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:03:05.007 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.748541+0000", 2024-09-21T10:03:05.007 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00018932500000000001 2024-09-21T10:03:05.007 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.007 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.007 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:03:05.007 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.748583+0000", 2024-09-21T10:03:05.007 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2478999999999998e-05 2024-09-21T10:03:05.007 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.007 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:05.007 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.008 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.008 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.008 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:101221 9.6 9:6e7b28d1:::benchmark_data_smithi107_152010_object101220:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-21T10:03:05.008 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:09.473703+0000", 2024-09-21T10:03:05.008 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 55.457941718999997, 2024-09-21T10:03:05.008 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.27788642400000002, 2024-09-21T10:03:05.008 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:03:05.008 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:03:05.008 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:03:05.008 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:03:05.008 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:03:05.008 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 101221 2024-09-21T10:03:05.008 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.008 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:03:05.008 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.009 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:03:05.009 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473703+0000", 2024-09-21T10:03:05.009 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:05.009 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.009 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.009 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:03:05.009 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473703+0000", 2024-09-21T10:03:05.009 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:05.009 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.009 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.009 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:03:05.009 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473703+0000", 2024-09-21T10:03:05.009 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:03:05.009 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.010 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.010 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:03:05.010 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473759+0000", 2024-09-21T10:03:05.010 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5989999999999998e-05 2024-09-21T10:03:05.010 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.010 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.010 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:03:05.010 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473768+0000", 2024-09-21T10:03:05.010 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.0119999999999994e-06 2024-09-21T10:03:05.010 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.010 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.010 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:03:05.010 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473771+0000", 2024-09-21T10:03:05.010 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6440000000000002e-06 2024-09-21T10:03:05.011 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.011 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.011 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:05.011 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473801+0000", 2024-09-21T10:03:05.011 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.9403e-05 2024-09-21T10:03:05.011 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.011 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.011 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:03:05.011 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473906+0000", 2024-09-21T10:03:05.011 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000105516 2024-09-21T10:03:05.011 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.011 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.011 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:03:05.011 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473986+0000", 2024-09-21T10:03:05.012 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9802000000000006e-05 2024-09-21T10:03:05.012 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.012 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.012 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:03:05.012 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.480993+0000", 2024-09-21T10:03:05.012 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.007007231 2024-09-21T10:03:05.012 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.012 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.012 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:05.012 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.481030+0000", 2024-09-21T10:03:05.012 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6848999999999999e-05 2024-09-21T10:03:05.012 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.012 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.012 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:05.012 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.751491+0000", 2024-09-21T10:03:05.013 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.27046129299999999 2024-09-21T10:03:05.013 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.013 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.013 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:03:05.013 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.751548+0000", 2024-09-21T10:03:05.013 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6422999999999998e-05 2024-09-21T10:03:05.013 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.013 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.013 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:03:05.013 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.751590+0000", 2024-09-21T10:03:05.013 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1850000000000001e-05 2024-09-21T10:03:05.013 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.013 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:05.013 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.013 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.014 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.014 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:101223 9.6 9:66631bd2:::benchmark_data_smithi107_152010_object101222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-21T10:03:05.014 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:09.473791+0000", 2024-09-21T10:03:05.014 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 55.457853948999997, 2024-09-21T10:03:05.014 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.283198595, 2024-09-21T10:03:05.014 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:03:05.014 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:03:05.014 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:03:05.014 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:03:05.014 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:03:05.014 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 101223 2024-09-21T10:03:05.014 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.014 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:03:05.014 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.014 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:03:05.015 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473791+0000", 2024-09-21T10:03:05.015 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:05.015 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.015 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.015 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:03:05.015 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473791+0000", 2024-09-21T10:03:05.015 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:05.015 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.015 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.015 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:03:05.015 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473790+0000", 2024-09-21T10:03:05.015 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:03:05.015 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.015 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.015 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:03:05.016 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473870+0000", 2024-09-21T10:03:05.016 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9717999999999998e-05 2024-09-21T10:03:05.016 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.016 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.016 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:03:05.016 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473878+0000", 2024-09-21T10:03:05.016 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9589999999999995e-06 2024-09-21T10:03:05.016 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.016 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.016 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:03:05.016 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473883+0000", 2024-09-21T10:03:05.016 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8960000000000002e-06 2024-09-21T10:03:05.016 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.016 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.017 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:05.017 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.476473+0000", 2024-09-21T10:03:05.017 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0025897509999999999 2024-09-21T10:03:05.017 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.017 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.017 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:03:05.017 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.476556+0000", 2024-09-21T10:03:05.017 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.3242e-05 2024-09-21T10:03:05.017 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.017 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.017 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:03:05.017 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.476636+0000", 2024-09-21T10:03:05.017 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.0174999999999997e-05 2024-09-21T10:03:05.017 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.017 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.018 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:03:05.018 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.480996+0000", 2024-09-21T10:03:05.018 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0043602459999999999 2024-09-21T10:03:05.018 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.018 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.018 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:05.018 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.488327+0000", 2024-09-21T10:03:05.018 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0073310729999999996 2024-09-21T10:03:05.018 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.018 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.018 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:05.018 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.756861+0000", 2024-09-21T10:03:05.018 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.26853359700000001 2024-09-21T10:03:05.018 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.019 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.019 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:03:05.019 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.756947+0000", 2024-09-21T10:03:05.019 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6316999999999994e-05 2024-09-21T10:03:05.019 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.019 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.019 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:03:05.019 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.756990+0000", 2024-09-21T10:03:05.019 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2259000000000003e-05 2024-09-21T10:03:05.019 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.019 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:05.019 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.019 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.019 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.020 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:101225 9.6 9:6bf868dc:::benchmark_data_smithi107_152010_object101224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-21T10:03:05.020 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:09.474216+0000", 2024-09-21T10:03:05.020 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 55.457429290999997, 2024-09-21T10:03:05.020 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28288837500000003, 2024-09-21T10:03:05.020 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:03:05.020 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:03:05.020 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:03:05.020 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:03:05.020 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:03:05.020 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 101225 2024-09-21T10:03:05.020 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.020 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:03:05.020 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.020 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:03:05.020 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474216+0000", 2024-09-21T10:03:05.021 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:05.021 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.021 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.021 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:03:05.021 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474216+0000", 2024-09-21T10:03:05.021 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:05.021 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.021 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.021 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:03:05.021 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474215+0000", 2024-09-21T10:03:05.021 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:03:05.021 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.021 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.021 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:03:05.022 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474301+0000", 2024-09-21T10:03:05.022 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6462e-05 2024-09-21T10:03:05.022 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.022 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.022 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:03:05.022 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474313+0000", 2024-09-21T10:03:05.022 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1685000000000001e-05 2024-09-21T10:03:05.022 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.022 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.022 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:03:05.022 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474318+0000", 2024-09-21T10:03:05.022 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2190000000000003e-06 2024-09-21T10:03:05.022 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.022 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.023 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:05.023 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.478980+0000", 2024-09-21T10:03:05.023 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0046614480000000003 2024-09-21T10:03:05.023 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.023 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.023 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:03:05.023 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.479090+0000", 2024-09-21T10:03:05.023 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000110468 2024-09-21T10:03:05.023 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.023 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.023 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:03:05.023 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.479168+0000", 2024-09-21T10:03:05.023 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.7837999999999996e-05 2024-09-21T10:03:05.023 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.024 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.024 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:03:05.024 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.481151+0000", 2024-09-21T10:03:05.024 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0019829729999999999 2024-09-21T10:03:05.024 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.024 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.024 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:05.024 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.488352+0000", 2024-09-21T10:03:05.024 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0072014410000000003 2024-09-21T10:03:05.024 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.024 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.024 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:05.024 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.757027+0000", 2024-09-21T10:03:05.024 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.26867438300000002 2024-09-21T10:03:05.025 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.025 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.025 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:03:05.025 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.757069+0000", 2024-09-21T10:03:05.025 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1835000000000002e-05 2024-09-21T10:03:05.025 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.025 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.025 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:03:05.025 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.757104+0000", 2024-09-21T10:03:05.025 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5410000000000001e-05 2024-09-21T10:03:05.025 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.025 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:05.025 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.025 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.025 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.025 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:126108 9.6 9:6f7b6afe:::benchmark_data_smithi107_152010_object16171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-21T10:03:05.026 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:38.616365+0000", 2024-09-21T10:03:05.026 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 26.315279729, 2024-09-21T10:03:05.026 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.213592333, 2024-09-21T10:03:05.026 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:03:05.026 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:03:05.026 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:03:05.026 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:03:05.026 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:03:05.026 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 126108 2024-09-21T10:03:05.026 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.026 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:03:05.026 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.026 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:03:05.026 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616365+0000", 2024-09-21T10:03:05.027 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:05.027 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.027 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.027 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:03:05.027 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616365+0000", 2024-09-21T10:03:05.027 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:05.027 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.027 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.027 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:03:05.027 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616364+0000", 2024-09-21T10:03:05.027 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:03:05.027 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.027 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.027 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:03:05.027 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616371+0000", 2024-09-21T10:03:05.028 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9840000000000004e-06 2024-09-21T10:03:05.028 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.028 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.028 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:03:05.028 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616390+0000", 2024-09-21T10:03:05.028 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.9476000000000001e-05 2024-09-21T10:03:05.028 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.028 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.028 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:03:05.028 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616397+0000", 2024-09-21T10:03:05.028 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4130000000000003e-06 2024-09-21T10:03:05.028 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.028 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.028 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:05.029 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616446+0000", 2024-09-21T10:03:05.029 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8998000000000001e-05 2024-09-21T10:03:05.029 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.029 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.029 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:03:05.029 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616569+0000", 2024-09-21T10:03:05.029 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000123318 2024-09-21T10:03:05.029 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.029 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.029 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:03:05.029 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616671+0000", 2024-09-21T10:03:05.029 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010222599999999999 2024-09-21T10:03:05.029 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.029 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.029 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:03:05.029 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.620447+0000", 2024-09-21T10:03:05.030 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0037759579999999998 2024-09-21T10:03:05.030 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.030 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.030 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:05.030 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.829942+0000", 2024-09-21T10:03:05.030 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.20949464800000001 2024-09-21T10:03:05.030 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.030 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.030 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:03:05.030 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.829958+0000", 2024-09-21T10:03:05.030 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5863999999999999e-05 2024-09-21T10:03:05.030 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.030 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:05.031 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.031 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.031 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.031 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop_reply(client.45761.0:126108 9.6 e162/141)", 2024-09-21T10:03:05.031 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:38.629127+0000", 2024-09-21T10:03:05.031 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 26.302517872999999, 2024-09-21T10:03:05.031 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.20087063799999999, 2024-09-21T10:03:05.031 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:03:05.031 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:03:05.031 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:03:05.031 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.031 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:03:05.031 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.629127+0000", 2024-09-21T10:03:05.032 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:05.032 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.032 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.032 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:03:05.032 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.629127+0000", 2024-09-21T10:03:05.032 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:05.032 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.032 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.032 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:03:05.032 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.629126+0000", 2024-09-21T10:03:05.032 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:03:05.032 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.032 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.032 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:03:05.033 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.629132+0000", 2024-09-21T10:03:05.033 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.781e-06 2024-09-21T10:03:05.033 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.033 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.033 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:03:05.033 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.629148+0000", 2024-09-21T10:03:05.033 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5446000000000001e-05 2024-09-21T10:03:05.033 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.033 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.033 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:03:05.033 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.629154+0000", 2024-09-21T10:03:05.033 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7989999999999999e-06 2024-09-21T10:03:05.034 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.034 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.034 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:05.034 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.829983+0000", 2024-09-21T10:03:05.034 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.20082974300000001 2024-09-21T10:03:05.034 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.034 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.034 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:03:05.034 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.829998+0000", 2024-09-21T10:03:05.034 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4307999999999999e-05 2024-09-21T10:03:05.034 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.034 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:05.034 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.034 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.035 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.035 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:126108 9.16 9:6f7b6afe:::benchmark_data_smithi107_152010_object16171:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163)", 2024-09-21T10:03:05.035 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:38.641118+0000", 2024-09-21T10:03:05.035 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 26.290527057999999, 2024-09-21T10:03:05.035 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.03992945, 2024-09-21T10:03:05.035 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:03:05.035 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:03:05.035 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:03:05.035 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:03:05.035 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:03:05.035 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 126108 2024-09-21T10:03:05.035 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.035 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:03:05.035 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.036 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:03:05.036 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.641118+0000", 2024-09-21T10:03:05.036 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:05.036 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.036 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.036 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:03:05.036 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.641118+0000", 2024-09-21T10:03:05.036 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:05.036 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.036 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.036 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:03:05.036 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.641117+0000", 2024-09-21T10:03:05.036 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:03:05.036 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.037 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.037 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:03:05.037 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.641124+0000", 2024-09-21T10:03:05.037 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5200000000000003e-06 2024-09-21T10:03:05.037 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.037 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.037 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:03:05.037 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.641133+0000", 2024-09-21T10:03:05.037 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.7790000000000007e-06 2024-09-21T10:03:05.037 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.037 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.037 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:03:05.037 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.641139+0000", 2024-09-21T10:03:05.037 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8510000000000004e-06 2024-09-21T10:03:05.037 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.038 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.038 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:05.038 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.830197+0000", 2024-09-21T10:03:05.038 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.18905825400000001 2024-09-21T10:03:05.038 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.038 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.038 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:03:05.038 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.830204+0000", 2024-09-21T10:03:05.038 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9940000000000003e-06 2024-09-21T10:03:05.038 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.038 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.038 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:05.038 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:39.680990+0000", 2024-09-21T10:03:05.038 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.85078537099999996 2024-09-21T10:03:05.039 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.039 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.039 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:03:05.039 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:39.681047+0000", 2024-09-21T10:03:05.039 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7512000000000001e-05 2024-09-21T10:03:05.039 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.039 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:05.039 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.039 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.039 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.039 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:126119 9.a 9:5778ec59:::benchmark_data_smithi107_152010_object16182:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-21T10:03:05.039 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:38.652541+0000", 2024-09-21T10:03:05.039 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 26.279103983999999, 2024-09-21T10:03:05.039 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.413684987, 2024-09-21T10:03:05.039 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:03:05.040 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:03:05.040 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:03:05.040 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:03:05.040 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:03:05.040 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 126119 2024-09-21T10:03:05.040 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.040 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:03:05.040 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.040 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:03:05.040 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652541+0000", 2024-09-21T10:03:05.040 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:05.040 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.040 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.040 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:03:05.041 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652541+0000", 2024-09-21T10:03:05.041 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:05.041 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.041 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.041 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:03:05.041 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652539+0000", 2024-09-21T10:03:05.041 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999976 2024-09-21T10:03:05.041 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.041 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.041 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:03:05.041 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652548+0000", 2024-09-21T10:03:05.041 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.5810000000000003e-06 2024-09-21T10:03:05.041 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.041 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.042 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:03:05.042 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652568+0000", 2024-09-21T10:03:05.042 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.9656999999999998e-05 2024-09-21T10:03:05.042 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.042 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.042 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:03:05.042 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652575+0000", 2024-09-21T10:03:05.042 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2960000000000001e-06 2024-09-21T10:03:05.042 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.042 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.042 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:05.042 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652636+0000", 2024-09-21T10:03:05.042 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0293e-05 2024-09-21T10:03:05.042 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.043 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.043 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:03:05.043 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652783+0000", 2024-09-21T10:03:05.043 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014715 2024-09-21T10:03:05.043 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.043 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.043 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:03:05.043 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652898+0000", 2024-09-21T10:03:05.043 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011513700000000001 2024-09-21T10:03:05.043 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.043 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.043 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:03:05.043 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.657580+0000", 2024-09-21T10:03:05.043 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0046820580000000002 2024-09-21T10:03:05.043 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.044 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.044 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:05.044 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.895867+0000", 2024-09-21T10:03:05.044 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23828700799999999 2024-09-21T10:03:05.044 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.044 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.044 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:05.044 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:39.066119+0000", 2024-09-21T10:03:05.044 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.170251927 2024-09-21T10:03:05.044 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.044 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.044 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:03:05.044 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:39.066160+0000", 2024-09-21T10:03:05.044 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.074e-05 2024-09-21T10:03:05.045 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.045 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.045 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:03:05.045 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:39.066226+0000", 2024-09-21T10:03:05.045 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6333000000000003e-05 2024-09-21T10:03:05.045 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.045 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:05.045 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.045 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.045 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.045 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:150450 9.16 9:6ac3e7ad:::benchmark_data_smithi107_152010_object40513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-21T10:03:05.045 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:03:04.016611+0000", 2024-09-21T10:03:05.045 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.91503422999999995, 2024-09-21T10:03:05.046 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.19595029, 2024-09-21T10:03:05.046 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:03:05.046 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:03:05.046 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:03:05.046 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:03:05.046 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:03:05.046 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 150450 2024-09-21T10:03:05.046 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.046 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:03:05.046 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.046 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:03:05.046 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.016611+0000", 2024-09-21T10:03:05.046 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:05.047 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.047 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.047 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:03:05.047 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.016611+0000", 2024-09-21T10:03:05.047 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:05.047 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.047 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.047 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:03:05.047 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.016609+0000", 2024-09-21T10:03:05.047 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:03:05.047 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.047 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.047 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:03:05.047 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.016616+0000", 2024-09-21T10:03:05.047 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2749999999999998e-06 2024-09-21T10:03:05.048 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.048 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.048 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:03:05.048 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.016641+0000", 2024-09-21T10:03:05.048 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.482e-05 2024-09-21T10:03:05.048 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.048 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.048 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:03:05.048 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.016647+0000", 2024-09-21T10:03:05.048 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9340000000000003e-06 2024-09-21T10:03:05.048 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.048 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.048 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:05.048 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.016688+0000", 2024-09-21T10:03:05.048 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1026999999999999e-05 2024-09-21T10:03:05.049 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.049 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.049 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:03:05.049 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.016780+0000", 2024-09-21T10:03:05.049 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.1557000000000004e-05 2024-09-21T10:03:05.049 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.049 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.049 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:03:05.049 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.017094+0000", 2024-09-21T10:03:05.049 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00031382900000000001 2024-09-21T10:03:05.049 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.049 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.049 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:03:05.049 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.019022+0000", 2024-09-21T10:03:05.049 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0019287950000000001 2024-09-21T10:03:05.050 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.050 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.050 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:05.050 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.022960+0000", 2024-09-21T10:03:05.050 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0039376890000000003 2024-09-21T10:03:05.050 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.050 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.050 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:05.050 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.212489+0000", 2024-09-21T10:03:05.050 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.18952859999999999 2024-09-21T10:03:05.050 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.050 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.050 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:03:05.050 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.212534+0000", 2024-09-21T10:03:05.050 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5164999999999999e-05 2024-09-21T10:03:05.051 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.051 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.051 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:03:05.051 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.212561+0000", 2024-09-21T10:03:05.051 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7160000000000001e-05 2024-09-21T10:03:05.051 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.051 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:05.051 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.051 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.051 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.051 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:150455 9.7 9:e1e071f1:::benchmark_data_smithi107_152010_object40518:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e168)", 2024-09-21T10:03:05.051 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:03:04.020599+0000", 2024-09-21T10:03:05.051 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.91104630499999995, 2024-09-21T10:03:05.051 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.19509716499999999, 2024-09-21T10:03:05.052 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:03:05.052 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:03:05.052 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:03:05.052 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:03:05.052 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:03:05.052 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 150455 2024-09-21T10:03:05.052 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.052 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:03:05.052 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.052 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:03:05.052 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.020599+0000", 2024-09-21T10:03:05.052 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:05.052 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.052 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.052 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:03:05.053 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.020599+0000", 2024-09-21T10:03:05.053 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:03:05.053 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.053 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.053 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:03:05.053 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.020597+0000", 2024-09-21T10:03:05.053 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:03:05.053 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.053 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.053 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:03:05.053 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.020603+0000", 2024-09-21T10:03:05.053 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0630000000000002e-06 2024-09-21T10:03:05.053 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.053 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.053 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:03:05.053 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.020615+0000", 2024-09-21T10:03:05.054 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2116999999999999e-05 2024-09-21T10:03:05.054 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.054 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.054 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:03:05.054 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.020621+0000", 2024-09-21T10:03:05.054 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3369999999999999e-06 2024-09-21T10:03:05.054 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.054 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.054 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:03:05.054 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.020663+0000", 2024-09-21T10:03:05.054 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2040999999999999e-05 2024-09-21T10:03:05.054 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.054 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.054 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:03:05.055 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.020788+0000", 2024-09-21T10:03:05.055 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012577399999999999 2024-09-21T10:03:05.055 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.055 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.055 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:03:05.055 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.021243+0000", 2024-09-21T10:03:05.055 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00045419299999999998 2024-09-21T10:03:05.055 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.055 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.055 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:03:05.055 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.023521+0000", 2024-09-21T10:03:05.055 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0022787380000000002 2024-09-21T10:03:05.055 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.055 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.056 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:05.056 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.025036+0000", 2024-09-21T10:03:05.056 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0015144449999999999 2024-09-21T10:03:05.056 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.056 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.056 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:05.056 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.215536+0000", 2024-09-21T10:03:05.056 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.19050009300000001 2024-09-21T10:03:05.056 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.056 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.056 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:03:05.056 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.215569+0000", 2024-09-21T10:03:05.056 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3377999999999998e-05 2024-09-21T10:03:05.056 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:03:05.056 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:03:05.057 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:03:05.057 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:03:04.215696+0000", 2024-09-21T10:03:05.057 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012659999999999999 2024-09-21T10:03:05.057 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.057 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:05.057 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.057 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:03:05.057 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:03:05.057 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:03:05.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:05.058 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:05.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:05.142 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:05.142 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:05.159 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:05.183 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:05.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:05.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:05.284 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:05.312 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:05.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:05.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:05.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:05 smithi107 ceph-mon[102221]: pgmap v594: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 15/218709 objects degraded (0.007%); 40877/218709 objects misplaced (18.690%) 2024-09-21T10:03:05.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:05 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/218709 objects degraded (0.007%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:05.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:05 smithi107 ceph-mon[105022]: pgmap v594: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 15/218709 objects degraded (0.007%); 40877/218709 objects misplaced (18.690%) 2024-09-21T10:03:05.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:05 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/218709 objects degraded (0.007%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:05.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:05.483 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:05.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:05.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:05.528 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:05 smithi129 ceph-mon[88471]: pgmap v594: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 15/218709 objects degraded (0.007%); 40877/218709 objects misplaced (18.690%) 2024-09-21T10:03:05.528 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:05 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/218709 objects degraded (0.007%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:05.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:05.623 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:05.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:05.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:05.724 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:05.753 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:05.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:05.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:05.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:05.897 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:05.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:05.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:05.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:06.016 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:03:06.042 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:06.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:06.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:06.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:06.180 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:06.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:06.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:06.264 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:06 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4293935821' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:03:06.265 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:06 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4293935821' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:03:06.282 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:06.332 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:03:06.335 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:06.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:06.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:06.342 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:03:06.342 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:03:06.342 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:03:06.342 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:03:06.342 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:03:06.342 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:03:06.342 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:03:06.342 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:03:06.343 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:03:06.343 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:03:06.343 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:03:06.343 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:03:06.343 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:03:06.343 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:03:06.343 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T10:03:06.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:06 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4293935821' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:03:06.436 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:06.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:06.466 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:06.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:06.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:06.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:06.625 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:06.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:06.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:06.759 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:06.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:06.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:06.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:06.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:06.909 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:06.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:07.010 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:07.034 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:07.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:07.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:07.136 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:07.161 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:07.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:07.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:07.263 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:07.270 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:07 smithi107 ceph-mon[102221]: pgmap v595: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 15/207426 objects degraded (0.007%); 38863/207426 objects misplaced (18.736%) 2024-09-21T10:03:07.308 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:07.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:07.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:07.320 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:03:07.320 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:03:07.409 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:07.440 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:07.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:07.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:07.520 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:07 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:07.520 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:07 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:07.520 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:07 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:07.520 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:07 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:07.520 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:07 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:07.521 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:07 smithi107 ceph-mon[105022]: pgmap v595: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 15/207426 objects degraded (0.007%); 38863/207426 objects misplaced (18.736%) 2024-09-21T10:03:07.521 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:07 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:07.521 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:07 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:07.521 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:07 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:07.521 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:07 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:07.521 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:07 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:07.541 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:07.566 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:07.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:07.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:07.668 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:07.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:07 smithi129 ceph-mon[88471]: pgmap v595: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 15/207426 objects degraded (0.007%); 38863/207426 objects misplaced (18.736%) 2024-09-21T10:03:07.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:07 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:07.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:07 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:07.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:07 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:07.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:07 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:07.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:07 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:07.696 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:07.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:07.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:07.798 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:07.821 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:07.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:07.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:07.923 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:07.949 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:07.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:07.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:08.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:08.083 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:08.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:08.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:08.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:08.219 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:08.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:08.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:08.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:08.364 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:08.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:08.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:08.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:08.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:08.506 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:08.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:08.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:08.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:08 smithi107 ceph-mon[105022]: pgmap v596: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 15/207426 objects degraded (0.007%); 38863/207426 objects misplaced (18.736%) 2024-09-21T10:03:08.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:08 smithi107 ceph-mon[105022]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-21T10:03:08.614 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:08 smithi107 ceph-mon[102221]: pgmap v596: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 15/207426 objects degraded (0.007%); 38863/207426 objects misplaced (18.736%) 2024-09-21T10:03:08.614 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:08 smithi107 ceph-mon[102221]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-21T10:03:08.653 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:08.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:08.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:08.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:08 smithi129 ceph-mon[88471]: pgmap v596: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 15/207426 objects degraded (0.007%); 38863/207426 objects misplaced (18.736%) 2024-09-21T10:03:08.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:08 smithi129 ceph-mon[88471]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-21T10:03:08.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:08.786 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:08.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:08.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:08.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:08.929 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:08.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:08.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:09.031 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:09.055 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:09.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:09.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:09.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:09.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:09.191 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:09.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:09.292 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:09.318 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:09.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:09.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:09.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:03:09.420 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:09.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:09.489 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:09.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:09.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:09.634 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:09.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:09.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:09.736 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:09.764 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:09.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:09.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:09.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:09.900 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:09.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:09.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:10.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:10.037 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:10.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:10.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:10.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:10.177 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:10.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:10.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:10.271 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:03:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:03:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:10.279 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:10.312 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:10.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:10.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:10.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:10.443 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:10.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:10.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:10.527 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:03:10.528 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:03:10.528 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:10 smithi129 ceph-mon[88471]: pgmap v597: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 15/199725 objects degraded (0.008%); 37278/199725 objects misplaced (18.665%) 2024-09-21T10:03:10.528 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:10 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/199725 objects degraded (0.008%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:10.544 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:10.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:10.574 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:10.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:10.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:10 smithi107 ceph-mon[102221]: pgmap v597: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 15/199725 objects degraded (0.008%); 37278/199725 objects misplaced (18.665%) 2024-09-21T10:03:10.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:10 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/199725 objects degraded (0.008%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:10.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:10 smithi107 ceph-mon[105022]: pgmap v597: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 15/199725 objects degraded (0.008%); 37278/199725 objects misplaced (18.665%) 2024-09-21T10:03:10.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:10 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/199725 objects degraded (0.008%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:10.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:10.701 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:10.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:10.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:10.803 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:10.828 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:10.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:10.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:10.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:10.974 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:10.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:11.076 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:11.105 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:11.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:11.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:11.185 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:03:11.207 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:11.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:11.258 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:11.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:11.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:11.401 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:11.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:11.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:11.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:11.537 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:11.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:11.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:11.539 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:03:11.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:11.680 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:11.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:11.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:11.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:11.824 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:11.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:11.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:11.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:11.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:11.972 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:11.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:12.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:12.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:03:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:03:11] "GET /metrics HTTP/1.1" 200 36663 "" "Prometheus/2.51.0" 2024-09-21T10:03:12.111 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:12.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:12.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:12.233 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:12.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:12.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:12.335 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:12.357 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:12 smithi129 ceph-mon[88471]: pgmap v598: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 15/191883 objects degraded (0.008%); 35991/191883 objects misplaced (18.757%) 2024-09-21T10:03:12.357 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:12.357 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:12.367 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:12.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:12.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:12.469 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:12.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:12.499 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:12.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:12.600 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:12.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:12 smithi107 ceph-mon[102221]: pgmap v598: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 15/191883 objects degraded (0.008%); 35991/191883 objects misplaced (18.757%) 2024-09-21T10:03:12.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:12.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:12.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:12 smithi107 ceph-mon[105022]: pgmap v598: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.9 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.15k op/s; 15/191883 objects degraded (0.008%); 35991/191883 objects misplaced (18.757%) 2024-09-21T10:03:12.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:12.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:12.624 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:12.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:12.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:12.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:12.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:12.763 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:12.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:12.864 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:12.893 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:12.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:12.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:12.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:13.034 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:13.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:13.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:13.135 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:13.163 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:13.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:13.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:13.181 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:03:13.265 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:13.289 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:13.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:13.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:13.390 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:13.416 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:13.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:13.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:13 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1500882723' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:03:13.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:13 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:03:13.518 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:13.546 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:13.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:13.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:13.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:13 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1500882723' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:03:13.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:13 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:03:13.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:13 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1500882723' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:03:13.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:13 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:03:13.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:13.682 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:13.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:13.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:13.770 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:03:13.770 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:03:13.784 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:13.806 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:13.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:13.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:13.907 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:13.935 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:13.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:13.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:14.037 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:14.065 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:14.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:14.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:14.166 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:14.192 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:14.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:14.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:14.294 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:14.321 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:14.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:14.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:14.404 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:14 smithi129 ceph-mon[88471]: pgmap v599: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 15/187116 objects degraded (0.008%); 35013/187116 objects misplaced (18.712%) 2024-09-21T10:03:14.405 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:14 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:03:14.405 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:14 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:03:14.405 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:14 smithi129 ceph-mon[88471]: osdmap e169: 8 total, 7 up, 8 in 2024-09-21T10:03:14.422 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:14.432 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:03:14.459 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:14.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:14.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:14.561 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:14.593 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:14.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:14.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:14 smithi107 ceph-mon[102221]: pgmap v599: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 15/187116 objects degraded (0.008%); 35013/187116 objects misplaced (18.712%) 2024-09-21T10:03:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:14 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:03:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:14 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:03:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:14 smithi107 ceph-mon[102221]: osdmap e169: 8 total, 7 up, 8 in 2024-09-21T10:03:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:14 smithi107 ceph-mon[105022]: pgmap v599: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 15/187116 objects degraded (0.008%); 35013/187116 objects misplaced (18.712%) 2024-09-21T10:03:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:14 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:03:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:14 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:03:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:14 smithi107 ceph-mon[105022]: osdmap e169: 8 total, 7 up, 8 in 2024-09-21T10:03:14.695 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:14.725 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:14.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:14.769 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:03:14.827 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:14.854 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:14.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:14.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:14.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:14.983 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:14.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:14.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:15.085 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:15.114 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:15.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:15.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:15.216 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:15.237 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:15.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:15.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:15.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:15.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:15.371 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:15.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:15.474 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:15.508 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:15.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:15.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:15 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/187116 objects degraded (0.008%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:15.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:15 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/187116 objects degraded (0.008%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:15.610 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:15.635 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:15.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:15.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:15.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:15 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/187116 objects degraded (0.008%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:15.736 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:15.772 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:15.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:15.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:15.874 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:15.898 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:15.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:15.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:16.000 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:16.021 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:16.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:16.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:16.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:16.161 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:16.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:16.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:16.263 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:16.299 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:16 smithi129 ceph-mon[88471]: pgmap v601: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 956 op/s; 15/181095 objects degraded (0.008%); 34000/181095 objects misplaced (18.775%) 2024-09-21T10:03:16.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:16.305 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:16.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:16.406 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:16.434 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:16.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:16.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:16.536 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:16.558 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:16 smithi107 ceph-mon[102221]: pgmap v601: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 956 op/s; 15/181095 objects degraded (0.008%); 34000/181095 objects misplaced (18.775%) 2024-09-21T10:03:16.559 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:16 smithi107 ceph-mon[105022]: pgmap v601: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 956 op/s; 15/181095 objects degraded (0.008%); 34000/181095 objects misplaced (18.775%) 2024-09-21T10:03:16.561 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:16.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:16.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:16.564 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:03:16.564 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:03:16.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:16.700 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:16.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:16.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:16.802 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:16.832 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:16.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:16.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:16.934 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:16.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:16.962 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:16.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:17.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:17.105 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:17.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:17.207 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:17.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:03:17.237 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:17.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:17.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:17.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:17.390 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:17.391 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:17.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:17.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:17.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:17.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:17.545 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:17.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:17.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:17.698 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:17.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:17.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:17.800 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:17.829 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:17.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:17.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:17.931 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:17.956 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:17.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:17.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:18.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:18.082 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:18.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:18.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:18.183 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:18.191 INFO:teuthology.orchestra.run.smithi107.stderr:set pool 1 pg_num to 21 2024-09-21T10:03:18.208 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:18.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:18.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:18.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:18.350 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:18.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:18.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:18.452 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:18.461 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:18 smithi107 ceph-mon[102221]: pgmap v602: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 15/176379 objects degraded (0.009%); 33036/176379 objects misplaced (18.730%) 2024-09-21T10:03:18.461 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:18 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4249322117' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T10:03:18.461 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:18 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T10:03:18.461 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:18 smithi107 ceph-mon[105022]: pgmap v602: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 15/176379 objects degraded (0.009%); 33036/176379 objects misplaced (18.730%) 2024-09-21T10:03:18.461 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:18 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4249322117' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T10:03:18.462 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:18 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T10:03:18.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:18.478 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:18.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:18.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:18.623 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:18.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:18.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:18 smithi129 ceph-mon[88471]: pgmap v602: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 15/176379 objects degraded (0.009%); 33036/176379 objects misplaced (18.730%) 2024-09-21T10:03:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:18 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4249322117' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T10:03:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:18 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T10:03:18.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:18.761 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:18.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:18.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:18.863 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:18.893 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:18.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:18.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:18.995 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:19.028 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:19.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:19.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:19.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:03:19.130 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:19.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:19.157 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:19.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:19.258 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:19.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:19.313 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:19.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:19.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:19 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-21T10:03:19.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:19 smithi107 ceph-mon[105022]: osdmap e170: 8 total, 7 up, 8 in 2024-09-21T10:03:19.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:19 smithi107 ceph-mon[105022]: 9.17 scrub starts 2024-09-21T10:03:19.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:19 smithi107 ceph-mon[105022]: 9.17 scrub ok 2024-09-21T10:03:19.362 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:19 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T10:03:19.362 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:19 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-21T10:03:19.362 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:19 smithi107 ceph-mon[102221]: osdmap e170: 8 total, 7 up, 8 in 2024-09-21T10:03:19.362 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:19 smithi107 ceph-mon[102221]: 9.17 scrub starts 2024-09-21T10:03:19.362 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:19 smithi107 ceph-mon[102221]: 9.17 scrub ok 2024-09-21T10:03:19.362 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:19 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T10:03:19.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:19.463 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:19.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:19.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:19.565 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:19.598 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:19.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:19.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:19 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-21T10:03:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:19 smithi129 ceph-mon[88471]: osdmap e170: 8 total, 7 up, 8 in 2024-09-21T10:03:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:19 smithi129 ceph-mon[88471]: 9.17 scrub starts 2024-09-21T10:03:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:19 smithi129 ceph-mon[88471]: 9.17 scrub ok 2024-09-21T10:03:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:19 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T10:03:19.700 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:19.727 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:19.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:19.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:19.829 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:19.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:19.853 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:19.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:19.859 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:19.954 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:19.999 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:19.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:20.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:20.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:20.139 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:20.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:20.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:20.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:20.285 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:20.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:20.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:20.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:20.428 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:20.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:20.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:20.503 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:20 smithi107 ceph-mon[105022]: pgmap v604: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 968 op/s; 15/168636 objects degraded (0.009%); 31463/168636 objects misplaced (18.657%) 2024-09-21T10:03:20.503 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:20 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-21T10:03:20.503 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:20 smithi107 ceph-mon[105022]: osdmap e171: 8 total, 7 up, 8 in 2024-09-21T10:03:20.503 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:20 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/168636 objects degraded (0.009%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:20.503 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:20 smithi107 ceph-mon[102221]: pgmap v604: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 968 op/s; 15/168636 objects degraded (0.009%); 31463/168636 objects misplaced (18.657%) 2024-09-21T10:03:20.504 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:20 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-21T10:03:20.504 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:20 smithi107 ceph-mon[102221]: osdmap e171: 8 total, 7 up, 8 in 2024-09-21T10:03:20.504 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:20 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/168636 objects degraded (0.009%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:20.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:20.569 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:20.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:20.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:20.587 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:03:20.587 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:03:20.622 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:03:20.623 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:03:20.660 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:03:20.660 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:03:20.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:20 smithi129 ceph-mon[88471]: pgmap v604: 109 pgs: 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 968 op/s; 15/168636 objects degraded (0.009%); 31463/168636 objects misplaced (18.657%) 2024-09-21T10:03:20.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:20 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-21T10:03:20.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:20 smithi129 ceph-mon[88471]: osdmap e171: 8 total, 7 up, 8 in 2024-09-21T10:03:20.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:20 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/168636 objects degraded (0.009%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:03:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:03:20.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:20.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:20.702 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:03:20.702 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:03:20.712 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:20.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:20.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:20.731 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:03:20.731 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:03:20.766 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:03:20.766 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:03:20.801 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:03:20.801 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:03:20.801 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T10:03:20.815 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:20.839 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:20.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:20.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:20.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:20.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:20.979 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:20.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:21.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:21.118 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:21.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:21.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:21.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:21.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:21.273 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:21.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:21.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:21 smithi107 ceph-mon[102221]: osdmap e172: 8 total, 7 up, 8 in 2024-09-21T10:03:21.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:21 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3244196122' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:03:21.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:21 smithi107 ceph-mon[105022]: osdmap e172: 8 total, 7 up, 8 in 2024-09-21T10:03:21.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:21 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3244196122' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:03:21.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:21.424 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:21.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:21.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:21.530 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:21.556 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:21.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:21.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:21.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:21.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:21 smithi129 ceph-mon[88471]: osdmap e172: 8 total, 7 up, 8 in 2024-09-21T10:03:21.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:21 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3244196122' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:03:21.692 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:21.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:21.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:21.734 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:21.771 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:03:21.771 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:03:21.794 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:21.818 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:21.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:21.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:21.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:21.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:21.964 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:21.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:22.045 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:03:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:03:21] "GET /metrics HTTP/1.1" 200 36663 "" "Prometheus/2.51.0" 2024-09-21T10:03:22.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:22.101 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:22.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:22.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:22.203 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:22.229 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:22.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:22.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:22.331 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:22.356 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:22.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:22.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:22.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:22 smithi107 ceph-mon[102221]: 9.0 scrub starts 2024-09-21T10:03:22.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:22 smithi107 ceph-mon[102221]: pgmap v607: 119 pgs: 10 unknown, 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s; 15/160611 objects degraded (0.009%); 30124/160611 objects misplaced (18.756%) 2024-09-21T10:03:22.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:22 smithi107 ceph-mon[102221]: 9.0 scrub ok 2024-09-21T10:03:22.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:22 smithi107 ceph-mon[105022]: 9.0 scrub starts 2024-09-21T10:03:22.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:22 smithi107 ceph-mon[105022]: pgmap v607: 119 pgs: 10 unknown, 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s; 15/160611 objects degraded (0.009%); 30124/160611 objects misplaced (18.756%) 2024-09-21T10:03:22.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:22 smithi107 ceph-mon[105022]: 9.0 scrub ok 2024-09-21T10:03:22.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:22 smithi129 ceph-mon[88471]: 9.0 scrub starts 2024-09-21T10:03:22.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:22 smithi129 ceph-mon[88471]: pgmap v607: 119 pgs: 10 unknown, 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s; 15/160611 objects degraded (0.009%); 30124/160611 objects misplaced (18.756%) 2024-09-21T10:03:22.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:22 smithi129 ceph-mon[88471]: 9.0 scrub ok 2024-09-21T10:03:22.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:22.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:22.496 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:22.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:22.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:22.648 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:22.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:22.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:22.751 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:22.780 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:22.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:22.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:22.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:22.916 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:22.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:22.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:23.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:23.064 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:23.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:23.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:23.166 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:23.192 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:23.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:23.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:23.294 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:23.321 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:23.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:23.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:23.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:23.450 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:23.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:23.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:23.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:23.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:23.577 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:23.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:23.678 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:23.703 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:23.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:23.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:23.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:03:23.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:23.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:23.856 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:23.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:23.957 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:24.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:24.028 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:24.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:24.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:24.170 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:24.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:24.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:24.271 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:24.296 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:24.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:24.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:24.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:24 smithi107 ceph-mon[102221]: pgmap v608: 119 pgs: 10 unknown, 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 296 op/s; 15/155763 objects degraded (0.010%); 29203/155763 objects misplaced (18.748%) 2024-09-21T10:03:24.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:24 smithi107 ceph-mon[105022]: pgmap v608: 119 pgs: 10 unknown, 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 296 op/s; 15/155763 objects degraded (0.010%); 29203/155763 objects misplaced (18.748%) 2024-09-21T10:03:24.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:24.435 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:24.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:24.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:24.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:24.575 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:24.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:24.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:24.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:24 smithi129 ceph-mon[88471]: pgmap v608: 119 pgs: 10 unknown, 14 active+clean+remapped, 30 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 296 op/s; 15/155763 objects degraded (0.010%); 29203/155763 objects misplaced (18.748%) 2024-09-21T10:03:24.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:24.712 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:24.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:24.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:24.805 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-09-21T10:03:24.806 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-09-21T10:03:24.806 INFO:tasks.cephadm.osd.7:Restarting osd.7 (starting--it wasn't running)... 2024-09-21T10:03:24.806 DEBUG:teuthology.orchestra.run.smithi129:osd.7> sudo journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.7.service 2024-09-21T10:03:24.809 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl start ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.7 2024-09-21T10:03:24.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:24.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:24.855 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:24.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:24.956 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:24.987 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:24.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:24.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:25.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:25.115 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:25.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:25.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:25.169 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:24 smithi129 systemd[1]: Starting Ceph osd.7 for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:03:25.216 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:25.241 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:25.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:25.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:25.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:25.379 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:25.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:25.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:25.481 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:25.513 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:25.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:25.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:25.562 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:25 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/155763 objects degraded (0.010%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:25.562 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:25 smithi129 podman[112444]: 2024-09-21 10:03:25.078502951 +0000 UTC m=+0.024019821 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:03:25.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:25 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/155763 objects degraded (0.010%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:25.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:25 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/155763 objects degraded (0.010%), 10 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:25.615 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:25.643 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:25.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:25.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:25.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:25.781 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:25.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:25.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:25.816 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:25 smithi129 podman[112444]: 2024-09-21 10:03:25.560863286 +0000 UTC m=+0.506380146 container create 26d8224ae8a37b7609ca972686be8b881cf803d853a6373fe2dc6f010536ff57 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-activate, RELEASE=quincy-fbd1a6a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux ) 2024-09-21T10:03:25.816 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:25 smithi129 podman[112444]: 2024-09-21 10:03:25.814234511 +0000 UTC m=+0.759751365 container init 26d8224ae8a37b7609ca972686be8b881cf803d853a6373fe2dc6f010536ff57 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-activate, CEPH_POINT_RELEASE=, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True) 2024-09-21T10:03:25.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:25.928 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:25.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:25.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:26.029 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:26.056 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:26.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:26.138 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:25 smithi129 podman[112444]: 2024-09-21 10:03:25.818890405 +0000 UTC m=+0.764407263 container start 26d8224ae8a37b7609ca972686be8b881cf803d853a6373fe2dc6f010536ff57 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-activate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a) 2024-09-21T10:03:26.138 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:25 smithi129 podman[112444]: 2024-09-21 10:03:25.922350291 +0000 UTC m=+0.867867151 container attach 26d8224ae8a37b7609ca972686be8b881cf803d853a6373fe2dc6f010536ff57 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-activate, RELEASE=quincy-fbd1a6a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-21T10:03:26.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:26.197 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:26.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:26.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:26.299 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:26.327 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:26.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:26.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:26.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:26 smithi107 ceph-mon[102221]: pgmap v609: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 15/150012 objects degraded (0.010%); 28223/150012 objects misplaced (18.814%) 2024-09-21T10:03:26.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:26.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:26 smithi107 ceph-mon[105022]: pgmap v609: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 15/150012 objects degraded (0.010%); 28223/150012 objects misplaced (18.814%) 2024-09-21T10:03:26.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:26.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:26 smithi129 ceph-mon[88471]: pgmap v609: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 15/150012 objects degraded (0.010%); 28223/150012 objects misplaced (18.814%) 2024-09-21T10:03:26.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:26.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-activate[112510]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-21T10:03:26.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:26 smithi129 bash[112444]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-21T10:03:26.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-activate[112510]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-21T10:03:26.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:26 smithi129 bash[112444]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-21T10:03:26.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-activate[112510]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-21T10:03:26.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:26 smithi129 bash[112444]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-21T10:03:26.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-activate[112510]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-21T10:03:26.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:26 smithi129 bash[112444]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-21T10:03:26.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-activate[112510]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-21T10:03:26.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:26 smithi129 bash[112444]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-21T10:03:26.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-activate[112510]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-21T10:03:26.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:26 smithi129 bash[112444]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-21T10:03:26.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-activate[112510]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-21T10:03:26.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:26 smithi129 bash[112444]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-21T10:03:26.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:26 smithi129 podman[112444]: 2024-09-21 10:03:26.193140393 +0000 UTC m=+1.138657257 container died 26d8224ae8a37b7609ca972686be8b881cf803d853a6373fe2dc6f010536ff57 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-activate, RELEASE=quincy-fbd1a6a, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS) 2024-09-21T10:03:26.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:26.470 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:26.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:26.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:26.572 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:26.593 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:26.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:26.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:26.694 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:26.730 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:26.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:26.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:26.832 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:26.859 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:26.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:26.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:26.912 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:03:26.912 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:03:26.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.913 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.45761.0:173727 9.c e172/153 9:30870b2d:::benchmark_data_smithi107_152010_object63790:head v 172'11001, mlcod=172'10998)", 2024-09-21T10:03:26.913 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:03:26.904134+0000", 2024-09-21T10:03:26.913 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.005418735, 2024-09-21T10:03:26.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054466100000000002, 2024-09-21T10:03:26.913 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:03:26.913 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:03:26.913 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:03:26.913 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:03:26.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:03:26.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.904134+0000", 2024-09-21T10:03:26.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:03:26.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:03:26.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.904134+0000", 2024-09-21T10:03:26.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:03:26.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:03:26.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.904135+0000", 2024-09-21T10:03:26.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9800000000000002e-07 2024-09-21T10:03:26.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:03:26.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.904142+0000", 2024-09-21T10:03:26.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1289999999999998e-06 2024-09-21T10:03:26.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:03:26.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.904158+0000", 2024-09-21T10:03:26.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6344000000000001e-05 2024-09-21T10:03:26.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.916 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:03:26.916 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.904167+0000", 2024-09-21T10:03:26.916 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1410000000000005e-06 2024-09-21T10:03:26.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.916 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:03:26.916 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.908312+0000", 2024-09-21T10:03:26.916 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041456799999999997 2024-09-21T10:03:26.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.916 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:03:26.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.908335+0000", 2024-09-21T10:03:26.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2653000000000001e-05 2024-09-21T10:03:26.917 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:03:26.917 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:03:26.917 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:03:26.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.917 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.45761.0:173743 9.e e172/153 9:7a8ed31e:::benchmark_data_smithi107_152010_object63806:head v 172'10779, mlcod=172'10777)", 2024-09-21T10:03:26.917 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:03:26.905133+0000", 2024-09-21T10:03:26.917 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0044201120000000003, 2024-09-21T10:03:26.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045022589999999998, 2024-09-21T10:03:26.917 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:03:26.917 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:03:26.918 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:03:26.918 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:03:26.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.918 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:03:26.918 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.905133+0000", 2024-09-21T10:03:26.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:03:26.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.918 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:03:26.918 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.905133+0000", 2024-09-21T10:03:26.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:03:26.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.918 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:03:26.919 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.905133+0000", 2024-09-21T10:03:26.919 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5499999999999997e-07 2024-09-21T10:03:26.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.919 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.919 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:03:26.919 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.905140+0000", 2024-09-21T10:03:26.919 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2609999999999999e-06 2024-09-21T10:03:26.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.919 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.919 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:03:26.919 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.905155+0000", 2024-09-21T10:03:26.919 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5068999999999999e-05 2024-09-21T10:03:26.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.919 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.920 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:03:26.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.905162+0000", 2024-09-21T10:03:26.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.934e-06 2024-09-21T10:03:26.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.920 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.920 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:03:26.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.905234+0000", 2024-09-21T10:03:26.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2384000000000004e-05 2024-09-21T10:03:26.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.920 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.920 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:03:26.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.905259+0000", 2024-09-21T10:03:26.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5020000000000001e-05 2024-09-21T10:03:26.920 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:03:26.921 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:03:26.921 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:03:26.921 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.921 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45761.0:173738 9.5 9:a46053e7:::benchmark_data_smithi107_152010_object63801:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-09-21T10:03:26.921 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:03:26.898774+0000", 2024-09-21T10:03:26.921 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.010778995, 2024-09-21T10:03:26.921 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01091396, 2024-09-21T10:03:26.921 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:03:26.921 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:03:26.921 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:03:26.921 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:03:26.921 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45761", 2024-09-21T10:03:26.921 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:03:26.922 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 173738 2024-09-21T10:03:26.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.922 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:03:26.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.922 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:03:26.922 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.898774+0000", 2024-09-21T10:03:26.922 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:03:26.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.922 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:03:26.922 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.898774+0000", 2024-09-21T10:03:26.922 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:03:26.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:03:26.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.898776+0000", 2024-09-21T10:03:26.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.717e-06 2024-09-21T10:03:26.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:03:26.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.898785+0000", 2024-09-21T10:03:26.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3000000000000002e-06 2024-09-21T10:03:26.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:03:26.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.898807+0000", 2024-09-21T10:03:26.924 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.179e-05 2024-09-21T10:03:26.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.924 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:03:26.924 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.898817+0000", 2024-09-21T10:03:26.924 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0081000000000001e-05 2024-09-21T10:03:26.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.924 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:03:26.924 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.898893+0000", 2024-09-21T10:03:26.924 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6600999999999994e-05 2024-09-21T10:03:26.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.924 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:03:26.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.899052+0000", 2024-09-21T10:03:26.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015847699999999999 2024-09-21T10:03:26.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.925 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:03:26.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.899500+0000", 2024-09-21T10:03:26.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044858300000000001 2024-09-21T10:03:26.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.925 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:26.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.902371+0000", 2024-09-21T10:03:26.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028707049999999999 2024-09-21T10:03:26.925 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:26.925 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:26.926 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:03:26.926 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:26.902454+0000", 2024-09-21T10:03:26.926 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2688000000000005e-05 2024-09-21T10:03:26.926 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:03:26.926 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:03:26.926 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:03:26.926 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:03:26.926 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-09-21T10:03:26.926 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 3 2024-09-21T10:03:26.926 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:03:26.961 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:27.001 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:27.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:27.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:27.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:27.131 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:26 smithi129 podman[112670]: 2024-09-21 10:03:26.86177528 +0000 UTC m=+0.662131303 container remove 26d8224ae8a37b7609ca972686be8b881cf803d853a6373fe2dc6f010536ff57 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-activate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2) 2024-09-21T10:03:27.131 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:27 smithi129 podman[112784]: 2024-09-21 10:03:27.002817093 +0000 UTC m=+0.019810906 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:03:27.131 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:27 smithi129 podman[112784]: 2024-09-21 10:03:27.129769718 +0000 UTC m=+0.146763511 container create 0c874265bc9e1d7ff6f45c9ced6c6a97bf293d1e305476f344d830c03fc5f31d (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-21T10:03:27.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:27.150 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:27.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:27.251 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:27.290 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:27.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:27.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:27.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:27.395 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:27 smithi129 podman[112784]: 2024-09-21 10:03:27.366098475 +0000 UTC m=+0.383092270 container init 0c874265bc9e1d7ff6f45c9ced6c6a97bf293d1e305476f344d830c03fc5f31d (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=quincy-fbd1a6a, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-21T10:03:27.396 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:27 smithi129 podman[112784]: 2024-09-21 10:03:27.372946088 +0000 UTC m=+0.389939884 container start 0c874265bc9e1d7ff6f45c9ced6c6a97bf293d1e305476f344d830c03fc5f31d (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-21T10:03:27.427 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:27.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:27.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:27.483 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:03:27.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:27.570 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:27.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:27.669 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:27 smithi129 ceph-osd[112820]: -- 172.21.15.129:0/2665980798 <== mon.1 v2:172.21.15.129:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a63705ad00 con 0x55a63702d000 2024-09-21T10:03:27.669 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:27 smithi129 bash[112784]: 0c874265bc9e1d7ff6f45c9ced6c6a97bf293d1e305476f344d830c03fc5f31d 2024-09-21T10:03:27.669 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:27 smithi129 systemd[1]: Started Ceph osd.7 for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T10:03:27.672 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:27.712 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:27.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:27.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:03:27.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:27.826 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:03:27.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:27.878 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:27.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:27.979 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:28.013 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:28.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:28.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:28.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:28.162 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:28.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:28.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:28.169 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:27 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:03:27.908+0000 7f61644c7440 -1 Falling back to public interface 2024-09-21T10:03:28.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:28.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:28.320 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:28.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:28.359 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:28.423 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:28.437 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:28.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:28.482 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:28.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:28.553 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:28 smithi107 ceph-mon[102221]: pgmap v610: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 15/145509 objects degraded (0.010%); 27351/145509 objects misplaced (18.797%) 2024-09-21T10:03:28.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:28 smithi107 ceph-mon[105022]: pgmap v610: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 15/145509 objects degraded (0.010%); 27351/145509 objects misplaced (18.797%) 2024-09-21T10:03:28.583 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:28.608 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:28.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:28.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:28.632 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:28 smithi129 ceph-mon[88471]: pgmap v610: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 15/145509 objects degraded (0.010%); 27351/145509 objects misplaced (18.797%) 2024-09-21T10:03:28.709 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:28.754 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:28.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:28.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:28.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:28.883 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:28.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:28.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:28.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:29.028 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:29.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:29.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:29.130 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:29.155 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:29.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:29.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:29.257 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:29.284 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:29.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:29.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:29.287 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:03:29.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:29.425 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:29.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:29.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:29.527 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:29.554 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:29.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:29.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:29.595 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:29 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2008486684' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:03:29.596 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:29 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:03:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:29 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2008486684' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:03:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:29 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:03:29.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:29 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2008486684' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:03:29.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:29 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:03:29.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:29.690 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:29.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:29.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:29.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:29.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:29.827 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:29.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:29.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:29.949 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:29.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:29.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:30.051 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:30.087 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:30.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:30.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:30.182 INFO:teuthology.orchestra.run.smithi129.stderr:no valid command found; 10 closest matches: 2024-09-21T10:03:30.182 INFO:teuthology.orchestra.run.smithi129.stderr:0 2024-09-21T10:03:30.182 INFO:teuthology.orchestra.run.smithi129.stderr:1 2024-09-21T10:03:30.182 INFO:teuthology.orchestra.run.smithi129.stderr:2 2024-09-21T10:03:30.182 INFO:teuthology.orchestra.run.smithi129.stderr:abort 2024-09-21T10:03:30.182 INFO:teuthology.orchestra.run.smithi129.stderr:assert 2024-09-21T10:03:30.182 INFO:teuthology.orchestra.run.smithi129.stderr:config diff 2024-09-21T10:03:30.182 INFO:teuthology.orchestra.run.smithi129.stderr:config diff get 2024-09-21T10:03:30.182 INFO:teuthology.orchestra.run.smithi129.stderr:config get 2024-09-21T10:03:30.182 INFO:teuthology.orchestra.run.smithi129.stderr:config help [] 2024-09-21T10:03:30.182 INFO:teuthology.orchestra.run.smithi129.stderr:config set ... 2024-09-21T10:03:30.182 INFO:teuthology.orchestra.run.smithi129.stderr:admin_socket: invalid command 2024-09-21T10:03:30.189 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:30.218 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:30.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:30.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:30.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:30.360 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:30.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:30.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:30.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:30.499 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:30.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:30.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:30.513 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:30 smithi129 ceph-mon[88471]: pgmap v611: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 15/138312 objects degraded (0.011%); 25911/138312 objects misplaced (18.734%) 2024-09-21T10:03:30.513 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:30 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:03:30.514 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:30 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:03:30.514 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:30 smithi129 ceph-mon[88471]: osdmap e173: 8 total, 7 up, 8 in 2024-09-21T10:03:30.514 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:30 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/138312 objects degraded (0.011%), 10 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:30.518 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:03:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:03:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:30.547 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:30 smithi107 ceph-mon[102221]: pgmap v611: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 15/138312 objects degraded (0.011%); 25911/138312 objects misplaced (18.734%) 2024-09-21T10:03:30.547 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:30 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:03:30.547 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:30 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:03:30.547 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:30 smithi107 ceph-mon[102221]: osdmap e173: 8 total, 7 up, 8 in 2024-09-21T10:03:30.547 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:30 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/138312 objects degraded (0.011%), 10 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:30.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:30 smithi107 ceph-mon[105022]: pgmap v611: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 15/138312 objects degraded (0.011%); 25911/138312 objects misplaced (18.734%) 2024-09-21T10:03:30.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:30 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:03:30.548 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:30 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:03:30.548 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:30 smithi107 ceph-mon[105022]: osdmap e173: 8 total, 7 up, 8 in 2024-09-21T10:03:30.548 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:30 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/138312 objects degraded (0.011%), 10 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:30.601 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:30.639 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:30.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:30.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:30.740 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:30.769 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:30.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:30.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:30.859 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:03:30.859 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-09-21T10:03:30.872 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:30.899 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:30.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:30.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:31.000 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:31.035 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:31.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:31.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:31.136 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:31.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:31.171 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:31.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:31.272 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:31.310 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:31.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:31.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:31.411 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:31.450 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:31.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:31.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:31.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:31.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:31.589 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:31.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:31.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:31.725 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:31.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:31.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:31.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:31.863 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:31.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:31.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:31.965 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:31.995 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:31.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:31.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:32.020 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:03:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:03:31] "GET /metrics HTTP/1.1" 200 36663 "" "Prometheus/2.51.0" 2024-09-21T10:03:32.096 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:32.126 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:32.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:32.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:32.228 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:32.257 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:32.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:32.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:32.358 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:32.387 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:32.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:32.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:32.489 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:32.525 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:32.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:32.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:32.575 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:32 smithi107 ceph-mon[102221]: pgmap v613: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 15/130935 objects degraded (0.011%); 24656/130935 objects misplaced (18.831%) 2024-09-21T10:03:32.575 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:32 smithi107 ceph-mon[105022]: pgmap v613: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 15/130935 objects degraded (0.011%); 24656/130935 objects misplaced (18.831%) 2024-09-21T10:03:32.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:32.667 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:32.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:32.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:32.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:32 smithi129 ceph-mon[88471]: pgmap v613: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 15/130935 objects degraded (0.011%); 24656/130935 objects misplaced (18.831%) 2024-09-21T10:03:32.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:32.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:32.804 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:32.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:32.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:32.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:32.963 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:32.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:33.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:33.106 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:33.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:33.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:33.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:33.258 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:33.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:33.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:33.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:33.401 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:33.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:33.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:33.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:33.539 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:33.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:33.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:33.641 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:33.669 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:33.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:33.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:33.771 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:33.797 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:33.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:33.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:33.822 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:03:33.822 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:03:33.890 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:03:33.890 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:03:33.890 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:03:33.890 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:03:33.891 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:03:33.891 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:03:33.891 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:03:33.891 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:03:33.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:33.915 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:03:33.915 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:03:33.930 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:33.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:33.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:33.959 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:03:33.960 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:03:33.992 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:03:33.992 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:03:33.994 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T10:03:34.032 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:34.058 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:34.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:34.160 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:34.190 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:34.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:34.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:34.292 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:34.321 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:34.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:34.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:34.423 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:34.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:34.465 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:34.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:34.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:34.599 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:34.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:34.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:34.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:34 smithi107 ceph-mon[102221]: pgmap v614: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 452 op/s; 15/126207 objects degraded (0.012%); 23710/126207 objects misplaced (18.787%) 2024-09-21T10:03:34.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:34 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2058619224' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:03:34.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:34 smithi107 ceph-mon[105022]: pgmap v614: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 452 op/s; 15/126207 objects degraded (0.012%); 23710/126207 objects misplaced (18.787%) 2024-09-21T10:03:34.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:34 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2058619224' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:03:34.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:34 smithi129 ceph-mon[88471]: pgmap v614: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 452 op/s; 15/126207 objects degraded (0.012%); 23710/126207 objects misplaced (18.787%) 2024-09-21T10:03:34.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:34 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2058619224' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:03:34.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:34.740 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:34.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:34.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:34.805 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:03:34.806 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:03:34.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:34.881 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:34.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:34.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:03:34.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:35.069 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:35.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:35.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:35.171 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:35.199 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:35.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:35.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:35.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:35.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:03:35.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:35.365 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:35.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:35.466 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:35.473 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:35.492 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:35.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:35.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:35.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:35.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:35 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 15/126207 objects degraded (0.012%), 10 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:35.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:35 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 15/126207 objects degraded (0.012%), 10 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:35.623 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:35 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 15/126207 objects degraded (0.012%), 10 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:35.634 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:35.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:35.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:35.736 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:35.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:35.786 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:35.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:35.861 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:03:35.887 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:35.888 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:35 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:03:35.620+0000 7f61644c7440 -1 osd.7 151 log_to_monitors true 2024-09-21T10:03:35.913 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:35.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:35.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:36.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:36.053 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:36.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:36.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:36.154 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:36.186 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:36.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:36.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:36.206 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:03:36.288 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:36.315 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:36.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:36.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:36.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:36.463 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:36.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:36.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:36.565 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:36.604 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:36.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:36.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:36.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:36 smithi107 ceph-mon[102221]: pgmap v615: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 989 op/s; 15/120309 objects degraded (0.012%); 22726/120309 objects misplaced (18.890%) 2024-09-21T10:03:36.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:36 smithi107 ceph-mon[102221]: from='osd.7 [v2:172.21.15.129:6824/702981256,v1:172.21.15.129:6825/702981256]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T10:03:36.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:36 smithi107 ceph-mon[102221]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T10:03:36.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:36 smithi107 ceph-mon[105022]: pgmap v615: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 989 op/s; 15/120309 objects degraded (0.012%); 22726/120309 objects misplaced (18.890%) 2024-09-21T10:03:36.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:36 smithi107 ceph-mon[105022]: from='osd.7 [v2:172.21.15.129:6824/702981256,v1:172.21.15.129:6825/702981256]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T10:03:36.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:36 smithi107 ceph-mon[105022]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T10:03:36.636 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:36 smithi129 ceph-mon[88471]: pgmap v615: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 2.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 989 op/s; 15/120309 objects degraded (0.012%); 22726/120309 objects misplaced (18.890%) 2024-09-21T10:03:36.636 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:36 smithi129 ceph-mon[88471]: from='osd.7 [v2:172.21.15.129:6824/702981256,v1:172.21.15.129:6825/702981256]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T10:03:36.636 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:36 smithi129 ceph-mon[88471]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T10:03:36.636 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:03:36 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:03:36.334+0000 7f6159a6e640 -1 osd.7 151 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T10:03:36.706 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:36.747 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:36.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:36.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:36.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:03:36.848 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:36.854 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:36.872 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:36.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:36.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:36.974 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:37.023 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:37.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:37.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:37.124 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:37.155 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:37.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:37.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:37.257 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:37.280 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:37.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:37.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:37.382 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:37.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:37.406 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:37.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:37.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:37 smithi129 ceph-mon[88471]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T10:03:37.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:37 smithi129 ceph-mon[88471]: osdmap e174: 8 total, 7 up, 8 in 2024-09-21T10:03:37.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:37 smithi129 ceph-mon[88471]: from='osd.7 [v2:172.21.15.129:6824/702981256,v1:172.21.15.129:6825/702981256]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T10:03:37.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:37 smithi129 ceph-mon[88471]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T10:03:37.461 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:37 smithi107 ceph-mon[102221]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T10:03:37.461 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:37 smithi107 ceph-mon[102221]: osdmap e174: 8 total, 7 up, 8 in 2024-09-21T10:03:37.461 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:37 smithi107 ceph-mon[102221]: from='osd.7 [v2:172.21.15.129:6824/702981256,v1:172.21.15.129:6825/702981256]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T10:03:37.461 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:37 smithi107 ceph-mon[102221]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T10:03:37.462 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:37 smithi107 ceph-mon[105022]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T10:03:37.462 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:37 smithi107 ceph-mon[105022]: osdmap e174: 8 total, 7 up, 8 in 2024-09-21T10:03:37.462 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:37 smithi107 ceph-mon[105022]: from='osd.7 [v2:172.21.15.129:6824/702981256,v1:172.21.15.129:6825/702981256]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T10:03:37.462 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:37 smithi107 ceph-mon[105022]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi129", "root=default"]} : dispatch 2024-09-21T10:03:37.507 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:37.529 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:37.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:37.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:37.631 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:37.652 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:37.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:37.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:37.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:37.776 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:37.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:37.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:37.877 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:37.899 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:37.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:37.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:37.933 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:03:37.933 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:03:37.933 INFO:teuthology.orchestra.run.smithi129.stdout: "num_ops": 0 2024-09-21T10:03:37.934 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:03:38.000 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:38.021 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:38.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:38.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:38.112 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:38.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:38.144 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:38.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:38.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:38.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:38.268 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:38.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:38.341 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:03:38.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:38 smithi107 ceph-mon[102221]: pgmap v617: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 964 op/s; 15/115149 objects degraded (0.013%); 21699/115149 objects misplaced (18.844%) 2024-09-21T10:03:38.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:38 smithi107 ceph-mon[102221]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:03:38.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:38 smithi107 ceph-mon[102221]: osd.7 [v2:172.21.15.129:6824/702981256,v1:172.21.15.129:6825/702981256] boot 2024-09-21T10:03:38.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:38 smithi107 ceph-mon[102221]: osdmap e175: 8 total, 8 up, 8 in 2024-09-21T10:03:38.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T10:03:38.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:38 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3720579862' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:03:38.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:38 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:03:38.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:38 smithi107 ceph-mon[105022]: pgmap v617: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 964 op/s; 15/115149 objects degraded (0.013%); 21699/115149 objects misplaced (18.844%) 2024-09-21T10:03:38.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:38 smithi107 ceph-mon[105022]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:03:38.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:38 smithi107 ceph-mon[105022]: osd.7 [v2:172.21.15.129:6824/702981256,v1:172.21.15.129:6825/702981256] boot 2024-09-21T10:03:38.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:38 smithi107 ceph-mon[105022]: osdmap e175: 8 total, 8 up, 8 in 2024-09-21T10:03:38.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T10:03:38.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:38 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3720579862' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:03:38.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:38 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:03:38.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:38.396 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:38.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:38.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:38.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:38.523 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:38.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:38.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:38.625 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:38.649 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:38.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:38.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:38.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:38 smithi129 ceph-mon[88471]: pgmap v617: 119 pgs: 14 active+clean+remapped, 40 active+undersized, 10 active+undersized+degraded, 55 active+clean; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 964 op/s; 15/115149 objects degraded (0.013%); 21699/115149 objects misplaced (18.844%) 2024-09-21T10:03:38.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:38 smithi129 ceph-mon[88471]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:03:38.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:38 smithi129 ceph-mon[88471]: osd.7 [v2:172.21.15.129:6824/702981256,v1:172.21.15.129:6825/702981256] boot 2024-09-21T10:03:38.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:38 smithi129 ceph-mon[88471]: osdmap e175: 8 total, 8 up, 8 in 2024-09-21T10:03:38.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T10:03:38.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:38 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3720579862' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:03:38.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:38 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:03:38.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:38.776 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:38.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:38.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:38.877 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:38.900 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:38.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:38.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:39.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:39.026 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:39.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:39.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:39.149 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:39.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:39.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:39.251 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:39.274 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:39.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:39.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:39.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:39.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:39.413 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:39.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:39.514 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:39.535 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:39.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:39.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:39.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T10:03:39.597 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:39 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:03:39.598 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:39 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:03:39.598 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:39 smithi107 ceph-mon[102221]: 9.c starting backfill to osd.7 from (0'0,151'2952] MAX to 174'11695 2024-09-21T10:03:39.598 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:39 smithi107 ceph-mon[102221]: osdmap e176: 8 total, 8 up, 8 in 2024-09-21T10:03:39.598 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:39 smithi107 ceph-mon[102221]: 9.d starting backfill to osd.7 from (0'0,151'3016] MAX to 174'11772 2024-09-21T10:03:39.598 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:39 smithi107 ceph-mon[102221]: 9.a starting backfill to osd.7 from (0'0,151'2939] MAX to 174'11310 2024-09-21T10:03:39.598 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:39 smithi107 ceph-mon[102221]: 9.e starting backfill to osd.7 from (0'0,151'2966] MAX to 174'11446 2024-09-21T10:03:39.598 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:39 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:03:39.598 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:39 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:03:39.598 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:39 smithi107 ceph-mon[105022]: 9.c starting backfill to osd.7 from (0'0,151'2952] MAX to 174'11695 2024-09-21T10:03:39.598 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:39 smithi107 ceph-mon[105022]: osdmap e176: 8 total, 8 up, 8 in 2024-09-21T10:03:39.599 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:39 smithi107 ceph-mon[105022]: 9.d starting backfill to osd.7 from (0'0,151'3016] MAX to 174'11772 2024-09-21T10:03:39.599 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:39 smithi107 ceph-mon[105022]: 9.a starting backfill to osd.7 from (0'0,151'2939] MAX to 174'11310 2024-09-21T10:03:39.599 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:39 smithi107 ceph-mon[105022]: 9.e starting backfill to osd.7 from (0'0,151'2966] MAX to 174'11446 2024-09-21T10:03:39.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:39.664 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:39.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:39.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:39 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:03:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:39 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:03:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:39 smithi129 ceph-mon[88471]: 9.c starting backfill to osd.7 from (0'0,151'2952] MAX to 174'11695 2024-09-21T10:03:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:39 smithi129 ceph-mon[88471]: osdmap e176: 8 total, 8 up, 8 in 2024-09-21T10:03:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:39 smithi129 ceph-mon[88471]: 9.d starting backfill to osd.7 from (0'0,151'3016] MAX to 174'11772 2024-09-21T10:03:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:39 smithi129 ceph-mon[88471]: 9.a starting backfill to osd.7 from (0'0,151'2939] MAX to 174'11310 2024-09-21T10:03:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:39 smithi129 ceph-mon[88471]: 9.e starting backfill to osd.7 from (0'0,151'2966] MAX to 174'11446 2024-09-21T10:03:39.768 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:39.801 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:39.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:39.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:39.902 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:39.944 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:39.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:39.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:40.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:40.086 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:40.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:40.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:40.103 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:03:40.103 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:03:40.188 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:40.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:40.232 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:40.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:40.333 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:40.376 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:40.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:40.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:40.478 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:40.517 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:40.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:40.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:40.538 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:40 smithi129 ceph-mon[88471]: pgmap v620: 119 pgs: 7 remapped+peering, 4 peering, 7 active+clean+remapped, 32 active+undersized, 9 active+undersized+degraded, 60 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 963 op/s; 14/108858 objects degraded (0.013%); 11578/108858 objects misplaced (10.636%) 2024-09-21T10:03:40.538 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:40 smithi129 ceph-mon[88471]: osdmap e177: 8 total, 8 up, 8 in 2024-09-21T10:03:40.538 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:40 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 14/108858 objects degraded (0.013%), 9 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:40.539 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:03:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:03:40.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:40 smithi107 ceph-mon[102221]: pgmap v620: 119 pgs: 7 remapped+peering, 4 peering, 7 active+clean+remapped, 32 active+undersized, 9 active+undersized+degraded, 60 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 963 op/s; 14/108858 objects degraded (0.013%); 11578/108858 objects misplaced (10.636%) 2024-09-21T10:03:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:40 smithi107 ceph-mon[102221]: osdmap e177: 8 total, 8 up, 8 in 2024-09-21T10:03:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:40 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 14/108858 objects degraded (0.013%), 9 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:40.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:40 smithi107 ceph-mon[105022]: pgmap v620: 119 pgs: 7 remapped+peering, 4 peering, 7 active+clean+remapped, 32 active+undersized, 9 active+undersized+degraded, 60 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 963 op/s; 14/108858 objects degraded (0.013%); 11578/108858 objects misplaced (10.636%) 2024-09-21T10:03:40.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:40 smithi107 ceph-mon[105022]: osdmap e177: 8 total, 8 up, 8 in 2024-09-21T10:03:40.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:40 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 14/108858 objects degraded (0.013%), 9 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-09-21T10:03:40.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:40.650 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:40.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:40.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:40.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:40.783 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:40.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:40.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:40.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:03:40.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:40.951 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:40.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:40.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:41.053 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:41.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:41.097 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:41.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:41.198 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:41.241 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:41.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:41.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:41.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:41.354 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:41.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:41.405 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:41.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:41.505 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:41.540 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:41.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:41.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:41.642 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:41.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:41.680 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:41.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:41.781 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:41.822 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:41.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:41.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:41.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:41.963 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:41.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:41.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:42.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:42.093 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:03:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:03:41] "GET /metrics HTTP/1.1" 200 36525 "" "Prometheus/2.51.0" 2024-09-21T10:03:42.105 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:42.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:42.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:42.207 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:42.252 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:42.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:42.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:42.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:42.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:42 smithi107 ceph-mon[102221]: pgmap v622: 119 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 7 remapped+peering, 4 peering, 1 active+clean+remapped, 2 active+remapped+backfill_wait, 26 active+undersized, 8 active+undersized+degraded, 67 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 13637/105138 objects degraded (12.971%); 6201/105138 objects misplaced (5.898%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:03:42.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:42.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:42.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:42 smithi107 ceph-mon[105022]: pgmap v622: 119 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 7 remapped+peering, 4 peering, 1 active+clean+remapped, 2 active+remapped+backfill_wait, 26 active+undersized, 8 active+undersized+degraded, 67 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 13637/105138 objects degraded (12.971%); 6201/105138 objects misplaced (5.898%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:03:42.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:42.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:42.397 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:42.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:42.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:42 smithi129 ceph-mon[88471]: pgmap v622: 119 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 7 remapped+peering, 4 peering, 1 active+clean+remapped, 2 active+remapped+backfill_wait, 26 active+undersized, 8 active+undersized+degraded, 67 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 13637/105138 objects degraded (12.971%); 6201/105138 objects misplaced (5.898%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:03:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:03:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:42.498 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:42.534 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:42.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:42.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:42.613 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:03:42.613 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:03:42.613 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:03:42.613 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:03:42.613 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:03:42.613 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:03:42.613 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:03:42.613 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:03:42.614 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:03:42.614 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:03:42.621 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:42.636 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:42.640 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:03:42.640 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:03:42.667 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:42.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:42.671 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:03:42.671 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:03:42.713 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:03:42.714 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:03:42.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:42.809 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:42.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:42.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:42.911 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:42.953 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:42.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:42.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:43.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:43.099 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:43.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:43.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:43.200 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:43.221 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:43.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:43.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:43.323 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:43.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:43 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1998787595' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:03:43.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:43 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1998787595' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:03:43.363 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:43.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:43.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:43.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:43 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1998787595' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:03:43.466 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:43.505 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:43.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:43.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:43.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:43.643 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:43.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:43.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:43.745 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:43.780 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:43.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:43.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:43.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:03:43.882 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:43.919 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:43.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:43.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:44.021 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:44.061 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:44.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:44.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:44.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:44.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:44.199 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:44.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:44.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:44.338 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:44.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:44.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:44.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:44 smithi107 ceph-mon[102221]: pgmap v623: 119 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 remapped+peering, 4 active+remapped+backfill_wait, 105 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 17023/102630 objects degraded (16.587%); 9114/102630 objects misplaced (8.880%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:03:44.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:44 smithi107 ceph-mon[105022]: pgmap v623: 119 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 remapped+peering, 4 active+remapped+backfill_wait, 105 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 17023/102630 objects degraded (16.587%); 9114/102630 objects misplaced (8.880%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:03:44.440 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:44.479 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:44.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:44.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:44.581 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:44.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:44.617 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:44.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:44 smithi129 ceph-mon[88471]: pgmap v623: 119 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 4 remapped+peering, 4 active+remapped+backfill_wait, 105 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 17023/102630 objects degraded (16.587%); 9114/102630 objects misplaced (8.880%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:03:44.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:44.761 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:44.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:44.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:44.863 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:44.900 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:44.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:44.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:45.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:45.033 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:45.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:45.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:45.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:45.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:45.183 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:45.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:45.284 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:45.315 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:45.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:45.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:45 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 17023/102630 objects degraded (16.587%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:03:45.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:45 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 17023/102630 objects degraded (16.587%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:03:45.416 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:45.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:45.454 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:45.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:45.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:45.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:45.594 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:45.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:45 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 17023/102630 objects degraded (16.587%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:03:45.694 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:45.731 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:45.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:45.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:45.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:45.865 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:45.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:45.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:45.966 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:46.001 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:46.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:46.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:46.084 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:46.103 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:46.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:03:46.134 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:46.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:46.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:46.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:46.279 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:46.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:46.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:46.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:46 smithi107 ceph-mon[102221]: pgmap v624: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 707 op/s; 30706/97665 objects degraded (31.440%); 8877/97665 objects misplaced (9.089%); 4.6 MiB/s, 73 objects/s recovering 2024-09-21T10:03:46.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:46 smithi107 ceph-mon[105022]: pgmap v624: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 707 op/s; 30706/97665 objects degraded (31.440%); 8877/97665 objects misplaced (9.089%); 4.6 MiB/s, 73 objects/s recovering 2024-09-21T10:03:46.381 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:46.422 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:46.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:46.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:46.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:46.544 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:46.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:46.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:46.646 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:46.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:46 smithi129 ceph-mon[88471]: pgmap v624: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 707 op/s; 30706/97665 objects degraded (31.440%); 8877/97665 objects misplaced (9.089%); 4.6 MiB/s, 73 objects/s recovering 2024-09-21T10:03:46.679 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:46.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:46.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:46.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:46.809 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:46.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:46.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:46.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:46.940 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:46.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:46.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:47.041 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:47.064 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:47.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:47.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:47.165 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:47.197 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:47.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:47.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:47.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:47.332 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:47.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:47.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:47.434 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:47.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:47.471 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:47.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:47.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:47.604 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:47.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:47.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:47.706 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:47.746 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:47.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:47.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:47.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:47.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:47.891 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:47.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:47.992 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:48.023 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:48.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:48.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:48.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:48.159 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:48.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:48.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:48.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:48.281 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:48.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:48.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:48.382 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:48.419 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:48.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:48.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:48.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:48.555 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:48.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:48.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:48 smithi107 ceph-mon[102221]: pgmap v625: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 670 op/s; 30706/92979 objects degraded (33.025%); 8558/92979 objects misplaced (9.204%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T10:03:48.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:48 smithi107 ceph-mon[105022]: pgmap v625: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 670 op/s; 30706/92979 objects degraded (33.025%); 8558/92979 objects misplaced (9.204%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T10:03:48.657 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:48.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:48 smithi129 ceph-mon[88471]: pgmap v625: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.9 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 670 op/s; 30706/92979 objects degraded (33.025%); 8558/92979 objects misplaced (9.204%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T10:03:48.694 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:48.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:48.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:48.796 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:48.831 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:48.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:48.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:48.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:48.975 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:48.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:48.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:49.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:49.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:49.110 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:49.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:49.211 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:49.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:49.254 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:49.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:49.355 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:49.388 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:49.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:49.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:49.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:49.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:49.524 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:49.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:49.625 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:49.647 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:49.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:49.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:49.749 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:49.786 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:49.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:49.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:49.887 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:49.924 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:49.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:49.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:50.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:50.062 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:50.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:50.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:50.163 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:50.199 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:50.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:50.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:50.300 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:50.339 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:50.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:50.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:50.440 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:50.478 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:50.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:50.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:50.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:50 smithi107 ceph-mon[102221]: pgmap v626: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 773 op/s; 30706/85647 objects degraded (35.852%); 7689/85647 objects misplaced (8.978%); 5.2 MiB/s, 82 objects/s recovering 2024-09-21T10:03:50.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:50 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 30706/85647 objects degraded (35.852%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:03:50.532 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:50 smithi107 ceph-mon[105022]: pgmap v626: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 773 op/s; 30706/85647 objects degraded (35.852%); 7689/85647 objects misplaced (8.978%); 5.2 MiB/s, 82 objects/s recovering 2024-09-21T10:03:50.532 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:50 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 30706/85647 objects degraded (35.852%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:03:50.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:50.606 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:50.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:50.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:03:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:03:50.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:50 smithi129 ceph-mon[88471]: pgmap v626: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.7 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 773 op/s; 30706/85647 objects degraded (35.852%); 7689/85647 objects misplaced (8.978%); 5.2 MiB/s, 82 objects/s recovering 2024-09-21T10:03:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:50 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 30706/85647 objects degraded (35.852%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:03:50.707 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:50.741 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:50.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:50.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:50.842 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:50.874 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:50.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:50.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:50.976 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:50.999 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:50.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:51.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:51.100 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:51.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:51.137 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:51.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:51.238 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:51.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:51.273 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:51.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:51.374 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:51.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:51.402 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:51.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:51.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:51.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:51.526 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:51.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:51.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:51.662 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:51.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:51.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:51.764 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:51.802 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:51.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:51.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:51.904 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:51.936 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:51.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:51.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:52.038 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:52.080 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:52.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:52.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:52.108 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:03:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:03:51] "GET /metrics HTTP/1.1" 200 36525 "" "Prometheus/2.51.0" 2024-09-21T10:03:52.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:52.215 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:52.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:52.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:52.316 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:52.356 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:52.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:52.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:52.458 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:52.491 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:52.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:52.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:52.593 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:52.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:52 smithi107 ceph-mon[102221]: pgmap v627: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 863 op/s; 30706/78393 objects degraded (39.169%); 7100/78393 objects misplaced (9.057%); 4.4 MiB/s, 70 objects/s recovering 2024-09-21T10:03:52.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:52 smithi107 ceph-mon[105022]: pgmap v627: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 863 op/s; 30706/78393 objects degraded (39.169%); 7100/78393 objects misplaced (9.057%); 4.4 MiB/s, 70 objects/s recovering 2024-09-21T10:03:52.631 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:52.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:52.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:52.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:52 smithi129 ceph-mon[88471]: pgmap v627: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 863 op/s; 30706/78393 objects degraded (39.169%); 7100/78393 objects misplaced (9.057%); 4.4 MiB/s, 70 objects/s recovering 2024-09-21T10:03:52.732 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:52.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:52.776 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:52.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:52.877 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:52.909 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:52.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:52.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:53.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:53.041 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:53.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:53.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:53.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:53.169 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:53.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:53.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:53.271 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:53.315 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:53.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:53.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:53.417 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:53.458 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:53.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:53.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:53.559 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:53.591 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:53.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:53.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:53.692 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:53.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:53.727 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:53.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:53.795 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:03:53.795 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:03:53.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.795 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.45761.0:197057 9.4 9:2676887e:::benchmark_data_smithi107_152010_object87120:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-09-21T10:03:53.796 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:03:53.083387+0000", 2024-09-21T10:03:53.796 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.061686390000000001, 2024-09-21T10:03:53.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.061717820999999999, 2024-09-21T10:03:53.796 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:03:53.796 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:03:53.796 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:03:53.796 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:03:53.796 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.45761", 2024-09-21T10:03:53.796 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:03:53.796 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 197057 2024-09-21T10:03:53.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.796 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:03:53.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:03:53.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.083387+0000", 2024-09-21T10:03:53.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:03:53.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:03:53.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.083387+0000", 2024-09-21T10:03:53.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:03:53.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:03:53.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.083388+0000", 2024-09-21T10:03:53.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.584e-06 2024-09-21T10:03:53.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:03:53.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.083393+0000", 2024-09-21T10:03:53.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1889999999999998e-06 2024-09-21T10:03:53.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:03:53.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.083403+0000", 2024-09-21T10:03:53.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9070000000000002e-06 2024-09-21T10:03:53.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:03:53.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.083411+0000", 2024-09-21T10:03:53.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8820000000000001e-06 2024-09-21T10:03:53.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:03:53.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.083467+0000", 2024-09-21T10:03:53.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6156000000000002e-05 2024-09-21T10:03:53.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:03:53.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.083576+0000", 2024-09-21T10:03:53.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010822700000000001 2024-09-21T10:03:53.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:03:53.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.084068+0000", 2024-09-21T10:03:53.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049198000000000004 2024-09-21T10:03:53.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:03:53.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.090348+0000", 2024-09-21T10:03:53.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006280117 2024-09-21T10:03:53.800 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:03:53.800 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:03:53.800 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:03:53.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.801 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.45761.0:197059 9.f e177/141 9:f1425c8c:::benchmark_data_smithi107_152010_object87122:head v 177'12095, mlcod=177'12093)", 2024-09-21T10:03:53.801 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:03:53.086677+0000", 2024-09-21T10:03:53.801 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.058395784999999999, 2024-09-21T10:03:53.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.058471489000000001, 2024-09-21T10:03:53.801 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:03:53.801 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:03:53.801 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:03:53.801 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:03:53.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:03:53.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.086677+0000", 2024-09-21T10:03:53.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:03:53.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:03:53.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.086677+0000", 2024-09-21T10:03:53.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:03:53.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:03:53.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.086678+0000", 2024-09-21T10:03:53.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.037e-06 2024-09-21T10:03:53.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:03:53.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.086686+0000", 2024-09-21T10:03:53.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2620000000000001e-06 2024-09-21T10:03:53.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:03:53.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.086697+0000", 2024-09-21T10:03:53.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1718e-05 2024-09-21T10:03:53.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:03:53.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.086705+0000", 2024-09-21T10:03:53.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.34e-06 2024-09-21T10:03:53.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:03:53.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.086786+0000", 2024-09-21T10:03:53.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1391999999999996e-05 2024-09-21T10:03:53.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:03:53.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.086804+0000", 2024-09-21T10:03:53.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7609e-05 2024-09-21T10:03:53.804 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:03:53.804 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:03:53.804 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:03:53.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.804 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.45761.0:197062 9.3 e177/176 9:c34de7a1:::benchmark_data_smithi107_152010_object87125:head v 177'9958, mlcod=155'4570)", 2024-09-21T10:03:53.805 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:03:53.086955+0000", 2024-09-21T10:03:53.805 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.058117717999999999, 2024-09-21T10:03:53.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.058221704999999999, 2024-09-21T10:03:53.805 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:03:53.805 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:03:53.805 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:03:53.805 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:03:53.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:03:53.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.086955+0000", 2024-09-21T10:03:53.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:03:53.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:03:53.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.086955+0000", 2024-09-21T10:03:53.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:03:53.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:03:53.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.086956+0000", 2024-09-21T10:03:53.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7000000000000004e-07 2024-09-21T10:03:53.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:03:53.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.086963+0000", 2024-09-21T10:03:53.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0979999999999996e-06 2024-09-21T10:03:53.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:03:53.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.086975+0000", 2024-09-21T10:03:53.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1642e-05 2024-09-21T10:03:53.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:03:53.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.086982+0000", 2024-09-21T10:03:53.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1110000000000002e-06 2024-09-21T10:03:53.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:03:53.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.087055+0000", 2024-09-21T10:03:53.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2663000000000005e-05 2024-09-21T10:03:53.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:03:53.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.087076+0000", 2024-09-21T10:03:53.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1280999999999998e-05 2024-09-21T10:03:53.808 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:03:53.808 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:03:53.808 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:03:53.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.808 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.45761.0:197063 9.6 e177/163)", 2024-09-21T10:03:53.808 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:03:53.089047+0000", 2024-09-21T10:03:53.808 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.056025806999999997, 2024-09-21T10:03:53.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.056153729999999999, 2024-09-21T10:03:53.809 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:03:53.809 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:03:53.809 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2024-09-21T10:03:53.809 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:03:53.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:03:53.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.089047+0000", 2024-09-21T10:03:53.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:03:53.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:03:53.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.089047+0000", 2024-09-21T10:03:53.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:03:53.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:03:53.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.089048+0000", 2024-09-21T10:03:53.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.145e-06 2024-09-21T10:03:53.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:03:53.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.089057+0000", 2024-09-21T10:03:53.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1640000000000001e-06 2024-09-21T10:03:53.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:03:53.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.089070+0000", 2024-09-21T10:03:53.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3298999999999999e-05 2024-09-21T10:03:53.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:03:53.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.089077+0000", 2024-09-21T10:03:53.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4070000000000004e-06 2024-09-21T10:03:53.811 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:03:53.811 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:03:53.811 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:03:53.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.811 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.45761.0:197065 9.13 e177/176)", 2024-09-21T10:03:53.811 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:03:53.090364+0000", 2024-09-21T10:03:53.811 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.054709348999999997, 2024-09-21T10:03:53.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.054857553000000003, 2024-09-21T10:03:53.812 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:03:53.812 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:03:53.812 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2024-09-21T10:03:53.812 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:03:53.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:03:53.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.090364+0000", 2024-09-21T10:03:53.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:03:53.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:03:53.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.090364+0000", 2024-09-21T10:03:53.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:03:53.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:03:53.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.090365+0000", 2024-09-21T10:03:53.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1999999999999998e-07 2024-09-21T10:03:53.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:03:53.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.090372+0000", 2024-09-21T10:03:53.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.097e-06 2024-09-21T10:03:53.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:03:53.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.090383+0000", 2024-09-21T10:03:53.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1695000000000001e-05 2024-09-21T10:03:53.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:03:53.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:03:53.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:03:53.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:03:53.090390+0000", 2024-09-21T10:03:53.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5050000000000004e-06 2024-09-21T10:03:53.814 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:03:53.814 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:03:53.814 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:03:53.814 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:03:53.814 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-09-21T10:03:53.814 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 5 2024-09-21T10:03:53.814 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:03:53.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:53.865 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:53.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:53.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:53.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:54.002 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:54.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:54.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:54.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:54.144 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:54.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:54.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:54.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:54.288 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:54.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:54.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:54.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:54.429 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:54.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:54.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:54.527 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:54 smithi107 ceph-mon[102221]: pgmap v628: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.5 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 903 op/s; 30706/72612 objects degraded (42.288%); 6225/72612 objects misplaced (8.573%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T10:03:54.528 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:54 smithi107 ceph-mon[105022]: pgmap v628: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.5 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 903 op/s; 30706/72612 objects degraded (42.288%); 6225/72612 objects misplaced (8.573%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T10:03:54.531 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:54.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:03:54.579 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:54.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:54.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:54 smithi129 ceph-mon[88471]: pgmap v628: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.5 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 903 op/s; 30706/72612 objects degraded (42.288%); 6225/72612 objects misplaced (8.573%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T10:03:54.682 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:54.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:54.725 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:54.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:54.826 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:54.863 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:54.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:54.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:54.964 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:55.015 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:55.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:55.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:55.117 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:55.152 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:55.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:55.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:55.254 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:55.288 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:55.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:55.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:55.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:55.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:55 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 30706/72612 objects degraded (42.288%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:03:55.442 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:55 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 30706/72612 objects degraded (42.288%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:03:55.444 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:55.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:55.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:55.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:55.584 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:55.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:55.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:55 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 30706/72612 objects degraded (42.288%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:03:55.685 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:55.717 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:55.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:55.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:55.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:55.859 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:55.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:55.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:55.961 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:56.003 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:56.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:56.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:56.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:56.142 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:56.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:56.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:56.199 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:56.211 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:56.244 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:56.261 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T10:03:56.321 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:56.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:56.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:56.422 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:56.462 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:56.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:56.564 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:56.593 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:56.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:56.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:56 smithi107 ceph-mon[102221]: pgmap v629: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.4 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 30706/70194 objects degraded (43.744%); 6001/70194 objects misplaced (8.549%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T10:03:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:56 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3886807924' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:03:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:56 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:03:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:56 smithi107 ceph-mon[105022]: pgmap v629: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.4 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 30706/70194 objects degraded (43.744%); 6001/70194 objects misplaced (8.549%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T10:03:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:56 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3886807924' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:03:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:56 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:03:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:56 smithi129 ceph-mon[88471]: pgmap v629: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.4 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 30706/70194 objects degraded (43.744%); 6001/70194 objects misplaced (8.549%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T10:03:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:56 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3886807924' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:03:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:56 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:03:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:56.696 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:56.730 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:56.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:56.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:56.832 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:56.874 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:56.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:56.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:56.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:57.014 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:57.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:57.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:57.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:57.155 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:57.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:57.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:57.257 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:57.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T10:03:57.285 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:57.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:57.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:57.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:57.419 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:57.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:57.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:57.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:03:57.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:57.564 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:57.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:03:57.612 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:57 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:03:57.612 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:57 smithi107 ceph-mon[102221]: osdmap e178: 8 total, 8 up, 8 in 2024-09-21T10:03:57.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:57 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:03:57.612 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:57 smithi107 ceph-mon[105022]: osdmap e178: 8 total, 8 up, 8 in 2024-09-21T10:03:57.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:57 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:03:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:57 smithi129 ceph-mon[88471]: osdmap e178: 8 total, 8 up, 8 in 2024-09-21T10:03:57.697 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:57.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:57.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:57.799 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:57.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:57.836 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:57.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:57.936 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:57.970 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:57.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:57.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:58.072 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:58.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:58.113 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:58.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:58.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:58.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:58.261 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:58.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:58.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:58.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:58.407 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:58.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:58.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:58.540 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:58.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:58.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:58.574 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:03:58 smithi107 ceph-mon[102221]: pgmap v631: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.3 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 978 op/s; 30706/63612 objects degraded (48.271%); 5740/63612 objects misplaced (9.023%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T10:03:58.575 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:03:58 smithi107 ceph-mon[105022]: pgmap v631: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.3 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 978 op/s; 30706/63612 objects degraded (48.271%); 5740/63612 objects misplaced (9.023%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T10:03:58.642 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:58.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:03:58 smithi129 ceph-mon[88471]: pgmap v631: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.3 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 978 op/s; 30706/63612 objects degraded (48.271%); 5740/63612 objects misplaced (9.023%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T10:03:58.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:58.675 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:58.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:58.776 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:58.811 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:58.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:58.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:03:58.913 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:03:58.950 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:58.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:58.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:03:59.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:03:59.088 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:59.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:59.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:03:59.190 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:03:59.220 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:59.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:59.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:03:59.322 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:59.366 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:59.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:59.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:59.468 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:03:59.502 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:59.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:59.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:03:59.604 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:03:59.641 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:03:59.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:59.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:03:59.743 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:03:59.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:59.782 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:59.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:03:59.883 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:03:59.914 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:03:59.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:03:59.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:00.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:00.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:00.057 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:00.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:00.158 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:00.200 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:00.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:00.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:00.308 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:04:00.347 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:00.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:00.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:00.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:00.482 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:00.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:00.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:00.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:00.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:00 smithi107 ceph-mon[102221]: pgmap v632: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 870 op/s; 30706/59529 objects degraded (51.582%); 4799/59529 objects misplaced (8.062%); 4.2 MiB/s, 66 objects/s recovering 2024-09-21T10:04:00.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:00 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 30706/59529 objects degraded (51.582%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:00.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:00 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2483732925' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:04:00.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:00 smithi107 ceph-mon[105022]: pgmap v632: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 870 op/s; 30706/59529 objects degraded (51.582%); 4799/59529 objects misplaced (8.062%); 4.2 MiB/s, 66 objects/s recovering 2024-09-21T10:04:00.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:00 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 30706/59529 objects degraded (51.582%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:00.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:00 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2483732925' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:04:00.620 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:00.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:00.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:00 smithi129 ceph-mon[88471]: pgmap v632: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 870 op/s; 30706/59529 objects degraded (51.582%); 4799/59529 objects misplaced (8.062%); 4.2 MiB/s, 66 objects/s recovering 2024-09-21T10:04:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:00 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 30706/59529 objects degraded (51.582%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:00 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2483732925' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:04:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:04:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:04:00.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:00.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:00.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:00.749 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:00.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:00.851 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:00.896 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:00.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:00.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:00.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:01.020 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:01.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:01.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:01.122 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:01.160 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:01.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:01.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:01.262 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:01.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:01.296 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:01.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:01.397 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:01.437 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:01.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:01.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:01.539 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:01.569 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:01.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:01.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:01.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:01 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:04:01.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:01 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2483732925' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:04:01.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:01 smithi107 ceph-mon[102221]: osdmap e179: 8 total, 8 up, 8 in 2024-09-21T10:04:01.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:01 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:04:01.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:01 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2483732925' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:04:01.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:01 smithi107 ceph-mon[105022]: osdmap e179: 8 total, 8 up, 8 in 2024-09-21T10:04:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:01 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:04:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:01 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2483732925' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:04:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:01 smithi129 ceph-mon[88471]: osdmap e179: 8 total, 8 up, 8 in 2024-09-21T10:04:01.671 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:01.712 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:01.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:01.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:01.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:01.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:01.853 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:01.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:01.954 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:01.989 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:01.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:01.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:02.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:02.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:04:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:04:01] "GET /metrics HTTP/1.1" 200 36676 "" "Prometheus/2.51.0" 2024-09-21T10:04:02.116 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:02.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:02.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:02.217 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:02.248 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:02.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:02.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:02.350 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:02.392 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:02.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:02.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:02.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:02.539 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:02.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:02.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:02.575 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:02 smithi107 ceph-mon[102221]: pgmap v634: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.1 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 842 op/s; 30706/52383 objects degraded (58.618%); 4241/52383 objects misplaced (8.096%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T10:04:02.575 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:02 smithi107 ceph-mon[105022]: pgmap v634: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.1 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 842 op/s; 30706/52383 objects degraded (58.618%); 4241/52383 objects misplaced (8.096%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T10:04:02.642 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:02 smithi129 ceph-mon[88471]: pgmap v634: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 1.1 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 842 op/s; 30706/52383 objects degraded (58.618%); 4241/52383 objects misplaced (8.096%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T10:04:02.684 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:02.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:02.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:02.785 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:02.820 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:02.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:02.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:02.922 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:02.960 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:02.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:02.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:03.061 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:03.102 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:03.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:03.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:03.204 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:03.236 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:03.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:03.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:03.337 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:03.370 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:03.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:03.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:03.472 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:03.503 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:03.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:03.604 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:03.645 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:03.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:03.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:03.654 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:04:03.654 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:04:03.654 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-09-21T10:04:03.654 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-09-21T10:04:03.654 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:04:03.747 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:03.779 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:03.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:03.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:03.872 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:03.880 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:03.909 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:03.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:03.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:04.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:04.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:04.068 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:04.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:04.169 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:04.220 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:04.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:04.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:04.322 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:04.361 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:04.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:04.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:04.463 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:04.517 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:04:04.553 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:04.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:04.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:04.598 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:04 smithi107 ceph-mon[105022]: pgmap v635: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 939 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 30706/45138 objects degraded (68.027%); 3535/45138 objects misplaced (7.832%); 4.0 MiB/s, 90 objects/s recovering 2024-09-21T10:04:04.598 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:04 smithi107 ceph-mon[102221]: pgmap v635: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 939 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 30706/45138 objects degraded (68.027%); 3535/45138 objects misplaced (7.832%); 4.0 MiB/s, 90 objects/s recovering 2024-09-21T10:04:04.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:04 smithi129 ceph-mon[88471]: pgmap v635: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 939 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 30706/45138 objects degraded (68.027%); 3535/45138 objects misplaced (7.832%); 4.0 MiB/s, 90 objects/s recovering 2024-09-21T10:04:04.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:04.729 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:04.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:04.791 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:04:04.791 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:04:04.831 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:04.842 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:04:04.842 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:04:04.877 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:04:04.878 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:04:04.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:04.879 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:04.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:04.914 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:04:04.915 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:04:04.942 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:04:04.943 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:04:04.978 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:04:04.978 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:04:04.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:05.008 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:04:05.008 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:04:05.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:04:05.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:05.022 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:05.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:05.065 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:04:05.066 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:04:05.123 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:05.160 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:05.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:05.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:05.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:05.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:05.312 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:05.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:05.413 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:05.447 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:05.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:05.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:05.549 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:05.588 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:05.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:05.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:05.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:05 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 30706/45138 objects degraded (68.027%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:05.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:05 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2784483427' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:05.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:05 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 30706/45138 objects degraded (68.027%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:05.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:05 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2784483427' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:05.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:05 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 30706/45138 objects degraded (68.027%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:05.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:05 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2784483427' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:05.689 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:05.736 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:05.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:05.838 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:05.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:05.872 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:05.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:05.973 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:06.008 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:06.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:06.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:06.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:06.130 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:06.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:06.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:06.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:06.269 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:06.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:06.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:06.370 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:06.400 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:06.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:06.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:06.502 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:06.534 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:06.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:06.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:06 smithi107 ceph-mon[102221]: pgmap v636: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 883 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 889 op/s; 30706/42453 objects degraded (72.329%); 3296/42453 objects misplaced (7.764%); 3.6 MiB/s, 81 objects/s recovering 2024-09-21T10:04:06.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:06 smithi107 ceph-mon[105022]: pgmap v636: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 883 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 889 op/s; 30706/42453 objects degraded (72.329%); 3296/42453 objects misplaced (7.764%); 3.6 MiB/s, 81 objects/s recovering 2024-09-21T10:04:06.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:06.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:06 smithi129 ceph-mon[88471]: pgmap v636: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 883 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 889 op/s; 30706/42453 objects degraded (72.329%); 3296/42453 objects misplaced (7.764%); 3.6 MiB/s, 81 objects/s recovering 2024-09-21T10:04:06.672 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:06.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:06.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:06.773 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:06.814 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:06.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:06.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:06.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:06.946 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:06.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:06.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:07.048 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:07.081 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:07.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:07.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:07.183 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:07.215 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:07.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:07.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:07.317 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:07.374 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:07.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:07.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:07.418 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:07 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:04:07.419 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:07 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:04:07.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:07.529 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:07.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:07.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:07.631 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:07.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:07 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:04:07.676 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:07.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:07.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:07.778 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:07.816 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:07.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:07.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:07.917 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:07.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:07.983 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:07.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:08.084 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:08.148 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:08.168 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:08.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:08.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:08.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:08.306 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:08.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:08.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:08.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:08.440 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:08.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:08.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:08.487 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:08.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:08.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:08.594 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:08.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:08.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:08 smithi107 ceph-mon[102221]: pgmap v637: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 741 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 931 op/s; 30706/35640 objects degraded (86.156%); 2996/35640 objects misplaced (8.406%); 3.2 MiB/s, 72 objects/s recovering 2024-09-21T10:04:08.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:08 smithi107 ceph-mon[105022]: pgmap v637: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 741 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 931 op/s; 30706/35640 objects degraded (86.156%); 2996/35640 objects misplaced (8.406%); 3.2 MiB/s, 72 objects/s recovering 2024-09-21T10:04:08.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:08 smithi129 ceph-mon[88471]: pgmap v637: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 741 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 931 op/s; 30706/35640 objects degraded (86.156%); 2996/35640 objects misplaced (8.406%); 3.2 MiB/s, 72 objects/s recovering 2024-09-21T10:04:08.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:08.731 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:08.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:08.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:08.832 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:08.878 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:08.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:08.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:08.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:09.004 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:09.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:09.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:09.105 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:09.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:09.143 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:09.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:09.244 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:09.274 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:09.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:09.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:09.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:09.411 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:09.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:09.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:09.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:09.550 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:09.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:09.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:09.652 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:09.691 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:09.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:09.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:09.793 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:09.828 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:09.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:09.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:09.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:09.975 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:09.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:09.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:10.076 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:10.098 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:10.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:10.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:10.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:04:10.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:10.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:10.255 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:10.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:10.350 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T10:04:10.356 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:10.404 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:10.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:10.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:10.471 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:10 smithi129 ceph-mon[88471]: pgmap v638: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 650 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 941 op/s; 30706/31272 objects degraded (98.190%); 2446/31272 objects misplaced (7.822%); 844 KiB/s, 74 objects/s recovering 2024-09-21T10:04:10.471 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:10 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 30706/31272 objects degraded (98.190%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:10.471 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:10 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1364896300' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:04:10.471 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:04:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:04:10.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:10.506 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:10.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:10.539 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:10.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:10.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:10 smithi107 ceph-mon[105022]: pgmap v638: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 650 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 941 op/s; 30706/31272 objects degraded (98.190%); 2446/31272 objects misplaced (7.822%); 844 KiB/s, 74 objects/s recovering 2024-09-21T10:04:10.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:10 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 30706/31272 objects degraded (98.190%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:10.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:10 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1364896300' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:04:10.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:10 smithi107 ceph-mon[102221]: pgmap v638: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 650 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 941 op/s; 30706/31272 objects degraded (98.190%); 2446/31272 objects misplaced (7.822%); 844 KiB/s, 74 objects/s recovering 2024-09-21T10:04:10.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:10 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 30706/31272 objects degraded (98.190%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:10.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:10 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1364896300' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:04:10.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:10.670 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:10.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:10.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:10.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:10.795 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:10.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:10.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:10.897 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:10.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:10.933 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:10.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:11.034 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:11.075 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:11.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:11.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:11.177 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:11.202 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:11.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:11.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:11.304 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:11.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:11.340 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:11.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:11.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:11.478 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:11.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:11.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:11.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:11.609 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:11.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:11.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:11.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:11 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1364896300' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:04:11.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:11 smithi129 ceph-mon[88471]: osdmap e180: 8 total, 8 up, 8 in 2024-09-21T10:04:11.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:11.711 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:11.746 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:11.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:11.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:11.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:11 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1364896300' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:04:11.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:11 smithi107 ceph-mon[102221]: osdmap e180: 8 total, 8 up, 8 in 2024-09-21T10:04:11.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:11.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:11 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1364896300' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:04:11.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:11 smithi107 ceph-mon[105022]: osdmap e180: 8 total, 8 up, 8 in 2024-09-21T10:04:11.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:11.848 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:11.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:11.886 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:11.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:11.988 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:12.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:12.025 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:12.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:12.052 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:04:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:04:11] "GET /metrics HTTP/1.1" 200 36670 "" "Prometheus/2.51.0" 2024-09-21T10:04:12.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:12.158 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:12.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:12.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:12.261 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:12.303 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:12.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:12.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:12.405 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:12.453 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:12.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:12.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:12.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:12.592 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:12.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:12.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:12 smithi107 ceph-mon[102221]: pgmap v640: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 509 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 929 op/s; 30706/24468 objects degraded (125.495%); 1947/24468 objects misplaced (7.957%); 844 KiB/s, 74 objects/s recovering 2024-09-21T10:04:12.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:12 smithi107 ceph-mon[105022]: pgmap v640: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 509 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 929 op/s; 30706/24468 objects degraded (125.495%); 1947/24468 objects misplaced (7.957%); 844 KiB/s, 74 objects/s recovering 2024-09-21T10:04:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:12 smithi129 ceph-mon[88471]: pgmap v640: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 509 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 929 op/s; 30706/24468 objects degraded (125.495%); 1947/24468 objects misplaced (7.957%); 844 KiB/s, 74 objects/s recovering 2024-09-21T10:04:12.694 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:12.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:12.732 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:12.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:12.833 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:12.871 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:12.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:12.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:12.972 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:13.030 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:13.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:13.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:13.031 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:04:13.031 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:04:13.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:13.171 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:13.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:13.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:13.273 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:13.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:13.320 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:13.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:13.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T10:04:13.421 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:13.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:13.472 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:13.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:13.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:13.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:13.620 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:13.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:13.721 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:13.767 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:13.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:13.868 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:13.901 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:13.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:13.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:14.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:14.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:04:14.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:14.053 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:14.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:14.154 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:14.205 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:14.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:14.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:14.306 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:14.349 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:14.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:14.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:14.450 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:14.480 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:14.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:14.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:14.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:14.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:14.633 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:14.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:14 smithi129 ceph-mon[88471]: pgmap v641: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 375 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 902 op/s; 30706/18051 objects degraded (170.107%); 1408/18051 objects misplaced (7.800%); 0 B/s, 78 objects/s recovering 2024-09-21T10:04:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:14 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 30706/18051 objects degraded (170.107%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:14.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:14.781 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:14.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:14.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:14.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:14 smithi107 ceph-mon[102221]: pgmap v641: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 375 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 902 op/s; 30706/18051 objects degraded (170.107%); 1408/18051 objects misplaced (7.800%); 0 B/s, 78 objects/s recovering 2024-09-21T10:04:14.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:14 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 30706/18051 objects degraded (170.107%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:14.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:14 smithi107 ceph-mon[105022]: pgmap v641: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 375 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 902 op/s; 30706/18051 objects degraded (170.107%); 1408/18051 objects misplaced (7.800%); 0 B/s, 78 objects/s recovering 2024-09-21T10:04:14.795 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:14 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 30706/18051 objects degraded (170.107%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:14.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:14.923 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:14.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:14.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:14.954 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:15.025 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:15.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:15.059 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:15.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:15.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:15.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:15.219 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:15.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:15.319 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:15.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:15.363 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:15.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:15.464 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:15.498 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:15.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:15.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:15.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:15.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:15.642 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:15.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:15.743 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:15.784 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:15.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:15.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:15.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:15.945 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:15.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:15.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:16.047 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:16.096 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:16.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:16.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:16.198 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:16.240 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:16.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:16.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:16.342 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:16.375 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:16.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:16.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:16.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:16.513 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:16.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:16.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:16.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:16 smithi107 ceph-mon[102221]: pgmap v642: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 322 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 897 op/s; 30706/15510 objects degraded (197.975%); 1191/15510 objects misplaced (7.679%); 0 B/s, 78 objects/s recovering 2024-09-21T10:04:16.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:16 smithi107 ceph-mon[105022]: pgmap v642: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 322 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 897 op/s; 30706/15510 objects degraded (197.975%); 1191/15510 objects misplaced (7.679%); 0 B/s, 78 objects/s recovering 2024-09-21T10:04:16.609 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:16.615 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:16.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:16 smithi129 ceph-mon[88471]: pgmap v642: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 322 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 897 op/s; 30706/15510 objects degraded (197.975%); 1191/15510 objects misplaced (7.679%); 0 B/s, 78 objects/s recovering 2024-09-21T10:04:16.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:16.678 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:16.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:16.779 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:16.817 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:16.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:16.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:16.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:16.956 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:16.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:17.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:17.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:17.095 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:17.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:17.195 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:17.245 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:17.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:17.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:17.347 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:17.397 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:17.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:17.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:17.413 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:04:17.499 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:17.546 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:17.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:17.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:17.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:17 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1464020507' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:04:17.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:17 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1464020507' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:04:17.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:17.655 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:17.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:17 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1464020507' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:04:17.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:17.689 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:17.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:17.790 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:17.825 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:17.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:17.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:17.927 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:17.992 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:17.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:17.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:18.094 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:18.102 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Removed 109934 objects 2024-09-21T10:04:18.102 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Clean up completed and total clean up time :118.607 2024-09-21T10:04:18.117 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:18.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:18.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:18.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:18.242 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:18.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:18.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:18.344 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:18.367 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:18.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:18.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:18.469 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:18.490 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:18.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:18.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:18.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:18.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:18 smithi107 ceph-mon[102221]: pgmap v643: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 179 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 899 op/s; 30706/8646 objects degraded (355.147%); 895/8646 objects misplaced (10.352%); 0 B/s, 78 objects/s recovering 2024-09-21T10:04:18.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:18 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:04:18.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:18 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1464020507' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:04:18.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:18 smithi107 ceph-mon[102221]: osdmap e181: 8 total, 8 up, 8 in 2024-09-21T10:04:18.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:18 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:18.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:18 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:18.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:18 smithi107 ceph-mon[105022]: pgmap v643: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 179 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 899 op/s; 30706/8646 objects degraded (355.147%); 895/8646 objects misplaced (10.352%); 0 B/s, 78 objects/s recovering 2024-09-21T10:04:18.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:18 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:04:18.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:18 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1464020507' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:04:18.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:18 smithi107 ceph-mon[105022]: osdmap e181: 8 total, 8 up, 8 in 2024-09-21T10:04:18.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:18 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:18.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:18 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:18.614 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:18.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:18.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:18 smithi129 ceph-mon[88471]: pgmap v643: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 179 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 899 op/s; 30706/8646 objects degraded (355.147%); 895/8646 objects misplaced (10.352%); 0 B/s, 78 objects/s recovering 2024-09-21T10:04:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:18 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:04:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:18 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1464020507' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:04:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:18 smithi129 ceph-mon[88471]: osdmap e181: 8 total, 8 up, 8 in 2024-09-21T10:04:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:18 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:18 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:18.716 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:18.737 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:18.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:18.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:18.839 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:18.862 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:18.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:18.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:18.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:18.985 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:18.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:19.086 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:19.109 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:19.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:19.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:19.211 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:19.233 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:19.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:19.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:19.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:19.357 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:19.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:19.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:19.459 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:19.481 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:19.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:19.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:19.582 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:19.605 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:19.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:19.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:19.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:19 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/162213452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:19.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:19 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 30706/8646 objects degraded (355.147%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:19.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:19 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/162213452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:19.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:19 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 30706/8646 objects degraded (355.147%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:19.707 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:19.729 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:19.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:19.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:19.830 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:19.854 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:19.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:19.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:19 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/162213452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:19 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 30706/8646 objects degraded (355.147%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:19.932 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:04:19.932 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:04:19.932 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:04:19.932 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:04:19.932 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:04:19.932 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:04:19.932 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:04:19.933 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:04:19.933 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:04:19.933 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:04:19.933 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:04:19.933 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:04:19.934 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:04:19.934 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:04:19.934 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:04:19.934 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:04:19.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:19.978 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:19.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:19.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:20.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:20.104 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:20.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:20.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:20.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:20.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:20.228 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:20.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:20.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:04:20.330 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:20.354 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:20.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:20.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:20.455 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:20.477 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:20.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:20.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:20.579 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:20.599 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:20.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:20.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:20.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:20 smithi107 ceph-mon[102221]: pgmap v645: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 93 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 919 op/s; 30706/4503 objects degraded (681.901%); 329/4503 objects misplaced (7.306%); 0 B/s, 89 objects/s recovering 2024-09-21T10:04:20.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:20 smithi107 ceph-mon[105022]: pgmap v645: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 93 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 919 op/s; 30706/4503 objects degraded (681.901%); 329/4503 objects misplaced (7.306%); 0 B/s, 89 objects/s recovering 2024-09-21T10:04:20.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:20 smithi129 ceph-mon[88471]: pgmap v645: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 93 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 919 op/s; 30706/4503 objects degraded (681.901%); 329/4503 objects misplaced (7.306%); 0 B/s, 89 objects/s recovering 2024-09-21T10:04:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:04:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:04:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:20.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:20.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:20.724 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:20.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:20.824 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:20.846 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:20.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:20.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:20.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:20.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:20.972 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:20.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:21.073 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:21.094 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:21.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:21.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:21.195 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:21.216 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:21.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:21.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:21.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:21.343 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:21.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:21.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:21.444 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:21.466 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:21.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:21.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:21.553 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:21 smithi107 ceph-mon[102221]: osdmap e182: 8 total, 8 up, 8 in 2024-09-21T10:04:21.553 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:21 smithi107 ceph-mon[105022]: osdmap e182: 8 total, 8 up, 8 in 2024-09-21T10:04:21.568 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:21.581 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:04:21.581 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:04:21.581 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-09-21T10:04:21.581 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:04:21.589 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:21.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:21.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:21.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:21.713 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:21.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:21.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:21.815 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:21.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:21.838 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:21.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:21.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:04:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:04:21] "GET /metrics HTTP/1.1" 200 36670 "" "Prometheus/2.51.0" 2024-09-21T10:04:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:21 smithi129 ceph-mon[88471]: osdmap e182: 8 total, 8 up, 8 in 2024-09-21T10:04:21.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:21.964 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:21.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:21.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:22.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:22.087 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:22.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:22.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:22.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:22.212 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:22.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:22.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:22.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:22.336 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:22.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:22.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:22.437 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:22.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:22.459 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:22.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:22.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:22.582 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:22.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:22.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:22.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:22.705 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:22.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:22.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:22.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:22.828 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:22.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:22.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:22.829 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-21T10:04:22.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-21T10:04:22.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:22 smithi107 ceph-mon[102221]: pgmap v647: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 27 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 711 op/s; 30706/1347 objects degraded (2279.584%); 67/1347 objects misplaced (4.974%); 0 B/s, 50 objects/s recovering 2024-09-21T10:04:22.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:22 smithi107 ceph-mon[102221]: osdmap e183: 8 total, 8 up, 8 in 2024-09-21T10:04:22.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:22 smithi107 ceph-mon[105022]: pgmap v647: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 27 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 711 op/s; 30706/1347 objects degraded (2279.584%); 67/1347 objects misplaced (4.974%); 0 B/s, 50 objects/s recovering 2024-09-21T10:04:22.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:22 smithi107 ceph-mon[105022]: osdmap e183: 8 total, 8 up, 8 in 2024-09-21T10:04:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:22 smithi129 ceph-mon[88471]: pgmap v647: 119 pgs: 1 active+recovering+undersized+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 105 active+clean; 27 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 711 op/s; 30706/1347 objects degraded (2279.584%); 67/1347 objects misplaced (4.974%); 0 B/s, 50 objects/s recovering 2024-09-21T10:04:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:22 smithi129 ceph-mon[88471]: osdmap e183: 8 total, 8 up, 8 in 2024-09-21T10:04:22.929 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:22.950 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:22.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:22.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:22.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:04:23.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:23.074 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:23.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:23.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:23.176 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:23.197 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:23.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:23.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:23.299 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:23.320 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:23.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:23.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:23.422 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:23.443 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:23.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:23.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:23.545 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:23.566 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:23.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:23.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:23.668 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:23.689 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:23.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:23.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:23.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:23.812 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:23.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:23.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:23.914 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:23.936 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:23.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:23.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:24.038 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:24.060 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:24.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:24.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:24.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:24.184 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:24.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:24.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:24.285 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:24.307 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:24.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:24.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:24.408 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:24.430 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:24.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:24.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:24.532 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:24.552 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:24.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:24.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:24.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:24 smithi107 ceph-mon[105022]: pgmap v649: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 481 op/s; 28394/390 objects degraded (7280.513%); 0 B/s, 276 objects/s recovering 2024-09-21T10:04:24.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:24 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 28394/390 objects degraded (7280.513%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:24.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:24 smithi107 ceph-mon[102221]: pgmap v649: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 481 op/s; 28394/390 objects degraded (7280.513%); 0 B/s, 276 objects/s recovering 2024-09-21T10:04:24.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:24 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 28394/390 objects degraded (7280.513%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:24.654 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:24.675 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:24.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:24.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:24.777 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:24.798 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:24.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:24.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:24.900 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:24.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:24 smithi129 ceph-mon[88471]: pgmap v649: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 481 op/s; 28394/390 objects degraded (7280.513%); 0 B/s, 276 objects/s recovering 2024-09-21T10:04:24.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:24 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 28394/390 objects degraded (7280.513%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:24.921 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:24.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:24.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:25.023 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:25.044 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:25.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:25.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:25.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:25.168 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:25.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:25.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:25.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:25.291 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:25.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:25.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:25.393 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:25.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:25.415 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:25.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:25.516 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:25.537 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:25.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:25.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:25.638 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:25.660 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:25.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:25.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:25.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:25.782 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:25.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:25.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:25.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:25.907 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:25.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:25.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:26.009 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:26.030 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:26.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:26.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:26.132 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:26.153 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:26.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:26.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:26.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:26.276 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:26.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:26.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:26.378 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:26.400 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:26.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:26.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:26.501 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:26.523 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:26.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:26.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:26.625 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:26.646 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:26.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:26.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:26.748 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:26.769 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:26.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:26.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:26 smithi107 ceph-mon[105022]: pgmap v650: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 336 op/s; 28394/390 objects degraded (7280.513%); 0 B/s, 215 objects/s recovering 2024-09-21T10:04:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:26 smithi107 ceph-mon[102221]: pgmap v650: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 336 op/s; 28394/390 objects degraded (7280.513%); 0 B/s, 215 objects/s recovering 2024-09-21T10:04:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:26.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:26.892 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:26.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:26.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:26 smithi129 ceph-mon[88471]: pgmap v650: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 336 op/s; 28394/390 objects degraded (7280.513%); 0 B/s, 215 objects/s recovering 2024-09-21T10:04:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:26.993 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:27.016 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:27.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:27.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:27.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:27.139 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:27.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:27.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:27.240 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:27.261 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:27.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:27.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:27.363 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:27.384 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:27.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:27.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:27.486 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:27.508 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:27.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:27.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:27.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:27.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:27.631 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:27.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:27.732 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:27.754 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:27.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:27.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:27.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:27.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:04:27.880 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:27.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:27.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:27.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:28.005 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:28.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:28.107 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:28.129 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:28.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:28.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:28.231 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:28.252 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:28.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:28.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:28.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:28.375 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:28.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:28.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:28.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:28.498 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:28.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:28.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:28.600 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:28.621 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:28.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:28.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:28.723 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:28.744 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:28.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:28.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:28.791 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:28 smithi107 ceph-mon[102221]: pgmap v651: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 188 op/s; 28394/390 objects degraded (7280.513%); 0 B/s, 297 objects/s recovering 2024-09-21T10:04:28.792 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:28 smithi107 ceph-mon[105022]: pgmap v651: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 188 op/s; 28394/390 objects degraded (7280.513%); 0 B/s, 297 objects/s recovering 2024-09-21T10:04:28.844 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:28.846 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:28.848 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:28.867 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:28.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:28.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:28.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:28 smithi129 ceph-mon[88471]: pgmap v651: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 188 op/s; 28394/390 objects degraded (7280.513%); 0 B/s, 297 objects/s recovering 2024-09-21T10:04:28.969 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:28.990 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:28.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:28.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:29.092 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:29.113 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:29.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:29.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:29.122 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:29.214 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:29.235 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:29.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:29.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:29.337 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:29.359 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:29.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:29.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:29.461 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:29.483 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:29.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:29.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:29.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:29.606 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:29.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:29.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:29.708 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:29.729 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:29.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:29.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:29.831 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:29.853 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:29.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:29.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:29.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:29.976 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:29.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:29.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:30.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:30.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:30.112 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:30.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:30.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:30.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:30.236 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:30.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:30.337 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:30.359 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:30.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:30.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:30.460 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:30.482 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:30.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:30.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:30.584 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:30.605 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:30.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:30.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:30 smithi129 ceph-mon[88471]: pgmap v652: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 107 op/s; 28394/390 objects degraded (7280.513%); 0 B/s, 273 objects/s recovering 2024-09-21T10:04:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:04:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:04:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:30.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:30.729 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:30.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:30.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:30.830 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:30.851 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:30.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:30.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:30.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:30 smithi107 ceph-mon[102221]: pgmap v652: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 107 op/s; 28394/390 objects degraded (7280.513%); 0 B/s, 273 objects/s recovering 2024-09-21T10:04:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:30 smithi107 ceph-mon[105022]: pgmap v652: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 107 op/s; 28394/390 objects degraded (7280.513%); 0 B/s, 273 objects/s recovering 2024-09-21T10:04:30.953 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:30.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:30.975 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:30.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:31.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:31.100 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:31.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:31.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:31.201 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:31.222 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:31.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:31.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:31.324 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:31.345 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:31.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:31.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:31.449 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:31.470 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:31.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:31.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:31.572 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:31.593 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:31.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:31.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:31.694 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:31.716 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:31.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:31.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:31.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:31.840 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:31.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:31.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:31.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:31.964 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:31.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:31.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:32.066 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:32.087 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:32.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:32.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:32.108 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:04:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:04:31] "GET /metrics HTTP/1.1" 200 36653 "" "Prometheus/2.51.0" 2024-09-21T10:04:32.189 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:32.210 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:32.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:32.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:32.312 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:32.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:32.335 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:32.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:32.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:32.460 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:32.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:32.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:32.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:32.584 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:32.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:32.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:32.686 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:32.709 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:32.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:32.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:32.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:32 smithi107 ceph-mon[102221]: pgmap v653: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 32 op/s; 28394/390 objects degraded (7280.513%); 0 B/s, 237 objects/s recovering 2024-09-21T10:04:32.794 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:32 smithi107 ceph-mon[105022]: pgmap v653: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 32 op/s; 28394/390 objects degraded (7280.513%); 0 B/s, 237 objects/s recovering 2024-09-21T10:04:32.811 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:32.832 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:32.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:32.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:32 smithi129 ceph-mon[88471]: pgmap v653: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 32 op/s; 28394/390 objects degraded (7280.513%); 0 B/s, 237 objects/s recovering 2024-09-21T10:04:32.934 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:32.955 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:32.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:32.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:33.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:33.078 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:33.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:33.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:33.180 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:33.201 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:33.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:33.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:33.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:33.326 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:33.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:33.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:33.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:33.451 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:33.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:33.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:33.473 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T10:04:33.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:33.576 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:33.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:33.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:33.678 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:33.700 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:33.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:33.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:33.801 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:33.823 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:33.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:33.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:33.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:33 smithi107 ceph-mon[102221]: osdmap e184: 8 total, 8 up, 8 in 2024-09-21T10:04:33.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:33 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3873271254' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:04:33.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:33 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:04:33.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:33 smithi107 ceph-mon[105022]: osdmap e184: 8 total, 8 up, 8 in 2024-09-21T10:04:33.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:33 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3873271254' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:04:33.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:33 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:04:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:33 smithi129 ceph-mon[88471]: osdmap e184: 8 total, 8 up, 8 in 2024-09-21T10:04:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:33 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3873271254' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:04:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:33 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:04:33.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:33.945 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:33.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:33.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:34.047 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:34.068 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:34.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:34.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:34.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:34.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:34.193 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:34.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:34.294 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:34.315 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:34.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:34.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:34.417 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:34.438 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:34.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:34.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:34.540 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:34.562 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:34.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:34.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:34.664 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:34.685 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:34.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:34.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:34.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:34.811 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:34.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:34.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:34.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:34 smithi107 ceph-mon[102221]: pgmap v655: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 27527/390 objects degraded (7058.205%); 0 B/s, 281 objects/s recovering 2024-09-21T10:04:34.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:34 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 27527/390 objects degraded (7058.205%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:34.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:34 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:04:34.836 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:34 smithi107 ceph-mon[102221]: osdmap e185: 8 total, 8 up, 8 in 2024-09-21T10:04:34.837 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:34 smithi107 ceph-mon[105022]: pgmap v655: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 27527/390 objects degraded (7058.205%); 0 B/s, 281 objects/s recovering 2024-09-21T10:04:34.837 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:34 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 27527/390 objects degraded (7058.205%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:34.837 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:34 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:04:34.837 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:34 smithi107 ceph-mon[105022]: osdmap e185: 8 total, 8 up, 8 in 2024-09-21T10:04:34.913 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:34.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:34 smithi129 ceph-mon[88471]: pgmap v655: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 27527/390 objects degraded (7058.205%); 0 B/s, 281 objects/s recovering 2024-09-21T10:04:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:34 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 27527/390 objects degraded (7058.205%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:34 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:04:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:34 smithi129 ceph-mon[88471]: osdmap e185: 8 total, 8 up, 8 in 2024-09-21T10:04:34.934 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:34.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:34.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:35.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:35.058 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:35.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:35.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:35.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:35.181 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:35.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:35.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:35.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:35.306 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:35.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:35.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:35.407 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:35.429 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:35.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:35.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:35.531 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:35.552 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:35.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:35.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:35.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:35.675 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:35.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:35.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:35.777 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:35.798 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:35.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:35.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:35.845 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T10:04:35.900 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:35.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:35.921 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:35.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:36.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:36.045 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:36.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:36.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:36.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:36.170 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:36.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:36.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:36.272 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:36.295 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:36.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:36.398 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:36.419 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:36.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:36.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:36.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:36.543 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:36.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:36.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:36.645 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:36.667 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:36.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:36.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:36.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:36 smithi107 ceph-mon[102221]: pgmap v657: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 27527/390 objects degraded (7058.205%); 0 B/s, 211 objects/s recovering 2024-09-21T10:04:36.673 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:36 smithi107 ceph-mon[105022]: pgmap v657: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 27527/390 objects degraded (7058.205%); 0 B/s, 211 objects/s recovering 2024-09-21T10:04:36.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:36.789 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:36.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:36.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:36.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:36.913 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:36.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:36.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:36 smithi129 ceph-mon[88471]: pgmap v657: 119 pgs: 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 106 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 27527/390 objects degraded (7058.205%); 0 B/s, 211 objects/s recovering 2024-09-21T10:04:37.015 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:37.036 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:37.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:37.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:37.138 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:37.159 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:37.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:37.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:37.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:37.284 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:37.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:37.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:37.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:37.407 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:37.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:37.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:37.474 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:04:37.509 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:37.531 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:37.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:37.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:37.632 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:37.654 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:37.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:37.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:37.755 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:37.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:37.778 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:37.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:37 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/331518415' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:04:37.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:37 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/331518415' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:04:37.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:37.902 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:37.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:37.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:37 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/331518415' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:04:38.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:38.027 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:38.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:38.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:38.151 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:38.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:38.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:38.253 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:38.275 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:38.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:38.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:38.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:38.397 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:38.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:38.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:38.499 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:38.520 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:38.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:38.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:38.621 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:38.642 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:38.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:38.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:38.744 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:38.765 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:38.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:38.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:38.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:38 smithi107 ceph-mon[105022]: pgmap v658: 119 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 107 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 24962/390 objects degraded (6400.513%); 0 B/s, 373 objects/s recovering 2024-09-21T10:04:38.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:38 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:04:38.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:38 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/331518415' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:04:38.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:38 smithi107 ceph-mon[105022]: osdmap e186: 8 total, 8 up, 8 in 2024-09-21T10:04:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:38 smithi107 ceph-mon[102221]: pgmap v658: 119 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 107 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 24962/390 objects degraded (6400.513%); 0 B/s, 373 objects/s recovering 2024-09-21T10:04:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:38 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:04:38.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:38 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/331518415' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:04:38.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:38 smithi107 ceph-mon[102221]: osdmap e186: 8 total, 8 up, 8 in 2024-09-21T10:04:38.866 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:38.887 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:38.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:38.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:38.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:38 smithi129 ceph-mon[88471]: pgmap v658: 119 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 107 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 24962/390 objects degraded (6400.513%); 0 B/s, 373 objects/s recovering 2024-09-21T10:04:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:38 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:04:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:38 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/331518415' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:04:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:38 smithi129 ceph-mon[88471]: osdmap e186: 8 total, 8 up, 8 in 2024-09-21T10:04:38.989 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:39.013 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:39.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:39.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:39.115 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:39.136 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:39.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:39.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:39.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:39.260 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:39.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:39.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:39.383 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:39.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:39.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:39.477 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_4' removed 2024-09-21T10:04:39.486 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:39.510 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:39.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:39.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:39.612 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:39.632 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:39.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:39.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:39.734 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:39.755 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:39.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:39.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:39.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:39.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:39 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 24962/390 objects degraded (6400.513%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:39.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:39 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/281818596' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:04:39.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:39 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 24962/390 objects degraded (6400.513%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:39.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:39 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/281818596' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:04:39.881 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:39.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:39.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:39.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:39 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 24962/390 objects degraded (6400.513%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:04:39.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:39 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/281818596' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:04:39.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:40.004 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:40.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:40.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:40.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:40.127 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:40.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:40.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:40.229 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:40.250 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:40.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:40.354 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:40.379 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:40.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:40.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:40.478 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:04:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:04:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:40.480 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:40.502 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:40.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:40.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:40.604 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:40.626 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:40.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:40.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:40.728 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:40.744 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:40 smithi129 ceph-mon[88471]: pgmap v660: 119 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 107 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 24962/390 objects degraded (6400.513%); 0 B/s, 195 objects/s recovering 2024-09-21T10:04:40.745 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:40 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/281818596' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:04:40.745 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:40 smithi129 ceph-mon[88471]: osdmap e187: 8 total, 8 up, 8 in 2024-09-21T10:04:40.751 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:40.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:40.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:40.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:40 smithi107 ceph-mon[105022]: pgmap v660: 119 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 107 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 24962/390 objects degraded (6400.513%); 0 B/s, 195 objects/s recovering 2024-09-21T10:04:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:40 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/281818596' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:04:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:40 smithi107 ceph-mon[105022]: osdmap e187: 8 total, 8 up, 8 in 2024-09-21T10:04:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:40 smithi107 ceph-mon[102221]: pgmap v660: 119 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 107 active+clean; 6.9 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 24962/390 objects degraded (6400.513%); 0 B/s, 195 objects/s recovering 2024-09-21T10:04:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:40 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/281818596' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:04:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:40 smithi107 ceph-mon[102221]: osdmap e187: 8 total, 8 up, 8 in 2024-09-21T10:04:40.880 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:40.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:40.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:40.982 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:41.004 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:41.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:41.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:41.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:41.128 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:41.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:41.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:41.229 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:41.251 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:41.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:41.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:41.353 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:41.374 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:41.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:41.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:41.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:41.508 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:41.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:41.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:41.610 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:41.631 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:41.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:41.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:41.732 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:41.757 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:41.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:41.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:41.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T10:04:41.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:41.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:41.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T10:04:41.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:41.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:41.787 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T10:04:41.787 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:41.787 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:41.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:41.886 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:41.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:41.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:41.987 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:42.013 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:42.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:42.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:42.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:04:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:04:41] "GET /metrics HTTP/1.1" 200 34293 "" "Prometheus/2.51.0" 2024-09-21T10:04:42.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:42.140 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:42.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:42.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:42.242 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:42.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:42.264 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:42.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:42.365 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:42.386 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:42.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:42.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:42.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:42.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:42.514 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:42.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:42.615 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:42.636 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:42.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:42.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:42.738 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:42.759 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:42.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:42.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:42.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:42 smithi107 ceph-mon[105022]: pgmap v662: 93 pgs: 93 active+clean; 582 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:42.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:42 smithi107 ceph-mon[105022]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24962/390 objects degraded (6400.513%), 8 pgs degraded) 2024-09-21T10:04:42.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:42 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:04:42.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T10:04:42.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:42 smithi107 ceph-mon[105022]: osdmap e188: 8 total, 8 up, 8 in 2024-09-21T10:04:42.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:42 smithi107 ceph-mon[102221]: pgmap v662: 93 pgs: 93 active+clean; 582 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:42.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:42 smithi107 ceph-mon[102221]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24962/390 objects degraded (6400.513%), 8 pgs degraded) 2024-09-21T10:04:42.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:42 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:04:42.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T10:04:42.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:42 smithi107 ceph-mon[102221]: osdmap e188: 8 total, 8 up, 8 in 2024-09-21T10:04:42.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:42.885 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:42.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:42.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:42 smithi129 ceph-mon[88471]: pgmap v662: 93 pgs: 93 active+clean; 582 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:42 smithi129 ceph-mon[88471]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24962/390 objects degraded (6400.513%), 8 pgs degraded) 2024-09-21T10:04:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:42 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:04:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T10:04:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:42 smithi129 ceph-mon[88471]: osdmap e188: 8 total, 8 up, 8 in 2024-09-21T10:04:42.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:43.013 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:43.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:43.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:43.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:43.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:43.142 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:43.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:43.243 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:43.265 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:43.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:43.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:43.367 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:43.388 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:43.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:43.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:43.490 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:43.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:43.511 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:43.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:43.612 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:43.634 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:43.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:43.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:43.736 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:43.761 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:43.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:43.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:43 smithi107 ceph-mon[102221]: osdmap e189: 8 total, 8 up, 8 in 2024-09-21T10:04:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:43 smithi107 ceph-mon[102221]: 1.11 scrub starts 2024-09-21T10:04:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:43 smithi107 ceph-mon[102221]: 1.11 scrub ok 2024-09-21T10:04:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:43 smithi107 ceph-mon[102221]: 1.14 scrub starts 2024-09-21T10:04:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:43 smithi107 ceph-mon[102221]: 1.14 scrub ok 2024-09-21T10:04:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:43 smithi107 ceph-mon[102221]: 1.f scrub starts 2024-09-21T10:04:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:43 smithi107 ceph-mon[102221]: 1.f scrub ok 2024-09-21T10:04:43.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:43 smithi107 ceph-mon[105022]: osdmap e189: 8 total, 8 up, 8 in 2024-09-21T10:04:43.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:43 smithi107 ceph-mon[105022]: 1.11 scrub starts 2024-09-21T10:04:43.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:43 smithi107 ceph-mon[105022]: 1.11 scrub ok 2024-09-21T10:04:43.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:43 smithi107 ceph-mon[105022]: 1.14 scrub starts 2024-09-21T10:04:43.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:43 smithi107 ceph-mon[105022]: 1.14 scrub ok 2024-09-21T10:04:43.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:43 smithi107 ceph-mon[105022]: 1.f scrub starts 2024-09-21T10:04:43.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:43 smithi107 ceph-mon[105022]: 1.f scrub ok 2024-09-21T10:04:43.863 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:43.889 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:43.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:43.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:43 smithi129 ceph-mon[88471]: osdmap e189: 8 total, 8 up, 8 in 2024-09-21T10:04:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:43 smithi129 ceph-mon[88471]: 1.11 scrub starts 2024-09-21T10:04:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:43 smithi129 ceph-mon[88471]: 1.11 scrub ok 2024-09-21T10:04:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:43 smithi129 ceph-mon[88471]: 1.14 scrub starts 2024-09-21T10:04:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:43 smithi129 ceph-mon[88471]: 1.14 scrub ok 2024-09-21T10:04:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:43 smithi129 ceph-mon[88471]: 1.f scrub starts 2024-09-21T10:04:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:43 smithi129 ceph-mon[88471]: 1.f scrub ok 2024-09-21T10:04:43.990 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:44.014 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:44.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:44.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:44.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:44.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:44.143 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:44.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:44.244 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:44.269 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:44.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:44.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:44.371 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:44.397 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:44.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:44.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:44.499 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:44.525 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:44.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:44.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:44.627 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:44.651 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:44.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:44.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:44.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:44.776 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:44.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:44.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:44.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:44 smithi107 ceph-mon[102221]: 1.4 scrub starts 2024-09-21T10:04:44.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:44 smithi107 ceph-mon[102221]: 1.4 scrub ok 2024-09-21T10:04:44.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:44 smithi107 ceph-mon[102221]: pgmap v665: 93 pgs: 5 peering, 88 active+clean; 530 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:44.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:44 smithi107 ceph-mon[102221]: 1.10 scrub starts 2024-09-21T10:04:44.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:44 smithi107 ceph-mon[102221]: 1.10 scrub ok 2024-09-21T10:04:44.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:44 smithi107 ceph-mon[102221]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:04:44.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:44 smithi107 ceph-mon[105022]: 1.4 scrub starts 2024-09-21T10:04:44.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:44 smithi107 ceph-mon[105022]: 1.4 scrub ok 2024-09-21T10:04:44.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:44 smithi107 ceph-mon[105022]: pgmap v665: 93 pgs: 5 peering, 88 active+clean; 530 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:44.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:44 smithi107 ceph-mon[105022]: 1.10 scrub starts 2024-09-21T10:04:44.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:44 smithi107 ceph-mon[105022]: 1.10 scrub ok 2024-09-21T10:04:44.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:44 smithi107 ceph-mon[105022]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:04:44.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:44.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:44.903 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:44.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:44 smithi129 ceph-mon[88471]: 1.4 scrub starts 2024-09-21T10:04:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:44 smithi129 ceph-mon[88471]: 1.4 scrub ok 2024-09-21T10:04:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:44 smithi129 ceph-mon[88471]: pgmap v665: 93 pgs: 5 peering, 88 active+clean; 530 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:44 smithi129 ceph-mon[88471]: 1.10 scrub starts 2024-09-21T10:04:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:44 smithi129 ceph-mon[88471]: 1.10 scrub ok 2024-09-21T10:04:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:44 smithi129 ceph-mon[88471]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:04:45.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:45.028 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:45.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:45.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:45.130 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:45.156 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:45.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:45.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:45.258 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:45.284 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:45.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:45.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:45.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:45.405 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:04:45.406 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:04:45.406 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-09-21T10:04:45.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_5 16 2024-09-21T10:04:45.410 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:45.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:45.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:45.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:45.533 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:45.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:45.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:45.635 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:45.657 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:45.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:45.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:45.699 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:45 smithi107 ceph-mon[102221]: 1.1 scrub starts 2024-09-21T10:04:45.700 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:45 smithi107 ceph-mon[102221]: 1.1 scrub ok 2024-09-21T10:04:45.700 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:45 smithi107 ceph-mon[102221]: 1.5 scrub starts 2024-09-21T10:04:45.700 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:45 smithi107 ceph-mon[102221]: 1.5 scrub ok 2024-09-21T10:04:45.700 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:45 smithi107 ceph-mon[102221]: 1.7 deep-scrub starts 2024-09-21T10:04:45.700 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:45 smithi107 ceph-mon[102221]: 1.7 deep-scrub ok 2024-09-21T10:04:45.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:45 smithi107 ceph-mon[105022]: 1.1 scrub starts 2024-09-21T10:04:45.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:45 smithi107 ceph-mon[105022]: 1.1 scrub ok 2024-09-21T10:04:45.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:45 smithi107 ceph-mon[105022]: 1.5 scrub starts 2024-09-21T10:04:45.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:45 smithi107 ceph-mon[105022]: 1.5 scrub ok 2024-09-21T10:04:45.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:45 smithi107 ceph-mon[105022]: 1.7 deep-scrub starts 2024-09-21T10:04:45.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:45 smithi107 ceph-mon[105022]: 1.7 deep-scrub ok 2024-09-21T10:04:45.759 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:45.785 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:45.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:45.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:45.887 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:45.912 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:45.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:45.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:45.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:45 smithi129 ceph-mon[88471]: 1.1 scrub starts 2024-09-21T10:04:45.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:45 smithi129 ceph-mon[88471]: 1.1 scrub ok 2024-09-21T10:04:45.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:45 smithi129 ceph-mon[88471]: 1.5 scrub starts 2024-09-21T10:04:45.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:45 smithi129 ceph-mon[88471]: 1.5 scrub ok 2024-09-21T10:04:45.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:45 smithi129 ceph-mon[88471]: 1.7 deep-scrub starts 2024-09-21T10:04:45.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:45 smithi129 ceph-mon[88471]: 1.7 deep-scrub ok 2024-09-21T10:04:46.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:46.035 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:46.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:46.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:46.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:46.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:46.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:46.261 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:46.288 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:46.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:46.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:46.389 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:46.414 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:46.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:46.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:46.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:46.537 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:46.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:46.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:46.604 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:04:46.604 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:04:46.604 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-09-21T10:04:46.605 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-09-21T10:04:46.605 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:04:46.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:46.660 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:46.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:46.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:46.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:46.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:46.783 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:46.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:46.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:46 smithi107 ceph-mon[102221]: pgmap v666: 93 pgs: 5 peering, 88 active+clean; 512 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:46.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:46 smithi107 ceph-mon[102221]: 1.12 scrub starts 2024-09-21T10:04:46.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:46 smithi107 ceph-mon[102221]: 1.12 scrub ok 2024-09-21T10:04:46.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:46 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:46 smithi107 ceph-mon[105022]: pgmap v666: 93 pgs: 5 peering, 88 active+clean; 512 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:46 smithi107 ceph-mon[105022]: 1.12 scrub starts 2024-09-21T10:04:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:46 smithi107 ceph-mon[105022]: 1.12 scrub ok 2024-09-21T10:04:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:46 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:46.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:46.906 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:46.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:46.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:46 smithi129 ceph-mon[88471]: pgmap v666: 93 pgs: 5 peering, 88 active+clean; 512 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:46 smithi129 ceph-mon[88471]: 1.12 scrub starts 2024-09-21T10:04:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:46 smithi129 ceph-mon[88471]: 1.12 scrub ok 2024-09-21T10:04:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:46 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:47.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:47.034 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:47.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:47.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:47.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:47.157 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:47.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:47.259 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:47.285 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:47.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:47.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:47.341 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:04:47.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:47.408 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:47.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:47.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:47.442 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:47.510 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:47.533 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:47.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:47.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:47.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:47 smithi107 ceph-mon[102221]: 1.c scrub starts 2024-09-21T10:04:47.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:47 smithi107 ceph-mon[102221]: 1.c scrub ok 2024-09-21T10:04:47.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:47 smithi107 ceph-mon[105022]: 1.c scrub starts 2024-09-21T10:04:47.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:47 smithi107 ceph-mon[105022]: 1.c scrub ok 2024-09-21T10:04:47.635 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:47.661 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:47.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:47.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:47.763 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:47.774 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:47.788 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:47.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:47.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:47.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:47.912 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:47.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:47.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:47.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:47 smithi129 ceph-mon[88471]: 1.c scrub starts 2024-09-21T10:04:47.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:47 smithi129 ceph-mon[88471]: 1.c scrub ok 2024-09-21T10:04:48.013 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:48.036 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:48.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:48.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:48.039 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:48.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:48.160 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:48.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:48.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:48.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:04:48.262 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:48.288 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:48.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:48.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:48.390 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:48.413 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:48.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:48.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:48.515 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:48.540 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:48.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:48.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:48.642 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:48.666 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:48.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:48.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:48.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:48.789 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:48.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:48.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:48.800 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:48 smithi107 ceph-mon[105022]: pgmap v667: 93 pgs: 5 peering, 88 active+clean; 771 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:48.800 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:48 smithi107 ceph-mon[105022]: 1.0 scrub starts 2024-09-21T10:04:48.800 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:48 smithi107 ceph-mon[105022]: 1.0 scrub ok 2024-09-21T10:04:48.800 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:48 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:48.800 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:48 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:48.800 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:48 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:48.801 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:48 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:04:48.801 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:48 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:48.801 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:48 smithi107 ceph-mon[102221]: pgmap v667: 93 pgs: 5 peering, 88 active+clean; 771 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:48.801 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:48 smithi107 ceph-mon[102221]: 1.0 scrub starts 2024-09-21T10:04:48.801 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:48 smithi107 ceph-mon[102221]: 1.0 scrub ok 2024-09-21T10:04:48.801 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:48 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:48.801 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:48 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:48.801 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:48 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:48.801 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:48 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:04:48.801 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:48 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:48.891 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:48.917 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:48.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:48.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:48.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:48 smithi129 ceph-mon[88471]: pgmap v667: 93 pgs: 5 peering, 88 active+clean; 771 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:48.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:48 smithi129 ceph-mon[88471]: 1.0 scrub starts 2024-09-21T10:04:48.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:48 smithi129 ceph-mon[88471]: 1.0 scrub ok 2024-09-21T10:04:48.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:48 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:48.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:48 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:48.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:48 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:48.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:48 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:04:48.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:48 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:49.019 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:49.044 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:49.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:49.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:49.145 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:49.171 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:49.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:49.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:49.272 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:49.294 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:49.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:49.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:49.396 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:49.422 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:49.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:49.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:49.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:49.545 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:49.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:49.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:49.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:49.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:49 smithi107 ceph-mon[102221]: 1.a scrub starts 2024-09-21T10:04:49.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:49 smithi107 ceph-mon[102221]: 1.a scrub ok 2024-09-21T10:04:49.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:49 smithi107 ceph-mon[102221]: 1.e scrub starts 2024-09-21T10:04:49.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:49 smithi107 ceph-mon[102221]: 1.e scrub ok 2024-09-21T10:04:49.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:49 smithi107 ceph-mon[102221]: 1.6 scrub starts 2024-09-21T10:04:49.651 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:49 smithi107 ceph-mon[102221]: 1.6 scrub ok 2024-09-21T10:04:49.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:49 smithi107 ceph-mon[105022]: 1.a scrub starts 2024-09-21T10:04:49.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:49 smithi107 ceph-mon[105022]: 1.a scrub ok 2024-09-21T10:04:49.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:49 smithi107 ceph-mon[105022]: 1.e scrub starts 2024-09-21T10:04:49.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:49 smithi107 ceph-mon[105022]: 1.e scrub ok 2024-09-21T10:04:49.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:49 smithi107 ceph-mon[105022]: 1.6 scrub starts 2024-09-21T10:04:49.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:49 smithi107 ceph-mon[105022]: 1.6 scrub ok 2024-09-21T10:04:49.669 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:49.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:49.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:49.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:49.792 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:49.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:49.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:49.894 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:49.915 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:49.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:49.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:49 smithi129 ceph-mon[88471]: 1.a scrub starts 2024-09-21T10:04:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:49 smithi129 ceph-mon[88471]: 1.a scrub ok 2024-09-21T10:04:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:49 smithi129 ceph-mon[88471]: 1.e scrub starts 2024-09-21T10:04:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:49 smithi129 ceph-mon[88471]: 1.e scrub ok 2024-09-21T10:04:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:49 smithi129 ceph-mon[88471]: 1.6 scrub starts 2024-09-21T10:04:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:49 smithi129 ceph-mon[88471]: 1.6 scrub ok 2024-09-21T10:04:50.018 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:50.043 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:50.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:50.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:50.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:50.166 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:50.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:50.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:50.268 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:50.294 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:50.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:50.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:50.394 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:50.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:50.396 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:50.419 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:50.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:50.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:50.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:50.542 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:50.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:50.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:50.568 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:04:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:04:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:50.644 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:50.670 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:50.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:50.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:50.747 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:50 smithi107 ceph-mon[102221]: pgmap v668: 93 pgs: 93 active+clean; 666 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:50.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:50 smithi107 ceph-mon[102221]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-21T10:04:50.748 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:50 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:04:50.748 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:50 smithi107 ceph-mon[105022]: pgmap v668: 93 pgs: 93 active+clean; 666 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:50.748 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:50 smithi107 ceph-mon[105022]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-21T10:04:50.748 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:50 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:04:50.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:50.796 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:50.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:50.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:50.898 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:50.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:50 smithi129 ceph-mon[88471]: pgmap v668: 93 pgs: 93 active+clean; 666 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:50.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:50 smithi129 ceph-mon[88471]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-21T10:04:50.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:50 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:04:50.924 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:50.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:50.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:51.026 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:51.051 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:51.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:51.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:51.152 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:51.176 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:51.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:51.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:51.278 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:51.301 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:51.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:51.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:51.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:51.424 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:51.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:51.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:51.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:51.549 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:51.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:51.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:51.573 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 10:04:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=cleanup t=2024-09-21T10:04:51.44967151Z level=info msg="Completed cleanup jobs" duration=27.880911ms 2024-09-21T10:04:51.651 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:51.675 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:51.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:51.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:51.768 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:51 smithi107 ceph-mon[105022]: 1.8 scrub starts 2024-09-21T10:04:51.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:51 smithi107 ceph-mon[105022]: 1.8 scrub ok 2024-09-21T10:04:51.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:51 smithi107 ceph-mon[105022]: 1.13 scrub starts 2024-09-21T10:04:51.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:51 smithi107 ceph-mon[105022]: 1.13 scrub ok 2024-09-21T10:04:51.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:51 smithi107 ceph-mon[102221]: 1.8 scrub starts 2024-09-21T10:04:51.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:51 smithi107 ceph-mon[102221]: 1.8 scrub ok 2024-09-21T10:04:51.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:51 smithi107 ceph-mon[102221]: 1.13 scrub starts 2024-09-21T10:04:51.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:51 smithi107 ceph-mon[102221]: 1.13 scrub ok 2024-09-21T10:04:51.777 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:51.799 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:51.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:51.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:51.855 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:51 smithi129 ceph-mon[88471]: 1.8 scrub starts 2024-09-21T10:04:51.855 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:51 smithi129 ceph-mon[88471]: 1.8 scrub ok 2024-09-21T10:04:51.855 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:51 smithi129 ceph-mon[88471]: 1.13 scrub starts 2024-09-21T10:04:51.855 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:51 smithi129 ceph-mon[88471]: 1.13 scrub ok 2024-09-21T10:04:51.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:51.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:51.926 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:51.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:52.028 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:52.050 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:52.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:52.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:52.101 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:04:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:04:51] "GET /metrics HTTP/1.1" 200 34293 "" "Prometheus/2.51.0" 2024-09-21T10:04:52.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:52.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:52.174 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:52.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:52.203 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:04:52.203 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:04:52.219 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:04:52.219 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:04:52.235 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:04:52.235 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:04:52.251 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:04:52.251 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:04:52.266 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:04:52.267 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:04:52.275 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:52.282 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:04:52.282 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:04:52.297 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:52.297 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:04:52.298 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:04:52.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:52.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:52.312 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:04:52.312 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:04:52.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:52.421 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:52.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:52.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:52.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:52.543 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:52.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:52.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:52.645 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:52.667 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:52.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:52.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:52.752 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:52 smithi107 ceph-mon[102221]: pgmap v669: 93 pgs: 93 active+clean; 666 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:52.752 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:52 smithi107 ceph-mon[102221]: 1.3 scrub starts 2024-09-21T10:04:52.752 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:52 smithi107 ceph-mon[102221]: 1.3 scrub ok 2024-09-21T10:04:52.752 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:52 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/320202235' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:52.752 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:52 smithi107 ceph-mon[105022]: pgmap v669: 93 pgs: 93 active+clean; 666 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:52.752 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:52 smithi107 ceph-mon[105022]: 1.3 scrub starts 2024-09-21T10:04:52.752 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:52 smithi107 ceph-mon[105022]: 1.3 scrub ok 2024-09-21T10:04:52.752 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:52 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/320202235' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:52.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:52.790 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:52.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:52.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:52.892 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:52.914 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:52.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:52.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:52 smithi129 ceph-mon[88471]: pgmap v669: 93 pgs: 93 active+clean; 666 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:52 smithi129 ceph-mon[88471]: 1.3 scrub starts 2024-09-21T10:04:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:52 smithi129 ceph-mon[88471]: 1.3 scrub ok 2024-09-21T10:04:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:52 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/320202235' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:04:53.017 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:53.039 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:53.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:53.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:53.140 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:53.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:53.163 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:53.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:53.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:53.285 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:53.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:53.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:53.387 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:53.411 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:53.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:53.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:53.512 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:53.536 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:53.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:53.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:53.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:53 smithi107 ceph-mon[105022]: 1.2 scrub starts 2024-09-21T10:04:53.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:53 smithi107 ceph-mon[105022]: 1.2 scrub ok 2024-09-21T10:04:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:53 smithi107 ceph-mon[102221]: 1.2 scrub starts 2024-09-21T10:04:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:53 smithi107 ceph-mon[102221]: 1.2 scrub ok 2024-09-21T10:04:53.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:53.660 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:53.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:53.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:53.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:53.783 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:53.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:53.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:53.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:53.907 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:53.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:53.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:53 smithi129 ceph-mon[88471]: 1.2 scrub starts 2024-09-21T10:04:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:53 smithi129 ceph-mon[88471]: 1.2 scrub ok 2024-09-21T10:04:54.009 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:54.031 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:54.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:54.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:54.132 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:54.158 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:54.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:54.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:54.260 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:54.284 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:54.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:54.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:54.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:54.408 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:54.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:54.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:54.510 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:54.532 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:54.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:54.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:54.552 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:04:54.552 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:04:54.591 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_5' created 2024-09-21T10:04:54.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:54.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:54.657 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:54.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:54.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:54.780 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:54.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:54.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:54 smithi107 ceph-mon[102221]: pgmap v670: 93 pgs: 93 active+clean; 739 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:54 smithi107 ceph-mon[102221]: 1.9 scrub starts 2024-09-21T10:04:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:54 smithi107 ceph-mon[102221]: 1.9 scrub ok 2024-09-21T10:04:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:54 smithi107 ceph-mon[102221]: 1.b scrub starts 2024-09-21T10:04:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:54 smithi107 ceph-mon[102221]: 1.b scrub ok 2024-09-21T10:04:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:54 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3154502302' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-21T10:04:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:54 smithi107 ceph-mon[105022]: pgmap v670: 93 pgs: 93 active+clean; 739 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:54.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:54 smithi107 ceph-mon[105022]: 1.9 scrub starts 2024-09-21T10:04:54.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:54 smithi107 ceph-mon[105022]: 1.9 scrub ok 2024-09-21T10:04:54.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:54 smithi107 ceph-mon[105022]: 1.b scrub starts 2024-09-21T10:04:54.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:54 smithi107 ceph-mon[105022]: 1.b scrub ok 2024-09-21T10:04:54.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:54 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3154502302' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-21T10:04:54.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:54.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:54.904 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:54.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:54 smithi129 ceph-mon[88471]: pgmap v670: 93 pgs: 93 active+clean; 739 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:54 smithi129 ceph-mon[88471]: 1.9 scrub starts 2024-09-21T10:04:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:54 smithi129 ceph-mon[88471]: 1.9 scrub ok 2024-09-21T10:04:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:54 smithi129 ceph-mon[88471]: 1.b scrub starts 2024-09-21T10:04:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:54 smithi129 ceph-mon[88471]: 1.b scrub ok 2024-09-21T10:04:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:54 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3154502302' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-21T10:04:55.005 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:55.031 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:55.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:55.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:55.133 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:55.159 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:55.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:55.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:55.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:55.281 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:55.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:55.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:55.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:04:55.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:55.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:55.415 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:55.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:55.516 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:55.542 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:55.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:55.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:55.644 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:55.670 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:55.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:55.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:55.771 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:55.797 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:55.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:55.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:55 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3154502302' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-21T10:04:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:55 smithi107 ceph-mon[102221]: osdmap e190: 8 total, 8 up, 8 in 2024-09-21T10:04:55.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:55 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3154502302' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-21T10:04:55.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:55 smithi107 ceph-mon[105022]: osdmap e190: 8 total, 8 up, 8 in 2024-09-21T10:04:55.899 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:55.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:55 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3154502302' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-21T10:04:55.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:55 smithi129 ceph-mon[88471]: osdmap e190: 8 total, 8 up, 8 in 2024-09-21T10:04:55.924 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:55.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:55.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:55.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-09-21T10:04:56.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:56.047 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:56.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:56.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:56.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:56.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:56.170 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:56.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:56.271 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:56.296 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:56.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:56.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:56.398 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:56.423 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:56.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:56.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:56.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:56.546 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:56.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:56.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:56.648 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:04:56.672 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:56.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:56.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:04:56.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:56.795 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:56.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:56.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:56 smithi107 ceph-mon[102221]: pgmap v672: 109 pgs: 3 creating+peering, 13 unknown, 93 active+clean; 687 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:56 smithi107 ceph-mon[102221]: 1.d scrub starts 2024-09-21T10:04:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:56 smithi107 ceph-mon[102221]: 1.d scrub ok 2024-09-21T10:04:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:56 smithi107 ceph-mon[102221]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:04:56.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:56 smithi107 ceph-mon[102221]: osdmap e191: 8 total, 8 up, 8 in 2024-09-21T10:04:56.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:56.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:56.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:56.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:56 smithi107 ceph-mon[105022]: pgmap v672: 109 pgs: 3 creating+peering, 13 unknown, 93 active+clean; 687 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:56.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:56 smithi107 ceph-mon[105022]: 1.d scrub starts 2024-09-21T10:04:56.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:56 smithi107 ceph-mon[105022]: 1.d scrub ok 2024-09-21T10:04:56.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:56 smithi107 ceph-mon[105022]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:04:56.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:56 smithi107 ceph-mon[105022]: osdmap e191: 8 total, 8 up, 8 in 2024-09-21T10:04:56.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:56.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:56.862 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:56.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:56 smithi129 ceph-mon[88471]: pgmap v672: 109 pgs: 3 creating+peering, 13 unknown, 93 active+clean; 687 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:56 smithi129 ceph-mon[88471]: 1.d scrub starts 2024-09-21T10:04:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:56 smithi129 ceph-mon[88471]: 1.d scrub ok 2024-09-21T10:04:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:56 smithi129 ceph-mon[88471]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:04:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:56 smithi129 ceph-mon[88471]: osdmap e191: 8 total, 8 up, 8 in 2024-09-21T10:04:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:56.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:04:56.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:56.920 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:56.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:57.022 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:57.046 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:57.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:57.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:57.147 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:57.171 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:57.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:57.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:57.273 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:57.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:57.296 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:57.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:57.397 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:57.421 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:57.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:57.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:57.523 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:57.545 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:57.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:57.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:57.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:04:57.668 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:57.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:57.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:04:57.682 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:57 smithi107 ceph-mon[102221]: osdmap e192: 8 total, 8 up, 8 in 2024-09-21T10:04:57.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:57 smithi107 ceph-mon[105022]: osdmap e192: 8 total, 8 up, 8 in 2024-09-21T10:04:57.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:57.792 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:57.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:57.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:57.839 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:57.894 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:04:57.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:57 smithi129 ceph-mon[88471]: osdmap e192: 8 total, 8 up, 8 in 2024-09-21T10:04:57.920 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:57.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:57.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:04:58.021 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:58.046 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:58.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:58.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:58.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:58.170 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:58.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:58.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:58.272 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:58.298 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:58.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:58.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:58.400 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:58.425 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:58.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:58.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:58.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:58.549 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:58.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:58.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:58.613 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T10:04:58.618 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:04:58.652 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:58.672 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:58.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:58.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:58.774 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:04:58.801 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:58.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:58.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:04:58.806 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:58 smithi107 ceph-mon[102221]: pgmap v675: 109 pgs: 3 creating+peering, 10 unknown, 96 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:58.806 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:58 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3970196423' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:04:58.806 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:58 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:04:58.806 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:58 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2337468540' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:04:58.806 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:58 smithi107 ceph-mon[105022]: pgmap v675: 109 pgs: 3 creating+peering, 10 unknown, 96 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:58.807 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:58 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3970196423' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:04:58.807 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:58 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:04:58.807 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:58 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2337468540' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:04:58.902 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:58 smithi129 ceph-mon[88471]: pgmap v675: 109 pgs: 3 creating+peering, 10 unknown, 96 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:58 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3970196423' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:04:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:58 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:04:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:58 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2337468540' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:04:58.924 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:58.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:58.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:59.025 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:59.051 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:59.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:59.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:59.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:59.175 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:59.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:59.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:59.277 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:04:59.298 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:59.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:04:59.401 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:04:59.427 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:04:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:59.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:04:59.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:59.550 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:59.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:59.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:59.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:59.672 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:59.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:59.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:59.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:04:59.796 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:59.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:59.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:04:59.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:04:59.919 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:04:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:59 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:04:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:59 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:04:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:59 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2337468540' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:04:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:04:59 smithi129 ceph-mon[88471]: osdmap e193: 8 total, 8 up, 8 in 2024-09-21T10:04:59.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:04:59.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:04:59.986 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:59 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:04:59.986 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:59 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:04:59.986 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:59 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2337468540' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:04:59.986 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:04:59 smithi107 ceph-mon[102221]: osdmap e193: 8 total, 8 up, 8 in 2024-09-21T10:04:59.986 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:59 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:04:59.986 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:59 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:04:59.986 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:59 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2337468540' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:04:59.986 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:04:59 smithi107 ceph-mon[105022]: osdmap e193: 8 total, 8 up, 8 in 2024-09-21T10:05:00.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:00.042 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:00.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:00.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:00.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:00.164 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:00.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:00.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:00.266 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:00.292 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:00.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:00.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:00.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:05:00.393 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:00.418 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:00.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:00.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:00.500 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:00.520 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:00.544 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:00.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:00.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:00.619 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:05:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:05:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:00.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:00.667 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:00.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:00.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:00.768 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:00.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:00.795 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:00.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:00.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:00.903 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:00 smithi107 ceph-mon[102221]: pgmap v677: 109 pgs: 3 creating+peering, 106 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:05:00.903 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:00 smithi107 ceph-mon[105022]: pgmap v677: 109 pgs: 3 creating+peering, 106 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:05:00.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:00 smithi129 ceph-mon[88471]: pgmap v677: 109 pgs: 3 creating+peering, 106 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:05:00.920 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:00.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:00.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:00.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T10:05:01.021 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:01.047 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:01.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:01.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:01.149 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:01.173 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:01.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:01.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:01.275 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:01.299 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:01.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:01.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:01.400 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:01.424 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:01.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:01.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:01.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:01.550 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:01.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:01.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:01.652 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:01.675 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:01.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:01.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:01.733 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:05:01.734 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:05:01.777 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:01.801 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:01.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:01.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:01.902 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:01.926 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:01.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:01.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:02.028 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:02.045 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:05:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:05:01] "GET /metrics HTTP/1.1" 200 36642 "" "Prometheus/2.51.0" 2024-09-21T10:05:02.051 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:02.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:02.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:02.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:02.174 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:02.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:02.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:02.276 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:02.297 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:02.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:02.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:02.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:02.421 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:02.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:02.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:02.475 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:05:02.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:05:02.523 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:02.583 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:02.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:02.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:02.686 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:02.708 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:02.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:02.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:02.810 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:02.836 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:02.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:02.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:02 smithi107 ceph-mon[102221]: pgmap v678: 109 pgs: 109 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:05:02.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:02 smithi107 ceph-mon[105022]: pgmap v678: 109 pgs: 109 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:05:02.912 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:02.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:02 smithi129 ceph-mon[88471]: pgmap v678: 109 pgs: 109 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:05:02.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:02.959 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:02.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:02.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:03.061 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:03.086 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:03.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:03.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:03.188 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:03.212 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:03.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:03.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:03.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:03.336 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:03.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:03.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:03.438 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:03.440 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:03.464 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:03.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:03.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:03.565 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:03.591 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:03.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:03.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:03.692 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:03.716 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:03.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:03.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:03.818 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:03.842 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:03.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:03.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:03.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:03.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:03.966 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:03.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:04.067 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:04.094 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:04.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:04.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:04.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:04.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:04.218 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:04.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:04.319 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:04.345 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:04.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:04.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:04.447 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:04.468 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:04.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:04.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:04.570 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:04.596 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:04.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:04.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:04.697 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:04.723 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:04.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:04.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:04.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:04.846 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:04.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:04.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:04 smithi129 ceph-mon[88471]: pgmap v679: 109 pgs: 109 active+clean; 582 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:05:04.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:04.969 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:04.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:04.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:05.064 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:04 smithi107 ceph-mon[102221]: pgmap v679: 109 pgs: 109 active+clean; 582 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:05:05.064 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:04 smithi107 ceph-mon[105022]: pgmap v679: 109 pgs: 109 active+clean; 582 KiB data, 4.7 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:05:05.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:05.092 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:05.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:05.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:05.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:05.216 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:05.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:05.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:05.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:05.340 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:05.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:05.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:05.441 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:05.468 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:05.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:05.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:05.570 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:05.595 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:05.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:05.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:05.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:05.718 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:05.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:05.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:05.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:05.841 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:05.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:05.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:05.934 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:06.261 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:06.287 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:06.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:06.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:06.389 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:06.415 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:06.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:06.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:06.516 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:06.540 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:06.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:06.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:06.631 INFO:teuthology.orchestra.run.smithi107.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-09-21T10:05:06.641 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:06.664 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:06.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:06.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:06.765 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:06.788 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:06.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:06.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:06.878 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:06 smithi107 ceph-mon[102221]: pgmap v680: 109 pgs: 109 active+clean; 582 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:05:06.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:06 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3155192913' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:06.879 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:06 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:05:06.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:06 smithi107 ceph-mon[105022]: pgmap v680: 109 pgs: 109 active+clean; 582 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:05:06.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:06 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3155192913' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:06.879 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:06 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:05:06.889 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:06.911 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:06.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:06.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:06.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:06 smithi129 ceph-mon[88471]: pgmap v680: 109 pgs: 109 active+clean; 582 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:05:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:06 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3155192913' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:05:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:06 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:05:07.013 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:07.036 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:07.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:07.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:07.138 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:07.161 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:07.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:07.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:07.263 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:07.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:07.285 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:07.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:07.386 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:07.409 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:07.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:07.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:07.511 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:07.534 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:07.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:07.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:07.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:07.637 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T10:05:07.661 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:07.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:07.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:07.762 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:07.787 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:07.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:07.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:07 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3155192913' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:07 smithi107 ceph-mon[102221]: osdmap e194: 8 total, 8 up, 8 in 2024-09-21T10:05:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:07 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/448376243' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:05:07.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:07 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3155192913' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:07.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:07 smithi107 ceph-mon[105022]: osdmap e194: 8 total, 8 up, 8 in 2024-09-21T10:05:07.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:07 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/448376243' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:05:07.889 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:07.912 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:07.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:07.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:07 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3155192913' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:05:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:07 smithi129 ceph-mon[88471]: osdmap e194: 8 total, 8 up, 8 in 2024-09-21T10:05:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:07 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/448376243' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:05:08.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:08.035 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:08.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:08.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:08.066 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:05:08.066 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:05:08.081 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:05:08.082 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:05:08.097 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:05:08.097 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:05:08.116 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:05:08.116 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:05:08.132 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:05:08.133 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:05:08.137 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:08.148 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:05:08.149 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:05:08.161 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:08.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:08.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:08.164 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:05:08.164 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:05:08.179 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:05:08.179 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:05:08.262 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:08.285 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:08.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:08.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:08.387 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:08.410 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:08.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:08.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:08.513 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:08.536 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:08.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:08.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:08.637 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:08.660 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:08.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:08.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:08.783 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:08.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:08.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:08.838 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:08 smithi107 ceph-mon[105022]: pgmap v682: 109 pgs: 109 active+clean; 582 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:05:08.838 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:08 smithi107 ceph-mon[105022]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:05:08.838 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:08 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/448376243' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:05:08.838 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:08 smithi107 ceph-mon[105022]: osdmap e195: 8 total, 8 up, 8 in 2024-09-21T10:05:08.839 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:08 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2286720282' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:08.839 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:08 smithi107 ceph-mon[102221]: pgmap v682: 109 pgs: 109 active+clean; 582 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:05:08.839 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:08 smithi107 ceph-mon[102221]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:05:08.839 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:08 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/448376243' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:05:08.839 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:08 smithi107 ceph-mon[102221]: osdmap e195: 8 total, 8 up, 8 in 2024-09-21T10:05:08.839 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:08 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2286720282' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:08.885 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:08.909 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:08.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:08.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:08 smithi129 ceph-mon[88471]: pgmap v682: 109 pgs: 109 active+clean; 582 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:05:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:08 smithi129 ceph-mon[88471]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:05:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:08 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/448376243' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:05:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:08 smithi129 ceph-mon[88471]: osdmap e195: 8 total, 8 up, 8 in 2024-09-21T10:05:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:08 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2286720282' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:09.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:09.032 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:09.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:09.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:09.134 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:09.159 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:09.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:09.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:09.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:09.282 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:09.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:09.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:09.384 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:09.410 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:09.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:09.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:09.512 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:09.530 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:09.536 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:09.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:09.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:09.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T10:05:09.578 DEBUG:teuthology.orchestra.run.smithi107:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-09-21T10:05:09.600 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:05:09.600 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:05:09.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:09.651 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:hints = 1 2024-09-21T10:05:09.651 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:05:09.651 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object prefix: benchmark_data_smithi107_184135 2024-09-21T10:05:09.651 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:05:09.651 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:05:09.668 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:09.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:09.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:09.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:09.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:09.819 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:09.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:09.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:09.958 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:09.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:09.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:10.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:10.094 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:10.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:10.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:10.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:10.231 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:10.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:10.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:10.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:10.369 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:10.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:10.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:10.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:10.505 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:10.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:10.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:10.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:10.642 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:05:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:05:10.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:10.643 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:10.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:10.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:10.651 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1 16 3916 3900 243.736 243.75 0.00523505 0.00408934 2024-09-21T10:05:10.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:10.780 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:10.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:10.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:10.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:10.917 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:10.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:10.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:10 smithi129 ceph-mon[88471]: pgmap v684: 109 pgs: 109 active+clean; 582 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:05:11.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:11.048 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:11.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:11.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:11.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:10 smithi107 ceph-mon[102221]: pgmap v684: 109 pgs: 109 active+clean; 582 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:05:11.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:10 smithi107 ceph-mon[105022]: pgmap v684: 109 pgs: 109 active+clean; 582 KiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:05:11.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:11.188 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:11.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:11.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:11.290 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:11.327 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:11.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:11.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:11.429 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:11.462 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:11.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:11.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:11.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:11.599 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:11.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:11.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:11.651 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2 16 7505 7489 234.014 224.312 0.00460911 0.00426451 2024-09-21T10:05:11.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:11.738 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:11.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:11.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:11.840 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:11.876 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:11.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:11.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:11.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:12.014 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:12.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:12.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:12.108 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:05:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:05:11] "GET /metrics HTTP/1.1" 200 36677 "" "Prometheus/2.51.0" 2024-09-21T10:05:12.116 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:12.153 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:12.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:12.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:12.254 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:12.289 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:12.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:12.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:12.391 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:12 smithi129 ceph-mon[88471]: pgmap v685: 109 pgs: 109 active+clean; 6.0 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 696 KiB/s wr, 10 op/s 2024-09-21T10:05:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:05:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:12.426 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:12.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:12.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:12.473 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:12 smithi107 ceph-mon[102221]: pgmap v685: 109 pgs: 109 active+clean; 6.0 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 696 KiB/s wr, 10 op/s 2024-09-21T10:05:12.473 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:05:12.473 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:12.473 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:12 smithi107 ceph-mon[105022]: pgmap v685: 109 pgs: 109 active+clean; 6.0 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 696 KiB/s wr, 10 op/s 2024-09-21T10:05:12.473 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:05:12.473 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:12.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:12.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:12.567 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:12.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:12.651 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3 16 11041 11025 229.67 221 0.00301239 0.00434775 2024-09-21T10:05:12.669 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:12.710 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:12.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:12.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:12.812 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:12.845 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:12.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:12.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:12.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:12.961 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:12.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:12.997 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:12.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:13.098 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:13.136 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:13.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:13.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:13.238 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:13.272 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:13.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:13.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:13.373 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:13.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:13.408 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:13.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:13.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:05:13.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:13.555 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:13.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:13.651 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 4 16 14190 14174 221.451 196.812 0.00315485 0.00450858 2024-09-21T10:05:13.657 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:13.689 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:13.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:13.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:13.791 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:13.820 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:13.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:13.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:13.922 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:13.951 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:13.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:13.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:14.053 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:14.087 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:14.088 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:05:14.088 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:05:14.088 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-09-21T10:05:14.088 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-09-21T10:05:14.088 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:05:14.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:14.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:14.168 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:05:14.189 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:14.223 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:14.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:14.294 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:14 smithi107 ceph-mon[105022]: pgmap v686: 109 pgs: 109 active+clean; 496 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 62 MiB/s wr, 991 op/s 2024-09-21T10:05:14.294 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:14 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3255954306' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:05:14.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:14 smithi107 ceph-mon[102221]: pgmap v686: 109 pgs: 109 active+clean; 496 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 62 MiB/s wr, 991 op/s 2024-09-21T10:05:14.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:14 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3255954306' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:05:14.326 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:14.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:14.361 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:14.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:14.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:14 smithi129 ceph-mon[88471]: pgmap v686: 109 pgs: 109 active+clean; 496 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 62 MiB/s wr, 991 op/s 2024-09-21T10:05:14.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:14 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3255954306' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:05:14.462 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:14.498 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:14.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:14.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:14.600 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:14.634 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:14.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:14.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:14.651 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 5 16 17012 16996 212.432 176.375 0.0016309 0.0046908 2024-09-21T10:05:14.736 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:14.769 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:14.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:14.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:14.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:14.911 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:14.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:14.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:15.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:15.042 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:15.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:15.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:15.144 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:15.182 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:15.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:15.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:15.284 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:15.319 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:15.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:15.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:15.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:15 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:05:15.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:15 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3255954306' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:05:15.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:15 smithi129 ceph-mon[88471]: osdmap e196: 8 total, 8 up, 8 in 2024-09-21T10:05:15.420 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:15.455 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:15.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:15.534 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:15 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:05:15.534 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:15 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3255954306' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:05:15.534 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:15 smithi107 ceph-mon[102221]: osdmap e196: 8 total, 8 up, 8 in 2024-09-21T10:05:15.535 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:15 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:05:15.535 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:15 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3255954306' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:05:15.535 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:15 smithi107 ceph-mon[105022]: osdmap e196: 8 total, 8 up, 8 in 2024-09-21T10:05:15.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:15.594 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:15.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:15.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:15.651 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 6 16 19668 19652 204.691 166 0.00397304 0.00487991 2024-09-21T10:05:15.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:15.733 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:15.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:15.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:15.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:15.871 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:15.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:15.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:15.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:16.005 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:16.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:16.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:16.107 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:16.145 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:16.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:16.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:16.248 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:16.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:05:16.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:16.303 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:16.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:16.404 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:16.441 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:16.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:16.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:16.545 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:16.580 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:16.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:16.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:16.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:16 smithi107 ceph-mon[102221]: pgmap v688: 109 pgs: 109 active+clean; 688 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-21T10:05:16.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:16 smithi107 ceph-mon[105022]: pgmap v688: 109 pgs: 109 active+clean; 688 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-21T10:05:16.652 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 7 16 22490 22474 200.643 176.375 0.00409352 0.00497848 2024-09-21T10:05:16.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:16 smithi129 ceph-mon[88471]: pgmap v688: 109 pgs: 109 active+clean; 688 MiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-21T10:05:16.682 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:16.717 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:16.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:16.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:16.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:16.874 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:16.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:16.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:16.976 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:17.010 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:17.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:17.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:17.112 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:17.149 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:17.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:17.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:17.251 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:17.285 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:17.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:17.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:17.343 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:17.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:17.408 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:17.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:17.428 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:17.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:17.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:17.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:17.566 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:17.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:17.652 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 8 16 25072 25056 195.733 161.375 0.00698927 0.00510384 2024-09-21T10:05:17.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:17.695 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:17.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:17.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:17.797 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:17.834 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:17.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:17.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:17.936 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:17.972 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:17.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:17.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:18.074 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:18.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:18.111 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:18.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:18.212 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:18.213 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:18 smithi107 ceph-mon[102221]: pgmap v689: 109 pgs: 109 active+clean; 859 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-09-21T10:05:18.213 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:18 smithi107 ceph-mon[105022]: pgmap v689: 109 pgs: 109 active+clean; 859 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-09-21T10:05:18.252 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:18.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:18.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:18.354 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:18.389 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:18.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:18.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:18.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:18.525 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:18.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:18.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:18.627 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:18.651 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 9 16 27374 27358 189.97 143.875 0.00847492 0.00525877 2024-09-21T10:05:18.660 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:18.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:18.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:18.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:18 smithi129 ceph-mon[88471]: pgmap v689: 109 pgs: 109 active+clean; 859 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 90 MiB/s wr, 1.45k op/s 2024-09-21T10:05:18.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:18.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:18.801 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:18.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:18.873 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:05:18.902 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:18.957 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:18.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:18.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:19.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:19.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:19.097 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:19.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:19.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:19.252 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:19.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:19.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:19.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:19.396 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:19.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:19.434 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:19 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2338053559' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:05:19.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:19 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:05:19.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:19 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2338053559' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:05:19.435 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:19 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:05:19.498 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:19.534 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:19.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:19.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:19.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:19.652 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 10 16 29718 29702 185.617 146.5 0.00816635 0.0053823 2024-09-21T10:05:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:19 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2338053559' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:05:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:19 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:05:19.670 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:19.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:19.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:19.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:19.811 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:19.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:19.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:19.913 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:19.949 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:19.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:19.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:20.050 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:20.086 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:20.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:20.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:20.188 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:20.225 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:20.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:20.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:20.327 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:20.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:20 smithi107 ceph-mon[102221]: pgmap v690: 109 pgs: 109 active+clean; 1.4 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-09-21T10:05:20.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:20 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:05:20.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:20 smithi107 ceph-mon[102221]: osdmap e197: 8 total, 8 up, 8 in 2024-09-21T10:05:20.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:20 smithi107 ceph-mon[105022]: pgmap v690: 109 pgs: 109 active+clean; 1.4 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-09-21T10:05:20.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:20 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:05:20.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:20 smithi107 ceph-mon[105022]: osdmap e197: 8 total, 8 up, 8 in 2024-09-21T10:05:20.362 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:20.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:20.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:20.464 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:20.491 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:20.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:20.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:20.593 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:20.632 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:20.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:20.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:20.652 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 11 16 31931 31915 181.316 138.312 0.00568364 0.0055096 2024-09-21T10:05:20.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:20 smithi129 ceph-mon[88471]: pgmap v690: 109 pgs: 109 active+clean; 1.4 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 142 MiB/s wr, 2.27k op/s 2024-09-21T10:05:20.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:20 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:05:20.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:20 smithi129 ceph-mon[88471]: osdmap e197: 8 total, 8 up, 8 in 2024-09-21T10:05:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:05:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:05:20.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:20.735 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:20.756 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:20.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:20.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:20.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:20.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:20.882 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:20.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:20.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:21.015 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:21.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:21.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:21.117 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:21.152 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:21.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:21.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:21.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:21.299 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:21.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:21.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:21.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:21.430 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:21.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:21.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:21.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:21.567 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:21.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:21.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:21.652 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 12 16 33659 33643 175.206 108 0.00406334 0.00570248 2024-09-21T10:05:21.669 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:21.705 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:21.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:21.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:21.807 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:21.829 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:21.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:21.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:21.931 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:21.960 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:21.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:22.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:22.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:22.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:22.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:05:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:05:21] "GET /metrics HTTP/1.1" 200 36677 "" "Prometheus/2.51.0" 2024-09-21T10:05:22.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:22.229 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:22.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:22.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:22.330 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:22.365 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:22.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:22.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:22.467 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:22.503 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:22.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:22.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:22.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:22.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:22 smithi107 ceph-mon[102221]: pgmap v692: 109 pgs: 109 active+clean; 1.5 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 134 MiB/s wr, 2.15k op/s 2024-09-21T10:05:22.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:22 smithi107 ceph-mon[105022]: pgmap v692: 109 pgs: 109 active+clean; 1.5 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 134 MiB/s wr, 2.15k op/s 2024-09-21T10:05:22.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:22.641 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:22.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:22.652 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 13 16 35436 35420 170.271 111.062 0.00700919 0.0058673 2024-09-21T10:05:22.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:22 smithi129 ceph-mon[88471]: pgmap v692: 109 pgs: 109 active+clean; 1.5 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 134 MiB/s wr, 2.15k op/s 2024-09-21T10:05:22.742 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:22.763 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:22.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:22.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:22.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:22.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:22.901 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:22.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:23.002 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:23.045 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:23.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:23.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:23.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:23.168 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:23.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:23.269 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:23.304 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:23.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:23.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:23.406 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:23.438 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:23.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:23.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:23.540 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:23.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:23.571 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:23.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:23.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:23 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4155192880' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:23.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:23 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4155192880' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:23.652 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 14 16 37012 36996 165.144 98.5 0.00593975 0.00604982 2024-09-21T10:05:23.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:23 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4155192880' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:23.672 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:23.705 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:23.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:23.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:23.732 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:05:23.732 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:05:23.732 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:05:23.732 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:05:23.732 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:05:23.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:05:23.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:05:23.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:05:23.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:05:23.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:05:23.733 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:05:23.733 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:05:23.733 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:05:23.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:05:23.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:05:23.734 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:05:23.807 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:23.838 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:23.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:23.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:23.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:23.971 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:23.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:23.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:24.073 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:24.109 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:24.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:24.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:24.211 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:24.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:24.249 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:24.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:24.350 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:24.382 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:24.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:24.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:24.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:24.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:24.519 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:24.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:24.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:24 smithi107 ceph-mon[102221]: pgmap v693: 109 pgs: 109 active+clean; 2.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 180 MiB/s wr, 2.88k op/s 2024-09-21T10:05:24.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:24 smithi107 ceph-mon[105022]: pgmap v693: 109 pgs: 109 active+clean; 2.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 180 MiB/s wr, 2.88k op/s 2024-09-21T10:05:24.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:24.652 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 15 16 39003 38987 162.429 124.438 0.00909828 0.00615168 2024-09-21T10:05:24.655 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:24.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:24.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:24.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:24 smithi129 ceph-mon[88471]: pgmap v693: 109 pgs: 109 active+clean; 2.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 180 MiB/s wr, 2.88k op/s 2024-09-21T10:05:24.757 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:24.793 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:24.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:24.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:24.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:24.934 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:24.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:24.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:25.036 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:25.069 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:25.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:25.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:25.170 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:25.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:25.207 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:25.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:25.308 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:25.344 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:25.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:25.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:25.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:25.485 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:25.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:25.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:25.586 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:25.619 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:25.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:25.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:25.652 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 16 16 41005 40989 160.097 125.125 0.00354437 0.00624112 2024-09-21T10:05:25.721 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:25.742 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:25.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:25.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:25.844 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:25.877 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:25.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:25.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:25.979 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:26.000 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:26.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:26.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:26.102 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:26.123 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:26.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:26.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:26.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:26.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:26.250 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:26.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:26.352 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:26.382 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:26.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:26.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:26.484 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:26.517 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:26.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:26.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:26.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:26 smithi107 ceph-mon[102221]: pgmap v694: 109 pgs: 109 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 153 MiB/s wr, 2.44k op/s 2024-09-21T10:05:26.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:26.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:26 smithi107 ceph-mon[105022]: pgmap v694: 109 pgs: 109 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 153 MiB/s wr, 2.44k op/s 2024-09-21T10:05:26.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:26.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:26.653 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 17 16 42109 42093 154.738 69 0.00448354 0.00645677 2024-09-21T10:05:26.655 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:26.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:26.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:26.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:26 smithi129 ceph-mon[88471]: pgmap v694: 109 pgs: 109 active+clean; 2.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 153 MiB/s wr, 2.44k op/s 2024-09-21T10:05:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:26.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:26.794 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:26.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:26.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:26.896 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:26.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:26.937 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:26.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:27.038 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:27.074 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:27.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:27.176 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:27.207 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:27.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:27.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:27.308 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:27.345 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:27.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:27.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:27.446 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:27.478 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:27.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:27.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:27.580 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:27.616 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:27.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:27.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:27.652 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 18 16 43989 43973 152.669 117.5 0.00671329 0.00654473 2024-09-21T10:05:27.718 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:27.751 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:27.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:27.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:27.853 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:27.888 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:27.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:27.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:27.908 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T10:05:27.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:28.030 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:28.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:28.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:28.132 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:28.168 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:28.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:28.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:28.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:28.307 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:28.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:28.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:28.409 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:28.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:28.446 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:28.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:28.534 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:28 smithi107 ceph-mon[102221]: pgmap v695: 109 pgs: 109 active+clean; 2.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 146 MiB/s wr, 2.33k op/s 2024-09-21T10:05:28.534 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:28 smithi107 ceph-mon[105022]: pgmap v695: 109 pgs: 109 active+clean; 2.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 146 MiB/s wr, 2.33k op/s 2024-09-21T10:05:28.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:28.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T10:05:28.591 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:28.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:28.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:28.595 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:28.653 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 19 16 45962 45946 151.121 123.312 0.00648449 0.00661229 2024-09-21T10:05:28.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:28 smithi129 ceph-mon[88471]: pgmap v695: 109 pgs: 109 active+clean; 2.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 146 MiB/s wr, 2.33k op/s 2024-09-21T10:05:28.695 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:28.728 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:28.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:28.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:28.829 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:28.865 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:28.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:28.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:28.967 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:29.000 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:29.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:29.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:29.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:29.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:29.131 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:29.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:29.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:29.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:29.272 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:29.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:29.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:29.405 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:29.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:29.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:29.506 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:29.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:29.542 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:29.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:29.575 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:05:29.575 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:05:29.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:29.653 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:05:29.652431+0000 min lat: 0.000982552 max lat: 0.416605 avg lat: 0.00670032 2024-09-21T10:05:29.653 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:05:29.653 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 20 16 47747 47731 149.142 111.562 0.00640949 0.00670032 2024-09-21T10:05:29.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:29.680 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:29.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:29.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:29.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:05:29.820 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:29.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:29.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:29.923 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:29.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:29.962 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:29.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:30.064 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:30.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:30.101 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:30.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:30.202 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:30.233 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:30.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:30.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:30.335 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:30.370 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:30.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:30.472 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:30.498 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:30.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:30.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:30.600 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:30.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:30 smithi107 ceph-mon[102221]: pgmap v696: 109 pgs: 109 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 123 MiB/s wr, 1.97k op/s 2024-09-21T10:05:30.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:30 smithi107 ceph-mon[105022]: pgmap v696: 109 pgs: 109 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 123 MiB/s wr, 1.97k op/s 2024-09-21T10:05:30.639 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:30.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:30.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:30.653 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 21 16 49396 49380 146.948 103.062 0.00758174 0.00679991 2024-09-21T10:05:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:30 smithi129 ceph-mon[88471]: pgmap v696: 109 pgs: 109 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 123 MiB/s wr, 1.97k op/s 2024-09-21T10:05:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:05:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:05:30.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:30.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:30.777 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:30.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:30.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:30.879 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:30.914 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:30.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:30.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:31.016 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:31.051 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:31.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:31.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:31.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:31.186 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:31.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:31.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:31.288 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:31.324 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:31.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:31.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:31.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:31.464 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:31.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:31.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:31.534 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:05:31.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:31.586 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:31.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:31.623 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:31.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:31.653 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 22 16 51233 51217 145.486 114.812 0.0129785 0.00686813 2024-09-21T10:05:31.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:31.760 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:31.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:31.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:31.861 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:31.923 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:31.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:31.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:31.977 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:05:32.027 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:32.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:32.061 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:32.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:32.111 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:05:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:05:31] "GET /metrics HTTP/1.1" 200 36687 "" "Prometheus/2.51.0" 2024-09-21T10:05:32.164 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:32.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:32.210 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:32.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:32.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:32.351 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:32.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:32.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:32.382 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:32 smithi129 ceph-mon[88471]: pgmap v697: 109 pgs: 109 active+clean; 2.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-21T10:05:32.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:32 smithi107 ceph-mon[102221]: pgmap v697: 109 pgs: 109 active+clean; 2.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-21T10:05:32.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:32 smithi107 ceph-mon[105022]: pgmap v697: 109 pgs: 109 active+clean; 2.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s 2024-09-21T10:05:32.452 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:32.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:32.480 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:32.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:32.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:32.625 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:32.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:32.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:32.653 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 23 16 52801 52785 143.421 98 0.0128453 0.00696834 2024-09-21T10:05:32.727 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:32.749 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:32.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:32.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:32.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:32.887 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:32.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:32.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:32.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:33.025 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:33.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:33.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:33.027 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:33.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:33.164 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:33.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:33.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:33.265 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:33.301 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:33.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:33.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:33.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:33.441 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:33.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:33.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:33.543 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:33.574 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:33.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:33.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:33.654 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 24 15 54373 54358 141.541 98.3125 0.0172653 0.00706023 2024-09-21T10:05:33.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:33.719 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:33.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:33.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:33.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:33.854 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:33.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:33.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:33.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:33.995 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:33.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:33.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:34.098 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:34.130 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:34.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:34.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:34.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:34.274 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:34.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:34.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:34.323 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:34 smithi129 ceph-mon[88471]: pgmap v698: 109 pgs: 109 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-21T10:05:34.375 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:34.408 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:34.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:34.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:34.479 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:05:34.479 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:05:34.509 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:34.546 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:34.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:34.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:34.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:34 smithi107 ceph-mon[102221]: pgmap v698: 109 pgs: 109 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-21T10:05:34.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:34 smithi107 ceph-mon[105022]: pgmap v698: 109 pgs: 109 active+clean; 3.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-21T10:05:34.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:34.653 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 25 16 55979 55963 139.892 100.312 0.00904547 0.00714319 2024-09-21T10:05:34.683 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:34.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:34.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:34.785 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:34.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:34.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:34.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:34.943 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:34.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:34.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:35.045 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:35.077 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:35.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:35.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:35.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:35.181 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:05:35.201 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:35.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:35.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:35.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:35.337 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:35.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:35.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:35.440 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:35.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:35.481 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:35.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:35.561 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:05:35.582 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:35.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:35.622 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:35.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:35.654 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 26 16 57449 57433 138.045 91.875 0.0112505 0.00723918 2024-09-21T10:05:35.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:35.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:35.761 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:35.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:35.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:35.883 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:35.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:35.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:35.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:36.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:36.018 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:36.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:36.119 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:36.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:36.151 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:36.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:36.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:36.281 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:36.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:36.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:36.382 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:36.418 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:36.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:36.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:36.520 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:36.548 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:36.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:36.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:36.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:36 smithi107 ceph-mon[102221]: pgmap v699: 109 pgs: 109 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-21T10:05:36.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:36 smithi107 ceph-mon[105022]: pgmap v699: 109 pgs: 109 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-21T10:05:36.649 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:36.654 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 27 16 58755 58739 135.955 81.625 0.009209 0.00735041 2024-09-21T10:05:36.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:36 smithi129 ceph-mon[88471]: pgmap v699: 109 pgs: 109 active+clean; 3.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-21T10:05:36.684 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:36.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:36.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:36.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:36.807 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:36.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:36.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:36.909 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:36.948 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:36.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:36.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:37.049 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:37.087 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:37.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:37.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:37.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:37.231 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:37.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:37.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:37.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:37.384 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:37.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:37.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:37.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:37.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:37.523 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:37.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:37.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:37.654 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 28 15 60096 60081 134.095 83.875 0.0084558 0.00745278 2024-09-21T10:05:37.662 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:37.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:37.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:37.764 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:37.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:37.802 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:37.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:37.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:37.941 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:37.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:38.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:38.092 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:38.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:38.194 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:38.232 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:38.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:38.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:38.252 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:05:38.258 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T10:05:38.282 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:05:38.282 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:05:38.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:38.371 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:38.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:38.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:38.399 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:38 smithi107 ceph-mon[102221]: pgmap v700: 109 pgs: 109 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:05:38.399 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:38 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1727056466' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:05:38.399 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:38 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2800331350' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:05:38.399 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:38 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:05:38.399 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:38 smithi107 ceph-mon[105022]: pgmap v700: 109 pgs: 109 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:05:38.399 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:38 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1727056466' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:05:38.399 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:38 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2800331350' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:05:38.399 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:38 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:05:38.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:38.511 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:38.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:38.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:38.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:38.636 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:38 smithi129 ceph-mon[88471]: pgmap v700: 109 pgs: 109 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:05:38.636 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:38 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1727056466' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:05:38.636 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:38 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2800331350' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:05:38.636 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:38 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:05:38.647 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:38.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:38.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:38.654 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 29 16 61636 61620 132.787 96.1875 0.0159287 0.0075254 2024-09-21T10:05:38.749 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:38.784 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:38.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:38.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:38.886 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:38.921 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:38.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:38.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:38.982 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:05:39.023 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:39.071 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:39.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:39.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:39.173 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:39.215 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:39.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:39.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:39.274 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:39 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:05:39.275 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:39 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1727056466' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:05:39.275 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:39 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:05:39.275 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:39 smithi129 ceph-mon[88471]: osdmap e198: 8 total, 8 up, 8 in 2024-09-21T10:05:39.317 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:39.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:39 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:05:39.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:39 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1727056466' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:05:39.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:39 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:05:39.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:39 smithi107 ceph-mon[102221]: osdmap e198: 8 total, 8 up, 8 in 2024-09-21T10:05:39.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:39 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:05:39.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:39 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1727056466' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:05:39.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:39 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:05:39.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:39 smithi107 ceph-mon[105022]: osdmap e198: 8 total, 8 up, 8 in 2024-09-21T10:05:39.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:39.368 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:39.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:39.442 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:05:39.469 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:39.518 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:39.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:39.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:39.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:39.654 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:39.654 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 30 16 63186 63170 131.59 96.875 0.0149552 0.00759355 2024-09-21T10:05:39.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:39.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:39.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:39.790 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:39.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:39.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:39.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T10:05:39.892 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:39.927 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:39.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:39.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:40.028 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:40.060 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:40.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:40.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:40.149 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:05:40.150 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:05:40.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:40.188 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:05:40.189 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:05:40.196 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:40.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:40.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:40.227 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:05:40.228 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:05:40.270 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:05:40.270 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:05:40.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:40.302 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:05:40.302 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:05:40.317 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:40.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:40 smithi107 ceph-mon[102221]: pgmap v702: 109 pgs: 109 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-21T10:05:40.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:40 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3536816830' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:40.332 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:05:40.332 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:05:40.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:40 smithi107 ceph-mon[105022]: pgmap v702: 109 pgs: 109 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-21T10:05:40.332 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:40 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3536816830' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:40.346 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:40.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:40.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:40.361 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:05:40.362 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:05:40.394 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:05:40.394 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:05:40.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:40.485 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:40.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:40.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:40.587 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:40.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:40.627 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:40.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:40.654 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 31 16 64652 64636 130.301 91.625 0.0106211 0.00766929 2024-09-21T10:05:40.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:40 smithi129 ceph-mon[88471]: pgmap v702: 109 pgs: 109 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 103 MiB/s wr, 1.65k op/s 2024-09-21T10:05:40.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:40 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3536816830' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:40.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:05:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:05:40.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:40.728 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:40.772 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:40.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:40.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:40.874 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:40.918 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:40.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:40.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:41.019 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:41.060 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:41.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:41.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:41.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:05:41.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:41.199 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:41.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:41.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:41.301 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:41.343 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:41.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:41.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:41.445 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:41.484 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:05:41.484 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 20, 2024-09-21T10:05:41.484 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 600, 2024-09-21T10:05:41.484 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:05:41.484 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.485 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45320.0:60871 8.b 8:d15c6a5c:::benchmark_data_smithi107_135256_object60870:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e123)", 2024-09-21T10:05:41.485 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T09:55:49.417008+0000", 2024-09-21T10:05:41.485 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 592.06194807600002, 2024-09-21T10:05:41.485 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.013777819, 2024-09-21T10:05:41.485 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:41.485 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:05:41.485 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:41.485 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45320", 2024-09-21T10:05:41.485 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:05:41.485 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 60871 2024-09-21T10:05:41.485 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.485 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:41.485 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.486 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:41.486 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:55:49.417008+0000", 2024-09-21T10:05:41.486 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.486 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.486 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.486 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:41.486 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:55:49.417008+0000", 2024-09-21T10:05:41.486 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.486 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.486 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.486 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:41.486 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:55:49.417007+0000", 2024-09-21T10:05:41.486 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:05:41.486 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.487 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.487 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:41.487 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:55:49.417041+0000", 2024-09-21T10:05:41.487 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3980999999999999e-05 2024-09-21T10:05:41.487 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.487 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.487 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:41.487 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:55:49.417051+0000", 2024-09-21T10:05:41.487 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.8970000000000003e-06 2024-09-21T10:05:41.487 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.487 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.487 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:41.487 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:55:49.417057+0000", 2024-09-21T10:05:41.488 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.553e-06 2024-09-21T10:05:41.488 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.488 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.488 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.488 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:55:49.417246+0000", 2024-09-21T10:05:41.488 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00018948599999999999 2024-09-21T10:05:41.488 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.488 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.488 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:05:41.488 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:55:49.417252+0000", 2024-09-21T10:05:41.488 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2369999999999997e-06 2024-09-21T10:05:41.488 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.488 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.488 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.489 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:55:50.429287+0000", 2024-09-21T10:05:41.489 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.012034914 2024-09-21T10:05:41.489 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.489 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.489 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:05:41.489 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:55:50.430786+0000", 2024-09-21T10:05:41.489 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001498601 2024-09-21T10:05:41.489 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.489 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.489 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.489 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.489 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.489 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45320.0:142845 8.2 8:4336792e:::benchmark_data_smithi107_135256_object16961:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T10:05:41.489 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T09:57:09.879273+0000", 2024-09-21T10:05:41.490 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 511.59968293499998, 2024-09-21T10:05:41.490 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.127965938, 2024-09-21T10:05:41.490 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:41.490 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:41.490 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:41.490 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45320", 2024-09-21T10:05:41.490 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:05:41.490 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 142845 2024-09-21T10:05:41.490 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.490 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:41.490 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.490 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:41.490 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.879273+0000", 2024-09-21T10:05:41.490 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.491 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.491 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.491 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:41.491 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.879273+0000", 2024-09-21T10:05:41.491 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.491 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.491 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.491 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:41.491 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.879272+0000", 2024-09-21T10:05:41.491 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:05:41.491 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.491 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.491 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:41.492 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.879278+0000", 2024-09-21T10:05:41.492 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2859999999999996e-06 2024-09-21T10:05:41.492 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.492 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.492 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:41.492 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.879290+0000", 2024-09-21T10:05:41.492 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1534e-05 2024-09-21T10:05:41.492 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.492 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.492 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:41.492 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.879294+0000", 2024-09-21T10:05:41.492 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5090000000000004e-06 2024-09-21T10:05:41.492 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.492 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.493 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.493 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.879335+0000", 2024-09-21T10:05:41.493 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1128999999999999e-05 2024-09-21T10:05:41.493 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.493 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.493 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:41.493 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.879437+0000", 2024-09-21T10:05:41.493 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000101432 2024-09-21T10:05:41.493 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.493 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.493 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:05:41.493 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.879544+0000", 2024-09-21T10:05:41.493 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000107255 2024-09-21T10:05:41.493 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.494 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.494 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:05:41.494 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.887610+0000", 2024-09-21T10:05:41.494 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0080659019999999998 2024-09-21T10:05:41.494 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.494 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.494 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.494 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.887643+0000", 2024-09-21T10:05:41.494 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2951999999999999e-05 2024-09-21T10:05:41.494 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.494 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.494 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.494 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:10.007184+0000", 2024-09-21T10:05:41.495 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.119540761 2024-09-21T10:05:41.495 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.495 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.495 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:05:41.495 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:10.007216+0000", 2024-09-21T10:05:41.495 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2391999999999998e-05 2024-09-21T10:05:41.495 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.495 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.495 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:05:41.495 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:10.007239+0000", 2024-09-21T10:05:41.495 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.3096999999999999e-05 2024-09-21T10:05:41.495 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.495 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.495 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.495 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.496 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.496 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45320.0:142847 8.2 8:4a7aabe9:::benchmark_data_smithi107_135256_object16963:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-21T10:05:41.496 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T09:57:09.879477+0000", 2024-09-21T10:05:41.496 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 511.59947905400003, 2024-09-21T10:05:41.496 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.13070955000000001, 2024-09-21T10:05:41.496 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:41.496 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:41.496 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:41.496 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45320", 2024-09-21T10:05:41.496 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:05:41.496 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 142847 2024-09-21T10:05:41.496 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.496 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:41.496 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.497 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:41.497 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.879477+0000", 2024-09-21T10:05:41.497 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.497 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.497 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.497 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:41.497 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.879477+0000", 2024-09-21T10:05:41.497 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.497 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.497 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.497 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:41.497 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.879476+0000", 2024-09-21T10:05:41.497 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:05:41.498 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.498 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.498 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:41.498 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.879484+0000", 2024-09-21T10:05:41.498 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.7349999999999996e-06 2024-09-21T10:05:41.498 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.498 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.498 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:41.499 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:41.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:41.500 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.879494+0000", 2024-09-21T10:05:41.500 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0186e-05 2024-09-21T10:05:41.500 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.500 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.500 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:41.500 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.879498+0000", 2024-09-21T10:05:41.500 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3959999999999999e-06 2024-09-21T10:05:41.500 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.500 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.500 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.500 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.883576+0000", 2024-09-21T10:05:41.500 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0040776529999999997 2024-09-21T10:05:41.501 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.501 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.501 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:41.501 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.883689+0000", 2024-09-21T10:05:41.501 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000113233 2024-09-21T10:05:41.501 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.501 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.501 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:05:41.501 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.883795+0000", 2024-09-21T10:05:41.501 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000105942 2024-09-21T10:05:41.501 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.501 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.501 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:05:41.501 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.887745+0000", 2024-09-21T10:05:41.502 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0039496610000000001 2024-09-21T10:05:41.502 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.502 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.502 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.502 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:09.888548+0000", 2024-09-21T10:05:41.502 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000803289 2024-09-21T10:05:41.502 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.502 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.502 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.502 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:10.010116+0000", 2024-09-21T10:05:41.502 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.121568173 2024-09-21T10:05:41.502 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.502 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.502 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:05:41.503 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:10.010150+0000", 2024-09-21T10:05:41.503 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3952999999999998e-05 2024-09-21T10:05:41.503 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.503 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.503 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:05:41.503 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:10.010187+0000", 2024-09-21T10:05:41.503 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6191e-05 2024-09-21T10:05:41.503 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.503 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.503 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.503 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.503 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.503 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45320.0:189071 8.2 8:4ace8ae9:::benchmark_data_smithi107_135256_object63187:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-21T10:05:41.503 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T09:58:02.830968+0000", 2024-09-21T10:05:41.503 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 458.64798819800001, 2024-09-21T10:05:41.504 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.13139941699999999, 2024-09-21T10:05:41.504 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:41.504 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:41.504 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:41.504 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45320", 2024-09-21T10:05:41.504 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:05:41.504 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 189071 2024-09-21T10:05:41.504 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.504 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:41.504 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.504 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:41.504 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.830968+0000", 2024-09-21T10:05:41.504 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.504 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.505 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.505 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:41.505 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.830968+0000", 2024-09-21T10:05:41.505 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.505 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.505 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.505 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:41.505 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.830967+0000", 2024-09-21T10:05:41.505 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:05:41.505 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.505 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.505 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:41.505 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.830972+0000", 2024-09-21T10:05:41.505 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7740000000000001e-06 2024-09-21T10:05:41.506 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.506 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.506 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:41.506 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.830984+0000", 2024-09-21T10:05:41.506 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1218e-05 2024-09-21T10:05:41.506 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.506 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.506 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:41.506 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.830988+0000", 2024-09-21T10:05:41.506 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7589999999999997e-06 2024-09-21T10:05:41.506 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.506 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.506 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.506 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.831037+0000", 2024-09-21T10:05:41.507 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8562e-05 2024-09-21T10:05:41.507 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.507 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.507 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:41.507 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.949166+0000", 2024-09-21T10:05:41.507 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.11812872000000001 2024-09-21T10:05:41.507 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.507 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.507 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:05:41.507 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.949288+0000", 2024-09-21T10:05:41.507 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012243200000000001 2024-09-21T10:05:41.507 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.507 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.507 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.508 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.962087+0000", 2024-09-21T10:05:41.508 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.012799389 2024-09-21T10:05:41.508 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.508 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.508 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:05:41.508 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.962095+0000", 2024-09-21T10:05:41.508 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9589999999999995e-06 2024-09-21T10:05:41.508 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.508 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.508 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.508 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.962215+0000", 2024-09-21T10:05:41.508 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000119178 2024-09-21T10:05:41.508 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.508 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.508 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:05:41.509 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.962329+0000", 2024-09-21T10:05:41.509 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011404300000000001 2024-09-21T10:05:41.509 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.509 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.509 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:05:41.509 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.962367+0000", 2024-09-21T10:05:41.509 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8612000000000002e-05 2024-09-21T10:05:41.509 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.509 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.509 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.509 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.509 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.509 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45320.0:189072 8.2 8:417df2e5:::benchmark_data_smithi107_135256_object63188:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-21T10:05:41.509 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T09:58:02.831191+0000", 2024-09-21T10:05:41.509 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 458.64776513300001, 2024-09-21T10:05:41.510 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.13135862100000001, 2024-09-21T10:05:41.510 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:41.510 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:41.510 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:41.510 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45320", 2024-09-21T10:05:41.510 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:05:41.510 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 189072 2024-09-21T10:05:41.510 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.510 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:41.510 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.510 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:41.510 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.831191+0000", 2024-09-21T10:05:41.510 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.510 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.511 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.511 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:41.511 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.831191+0000", 2024-09-21T10:05:41.511 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.511 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.511 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.511 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:41.511 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.831190+0000", 2024-09-21T10:05:41.511 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:05:41.511 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.511 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.511 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:41.511 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.831194+0000", 2024-09-21T10:05:41.511 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4230000000000001e-06 2024-09-21T10:05:41.511 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.512 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.512 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:41.512 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.949108+0000", 2024-09-21T10:05:41.512 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.117914068 2024-09-21T10:05:41.512 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.512 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.512 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:41.512 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.949122+0000", 2024-09-21T10:05:41.512 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4075e-05 2024-09-21T10:05:41.512 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.512 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.512 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.512 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.953320+0000", 2024-09-21T10:05:41.512 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0041983749999999999 2024-09-21T10:05:41.512 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.513 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.513 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:41.513 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.953586+0000", 2024-09-21T10:05:41.513 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00026533200000000001 2024-09-21T10:05:41.513 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.513 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.513 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:05:41.513 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.953708+0000", 2024-09-21T10:05:41.513 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012233900000000001 2024-09-21T10:05:41.513 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.513 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.513 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:05:41.513 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.962101+0000", 2024-09-21T10:05:41.513 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0083932710000000008 2024-09-21T10:05:41.514 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.514 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.514 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.514 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.962181+0000", 2024-09-21T10:05:41.514 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9765999999999995e-05 2024-09-21T10:05:41.514 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.514 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.514 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.514 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.962466+0000", 2024-09-21T10:05:41.514 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00028497100000000002 2024-09-21T10:05:41.514 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.514 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.514 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:05:41.514 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.962514+0000", 2024-09-21T10:05:41.515 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8142000000000001e-05 2024-09-21T10:05:41.515 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.515 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.515 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:05:41.515 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.962550+0000", 2024-09-21T10:05:41.515 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5398999999999998e-05 2024-09-21T10:05:41.515 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.515 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.515 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.515 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.515 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.515 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.45320.0:189067 8.e e132/119 8:7dac1850:::benchmark_data_smithi107_135256_object63183:head v 132'11848, mlcod=132'11842)", 2024-09-21T10:05:41.515 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T09:58:02.833045+0000", 2024-09-21T10:05:41.515 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 458.64591090800002, 2024-09-21T10:05:41.515 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.12897449499999999, 2024-09-21T10:05:41.516 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:41.516 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:41.516 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:41.516 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.516 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:41.516 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.833045+0000", 2024-09-21T10:05:41.516 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.516 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.516 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.516 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:41.516 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.833045+0000", 2024-09-21T10:05:41.516 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.516 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.516 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.517 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:41.517 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.833044+0000", 2024-09-21T10:05:41.517 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:05:41.517 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.517 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.517 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:41.517 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.833051+0000", 2024-09-21T10:05:41.517 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4459999999999998e-06 2024-09-21T10:05:41.517 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.517 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.517 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:41.517 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.949127+0000", 2024-09-21T10:05:41.517 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.11607619 2024-09-21T10:05:41.517 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.517 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.518 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:41.518 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.949136+0000", 2024-09-21T10:05:41.518 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.5279999999999992e-06 2024-09-21T10:05:41.518 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.518 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.518 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.518 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.949200+0000", 2024-09-21T10:05:41.518 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4010000000000005e-05 2024-09-21T10:05:41.518 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.518 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.518 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:41.518 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.949221+0000", 2024-09-21T10:05:41.518 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.0818e-05 2024-09-21T10:05:41.518 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.519 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.519 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:05:41.519 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.961889+0000", 2024-09-21T10:05:41.519 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.012667715 2024-09-21T10:05:41.519 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.519 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.519 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:05:41.519 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:58:02.962020+0000", 2024-09-21T10:05:41.519 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013062600000000001 2024-09-21T10:05:41.519 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.519 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.519 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.519 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.519 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.520 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:59377 9.e 9:78629d4b:::benchmark_data_smithi107_152010_object59376:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-21T10:05:41.520 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:28.166811+0000", 2024-09-21T10:05:41.520 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 253.31214462200001, 2024-09-21T10:05:41.520 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.20589206400000001, 2024-09-21T10:05:41.520 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:41.520 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:41.520 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:41.520 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:05:41.520 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:05:41.520 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 59377 2024-09-21T10:05:41.520 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.520 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:41.520 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.520 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:41.521 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.166811+0000", 2024-09-21T10:05:41.521 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.521 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.521 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.521 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:41.521 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.166811+0000", 2024-09-21T10:05:41.521 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.521 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.521 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.521 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:41.521 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.166810+0000", 2024-09-21T10:05:41.521 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:05:41.521 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.521 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.521 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:41.522 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.166861+0000", 2024-09-21T10:05:41.522 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0497000000000001e-05 2024-09-21T10:05:41.522 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.522 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.522 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:41.522 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.166873+0000", 2024-09-21T10:05:41.522 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2191000000000001e-05 2024-09-21T10:05:41.522 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.522 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.522 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:41.522 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.166879+0000", 2024-09-21T10:05:41.522 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0290000000000002e-06 2024-09-21T10:05:41.522 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.522 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.523 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.523 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.166916+0000", 2024-09-21T10:05:41.523 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6696000000000002e-05 2024-09-21T10:05:41.523 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.523 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.523 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:41.523 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.167010+0000", 2024-09-21T10:05:41.523 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.4344000000000003e-05 2024-09-21T10:05:41.523 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.523 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.523 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:05:41.523 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.167108+0000", 2024-09-21T10:05:41.523 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.7974999999999996e-05 2024-09-21T10:05:41.523 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.523 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.524 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:05:41.524 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.168751+0000", 2024-09-21T10:05:41.524 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0016428359999999999 2024-09-21T10:05:41.524 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.524 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.524 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.524 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.176900+0000", 2024-09-21T10:05:41.524 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0081491519999999998 2024-09-21T10:05:41.524 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.524 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.524 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.524 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.372549+0000", 2024-09-21T10:05:41.524 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.195649398 2024-09-21T10:05:41.524 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.525 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.525 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:05:41.525 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.372605+0000", 2024-09-21T10:05:41.525 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6098999999999999e-05 2024-09-21T10:05:41.525 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.525 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.525 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:05:41.525 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.372704+0000", 2024-09-21T10:05:41.525 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.8083000000000003e-05 2024-09-21T10:05:41.525 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.525 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.525 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.525 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.525 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.526 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:59390 9.e 9:742efc15:::benchmark_data_smithi107_152010_object59389:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-21T10:05:41.526 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:28.180915+0000", 2024-09-21T10:05:41.526 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 253.29804111600001, 2024-09-21T10:05:41.526 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.192036757, 2024-09-21T10:05:41.526 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:41.526 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:41.526 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:41.526 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:05:41.526 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:05:41.526 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 59390 2024-09-21T10:05:41.526 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.526 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:41.526 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.526 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:41.527 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.180915+0000", 2024-09-21T10:05:41.527 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.527 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.527 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.527 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:41.527 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.180915+0000", 2024-09-21T10:05:41.527 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.527 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.527 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.527 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:41.527 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.180910+0000", 2024-09-21T10:05:41.527 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999948 2024-09-21T10:05:41.527 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.527 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.528 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:41.528 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.180980+0000", 2024-09-21T10:05:41.528 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.0110000000000005e-05 2024-09-21T10:05:41.528 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.528 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.528 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:41.528 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.180991+0000", 2024-09-21T10:05:41.528 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0947999999999999e-05 2024-09-21T10:05:41.528 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.528 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.528 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:41.528 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.180996+0000", 2024-09-21T10:05:41.528 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1109999999999999e-06 2024-09-21T10:05:41.528 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.529 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.529 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.529 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.181029+0000", 2024-09-21T10:05:41.529 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3013999999999999e-05 2024-09-21T10:05:41.529 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.529 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.529 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:41.529 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.181120+0000", 2024-09-21T10:05:41.529 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.1198000000000003e-05 2024-09-21T10:05:41.529 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.529 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.529 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:05:41.529 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.181216+0000", 2024-09-21T10:05:41.529 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.6146999999999999e-05 2024-09-21T10:05:41.529 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.530 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.530 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:05:41.530 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.182799+0000", 2024-09-21T10:05:41.530 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0015824260000000001 2024-09-21T10:05:41.530 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.530 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.530 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.530 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.184037+0000", 2024-09-21T10:05:41.530 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0012377480000000001 2024-09-21T10:05:41.530 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.530 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.530 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.530 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.372820+0000", 2024-09-21T10:05:41.530 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.18878344599999999 2024-09-21T10:05:41.531 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.531 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.531 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:05:41.531 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.372863+0000", 2024-09-21T10:05:41.531 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3506e-05 2024-09-21T10:05:41.531 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.531 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.531 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:05:41.531 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.372952+0000", 2024-09-21T10:05:41.531 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.8238000000000005e-05 2024-09-21T10:05:41.531 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.531 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.531 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.531 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.532 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.532 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:74508 9.e 9:73ae9622:::benchmark_data_smithi107_152010_object74507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:05:41.532 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:41.860213+0000", 2024-09-21T10:05:41.532 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 239.61874341199999, 2024-09-21T10:05:41.532 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.19656214699999999, 2024-09-21T10:05:41.532 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:41.532 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:41.532 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:41.532 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:05:41.532 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:05:41.532 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 74508 2024-09-21T10:05:41.532 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.532 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:41.532 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.532 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:41.533 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.860213+0000", 2024-09-21T10:05:41.533 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.533 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.533 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.533 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:41.533 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.860213+0000", 2024-09-21T10:05:41.533 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.533 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.533 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.533 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:41.533 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.860211+0000", 2024-09-21T10:05:41.533 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:05:41.533 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.533 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.533 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:41.534 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.860266+0000", 2024-09-21T10:05:41.534 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4657000000000002e-05 2024-09-21T10:05:41.534 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.534 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.534 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:41.534 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.860277+0000", 2024-09-21T10:05:41.534 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1646e-05 2024-09-21T10:05:41.534 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.534 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.534 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:41.534 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.860283+0000", 2024-09-21T10:05:41.534 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2460000000000003e-06 2024-09-21T10:05:41.534 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.534 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.535 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.535 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.860334+0000", 2024-09-21T10:05:41.535 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1134999999999997e-05 2024-09-21T10:05:41.535 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.535 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.535 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:41.535 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.860474+0000", 2024-09-21T10:05:41.535 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014038599999999999 2024-09-21T10:05:41.535 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.535 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.535 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:05:41.535 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.860628+0000", 2024-09-21T10:05:41.535 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015359299999999999 2024-09-21T10:05:41.535 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.535 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.536 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:05:41.536 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.862650+0000", 2024-09-21T10:05:41.536 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0020217939999999999 2024-09-21T10:05:41.536 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.536 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.536 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.536 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.863490+0000", 2024-09-21T10:05:41.536 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00084041499999999996 2024-09-21T10:05:41.536 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.536 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.536 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.536 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:42.056663+0000", 2024-09-21T10:05:41.536 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.19317345899999999 2024-09-21T10:05:41.536 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.536 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.537 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:05:41.537 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:42.056709+0000", 2024-09-21T10:05:41.537 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5275000000000003e-05 2024-09-21T10:05:41.537 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.537 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.537 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:05:41.537 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:42.056775+0000", 2024-09-21T10:05:41.537 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6149999999999995e-05 2024-09-21T10:05:41.537 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.537 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.537 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.537 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.537 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.537 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:76398 9.e 9:7c2c8617:::benchmark_data_smithi107_152010_object76397:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:05:41.537 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:43.735330+0000", 2024-09-21T10:05:41.538 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 237.743625638, 2024-09-21T10:05:41.538 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.13469748200000001, 2024-09-21T10:05:41.538 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:41.538 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:41.538 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:41.538 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:05:41.538 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:05:41.538 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 76398 2024-09-21T10:05:41.538 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.538 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:41.538 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.538 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:41.538 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:43.735330+0000", 2024-09-21T10:05:41.538 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.539 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.539 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.539 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:41.539 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:43.735330+0000", 2024-09-21T10:05:41.539 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.539 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.539 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.540 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:41.540 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:43.735329+0000", 2024-09-21T10:05:41.540 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:05:41.540 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.540 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.540 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:41.540 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:43.735384+0000", 2024-09-21T10:05:41.540 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5078999999999999e-05 2024-09-21T10:05:41.540 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.540 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.541 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:41.541 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:43.735403+0000", 2024-09-21T10:05:41.541 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.9114999999999999e-05 2024-09-21T10:05:41.541 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.541 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.541 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:41.541 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:43.735417+0000", 2024-09-21T10:05:41.541 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4474000000000001e-05 2024-09-21T10:05:41.541 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.541 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.541 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.541 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:43.735474+0000", 2024-09-21T10:05:41.541 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6798000000000001e-05 2024-09-21T10:05:41.542 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.542 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.542 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:41.542 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:43.735607+0000", 2024-09-21T10:05:41.542 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000133289 2024-09-21T10:05:41.542 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.542 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.542 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:05:41.542 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:43.735747+0000", 2024-09-21T10:05:41.542 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000140059 2024-09-21T10:05:41.542 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.542 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.542 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:05:41.542 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:43.738199+0000", 2024-09-21T10:05:41.543 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0024517359999999999 2024-09-21T10:05:41.543 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.543 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.543 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.543 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:43.739433+0000", 2024-09-21T10:05:41.543 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0012340039999999999 2024-09-21T10:05:41.543 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.543 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.543 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.543 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:43.869874+0000", 2024-09-21T10:05:41.543 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.130440853 2024-09-21T10:05:41.543 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.543 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.543 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:05:41.543 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:43.869927+0000", 2024-09-21T10:05:41.544 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2877000000000002e-05 2024-09-21T10:05:41.544 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.544 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.544 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:05:41.544 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:43.870028+0000", 2024-09-21T10:05:41.544 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000100949 2024-09-21T10:05:41.544 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.544 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.544 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.544 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.544 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.544 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:101224 9.e 9:7283d620:::benchmark_data_smithi107_152010_object101223:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-21T10:05:41.544 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:09.474070+0000", 2024-09-21T10:05:41.545 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 212.00488649600001, 2024-09-21T10:05:41.545 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28291996200000002, 2024-09-21T10:05:41.545 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:41.545 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:41.545 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:41.545 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:05:41.545 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:05:41.545 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 101224 2024-09-21T10:05:41.545 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.545 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:41.545 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.545 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:41.545 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474070+0000", 2024-09-21T10:05:41.546 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.546 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.546 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.546 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:41.546 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474070+0000", 2024-09-21T10:05:41.546 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.546 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.546 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.546 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:41.546 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474068+0000", 2024-09-21T10:05:41.546 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:05:41.546 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.546 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.546 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:41.547 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474170+0000", 2024-09-21T10:05:41.547 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000102308 2024-09-21T10:05:41.547 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.547 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.547 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:41.547 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474185+0000", 2024-09-21T10:05:41.547 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.448e-05 2024-09-21T10:05:41.547 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.547 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.547 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:41.547 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474190+0000", 2024-09-21T10:05:41.547 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5969999999999999e-06 2024-09-21T10:05:41.547 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.547 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474235+0000", 2024-09-21T10:05:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4521000000000003e-05 2024-09-21T10:05:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474327+0000", 2024-09-21T10:05:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.2227999999999999e-05 2024-09-21T10:05:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:05:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474424+0000", 2024-09-21T10:05:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.6824999999999995e-05 2024-09-21T10:05:41.548 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:05:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.477173+0000", 2024-09-21T10:05:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0027488759999999999 2024-09-21T10:05:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.481128+0000", 2024-09-21T10:05:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003955592 2024-09-21T10:05:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.756811+0000", 2024-09-21T10:05:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.27568240500000002 2024-09-21T10:05:41.549 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:05:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.756879+0000", 2024-09-21T10:05:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8109999999999997e-05 2024-09-21T10:05:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:05:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.756990+0000", 2024-09-21T10:05:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000110536 2024-09-21T10:05:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.550 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:101232 9.e 9:7d6e3112:::benchmark_data_smithi107_152010_object101231:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-21T10:05:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:09.490031+0000", 2024-09-21T10:05:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 211.98892537200001, 2024-09-21T10:05:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.27560621600000001, 2024-09-21T10:05:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:05:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:05:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 101232 2024-09-21T10:05:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:41.551 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.490031+0000", 2024-09-21T10:05:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.490031+0000", 2024-09-21T10:05:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.490029+0000", 2024-09-21T10:05:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:05:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.552 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.490091+0000", 2024-09-21T10:05:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1692999999999994e-05 2024-09-21T10:05:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.490105+0000", 2024-09-21T10:05:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3743999999999999e-05 2024-09-21T10:05:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.490110+0000", 2024-09-21T10:05:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1510000000000003e-06 2024-09-21T10:05:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.553 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.490155+0000", 2024-09-21T10:05:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5104e-05 2024-09-21T10:05:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.490253+0000", 2024-09-21T10:05:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.8528000000000003e-05 2024-09-21T10:05:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:05:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.490342+0000", 2024-09-21T10:05:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.8349000000000004e-05 2024-09-21T10:05:41.554 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:05:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.492828+0000", 2024-09-21T10:05:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002486055 2024-09-21T10:05:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.493332+0000", 2024-09-21T10:05:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00050429700000000001 2024-09-21T10:05:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.765486+0000", 2024-09-21T10:05:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.27215410099999998 2024-09-21T10:05:41.555 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:05:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.765551+0000", 2024-09-21T10:05:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4460000000000003e-05 2024-09-21T10:05:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:05:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.765637+0000", 2024-09-21T10:05:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6267999999999995e-05 2024-09-21T10:05:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.556 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:32042 10.1 10:8f222128:::benchmark_data_smithi107_184135_object32041:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-21T10:05:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:20.694851+0000", 2024-09-21T10:05:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 20.784104817999999, 2024-09-21T10:05:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.18394027099999999, 2024-09-21T10:05:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:05:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:05:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32042 2024-09-21T10:05:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:41.557 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.694851+0000", 2024-09-21T10:05:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.694851+0000", 2024-09-21T10:05:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.694850+0000", 2024-09-21T10:05:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:05:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.558 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.694939+0000", 2024-09-21T10:05:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.9457999999999996e-05 2024-09-21T10:05:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.694962+0000", 2024-09-21T10:05:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.2887e-05 2024-09-21T10:05:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.694968+0000", 2024-09-21T10:05:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1469999999999998e-06 2024-09-21T10:05:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.559 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.695030+0000", 2024-09-21T10:05:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2149000000000004e-05 2024-09-21T10:05:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.695139+0000", 2024-09-21T10:05:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010844399999999999 2024-09-21T10:05:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:05:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.695277+0000", 2024-09-21T10:05:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013883100000000001 2024-09-21T10:05:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.560 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:05:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.699209+0000", 2024-09-21T10:05:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0039318720000000003 2024-09-21T10:05:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.699268+0000", 2024-09-21T10:05:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8909999999999997e-05 2024-09-21T10:05:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.878713+0000", 2024-09-21T10:05:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.179445152 2024-09-21T10:05:41.561 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:05:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.878760+0000", 2024-09-21T10:05:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6140000000000002e-05 2024-09-21T10:05:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:05:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.878792+0000", 2024-09-21T10:05:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1965000000000003e-05 2024-09-21T10:05:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.562 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:32045 10.1 10:86a6431e:::benchmark_data_smithi107_184135_object32044:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-21T10:05:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:20.695623+0000", 2024-09-21T10:05:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 20.783332735999998, 2024-09-21T10:05:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.183322399, 2024-09-21T10:05:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:05:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:05:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32045 2024-09-21T10:05:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:41.563 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.695623+0000", 2024-09-21T10:05:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.695623+0000", 2024-09-21T10:05:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.695622+0000", 2024-09-21T10:05:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:05:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.564 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.695723+0000", 2024-09-21T10:05:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000100306 2024-09-21T10:05:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.695736+0000", 2024-09-21T10:05:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3681e-05 2024-09-21T10:05:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.695742+0000", 2024-09-21T10:05:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9000000000000003e-06 2024-09-21T10:05:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.565 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.696547+0000", 2024-09-21T10:05:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00080520600000000004 2024-09-21T10:05:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.696649+0000", 2024-09-21T10:05:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000101903 2024-09-21T10:05:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:05:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.696774+0000", 2024-09-21T10:05:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012425899999999999 2024-09-21T10:05:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.566 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:05:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.699222+0000", 2024-09-21T10:05:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0024478730000000001 2024-09-21T10:05:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.700488+0000", 2024-09-21T10:05:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0012663069999999999 2024-09-21T10:05:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.878865+0000", 2024-09-21T10:05:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.178377023 2024-09-21T10:05:41.567 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:05:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.878911+0000", 2024-09-21T10:05:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6149000000000001e-05 2024-09-21T10:05:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:05:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.878946+0000", 2024-09-21T10:05:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4724999999999997e-05 2024-09-21T10:05:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.568 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:32049 10.1 10:842495b3:::benchmark_data_smithi107_184135_object32048:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-21T10:05:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:20.697016+0000", 2024-09-21T10:05:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 20.781940275, 2024-09-21T10:05:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.186558366, 2024-09-21T10:05:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:05:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:05:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32049 2024-09-21T10:05:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.569 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.697016+0000", 2024-09-21T10:05:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.697016+0000", 2024-09-21T10:05:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.697015+0000", 2024-09-21T10:05:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:05:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.570 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.697209+0000", 2024-09-21T10:05:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000194336 2024-09-21T10:05:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.697223+0000", 2024-09-21T10:05:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3603e-05 2024-09-21T10:05:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.697229+0000", 2024-09-21T10:05:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.618e-06 2024-09-21T10:05:41.571 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.697859+0000", 2024-09-21T10:05:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00062961199999999999 2024-09-21T10:05:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.697957+0000", 2024-09-21T10:05:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.7594000000000001e-05 2024-09-21T10:05:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:05:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.698074+0000", 2024-09-21T10:05:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000117632 2024-09-21T10:05:41.572 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:05:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.699441+0000", 2024-09-21T10:05:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0013663359999999999 2024-09-21T10:05:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.701303+0000", 2024-09-21T10:05:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001862753 2024-09-21T10:05:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.883355+0000", 2024-09-21T10:05:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.18205179699999999 2024-09-21T10:05:41.573 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:05:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.883456+0000", 2024-09-21T10:05:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000101346 2024-09-21T10:05:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:05:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.883574+0000", 2024-09-21T10:05:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000117674 2024-09-21T10:05:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.574 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:32065 10.1 10:80121e10:::benchmark_data_smithi107_184135_object32064:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-21T10:05:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:20.704622+0000", 2024-09-21T10:05:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 20.774334461999999, 2024-09-21T10:05:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.179118943, 2024-09-21T10:05:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:05:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:05:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32065 2024-09-21T10:05:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.575 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.704622+0000", 2024-09-21T10:05:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.704622+0000", 2024-09-21T10:05:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.704621+0000", 2024-09-21T10:05:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:05:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.576 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.704675+0000", 2024-09-21T10:05:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4379999999999998e-05 2024-09-21T10:05:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.704685+0000", 2024-09-21T10:05:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.73e-06 2024-09-21T10:05:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.704689+0000", 2024-09-21T10:05:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8480000000000003e-06 2024-09-21T10:05:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.577 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.704728+0000", 2024-09-21T10:05:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.9357000000000003e-05 2024-09-21T10:05:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.704808+0000", 2024-09-21T10:05:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.0489999999999997e-05 2024-09-21T10:05:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:05:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.704906+0000", 2024-09-21T10:05:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.7820999999999997e-05 2024-09-21T10:05:41.578 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:05:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.706023+0000", 2024-09-21T10:05:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0011165770000000001 2024-09-21T10:05:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.707044+0000", 2024-09-21T10:05:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0010210320000000001 2024-09-21T10:05:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.579 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.883646+0000", 2024-09-21T10:05:41.580 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:41.580 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.176601919 2024-09-21T10:05:41.580 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.580 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.580 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:05:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.883694+0000", 2024-09-21T10:05:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8513999999999997e-05 2024-09-21T10:05:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:05:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.883741+0000", 2024-09-21T10:05:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6211000000000002e-05 2024-09-21T10:05:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:32083 10.1 10:86c9040f:::benchmark_data_smithi107_184135_object32082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-21T10:05:41.581 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:20.719568+0000", 2024-09-21T10:05:41.582 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 20.759387602, 2024-09-21T10:05:41.582 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.16431898, 2024-09-21T10:05:41.582 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:41.582 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:41.582 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:41.582 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:05:41.582 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:05:41.582 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32083 2024-09-21T10:05:41.582 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.582 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:41.582 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.582 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:41.582 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.719568+0000", 2024-09-21T10:05:41.582 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.719568+0000", 2024-09-21T10:05:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.719567+0000", 2024-09-21T10:05:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:05:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:41.583 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.719640+0000", 2024-09-21T10:05:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2805999999999994e-05 2024-09-21T10:05:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.719651+0000", 2024-09-21T10:05:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0812000000000001e-05 2024-09-21T10:05:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.719656+0000", 2024-09-21T10:05:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4670000000000003e-06 2024-09-21T10:05:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.584 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.719703+0000", 2024-09-21T10:05:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6672000000000003e-05 2024-09-21T10:05:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.719803+0000", 2024-09-21T10:05:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.9840000000000006e-05 2024-09-21T10:05:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:05:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.719887+0000", 2024-09-21T10:05:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.3755e-05 2024-09-21T10:05:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.585 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.586 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:05:41.586 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.720942+0000", 2024-09-21T10:05:41.586 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001054992 2024-09-21T10:05:41.586 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.586 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.586 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.586 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.721132+0000", 2024-09-21T10:05:41.586 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00018996400000000001 2024-09-21T10:05:41.586 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.586 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.586 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.586 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.883784+0000", 2024-09-21T10:05:41.586 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.162652297 2024-09-21T10:05:41.586 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:05:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.883823+0000", 2024-09-21T10:05:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8964000000000002e-05 2024-09-21T10:05:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:05:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.883887+0000", 2024-09-21T10:05:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4515000000000001e-05 2024-09-21T10:05:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.587 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:65710 10.e 10:757f5ac1:::benchmark_data_smithi107_184135_object65709:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-21T10:05:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:41.410687+0000", 2024-09-21T10:05:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.068269464000000002, 2024-09-21T10:05:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.026511092999999999, 2024-09-21T10:05:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:05:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:05:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 65710 2024-09-21T10:05:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:41.588 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.410687+0000", 2024-09-21T10:05:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.410687+0000", 2024-09-21T10:05:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.410685+0000", 2024-09-21T10:05:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:05:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:41.589 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.410720+0000", 2024-09-21T10:05:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5298e-05 2024-09-21T10:05:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.410730+0000", 2024-09-21T10:05:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.1959999999999997e-06 2024-09-21T10:05:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.410734+0000", 2024-09-21T10:05:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5859999999999998e-06 2024-09-21T10:05:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.590 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.414058+0000", 2024-09-21T10:05:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0033233450000000001 2024-09-21T10:05:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.414246+0000", 2024-09-21T10:05:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00018843899999999999 2024-09-21T10:05:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:05:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.414375+0000", 2024-09-21T10:05:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000128842 2024-09-21T10:05:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.591 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:05:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.419501+0000", 2024-09-21T10:05:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0051265349999999998 2024-09-21T10:05:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.419651+0000", 2024-09-21T10:05:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000149192 2024-09-21T10:05:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.437110+0000", 2024-09-21T10:05:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.017459817999999998 2024-09-21T10:05:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.592 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:05:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.437157+0000", 2024-09-21T10:05:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6848999999999998e-05 2024-09-21T10:05:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:05:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.437198+0000", 2024-09-21T10:05:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0438999999999997e-05 2024-09-21T10:05:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.593 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:65713 10.e 10:7e15c76a:::benchmark_data_smithi107_184135_object65712:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-21T10:05:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:41.410746+0000", 2024-09-21T10:05:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.068209838999999994, 2024-09-21T10:05:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.026553294000000002, 2024-09-21T10:05:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:05:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:05:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 65713 2024-09-21T10:05:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:41.594 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.410746+0000", 2024-09-21T10:05:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.410746+0000", 2024-09-21T10:05:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.410746+0000", 2024-09-21T10:05:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:05:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.595 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.410773+0000", 2024-09-21T10:05:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7495e-05 2024-09-21T10:05:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.410780+0000", 2024-09-21T10:05:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2929999999999997e-06 2024-09-21T10:05:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.410783+0000", 2024-09-21T10:05:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8580000000000002e-06 2024-09-21T10:05:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.596 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.416835+0000", 2024-09-21T10:05:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0060511200000000001 2024-09-21T10:05:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.416981+0000", 2024-09-21T10:05:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014616999999999999 2024-09-21T10:05:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:05:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.417126+0000", 2024-09-21T10:05:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000145663 2024-09-21T10:05:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.597 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.425061+0000", 2024-09-21T10:05:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0079347789999999994 2024-09-21T10:05:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:05:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.428696+0000", 2024-09-21T10:05:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0036348280000000001 2024-09-21T10:05:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.437231+0000", 2024-09-21T10:05:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0085354360000000004 2024-09-21T10:05:41.598 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:05:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.437269+0000", 2024-09-21T10:05:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8025999999999998e-05 2024-09-21T10:05:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:05:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.437300+0000", 2024-09-21T10:05:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0057e-05 2024-09-21T10:05:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.599 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:65759 10.e 10:78e8ff6d:::benchmark_data_smithi107_184135_object65758:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-21T10:05:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:41.447585+0000", 2024-09-21T10:05:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.031370784999999998, 2024-09-21T10:05:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.023969788999999998, 2024-09-21T10:05:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:05:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:05:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:05:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 65759 2024-09-21T10:05:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.600 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.447585+0000", 2024-09-21T10:05:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.447585+0000", 2024-09-21T10:05:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.447583+0000", 2024-09-21T10:05:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:05:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.601 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.447712+0000", 2024-09-21T10:05:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012832699999999999 2024-09-21T10:05:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.447722+0000", 2024-09-21T10:05:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0536999999999999e-05 2024-09-21T10:05:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.447727+0000", 2024-09-21T10:05:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9540000000000003e-06 2024-09-21T10:05:41.602 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.455656+0000", 2024-09-21T10:05:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0079291099999999996 2024-09-21T10:05:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.456955+0000", 2024-09-21T10:05:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0012990930000000001 2024-09-21T10:05:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:05:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.457388+0000", 2024-09-21T10:05:41.603 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00043272299999999998 2024-09-21T10:05:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:05:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.461977+0000", 2024-09-21T10:05:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0045890089999999998 2024-09-21T10:05:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.463434+0000", 2024-09-21T10:05:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001456611 2024-09-21T10:05:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:41.604 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.471469+0000", 2024-09-21T10:05:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0080359130000000004 2024-09-21T10:05:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:05:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.471520+0000", 2024-09-21T10:05:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0852e-05 2024-09-21T10:05:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:05:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:41.471555+0000", 2024-09-21T10:05:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4740000000000003e-05 2024-09-21T10:05:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.605 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:41.606 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:41.606 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:05:41.606 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:41.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:41.642 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:41.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:41.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:41.654 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 32 16 66010 65994 128.881 84.875 0.0153106 0.00775386 2024-09-21T10:05:41.744 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:41.790 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:41.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:41.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:41.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:41.959 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:41.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:41.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:42.049 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:05:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:05:41] "GET /metrics HTTP/1.1" 200 36695 "" "Prometheus/2.51.0" 2024-09-21T10:05:42.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:42.099 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:42.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:42.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:42.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:42.235 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:42.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:42.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:42.337 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:42.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:42 smithi107 ceph-mon[102221]: pgmap v703: 109 pgs: 109 active+clean; 3.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:05:42.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:42 smithi107 ceph-mon[105022]: pgmap v703: 109 pgs: 109 active+clean; 3.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:05:42.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:42.375 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:42.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:42.403 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:42 smithi129 ceph-mon[88471]: pgmap v703: 109 pgs: 109 active+clean; 3.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:05:42.448 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:05:42.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:42.516 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:42.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:42.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:42.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:42.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:42.649 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:42.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:42.654 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 33 16 67336 67320 127.487 82.875 0.0068682 0.00783854 2024-09-21T10:05:42.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:42.785 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:42.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:42.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:42.791 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:42.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:42.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:42.923 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:42.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:42.934 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:05:43.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:43.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:43.058 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:43.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:43.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:43.199 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:43.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:43.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:43.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:43.335 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:43.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:43.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:43.437 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:43.477 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:43.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:43.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:43.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:43.625 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:43.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:43.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:43.654 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 34 16 68659 68643 126.169 82.6875 0.00883876 0.00791977 2024-09-21T10:05:43.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:43.761 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:43.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:43.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:43.863 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:43.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:05:43.896 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:43.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:43.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:43.998 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:44.030 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:44.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:44.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:44.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:44.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:44.185 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:44.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:44.286 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:44.299 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T10:05:44.312 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:44.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:44.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:44.414 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:44.451 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:44.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:44.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:44.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:44.588 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:44.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:44.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:44 smithi107 ceph-mon[102221]: pgmap v704: 109 pgs: 109 active+clean; 4.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-21T10:05:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:44 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1516462707' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:05:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:44 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:05:44.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:44 smithi107 ceph-mon[105022]: pgmap v704: 109 pgs: 109 active+clean; 4.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-21T10:05:44.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:44 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1516462707' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:05:44.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:44 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:05:44.618 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:44 smithi129 ceph-mon[88471]: pgmap v704: 109 pgs: 109 active+clean; 4.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-21T10:05:44.618 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:44 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1516462707' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:05:44.618 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:44 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:05:44.655 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 35 16 69912 69896 124.801 78.3125 0.00744459 0.00800792 2024-09-21T10:05:44.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:44.726 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:44.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:44.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:44.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:44.872 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:44.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:44.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:44.974 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:45.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:45.010 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:45.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:45.086 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:05:45.086 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:05:45.087 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.087 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:70494 10.9 10:987b4cad:::benchmark_data_smithi107_184135_object70493:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-21T10:05:45.087 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:45.067941+0000", 2024-09-21T10:05:45.087 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.014371567999999999, 2024-09-21T10:05:45.087 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.014406131000000001, 2024-09-21T10:05:45.087 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:45.087 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:05:45.087 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:45.087 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:05:45.087 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:05:45.087 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 70494 2024-09-21T10:05:45.087 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.087 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:45.088 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.088 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:45.088 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.067941+0000", 2024-09-21T10:05:45.088 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:45.088 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.088 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.088 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:45.088 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.067941+0000", 2024-09-21T10:05:45.088 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:45.088 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.088 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.088 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:45.088 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.067940+0000", 2024-09-21T10:05:45.088 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:05:45.088 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.089 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.089 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:45.089 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.068020+0000", 2024-09-21T10:05:45.089 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.0412000000000002e-05 2024-09-21T10:05:45.089 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.089 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.089 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:45.089 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.068035+0000", 2024-09-21T10:05:45.089 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4917e-05 2024-09-21T10:05:45.089 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.089 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.090 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:45.090 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.068044+0000", 2024-09-21T10:05:45.090 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.3469999999999999e-06 2024-09-21T10:05:45.090 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.090 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.090 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:45.090 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.068097+0000", 2024-09-21T10:05:45.090 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3618000000000001e-05 2024-09-21T10:05:45.090 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.090 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.090 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:45.090 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.068212+0000", 2024-09-21T10:05:45.091 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000115232 2024-09-21T10:05:45.091 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.091 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.091 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:05:45.091 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.068327+0000", 2024-09-21T10:05:45.091 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000114199 2024-09-21T10:05:45.091 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.091 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.091 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:05:45.091 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.071205+0000", 2024-09-21T10:05:45.091 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0028786189999999998 2024-09-21T10:05:45.092 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.092 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.092 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:05:45.092 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.072094+0000", 2024-09-21T10:05:45.092 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00088853100000000002 2024-09-21T10:05:45.092 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:45.092 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:45.092 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:45.092 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.092 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.092 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:70499 10.9 10:99460443:::benchmark_data_smithi107_184135_object70498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-21T10:05:45.093 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:45.073170+0000", 2024-09-21T10:05:45.093 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.0091424829999999999, 2024-09-21T10:05:45.093 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.009276065, 2024-09-21T10:05:45.093 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:45.093 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:05:45.093 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:45.093 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:05:45.093 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:05:45.093 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 70499 2024-09-21T10:05:45.093 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.093 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:45.093 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.093 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:45.093 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.073170+0000", 2024-09-21T10:05:45.094 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:45.094 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.094 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.094 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:45.094 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.073170+0000", 2024-09-21T10:05:45.094 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:45.094 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.094 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.094 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:45.095 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.073168+0000", 2024-09-21T10:05:45.095 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:05:45.095 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.095 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.095 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:45.095 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.073226+0000", 2024-09-21T10:05:45.095 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7210000000000003e-05 2024-09-21T10:05:45.095 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.095 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.095 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:45.095 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.073241+0000", 2024-09-21T10:05:45.095 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4948e-05 2024-09-21T10:05:45.096 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.096 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.096 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:45.096 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.073254+0000", 2024-09-21T10:05:45.096 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3903999999999999e-05 2024-09-21T10:05:45.096 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.096 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.096 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:45.096 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.073356+0000", 2024-09-21T10:05:45.097 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000101952 2024-09-21T10:05:45.097 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.097 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.097 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:45.097 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.073507+0000", 2024-09-21T10:05:45.097 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000150209 2024-09-21T10:05:45.097 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.097 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.097 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:05:45.097 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.073647+0000", 2024-09-21T10:05:45.097 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014046600000000001 2024-09-21T10:05:45.097 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:45.097 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:45.097 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:45.098 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.098 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.098 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:70501 10.9 10:9d9c1413:::benchmark_data_smithi107_184135_object70500:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-21T10:05:45.098 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:45.073267+0000", 2024-09-21T10:05:45.098 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.0090462830000000004, 2024-09-21T10:05:45.098 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0092496760000000001, 2024-09-21T10:05:45.098 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:45.098 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:05:45.098 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:05:45.099 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:05:45.099 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:05:45.099 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 70501 2024-09-21T10:05:45.099 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.099 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:45.099 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.099 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:45.099 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.073267+0000", 2024-09-21T10:05:45.099 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:45.099 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.099 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.099 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:45.099 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.073267+0000", 2024-09-21T10:05:45.100 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:45.100 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.100 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.100 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:45.100 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.073266+0000", 2024-09-21T10:05:45.100 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:05:45.100 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.100 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.100 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:45.100 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.073310+0000", 2024-09-21T10:05:45.101 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3974999999999998e-05 2024-09-21T10:05:45.101 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.101 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.101 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:45.101 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.073331+0000", 2024-09-21T10:05:45.101 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.0863999999999999e-05 2024-09-21T10:05:45.101 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.101 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.101 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:45.101 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.073337+0000", 2024-09-21T10:05:45.102 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1240000000000002e-06 2024-09-21T10:05:45.102 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.102 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.102 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:05:45.102 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.077247+0000", 2024-09-21T10:05:45.102 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003910017 2024-09-21T10:05:45.102 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.102 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.102 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:05:45.103 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.077381+0000", 2024-09-21T10:05:45.103 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000134215 2024-09-21T10:05:45.103 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.103 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.103 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:05:45.103 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.077526+0000", 2024-09-21T10:05:45.103 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014453 2024-09-21T10:05:45.103 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:45.103 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:45.103 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:45.103 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.103 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.103 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop_reply(client.36210.0:70499 10.9 e199/190)", 2024-09-21T10:05:45.104 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:45.077168+0000", 2024-09-21T10:05:45.104 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.005145251, 2024-09-21T10:05:45.104 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0054113989999999999, 2024-09-21T10:05:45.104 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:45.104 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "queued for pg", 2024-09-21T10:05:45.104 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:45.104 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.104 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:45.104 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.077168+0000", 2024-09-21T10:05:45.104 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:45.104 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.104 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.104 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:45.105 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.077168+0000", 2024-09-21T10:05:45.105 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:45.105 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.105 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.105 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:45.105 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.077167+0000", 2024-09-21T10:05:45.105 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:05:45.105 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.105 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.105 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:45.105 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.077174+0000", 2024-09-21T10:05:45.105 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.6320000000000008e-06 2024-09-21T10:05:45.105 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.105 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.106 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:45.106 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.077191+0000", 2024-09-21T10:05:45.106 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.6113e-05 2024-09-21T10:05:45.106 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.106 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.106 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:45.106 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.077197+0000", 2024-09-21T10:05:45.106 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4860000000000002e-06 2024-09-21T10:05:45.106 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:45.106 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:45.106 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:45.106 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.106 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.106 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop_reply(client.36210.0:70501 10.9 e199/190)", 2024-09-21T10:05:45.107 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:45.079240+0000", 2024-09-21T10:05:45.107 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.0030726680000000002, 2024-09-21T10:05:45.107 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003378434, 2024-09-21T10:05:45.107 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:45.107 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "queued for pg", 2024-09-21T10:05:45.107 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:45.107 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.107 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:45.107 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.079240+0000", 2024-09-21T10:05:45.107 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:45.107 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.107 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.107 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:45.107 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.079240+0000", 2024-09-21T10:05:45.108 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:45.108 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.108 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.108 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:45.108 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.079240+0000", 2024-09-21T10:05:45.108 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:05:45.108 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.108 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.108 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:45.108 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.079244+0000", 2024-09-21T10:05:45.108 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2629999999999997e-06 2024-09-21T10:05:45.108 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.108 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.108 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:45.109 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.079255+0000", 2024-09-21T10:05:45.109 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1138000000000001e-05 2024-09-21T10:05:45.109 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.109 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.109 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:45.109 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.079260+0000", 2024-09-21T10:05:45.109 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5689999999999998e-06 2024-09-21T10:05:45.109 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:45.109 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:45.109 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:45.109 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.109 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.109 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop_reply(client.36210.0:70494 10.9 e199/190)", 2024-09-21T10:05:45.110 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:45.079400+0000", 2024-09-21T10:05:45.110 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.0029129490000000002, 2024-09-21T10:05:45.110 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0032596819999999999, 2024-09-21T10:05:45.110 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:45.110 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "queued for pg", 2024-09-21T10:05:45.110 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:45.110 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.110 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:45.110 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.079400+0000", 2024-09-21T10:05:45.110 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:45.110 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.110 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.110 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:45.110 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.079400+0000", 2024-09-21T10:05:45.110 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:45.111 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.111 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.111 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:45.111 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.079399+0000", 2024-09-21T10:05:45.111 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:05:45.111 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.111 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.111 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:45.111 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.079403+0000", 2024-09-21T10:05:45.111 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.642e-06 2024-09-21T10:05:45.111 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.111 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.111 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:45.111 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.079425+0000", 2024-09-21T10:05:45.112 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.2401000000000001e-05 2024-09-21T10:05:45.112 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.112 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.112 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:45.112 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.079431+0000", 2024-09-21T10:05:45.112 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7860000000000002e-06 2024-09-21T10:05:45.112 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:45.112 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:45.112 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:45.112 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.112 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.112 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop_reply(client.36210.0:70499 10.9 e199/190)", 2024-09-21T10:05:45.112 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:45.081619+0000", 2024-09-21T10:05:45.112 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.00069377900000000005, 2024-09-21T10:05:45.112 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001079726, 2024-09-21T10:05:45.113 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:45.113 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "queued for pg", 2024-09-21T10:05:45.113 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:45.113 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.113 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:45.113 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.081619+0000", 2024-09-21T10:05:45.113 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:45.113 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.113 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.113 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:45.113 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.081619+0000", 2024-09-21T10:05:45.113 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:45.113 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.113 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.114 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:45.114 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.081618+0000", 2024-09-21T10:05:45.114 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:05:45.114 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.114 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.114 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:45.114 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.081624+0000", 2024-09-21T10:05:45.114 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2449999999999998e-06 2024-09-21T10:05:45.114 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.114 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.114 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:45.114 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.081636+0000", 2024-09-21T10:05:45.114 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2775000000000001e-05 2024-09-21T10:05:45.114 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.115 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.115 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:45.115 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.081642+0000", 2024-09-21T10:05:45.115 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6749999999999999e-06 2024-09-21T10:05:45.115 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:45.115 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:45.115 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:45.115 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.115 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.115 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop_reply(client.36210.0:70501 10.9 e199/190)", 2024-09-21T10:05:45.115 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:45.082034+0000", 2024-09-21T10:05:45.115 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.00027889800000000003, 2024-09-21T10:05:45.115 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00070377999999999999, 2024-09-21T10:05:45.115 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:05:45.115 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "queued for pg", 2024-09-21T10:05:45.116 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:05:45.116 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.116 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:05:45.116 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.082034+0000", 2024-09-21T10:05:45.116 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:45.116 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.116 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.116 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:05:45.116 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.082034+0000", 2024-09-21T10:05:45.116 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:05:45.116 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.116 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.116 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:05:45.116 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.082033+0000", 2024-09-21T10:05:45.116 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:05:45.117 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.117 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.117 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:05:45.117 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.082037+0000", 2024-09-21T10:05:45.117 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.9940000000000002e-06 2024-09-21T10:05:45.117 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.117 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.117 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:05:45.117 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.082047+0000", 2024-09-21T10:05:45.117 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.1120000000000002e-06 2024-09-21T10:05:45.117 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:05:45.117 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:05:45.117 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:05:45.117 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:45.082051+0000", 2024-09-21T10:05:45.118 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8450000000000002e-06 2024-09-21T10:05:45.118 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:45.118 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:05:45.118 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:45.118 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:05:45.118 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-21T10:05:45.118 INFO:teuthology.orchestra.run.smithi129.stdout: "num_ops": 8 2024-09-21T10:05:45.118 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:05:45.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:45.152 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:45.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:45.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:45.254 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:45.287 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:45.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:45.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:45.389 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:45.426 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:45.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:45.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:45.477 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:45 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:05:45.477 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:45 smithi107 ceph-mon[102221]: osdmap e199: 8 total, 8 up, 8 in 2024-09-21T10:05:45.477 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:45 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:05:45.477 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:45 smithi107 ceph-mon[105022]: osdmap e199: 8 total, 8 up, 8 in 2024-09-21T10:05:45.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:45.567 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:45.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:45.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:45.583 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:45 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:05:45.583 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:45 smithi129 ceph-mon[88471]: osdmap e199: 8 total, 8 up, 8 in 2024-09-21T10:05:45.655 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 36 16 71333 71317 123.802 88.8125 0.0140305 0.00807253 2024-09-21T10:05:45.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:45.699 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:45.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:45.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:45.801 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:45.839 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:45.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:45.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:45.939 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T10:05:45.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:05:45.941 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:45.978 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:45.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:45.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:46.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:46.098 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:05:46.123 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:46.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:46.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:46.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:46.241 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:46.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:46.287 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:46.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:46.388 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:46.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:46.435 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:46.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:46.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:46 smithi107 ceph-mon[102221]: pgmap v706: 109 pgs: 109 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-09-21T10:05:46.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:46 smithi107 ceph-mon[105022]: pgmap v706: 109 pgs: 109 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-09-21T10:05:46.491 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:05:46.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:46.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:46.601 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:46.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:46.635 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:46 smithi129 ceph-mon[88471]: pgmap v706: 109 pgs: 109 active+clean; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-09-21T10:05:46.655 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 37 16 72608 72592 122.609 79.6875 0.0137823 0.00815043 2024-09-21T10:05:46.702 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:46.738 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:46.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:46.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:46.840 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:46.874 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:46.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:46.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:46.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:47.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:47.010 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:47.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:47.081 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:47.111 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:47.147 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:47.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:47.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:47.248 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:47.282 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:47.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:47.332 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:05:47.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:47.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:47.424 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:47.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:47.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:47.558 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:47.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:47.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:47.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:47 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/883055286' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:05:47.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:47 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/883055286' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:05:47.655 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 38 16 74040 74024 121.738 89.5 0.0127951 0.00820955 2024-09-21T10:05:47.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:47 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/883055286' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:05:47.702 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:47.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:47.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:47.804 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:47.839 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:47.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:47.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:47.941 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:47.969 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:47.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:47.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:48.071 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:48.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:48.108 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:48.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:48.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:48.241 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:48.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:48.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:48.343 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:48.378 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:48.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:48.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:48 smithi129 ceph-mon[88471]: pgmap v707: 109 pgs: 109 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-21T10:05:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:48 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:05:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:48 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:05:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:48 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/883055286' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:05:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:48 smithi129 ceph-mon[88471]: osdmap e200: 8 total, 8 up, 8 in 2024-09-21T10:05:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:48 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:48.480 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:48.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:48.517 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:48.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:48 smithi107 ceph-mon[102221]: pgmap v707: 109 pgs: 109 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-21T10:05:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:48 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:05:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:48 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:05:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:48 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/883055286' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:05:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:48 smithi107 ceph-mon[102221]: osdmap e200: 8 total, 8 up, 8 in 2024-09-21T10:05:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:48 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:48.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:48 smithi107 ceph-mon[105022]: pgmap v707: 109 pgs: 109 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-21T10:05:48.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:48 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:05:48.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:48 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:05:48.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:48 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/883055286' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:05:48.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:48 smithi107 ceph-mon[105022]: osdmap e200: 8 total, 8 up, 8 in 2024-09-21T10:05:48.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:48 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:48.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:48.655 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 39 16 75336 75320 120.692 81 0.0143173 0.00828048 2024-09-21T10:05:48.656 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:48.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:48.758 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:48.801 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:48.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:48.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:48.875 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:05:48.875 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:05:48.875 INFO:teuthology.orchestra.run.smithi129.stdout: "complaint_time": 30, 2024-09-21T10:05:48.875 INFO:teuthology.orchestra.run.smithi129.stdout: "num_blocked_ops": 0 2024-09-21T10:05:48.875 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:05:48.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:48.940 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:48.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:49.041 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:49.077 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:49.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:49.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:49.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:49.199 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:49.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:49.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:49.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:49.326 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:49.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:49.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:49.428 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:49.466 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:49.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:49.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:49.523 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:05:49.568 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:49.594 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:49.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:49.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:49.655 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:05:49.654323+0000 min lat: 0.000982552 max lat: 0.416605 avg lat: 0.0083774 2024-09-21T10:05:49.655 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:05:49.655 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 40 16 76382 76366 119.309 65.375 0.0184946 0.0083774 2024-09-21T10:05:49.696 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:49.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:49.747 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:49.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:49.848 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:49.874 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:49.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:49.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:49.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:50.010 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:50.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:50.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:50.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:50.146 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:50.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:50.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:50.248 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:50.271 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:50.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:50.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:50.406 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:50.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:50.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:50.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:50 smithi129 ceph-mon[88471]: pgmap v709: 109 pgs: 109 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:05:50.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:05:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:05:50.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:50.508 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:50.551 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:50.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:50.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:50.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:50 smithi107 ceph-mon[102221]: pgmap v709: 109 pgs: 109 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:05:50.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:50 smithi107 ceph-mon[105022]: pgmap v709: 109 pgs: 109 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:05:50.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:50.655 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 41 16 77480 77464 118.073 68.625 0.00327483 0.00845303 2024-09-21T10:05:50.674 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:50.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:50.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:50.776 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:50.820 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:50.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:50.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:50.922 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:50.957 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:50.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:50.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:51.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:51.099 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:51.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:51.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:51.201 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:51.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:51.239 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:51.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:51.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:51.369 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:51.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:51.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:51.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:51.505 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:51.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:51.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:51.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:51.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:51.642 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:51.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:51.656 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 42 16 78703 78687 117.08 76.4375 0.0132566 0.00853646 2024-09-21T10:05:51.743 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:51.765 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:51.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:51.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:51.867 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:51.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:51.906 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:51.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:52.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:52.024 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:05:52.035 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:52.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:52.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:52.092 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:05:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:05:51] "GET /metrics HTTP/1.1" 200 36695 "" "Prometheus/2.51.0" 2024-09-21T10:05:52.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:52.178 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:52.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:52.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:52.314 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:52.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:52.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:52.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:52 smithi107 ceph-mon[102221]: pgmap v710: 109 pgs: 109 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 62 MiB/s wr, 989 op/s 2024-09-21T10:05:52.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:52 smithi107 ceph-mon[105022]: pgmap v710: 109 pgs: 109 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 62 MiB/s wr, 989 op/s 2024-09-21T10:05:52.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:52.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:52 smithi129 ceph-mon[88471]: pgmap v710: 109 pgs: 109 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 62 MiB/s wr, 989 op/s 2024-09-21T10:05:52.438 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:52.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:52.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:52.540 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:52.570 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:52.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:52.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:52.656 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 43 16 79816 79800 115.975 69.5625 0.0244894 0.00861655 2024-09-21T10:05:52.672 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:52.722 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:52.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:52.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:52.824 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:52.853 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:52.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:52.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:52.955 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:52.992 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:52.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:52.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:53.094 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:53.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:53.131 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:53.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:53.207 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:05:53.210 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:05:53.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:53.284 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:53.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:53.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:53.387 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:53.414 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:53.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:53.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:53.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:53.545 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:05:53.545 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:05:53.553 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:53.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:53.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:53.568 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:05:53.568 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:05:53.597 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:05:53.597 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:05:53.629 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:05:53.629 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:05:53.656 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:53.657 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 44 16 80941 80925 114.937 70.3125 0.0078063 0.00869471 2024-09-21T10:05:53.658 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:05:53.658 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:05:53.689 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:05:53.689 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:05:53.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:53.690 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:53.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:53.724 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:05:53.724 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:05:53.755 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:05:53.756 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:05:53.791 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:53.832 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:53.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:53.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:53.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:05:53.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:54.029 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:54.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:54.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:54.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:54.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:54.176 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:54.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:54.277 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:54.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:54.310 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:54.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:54.411 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:54.451 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:54.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:54.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:54.553 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:54.575 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:54.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:54.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:54 smithi107 ceph-mon[102221]: pgmap v711: 109 pgs: 109 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-21T10:05:54.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:54 smithi107 ceph-mon[102221]: from='client.36255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:05:54.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:54 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2469970560' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:54.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:54 smithi107 ceph-mon[105022]: pgmap v711: 109 pgs: 109 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-21T10:05:54.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:54 smithi107 ceph-mon[105022]: from='client.36255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:05:54.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:54 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2469970560' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:54.656 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 45 16 81895 81879 113.707 59.625 0.00784084 0.00878848 2024-09-21T10:05:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:54 smithi129 ceph-mon[88471]: pgmap v711: 109 pgs: 109 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-21T10:05:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:54 smithi129 ceph-mon[88471]: from='client.36255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:05:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:54 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2469970560' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:05:54.676 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:54.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:54.711 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:54.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:54.812 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:54.844 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:54.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:54.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:54.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:54.982 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:54.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:54.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:55.084 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:55.132 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:55.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:55.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:55.234 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:55.273 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:55.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:55.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:55.374 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:55.428 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:55.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:55.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:55.530 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:55.567 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:55.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:55.656 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 46 16 83133 83117 112.918 77.375 0.0130141 0.00885055 2024-09-21T10:05:55.669 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:55.710 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:55.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:55.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:55.812 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:55.840 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:55.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:55.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:55.941 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:55.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:55.963 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:55.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:56.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:56.086 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:56.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:56.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:56.179 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":711,"stamp":"2024-09-21T10:05:53.149170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5164767181,"num_objects":78825,"num_object_clones":0,"num_object_copies":236475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78863,"num_write_kb":5043168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":9648,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80958,"ondisk_log_size":80958,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":19962320,"kb_used_data":15008396,"kb_used_omap":40,"kb_used_meta":4953815,"kb_avail":729998896,"statfs":{"total":767960285184,"available":747518869504,"internally_reserved":0,"allocated":15368597504,"data_stored":15360088550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41367,"internal_metadata":5072707177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,3,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":827457536,"num_objects":12626,"num_object_clones":0,"num_object_copies":37878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12626,"num_write_kb":808064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.853566"},"pg_stats":[{"pgid":"1.11","version":"77'100","reported_seq":270,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.344598+0000","last_change":"2024-09-21T10:04:42.586770+0000","last_active":"2024-09-21T10:05:47.344598+0000","last_peered":"2024-09-21T10:05:47.344598+0000","last_clean":"2024-09-21T10:05:47.344598+0000","last_became_active":"2024-09-21T10:04:42.544825+0000","last_became_peered":"2024-09-21T10:04:42.544825+0000","last_unstale":"2024-09-21T10:05:47.344598+0000","last_undegraded":"2024-09-21T10:05:47.344598+0000","last_fullsized":"2024-09-21T10:05:47.344598+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:42.586723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:42.586723+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:31.536325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":251,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.336729+0000","last_change":"2024-09-21T10:03:38.389577+0000","last_active":"2024-09-21T10:05:47.336729+0000","last_peered":"2024-09-21T10:05:47.336729+0000","last_clean":"2024-09-21T10:05:47.336729+0000","last_became_active":"2024-09-21T10:03:38.389292+0000","last_became_peered":"2024-09-21T10:03:38.389292+0000","last_unstale":"2024-09-21T10:05:47.336729+0000","last_undegraded":"2024-09-21T10:05:47.336729+0000","last_fullsized":"2024-09-21T10:05:47.336729+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:32.113472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":54,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296767+0000","last_change":"2024-09-21T10:03:38.404072+0000","last_active":"2024-09-21T10:05:44.296767+0000","last_peered":"2024-09-21T10:05:44.296767+0000","last_clean":"2024-09-21T10:05:44.296767+0000","last_became_active":"2024-09-21T10:03:38.403652+0000","last_became_peered":"2024-09-21T10:03:38.403652+0000","last_unstale":"2024-09-21T10:05:44.296767+0000","last_undegraded":"2024-09-21T10:05:44.296767+0000","last_fullsized":"2024-09-21T10:05:44.296767+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:13:33.708105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'100","reported_seq":31,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.353083+0000","last_change":"2024-09-21T10:04:43.321729+0000","last_active":"2024-09-21T10:05:44.353083+0000","last_peered":"2024-09-21T10:05:44.353083+0000","last_clean":"2024-09-21T10:05:44.353083+0000","last_became_active":"2024-09-21T10:04:42.546211+0000","last_became_peered":"2024-09-21T10:04:42.546211+0000","last_unstale":"2024-09-21T10:05:44.353083+0000","last_undegraded":"2024-09-21T10:05:44.353083+0000","last_fullsized":"2024-09-21T10:05:44.353083+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.321671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.321671+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:40:08.766043+0000","scrub_duration":0.0012263879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":201,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.353652+0000","last_change":"2024-09-21T10:03:38.372672+0000","last_active":"2024-09-21T10:05:44.353652+0000","last_peered":"2024-09-21T10:05:44.353652+0000","last_clean":"2024-09-21T10:05:44.353652+0000","last_became_active":"2024-09-21T10:03:38.372223+0000","last_became_peered":"2024-09-21T10:03:38.372223+0000","last_unstale":"2024-09-21T10:05:44.353652+0000","last_undegraded":"2024-09-21T10:05:44.353652+0000","last_fullsized":"2024-09-21T10:05:44.353652+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:05:25.460564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":300,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.346472+0000","last_change":"2024-09-21T10:03:38.378755+0000","last_active":"2024-09-21T10:05:47.346472+0000","last_peered":"2024-09-21T10:05:47.346472+0000","last_clean":"2024-09-21T10:05:47.346472+0000","last_became_active":"2024-09-21T10:03:38.378338+0000","last_became_peered":"2024-09-21T10:03:38.378338+0000","last_unstale":"2024-09-21T10:05:47.346472+0000","last_undegraded":"2024-09-21T10:05:47.346472+0000","last_fullsized":"2024-09-21T10:05:47.346472+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:17:12.051549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'100","reported_seq":42,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.341775+0000","last_change":"2024-09-21T10:04:50.433274+0000","last_active":"2024-09-21T10:05:47.341775+0000","last_peered":"2024-09-21T10:05:47.341775+0000","last_clean":"2024-09-21T10:05:47.341775+0000","last_became_active":"2024-09-21T10:04:42.545783+0000","last_became_peered":"2024-09-21T10:04:42.545783+0000","last_unstale":"2024-09-21T10:05:47.341775+0000","last_undegraded":"2024-09-21T10:05:47.341775+0000","last_fullsized":"2024-09-21T10:05:47.341775+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.433227+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:10:59.708360+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":301,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.348016+0000","last_change":"2024-09-21T10:03:38.379045+0000","last_active":"2024-09-21T10:05:47.348016+0000","last_peered":"2024-09-21T10:05:47.348016+0000","last_clean":"2024-09-21T10:05:47.348016+0000","last_became_active":"2024-09-21T10:03:38.377507+0000","last_became_peered":"2024-09-21T10:03:38.377507+0000","last_unstale":"2024-09-21T10:05:47.348016+0000","last_undegraded":"2024-09-21T10:05:47.348016+0000","last_fullsized":"2024-09-21T10:05:47.348016+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:07:26.215863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":48,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296677+0000","last_change":"2024-09-21T10:03:38.410388+0000","last_active":"2024-09-21T10:05:44.296677+0000","last_peered":"2024-09-21T10:05:44.296677+0000","last_clean":"2024-09-21T10:05:44.296677+0000","last_became_active":"2024-09-21T10:03:38.402465+0000","last_became_peered":"2024-09-21T10:03:38.402465+0000","last_unstale":"2024-09-21T10:05:44.296677+0000","last_undegraded":"2024-09-21T10:05:44.296677+0000","last_fullsized":"2024-09-21T10:05:44.296677+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:54:49.777097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"77'100","reported_seq":31,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.353191+0000","last_change":"2024-09-21T10:04:45.295392+0000","last_active":"2024-09-21T10:05:44.353191+0000","last_peered":"2024-09-21T10:05:44.353191+0000","last_clean":"2024-09-21T10:05:44.353191+0000","last_became_active":"2024-09-21T10:04:42.545479+0000","last_became_peered":"2024-09-21T10:04:42.545479+0000","last_unstale":"2024-09-21T10:05:44.353191+0000","last_undegraded":"2024-09-21T10:05:44.353191+0000","last_fullsized":"2024-09-21T10:05:44.353191+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.295331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.295331+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:20.770065+0000","scrub_duration":0.0016030269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":251,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.339229+0000","last_change":"2024-09-21T10:03:38.381552+0000","last_active":"2024-09-21T10:05:47.339229+0000","last_peered":"2024-09-21T10:05:47.339229+0000","last_clean":"2024-09-21T10:05:47.339229+0000","last_became_active":"2024-09-21T10:03:38.380898+0000","last_became_peered":"2024-09-21T10:03:38.380898+0000","last_unstale":"2024-09-21T10:05:47.339229+0000","last_undegraded":"2024-09-21T10:05:47.339229+0000","last_fullsized":"2024-09-21T10:05:47.339229+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:53.409606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":172,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.350473+0000","last_change":"2024-09-21T09:48:54.140948+0000","last_active":"2024-09-21T10:05:44.350473+0000","last_peered":"2024-09-21T10:05:44.350473+0000","last_clean":"2024-09-21T10:05:44.350473+0000","last_became_active":"2024-09-21T09:48:54.140641+0000","last_became_peered":"2024-09-21T09:48:54.140641+0000","last_unstale":"2024-09-21T10:05:44.350473+0000","last_undegraded":"2024-09-21T10:05:44.350473+0000","last_fullsized":"2024-09-21T10:05:44.350473+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:25.157599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"200'4873","reported_seq":4890,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:52.793482+0000","last_change":"2024-09-21T10:04:55.601105+0000","last_active":"2024-09-21T10:05:52.793482+0000","last_peered":"2024-09-21T10:05:52.793482+0000","last_clean":"2024-09-21T10:05:52.793482+0000","last_became_active":"2024-09-21T10:04:55.600833+0000","last_became_peered":"2024-09-21T10:04:55.600833+0000","last_unstale":"2024-09-21T10:05:52.793482+0000","last_undegraded":"2024-09-21T10:05:52.793482+0000","last_fullsized":"2024-09-21T10:05:52.793482+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":4873,"log_dups_size":0,"ondisk_log_size":4873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:45.109273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319356928,"num_objects":4873,"num_object_clones":0,"num_object_copies":14619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4873,"num_write_kb":311872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"77'100","reported_seq":74,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297150+0000","last_change":"2024-09-21T10:04:55.259467+0000","last_active":"2024-09-21T10:05:44.297150+0000","last_peered":"2024-09-21T10:05:44.297150+0000","last_clean":"2024-09-21T10:05:44.297150+0000","last_became_active":"2024-09-21T10:04:42.544426+0000","last_became_peered":"2024-09-21T10:04:42.544426+0000","last_unstale":"2024-09-21T10:05:44.297150+0000","last_undegraded":"2024-09-21T10:05:44.297150+0000","last_fullsized":"2024-09-21T10:05:44.297150+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:55.259385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:55.259385+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:50.367631+0000","scrub_duration":0.0015667260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":52,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296571+0000","last_change":"2024-09-21T10:03:38.399320+0000","last_active":"2024-09-21T10:05:44.296571+0000","last_peered":"2024-09-21T10:05:44.296571+0000","last_clean":"2024-09-21T10:05:44.296571+0000","last_became_active":"2024-09-21T10:03:38.383894+0000","last_became_peered":"2024-09-21T10:03:38.383894+0000","last_unstale":"2024-09-21T10:05:44.296571+0000","last_undegraded":"2024-09-21T10:05:44.296571+0000","last_fullsized":"2024-09-21T10:05:44.296571+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:09:55.613727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":184,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.353170+0000","last_change":"2024-09-21T10:03:38.408828+0000","last_active":"2024-09-21T10:05:44.353170+0000","last_peered":"2024-09-21T10:05:44.353170+0000","last_clean":"2024-09-21T10:05:44.353170+0000","last_became_active":"2024-09-21T10:03:38.408327+0000","last_became_peered":"2024-09-21T10:03:38.408327+0000","last_unstale":"2024-09-21T10:05:44.353170+0000","last_undegraded":"2024-09-21T10:05:44.353170+0000","last_fullsized":"2024-09-21T10:05:44.353170+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:26.872372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"200'4961","reported_seq":4979,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:52.296683+0000","last_change":"2024-09-21T10:04:55.802720+0000","last_active":"2024-09-21T10:05:52.296683+0000","last_peered":"2024-09-21T10:05:52.296683+0000","last_clean":"2024-09-21T10:05:52.296683+0000","last_became_active":"2024-09-21T10:04:55.802349+0000","last_became_peered":"2024-09-21T10:04:55.802349+0000","last_unstale":"2024-09-21T10:05:52.296683+0000","last_undegraded":"2024-09-21T10:05:52.296683+0000","last_fullsized":"2024-09-21T10:05:52.296683+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":4961,"log_dups_size":0,"ondisk_log_size":4961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:00:13.309694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325124096,"num_objects":4961,"num_object_clones":0,"num_object_copies":14883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4961,"num_write_kb":317504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"77'100","reported_seq":46,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.336552+0000","last_change":"2024-09-21T10:04:45.981392+0000","last_active":"2024-09-21T10:05:47.336552+0000","last_peered":"2024-09-21T10:05:47.336552+0000","last_clean":"2024-09-21T10:05:47.336552+0000","last_became_active":"2024-09-21T10:04:42.547751+0000","last_became_peered":"2024-09-21T10:04:42.547751+0000","last_unstale":"2024-09-21T10:05:47.336552+0000","last_undegraded":"2024-09-21T10:05:47.336552+0000","last_fullsized":"2024-09-21T10:05:47.336552+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.981336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.981336+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:26:26.732033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":52,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297397+0000","last_change":"2024-09-21T10:03:38.395125+0000","last_active":"2024-09-21T10:05:44.297397+0000","last_peered":"2024-09-21T10:05:44.297397+0000","last_clean":"2024-09-21T10:05:44.297397+0000","last_became_active":"2024-09-21T10:03:38.391916+0000","last_became_peered":"2024-09-21T10:03:38.391916+0000","last_unstale":"2024-09-21T10:05:44.297397+0000","last_undegraded":"2024-09-21T10:05:44.297397+0000","last_fullsized":"2024-09-21T10:05:44.297397+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:15.723022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":180,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296520+0000","last_change":"2024-09-21T09:56:12.576294+0000","last_active":"2024-09-21T10:05:44.296520+0000","last_peered":"2024-09-21T10:05:44.296520+0000","last_clean":"2024-09-21T10:05:44.296520+0000","last_became_active":"2024-09-21T09:56:12.575928+0000","last_became_peered":"2024-09-21T09:56:12.575928+0000","last_unstale":"2024-09-21T10:05:44.296520+0000","last_undegraded":"2024-09-21T10:05:44.296520+0000","last_fullsized":"2024-09-21T10:05:44.296520+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:23.227435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"200'5042","reported_seq":5072,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:51.931321+0000","last_change":"2024-09-21T10:04:55.801704+0000","last_active":"2024-09-21T10:05:51.931321+0000","last_peered":"2024-09-21T10:05:51.931321+0000","last_clean":"2024-09-21T10:05:51.931321+0000","last_became_active":"2024-09-21T10:04:55.801386+0000","last_became_peered":"2024-09-21T10:04:55.801386+0000","last_unstale":"2024-09-21T10:05:51.931321+0000","last_undegraded":"2024-09-21T10:05:51.931321+0000","last_fullsized":"2024-09-21T10:05:51.931321+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":5042,"log_dups_size":0,"ondisk_log_size":5042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:58.954309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330432512,"num_objects":5042,"num_object_clones":0,"num_object_copies":15126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5042,"num_write_kb":322688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"77'100","reported_seq":174,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.352021+0000","last_change":"2024-09-21T10:04:43.289920+0000","last_active":"2024-09-21T10:05:44.352021+0000","last_peered":"2024-09-21T10:05:44.352021+0000","last_clean":"2024-09-21T10:05:44.352021+0000","last_became_active":"2024-09-21T10:04:42.549133+0000","last_became_peered":"2024-09-21T10:04:42.549133+0000","last_unstale":"2024-09-21T10:05:44.352021+0000","last_undegraded":"2024-09-21T10:05:44.352021+0000","last_fullsized":"2024-09-21T10:05:44.352021+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.289859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.289859+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:58.310069+0000","scrub_duration":0.001251809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":52,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296957+0000","last_change":"2024-09-21T10:03:38.401018+0000","last_active":"2024-09-21T10:05:44.296957+0000","last_peered":"2024-09-21T10:05:44.296957+0000","last_clean":"2024-09-21T10:05:44.296957+0000","last_became_active":"2024-09-21T10:03:38.395207+0000","last_became_peered":"2024-09-21T10:03:38.395207+0000","last_unstale":"2024-09-21T10:05:44.296957+0000","last_undegraded":"2024-09-21T10:05:44.296957+0000","last_fullsized":"2024-09-21T10:05:44.296957+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:06.649204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":200,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.351692+0000","last_change":"2024-09-21T10:03:38.378384+0000","last_active":"2024-09-21T10:05:44.351692+0000","last_peered":"2024-09-21T10:05:44.351692+0000","last_clean":"2024-09-21T10:05:44.351692+0000","last_became_active":"2024-09-21T10:03:38.378072+0000","last_became_peered":"2024-09-21T10:03:38.378072+0000","last_unstale":"2024-09-21T10:05:44.351692+0000","last_undegraded":"2024-09-21T10:05:44.351692+0000","last_fullsized":"2024-09-21T10:05:44.351692+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:54:37.039622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"200'4668","reported_seq":4699,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:49.614864+0000","last_change":"2024-09-21T10:04:55.800926+0000","last_active":"2024-09-21T10:05:49.614864+0000","last_peered":"2024-09-21T10:05:49.614864+0000","last_clean":"2024-09-21T10:05:49.614864+0000","last_became_active":"2024-09-21T10:04:55.800384+0000","last_became_peered":"2024-09-21T10:04:55.800384+0000","last_unstale":"2024-09-21T10:05:49.614864+0000","last_undegraded":"2024-09-21T10:05:49.614864+0000","last_fullsized":"2024-09-21T10:05:49.614864+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":4668,"log_dups_size":0,"ondisk_log_size":4668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:06:54.058899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305922048,"num_objects":4668,"num_object_clones":0,"num_object_copies":14004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4668,"num_write_kb":298752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"77'100","reported_seq":31,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296852+0000","last_change":"2024-09-21T10:04:48.387150+0000","last_active":"2024-09-21T10:05:44.296852+0000","last_peered":"2024-09-21T10:05:44.296852+0000","last_clean":"2024-09-21T10:05:44.296852+0000","last_became_active":"2024-09-21T10:04:42.546121+0000","last_became_peered":"2024-09-21T10:04:42.546121+0000","last_unstale":"2024-09-21T10:05:44.296852+0000","last_undegraded":"2024-09-21T10:05:44.296852+0000","last_fullsized":"2024-09-21T10:05:44.296852+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:48.387088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:48.387088+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:29:48.630550+0000","scrub_duration":0.001663997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":165,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.355822+0000","last_change":"2024-09-21T09:56:12.590219+0000","last_active":"2024-09-21T10:05:44.355822+0000","last_peered":"2024-09-21T10:05:44.355822+0000","last_clean":"2024-09-21T10:05:44.355822+0000","last_became_active":"2024-09-21T09:56:12.589946+0000","last_became_peered":"2024-09-21T09:56:12.589946+0000","last_unstale":"2024-09-21T10:05:44.355822+0000","last_undegraded":"2024-09-21T10:05:44.355822+0000","last_fullsized":"2024-09-21T10:05:44.355822+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:48.919929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":276,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.345900+0000","last_change":"2024-09-21T09:50:45.503336+0000","last_active":"2024-09-21T10:05:47.345900+0000","last_peered":"2024-09-21T10:05:47.345900+0000","last_clean":"2024-09-21T10:05:47.345900+0000","last_became_active":"2024-09-21T09:50:45.503045+0000","last_became_peered":"2024-09-21T09:50:45.503045+0000","last_unstale":"2024-09-21T10:05:47.345900+0000","last_undegraded":"2024-09-21T10:05:47.345900+0000","last_fullsized":"2024-09-21T10:05:47.345900+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:46.728347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"200'4919","reported_seq":4950,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:50.019554+0000","last_change":"2024-09-21T10:04:55.603614+0000","last_active":"2024-09-21T10:05:50.019554+0000","last_peered":"2024-09-21T10:05:50.019554+0000","last_clean":"2024-09-21T10:05:50.019554+0000","last_became_active":"2024-09-21T10:04:55.603210+0000","last_became_peered":"2024-09-21T10:04:55.603210+0000","last_unstale":"2024-09-21T10:05:50.019554+0000","last_undegraded":"2024-09-21T10:05:50.019554+0000","last_fullsized":"2024-09-21T10:05:50.019554+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":4919,"log_dups_size":0,"ondisk_log_size":4919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:13:26.372076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322371584,"num_objects":4919,"num_object_clones":0,"num_object_copies":14757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4919,"num_write_kb":314816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":74,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296756+0000","last_change":"2024-09-21T10:04:53.355975+0000","last_active":"2024-09-21T10:05:44.296756+0000","last_peered":"2024-09-21T10:05:44.296756+0000","last_clean":"2024-09-21T10:05:44.296756+0000","last_became_active":"2024-09-21T10:04:42.547505+0000","last_became_peered":"2024-09-21T10:04:42.547505+0000","last_unstale":"2024-09-21T10:05:44.296756+0000","last_undegraded":"2024-09-21T10:05:44.296756+0000","last_fullsized":"2024-09-21T10:05:44.296756+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.355912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.355912+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:43.456830+0000","scrub_duration":0.0017980030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":214,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.336498+0000","last_change":"2024-09-21T09:56:12.575897+0000","last_active":"2024-09-21T10:05:47.336498+0000","last_peered":"2024-09-21T10:05:47.336498+0000","last_clean":"2024-09-21T10:05:47.336498+0000","last_became_active":"2024-09-21T09:56:12.575532+0000","last_became_peered":"2024-09-21T09:56:12.575532+0000","last_unstale":"2024-09-21T10:05:47.336498+0000","last_undegraded":"2024-09-21T10:05:47.336498+0000","last_fullsized":"2024-09-21T10:05:47.336498+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:01.302499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":163,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297402+0000","last_change":"2024-09-21T09:56:12.588824+0000","last_active":"2024-09-21T10:05:44.297402+0000","last_peered":"2024-09-21T10:05:44.297402+0000","last_clean":"2024-09-21T10:05:44.297402+0000","last_became_active":"2024-09-21T09:56:12.588561+0000","last_became_peered":"2024-09-21T09:56:12.588561+0000","last_unstale":"2024-09-21T10:05:44.297402+0000","last_undegraded":"2024-09-21T10:05:44.297402+0000","last_fullsized":"2024-09-21T10:05:44.297402+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:31:48.900623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"200'4813","reported_seq":4844,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:49.609223+0000","last_change":"2024-09-21T10:04:55.801441+0000","last_active":"2024-09-21T10:05:49.609223+0000","last_peered":"2024-09-21T10:05:49.609223+0000","last_clean":"2024-09-21T10:05:49.609223+0000","last_became_active":"2024-09-21T10:04:55.801163+0000","last_became_peered":"2024-09-21T10:04:55.801163+0000","last_unstale":"2024-09-21T10:05:49.609223+0000","last_undegraded":"2024-09-21T10:05:49.609223+0000","last_fullsized":"2024-09-21T10:05:49.609223+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":4813,"log_dups_size":0,"ondisk_log_size":4813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:03:10.019956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315424768,"num_objects":4813,"num_object_clones":0,"num_object_copies":14439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4813,"num_write_kb":308032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":38,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.353070+0000","last_change":"2024-09-21T10:04:50.312757+0000","last_active":"2024-09-21T10:05:44.353070+0000","last_peered":"2024-09-21T10:05:44.353070+0000","last_clean":"2024-09-21T10:05:44.353070+0000","last_became_active":"2024-09-21T10:04:42.548966+0000","last_became_peered":"2024-09-21T10:04:42.548966+0000","last_unstale":"2024-09-21T10:05:44.353070+0000","last_undegraded":"2024-09-21T10:05:44.353070+0000","last_fullsized":"2024-09-21T10:05:44.353070+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.312657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.312657+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:22:13.119698+0000","scrub_duration":0.001872255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":164,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.351845+0000","last_change":"2024-09-21T09:50:45.501596+0000","last_active":"2024-09-21T10:05:44.351845+0000","last_peered":"2024-09-21T10:05:44.351845+0000","last_clean":"2024-09-21T10:05:44.351845+0000","last_became_active":"2024-09-21T09:50:45.501186+0000","last_became_peered":"2024-09-21T09:50:45.501186+0000","last_unstale":"2024-09-21T10:05:44.351845+0000","last_undegraded":"2024-09-21T10:05:44.351845+0000","last_fullsized":"2024-09-21T10:05:44.351845+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:43:16.449425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":173,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297755+0000","last_change":"2024-09-21T09:51:30.639545+0000","last_active":"2024-09-21T10:05:44.297755+0000","last_peered":"2024-09-21T10:05:44.297755+0000","last_clean":"2024-09-21T10:05:44.297755+0000","last_became_active":"2024-09-21T09:51:30.639106+0000","last_became_peered":"2024-09-21T09:51:30.639106+0000","last_unstale":"2024-09-21T10:05:44.297755+0000","last_undegraded":"2024-09-21T10:05:44.297755+0000","last_fullsized":"2024-09-21T10:05:44.297755+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:19:19.607327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"200'4974","reported_seq":4991,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:52.314909+0000","last_change":"2024-09-21T10:04:55.604878+0000","last_active":"2024-09-21T10:05:52.314909+0000","last_peered":"2024-09-21T10:05:52.314909+0000","last_clean":"2024-09-21T10:05:52.314909+0000","last_became_active":"2024-09-21T10:04:55.604403+0000","last_became_peered":"2024-09-21T10:04:55.604403+0000","last_unstale":"2024-09-21T10:05:52.314909+0000","last_undegraded":"2024-09-21T10:05:52.314909+0000","last_fullsized":"2024-09-21T10:05:52.314909+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":4974,"log_dups_size":0,"ondisk_log_size":4974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:19.128055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325976064,"num_objects":4974,"num_object_clones":0,"num_object_copies":14922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4974,"num_write_kb":318336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"77'100","reported_seq":42,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.341669+0000","last_change":"2024-09-21T10:04:53.540367+0000","last_active":"2024-09-21T10:05:47.341669+0000","last_peered":"2024-09-21T10:05:47.341669+0000","last_clean":"2024-09-21T10:05:47.341669+0000","last_became_active":"2024-09-21T10:04:42.549016+0000","last_became_peered":"2024-09-21T10:04:42.549016+0000","last_unstale":"2024-09-21T10:05:47.341669+0000","last_undegraded":"2024-09-21T10:05:47.341669+0000","last_fullsized":"2024-09-21T10:05:47.341669+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.540323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.540323+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:30.672583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":201,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296711+0000","last_change":"2024-09-21T10:03:38.411439+0000","last_active":"2024-09-21T10:05:44.296711+0000","last_peered":"2024-09-21T10:05:44.296711+0000","last_clean":"2024-09-21T10:05:44.296711+0000","last_became_active":"2024-09-21T10:03:38.411108+0000","last_became_peered":"2024-09-21T10:03:38.411108+0000","last_unstale":"2024-09-21T10:05:44.296711+0000","last_undegraded":"2024-09-21T10:05:44.296711+0000","last_fullsized":"2024-09-21T10:05:44.296711+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:11:54.932822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":163,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297028+0000","last_change":"2024-09-21T09:50:45.510764+0000","last_active":"2024-09-21T10:05:44.297028+0000","last_peered":"2024-09-21T10:05:44.297028+0000","last_clean":"2024-09-21T10:05:44.297028+0000","last_became_active":"2024-09-21T09:50:45.510226+0000","last_became_peered":"2024-09-21T09:50:45.510226+0000","last_unstale":"2024-09-21T10:05:44.297028+0000","last_undegraded":"2024-09-21T10:05:44.297028+0000","last_fullsized":"2024-09-21T10:05:44.297028+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:31:13.491427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"200'4922","reported_seq":4940,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:52.329523+0000","last_change":"2024-09-21T10:04:55.597985+0000","last_active":"2024-09-21T10:05:52.329523+0000","last_peered":"2024-09-21T10:05:52.329523+0000","last_clean":"2024-09-21T10:05:52.329523+0000","last_became_active":"2024-09-21T10:04:55.597712+0000","last_became_peered":"2024-09-21T10:04:55.597712+0000","last_unstale":"2024-09-21T10:05:52.329523+0000","last_undegraded":"2024-09-21T10:05:52.329523+0000","last_fullsized":"2024-09-21T10:05:52.329523+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":4922,"log_dups_size":0,"ondisk_log_size":4922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:12:18.356027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322568192,"num_objects":4922,"num_object_clones":0,"num_object_copies":14766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4922,"num_write_kb":315008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":31,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.353114+0000","last_change":"2024-09-21T10:04:48.318076+0000","last_active":"2024-09-21T10:05:44.353114+0000","last_peered":"2024-09-21T10:05:44.353114+0000","last_clean":"2024-09-21T10:05:44.353114+0000","last_became_active":"2024-09-21T10:04:42.547894+0000","last_became_peered":"2024-09-21T10:04:42.547894+0000","last_unstale":"2024-09-21T10:05:44.353114+0000","last_undegraded":"2024-09-21T10:05:44.353114+0000","last_fullsized":"2024-09-21T10:05:44.353114+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:48.318010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:48.318010+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:15.411371+0000","scrub_duration":0.001567564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":252,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.336560+0000","last_change":"2024-09-21T10:03:38.386604+0000","last_active":"2024-09-21T10:05:47.336560+0000","last_peered":"2024-09-21T10:05:47.336560+0000","last_clean":"2024-09-21T10:05:47.336560+0000","last_became_active":"2024-09-21T10:03:38.386283+0000","last_became_peered":"2024-09-21T10:03:38.386283+0000","last_unstale":"2024-09-21T10:05:47.336560+0000","last_undegraded":"2024-09-21T10:05:47.336560+0000","last_fullsized":"2024-09-21T10:05:47.336560+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:27:05.944880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":247,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.342317+0000","last_change":"2024-09-21T09:56:12.575396+0000","last_active":"2024-09-21T10:05:47.342317+0000","last_peered":"2024-09-21T10:05:47.342317+0000","last_clean":"2024-09-21T10:05:47.342317+0000","last_became_active":"2024-09-21T09:56:12.574913+0000","last_became_peered":"2024-09-21T09:56:12.574913+0000","last_unstale":"2024-09-21T10:05:47.342317+0000","last_undegraded":"2024-09-21T10:05:47.342317+0000","last_fullsized":"2024-09-21T10:05:47.342317+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:05.462880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"200'5049","reported_seq":5067,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:52.307142+0000","last_change":"2024-09-21T10:04:55.602045+0000","last_active":"2024-09-21T10:05:52.307142+0000","last_peered":"2024-09-21T10:05:52.307142+0000","last_clean":"2024-09-21T10:05:52.307142+0000","last_became_active":"2024-09-21T10:04:55.601582+0000","last_became_peered":"2024-09-21T10:04:55.601582+0000","last_unstale":"2024-09-21T10:05:52.307142+0000","last_undegraded":"2024-09-21T10:05:52.307142+0000","last_fullsized":"2024-09-21T10:05:52.307142+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":5049,"log_dups_size":0,"ondisk_log_size":5049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:55:40.829352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330891264,"num_objects":5049,"num_object_clones":0,"num_object_copies":15147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5049,"num_write_kb":323136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":64,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296568+0000","last_change":"2024-09-21T10:04:47.393223+0000","last_active":"2024-09-21T10:05:44.296568+0000","last_peered":"2024-09-21T10:05:44.296568+0000","last_clean":"2024-09-21T10:05:44.296568+0000","last_became_active":"2024-09-21T10:03:38.384065+0000","last_became_peered":"2024-09-21T10:03:38.384065+0000","last_unstale":"2024-09-21T10:05:44.296568+0000","last_undegraded":"2024-09-21T10:05:44.296568+0000","last_fullsized":"2024-09-21T10:05:44.296568+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:47.393146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:47.393146+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:33:03.961914+0000","scrub_duration":0.001601086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":223,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.357051+0000","last_change":"2024-09-21T10:03:38.422217+0000","last_active":"2024-09-21T10:05:44.357051+0000","last_peered":"2024-09-21T10:05:44.357051+0000","last_clean":"2024-09-21T10:05:44.357051+0000","last_became_active":"2024-09-21T10:03:38.421901+0000","last_became_peered":"2024-09-21T10:03:38.421901+0000","last_unstale":"2024-09-21T10:05:44.357051+0000","last_undegraded":"2024-09-21T10:05:44.357051+0000","last_fullsized":"2024-09-21T10:05:44.357051+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:15:01.432624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039581700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":215,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.336957+0000","last_change":"2024-09-21T09:56:12.580057+0000","last_active":"2024-09-21T10:05:47.336957+0000","last_peered":"2024-09-21T10:05:47.336957+0000","last_clean":"2024-09-21T10:05:47.336957+0000","last_became_active":"2024-09-21T09:56:12.579700+0000","last_became_peered":"2024-09-21T09:56:12.579700+0000","last_unstale":"2024-09-21T10:05:47.336957+0000","last_undegraded":"2024-09-21T10:05:47.336957+0000","last_fullsized":"2024-09-21T10:05:47.336957+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:49.768872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":183,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.351906+0000","last_change":"2024-09-21T10:03:38.371444+0000","last_active":"2024-09-21T10:05:44.351906+0000","last_peered":"2024-09-21T10:05:44.351906+0000","last_clean":"2024-09-21T10:05:44.351906+0000","last_became_active":"2024-09-21T10:03:38.371174+0000","last_became_peered":"2024-09-21T10:03:38.371174+0000","last_unstale":"2024-09-21T10:05:44.351906+0000","last_undegraded":"2024-09-21T10:05:44.351906+0000","last_fullsized":"2024-09-21T10:05:44.351906+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:13:01.365129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"200'4803","reported_seq":4833,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:51.926528+0000","last_change":"2024-09-21T10:04:55.600777+0000","last_active":"2024-09-21T10:05:51.926528+0000","last_peered":"2024-09-21T10:05:51.926528+0000","last_clean":"2024-09-21T10:05:51.926528+0000","last_became_active":"2024-09-21T10:04:55.600435+0000","last_became_peered":"2024-09-21T10:04:55.600435+0000","last_unstale":"2024-09-21T10:05:51.926528+0000","last_undegraded":"2024-09-21T10:05:51.926528+0000","last_fullsized":"2024-09-21T10:05:51.926528+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":4803,"log_dups_size":0,"ondisk_log_size":4803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:05:26.161084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314769408,"num_objects":4803,"num_object_clones":0,"num_object_copies":14409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4803,"num_write_kb":307392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":42,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.342054+0000","last_change":"2024-09-21T10:04:51.462285+0000","last_active":"2024-09-21T10:05:47.342054+0000","last_peered":"2024-09-21T10:05:47.342054+0000","last_clean":"2024-09-21T10:05:47.342054+0000","last_became_active":"2024-09-21T10:04:42.546242+0000","last_became_peered":"2024-09-21T10:04:42.546242+0000","last_unstale":"2024-09-21T10:05:47.342054+0000","last_undegraded":"2024-09-21T10:05:47.342054+0000","last_fullsized":"2024-09-21T10:05:47.342054+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:51.462213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:51.462213+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:10.564260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":48,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297430+0000","last_change":"2024-09-21T10:03:38.393867+0000","last_active":"2024-09-21T10:05:44.297430+0000","last_peered":"2024-09-21T10:05:44.297430+0000","last_clean":"2024-09-21T10:05:44.297430+0000","last_became_active":"2024-09-21T10:03:38.392820+0000","last_became_peered":"2024-09-21T10:03:38.392820+0000","last_unstale":"2024-09-21T10:05:44.297430+0000","last_undegraded":"2024-09-21T10:05:44.297430+0000","last_fullsized":"2024-09-21T10:05:44.297430+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:19:54.697914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":276,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.344589+0000","last_change":"2024-09-21T09:56:12.573698+0000","last_active":"2024-09-21T10:05:47.344589+0000","last_peered":"2024-09-21T10:05:47.344589+0000","last_clean":"2024-09-21T10:05:47.344589+0000","last_became_active":"2024-09-21T09:56:12.573390+0000","last_became_peered":"2024-09-21T09:56:12.573390+0000","last_unstale":"2024-09-21T10:05:47.344589+0000","last_undegraded":"2024-09-21T10:05:47.344589+0000","last_fullsized":"2024-09-21T10:05:47.344589+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:55:12.937878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":287,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.347846+0000","last_change":"2024-09-21T09:51:30.666462+0000","last_active":"2024-09-21T10:05:47.347846+0000","last_peered":"2024-09-21T10:05:47.347846+0000","last_clean":"2024-09-21T10:05:47.347846+0000","last_became_active":"2024-09-21T09:51:30.666006+0000","last_became_peered":"2024-09-21T09:51:30.666006+0000","last_unstale":"2024-09-21T10:05:47.347846+0000","last_undegraded":"2024-09-21T10:05:47.347846+0000","last_fullsized":"2024-09-21T10:05:47.347846+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:51.481698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"200'5054","reported_seq":5071,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:52.806856+0000","last_change":"2024-09-21T10:04:55.602517+0000","last_active":"2024-09-21T10:05:52.806856+0000","last_peered":"2024-09-21T10:05:52.806856+0000","last_clean":"2024-09-21T10:05:52.806856+0000","last_became_active":"2024-09-21T10:04:55.602231+0000","last_became_peered":"2024-09-21T10:04:55.602231+0000","last_unstale":"2024-09-21T10:05:52.806856+0000","last_undegraded":"2024-09-21T10:05:52.806856+0000","last_fullsized":"2024-09-21T10:05:52.806856+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":5054,"log_dups_size":0,"ondisk_log_size":5054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:52:08.852510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331218944,"num_objects":5054,"num_object_clones":0,"num_object_copies":15162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5054,"num_write_kb":323456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":74,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297591+0000","last_change":"2024-09-21T10:04:52.361225+0000","last_active":"2024-09-21T10:05:44.297591+0000","last_peered":"2024-09-21T10:05:44.297591+0000","last_clean":"2024-09-21T10:05:44.297591+0000","last_became_active":"2024-09-21T10:04:42.543054+0000","last_became_peered":"2024-09-21T10:04:42.543054+0000","last_unstale":"2024-09-21T10:05:44.297591+0000","last_undegraded":"2024-09-21T10:05:44.297591+0000","last_fullsized":"2024-09-21T10:05:44.297591+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:52.361137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:52.361137+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:20:43.739146+0000","scrub_duration":0.001255358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":278,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.333837+0000","last_change":"2024-09-21T09:56:12.580659+0000","last_active":"2024-09-21T10:05:47.333837+0000","last_peered":"2024-09-21T10:05:47.333837+0000","last_clean":"2024-09-21T10:05:47.333837+0000","last_became_active":"2024-09-21T09:56:12.580300+0000","last_became_peered":"2024-09-21T09:56:12.580300+0000","last_unstale":"2024-09-21T10:05:47.333837+0000","last_undegraded":"2024-09-21T10:05:47.333837+0000","last_fullsized":"2024-09-21T10:05:47.333837+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:04.945404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":247,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.342546+0000","last_change":"2024-09-21T09:51:30.646466+0000","last_active":"2024-09-21T10:05:47.342546+0000","last_peered":"2024-09-21T10:05:47.342546+0000","last_clean":"2024-09-21T10:05:47.342546+0000","last_became_active":"2024-09-21T09:51:30.646209+0000","last_became_peered":"2024-09-21T09:51:30.646209+0000","last_unstale":"2024-09-21T10:05:47.342546+0000","last_undegraded":"2024-09-21T10:05:47.342546+0000","last_fullsized":"2024-09-21T10:05:47.342546+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:44:41.465418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":251,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.333731+0000","last_change":"2024-09-21T09:51:30.649056+0000","last_active":"2024-09-21T10:05:47.333731+0000","last_peered":"2024-09-21T10:05:47.333731+0000","last_clean":"2024-09-21T10:05:47.333731+0000","last_became_active":"2024-09-21T09:51:30.647245+0000","last_became_peered":"2024-09-21T09:51:30.647245+0000","last_unstale":"2024-09-21T10:05:47.333731+0000","last_undegraded":"2024-09-21T10:05:47.333731+0000","last_fullsized":"2024-09-21T10:05:47.333731+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:32.661847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"200'5060","reported_seq":5078,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:52.308872+0000","last_change":"2024-09-21T10:04:55.603587+0000","last_active":"2024-09-21T10:05:52.308872+0000","last_peered":"2024-09-21T10:05:52.308872+0000","last_clean":"2024-09-21T10:05:52.308872+0000","last_became_active":"2024-09-21T10:04:55.603173+0000","last_became_peered":"2024-09-21T10:04:55.603173+0000","last_unstale":"2024-09-21T10:05:52.308872+0000","last_undegraded":"2024-09-21T10:05:52.308872+0000","last_fullsized":"2024-09-21T10:05:52.308872+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":5060,"log_dups_size":0,"ondisk_log_size":5060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:12:45.473222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331612160,"num_objects":5060,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5060,"num_write_kb":323840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":31,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.353235+0000","last_change":"2024-09-21T10:04:44.288106+0000","last_active":"2024-09-21T10:05:44.353235+0000","last_peered":"2024-09-21T10:05:44.353235+0000","last_clean":"2024-09-21T10:05:44.353235+0000","last_became_active":"2024-09-21T10:04:42.542859+0000","last_became_peered":"2024-09-21T10:04:42.542859+0000","last_unstale":"2024-09-21T10:05:44.353235+0000","last_undegraded":"2024-09-21T10:05:44.353235+0000","last_fullsized":"2024-09-21T10:05:44.353235+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:44.288000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:44.288000+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:29.958780+0000","scrub_duration":0.0022491059999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":179,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.353505+0000","last_change":"2024-09-21T09:50:45.502094+0000","last_active":"2024-09-21T10:05:44.353505+0000","last_peered":"2024-09-21T10:05:44.353505+0000","last_clean":"2024-09-21T10:05:44.353505+0000","last_became_active":"2024-09-21T09:50:45.501603+0000","last_became_peered":"2024-09-21T09:50:45.501603+0000","last_unstale":"2024-09-21T10:05:44.353505+0000","last_undegraded":"2024-09-21T10:05:44.353505+0000","last_fullsized":"2024-09-21T10:05:44.353505+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:01.747067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091939799999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":164,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297188+0000","last_change":"2024-09-21T09:48:54.128617+0000","last_active":"2024-09-21T10:05:44.297188+0000","last_peered":"2024-09-21T10:05:44.297188+0000","last_clean":"2024-09-21T10:05:44.297188+0000","last_became_active":"2024-09-21T09:48:54.127878+0000","last_became_peered":"2024-09-21T09:48:54.127878+0000","last_unstale":"2024-09-21T10:05:44.297188+0000","last_undegraded":"2024-09-21T10:05:44.297188+0000","last_fullsized":"2024-09-21T10:05:44.297188+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:07.305638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":259,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.341027+0000","last_change":"2024-09-21T09:56:12.588992+0000","last_active":"2024-09-21T10:05:47.341027+0000","last_peered":"2024-09-21T10:05:47.341027+0000","last_clean":"2024-09-21T10:05:47.341027+0000","last_became_active":"2024-09-21T09:56:12.588665+0000","last_became_peered":"2024-09-21T09:56:12.588665+0000","last_unstale":"2024-09-21T10:05:47.341027+0000","last_undegraded":"2024-09-21T10:05:47.341027+0000","last_fullsized":"2024-09-21T10:05:47.341027+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:19:06.160456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"200'4838","reported_seq":4856,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:51.481792+0000","last_change":"2024-09-21T10:04:55.601700+0000","last_active":"2024-09-21T10:05:51.481792+0000","last_peered":"2024-09-21T10:05:51.481792+0000","last_clean":"2024-09-21T10:05:51.481792+0000","last_became_active":"2024-09-21T10:04:55.601163+0000","last_became_peered":"2024-09-21T10:04:55.601163+0000","last_unstale":"2024-09-21T10:05:51.481792+0000","last_undegraded":"2024-09-21T10:05:51.481792+0000","last_fullsized":"2024-09-21T10:05:51.481792+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":4838,"log_dups_size":0,"ondisk_log_size":4838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:15:47.805823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317063168,"num_objects":4838,"num_object_clones":0,"num_object_copies":14514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4838,"num_write_kb":309632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":174,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.352002+0000","last_change":"2024-09-21T10:04:45.242148+0000","last_active":"2024-09-21T10:05:44.352002+0000","last_peered":"2024-09-21T10:05:44.352002+0000","last_clean":"2024-09-21T10:05:44.352002+0000","last_became_active":"2024-09-21T10:04:42.549844+0000","last_became_peered":"2024-09-21T10:04:42.549844+0000","last_unstale":"2024-09-21T10:05:44.352002+0000","last_undegraded":"2024-09-21T10:05:44.352002+0000","last_fullsized":"2024-09-21T10:05:44.352002+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_deep_scrub":"77'100","last_deep_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.242073+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:03:42.250480+0000","scrub_duration":0.0023466239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":182,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.353786+0000","last_change":"2024-09-21T09:56:12.583280+0000","last_active":"2024-09-21T10:05:44.353786+0000","last_peered":"2024-09-21T10:05:44.353786+0000","last_clean":"2024-09-21T10:05:44.353786+0000","last_became_active":"2024-09-21T09:56:12.583007+0000","last_became_peered":"2024-09-21T09:56:12.583007+0000","last_unstale":"2024-09-21T10:05:44.353786+0000","last_undegraded":"2024-09-21T10:05:44.353786+0000","last_fullsized":"2024-09-21T10:05:44.353786+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:26.991596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":311,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.341192+0000","last_change":"2024-09-21T10:03:38.392622+0000","last_active":"2024-09-21T10:05:47.341192+0000","last_peered":"2024-09-21T10:05:47.341192+0000","last_clean":"2024-09-21T10:05:47.341192+0000","last_became_active":"2024-09-21T10:03:38.392290+0000","last_became_peered":"2024-09-21T10:03:38.392290+0000","last_unstale":"2024-09-21T10:05:47.341192+0000","last_undegraded":"2024-09-21T10:05:47.341192+0000","last_fullsized":"2024-09-21T10:05:47.341192+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:56:22.222321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":237,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.341329+0000","last_change":"2024-09-21T09:50:45.501349+0000","last_active":"2024-09-21T10:05:47.341329+0000","last_peered":"2024-09-21T10:05:47.341329+0000","last_clean":"2024-09-21T10:05:47.341329+0000","last_became_active":"2024-09-21T09:50:45.501065+0000","last_became_peered":"2024-09-21T09:50:45.501065+0000","last_unstale":"2024-09-21T10:05:47.341329+0000","last_undegraded":"2024-09-21T10:05:47.341329+0000","last_fullsized":"2024-09-21T10:05:47.341329+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:28:58.570586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"200'4881","reported_seq":4899,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:52.296593+0000","last_change":"2024-09-21T10:04:55.603954+0000","last_active":"2024-09-21T10:05:52.296593+0000","last_peered":"2024-09-21T10:05:52.296593+0000","last_clean":"2024-09-21T10:05:52.296593+0000","last_became_active":"2024-09-21T10:04:55.603666+0000","last_became_peered":"2024-09-21T10:04:55.603666+0000","last_unstale":"2024-09-21T10:05:52.296593+0000","last_undegraded":"2024-09-21T10:05:52.296593+0000","last_fullsized":"2024-09-21T10:05:52.296593+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":4881,"log_dups_size":0,"ondisk_log_size":4881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:33.958646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319881216,"num_objects":4881,"num_object_clones":0,"num_object_copies":14643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4881,"num_write_kb":312384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":174,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.351910+0000","last_change":"2024-09-21T10:04:49.226571+0000","last_active":"2024-09-21T10:05:44.351910+0000","last_peered":"2024-09-21T10:05:44.351910+0000","last_clean":"2024-09-21T10:05:44.351910+0000","last_became_active":"2024-09-21T10:04:42.547869+0000","last_became_peered":"2024-09-21T10:04:42.547869+0000","last_unstale":"2024-09-21T10:05:44.351910+0000","last_undegraded":"2024-09-21T10:05:44.351910+0000","last_fullsized":"2024-09-21T10:05:44.351910+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:49.226496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:49.226496+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:14:59.303054+0000","scrub_duration":0.001656865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":247,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.341102+0000","last_change":"2024-09-21T09:51:30.646531+0000","last_active":"2024-09-21T10:05:47.341102+0000","last_peered":"2024-09-21T10:05:47.341102+0000","last_clean":"2024-09-21T10:05:47.341102+0000","last_became_active":"2024-09-21T09:51:30.646248+0000","last_became_peered":"2024-09-21T09:51:30.646248+0000","last_unstale":"2024-09-21T10:05:47.341102+0000","last_undegraded":"2024-09-21T10:05:47.341102+0000","last_fullsized":"2024-09-21T10:05:47.341102+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:00.312924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":48,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297020+0000","last_change":"2024-09-21T10:03:38.402082+0000","last_active":"2024-09-21T10:05:44.297020+0000","last_peered":"2024-09-21T10:05:44.297020+0000","last_clean":"2024-09-21T10:05:44.297020+0000","last_became_active":"2024-09-21T10:03:38.395632+0000","last_became_peered":"2024-09-21T10:03:38.395632+0000","last_unstale":"2024-09-21T10:05:44.297020+0000","last_undegraded":"2024-09-21T10:05:44.297020+0000","last_fullsized":"2024-09-21T10:05:44.297020+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:29:00.631337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":291,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.345985+0000","last_change":"2024-09-21T10:03:38.411780+0000","last_active":"2024-09-21T10:05:47.345985+0000","last_peered":"2024-09-21T10:05:47.345985+0000","last_clean":"2024-09-21T10:05:47.345985+0000","last_became_active":"2024-09-21T10:03:38.411457+0000","last_became_peered":"2024-09-21T10:03:38.411457+0000","last_unstale":"2024-09-21T10:05:47.345985+0000","last_undegraded":"2024-09-21T10:05:47.345985+0000","last_fullsized":"2024-09-21T10:05:47.345985+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:19:08.177347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"200'4996","reported_seq":5014,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:52.314856+0000","last_change":"2024-09-21T10:04:55.603150+0000","last_active":"2024-09-21T10:05:52.314856+0000","last_peered":"2024-09-21T10:05:52.314856+0000","last_clean":"2024-09-21T10:05:52.314856+0000","last_became_active":"2024-09-21T10:04:55.602817+0000","last_became_peered":"2024-09-21T10:04:55.602817+0000","last_unstale":"2024-09-21T10:05:52.314856+0000","last_undegraded":"2024-09-21T10:05:52.314856+0000","last_fullsized":"2024-09-21T10:05:52.314856+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":4996,"log_dups_size":0,"ondisk_log_size":4996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:24:29.153232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327417856,"num_objects":4996,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4996,"num_write_kb":319744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":31,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297679+0000","last_change":"2024-09-21T10:04:44.475698+0000","last_active":"2024-09-21T10:05:44.297679+0000","last_peered":"2024-09-21T10:05:44.297679+0000","last_clean":"2024-09-21T10:05:44.297679+0000","last_became_active":"2024-09-21T10:04:42.549251+0000","last_became_peered":"2024-09-21T10:04:42.549251+0000","last_unstale":"2024-09-21T10:05:44.297679+0000","last_undegraded":"2024-09-21T10:05:44.297679+0000","last_fullsized":"2024-09-21T10:05:44.297679+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:44.475623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:44.475623+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:31:10.067467+0000","scrub_duration":0.0020471650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":247,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.339365+0000","last_change":"2024-09-21T10:03:38.381626+0000","last_active":"2024-09-21T10:05:47.339365+0000","last_peered":"2024-09-21T10:05:47.339365+0000","last_clean":"2024-09-21T10:05:47.339365+0000","last_became_active":"2024-09-21T10:03:38.381000+0000","last_became_peered":"2024-09-21T10:03:38.381000+0000","last_unstale":"2024-09-21T10:05:47.339365+0000","last_undegraded":"2024-09-21T10:05:47.339365+0000","last_fullsized":"2024-09-21T10:05:47.339365+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:34:37.047124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":246,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.341904+0000","last_change":"2024-09-21T09:50:45.513017+0000","last_active":"2024-09-21T10:05:47.341904+0000","last_peered":"2024-09-21T10:05:47.341904+0000","last_clean":"2024-09-21T10:05:47.341904+0000","last_became_active":"2024-09-21T09:50:45.512629+0000","last_became_peered":"2024-09-21T09:50:45.512629+0000","last_unstale":"2024-09-21T10:05:47.341904+0000","last_undegraded":"2024-09-21T10:05:47.341904+0000","last_fullsized":"2024-09-21T10:05:47.341904+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:47:08.421575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":236,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.337945+0000","last_change":"2024-09-21T10:03:38.386785+0000","last_active":"2024-09-21T10:05:47.337945+0000","last_peered":"2024-09-21T10:05:47.337945+0000","last_clean":"2024-09-21T10:05:47.337945+0000","last_became_active":"2024-09-21T10:03:38.386313+0000","last_became_peered":"2024-09-21T10:03:38.386313+0000","last_unstale":"2024-09-21T10:05:47.337945+0000","last_undegraded":"2024-09-21T10:05:47.337945+0000","last_fullsized":"2024-09-21T10:05:47.337945+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:10.633433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"200'4946","reported_seq":4964,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:52.296547+0000","last_change":"2024-09-21T10:04:55.802526+0000","last_active":"2024-09-21T10:05:52.296547+0000","last_peered":"2024-09-21T10:05:52.296547+0000","last_clean":"2024-09-21T10:05:52.296547+0000","last_became_active":"2024-09-21T10:04:55.802207+0000","last_became_peered":"2024-09-21T10:04:55.802207+0000","last_unstale":"2024-09-21T10:05:52.296547+0000","last_undegraded":"2024-09-21T10:05:52.296547+0000","last_fullsized":"2024-09-21T10:05:52.296547+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":4946,"log_dups_size":0,"ondisk_log_size":4946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:22.119648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324141056,"num_objects":4946,"num_object_clones":0,"num_object_copies":14838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4946,"num_write_kb":316544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":42,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.336448+0000","last_change":"2024-09-21T10:04:43.005569+0000","last_active":"2024-09-21T10:05:47.336448+0000","last_peered":"2024-09-21T10:05:47.336448+0000","last_clean":"2024-09-21T10:05:47.336448+0000","last_became_active":"2024-09-21T10:04:42.552636+0000","last_became_peered":"2024-09-21T10:04:42.552636+0000","last_unstale":"2024-09-21T10:05:47.336448+0000","last_undegraded":"2024-09-21T10:05:47.336448+0000","last_fullsized":"2024-09-21T10:05:47.336448+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.005477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.005477+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:06.276495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":279,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.348098+0000","last_change":"2024-09-21T09:50:45.510812+0000","last_active":"2024-09-21T10:05:47.348098+0000","last_peered":"2024-09-21T10:05:47.348098+0000","last_clean":"2024-09-21T10:05:47.348098+0000","last_became_active":"2024-09-21T09:50:45.510480+0000","last_became_peered":"2024-09-21T09:50:45.510480+0000","last_unstale":"2024-09-21T10:05:47.348098+0000","last_undegraded":"2024-09-21T10:05:47.348098+0000","last_fullsized":"2024-09-21T10:05:47.348098+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:05.408194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":213,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.352114+0000","last_change":"2024-09-21T10:03:38.392767+0000","last_active":"2024-09-21T10:05:44.352114+0000","last_peered":"2024-09-21T10:05:44.352114+0000","last_clean":"2024-09-21T10:05:44.352114+0000","last_became_active":"2024-09-21T10:03:38.392424+0000","last_became_peered":"2024-09-21T10:03:38.392424+0000","last_unstale":"2024-09-21T10:05:44.352114+0000","last_undegraded":"2024-09-21T10:05:44.352114+0000","last_fullsized":"2024-09-21T10:05:44.352114+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:25.358291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":146,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296639+0000","last_change":"2024-09-21T09:40:49.190763+0000","last_active":"2024-09-21T10:05:44.296639+0000","last_peered":"2024-09-21T10:05:44.296639+0000","last_clean":"2024-09-21T10:05:44.296639+0000","last_became_active":"2024-09-21T09:40:49.190463+0000","last_became_peered":"2024-09-21T09:40:49.190463+0000","last_unstale":"2024-09-21T10:05:44.296639+0000","last_undegraded":"2024-09-21T10:05:44.296639+0000","last_fullsized":"2024-09-21T10:05:44.296639+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:15.669404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":173,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297060+0000","last_change":"2024-09-21T09:51:30.642074+0000","last_active":"2024-09-21T10:05:44.297060+0000","last_peered":"2024-09-21T10:05:44.297060+0000","last_clean":"2024-09-21T10:05:44.297060+0000","last_became_active":"2024-09-21T09:51:30.641176+0000","last_became_peered":"2024-09-21T09:51:30.641176+0000","last_unstale":"2024-09-21T10:05:44.297060+0000","last_undegraded":"2024-09-21T10:05:44.297060+0000","last_fullsized":"2024-09-21T10:05:44.297060+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:18.766865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":227,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297484+0000","last_change":"2024-09-21T10:03:38.395742+0000","last_active":"2024-09-21T10:05:44.297484+0000","last_peered":"2024-09-21T10:05:44.297484+0000","last_clean":"2024-09-21T10:05:44.297484+0000","last_became_active":"2024-09-21T10:03:38.395439+0000","last_became_peered":"2024-09-21T10:03:38.395439+0000","last_unstale":"2024-09-21T10:05:44.297484+0000","last_undegraded":"2024-09-21T10:05:44.297484+0000","last_fullsized":"2024-09-21T10:05:44.297484+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:27:40.297701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":247,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.341125+0000","last_change":"2024-09-21T09:51:30.642334+0000","last_active":"2024-09-21T10:05:47.341125+0000","last_peered":"2024-09-21T10:05:47.341125+0000","last_clean":"2024-09-21T10:05:47.341125+0000","last_became_active":"2024-09-21T09:51:30.641267+0000","last_became_peered":"2024-09-21T09:51:30.641267+0000","last_unstale":"2024-09-21T10:05:47.341125+0000","last_undegraded":"2024-09-21T10:05:47.341125+0000","last_fullsized":"2024-09-21T10:05:47.341125+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:24.201971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":54,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297084+0000","last_change":"2024-09-21T10:03:38.402359+0000","last_active":"2024-09-21T10:05:44.297084+0000","last_peered":"2024-09-21T10:05:44.297084+0000","last_clean":"2024-09-21T10:05:44.297084+0000","last_became_active":"2024-09-21T10:03:38.400209+0000","last_became_peered":"2024-09-21T10:03:38.400209+0000","last_unstale":"2024-09-21T10:05:44.297084+0000","last_undegraded":"2024-09-21T10:05:44.297084+0000","last_fullsized":"2024-09-21T10:05:44.297084+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:42:04.508541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"77'100","reported_seq":42,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.333260+0000","last_change":"2024-09-21T10:04:43.015055+0000","last_active":"2024-09-21T10:05:47.333260+0000","last_peered":"2024-09-21T10:05:47.333260+0000","last_clean":"2024-09-21T10:05:47.333260+0000","last_became_active":"2024-09-21T10:04:42.545568+0000","last_became_peered":"2024-09-21T10:04:42.545568+0000","last_unstale":"2024-09-21T10:05:47.333260+0000","last_undegraded":"2024-09-21T10:05:47.333260+0000","last_fullsized":"2024-09-21T10:05:47.333260+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.014993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.014993+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:37:24.019431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":202,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.355914+0000","last_change":"2024-09-21T10:03:38.378365+0000","last_active":"2024-09-21T10:05:44.355914+0000","last_peered":"2024-09-21T10:05:44.355914+0000","last_clean":"2024-09-21T10:05:44.355914+0000","last_became_active":"2024-09-21T10:03:38.377388+0000","last_became_peered":"2024-09-21T10:03:38.377388+0000","last_unstale":"2024-09-21T10:05:44.355914+0000","last_undegraded":"2024-09-21T10:05:44.355914+0000","last_fullsized":"2024-09-21T10:05:44.355914+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:31.162683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":241,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.338031+0000","last_change":"2024-09-21T10:03:38.425111+0000","last_active":"2024-09-21T10:05:47.338031+0000","last_peered":"2024-09-21T10:05:47.338031+0000","last_clean":"2024-09-21T10:05:47.338031+0000","last_became_active":"2024-09-21T10:03:38.424704+0000","last_became_peered":"2024-09-21T10:03:38.424704+0000","last_unstale":"2024-09-21T10:05:47.338031+0000","last_undegraded":"2024-09-21T10:05:47.338031+0000","last_fullsized":"2024-09-21T10:05:47.338031+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:35.157076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":275,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.347826+0000","last_change":"2024-09-21T09:56:12.574472+0000","last_active":"2024-09-21T10:05:47.347826+0000","last_peered":"2024-09-21T10:05:47.347826+0000","last_clean":"2024-09-21T10:05:47.347826+0000","last_became_active":"2024-09-21T09:56:12.573966+0000","last_became_peered":"2024-09-21T09:56:12.573966+0000","last_unstale":"2024-09-21T10:05:47.347826+0000","last_undegraded":"2024-09-21T10:05:47.347826+0000","last_fullsized":"2024-09-21T10:05:47.347826+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:37:21.415933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":226,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.336131+0000","last_change":"2024-09-21T09:56:12.576756+0000","last_active":"2024-09-21T10:05:47.336131+0000","last_peered":"2024-09-21T10:05:47.336131+0000","last_clean":"2024-09-21T10:05:47.336131+0000","last_became_active":"2024-09-21T09:56:12.575752+0000","last_became_peered":"2024-09-21T09:56:12.575752+0000","last_unstale":"2024-09-21T10:05:47.336131+0000","last_undegraded":"2024-09-21T10:05:47.336131+0000","last_fullsized":"2024-09-21T10:05:47.336131+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:55.224839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":163,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.352055+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T10:05:44.352055+0000","last_peered":"2024-09-21T10:05:44.352055+0000","last_clean":"2024-09-21T10:05:44.352055+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T10:05:44.352055+0000","last_undegraded":"2024-09-21T10:05:44.352055+0000","last_fullsized":"2024-09-21T10:05:44.352055+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":244,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.336513+0000","last_change":"2024-09-21T09:56:12.571495+0000","last_active":"2024-09-21T10:05:47.336513+0000","last_peered":"2024-09-21T10:05:47.336513+0000","last_clean":"2024-09-21T10:05:47.336513+0000","last_became_active":"2024-09-21T09:56:12.571200+0000","last_became_peered":"2024-09-21T09:56:12.571200+0000","last_unstale":"2024-09-21T10:05:47.336513+0000","last_undegraded":"2024-09-21T10:05:47.336513+0000","last_fullsized":"2024-09-21T10:05:47.336513+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:53:58.851300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":302,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.347934+0000","last_change":"2024-09-21T10:03:38.380567+0000","last_active":"2024-09-21T10:05:47.347934+0000","last_peered":"2024-09-21T10:05:47.347934+0000","last_clean":"2024-09-21T10:05:47.347934+0000","last_became_active":"2024-09-21T10:03:38.378538+0000","last_became_peered":"2024-09-21T10:03:38.378538+0000","last_unstale":"2024-09-21T10:05:47.347934+0000","last_undegraded":"2024-09-21T10:05:47.347934+0000","last_fullsized":"2024-09-21T10:05:47.347934+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:16:25.373217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":246,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.340482+0000","last_change":"2024-09-21T09:56:12.580276+0000","last_active":"2024-09-21T10:05:47.340482+0000","last_peered":"2024-09-21T10:05:47.340482+0000","last_clean":"2024-09-21T10:05:47.340482+0000","last_became_active":"2024-09-21T09:56:12.579989+0000","last_became_peered":"2024-09-21T09:56:12.579989+0000","last_unstale":"2024-09-21T10:05:47.340482+0000","last_undegraded":"2024-09-21T10:05:47.340482+0000","last_fullsized":"2024-09-21T10:05:47.340482+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:14:56.361873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":225,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.337366+0000","last_change":"2024-09-21T09:56:12.582383+0000","last_active":"2024-09-21T10:05:47.337366+0000","last_peered":"2024-09-21T10:05:47.337366+0000","last_clean":"2024-09-21T10:05:47.337366+0000","last_became_active":"2024-09-21T09:56:12.581427+0000","last_became_peered":"2024-09-21T09:56:12.581427+0000","last_unstale":"2024-09-21T10:05:47.337366+0000","last_undegraded":"2024-09-21T10:05:47.337366+0000","last_fullsized":"2024-09-21T10:05:47.337366+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:24.275633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":182,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297245+0000","last_change":"2024-09-21T09:51:30.645644+0000","last_active":"2024-09-21T10:05:44.297245+0000","last_peered":"2024-09-21T10:05:44.297245+0000","last_clean":"2024-09-21T10:05:44.297245+0000","last_became_active":"2024-09-21T09:51:30.644883+0000","last_became_peered":"2024-09-21T09:51:30.644883+0000","last_unstale":"2024-09-21T10:05:44.297245+0000","last_undegraded":"2024-09-21T10:05:44.297245+0000","last_fullsized":"2024-09-21T10:05:44.297245+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:39.331936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":279,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.333413+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:05:47.333413+0000","last_peered":"2024-09-21T10:05:47.333413+0000","last_clean":"2024-09-21T10:05:47.333413+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:05:47.333413+0000","last_undegraded":"2024-09-21T10:05:47.333413+0000","last_fullsized":"2024-09-21T10:05:47.333413+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":292,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.346064+0000","last_change":"2024-09-21T10:03:38.407960+0000","last_active":"2024-09-21T10:05:47.346064+0000","last_peered":"2024-09-21T10:05:47.346064+0000","last_clean":"2024-09-21T10:05:47.346064+0000","last_became_active":"2024-09-21T10:03:38.407632+0000","last_became_peered":"2024-09-21T10:03:38.407632+0000","last_unstale":"2024-09-21T10:05:47.346064+0000","last_undegraded":"2024-09-21T10:05:47.346064+0000","last_fullsized":"2024-09-21T10:05:47.346064+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:51:23.967273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":180,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296996+0000","last_change":"2024-09-21T09:56:12.576148+0000","last_active":"2024-09-21T10:05:44.296996+0000","last_peered":"2024-09-21T10:05:44.296996+0000","last_clean":"2024-09-21T10:05:44.296996+0000","last_became_active":"2024-09-21T09:56:12.575813+0000","last_became_peered":"2024-09-21T09:56:12.575813+0000","last_unstale":"2024-09-21T10:05:44.296996+0000","last_undegraded":"2024-09-21T10:05:44.296996+0000","last_fullsized":"2024-09-21T10:05:44.296996+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:53:01.367816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":167,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.355745+0000","last_change":"2024-09-21T09:50:45.502447+0000","last_active":"2024-09-21T10:05:44.355745+0000","last_peered":"2024-09-21T10:05:44.355745+0000","last_clean":"2024-09-21T10:05:44.355745+0000","last_became_active":"2024-09-21T09:50:45.500800+0000","last_became_peered":"2024-09-21T09:50:45.500800+0000","last_unstale":"2024-09-21T10:05:44.355745+0000","last_undegraded":"2024-09-21T10:05:44.355745+0000","last_fullsized":"2024-09-21T10:05:44.355745+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:20:51.819410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":180,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297163+0000","last_change":"2024-09-21T09:56:12.576457+0000","last_active":"2024-09-21T10:05:44.297163+0000","last_peered":"2024-09-21T10:05:44.297163+0000","last_clean":"2024-09-21T10:05:44.297163+0000","last_became_active":"2024-09-21T09:56:12.576061+0000","last_became_peered":"2024-09-21T09:56:12.576061+0000","last_unstale":"2024-09-21T10:05:44.297163+0000","last_undegraded":"2024-09-21T10:05:44.297163+0000","last_fullsized":"2024-09-21T10:05:44.297163+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:51:46.676699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":150,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.353721+0000","last_change":"2024-09-21T09:40:49.094916+0000","last_active":"2024-09-21T10:05:44.353721+0000","last_peered":"2024-09-21T10:05:44.353721+0000","last_clean":"2024-09-21T10:05:44.353721+0000","last_became_active":"2024-09-21T09:40:49.094264+0000","last_became_peered":"2024-09-21T09:40:49.094264+0000","last_unstale":"2024-09-21T10:05:44.353721+0000","last_undegraded":"2024-09-21T10:05:44.353721+0000","last_fullsized":"2024-09-21T10:05:44.353721+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:53.663226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":252,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.337238+0000","last_change":"2024-09-21T09:56:12.577329+0000","last_active":"2024-09-21T10:05:47.337238+0000","last_peered":"2024-09-21T10:05:47.337238+0000","last_clean":"2024-09-21T10:05:47.337238+0000","last_became_active":"2024-09-21T09:56:12.576941+0000","last_became_peered":"2024-09-21T09:56:12.576941+0000","last_unstale":"2024-09-21T10:05:47.337238+0000","last_undegraded":"2024-09-21T10:05:47.337238+0000","last_fullsized":"2024-09-21T10:05:47.337238+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:35.235768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":237,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.337816+0000","last_change":"2024-09-21T09:56:12.577137+0000","last_active":"2024-09-21T10:05:47.337816+0000","last_peered":"2024-09-21T10:05:47.337816+0000","last_clean":"2024-09-21T10:05:47.337816+0000","last_became_active":"2024-09-21T09:56:12.576668+0000","last_became_peered":"2024-09-21T09:56:12.576668+0000","last_unstale":"2024-09-21T10:05:47.337816+0000","last_undegraded":"2024-09-21T10:05:47.337816+0000","last_fullsized":"2024-09-21T10:05:47.337816+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:56.772427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":163,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296384+0000","last_change":"2024-09-21T09:50:45.502819+0000","last_active":"2024-09-21T10:05:44.296384+0000","last_peered":"2024-09-21T10:05:44.296384+0000","last_clean":"2024-09-21T10:05:44.296384+0000","last_became_active":"2024-09-21T09:50:45.502483+0000","last_became_peered":"2024-09-21T09:50:45.502483+0000","last_unstale":"2024-09-21T10:05:44.296384+0000","last_undegraded":"2024-09-21T10:05:44.296384+0000","last_fullsized":"2024-09-21T10:05:44.296384+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:26.549958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":280,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.347528+0000","last_change":"2024-09-21T09:51:30.650116+0000","last_active":"2024-09-21T10:05:47.347528+0000","last_peered":"2024-09-21T10:05:47.347528+0000","last_clean":"2024-09-21T10:05:47.347528+0000","last_became_active":"2024-09-21T09:51:30.649798+0000","last_became_peered":"2024-09-21T09:51:30.649798+0000","last_unstale":"2024-09-21T10:05:47.347528+0000","last_undegraded":"2024-09-21T10:05:47.347528+0000","last_fullsized":"2024-09-21T10:05:47.347528+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:40.222590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5164171264,"num_objects":78799,"num_object_clones":0,"num_object_copies":236397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78799,"num_write_kb":5043136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15452864512,"data_stored":15452864512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78799,"ondisk_log_size":78799,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":9648,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23724,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626762,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2826912,"kb_used_data":2232856,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":90918240,"statfs":{"total":95995035648,"available":93100277760,"internally_reserved":0,"allocated":2286444544,"data_stored":2285376924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019220,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3404772,"kb_used_data":2509660,"kb_used_omap":1,"kb_used_meta":895102,"kb_avail":90340380,"statfs":{"total":95995035648,"available":92508549120,"internally_reserved":0,"allocated":2569891840,"data_stored":2568820173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916584792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":81,"seq":347892351182,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3808340,"kb_used_data":3138572,"kb_used_omap":5,"kb_used_meta":669754,"kb_avail":89936812,"statfs":{"total":95995035648,"available":92095295488,"internally_reserved":0,"allocated":3213897728,"data_stored":3212838183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":685828156},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":175,"seq":751619276828,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1576168,"kb_used_data":1241568,"kb_used_omap":12,"kb_used_meta":334579,"kb_avail":92168984,"statfs":{"total":95995035648,"available":94381039616,"internally_reserved":0,"allocated":1271365632,"data_stored":1270289955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12845,"internal_metadata":342609363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822991,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2503444,"kb_used_data":1594188,"kb_used_omap":1,"kb_used_meta":909246,"kb_avail":91241708,"statfs":{"total":95995035648,"available":93431508992,"internally_reserved":0,"allocated":1632448512,"data_stored":1631388920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931068248},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879414,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2048916,"kb_used_data":1546188,"kb_used_omap":4,"kb_used_meta":502715,"kb_avail":91696236,"statfs":{"total":95995035648,"available":93896945664,"internally_reserved":0,"allocated":1583296512,"data_stored":1582237001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":514780604},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566254,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2246104,"kb_used_data":1816464,"kb_used_omap":6,"kb_used_meta":429625,"kb_avail":91499048,"statfs":{"total":95995035648,"available":93695025152,"internally_reserved":0,"allocated":1860059136,"data_stored":1858996051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":439936423},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860599,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1547664,"kb_used_data":928900,"kb_used_omap":5,"kb_used_meta":618746,"kb_avail":92197488,"statfs":{"total":95995035648,"available":94410227712,"internally_reserved":0,"allocated":951193600,"data_stored":950141343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":633596143},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9976,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3225485312,"data_stored":3225485312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":956760064,"data_stored":956760064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1866137600,"data_stored":1866137600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1599340544,"data_stored":1599340544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2600009728,"data_stored":2600009728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1645215744,"data_stored":1645215744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2285109248,"data_stored":2285109248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1274806272,"data_stored":1274806272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:05:56.183 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:05:56.188 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:56.249 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:56.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:56.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:56.351 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:56.413 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:56.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:56.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:56.481 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:56 smithi107 ceph-mon[102221]: pgmap v712: 109 pgs: 109 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-21T10:05:56.481 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:05:56.481 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:56.481 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:56 smithi107 ceph-mon[105022]: pgmap v712: 109 pgs: 109 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-21T10:05:56.481 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:05:56.481 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:56.515 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:56.536 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:56.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:56.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:56.617 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:56 smithi129 ceph-mon[88471]: pgmap v712: 109 pgs: 109 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-21T10:05:56.617 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:05:56.617 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:56.638 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:56.656 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 47 16 83828 83812 111.439 43.4375 0.0120644 0.00896814 2024-09-21T10:05:56.681 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:56.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:56.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:56.773 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:05:56.774 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:05:56.783 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:56.785 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:56.817 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:56.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:56.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:56.918 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:56.953 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:56.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:56.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:57.055 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:57.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:57.092 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:57.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:57.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:57.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:57.236 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:57.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:57.337 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:57.387 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:57.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:57.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:57.488 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:57.523 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:57.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:57.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:57.625 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:57.656 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 48 16 85045 85029 110.702 76.0625 0.0253796 0.00902819 2024-09-21T10:05:57.691 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:57.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:57.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:57.785 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:05:57.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:57.852 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:05:57.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:57.868 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:57.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:57.969 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:58.004 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:58.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:58.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:58.106 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:58.121 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:58.137 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:58.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:58.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:58.239 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:05:58.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:58.278 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:58.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:05:58.379 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:05:58.409 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:58.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:58.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:05:58.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:58.543 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:58.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:58.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:58.600 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:05:58 smithi107 ceph-mon[102221]: pgmap v713: 109 pgs: 109 active+clean; 4.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-21T10:05:58.600 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:05:58 smithi107 ceph-mon[105022]: pgmap v713: 109 pgs: 109 active+clean; 4.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-21T10:05:58.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:05:58.656 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 49 16 86179 86163 109.889 70.875 0.0161963 0.0090957 2024-09-21T10:05:58.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:05:58 smithi129 ceph-mon[88471]: pgmap v713: 109 pgs: 109 active+clean; 4.9 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 75 MiB/s wr, 1.21k op/s 2024-09-21T10:05:58.682 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:58.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:58.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:05:58.784 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:58.818 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:58.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:58.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:58.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:58.947 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:05:58.952 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:58.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:58.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:59.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:05:59.089 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:59.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:59.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:05:59.191 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:59.224 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:59.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:59.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:59.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:05:59.356 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:05:59.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:59.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:05:59.458 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:59.492 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:59.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:59.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:05:59.594 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:05:59.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:59.642 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:59.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:05:59.657 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 50 16 87394 87378 109.21 75.9375 0.00953734 0.00915169 2024-09-21T10:05:59.743 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:05:59.785 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:59.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:59.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:05:59.887 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:05:59.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:05:59.920 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:05:59.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:00.021 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:00.048 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:00.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:00.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:00.148 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:00.150 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:00.181 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:00.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:00.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:00.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:00.345 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:00.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:00.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:00.447 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:00.449 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:00 smithi107 ceph-mon[102221]: pgmap v714: 109 pgs: 109 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-21T10:06:00.449 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:00 smithi107 ceph-mon[105022]: pgmap v714: 109 pgs: 109 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-21T10:06:00.473 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:00.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:00.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:00.542 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:00 smithi129 ceph-mon[88471]: pgmap v714: 109 pgs: 109 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-21T10:06:00.542 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:06:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:06:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:06:00.575 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:00.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:00.628 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:00.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:00.657 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 51 16 88508 88492 108.434 69.625 0.0107904 0.00921733 2024-09-21T10:06:00.729 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:00.768 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:00.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:00.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:00.870 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:00.908 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:00.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:00.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:01.010 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:01.059 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:01.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:01.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:01.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:01.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:01.199 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:01.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:01.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:01.324 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:06:01.325 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:06:01.336 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:01.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:01.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:01.386 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:06:01.438 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:01.480 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:01.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:01.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:01.582 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:01.591 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:06:01.591 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:06:01.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:01 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3232523085' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:06:01.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:01 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3232523085' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:06:01.620 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:01.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:01.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:01.657 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 52 16 89659 89643 107.732 71.9375 0.0160706 0.00927775 2024-09-21T10:06:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:01 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3232523085' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:06:01.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:01.763 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:01.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:01.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:01.865 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:01.899 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:01.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:01.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:01.937 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:06:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:06:01] "GET /metrics HTTP/1.1" 200 36695 "" "Prometheus/2.51.0" 2024-09-21T10:06:02.000 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:02.039 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:02.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:02.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:02.141 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:02.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:02.177 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:02.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:02.278 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:02.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:02.314 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:02.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:02.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:06:02.449 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:02.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:02.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:06:02.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:02.579 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:02.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:02.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:02.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:02 smithi107 ceph-mon[105022]: pgmap v715: 109 pgs: 109 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:06:02.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:02 smithi107 ceph-mon[105022]: from='client.36264 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:06:02.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:02 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:02.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:02 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3232523085' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:06:02.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:02 smithi107 ceph-mon[105022]: osdmap e201: 8 total, 8 up, 8 in 2024-09-21T10:06:02.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:02 smithi107 ceph-mon[102221]: pgmap v715: 109 pgs: 109 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:06:02.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:02 smithi107 ceph-mon[102221]: from='client.36264 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:06:02.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:02 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:02.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:02 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3232523085' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:06:02.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:02 smithi107 ceph-mon[102221]: osdmap e201: 8 total, 8 up, 8 in 2024-09-21T10:06:02.657 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 53 16 90910 90894 107.174 78.1875 0.00823392 0.00932589 2024-09-21T10:06:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:02 smithi129 ceph-mon[88471]: pgmap v715: 109 pgs: 109 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:06:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:02 smithi129 ceph-mon[88471]: from='client.36264 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:06:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:02 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:02 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3232523085' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:06:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:02 smithi129 ceph-mon[88471]: osdmap e201: 8 total, 8 up, 8 in 2024-09-21T10:06:02.680 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:02.711 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:02.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:02.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:02.813 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:02.842 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:02.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:02.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:02.906 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":715,"stamp":"2024-09-21T10:06:01.151439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5579937741,"num_objects":85160,"num_object_clones":0,"num_object_copies":255480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85198,"num_write_kb":5448608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":9648,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87293,"ondisk_log_size":87293,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21311308,"kb_used_data":16337932,"kb_used_omap":40,"kb_used_meta":4973271,"kb_avail":728649908,"statfs":{"total":767960285184,"available":746137505792,"internally_reserved":0,"allocated":16730042368,"data_stored":16721533414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41367,"internal_metadata":5092630121},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":804847616,"num_objects":12281,"num_object_clones":0,"num_object_copies":36843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12281,"num_write_kb":785984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003412"},"pg_stats":[{"pgid":"1.11","version":"77'100","reported_seq":270,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.344598+0000","last_change":"2024-09-21T10:04:42.586770+0000","last_active":"2024-09-21T10:05:47.344598+0000","last_peered":"2024-09-21T10:05:47.344598+0000","last_clean":"2024-09-21T10:05:47.344598+0000","last_became_active":"2024-09-21T10:04:42.544825+0000","last_became_peered":"2024-09-21T10:04:42.544825+0000","last_unstale":"2024-09-21T10:05:47.344598+0000","last_undegraded":"2024-09-21T10:05:47.344598+0000","last_fullsized":"2024-09-21T10:05:47.344598+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:42.586723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:42.586723+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:51:31.536325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":251,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.336729+0000","last_change":"2024-09-21T10:03:38.389577+0000","last_active":"2024-09-21T10:05:47.336729+0000","last_peered":"2024-09-21T10:05:47.336729+0000","last_clean":"2024-09-21T10:05:47.336729+0000","last_became_active":"2024-09-21T10:03:38.389292+0000","last_became_peered":"2024-09-21T10:03:38.389292+0000","last_unstale":"2024-09-21T10:05:47.336729+0000","last_undegraded":"2024-09-21T10:05:47.336729+0000","last_fullsized":"2024-09-21T10:05:47.336729+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:32.113472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":54,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296767+0000","last_change":"2024-09-21T10:03:38.404072+0000","last_active":"2024-09-21T10:05:44.296767+0000","last_peered":"2024-09-21T10:05:44.296767+0000","last_clean":"2024-09-21T10:05:44.296767+0000","last_became_active":"2024-09-21T10:03:38.403652+0000","last_became_peered":"2024-09-21T10:03:38.403652+0000","last_unstale":"2024-09-21T10:05:44.296767+0000","last_undegraded":"2024-09-21T10:05:44.296767+0000","last_fullsized":"2024-09-21T10:05:44.296767+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:13:33.708105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'100","reported_seq":31,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.353083+0000","last_change":"2024-09-21T10:04:43.321729+0000","last_active":"2024-09-21T10:05:44.353083+0000","last_peered":"2024-09-21T10:05:44.353083+0000","last_clean":"2024-09-21T10:05:44.353083+0000","last_became_active":"2024-09-21T10:04:42.546211+0000","last_became_peered":"2024-09-21T10:04:42.546211+0000","last_unstale":"2024-09-21T10:05:44.353083+0000","last_undegraded":"2024-09-21T10:05:44.353083+0000","last_fullsized":"2024-09-21T10:05:44.353083+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.321671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.321671+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:40:08.766043+0000","scrub_duration":0.0012263879999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":201,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.353652+0000","last_change":"2024-09-21T10:03:38.372672+0000","last_active":"2024-09-21T10:05:44.353652+0000","last_peered":"2024-09-21T10:05:44.353652+0000","last_clean":"2024-09-21T10:05:44.353652+0000","last_became_active":"2024-09-21T10:03:38.372223+0000","last_became_peered":"2024-09-21T10:03:38.372223+0000","last_unstale":"2024-09-21T10:05:44.353652+0000","last_undegraded":"2024-09-21T10:05:44.353652+0000","last_fullsized":"2024-09-21T10:05:44.353652+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:05:25.460564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":300,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.346472+0000","last_change":"2024-09-21T10:03:38.378755+0000","last_active":"2024-09-21T10:05:47.346472+0000","last_peered":"2024-09-21T10:05:47.346472+0000","last_clean":"2024-09-21T10:05:47.346472+0000","last_became_active":"2024-09-21T10:03:38.378338+0000","last_became_peered":"2024-09-21T10:03:38.378338+0000","last_unstale":"2024-09-21T10:05:47.346472+0000","last_undegraded":"2024-09-21T10:05:47.346472+0000","last_fullsized":"2024-09-21T10:05:47.346472+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:17:12.051549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'100","reported_seq":42,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.341775+0000","last_change":"2024-09-21T10:04:50.433274+0000","last_active":"2024-09-21T10:05:47.341775+0000","last_peered":"2024-09-21T10:05:47.341775+0000","last_clean":"2024-09-21T10:05:47.341775+0000","last_became_active":"2024-09-21T10:04:42.545783+0000","last_became_peered":"2024-09-21T10:04:42.545783+0000","last_unstale":"2024-09-21T10:05:47.341775+0000","last_undegraded":"2024-09-21T10:05:47.341775+0000","last_fullsized":"2024-09-21T10:05:47.341775+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.433227+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:10:59.708360+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":301,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.348016+0000","last_change":"2024-09-21T10:03:38.379045+0000","last_active":"2024-09-21T10:05:47.348016+0000","last_peered":"2024-09-21T10:05:47.348016+0000","last_clean":"2024-09-21T10:05:47.348016+0000","last_became_active":"2024-09-21T10:03:38.377507+0000","last_became_peered":"2024-09-21T10:03:38.377507+0000","last_unstale":"2024-09-21T10:05:47.348016+0000","last_undegraded":"2024-09-21T10:05:47.348016+0000","last_fullsized":"2024-09-21T10:05:47.348016+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:07:26.215863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":48,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296677+0000","last_change":"2024-09-21T10:03:38.410388+0000","last_active":"2024-09-21T10:05:44.296677+0000","last_peered":"2024-09-21T10:05:44.296677+0000","last_clean":"2024-09-21T10:05:44.296677+0000","last_became_active":"2024-09-21T10:03:38.402465+0000","last_became_peered":"2024-09-21T10:03:38.402465+0000","last_unstale":"2024-09-21T10:05:44.296677+0000","last_undegraded":"2024-09-21T10:05:44.296677+0000","last_fullsized":"2024-09-21T10:05:44.296677+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:54:49.777097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"77'100","reported_seq":31,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.353191+0000","last_change":"2024-09-21T10:04:45.295392+0000","last_active":"2024-09-21T10:05:44.353191+0000","last_peered":"2024-09-21T10:05:44.353191+0000","last_clean":"2024-09-21T10:05:44.353191+0000","last_became_active":"2024-09-21T10:04:42.545479+0000","last_became_peered":"2024-09-21T10:04:42.545479+0000","last_unstale":"2024-09-21T10:05:44.353191+0000","last_undegraded":"2024-09-21T10:05:44.353191+0000","last_fullsized":"2024-09-21T10:05:44.353191+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.295331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.295331+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:20.770065+0000","scrub_duration":0.0016030269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":251,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.339229+0000","last_change":"2024-09-21T10:03:38.381552+0000","last_active":"2024-09-21T10:05:47.339229+0000","last_peered":"2024-09-21T10:05:47.339229+0000","last_clean":"2024-09-21T10:05:47.339229+0000","last_became_active":"2024-09-21T10:03:38.380898+0000","last_became_peered":"2024-09-21T10:03:38.380898+0000","last_unstale":"2024-09-21T10:05:47.339229+0000","last_undegraded":"2024-09-21T10:05:47.339229+0000","last_fullsized":"2024-09-21T10:05:47.339229+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:53.409606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":172,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.350473+0000","last_change":"2024-09-21T09:48:54.140948+0000","last_active":"2024-09-21T10:05:44.350473+0000","last_peered":"2024-09-21T10:05:44.350473+0000","last_clean":"2024-09-21T10:05:44.350473+0000","last_became_active":"2024-09-21T09:48:54.140641+0000","last_became_peered":"2024-09-21T09:48:54.140641+0000","last_unstale":"2024-09-21T10:05:44.350473+0000","last_undegraded":"2024-09-21T10:05:44.350473+0000","last_fullsized":"2024-09-21T10:05:44.350473+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:25.157599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"200'5222","reported_seq":5240,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:57.786449+0000","last_change":"2024-09-21T10:04:55.601105+0000","last_active":"2024-09-21T10:05:57.786449+0000","last_peered":"2024-09-21T10:05:57.786449+0000","last_clean":"2024-09-21T10:05:57.786449+0000","last_became_active":"2024-09-21T10:04:55.600833+0000","last_became_peered":"2024-09-21T10:04:55.600833+0000","last_unstale":"2024-09-21T10:05:57.786449+0000","last_undegraded":"2024-09-21T10:05:57.786449+0000","last_fullsized":"2024-09-21T10:05:57.786449+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":5222,"log_dups_size":0,"ondisk_log_size":5222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:45.109273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342228992,"num_objects":5222,"num_object_clones":0,"num_object_copies":15666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5222,"num_write_kb":334208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"77'100","reported_seq":74,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297150+0000","last_change":"2024-09-21T10:04:55.259467+0000","last_active":"2024-09-21T10:05:44.297150+0000","last_peered":"2024-09-21T10:05:44.297150+0000","last_clean":"2024-09-21T10:05:44.297150+0000","last_became_active":"2024-09-21T10:04:42.544426+0000","last_became_peered":"2024-09-21T10:04:42.544426+0000","last_unstale":"2024-09-21T10:05:44.297150+0000","last_undegraded":"2024-09-21T10:05:44.297150+0000","last_fullsized":"2024-09-21T10:05:44.297150+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:55.259385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:55.259385+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:50.367631+0000","scrub_duration":0.0015667260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":52,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296571+0000","last_change":"2024-09-21T10:03:38.399320+0000","last_active":"2024-09-21T10:05:44.296571+0000","last_peered":"2024-09-21T10:05:44.296571+0000","last_clean":"2024-09-21T10:05:44.296571+0000","last_became_active":"2024-09-21T10:03:38.383894+0000","last_became_peered":"2024-09-21T10:03:38.383894+0000","last_unstale":"2024-09-21T10:05:44.296571+0000","last_undegraded":"2024-09-21T10:05:44.296571+0000","last_fullsized":"2024-09-21T10:05:44.296571+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:09:55.613727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":184,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.353170+0000","last_change":"2024-09-21T10:03:38.408828+0000","last_active":"2024-09-21T10:05:44.353170+0000","last_peered":"2024-09-21T10:05:44.353170+0000","last_clean":"2024-09-21T10:05:44.353170+0000","last_became_active":"2024-09-21T10:03:38.408327+0000","last_became_peered":"2024-09-21T10:03:38.408327+0000","last_unstale":"2024-09-21T10:05:44.353170+0000","last_undegraded":"2024-09-21T10:05:44.353170+0000","last_fullsized":"2024-09-21T10:05:44.353170+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:43:26.872372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"200'5288","reported_seq":5306,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:57.309147+0000","last_change":"2024-09-21T10:04:55.802720+0000","last_active":"2024-09-21T10:05:57.309147+0000","last_peered":"2024-09-21T10:05:57.309147+0000","last_clean":"2024-09-21T10:05:57.309147+0000","last_became_active":"2024-09-21T10:04:55.802349+0000","last_became_peered":"2024-09-21T10:04:55.802349+0000","last_unstale":"2024-09-21T10:05:57.309147+0000","last_undegraded":"2024-09-21T10:05:57.309147+0000","last_fullsized":"2024-09-21T10:05:57.309147+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":5288,"log_dups_size":0,"ondisk_log_size":5288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:00:13.309694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346554368,"num_objects":5288,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5288,"num_write_kb":338432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"77'100","reported_seq":46,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.336552+0000","last_change":"2024-09-21T10:04:45.981392+0000","last_active":"2024-09-21T10:05:47.336552+0000","last_peered":"2024-09-21T10:05:47.336552+0000","last_clean":"2024-09-21T10:05:47.336552+0000","last_became_active":"2024-09-21T10:04:42.547751+0000","last_became_peered":"2024-09-21T10:04:42.547751+0000","last_unstale":"2024-09-21T10:05:47.336552+0000","last_undegraded":"2024-09-21T10:05:47.336552+0000","last_fullsized":"2024-09-21T10:05:47.336552+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.981336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.981336+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:26:26.732033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":52,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297397+0000","last_change":"2024-09-21T10:03:38.395125+0000","last_active":"2024-09-21T10:05:44.297397+0000","last_peered":"2024-09-21T10:05:44.297397+0000","last_clean":"2024-09-21T10:05:44.297397+0000","last_became_active":"2024-09-21T10:03:38.391916+0000","last_became_peered":"2024-09-21T10:03:38.391916+0000","last_unstale":"2024-09-21T10:05:44.297397+0000","last_undegraded":"2024-09-21T10:05:44.297397+0000","last_fullsized":"2024-09-21T10:05:44.297397+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:15.723022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":180,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296520+0000","last_change":"2024-09-21T09:56:12.576294+0000","last_active":"2024-09-21T10:05:44.296520+0000","last_peered":"2024-09-21T10:05:44.296520+0000","last_clean":"2024-09-21T10:05:44.296520+0000","last_became_active":"2024-09-21T09:56:12.575928+0000","last_became_peered":"2024-09-21T09:56:12.575928+0000","last_unstale":"2024-09-21T10:05:44.296520+0000","last_undegraded":"2024-09-21T10:05:44.296520+0000","last_fullsized":"2024-09-21T10:05:44.296520+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:23.227435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"200'5339","reported_seq":5369,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:56.946952+0000","last_change":"2024-09-21T10:04:55.801704+0000","last_active":"2024-09-21T10:05:56.946952+0000","last_peered":"2024-09-21T10:05:56.946952+0000","last_clean":"2024-09-21T10:05:56.946952+0000","last_became_active":"2024-09-21T10:04:55.801386+0000","last_became_peered":"2024-09-21T10:04:55.801386+0000","last_unstale":"2024-09-21T10:05:56.946952+0000","last_undegraded":"2024-09-21T10:05:56.946952+0000","last_fullsized":"2024-09-21T10:05:56.946952+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":5339,"log_dups_size":0,"ondisk_log_size":5339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:58.954309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349896704,"num_objects":5339,"num_object_clones":0,"num_object_copies":16017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5339,"num_write_kb":341696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"77'100","reported_seq":174,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.352021+0000","last_change":"2024-09-21T10:04:43.289920+0000","last_active":"2024-09-21T10:05:44.352021+0000","last_peered":"2024-09-21T10:05:44.352021+0000","last_clean":"2024-09-21T10:05:44.352021+0000","last_became_active":"2024-09-21T10:04:42.549133+0000","last_became_peered":"2024-09-21T10:04:42.549133+0000","last_unstale":"2024-09-21T10:05:44.352021+0000","last_undegraded":"2024-09-21T10:05:44.352021+0000","last_fullsized":"2024-09-21T10:05:44.352021+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.289859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.289859+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:58.310069+0000","scrub_duration":0.001251809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":52,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296957+0000","last_change":"2024-09-21T10:03:38.401018+0000","last_active":"2024-09-21T10:05:44.296957+0000","last_peered":"2024-09-21T10:05:44.296957+0000","last_clean":"2024-09-21T10:05:44.296957+0000","last_became_active":"2024-09-21T10:03:38.395207+0000","last_became_peered":"2024-09-21T10:03:38.395207+0000","last_unstale":"2024-09-21T10:05:44.296957+0000","last_undegraded":"2024-09-21T10:05:44.296957+0000","last_fullsized":"2024-09-21T10:05:44.296957+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:06.649204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":200,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.351692+0000","last_change":"2024-09-21T10:03:38.378384+0000","last_active":"2024-09-21T10:05:44.351692+0000","last_peered":"2024-09-21T10:05:44.351692+0000","last_clean":"2024-09-21T10:05:44.351692+0000","last_became_active":"2024-09-21T10:03:38.378072+0000","last_became_peered":"2024-09-21T10:03:38.378072+0000","last_unstale":"2024-09-21T10:05:44.351692+0000","last_undegraded":"2024-09-21T10:05:44.351692+0000","last_fullsized":"2024-09-21T10:05:44.351692+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:54:37.039622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"200'5330","reported_seq":5361,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:59.604192+0000","last_change":"2024-09-21T10:04:55.800926+0000","last_active":"2024-09-21T10:05:59.604192+0000","last_peered":"2024-09-21T10:05:59.604192+0000","last_clean":"2024-09-21T10:05:59.604192+0000","last_became_active":"2024-09-21T10:04:55.800384+0000","last_became_peered":"2024-09-21T10:04:55.800384+0000","last_unstale":"2024-09-21T10:05:59.604192+0000","last_undegraded":"2024-09-21T10:05:59.604192+0000","last_fullsized":"2024-09-21T10:05:59.604192+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":5330,"log_dups_size":0,"ondisk_log_size":5330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:06:54.058899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349306880,"num_objects":5330,"num_object_clones":0,"num_object_copies":15990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5330,"num_write_kb":341120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"77'100","reported_seq":31,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296852+0000","last_change":"2024-09-21T10:04:48.387150+0000","last_active":"2024-09-21T10:05:44.296852+0000","last_peered":"2024-09-21T10:05:44.296852+0000","last_clean":"2024-09-21T10:05:44.296852+0000","last_became_active":"2024-09-21T10:04:42.546121+0000","last_became_peered":"2024-09-21T10:04:42.546121+0000","last_unstale":"2024-09-21T10:05:44.296852+0000","last_undegraded":"2024-09-21T10:05:44.296852+0000","last_fullsized":"2024-09-21T10:05:44.296852+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:48.387088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:48.387088+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:29:48.630550+0000","scrub_duration":0.001663997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":165,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.355822+0000","last_change":"2024-09-21T09:56:12.590219+0000","last_active":"2024-09-21T10:05:44.355822+0000","last_peered":"2024-09-21T10:05:44.355822+0000","last_clean":"2024-09-21T10:05:44.355822+0000","last_became_active":"2024-09-21T09:56:12.589946+0000","last_became_peered":"2024-09-21T09:56:12.589946+0000","last_unstale":"2024-09-21T10:05:44.355822+0000","last_undegraded":"2024-09-21T10:05:44.355822+0000","last_fullsized":"2024-09-21T10:05:44.355822+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:48.919929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":276,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.345900+0000","last_change":"2024-09-21T09:50:45.503336+0000","last_active":"2024-09-21T10:05:47.345900+0000","last_peered":"2024-09-21T10:05:47.345900+0000","last_clean":"2024-09-21T10:05:47.345900+0000","last_became_active":"2024-09-21T09:50:45.503045+0000","last_became_peered":"2024-09-21T09:50:45.503045+0000","last_unstale":"2024-09-21T10:05:47.345900+0000","last_undegraded":"2024-09-21T10:05:47.345900+0000","last_fullsized":"2024-09-21T10:05:47.345900+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:58:46.728347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"200'5587","reported_seq":5617,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:06:00.031735+0000","last_change":"2024-09-21T10:04:55.603614+0000","last_active":"2024-09-21T10:06:00.031735+0000","last_peered":"2024-09-21T10:06:00.031735+0000","last_clean":"2024-09-21T10:06:00.031735+0000","last_became_active":"2024-09-21T10:04:55.603210+0000","last_became_peered":"2024-09-21T10:04:55.603210+0000","last_unstale":"2024-09-21T10:06:00.031735+0000","last_undegraded":"2024-09-21T10:06:00.031735+0000","last_fullsized":"2024-09-21T10:06:00.031735+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":5587,"log_dups_size":0,"ondisk_log_size":5587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:13:26.372076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366149632,"num_objects":5587,"num_object_clones":0,"num_object_copies":16761,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5587,"num_write_kb":357568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":74,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296756+0000","last_change":"2024-09-21T10:04:53.355975+0000","last_active":"2024-09-21T10:05:44.296756+0000","last_peered":"2024-09-21T10:05:44.296756+0000","last_clean":"2024-09-21T10:05:44.296756+0000","last_became_active":"2024-09-21T10:04:42.547505+0000","last_became_peered":"2024-09-21T10:04:42.547505+0000","last_unstale":"2024-09-21T10:05:44.296756+0000","last_undegraded":"2024-09-21T10:05:44.296756+0000","last_fullsized":"2024-09-21T10:05:44.296756+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.355912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.355912+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:43.456830+0000","scrub_duration":0.0017980030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":214,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.336498+0000","last_change":"2024-09-21T09:56:12.575897+0000","last_active":"2024-09-21T10:05:47.336498+0000","last_peered":"2024-09-21T10:05:47.336498+0000","last_clean":"2024-09-21T10:05:47.336498+0000","last_became_active":"2024-09-21T09:56:12.575532+0000","last_became_peered":"2024-09-21T09:56:12.575532+0000","last_unstale":"2024-09-21T10:05:47.336498+0000","last_undegraded":"2024-09-21T10:05:47.336498+0000","last_fullsized":"2024-09-21T10:05:47.336498+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:01.302499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":163,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297402+0000","last_change":"2024-09-21T09:56:12.588824+0000","last_active":"2024-09-21T10:05:44.297402+0000","last_peered":"2024-09-21T10:05:44.297402+0000","last_clean":"2024-09-21T10:05:44.297402+0000","last_became_active":"2024-09-21T09:56:12.588561+0000","last_became_peered":"2024-09-21T09:56:12.588561+0000","last_unstale":"2024-09-21T10:05:44.297402+0000","last_undegraded":"2024-09-21T10:05:44.297402+0000","last_fullsized":"2024-09-21T10:05:44.297402+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:31:48.900623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"200'5535","reported_seq":5566,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:59.548714+0000","last_change":"2024-09-21T10:04:55.801441+0000","last_active":"2024-09-21T10:05:59.548714+0000","last_peered":"2024-09-21T10:05:59.548714+0000","last_clean":"2024-09-21T10:05:59.548714+0000","last_became_active":"2024-09-21T10:04:55.801163+0000","last_became_peered":"2024-09-21T10:04:55.801163+0000","last_unstale":"2024-09-21T10:05:59.548714+0000","last_undegraded":"2024-09-21T10:05:59.548714+0000","last_fullsized":"2024-09-21T10:05:59.548714+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":5535,"log_dups_size":0,"ondisk_log_size":5535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:03:10.019956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362741760,"num_objects":5535,"num_object_clones":0,"num_object_copies":16605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5535,"num_write_kb":354240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":38,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.353070+0000","last_change":"2024-09-21T10:04:50.312757+0000","last_active":"2024-09-21T10:05:44.353070+0000","last_peered":"2024-09-21T10:05:44.353070+0000","last_clean":"2024-09-21T10:05:44.353070+0000","last_became_active":"2024-09-21T10:04:42.548966+0000","last_became_peered":"2024-09-21T10:04:42.548966+0000","last_unstale":"2024-09-21T10:05:44.353070+0000","last_undegraded":"2024-09-21T10:05:44.353070+0000","last_fullsized":"2024-09-21T10:05:44.353070+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.312657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.312657+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:22:13.119698+0000","scrub_duration":0.001872255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":164,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.351845+0000","last_change":"2024-09-21T09:50:45.501596+0000","last_active":"2024-09-21T10:05:44.351845+0000","last_peered":"2024-09-21T10:05:44.351845+0000","last_clean":"2024-09-21T10:05:44.351845+0000","last_became_active":"2024-09-21T09:50:45.501186+0000","last_became_peered":"2024-09-21T09:50:45.501186+0000","last_unstale":"2024-09-21T10:05:44.351845+0000","last_undegraded":"2024-09-21T10:05:44.351845+0000","last_fullsized":"2024-09-21T10:05:44.351845+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:43:16.449425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":173,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297755+0000","last_change":"2024-09-21T09:51:30.639545+0000","last_active":"2024-09-21T10:05:44.297755+0000","last_peered":"2024-09-21T10:05:44.297755+0000","last_clean":"2024-09-21T10:05:44.297755+0000","last_became_active":"2024-09-21T09:51:30.639106+0000","last_became_peered":"2024-09-21T09:51:30.639106+0000","last_unstale":"2024-09-21T10:05:44.297755+0000","last_undegraded":"2024-09-21T10:05:44.297755+0000","last_fullsized":"2024-09-21T10:05:44.297755+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:19:19.607327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"200'5317","reported_seq":5335,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:57.274560+0000","last_change":"2024-09-21T10:04:55.604878+0000","last_active":"2024-09-21T10:05:57.274560+0000","last_peered":"2024-09-21T10:05:57.274560+0000","last_clean":"2024-09-21T10:05:57.274560+0000","last_became_active":"2024-09-21T10:04:55.604403+0000","last_became_peered":"2024-09-21T10:04:55.604403+0000","last_unstale":"2024-09-21T10:05:57.274560+0000","last_undegraded":"2024-09-21T10:05:57.274560+0000","last_fullsized":"2024-09-21T10:05:57.274560+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":5317,"log_dups_size":0,"ondisk_log_size":5317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:19.128055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348454912,"num_objects":5317,"num_object_clones":0,"num_object_copies":15951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5317,"num_write_kb":340288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"77'100","reported_seq":42,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.341669+0000","last_change":"2024-09-21T10:04:53.540367+0000","last_active":"2024-09-21T10:05:47.341669+0000","last_peered":"2024-09-21T10:05:47.341669+0000","last_clean":"2024-09-21T10:05:47.341669+0000","last_became_active":"2024-09-21T10:04:42.549016+0000","last_became_peered":"2024-09-21T10:04:42.549016+0000","last_unstale":"2024-09-21T10:05:47.341669+0000","last_undegraded":"2024-09-21T10:05:47.341669+0000","last_fullsized":"2024-09-21T10:05:47.341669+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.540323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.540323+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:30.672583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":201,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296711+0000","last_change":"2024-09-21T10:03:38.411439+0000","last_active":"2024-09-21T10:05:44.296711+0000","last_peered":"2024-09-21T10:05:44.296711+0000","last_clean":"2024-09-21T10:05:44.296711+0000","last_became_active":"2024-09-21T10:03:38.411108+0000","last_became_peered":"2024-09-21T10:03:38.411108+0000","last_unstale":"2024-09-21T10:05:44.296711+0000","last_undegraded":"2024-09-21T10:05:44.296711+0000","last_fullsized":"2024-09-21T10:05:44.296711+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:11:54.932822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":163,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297028+0000","last_change":"2024-09-21T09:50:45.510764+0000","last_active":"2024-09-21T10:05:44.297028+0000","last_peered":"2024-09-21T10:05:44.297028+0000","last_clean":"2024-09-21T10:05:44.297028+0000","last_became_active":"2024-09-21T09:50:45.510226+0000","last_became_peered":"2024-09-21T09:50:45.510226+0000","last_unstale":"2024-09-21T10:05:44.297028+0000","last_undegraded":"2024-09-21T10:05:44.297028+0000","last_fullsized":"2024-09-21T10:05:44.297028+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:31:13.491427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"200'5270","reported_seq":5288,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:57.341083+0000","last_change":"2024-09-21T10:04:55.597985+0000","last_active":"2024-09-21T10:05:57.341083+0000","last_peered":"2024-09-21T10:05:57.341083+0000","last_clean":"2024-09-21T10:05:57.341083+0000","last_became_active":"2024-09-21T10:04:55.597712+0000","last_became_peered":"2024-09-21T10:04:55.597712+0000","last_unstale":"2024-09-21T10:05:57.341083+0000","last_undegraded":"2024-09-21T10:05:57.341083+0000","last_fullsized":"2024-09-21T10:05:57.341083+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:12:18.356027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345374720,"num_objects":5270,"num_object_clones":0,"num_object_copies":15810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5270,"num_write_kb":337280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":31,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.353114+0000","last_change":"2024-09-21T10:04:48.318076+0000","last_active":"2024-09-21T10:05:44.353114+0000","last_peered":"2024-09-21T10:05:44.353114+0000","last_clean":"2024-09-21T10:05:44.353114+0000","last_became_active":"2024-09-21T10:04:42.547894+0000","last_became_peered":"2024-09-21T10:04:42.547894+0000","last_unstale":"2024-09-21T10:05:44.353114+0000","last_undegraded":"2024-09-21T10:05:44.353114+0000","last_fullsized":"2024-09-21T10:05:44.353114+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:48.318010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:48.318010+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:15.411371+0000","scrub_duration":0.001567564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":252,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.336560+0000","last_change":"2024-09-21T10:03:38.386604+0000","last_active":"2024-09-21T10:05:47.336560+0000","last_peered":"2024-09-21T10:05:47.336560+0000","last_clean":"2024-09-21T10:05:47.336560+0000","last_became_active":"2024-09-21T10:03:38.386283+0000","last_became_peered":"2024-09-21T10:03:38.386283+0000","last_unstale":"2024-09-21T10:05:47.336560+0000","last_undegraded":"2024-09-21T10:05:47.336560+0000","last_fullsized":"2024-09-21T10:05:47.336560+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:27:05.944880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":247,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.342317+0000","last_change":"2024-09-21T09:56:12.575396+0000","last_active":"2024-09-21T10:05:47.342317+0000","last_peered":"2024-09-21T10:05:47.342317+0000","last_clean":"2024-09-21T10:05:47.342317+0000","last_became_active":"2024-09-21T09:56:12.574913+0000","last_became_peered":"2024-09-21T09:56:12.574913+0000","last_unstale":"2024-09-21T10:05:47.342317+0000","last_undegraded":"2024-09-21T10:05:47.342317+0000","last_fullsized":"2024-09-21T10:05:47.342317+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:05.462880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"200'5388","reported_seq":5406,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:57.307760+0000","last_change":"2024-09-21T10:04:55.602045+0000","last_active":"2024-09-21T10:05:57.307760+0000","last_peered":"2024-09-21T10:05:57.307760+0000","last_clean":"2024-09-21T10:05:57.307760+0000","last_became_active":"2024-09-21T10:04:55.601582+0000","last_became_peered":"2024-09-21T10:04:55.601582+0000","last_unstale":"2024-09-21T10:05:57.307760+0000","last_undegraded":"2024-09-21T10:05:57.307760+0000","last_fullsized":"2024-09-21T10:05:57.307760+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":5388,"log_dups_size":0,"ondisk_log_size":5388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:55:40.829352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353107968,"num_objects":5388,"num_object_clones":0,"num_object_copies":16164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5388,"num_write_kb":344832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":64,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296568+0000","last_change":"2024-09-21T10:04:47.393223+0000","last_active":"2024-09-21T10:05:44.296568+0000","last_peered":"2024-09-21T10:05:44.296568+0000","last_clean":"2024-09-21T10:05:44.296568+0000","last_became_active":"2024-09-21T10:03:38.384065+0000","last_became_peered":"2024-09-21T10:03:38.384065+0000","last_unstale":"2024-09-21T10:05:44.296568+0000","last_undegraded":"2024-09-21T10:05:44.296568+0000","last_fullsized":"2024-09-21T10:05:44.296568+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:47.393146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:47.393146+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:33:03.961914+0000","scrub_duration":0.001601086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":223,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.357051+0000","last_change":"2024-09-21T10:03:38.422217+0000","last_active":"2024-09-21T10:05:44.357051+0000","last_peered":"2024-09-21T10:05:44.357051+0000","last_clean":"2024-09-21T10:05:44.357051+0000","last_became_active":"2024-09-21T10:03:38.421901+0000","last_became_peered":"2024-09-21T10:03:38.421901+0000","last_unstale":"2024-09-21T10:05:44.357051+0000","last_undegraded":"2024-09-21T10:05:44.357051+0000","last_fullsized":"2024-09-21T10:05:44.357051+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:15:01.432624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039581700000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":215,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.336957+0000","last_change":"2024-09-21T09:56:12.580057+0000","last_active":"2024-09-21T10:05:47.336957+0000","last_peered":"2024-09-21T10:05:47.336957+0000","last_clean":"2024-09-21T10:05:47.336957+0000","last_became_active":"2024-09-21T09:56:12.579700+0000","last_became_peered":"2024-09-21T09:56:12.579700+0000","last_unstale":"2024-09-21T10:05:47.336957+0000","last_undegraded":"2024-09-21T10:05:47.336957+0000","last_fullsized":"2024-09-21T10:05:47.336957+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:49.768872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":183,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.351906+0000","last_change":"2024-09-21T10:03:38.371444+0000","last_active":"2024-09-21T10:05:44.351906+0000","last_peered":"2024-09-21T10:05:44.351906+0000","last_clean":"2024-09-21T10:05:44.351906+0000","last_became_active":"2024-09-21T10:03:38.371174+0000","last_became_peered":"2024-09-21T10:03:38.371174+0000","last_unstale":"2024-09-21T10:05:44.351906+0000","last_undegraded":"2024-09-21T10:05:44.351906+0000","last_fullsized":"2024-09-21T10:05:44.351906+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:13:01.365129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"200'5121","reported_seq":5151,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:56.959883+0000","last_change":"2024-09-21T10:04:55.600777+0000","last_active":"2024-09-21T10:05:56.959883+0000","last_peered":"2024-09-21T10:05:56.959883+0000","last_clean":"2024-09-21T10:05:56.959883+0000","last_became_active":"2024-09-21T10:04:55.600435+0000","last_became_peered":"2024-09-21T10:04:55.600435+0000","last_unstale":"2024-09-21T10:05:56.959883+0000","last_undegraded":"2024-09-21T10:05:56.959883+0000","last_fullsized":"2024-09-21T10:05:56.959883+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":5121,"log_dups_size":0,"ondisk_log_size":5121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:05:26.161084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335609856,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5121,"num_write_kb":327744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":42,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.342054+0000","last_change":"2024-09-21T10:04:51.462285+0000","last_active":"2024-09-21T10:05:47.342054+0000","last_peered":"2024-09-21T10:05:47.342054+0000","last_clean":"2024-09-21T10:05:47.342054+0000","last_became_active":"2024-09-21T10:04:42.546242+0000","last_became_peered":"2024-09-21T10:04:42.546242+0000","last_unstale":"2024-09-21T10:05:47.342054+0000","last_undegraded":"2024-09-21T10:05:47.342054+0000","last_fullsized":"2024-09-21T10:05:47.342054+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:51.462213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:51.462213+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:50:10.564260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":48,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297430+0000","last_change":"2024-09-21T10:03:38.393867+0000","last_active":"2024-09-21T10:05:44.297430+0000","last_peered":"2024-09-21T10:05:44.297430+0000","last_clean":"2024-09-21T10:05:44.297430+0000","last_became_active":"2024-09-21T10:03:38.392820+0000","last_became_peered":"2024-09-21T10:03:38.392820+0000","last_unstale":"2024-09-21T10:05:44.297430+0000","last_undegraded":"2024-09-21T10:05:44.297430+0000","last_fullsized":"2024-09-21T10:05:44.297430+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:19:54.697914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":276,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.344589+0000","last_change":"2024-09-21T09:56:12.573698+0000","last_active":"2024-09-21T10:05:47.344589+0000","last_peered":"2024-09-21T10:05:47.344589+0000","last_clean":"2024-09-21T10:05:47.344589+0000","last_became_active":"2024-09-21T09:56:12.573390+0000","last_became_peered":"2024-09-21T09:56:12.573390+0000","last_unstale":"2024-09-21T10:05:47.344589+0000","last_undegraded":"2024-09-21T10:05:47.344589+0000","last_fullsized":"2024-09-21T10:05:47.344589+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:55:12.937878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":287,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.347846+0000","last_change":"2024-09-21T09:51:30.666462+0000","last_active":"2024-09-21T10:05:47.347846+0000","last_peered":"2024-09-21T10:05:47.347846+0000","last_clean":"2024-09-21T10:05:47.347846+0000","last_became_active":"2024-09-21T09:51:30.666006+0000","last_became_peered":"2024-09-21T09:51:30.666006+0000","last_unstale":"2024-09-21T10:05:47.347846+0000","last_undegraded":"2024-09-21T10:05:47.347846+0000","last_fullsized":"2024-09-21T10:05:47.347846+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:51.481698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"200'5368","reported_seq":5386,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:57.798108+0000","last_change":"2024-09-21T10:04:55.602517+0000","last_active":"2024-09-21T10:05:57.798108+0000","last_peered":"2024-09-21T10:05:57.798108+0000","last_clean":"2024-09-21T10:05:57.798108+0000","last_became_active":"2024-09-21T10:04:55.602231+0000","last_became_peered":"2024-09-21T10:04:55.602231+0000","last_unstale":"2024-09-21T10:05:57.798108+0000","last_undegraded":"2024-09-21T10:05:57.798108+0000","last_fullsized":"2024-09-21T10:05:57.798108+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":5368,"log_dups_size":0,"ondisk_log_size":5368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:52:08.852510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351797248,"num_objects":5368,"num_object_clones":0,"num_object_copies":16104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5368,"num_write_kb":343552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":74,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297591+0000","last_change":"2024-09-21T10:04:52.361225+0000","last_active":"2024-09-21T10:05:44.297591+0000","last_peered":"2024-09-21T10:05:44.297591+0000","last_clean":"2024-09-21T10:05:44.297591+0000","last_became_active":"2024-09-21T10:04:42.543054+0000","last_became_peered":"2024-09-21T10:04:42.543054+0000","last_unstale":"2024-09-21T10:05:44.297591+0000","last_undegraded":"2024-09-21T10:05:44.297591+0000","last_fullsized":"2024-09-21T10:05:44.297591+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:52.361137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:52.361137+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:20:43.739146+0000","scrub_duration":0.001255358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":279,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.333872+0000","last_change":"2024-09-21T09:56:12.580659+0000","last_active":"2024-09-21T10:05:47.333872+0000","last_peered":"2024-09-21T10:05:47.333872+0000","last_clean":"2024-09-21T10:05:47.333872+0000","last_became_active":"2024-09-21T09:56:12.580300+0000","last_became_peered":"2024-09-21T09:56:12.580300+0000","last_unstale":"2024-09-21T10:05:47.333872+0000","last_undegraded":"2024-09-21T10:05:47.333872+0000","last_fullsized":"2024-09-21T10:05:47.333872+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:04.945404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":247,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.342546+0000","last_change":"2024-09-21T09:51:30.646466+0000","last_active":"2024-09-21T10:05:47.342546+0000","last_peered":"2024-09-21T10:05:47.342546+0000","last_clean":"2024-09-21T10:05:47.342546+0000","last_became_active":"2024-09-21T09:51:30.646209+0000","last_became_peered":"2024-09-21T09:51:30.646209+0000","last_unstale":"2024-09-21T10:05:47.342546+0000","last_undegraded":"2024-09-21T10:05:47.342546+0000","last_fullsized":"2024-09-21T10:05:47.342546+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:44:41.465418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":251,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.333731+0000","last_change":"2024-09-21T09:51:30.649056+0000","last_active":"2024-09-21T10:05:47.333731+0000","last_peered":"2024-09-21T10:05:47.333731+0000","last_clean":"2024-09-21T10:05:47.333731+0000","last_became_active":"2024-09-21T09:51:30.647245+0000","last_became_peered":"2024-09-21T09:51:30.647245+0000","last_unstale":"2024-09-21T10:05:47.333731+0000","last_undegraded":"2024-09-21T10:05:47.333731+0000","last_fullsized":"2024-09-21T10:05:47.333731+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:32.661847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.a","version":"200'5381","reported_seq":5396,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:57.333443+0000","last_change":"2024-09-21T10:04:55.603587+0000","last_active":"2024-09-21T10:05:57.333443+0000","last_peered":"2024-09-21T10:05:57.333443+0000","last_clean":"2024-09-21T10:05:57.333443+0000","last_became_active":"2024-09-21T10:04:55.603173+0000","last_became_peered":"2024-09-21T10:04:55.603173+0000","last_unstale":"2024-09-21T10:05:57.333443+0000","last_undegraded":"2024-09-21T10:05:57.333443+0000","last_fullsized":"2024-09-21T10:05:57.333443+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":5381,"log_dups_size":0,"ondisk_log_size":5381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:12:45.473222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352649216,"num_objects":5381,"num_object_clones":0,"num_object_copies":16143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5381,"num_write_kb":344384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":31,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.353235+0000","last_change":"2024-09-21T10:04:44.288106+0000","last_active":"2024-09-21T10:05:44.353235+0000","last_peered":"2024-09-21T10:05:44.353235+0000","last_clean":"2024-09-21T10:05:44.353235+0000","last_became_active":"2024-09-21T10:04:42.542859+0000","last_became_peered":"2024-09-21T10:04:42.542859+0000","last_unstale":"2024-09-21T10:05:44.353235+0000","last_undegraded":"2024-09-21T10:05:44.353235+0000","last_fullsized":"2024-09-21T10:05:44.353235+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:44.288000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:44.288000+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:29.958780+0000","scrub_duration":0.0022491059999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":179,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.353505+0000","last_change":"2024-09-21T09:50:45.502094+0000","last_active":"2024-09-21T10:05:44.353505+0000","last_peered":"2024-09-21T10:05:44.353505+0000","last_clean":"2024-09-21T10:05:44.353505+0000","last_became_active":"2024-09-21T09:50:45.501603+0000","last_became_peered":"2024-09-21T09:50:45.501603+0000","last_unstale":"2024-09-21T10:05:44.353505+0000","last_undegraded":"2024-09-21T10:05:44.353505+0000","last_fullsized":"2024-09-21T10:05:44.353505+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:01.747067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091939799999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":164,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297188+0000","last_change":"2024-09-21T09:48:54.128617+0000","last_active":"2024-09-21T10:05:44.297188+0000","last_peered":"2024-09-21T10:05:44.297188+0000","last_clean":"2024-09-21T10:05:44.297188+0000","last_became_active":"2024-09-21T09:48:54.127878+0000","last_became_peered":"2024-09-21T09:48:54.127878+0000","last_unstale":"2024-09-21T10:05:44.297188+0000","last_undegraded":"2024-09-21T10:05:44.297188+0000","last_fullsized":"2024-09-21T10:05:44.297188+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:07.305638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":259,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.341027+0000","last_change":"2024-09-21T09:56:12.588992+0000","last_active":"2024-09-21T10:05:47.341027+0000","last_peered":"2024-09-21T10:05:47.341027+0000","last_clean":"2024-09-21T10:05:47.341027+0000","last_became_active":"2024-09-21T09:56:12.588665+0000","last_became_peered":"2024-09-21T09:56:12.588665+0000","last_unstale":"2024-09-21T10:05:47.341027+0000","last_undegraded":"2024-09-21T10:05:47.341027+0000","last_fullsized":"2024-09-21T10:05:47.341027+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:19:06.160456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"200'5185","reported_seq":5202,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:56.439683+0000","last_change":"2024-09-21T10:04:55.601700+0000","last_active":"2024-09-21T10:05:56.439683+0000","last_peered":"2024-09-21T10:05:56.439683+0000","last_clean":"2024-09-21T10:05:56.439683+0000","last_became_active":"2024-09-21T10:04:55.601163+0000","last_became_peered":"2024-09-21T10:04:55.601163+0000","last_unstale":"2024-09-21T10:05:56.439683+0000","last_undegraded":"2024-09-21T10:05:56.439683+0000","last_fullsized":"2024-09-21T10:05:56.439683+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":5185,"log_dups_size":0,"ondisk_log_size":5185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:15:47.805823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339804160,"num_objects":5185,"num_object_clones":0,"num_object_copies":15555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5185,"num_write_kb":331840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":174,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.352002+0000","last_change":"2024-09-21T10:04:45.242148+0000","last_active":"2024-09-21T10:05:44.352002+0000","last_peered":"2024-09-21T10:05:44.352002+0000","last_clean":"2024-09-21T10:05:44.352002+0000","last_became_active":"2024-09-21T10:04:42.549844+0000","last_became_peered":"2024-09-21T10:04:42.549844+0000","last_unstale":"2024-09-21T10:05:44.352002+0000","last_undegraded":"2024-09-21T10:05:44.352002+0000","last_fullsized":"2024-09-21T10:05:44.352002+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_deep_scrub":"77'100","last_deep_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.242073+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:03:42.250480+0000","scrub_duration":0.0023466239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":182,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.353786+0000","last_change":"2024-09-21T09:56:12.583280+0000","last_active":"2024-09-21T10:05:44.353786+0000","last_peered":"2024-09-21T10:05:44.353786+0000","last_clean":"2024-09-21T10:05:44.353786+0000","last_became_active":"2024-09-21T09:56:12.583007+0000","last_became_peered":"2024-09-21T09:56:12.583007+0000","last_unstale":"2024-09-21T10:05:44.353786+0000","last_undegraded":"2024-09-21T10:05:44.353786+0000","last_fullsized":"2024-09-21T10:05:44.353786+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:26.991596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":311,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.341192+0000","last_change":"2024-09-21T10:03:38.392622+0000","last_active":"2024-09-21T10:05:47.341192+0000","last_peered":"2024-09-21T10:05:47.341192+0000","last_clean":"2024-09-21T10:05:47.341192+0000","last_became_active":"2024-09-21T10:03:38.392290+0000","last_became_peered":"2024-09-21T10:03:38.392290+0000","last_unstale":"2024-09-21T10:05:47.341192+0000","last_undegraded":"2024-09-21T10:05:47.341192+0000","last_fullsized":"2024-09-21T10:05:47.341192+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:56:22.222321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":237,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.341329+0000","last_change":"2024-09-21T09:50:45.501349+0000","last_active":"2024-09-21T10:05:47.341329+0000","last_peered":"2024-09-21T10:05:47.341329+0000","last_clean":"2024-09-21T10:05:47.341329+0000","last_became_active":"2024-09-21T09:50:45.501065+0000","last_became_peered":"2024-09-21T09:50:45.501065+0000","last_unstale":"2024-09-21T10:05:47.341329+0000","last_undegraded":"2024-09-21T10:05:47.341329+0000","last_fullsized":"2024-09-21T10:05:47.341329+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:28:58.570586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"200'5199","reported_seq":5217,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:57.303683+0000","last_change":"2024-09-21T10:04:55.603954+0000","last_active":"2024-09-21T10:05:57.303683+0000","last_peered":"2024-09-21T10:05:57.303683+0000","last_clean":"2024-09-21T10:05:57.303683+0000","last_became_active":"2024-09-21T10:04:55.603666+0000","last_became_peered":"2024-09-21T10:04:55.603666+0000","last_unstale":"2024-09-21T10:05:57.303683+0000","last_undegraded":"2024-09-21T10:05:57.303683+0000","last_fullsized":"2024-09-21T10:05:57.303683+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":5199,"log_dups_size":0,"ondisk_log_size":5199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:33.958646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340721664,"num_objects":5199,"num_object_clones":0,"num_object_copies":15597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5199,"num_write_kb":332736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":174,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.351910+0000","last_change":"2024-09-21T10:04:49.226571+0000","last_active":"2024-09-21T10:05:44.351910+0000","last_peered":"2024-09-21T10:05:44.351910+0000","last_clean":"2024-09-21T10:05:44.351910+0000","last_became_active":"2024-09-21T10:04:42.547869+0000","last_became_peered":"2024-09-21T10:04:42.547869+0000","last_unstale":"2024-09-21T10:05:44.351910+0000","last_undegraded":"2024-09-21T10:05:44.351910+0000","last_fullsized":"2024-09-21T10:05:44.351910+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:49.226496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:49.226496+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:14:59.303054+0000","scrub_duration":0.001656865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":247,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.341102+0000","last_change":"2024-09-21T09:51:30.646531+0000","last_active":"2024-09-21T10:05:47.341102+0000","last_peered":"2024-09-21T10:05:47.341102+0000","last_clean":"2024-09-21T10:05:47.341102+0000","last_became_active":"2024-09-21T09:51:30.646248+0000","last_became_peered":"2024-09-21T09:51:30.646248+0000","last_unstale":"2024-09-21T10:05:47.341102+0000","last_undegraded":"2024-09-21T10:05:47.341102+0000","last_fullsized":"2024-09-21T10:05:47.341102+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:00.312924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":48,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297020+0000","last_change":"2024-09-21T10:03:38.402082+0000","last_active":"2024-09-21T10:05:44.297020+0000","last_peered":"2024-09-21T10:05:44.297020+0000","last_clean":"2024-09-21T10:05:44.297020+0000","last_became_active":"2024-09-21T10:03:38.395632+0000","last_became_peered":"2024-09-21T10:03:38.395632+0000","last_unstale":"2024-09-21T10:05:44.297020+0000","last_undegraded":"2024-09-21T10:05:44.297020+0000","last_fullsized":"2024-09-21T10:05:44.297020+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:29:00.631337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":291,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.345985+0000","last_change":"2024-09-21T10:03:38.411780+0000","last_active":"2024-09-21T10:05:47.345985+0000","last_peered":"2024-09-21T10:05:47.345985+0000","last_clean":"2024-09-21T10:05:47.345985+0000","last_became_active":"2024-09-21T10:03:38.411457+0000","last_became_peered":"2024-09-21T10:03:38.411457+0000","last_unstale":"2024-09-21T10:05:47.345985+0000","last_undegraded":"2024-09-21T10:05:47.345985+0000","last_fullsized":"2024-09-21T10:05:47.345985+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:19:08.177347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"200'5335","reported_seq":5353,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:57.349492+0000","last_change":"2024-09-21T10:04:55.603150+0000","last_active":"2024-09-21T10:05:57.349492+0000","last_peered":"2024-09-21T10:05:57.349492+0000","last_clean":"2024-09-21T10:05:57.349492+0000","last_became_active":"2024-09-21T10:04:55.602817+0000","last_became_peered":"2024-09-21T10:04:55.602817+0000","last_unstale":"2024-09-21T10:05:57.349492+0000","last_undegraded":"2024-09-21T10:05:57.349492+0000","last_fullsized":"2024-09-21T10:05:57.349492+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":5335,"log_dups_size":0,"ondisk_log_size":5335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:24:29.153232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349634560,"num_objects":5335,"num_object_clones":0,"num_object_copies":16005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5335,"num_write_kb":341440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":31,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297679+0000","last_change":"2024-09-21T10:04:44.475698+0000","last_active":"2024-09-21T10:05:44.297679+0000","last_peered":"2024-09-21T10:05:44.297679+0000","last_clean":"2024-09-21T10:05:44.297679+0000","last_became_active":"2024-09-21T10:04:42.549251+0000","last_became_peered":"2024-09-21T10:04:42.549251+0000","last_unstale":"2024-09-21T10:05:44.297679+0000","last_undegraded":"2024-09-21T10:05:44.297679+0000","last_fullsized":"2024-09-21T10:05:44.297679+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:44.475623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:44.475623+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:31:10.067467+0000","scrub_duration":0.0020471650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":247,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.339365+0000","last_change":"2024-09-21T10:03:38.381626+0000","last_active":"2024-09-21T10:05:47.339365+0000","last_peered":"2024-09-21T10:05:47.339365+0000","last_clean":"2024-09-21T10:05:47.339365+0000","last_became_active":"2024-09-21T10:03:38.381000+0000","last_became_peered":"2024-09-21T10:03:38.381000+0000","last_unstale":"2024-09-21T10:05:47.339365+0000","last_undegraded":"2024-09-21T10:05:47.339365+0000","last_fullsized":"2024-09-21T10:05:47.339365+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:34:37.047124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":247,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.341941+0000","last_change":"2024-09-21T09:50:45.513017+0000","last_active":"2024-09-21T10:05:47.341941+0000","last_peered":"2024-09-21T10:05:47.341941+0000","last_clean":"2024-09-21T10:05:47.341941+0000","last_became_active":"2024-09-21T09:50:45.512629+0000","last_became_peered":"2024-09-21T09:50:45.512629+0000","last_unstale":"2024-09-21T10:05:47.341941+0000","last_undegraded":"2024-09-21T10:05:47.341941+0000","last_fullsized":"2024-09-21T10:05:47.341941+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:47:08.421575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":236,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.337945+0000","last_change":"2024-09-21T10:03:38.386785+0000","last_active":"2024-09-21T10:05:47.337945+0000","last_peered":"2024-09-21T10:05:47.337945+0000","last_clean":"2024-09-21T10:05:47.337945+0000","last_became_active":"2024-09-21T10:03:38.386313+0000","last_became_peered":"2024-09-21T10:03:38.386313+0000","last_unstale":"2024-09-21T10:05:47.337945+0000","last_undegraded":"2024-09-21T10:05:47.337945+0000","last_fullsized":"2024-09-21T10:05:47.337945+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:10.633433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"200'5269","reported_seq":5287,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:57.292663+0000","last_change":"2024-09-21T10:04:55.802526+0000","last_active":"2024-09-21T10:05:57.292663+0000","last_peered":"2024-09-21T10:05:57.292663+0000","last_clean":"2024-09-21T10:05:57.292663+0000","last_became_active":"2024-09-21T10:04:55.802207+0000","last_became_peered":"2024-09-21T10:04:55.802207+0000","last_unstale":"2024-09-21T10:05:57.292663+0000","last_undegraded":"2024-09-21T10:05:57.292663+0000","last_fullsized":"2024-09-21T10:05:57.292663+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:04:54.578106+0000","last_clean_scrub_stamp":"2024-09-21T10:04:54.578106+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:22.119648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345309184,"num_objects":5269,"num_object_clones":0,"num_object_copies":15807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5269,"num_write_kb":337216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":42,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.336448+0000","last_change":"2024-09-21T10:04:43.005569+0000","last_active":"2024-09-21T10:05:47.336448+0000","last_peered":"2024-09-21T10:05:47.336448+0000","last_clean":"2024-09-21T10:05:47.336448+0000","last_became_active":"2024-09-21T10:04:42.552636+0000","last_became_peered":"2024-09-21T10:04:42.552636+0000","last_unstale":"2024-09-21T10:05:47.336448+0000","last_undegraded":"2024-09-21T10:05:47.336448+0000","last_fullsized":"2024-09-21T10:05:47.336448+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.005477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.005477+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:06.276495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":279,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.348098+0000","last_change":"2024-09-21T09:50:45.510812+0000","last_active":"2024-09-21T10:05:47.348098+0000","last_peered":"2024-09-21T10:05:47.348098+0000","last_clean":"2024-09-21T10:05:47.348098+0000","last_became_active":"2024-09-21T09:50:45.510480+0000","last_became_peered":"2024-09-21T09:50:45.510480+0000","last_unstale":"2024-09-21T10:05:47.348098+0000","last_undegraded":"2024-09-21T10:05:47.348098+0000","last_fullsized":"2024-09-21T10:05:47.348098+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:05.408194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":213,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.352114+0000","last_change":"2024-09-21T10:03:38.392767+0000","last_active":"2024-09-21T10:05:44.352114+0000","last_peered":"2024-09-21T10:05:44.352114+0000","last_clean":"2024-09-21T10:05:44.352114+0000","last_became_active":"2024-09-21T10:03:38.392424+0000","last_became_peered":"2024-09-21T10:03:38.392424+0000","last_unstale":"2024-09-21T10:05:44.352114+0000","last_undegraded":"2024-09-21T10:05:44.352114+0000","last_fullsized":"2024-09-21T10:05:44.352114+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:25.358291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":146,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296639+0000","last_change":"2024-09-21T09:40:49.190763+0000","last_active":"2024-09-21T10:05:44.296639+0000","last_peered":"2024-09-21T10:05:44.296639+0000","last_clean":"2024-09-21T10:05:44.296639+0000","last_became_active":"2024-09-21T09:40:49.190463+0000","last_became_peered":"2024-09-21T09:40:49.190463+0000","last_unstale":"2024-09-21T10:05:44.296639+0000","last_undegraded":"2024-09-21T10:05:44.296639+0000","last_fullsized":"2024-09-21T10:05:44.296639+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:15.669404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":173,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297060+0000","last_change":"2024-09-21T09:51:30.642074+0000","last_active":"2024-09-21T10:05:44.297060+0000","last_peered":"2024-09-21T10:05:44.297060+0000","last_clean":"2024-09-21T10:05:44.297060+0000","last_became_active":"2024-09-21T09:51:30.641176+0000","last_became_peered":"2024-09-21T09:51:30.641176+0000","last_unstale":"2024-09-21T10:05:44.297060+0000","last_undegraded":"2024-09-21T10:05:44.297060+0000","last_fullsized":"2024-09-21T10:05:44.297060+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:18.766865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":227,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297484+0000","last_change":"2024-09-21T10:03:38.395742+0000","last_active":"2024-09-21T10:05:44.297484+0000","last_peered":"2024-09-21T10:05:44.297484+0000","last_clean":"2024-09-21T10:05:44.297484+0000","last_became_active":"2024-09-21T10:03:38.395439+0000","last_became_peered":"2024-09-21T10:03:38.395439+0000","last_unstale":"2024-09-21T10:05:44.297484+0000","last_undegraded":"2024-09-21T10:05:44.297484+0000","last_fullsized":"2024-09-21T10:05:44.297484+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:27:40.297701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":247,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.341125+0000","last_change":"2024-09-21T09:51:30.642334+0000","last_active":"2024-09-21T10:05:47.341125+0000","last_peered":"2024-09-21T10:05:47.341125+0000","last_clean":"2024-09-21T10:05:47.341125+0000","last_became_active":"2024-09-21T09:51:30.641267+0000","last_became_peered":"2024-09-21T09:51:30.641267+0000","last_unstale":"2024-09-21T10:05:47.341125+0000","last_undegraded":"2024-09-21T10:05:47.341125+0000","last_fullsized":"2024-09-21T10:05:47.341125+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:24.201971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":54,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297084+0000","last_change":"2024-09-21T10:03:38.402359+0000","last_active":"2024-09-21T10:05:44.297084+0000","last_peered":"2024-09-21T10:05:44.297084+0000","last_clean":"2024-09-21T10:05:44.297084+0000","last_became_active":"2024-09-21T10:03:38.400209+0000","last_became_peered":"2024-09-21T10:03:38.400209+0000","last_unstale":"2024-09-21T10:05:44.297084+0000","last_undegraded":"2024-09-21T10:05:44.297084+0000","last_fullsized":"2024-09-21T10:05:44.297084+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:42:04.508541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"77'100","reported_seq":42,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.333260+0000","last_change":"2024-09-21T10:04:43.015055+0000","last_active":"2024-09-21T10:05:47.333260+0000","last_peered":"2024-09-21T10:05:47.333260+0000","last_clean":"2024-09-21T10:05:47.333260+0000","last_became_active":"2024-09-21T10:04:42.545568+0000","last_became_peered":"2024-09-21T10:04:42.545568+0000","last_unstale":"2024-09-21T10:05:47.333260+0000","last_undegraded":"2024-09-21T10:05:47.333260+0000","last_fullsized":"2024-09-21T10:05:47.333260+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.014993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.014993+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:37:24.019431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":202,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.355914+0000","last_change":"2024-09-21T10:03:38.378365+0000","last_active":"2024-09-21T10:05:44.355914+0000","last_peered":"2024-09-21T10:05:44.355914+0000","last_clean":"2024-09-21T10:05:44.355914+0000","last_became_active":"2024-09-21T10:03:38.377388+0000","last_became_peered":"2024-09-21T10:03:38.377388+0000","last_unstale":"2024-09-21T10:05:44.355914+0000","last_undegraded":"2024-09-21T10:05:44.355914+0000","last_fullsized":"2024-09-21T10:05:44.355914+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:31.162683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":241,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.338031+0000","last_change":"2024-09-21T10:03:38.425111+0000","last_active":"2024-09-21T10:05:47.338031+0000","last_peered":"2024-09-21T10:05:47.338031+0000","last_clean":"2024-09-21T10:05:47.338031+0000","last_became_active":"2024-09-21T10:03:38.424704+0000","last_became_peered":"2024-09-21T10:03:38.424704+0000","last_unstale":"2024-09-21T10:05:47.338031+0000","last_undegraded":"2024-09-21T10:05:47.338031+0000","last_fullsized":"2024-09-21T10:05:47.338031+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:35.157076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":275,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.347826+0000","last_change":"2024-09-21T09:56:12.574472+0000","last_active":"2024-09-21T10:05:47.347826+0000","last_peered":"2024-09-21T10:05:47.347826+0000","last_clean":"2024-09-21T10:05:47.347826+0000","last_became_active":"2024-09-21T09:56:12.573966+0000","last_became_peered":"2024-09-21T09:56:12.573966+0000","last_unstale":"2024-09-21T10:05:47.347826+0000","last_undegraded":"2024-09-21T10:05:47.347826+0000","last_fullsized":"2024-09-21T10:05:47.347826+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:37:21.415933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":226,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.336131+0000","last_change":"2024-09-21T09:56:12.576756+0000","last_active":"2024-09-21T10:05:47.336131+0000","last_peered":"2024-09-21T10:05:47.336131+0000","last_clean":"2024-09-21T10:05:47.336131+0000","last_became_active":"2024-09-21T09:56:12.575752+0000","last_became_peered":"2024-09-21T09:56:12.575752+0000","last_unstale":"2024-09-21T10:05:47.336131+0000","last_undegraded":"2024-09-21T10:05:47.336131+0000","last_fullsized":"2024-09-21T10:05:47.336131+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:55.224839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":163,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.352055+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T10:05:44.352055+0000","last_peered":"2024-09-21T10:05:44.352055+0000","last_clean":"2024-09-21T10:05:44.352055+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T10:05:44.352055+0000","last_undegraded":"2024-09-21T10:05:44.352055+0000","last_fullsized":"2024-09-21T10:05:44.352055+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":244,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.336513+0000","last_change":"2024-09-21T09:56:12.571495+0000","last_active":"2024-09-21T10:05:47.336513+0000","last_peered":"2024-09-21T10:05:47.336513+0000","last_clean":"2024-09-21T10:05:47.336513+0000","last_became_active":"2024-09-21T09:56:12.571200+0000","last_became_peered":"2024-09-21T09:56:12.571200+0000","last_unstale":"2024-09-21T10:05:47.336513+0000","last_undegraded":"2024-09-21T10:05:47.336513+0000","last_fullsized":"2024-09-21T10:05:47.336513+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:53:58.851300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":302,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.347934+0000","last_change":"2024-09-21T10:03:38.380567+0000","last_active":"2024-09-21T10:05:47.347934+0000","last_peered":"2024-09-21T10:05:47.347934+0000","last_clean":"2024-09-21T10:05:47.347934+0000","last_became_active":"2024-09-21T10:03:38.378538+0000","last_became_peered":"2024-09-21T10:03:38.378538+0000","last_unstale":"2024-09-21T10:05:47.347934+0000","last_undegraded":"2024-09-21T10:05:47.347934+0000","last_fullsized":"2024-09-21T10:05:47.347934+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:16:25.373217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":246,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.340482+0000","last_change":"2024-09-21T09:56:12.580276+0000","last_active":"2024-09-21T10:05:47.340482+0000","last_peered":"2024-09-21T10:05:47.340482+0000","last_clean":"2024-09-21T10:05:47.340482+0000","last_became_active":"2024-09-21T09:56:12.579989+0000","last_became_peered":"2024-09-21T09:56:12.579989+0000","last_unstale":"2024-09-21T10:05:47.340482+0000","last_undegraded":"2024-09-21T10:05:47.340482+0000","last_fullsized":"2024-09-21T10:05:47.340482+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:14:56.361873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":225,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.337366+0000","last_change":"2024-09-21T09:56:12.582383+0000","last_active":"2024-09-21T10:05:47.337366+0000","last_peered":"2024-09-21T10:05:47.337366+0000","last_clean":"2024-09-21T10:05:47.337366+0000","last_became_active":"2024-09-21T09:56:12.581427+0000","last_became_peered":"2024-09-21T09:56:12.581427+0000","last_unstale":"2024-09-21T10:05:47.337366+0000","last_undegraded":"2024-09-21T10:05:47.337366+0000","last_fullsized":"2024-09-21T10:05:47.337366+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:24.275633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":182,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297245+0000","last_change":"2024-09-21T09:51:30.645644+0000","last_active":"2024-09-21T10:05:44.297245+0000","last_peered":"2024-09-21T10:05:44.297245+0000","last_clean":"2024-09-21T10:05:44.297245+0000","last_became_active":"2024-09-21T09:51:30.644883+0000","last_became_peered":"2024-09-21T09:51:30.644883+0000","last_unstale":"2024-09-21T10:05:44.297245+0000","last_undegraded":"2024-09-21T10:05:44.297245+0000","last_fullsized":"2024-09-21T10:05:44.297245+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:39.331936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":279,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.333413+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:05:47.333413+0000","last_peered":"2024-09-21T10:05:47.333413+0000","last_clean":"2024-09-21T10:05:47.333413+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:05:47.333413+0000","last_undegraded":"2024-09-21T10:05:47.333413+0000","last_fullsized":"2024-09-21T10:05:47.333413+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":292,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.346064+0000","last_change":"2024-09-21T10:03:38.407960+0000","last_active":"2024-09-21T10:05:47.346064+0000","last_peered":"2024-09-21T10:05:47.346064+0000","last_clean":"2024-09-21T10:05:47.346064+0000","last_became_active":"2024-09-21T10:03:38.407632+0000","last_became_peered":"2024-09-21T10:03:38.407632+0000","last_unstale":"2024-09-21T10:05:47.346064+0000","last_undegraded":"2024-09-21T10:05:47.346064+0000","last_fullsized":"2024-09-21T10:05:47.346064+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:51:23.967273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":180,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296996+0000","last_change":"2024-09-21T09:56:12.576148+0000","last_active":"2024-09-21T10:05:44.296996+0000","last_peered":"2024-09-21T10:05:44.296996+0000","last_clean":"2024-09-21T10:05:44.296996+0000","last_became_active":"2024-09-21T09:56:12.575813+0000","last_became_peered":"2024-09-21T09:56:12.575813+0000","last_unstale":"2024-09-21T10:05:44.296996+0000","last_undegraded":"2024-09-21T10:05:44.296996+0000","last_fullsized":"2024-09-21T10:05:44.296996+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:53:01.367816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":167,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.355745+0000","last_change":"2024-09-21T09:50:45.502447+0000","last_active":"2024-09-21T10:05:44.355745+0000","last_peered":"2024-09-21T10:05:44.355745+0000","last_clean":"2024-09-21T10:05:44.355745+0000","last_became_active":"2024-09-21T09:50:45.500800+0000","last_became_peered":"2024-09-21T09:50:45.500800+0000","last_unstale":"2024-09-21T10:05:44.355745+0000","last_undegraded":"2024-09-21T10:05:44.355745+0000","last_fullsized":"2024-09-21T10:05:44.355745+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:20:51.819410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":180,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.297163+0000","last_change":"2024-09-21T09:56:12.576457+0000","last_active":"2024-09-21T10:05:44.297163+0000","last_peered":"2024-09-21T10:05:44.297163+0000","last_clean":"2024-09-21T10:05:44.297163+0000","last_became_active":"2024-09-21T09:56:12.576061+0000","last_became_peered":"2024-09-21T09:56:12.576061+0000","last_unstale":"2024-09-21T10:05:44.297163+0000","last_undegraded":"2024-09-21T10:05:44.297163+0000","last_fullsized":"2024-09-21T10:05:44.297163+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:51:46.676699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":150,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.353721+0000","last_change":"2024-09-21T09:40:49.094916+0000","last_active":"2024-09-21T10:05:44.353721+0000","last_peered":"2024-09-21T10:05:44.353721+0000","last_clean":"2024-09-21T10:05:44.353721+0000","last_became_active":"2024-09-21T09:40:49.094264+0000","last_became_peered":"2024-09-21T09:40:49.094264+0000","last_unstale":"2024-09-21T10:05:44.353721+0000","last_undegraded":"2024-09-21T10:05:44.353721+0000","last_fullsized":"2024-09-21T10:05:44.353721+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:53.663226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":252,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.337238+0000","last_change":"2024-09-21T09:56:12.577329+0000","last_active":"2024-09-21T10:05:47.337238+0000","last_peered":"2024-09-21T10:05:47.337238+0000","last_clean":"2024-09-21T10:05:47.337238+0000","last_became_active":"2024-09-21T09:56:12.576941+0000","last_became_peered":"2024-09-21T09:56:12.576941+0000","last_unstale":"2024-09-21T10:05:47.337238+0000","last_undegraded":"2024-09-21T10:05:47.337238+0000","last_fullsized":"2024-09-21T10:05:47.337238+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:35.235768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":237,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.337816+0000","last_change":"2024-09-21T09:56:12.577137+0000","last_active":"2024-09-21T10:05:47.337816+0000","last_peered":"2024-09-21T10:05:47.337816+0000","last_clean":"2024-09-21T10:05:47.337816+0000","last_became_active":"2024-09-21T09:56:12.576668+0000","last_became_peered":"2024-09-21T09:56:12.576668+0000","last_unstale":"2024-09-21T10:05:47.337816+0000","last_undegraded":"2024-09-21T10:05:47.337816+0000","last_fullsized":"2024-09-21T10:05:47.337816+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:56.772427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":163,"reported_epoch":199,"state":"active+clean","last_fresh":"2024-09-21T10:05:44.296384+0000","last_change":"2024-09-21T09:50:45.502819+0000","last_active":"2024-09-21T10:05:44.296384+0000","last_peered":"2024-09-21T10:05:44.296384+0000","last_clean":"2024-09-21T10:05:44.296384+0000","last_became_active":"2024-09-21T09:50:45.502483+0000","last_became_peered":"2024-09-21T09:50:45.502483+0000","last_unstale":"2024-09-21T10:05:44.296384+0000","last_undegraded":"2024-09-21T10:05:44.296384+0000","last_fullsized":"2024-09-21T10:05:44.296384+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:26.549958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":280,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-09-21T10:05:47.347528+0000","last_change":"2024-09-21T09:51:30.650116+0000","last_active":"2024-09-21T10:05:47.347528+0000","last_peered":"2024-09-21T10:05:47.347528+0000","last_clean":"2024-09-21T10:05:47.347528+0000","last_became_active":"2024-09-21T09:51:30.649798+0000","last_became_peered":"2024-09-21T09:51:30.649798+0000","last_unstale":"2024-09-21T10:05:47.347528+0000","last_undegraded":"2024-09-21T10:05:47.347528+0000","last_fullsized":"2024-09-21T10:05:47.347528+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:40.222590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5579341824,"num_objects":85134,"num_object_clones":0,"num_object_copies":255402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85134,"num_write_kb":5448576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16818831360,"data_stored":16818831360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85134,"ondisk_log_size":85134,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":9648,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23724,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626763,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2966880,"kb_used_data":2381144,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":90778272,"statfs":{"total":95995035648,"available":92956950528,"internally_reserved":0,"allocated":2438291456,"data_stored":2437223836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019221,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3570788,"kb_used_data":2679836,"kb_used_omap":1,"kb_used_meta":890942,"kb_avail":90174364,"statfs":{"total":95995035648,"available":92338548736,"internally_reserved":0,"allocated":2744152064,"data_stored":2743080397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912324952},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":81,"seq":347892351183,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4012884,"kb_used_data":3347276,"kb_used_omap":5,"kb_used_meta":665594,"kb_avail":89732268,"statfs":{"total":95995035648,"available":91885842432,"internally_reserved":0,"allocated":3427610624,"data_stored":3426551079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6084,"internal_metadata":681568316},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":175,"seq":751619276829,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1666152,"kb_used_data":1327392,"kb_used_omap":12,"kb_used_meta":338739,"kb_avail":92079000,"statfs":{"total":95995035648,"available":94288896000,"internally_reserved":0,"allocated":1359249408,"data_stored":1358173731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12845,"internal_metadata":346869203},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822992,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2598356,"kb_used_data":1697420,"kb_used_omap":1,"kb_used_meta":900926,"kb_avail":91146796,"statfs":{"total":95995035648,"available":93334319104,"internally_reserved":0,"allocated":1738158080,"data_stored":1737098488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922548568},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879416,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2283668,"kb_used_data":1767052,"kb_used_omap":4,"kb_used_meta":516603,"kb_avail":91461484,"statfs":{"total":95995035648,"available":93656559616,"internally_reserved":0,"allocated":1809461248,"data_stored":1808401737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":529001916},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566256,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2523224,"kb_used_data":2075536,"kb_used_omap":6,"kb_used_meta":447673,"kb_avail":91221928,"statfs":{"total":95995035648,"available":93411254272,"internally_reserved":0,"allocated":2125348864,"data_stored":2124285779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":458417575},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860601,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1689356,"kb_used_data":1062276,"kb_used_omap":5,"kb_used_meta":627066,"kb_avail":92055796,"statfs":{"total":95995035648,"available":94265135104,"internally_reserved":0,"allocated":1087770624,"data_stored":1086718367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":642115823},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4380,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9976,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3438084096,"data_stored":3438084096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1092812800,"data_stored":1092812800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2139422720,"data_stored":2139422720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1825374208,"data_stored":1825374208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2773417984,"data_stored":2773417984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1752694784,"data_stored":1752694784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2436366336,"data_stored":2436366336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1360658432,"data_stored":1360658432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:06:02.910 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T10:06:02.910 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:06:02.910 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:06:02.911 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-21T10:06:02.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.0 2024-09-21T10:06:02.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:02.995 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:02.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:02.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:03.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:03.136 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:03.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:03.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:03.211 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:06:03 smithi107 systemd[1]: Stopping Ceph osd.0 for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:06:03.238 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:03.266 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:03.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:03.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:03.368 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:03.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:03.410 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:03.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:03.511 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:03.545 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:03.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:03.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:03.647 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:03.657 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 54 16 92060 92044 106.52 71.875 0.0039735 0.00937892 2024-09-21T10:06:03.668 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:03.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:03.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:03.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:03.804 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:03.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:03.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:03.907 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:03.936 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:03.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:03.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:04.038 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:04.073 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:04.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:04.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:04.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:04.211 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:04.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:04.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:04.313 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:04.346 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:04.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:04.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:04.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:06:04.448 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:04.474 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:04.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:04.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:04.576 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:04.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:04.610 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:04.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:04.658 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 55 16 93234 93218 105.917 73.375 0.0117395 0.00943663 2024-09-21T10:06:04.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:04 smithi129 ceph-mon[88471]: pgmap v717: 109 pgs: 109 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-21T10:06:04.711 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:04.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:04.755 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:04.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:04.812 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:06:04.856 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:04 smithi107 ceph-mon[102221]: pgmap v717: 109 pgs: 109 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-21T10:06:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:04 smithi107 ceph-mon[105022]: pgmap v717: 109 pgs: 109 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-21T10:06:04.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:04.905 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:04.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:05.006 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:05.037 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:05.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:05.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:05.139 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:05.174 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:05.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:05.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:05.276 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:05.355 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:06:05.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:05.356 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:05.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:05.457 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:05.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:05.493 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:05.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:05.594 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:05.609 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:06:05 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0[111891]: 2024-09-21T10:06:05.455+0000 7efd1d129640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:06:05.609 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:06:05 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0[111891]: 2024-09-21T10:06:05.455+0000 7efd1d129640 -1 osd.0 201 *** Got signal Terminated *** 2024-09-21T10:06:05.609 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:06:05 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0[111891]: 2024-09-21T10:06:05.455+0000 7efd1d129640 -1 osd.0 201 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:06:05.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:05.641 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:05.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:05.657 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 56 16 94300 94284 105.215 66.625 0.0158934 0.00949881 2024-09-21T10:06:05.672 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:05.704 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:06:05.704 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:06:05.739 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:06:05.740 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:06:05.742 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:05.769 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:06:05.770 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:06:05.783 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:05.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:05.804 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:06:05.805 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:06:05.835 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:06:05.835 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:06:05.865 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:06:05.865 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:06:05.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:05.896 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:06:05.896 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:06:05.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:05.917 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:05.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:05.925 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:06:05.925 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:06:06.017 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:06.049 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:06.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:06.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:06.194 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:06.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:06.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:06.296 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:06.336 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:06.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:06.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:06.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:06.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:06.460 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:06.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:06.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:06.581 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:06.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:06.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:06.605 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:06 smithi129 ceph-mon[88471]: pgmap v718: 109 pgs: 109 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T10:06:06.605 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:06 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:06.605 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:06 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:06.606 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:06 smithi129 ceph-mon[88471]: osd.0 marked itself down and dead 2024-09-21T10:06:06.606 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:06 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3212895825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:06.658 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 57 16 95110 95094 104.257 50.625 0.0115255 0.00954306 2024-09-21T10:06:06.683 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:06.706 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:06.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:06.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:06.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:06.828 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:06 smithi107 ceph-mon[102221]: pgmap v718: 109 pgs: 109 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T10:06:06.828 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:06 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:06.828 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:06 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:06.828 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:06 smithi107 ceph-mon[102221]: osd.0 marked itself down and dead 2024-09-21T10:06:06.828 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:06 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3212895825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:06.828 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:06 smithi107 ceph-mon[105022]: pgmap v718: 109 pgs: 109 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T10:06:06.828 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:06 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:06.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:06 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:06.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:06 smithi107 ceph-mon[105022]: osd.0 marked itself down and dead 2024-09-21T10:06:06.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:06 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3212895825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:06.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:06.829 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:06.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:06.930 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:06.952 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:06.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:07.054 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:07.075 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:07.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:07.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:07.177 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:07.193 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:06:07.193 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:06:07.193 INFO:teuthology.orchestra.run.smithi129.stdout: "complaint_time": 30, 2024-09-21T10:06:07.193 INFO:teuthology.orchestra.run.smithi129.stdout: "num_blocked_ops": 0 2024-09-21T10:06:07.193 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:06:07.199 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:07.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:07.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:07.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:07.322 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:07.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:07.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:07.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:07.428 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:06:07 smithi107 podman[190669]: 2024-09-21 10:06:07.130611384 +0000 UTC m=+2.105049235 container died 378577ff0aedbcd86ce6c31c663fb74a9731f523a43c0d88baf7211c935b2aff (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=main-62da54b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-21T10:06:07.428 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:07 smithi107 ceph-mon[102221]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:06:07.428 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:07 smithi107 ceph-mon[102221]: osdmap e202: 8 total, 7 up, 8 in 2024-09-21T10:06:07.428 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:07 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:07.428 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:07 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:07.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:07 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:07.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:07 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:06:07.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:07 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:07.429 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:07 smithi107 ceph-mon[105022]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:06:07.429 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:07 smithi107 ceph-mon[105022]: osdmap e202: 8 total, 7 up, 8 in 2024-09-21T10:06:07.429 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:07 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:07.429 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:07 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:07.429 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:07 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:07.429 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:07 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:06:07.429 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:07 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:07.456 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:07.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:07.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:07.491 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:07 smithi129 ceph-mon[88471]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:06:07.491 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:07 smithi129 ceph-mon[88471]: osdmap e202: 8 total, 7 up, 8 in 2024-09-21T10:06:07.491 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:07 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:07.491 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:07 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:07.491 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:07 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:07.491 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:07 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:06:07.491 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:07 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:07.558 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:07.596 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:07.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:07.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:07.658 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 58 16 95429 95413 102.803 19.9375 0.00834427 0.0097221 2024-09-21T10:06:07.697 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:07.731 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:07.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:07.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:07.770 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:06:07 smithi107 podman[190669]: 2024-09-21 10:06:07.642627399 +0000 UTC m=+2.617065250 container cleanup 378577ff0aedbcd86ce6c31c663fb74a9731f523a43c0d88baf7211c935b2aff (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-21T10:06:07.770 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:06:07 smithi107 bash[190669]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0 2024-09-21T10:06:07.824 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:06:07.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:07.860 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:07.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:07.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:07.962 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:07.999 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:08.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:08.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:08.101 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:08.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:08.158 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:08.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:08.259 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:08.275 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:06:08.313 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:08.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:08.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:08.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:08.440 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:08.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:08.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:08.542 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:08.774 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 59 16 96602 96586 102.303 73.3125 0.00649677 0.00976986 2024-09-21T10:06:08.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:08 smithi107 ceph-mon[102221]: pgmap v720: 109 pgs: 8 peering, 12 stale+active+clean, 89 active+clean; 5.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 56 MiB/s wr, 903 op/s 2024-09-21T10:06:08.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:08 smithi107 ceph-mon[102221]: osdmap e203: 8 total, 7 up, 8 in 2024-09-21T10:06:08.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:08 smithi107 ceph-mon[102221]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:06:08.775 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:08 smithi107 ceph-mon[105022]: pgmap v720: 109 pgs: 8 peering, 12 stale+active+clean, 89 active+clean; 5.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 56 MiB/s wr, 903 op/s 2024-09-21T10:06:08.775 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:08 smithi107 ceph-mon[105022]: osdmap e203: 8 total, 7 up, 8 in 2024-09-21T10:06:08.775 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:08 smithi107 ceph-mon[105022]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:06:08.776 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:08 smithi129 ceph-mon[88471]: pgmap v720: 109 pgs: 8 peering, 12 stale+active+clean, 89 active+clean; 5.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 56 MiB/s wr, 903 op/s 2024-09-21T10:06:08.776 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:08 smithi129 ceph-mon[88471]: osdmap e203: 8 total, 7 up, 8 in 2024-09-21T10:06:08.776 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:08 smithi129 ceph-mon[88471]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:06:08.818 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:08.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:08.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:08.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:08.946 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:08.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:08.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:09.048 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:09.087 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:09.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:09.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:09.092 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:06:08 smithi107 podman[191051]: 2024-09-21 10:06:08.766975576 +0000 UTC m=+1.691963275 container remove 378577ff0aedbcd86ce6c31c663fb74a9731f523a43c0d88baf7211c935b2aff (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=main-62da54b) 2024-09-21T10:06:09.189 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:09.217 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:09.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:09.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:09.319 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:09.359 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:09.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:09.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:09.461 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:09.501 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:09.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:09.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:09.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:09.629 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:09.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:09.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:09.658 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:06:09.657232+0000 min lat: 0.000982552 max lat: 1.05751 avg lat: 0.00981618 2024-09-21T10:06:09.658 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:06:09.658 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 60 16 97777 97761 101.822 73.4375 0.00580176 0.00981618 2024-09-21T10:06:09.731 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:09.769 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:09.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:09.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:09.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:09.901 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:09.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:09.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:10.003 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:10.044 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:10.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:10.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:10.147 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:10.181 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:10.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:10.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:10.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:10.310 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:10.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:10.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:10.402 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:06:10.412 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:10.452 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:10.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:10.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:10.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:10.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:10 smithi107 ceph-mon[102221]: pgmap v722: 109 pgs: 3 active+undersized, 19 peering, 5 stale+active+clean, 3 active+undersized+degraded, 79 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 4.4 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 5864/283734 objects degraded (2.067%); 6.7 KiB/s, 0 objects/s recovering 2024-09-21T10:06:10.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:10 smithi107 ceph-mon[102221]: Health check failed: Degraded data redundancy: 5864/283734 objects degraded (2.067%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:10.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:10 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2976622385' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:10.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:10 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:10.577 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:06:10 smithi107 podman[191392]: 2024-09-21 10:06:10.228836768 +0000 UTC m=+0.026457939 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:06:10.578 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:06:10 smithi107 podman[191392]: 2024-09-21 10:06:10.383168134 +0000 UTC m=+0.180789299 container create 17402d49bcdf631376ce51847f37c5297ba4b5810e7eab3a660a5064dd73469c (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-deactivate, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:06:10.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:10 smithi107 ceph-mon[105022]: pgmap v722: 109 pgs: 3 active+undersized, 19 peering, 5 stale+active+clean, 3 active+undersized+degraded, 79 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 4.4 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 5864/283734 objects degraded (2.067%); 6.7 KiB/s, 0 objects/s recovering 2024-09-21T10:06:10.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:10 smithi107 ceph-mon[105022]: Health check failed: Degraded data redundancy: 5864/283734 objects degraded (2.067%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:10.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:10 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2976622385' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:10.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:10 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:10.581 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:10.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:10.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:10.658 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 61 16 99041 99025 101.448 79 0.0167766 0.00985253 2024-09-21T10:06:10.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:10 smithi129 ceph-mon[88471]: pgmap v722: 109 pgs: 3 active+undersized, 19 peering, 5 stale+active+clean, 3 active+undersized+degraded, 79 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 4.4 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 5864/283734 objects degraded (2.067%); 6.7 KiB/s, 0 objects/s recovering 2024-09-21T10:06:10.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:10 smithi129 ceph-mon[88471]: Health check failed: Degraded data redundancy: 5864/283734 objects degraded (2.067%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:10.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:10 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2976622385' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:10.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:10 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:10.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:06:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:06:10.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:06:10.683 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:10.734 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:10.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:10.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:10.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:10.861 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:10.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:10.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:10.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:10.993 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:10.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:10.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:11.095 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:11.109 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:06:10 smithi107 podman[191392]: 2024-09-21 10:06:10.855215661 +0000 UTC m=+0.652836845 container init 17402d49bcdf631376ce51847f37c5297ba4b5810e7eab3a660a5064dd73469c (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:06:11.109 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:06:10 smithi107 podman[191392]: 2024-09-21 10:06:10.863721908 +0000 UTC m=+0.661343073 container start 17402d49bcdf631376ce51847f37c5297ba4b5810e7eab3a660a5064dd73469c (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:06:11.109 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:06:11 smithi107 podman[191392]: 2024-09-21 10:06:10.996836233 +0000 UTC m=+0.794457395 container attach 17402d49bcdf631376ce51847f37c5297ba4b5810e7eab3a660a5064dd73469c (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-62da54b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-21T10:06:11.143 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:11.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:11.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:11.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:11.270 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:11.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:11.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:11.372 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:11.413 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:06:11.413 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 20, 2024-09-21T10:06:11.413 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 600, 2024-09-21T10:06:11.413 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:06:11.413 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.413 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45320.0:168886 8.5 8:ad5a418f:::benchmark_data_smithi107_135256_object43002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-21T10:06:11.413 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T09:57:39.997418+0000", 2024-09-21T10:06:11.413 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 511.410817786, 2024-09-21T10:06:11.413 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.152188359, 2024-09-21T10:06:11.413 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:06:11.414 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:11.414 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:06:11.414 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45320", 2024-09-21T10:06:11.414 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:06:11.414 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 168886 2024-09-21T10:06:11.414 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.414 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:06:11.414 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.414 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:06:11.414 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:39.997418+0000", 2024-09-21T10:06:11.414 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.414 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.414 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.414 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:06:11.414 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:39.997418+0000", 2024-09-21T10:06:11.415 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.415 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.415 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.415 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:06:11.415 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:39.997416+0000", 2024-09-21T10:06:11.415 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:06:11.415 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.415 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.415 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:06:11.415 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:39.997423+0000", 2024-09-21T10:06:11.415 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.0809999999999996e-06 2024-09-21T10:06:11.415 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.415 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.415 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:06:11.415 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:39.997439+0000", 2024-09-21T10:06:11.415 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.6160000000000001e-05 2024-09-21T10:06:11.416 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.416 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.416 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:06:11.416 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:39.997445+0000", 2024-09-21T10:06:11.416 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5250000000000002e-06 2024-09-21T10:06:11.416 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.416 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.416 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.416 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:39.997700+0000", 2024-09-21T10:06:11.416 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000254496 2024-09-21T10:06:11.416 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.416 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.416 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:06:11.416 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:39.997971+0000", 2024-09-21T10:06:11.416 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00027095199999999998 2024-09-21T10:06:11.417 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.417 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.417 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:06:11.417 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:39.998301+0000", 2024-09-21T10:06:11.417 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00033006300000000001 2024-09-21T10:06:11.417 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.417 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.417 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.417 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:40.010093+0000", 2024-09-21T10:06:11.417 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.01179174 2024-09-21T10:06:11.417 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.417 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.417 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:06:11.417 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:40.010154+0000", 2024-09-21T10:06:11.417 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1750999999999999e-05 2024-09-21T10:06:11.417 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.418 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.418 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.418 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:40.149492+0000", 2024-09-21T10:06:11.418 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.13933725699999999 2024-09-21T10:06:11.418 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.418 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.418 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:06:11.418 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:40.149560+0000", 2024-09-21T10:06:11.418 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8685999999999999e-05 2024-09-21T10:06:11.418 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.418 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.418 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:06:11.419 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:40.149606+0000", 2024-09-21T10:06:11.419 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5586e-05 2024-09-21T10:06:11.419 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.419 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.419 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.419 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.419 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.419 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45320.0:174265 8.9 8:979a20b7:::benchmark_data_smithi107_135256_object48381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-21T10:06:11.419 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T09:57:46.167336+0000", 2024-09-21T10:06:11.419 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 505.24089933699997, 2024-09-21T10:06:11.419 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.15221831499999999, 2024-09-21T10:06:11.419 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:06:11.419 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:11.419 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:06:11.419 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45320", 2024-09-21T10:06:11.420 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:06:11.420 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 174265 2024-09-21T10:06:11.420 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.420 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:06:11.420 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.420 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:06:11.420 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.167336+0000", 2024-09-21T10:06:11.420 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.420 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.420 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.420 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:06:11.420 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.167336+0000", 2024-09-21T10:06:11.420 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.420 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.421 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.421 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:06:11.421 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.167335+0000", 2024-09-21T10:06:11.421 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:06:11.421 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.421 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.421 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:06:11.421 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.167339+0000", 2024-09-21T10:06:11.421 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1980000000000003e-06 2024-09-21T10:06:11.421 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.421 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.421 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:06:11.421 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.167349+0000", 2024-09-21T10:06:11.422 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.4599999999999992e-06 2024-09-21T10:06:11.422 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.422 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.422 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:06:11.422 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.167354+0000", 2024-09-21T10:06:11.422 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7149999999999997e-06 2024-09-21T10:06:11.422 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.422 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.422 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.422 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.174887+0000", 2024-09-21T10:06:11.422 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0075330290000000001 2024-09-21T10:06:11.422 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.422 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.422 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:06:11.422 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.174995+0000", 2024-09-21T10:06:11.423 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010876500000000001 2024-09-21T10:06:11.423 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.423 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.423 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:11.423 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.175103+0000", 2024-09-21T10:06:11.423 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010806000000000001 2024-09-21T10:06:11.423 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.423 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.423 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:06:11.423 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.183967+0000", 2024-09-21T10:06:11.423 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0088636070000000008 2024-09-21T10:06:11.423 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.424 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.424 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.424 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.317267+0000", 2024-09-21T10:06:11.424 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.13329955099999999 2024-09-21T10:06:11.424 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.424 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.424 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.424 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.319431+0000", 2024-09-21T10:06:11.424 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0021647480000000002 2024-09-21T10:06:11.424 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.424 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.424 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:06:11.424 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.319516+0000", 2024-09-21T10:06:11.424 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.4722999999999995e-05 2024-09-21T10:06:11.424 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.425 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.425 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:06:11.425 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.319554+0000", 2024-09-21T10:06:11.425 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8192999999999998e-05 2024-09-21T10:06:11.425 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.425 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.425 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.425 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.425 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.425 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45320.0:174283 8.5 8:a87d4287:::benchmark_data_smithi107_135256_object48399:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-21T10:06:11.425 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T09:57:46.186467+0000", 2024-09-21T10:06:11.425 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 505.22176804700001, 2024-09-21T10:06:11.425 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.15315779800000001, 2024-09-21T10:06:11.425 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:06:11.425 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:11.426 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:06:11.426 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45320", 2024-09-21T10:06:11.426 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3933601208", 2024-09-21T10:06:11.426 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 174283 2024-09-21T10:06:11.426 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.426 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:06:11.426 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.427 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:06:11.427 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.186467+0000", 2024-09-21T10:06:11.427 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.427 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.427 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.427 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:06:11.427 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.186467+0000", 2024-09-21T10:06:11.427 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.427 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.427 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.427 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:06:11.427 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.186467+0000", 2024-09-21T10:06:11.427 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:11.427 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.428 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.428 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:06:11.428 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.186470+0000", 2024-09-21T10:06:11.428 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0620000000000001e-06 2024-09-21T10:06:11.428 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.428 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.428 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:06:11.428 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.186476+0000", 2024-09-21T10:06:11.428 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0769999999999996e-06 2024-09-21T10:06:11.428 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.428 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.428 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:06:11.428 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.186480+0000", 2024-09-21T10:06:11.428 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.9750000000000001e-06 2024-09-21T10:06:11.428 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.429 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.429 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.429 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.319168+0000", 2024-09-21T10:06:11.429 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.13268806399999999 2024-09-21T10:06:11.429 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.429 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.429 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:06:11.429 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.319397+0000", 2024-09-21T10:06:11.429 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000228808 2024-09-21T10:06:11.429 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.429 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.429 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:06:11.429 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.319523+0000", 2024-09-21T10:06:11.429 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012617600000000001 2024-09-21T10:06:11.429 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.430 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.430 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:06:11.430 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.334673+0000", 2024-09-21T10:06:11.430 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.015149770999999999 2024-09-21T10:06:11.430 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.430 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.430 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.430 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.339265+0000", 2024-09-21T10:06:11.430 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0045926220000000002 2024-09-21T10:06:11.430 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.430 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.430 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.430 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.339398+0000", 2024-09-21T10:06:11.431 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000132294 2024-09-21T10:06:11.431 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.431 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.431 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:06:11.431 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.339502+0000", 2024-09-21T10:06:11.431 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010426500000000001 2024-09-21T10:06:11.431 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.431 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.431 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:06:11.431 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T09:57:46.339625+0000", 2024-09-21T10:06:11.431 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000123158 2024-09-21T10:06:11.431 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.431 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.431 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.432 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.432 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.432 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75905 10.b 10:df33c370:::benchmark_data_smithi107_184135_object75904:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:06:11.432 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.065777+0000", 2024-09-21T10:06:11.432 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 22.342458499999999, 2024-09-21T10:06:11.432 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.253965828, 2024-09-21T10:06:11.432 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:06:11.432 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:11.432 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:06:11.432 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:06:11.432 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:11.432 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75905 2024-09-21T10:06:11.432 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.432 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:06:11.432 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.433 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:06:11.433 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.065777+0000", 2024-09-21T10:06:11.433 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.433 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.433 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.433 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:06:11.433 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.065777+0000", 2024-09-21T10:06:11.433 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.433 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.433 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.433 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:06:11.433 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.065776+0000", 2024-09-21T10:06:11.433 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:06:11.433 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.434 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.434 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:06:11.434 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.065834+0000", 2024-09-21T10:06:11.434 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7809000000000002e-05 2024-09-21T10:06:11.434 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.434 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.434 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:06:11.434 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.065842+0000", 2024-09-21T10:06:11.434 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.1480000000000006e-06 2024-09-21T10:06:11.434 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.434 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.434 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:06:11.434 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.065847+0000", 2024-09-21T10:06:11.435 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8230000000000002e-06 2024-09-21T10:06:11.435 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.435 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.435 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.435 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.069759+0000", 2024-09-21T10:06:11.435 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0039118210000000002 2024-09-21T10:06:11.435 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.435 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.435 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:06:11.435 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.069860+0000", 2024-09-21T10:06:11.435 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010105199999999999 2024-09-21T10:06:11.435 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.435 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.435 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:06:11.436 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.069976+0000", 2024-09-21T10:06:11.436 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011605899999999999 2024-09-21T10:06:11.436 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.436 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.436 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.436 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.076580+0000", 2024-09-21T10:06:11.436 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.006603996 2024-09-21T10:06:11.436 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.436 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.436 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:06:11.436 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.080422+0000", 2024-09-21T10:06:11.436 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00384287 2024-09-21T10:06:11.436 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.437 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.437 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.437 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.319673+0000", 2024-09-21T10:06:11.437 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.239250085 2024-09-21T10:06:11.437 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.437 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.437 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:06:11.437 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.319715+0000", 2024-09-21T10:06:11.437 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2398999999999998e-05 2024-09-21T10:06:11.437 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.437 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.437 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:06:11.437 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.319743+0000", 2024-09-21T10:06:11.437 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7674999999999998e-05 2024-09-21T10:06:11.438 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.438 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.438 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.438 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.438 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.438 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75913 10.b 10:d690a9d4:::benchmark_data_smithi107_184135_object75912:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:06:11.438 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.071867+0000", 2024-09-21T10:06:11.438 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 22.336368390000001, 2024-09-21T10:06:11.438 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.247963933, 2024-09-21T10:06:11.438 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:06:11.438 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:11.438 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:06:11.438 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:06:11.438 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:11.439 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75913 2024-09-21T10:06:11.439 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.439 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:06:11.439 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.439 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:06:11.439 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.071867+0000", 2024-09-21T10:06:11.439 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.439 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.439 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.439 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:06:11.439 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.071867+0000", 2024-09-21T10:06:11.439 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.439 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.440 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.440 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:06:11.440 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.071866+0000", 2024-09-21T10:06:11.440 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:06:11.440 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.440 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.440 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:06:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:11.441 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:11.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:11.442 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.072104+0000", 2024-09-21T10:06:11.442 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00023810199999999999 2024-09-21T10:06:11.442 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.442 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.442 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:06:11.442 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.072116+0000", 2024-09-21T10:06:11.442 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.171e-05 2024-09-21T10:06:11.442 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.442 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.442 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:06:11.442 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.072120+0000", 2024-09-21T10:06:11.443 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3089999999999997e-06 2024-09-21T10:06:11.443 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.443 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.443 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.443 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.072705+0000", 2024-09-21T10:06:11.443 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00058508799999999999 2024-09-21T10:06:11.443 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.443 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.443 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:06:11.443 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.072824+0000", 2024-09-21T10:06:11.443 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000119162 2024-09-21T10:06:11.443 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.443 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.444 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:06:11.444 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.072952+0000", 2024-09-21T10:06:11.444 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012725599999999999 2024-09-21T10:06:11.444 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.444 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.444 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:06:11.444 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.080427+0000", 2024-09-21T10:06:11.444 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.007475359 2024-09-21T10:06:11.444 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.444 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.444 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.444 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.081859+0000", 2024-09-21T10:06:11.444 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001432399 2024-09-21T10:06:11.444 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.445 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.445 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.445 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.319776+0000", 2024-09-21T10:06:11.445 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23791665200000001 2024-09-21T10:06:11.445 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.445 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.445 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:06:11.445 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.319809+0000", 2024-09-21T10:06:11.445 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2802000000000002e-05 2024-09-21T10:06:11.445 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.445 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.445 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:06:11.445 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.319831+0000", 2024-09-21T10:06:11.445 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1970000000000001e-05 2024-09-21T10:06:11.446 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.446 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.446 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.446 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.446 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.446 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75917 10.0 10:04450359:::benchmark_data_smithi107_184135_object75916:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:06:11.446 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.074183+0000", 2024-09-21T10:06:11.446 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 22.334052152000002, 2024-09-21T10:06:11.446 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25040411400000001, 2024-09-21T10:06:11.446 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:06:11.446 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:11.446 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:06:11.446 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:06:11.446 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:11.446 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75917 2024-09-21T10:06:11.447 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.447 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:06:11.447 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.447 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:06:11.447 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074183+0000", 2024-09-21T10:06:11.447 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.447 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.447 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.447 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:06:11.447 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074183+0000", 2024-09-21T10:06:11.447 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.447 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.447 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.447 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:06:11.448 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074182+0000", 2024-09-21T10:06:11.448 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:06:11.448 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.448 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.448 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:06:11.448 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074235+0000", 2024-09-21T10:06:11.448 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2439999999999999e-05 2024-09-21T10:06:11.448 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.448 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.448 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:06:11.448 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074245+0000", 2024-09-21T10:06:11.448 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0042e-05 2024-09-21T10:06:11.448 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.448 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.448 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:06:11.449 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074260+0000", 2024-09-21T10:06:11.449 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5486999999999999e-05 2024-09-21T10:06:11.449 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.449 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.449 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.449 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074390+0000", 2024-09-21T10:06:11.449 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012976099999999999 2024-09-21T10:06:11.449 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.449 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.449 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:06:11.449 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074502+0000", 2024-09-21T10:06:11.449 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011167000000000001 2024-09-21T10:06:11.449 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.449 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.449 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:06:11.450 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074616+0000", 2024-09-21T10:06:11.450 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000114481 2024-09-21T10:06:11.450 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.450 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.450 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:06:11.450 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.078114+0000", 2024-09-21T10:06:11.450 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003497413 2024-09-21T10:06:11.450 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.450 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.450 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.450 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.078322+0000", 2024-09-21T10:06:11.450 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00020845799999999999 2024-09-21T10:06:11.450 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.451 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.451 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.451 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.324518+0000", 2024-09-21T10:06:11.451 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.246196096 2024-09-21T10:06:11.451 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.451 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.451 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:06:11.451 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.324553+0000", 2024-09-21T10:06:11.451 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4459999999999999e-05 2024-09-21T10:06:11.451 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.451 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.451 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:06:11.451 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.324587+0000", 2024-09-21T10:06:11.451 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4591999999999997e-05 2024-09-21T10:06:11.452 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.452 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.452 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.452 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.452 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.452 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75920 10.b 10:da23dad2:::benchmark_data_smithi107_184135_object75919:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:06:11.452 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.074352+0000", 2024-09-21T10:06:11.452 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 22.333883800999999, 2024-09-21T10:06:11.452 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.24755237099999999, 2024-09-21T10:06:11.452 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:06:11.452 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:11.452 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:06:11.452 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:06:11.452 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:11.452 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75920 2024-09-21T10:06:11.453 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.453 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:06:11.453 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.453 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:06:11.453 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074352+0000", 2024-09-21T10:06:11.453 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.453 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.453 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.453 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:06:11.453 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074352+0000", 2024-09-21T10:06:11.453 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.453 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.454 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.454 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:06:11.454 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074351+0000", 2024-09-21T10:06:11.454 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:11.454 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.454 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.454 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:06:11.454 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074380+0000", 2024-09-21T10:06:11.454 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.8773999999999999e-05 2024-09-21T10:06:11.454 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.454 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.454 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:06:11.454 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074387+0000", 2024-09-21T10:06:11.454 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7440000000000001e-06 2024-09-21T10:06:11.455 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.455 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.455 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:06:11.455 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074390+0000", 2024-09-21T10:06:11.455 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8539999999999999e-06 2024-09-21T10:06:11.455 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.455 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.455 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.455 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.077335+0000", 2024-09-21T10:06:11.455 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0029443210000000002 2024-09-21T10:06:11.455 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.455 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.455 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:06:11.455 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.077456+0000", 2024-09-21T10:06:11.455 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000121178 2024-09-21T10:06:11.456 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.456 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.456 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:06:11.456 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.077583+0000", 2024-09-21T10:06:11.456 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012657599999999999 2024-09-21T10:06:11.456 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.456 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.456 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:06:11.456 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.080555+0000", 2024-09-21T10:06:11.456 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0029719970000000001 2024-09-21T10:06:11.456 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.456 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.456 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.456 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.081977+0000", 2024-09-21T10:06:11.457 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0014220159999999999 2024-09-21T10:06:11.457 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.457 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.457 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.457 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.321814+0000", 2024-09-21T10:06:11.457 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23983771600000001 2024-09-21T10:06:11.457 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.457 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.457 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:06:11.457 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.321867+0000", 2024-09-21T10:06:11.457 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2587999999999999e-05 2024-09-21T10:06:11.457 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.457 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.457 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:06:11.457 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.321904+0000", 2024-09-21T10:06:11.458 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7024999999999999e-05 2024-09-21T10:06:11.458 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.458 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.458 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.458 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.458 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.458 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75931 10.0 10:02f906db:::benchmark_data_smithi107_184135_object75930:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:06:11.458 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.084617+0000", 2024-09-21T10:06:11.458 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 22.323618536000001, 2024-09-21T10:06:11.458 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.24220968000000001, 2024-09-21T10:06:11.458 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:06:11.458 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:11.459 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:06:11.459 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:06:11.459 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:11.459 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75931 2024-09-21T10:06:11.459 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.459 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:06:11.459 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.459 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:06:11.459 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.084617+0000", 2024-09-21T10:06:11.459 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.459 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.459 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.459 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:06:11.459 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.084617+0000", 2024-09-21T10:06:11.460 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.460 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.460 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.460 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:06:11.460 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.084616+0000", 2024-09-21T10:06:11.460 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:06:11.460 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.460 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.460 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:06:11.460 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.084703+0000", 2024-09-21T10:06:11.460 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6710000000000002e-05 2024-09-21T10:06:11.460 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.460 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.460 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:06:11.461 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.084715+0000", 2024-09-21T10:06:11.461 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2319e-05 2024-09-21T10:06:11.461 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.461 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.461 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:06:11.461 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.084721+0000", 2024-09-21T10:06:11.461 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2339999999999999e-06 2024-09-21T10:06:11.461 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.461 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.461 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.461 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.084768+0000", 2024-09-21T10:06:11.461 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6458999999999997e-05 2024-09-21T10:06:11.461 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.462 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.462 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:06:11.462 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.084925+0000", 2024-09-21T10:06:11.462 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015770700000000001 2024-09-21T10:06:11.462 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.462 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.462 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:06:11.462 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.085032+0000", 2024-09-21T10:06:11.462 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000106436 2024-09-21T10:06:11.462 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.462 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.462 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.462 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.090962+0000", 2024-09-21T10:06:11.462 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0059300890000000004 2024-09-21T10:06:11.463 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.463 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.463 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:06:11.463 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.091650+0000", 2024-09-21T10:06:11.463 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00068853600000000001 2024-09-21T10:06:11.463 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.463 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.463 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.463 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.326760+0000", 2024-09-21T10:06:11.463 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23510992 2024-09-21T10:06:11.463 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.463 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.463 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:06:11.463 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.326794+0000", 2024-09-21T10:06:11.463 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3200000000000001e-05 2024-09-21T10:06:11.464 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.464 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.464 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:06:11.464 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.326826+0000", 2024-09-21T10:06:11.464 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.294e-05 2024-09-21T10:06:11.464 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.464 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.464 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.464 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.464 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.464 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75937 10.0 10:0c60d6b6:::benchmark_data_smithi107_184135_object75936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:06:11.464 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.087439+0000", 2024-09-21T10:06:11.464 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 22.320796369, 2024-09-21T10:06:11.464 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.239497922, 2024-09-21T10:06:11.465 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:06:11.465 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:11.465 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:06:11.465 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:06:11.465 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:11.465 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75937 2024-09-21T10:06:11.465 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.465 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:06:11.465 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.465 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:06:11.465 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.087439+0000", 2024-09-21T10:06:11.465 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.465 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.465 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.465 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:06:11.466 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.087439+0000", 2024-09-21T10:06:11.466 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.466 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.466 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.466 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:06:11.466 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.087439+0000", 2024-09-21T10:06:11.466 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:11.466 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.466 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.466 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:06:11.466 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.087460+0000", 2024-09-21T10:06:11.466 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1469999999999999e-05 2024-09-21T10:06:11.466 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.466 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.467 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:06:11.467 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.087464+0000", 2024-09-21T10:06:11.467 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4270000000000001e-06 2024-09-21T10:06:11.467 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.467 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.467 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:06:11.467 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.087467+0000", 2024-09-21T10:06:11.467 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.553e-06 2024-09-21T10:06:11.467 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.467 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.467 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.467 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.087675+0000", 2024-09-21T10:06:11.468 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000207906 2024-09-21T10:06:11.468 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.468 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.468 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:06:11.468 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.087772+0000", 2024-09-21T10:06:11.468 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.6756999999999995e-05 2024-09-21T10:06:11.468 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.468 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.468 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:06:11.468 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.087892+0000", 2024-09-21T10:06:11.468 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000120674 2024-09-21T10:06:11.468 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.468 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.469 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.469 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.091273+0000", 2024-09-21T10:06:11.469 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003380802 2024-09-21T10:06:11.469 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.469 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.469 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:06:11.469 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.091668+0000", 2024-09-21T10:06:11.470 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000394931 2024-09-21T10:06:11.470 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.470 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.470 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.470 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.326870+0000", 2024-09-21T10:06:11.470 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.235202349 2024-09-21T10:06:11.470 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.470 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.470 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:06:11.470 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.326900+0000", 2024-09-21T10:06:11.470 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.9692e-05 2024-09-21T10:06:11.470 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.470 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.471 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:06:11.471 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.326937+0000", 2024-09-21T10:06:11.471 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6739000000000002e-05 2024-09-21T10:06:11.471 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.471 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.471 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.471 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.471 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.471 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75939 10.b 10:d4e3f307:::benchmark_data_smithi107_184135_object75938:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:06:11.471 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.089180+0000", 2024-09-21T10:06:11.471 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 22.319055111000001, 2024-09-21T10:06:11.471 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23286099399999999, 2024-09-21T10:06:11.472 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:06:11.472 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:11.472 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:06:11.472 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:06:11.472 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:11.472 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75939 2024-09-21T10:06:11.472 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.472 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:06:11.472 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.472 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:06:11.472 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.089180+0000", 2024-09-21T10:06:11.472 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.472 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.472 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.472 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:06:11.473 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.089180+0000", 2024-09-21T10:06:11.473 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.473 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.473 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.473 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:06:11.473 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.089179+0000", 2024-09-21T10:06:11.473 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:06:11.473 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.473 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.473 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:06:11.473 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.089230+0000", 2024-09-21T10:06:11.473 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1076999999999999e-05 2024-09-21T10:06:11.473 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.473 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.473 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:06:11.474 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.089247+0000", 2024-09-21T10:06:11.474 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.6375000000000002e-05 2024-09-21T10:06:11.474 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.474 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.474 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:06:11.474 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.089252+0000", 2024-09-21T10:06:11.474 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3900000000000001e-06 2024-09-21T10:06:11.474 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.474 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.474 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.474 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.090734+0000", 2024-09-21T10:06:11.474 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0014821789999999999 2024-09-21T10:06:11.474 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.475 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.475 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:06:11.475 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.090992+0000", 2024-09-21T10:06:11.475 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000258266 2024-09-21T10:06:11.475 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.475 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.475 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:06:11.475 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.091136+0000", 2024-09-21T10:06:11.475 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014327400000000001 2024-09-21T10:06:11.475 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.475 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.475 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:06:11.475 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.096462+0000", 2024-09-21T10:06:11.475 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0053259020000000004 2024-09-21T10:06:11.475 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.476 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.476 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.476 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.096500+0000", 2024-09-21T10:06:11.476 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8269000000000002e-05 2024-09-21T10:06:11.476 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.476 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.476 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.476 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.321962+0000", 2024-09-21T10:06:11.476 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.225462052 2024-09-21T10:06:11.476 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.476 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.476 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:06:11.476 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.322007+0000", 2024-09-21T10:06:11.476 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5615000000000003e-05 2024-09-21T10:06:11.477 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.477 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.477 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:06:11.477 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.322041+0000", 2024-09-21T10:06:11.477 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3738999999999997e-05 2024-09-21T10:06:11.477 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.477 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.477 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.477 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.477 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.477 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75942 10.b 10:dbcbfc02:::benchmark_data_smithi107_184135_object75941:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:06:11.477 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.092155+0000", 2024-09-21T10:06:11.477 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 22.316079822999999, 2024-09-21T10:06:11.477 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23667438599999999, 2024-09-21T10:06:11.478 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:06:11.478 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:11.478 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:06:11.478 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:06:11.478 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:11.478 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75942 2024-09-21T10:06:11.478 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.478 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:06:11.478 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.478 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:06:11.478 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.092155+0000", 2024-09-21T10:06:11.478 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.478 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.478 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.478 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:06:11.479 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.092155+0000", 2024-09-21T10:06:11.479 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.479 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.479 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.479 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:06:11.479 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.092154+0000", 2024-09-21T10:06:11.479 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:06:11.479 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.479 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.479 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:06:11.479 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.092207+0000", 2024-09-21T10:06:11.479 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3140999999999998e-05 2024-09-21T10:06:11.479 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.479 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.480 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:06:11.480 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.092220+0000", 2024-09-21T10:06:11.480 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2267e-05 2024-09-21T10:06:11.480 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.480 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.480 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:06:11.480 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.092226+0000", 2024-09-21T10:06:11.480 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2049999999999999e-06 2024-09-21T10:06:11.480 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.480 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.480 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.480 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.093742+0000", 2024-09-21T10:06:11.480 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0015158979999999999 2024-09-21T10:06:11.480 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.481 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.481 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:06:11.481 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.093852+0000", 2024-09-21T10:06:11.481 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011042700000000001 2024-09-21T10:06:11.481 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.481 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.481 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:06:11.481 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.093994+0000", 2024-09-21T10:06:11.481 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000141877 2024-09-21T10:06:11.481 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.481 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.481 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:06:11.481 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.096506+0000", 2024-09-21T10:06:11.482 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0025119230000000001 2024-09-21T10:06:11.482 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.482 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.482 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.482 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.096533+0000", 2024-09-21T10:06:11.482 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7387000000000001e-05 2024-09-21T10:06:11.482 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.482 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.482 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.482 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.328717+0000", 2024-09-21T10:06:11.482 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.232183787 2024-09-21T10:06:11.482 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.483 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.483 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:06:11.483 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.328793+0000", 2024-09-21T10:06:11.483 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.5401000000000005e-05 2024-09-21T10:06:11.483 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.483 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.483 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:06:11.483 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.328830+0000", 2024-09-21T10:06:11.483 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7215000000000002e-05 2024-09-21T10:06:11.483 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.483 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.483 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.483 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.483 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.484 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75961 10.0 10:0cdaf837:::benchmark_data_smithi107_184135_object75960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:06:11.484 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.112151+0000", 2024-09-21T10:06:11.484 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 22.296084665999999, 2024-09-21T10:06:11.484 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21492383800000001, 2024-09-21T10:06:11.484 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:06:11.484 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:11.484 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:06:11.484 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:06:11.484 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:11.484 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75961 2024-09-21T10:06:11.484 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.484 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:06:11.484 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.484 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:06:11.485 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112151+0000", 2024-09-21T10:06:11.485 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.485 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.485 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.485 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:06:11.485 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112151+0000", 2024-09-21T10:06:11.485 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.485 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.485 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.485 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:06:11.485 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112150+0000", 2024-09-21T10:06:11.485 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:06:11.485 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.485 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.486 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:06:11.486 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112206+0000", 2024-09-21T10:06:11.486 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6582000000000001e-05 2024-09-21T10:06:11.486 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.486 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.486 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:06:11.486 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112216+0000", 2024-09-21T10:06:11.486 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.7939999999999997e-06 2024-09-21T10:06:11.486 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.486 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.486 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:06:11.486 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112221+0000", 2024-09-21T10:06:11.486 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7709999999999997e-06 2024-09-21T10:06:11.486 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.487 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.487 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.487 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112281+0000", 2024-09-21T10:06:11.487 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0659000000000003e-05 2024-09-21T10:06:11.487 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.487 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.487 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:06:11.487 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112395+0000", 2024-09-21T10:06:11.487 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000113105 2024-09-21T10:06:11.487 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.487 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.487 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:06:11.487 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112490+0000", 2024-09-21T10:06:11.488 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.5764e-05 2024-09-21T10:06:11.488 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.488 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.488 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:06:11.488 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.116884+0000", 2024-09-21T10:06:11.488 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0043935170000000004 2024-09-21T10:06:11.488 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.488 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.488 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.488 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.116920+0000", 2024-09-21T10:06:11.488 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6304999999999999e-05 2024-09-21T10:06:11.488 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.488 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.488 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.489 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.326981+0000", 2024-09-21T10:06:11.489 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21006082300000001 2024-09-21T10:06:11.489 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.489 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.489 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:06:11.489 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.327031+0000", 2024-09-21T10:06:11.489 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9876000000000002e-05 2024-09-21T10:06:11.489 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.489 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.489 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:06:11.489 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.327074+0000", 2024-09-21T10:06:11.489 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3628000000000003e-05 2024-09-21T10:06:11.489 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.489 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.489 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.490 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.490 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.490 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75962 10.0 10:098753a3:::benchmark_data_smithi107_184135_object75961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:06:11.490 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.112237+0000", 2024-09-21T10:06:11.490 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 22.295997836000002, 2024-09-21T10:06:11.490 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.218636311, 2024-09-21T10:06:11.490 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:06:11.490 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:11.490 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:06:11.490 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:06:11.490 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:11.490 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75962 2024-09-21T10:06:11.490 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.491 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:06:11.491 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.491 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:06:11.491 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112237+0000", 2024-09-21T10:06:11.491 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.491 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.491 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.491 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:06:11.491 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112237+0000", 2024-09-21T10:06:11.491 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.491 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.491 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.492 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:06:11.492 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112237+0000", 2024-09-21T10:06:11.492 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:06:11.492 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.492 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.492 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:06:11.492 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112298+0000", 2024-09-21T10:06:11.492 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1126999999999999e-05 2024-09-21T10:06:11.492 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.492 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.492 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:06:11.492 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112309+0000", 2024-09-21T10:06:11.492 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1430000000000001e-05 2024-09-21T10:06:11.492 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.493 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.493 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:06:11.493 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112316+0000", 2024-09-21T10:06:11.493 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4099999999999996e-06 2024-09-21T10:06:11.493 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.493 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.493 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.493 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.114729+0000", 2024-09-21T10:06:11.493 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0024129080000000001 2024-09-21T10:06:11.493 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.493 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.493 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:06:11.493 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.114803+0000", 2024-09-21T10:06:11.493 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.4084000000000004e-05 2024-09-21T10:06:11.493 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.494 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.494 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:06:11.494 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.114936+0000", 2024-09-21T10:06:11.494 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013269900000000001 2024-09-21T10:06:11.494 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.494 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.494 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.494 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.117014+0000", 2024-09-21T10:06:11.494 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002078608 2024-09-21T10:06:11.494 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.494 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.494 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:06:11.494 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.117082+0000", 2024-09-21T10:06:11.494 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7507000000000003e-05 2024-09-21T10:06:11.495 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.495 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.495 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.495 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.330702+0000", 2024-09-21T10:06:11.495 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21362015200000001 2024-09-21T10:06:11.495 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.495 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.495 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:06:11.496 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.330774+0000", 2024-09-21T10:06:11.496 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2231999999999996e-05 2024-09-21T10:06:11.496 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.496 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.496 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:06:11.496 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.330874+0000", 2024-09-21T10:06:11.496 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.9710000000000006e-05 2024-09-21T10:06:11.496 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.496 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.496 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.496 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.496 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.496 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75976 10.0 10:09b9cbd0:::benchmark_data_smithi107_184135_object75975:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:06:11.496 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.149392+0000", 2024-09-21T10:06:11.497 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 22.258842848, 2024-09-21T10:06:11.497 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.18157615999999999, 2024-09-21T10:06:11.498 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:06:11.498 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:11.498 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:06:11.498 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:06:11.498 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:11.498 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75976 2024-09-21T10:06:11.498 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.498 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:06:11.499 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.499 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:06:11.499 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.149392+0000", 2024-09-21T10:06:11.499 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.499 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.499 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.499 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:06:11.499 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.149392+0000", 2024-09-21T10:06:11.499 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.499 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.499 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.499 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:06:11.499 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.149392+0000", 2024-09-21T10:06:11.500 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:06:11.500 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.500 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.500 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:06:11.500 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.149444+0000", 2024-09-21T10:06:11.500 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2596000000000003e-05 2024-09-21T10:06:11.500 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.500 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.501 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:06:11.501 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.149456+0000", 2024-09-21T10:06:11.501 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1612e-05 2024-09-21T10:06:11.501 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.501 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.501 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:06:11.501 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.149460+0000", 2024-09-21T10:06:11.501 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.51e-06 2024-09-21T10:06:11.501 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.501 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.501 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.501 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.149497+0000", 2024-09-21T10:06:11.501 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6826000000000003e-05 2024-09-21T10:06:11.502 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.502 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.502 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:06:11.502 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.149592+0000", 2024-09-21T10:06:11.502 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.5091000000000001e-05 2024-09-21T10:06:11.502 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.502 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.502 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:06:11.502 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.149679+0000", 2024-09-21T10:06:11.502 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.7156000000000004e-05 2024-09-21T10:06:11.503 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.503 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.503 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:06:11.503 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.151695+0000", 2024-09-21T10:06:11.503 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0020155490000000002 2024-09-21T10:06:11.503 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.503 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.503 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.503 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.151900+0000", 2024-09-21T10:06:11.503 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00020464800000000001 2024-09-21T10:06:11.504 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.504 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.504 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.504 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.330904+0000", 2024-09-21T10:06:11.504 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.17900391900000001 2024-09-21T10:06:11.504 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.504 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.504 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:06:11.504 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.330941+0000", 2024-09-21T10:06:11.504 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7086999999999999e-05 2024-09-21T10:06:11.505 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.505 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.505 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:06:11.505 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.330969+0000", 2024-09-21T10:06:11.505 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.8031999999999999e-05 2024-09-21T10:06:11.505 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.505 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.505 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.505 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.505 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.505 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:77455 10.b 10:d8f81762:::benchmark_data_smithi107_184135_object77454:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:06:11.506 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:50.583334+0000", 2024-09-21T10:06:11.506 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 20.824901486000002, 2024-09-21T10:06:11.506 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.161961298, 2024-09-21T10:06:11.506 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:06:11.506 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:11.506 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:06:11.506 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:06:11.506 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:11.507 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 77455 2024-09-21T10:06:11.507 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.507 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:06:11.507 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.507 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:06:11.507 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.583334+0000", 2024-09-21T10:06:11.507 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.507 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.507 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.507 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:06:11.508 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.583334+0000", 2024-09-21T10:06:11.508 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.508 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.508 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.508 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:06:11.508 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.583332+0000", 2024-09-21T10:06:11.508 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:06:11.508 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.508 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.508 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:06:11.509 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.583389+0000", 2024-09-21T10:06:11.509 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.681e-05 2024-09-21T10:06:11.509 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.509 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.509 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:06:11.509 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.583398+0000", 2024-09-21T10:06:11.509 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.9840000000000007e-06 2024-09-21T10:06:11.509 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.509 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.509 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:06:11.509 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.583403+0000", 2024-09-21T10:06:11.509 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1599999999999997e-06 2024-09-21T10:06:11.509 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.509 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.510 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.510 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.586469+0000", 2024-09-21T10:06:11.510 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003065679 2024-09-21T10:06:11.510 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.510 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.510 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:06:11.510 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.586584+0000", 2024-09-21T10:06:11.510 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000114918 2024-09-21T10:06:11.510 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.510 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.510 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:06:11.510 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.586703+0000", 2024-09-21T10:06:11.510 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000118984 2024-09-21T10:06:11.510 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.511 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.511 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:06:11.511 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.589380+0000", 2024-09-21T10:06:11.511 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002676819 2024-09-21T10:06:11.511 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.511 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.511 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.511 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.591861+0000", 2024-09-21T10:06:11.511 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0024809020000000001 2024-09-21T10:06:11.511 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.511 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.511 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.512 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.745236+0000", 2024-09-21T10:06:11.512 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.15337482 2024-09-21T10:06:11.512 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.512 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.512 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:06:11.512 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.745272+0000", 2024-09-21T10:06:11.512 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6826999999999998e-05 2024-09-21T10:06:11.512 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.512 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.512 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:06:11.512 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.745295+0000", 2024-09-21T10:06:11.512 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.2796e-05 2024-09-21T10:06:11.512 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.512 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.513 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.513 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.513 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.513 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:77460 10.a 10:5c8c5bfa:::benchmark_data_smithi107_184135_object77459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:06:11.513 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:50.587204+0000", 2024-09-21T10:06:11.513 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 20.821031173000002, 2024-09-21T10:06:11.513 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.16102037699999999, 2024-09-21T10:06:11.513 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:06:11.513 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:11.513 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:06:11.513 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:06:11.513 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:11.513 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 77460 2024-09-21T10:06:11.513 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.513 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:06:11.513 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.514 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:06:11.514 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.587204+0000", 2024-09-21T10:06:11.514 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.514 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.514 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.514 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:06:11.514 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.587204+0000", 2024-09-21T10:06:11.514 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.514 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.514 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.514 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:06:11.514 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.587203+0000", 2024-09-21T10:06:11.514 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:06:11.514 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.514 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.515 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:06:11.515 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.587232+0000", 2024-09-21T10:06:11.515 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.9666e-05 2024-09-21T10:06:11.515 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.515 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.515 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:06:11.515 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.587241+0000", 2024-09-21T10:06:11.515 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6789999999999994e-06 2024-09-21T10:06:11.515 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.515 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.515 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:06:11.515 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.587245+0000", 2024-09-21T10:06:11.515 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2309999999999998e-06 2024-09-21T10:06:11.515 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.515 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.515 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.516 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.587949+0000", 2024-09-21T10:06:11.516 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000703705 2024-09-21T10:06:11.516 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.516 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.516 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:06:11.516 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.588519+0000", 2024-09-21T10:06:11.516 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00057045499999999999 2024-09-21T10:06:11.516 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.516 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.516 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:11.516 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.588675+0000", 2024-09-21T10:06:11.516 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015525700000000001 2024-09-21T10:06:11.516 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.516 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.516 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:06:11.517 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.591805+0000", 2024-09-21T10:06:11.517 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0031302349999999999 2024-09-21T10:06:11.517 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.517 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.517 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.517 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.595329+0000", 2024-09-21T10:06:11.517 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003523862 2024-09-21T10:06:11.517 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.517 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.517 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.517 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.748128+0000", 2024-09-21T10:06:11.517 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.152799562 2024-09-21T10:06:11.517 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.517 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.517 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:06:11.517 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.748187+0000", 2024-09-21T10:06:11.518 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9233e-05 2024-09-21T10:06:11.518 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.518 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.518 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:06:11.518 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.748225+0000", 2024-09-21T10:06:11.518 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7110000000000002e-05 2024-09-21T10:06:11.518 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.518 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.518 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.518 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.518 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.518 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:77469 10.a 10:50d5c447:::benchmark_data_smithi107_184135_object77468:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:06:11.518 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:50.591777+0000", 2024-09-21T10:06:11.518 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 20.816458081, 2024-09-21T10:06:11.518 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.15654409399999999, 2024-09-21T10:06:11.519 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:06:11.519 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:11.519 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:06:11.519 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:06:11.519 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:11.519 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 77469 2024-09-21T10:06:11.519 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.519 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:06:11.519 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.519 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:06:11.519 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.591777+0000", 2024-09-21T10:06:11.519 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.519 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.519 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.519 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:06:11.520 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.591777+0000", 2024-09-21T10:06:11.520 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.520 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.520 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.520 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:06:11.520 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.591776+0000", 2024-09-21T10:06:11.520 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:06:11.520 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.520 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.520 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:06:11.520 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.591807+0000", 2024-09-21T10:06:11.520 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0667000000000003e-05 2024-09-21T10:06:11.520 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.520 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.521 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:06:11.521 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.591816+0000", 2024-09-21T10:06:11.521 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.9080000000000001e-06 2024-09-21T10:06:11.521 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.521 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.521 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:06:11.521 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.591821+0000", 2024-09-21T10:06:11.521 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1120000000000004e-06 2024-09-21T10:06:11.521 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.521 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.521 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.521 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.591892+0000", 2024-09-21T10:06:11.521 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.1469000000000003e-05 2024-09-21T10:06:11.521 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.522 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.522 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:06:11.522 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.592004+0000", 2024-09-21T10:06:11.522 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000111716 2024-09-21T10:06:11.522 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.522 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.522 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:06:11.522 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.592144+0000", 2024-09-21T10:06:11.522 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014016199999999999 2024-09-21T10:06:11.522 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.522 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.522 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:06:11.522 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.595333+0000", 2024-09-21T10:06:11.522 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0031884439999999999 2024-09-21T10:06:11.523 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.523 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.523 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.523 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.596307+0000", 2024-09-21T10:06:11.523 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00097400700000000004 2024-09-21T10:06:11.523 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.523 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.523 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.523 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.748256+0000", 2024-09-21T10:06:11.523 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.15194943799999999 2024-09-21T10:06:11.523 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.523 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.523 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:06:11.523 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.748293+0000", 2024-09-21T10:06:11.523 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7421e-05 2024-09-21T10:06:11.523 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.524 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.524 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:06:11.524 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:50.748321+0000", 2024-09-21T10:06:11.524 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7880000000000001e-05 2024-09-21T10:06:11.524 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.524 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.524 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.524 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.524 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.524 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95103 10.d 10:b3c31b59:::benchmark_data_smithi107_184135_object95102:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T10:06:11.524 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.391296+0000", 2024-09-21T10:06:11.524 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 5.0169388970000002, 2024-09-21T10:06:11.524 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0224283670000001, 2024-09-21T10:06:11.524 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:06:11.525 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:06:11.525 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:06:11.525 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:06:11.525 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:11.525 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95103 2024-09-21T10:06:11.525 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.525 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:06:11.525 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.525 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:06:11.525 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.391296+0000", 2024-09-21T10:06:11.525 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.525 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.525 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.526 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:06:11.526 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.391296+0000", 2024-09-21T10:06:11.526 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.526 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.526 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.526 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:06:11.526 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.391295+0000", 2024-09-21T10:06:11.526 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:06:11.526 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.526 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.526 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:06:11.526 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.391325+0000", 2024-09-21T10:06:11.526 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0151e-05 2024-09-21T10:06:11.527 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.527 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.527 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:06:11.527 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.391336+0000", 2024-09-21T10:06:11.527 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0859000000000001e-05 2024-09-21T10:06:11.527 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.527 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.527 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:06:11.527 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.391342+0000", 2024-09-21T10:06:11.527 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7860000000000002e-06 2024-09-21T10:06:11.527 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.527 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.527 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.527 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.391596+0000", 2024-09-21T10:06:11.528 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00025450400000000001 2024-09-21T10:06:11.528 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.528 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.528 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:06:11.528 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.391763+0000", 2024-09-21T10:06:11.528 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00016683600000000001 2024-09-21T10:06:11.528 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.528 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.528 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:06:11.528 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.392522+0000", 2024-09-21T10:06:11.528 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00075902900000000004 2024-09-21T10:06:11.528 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.529 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.529 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.529 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407862+0000", 2024-09-21T10:06:11.529 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.015340354 2024-09-21T10:06:11.529 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.529 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.529 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:06:11.529 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407869+0000", 2024-09-21T10:06:11.529 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7900000000000002e-06 2024-09-21T10:06:11.529 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.529 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.529 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.530 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.413651+0000", 2024-09-21T10:06:11.530 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0057817849999999 2024-09-21T10:06:11.530 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.530 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.530 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:06:11.530 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.413725+0000", 2024-09-21T10:06:11.530 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3898000000000003e-05 2024-09-21T10:06:11.530 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.530 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.530 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.530 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.530 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.530 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95107 10.d 10:b5c30b02:::benchmark_data_smithi107_184135_object95106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-21T10:06:11.531 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.406752+0000", 2024-09-21T10:06:11.531 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 5.0014836420000002, 2024-09-21T10:06:11.531 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0153879509999999, 2024-09-21T10:06:11.531 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:06:11.531 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:11.531 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:06:11.531 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:06:11.531 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:11.531 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95107 2024-09-21T10:06:11.531 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.531 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:06:11.531 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.531 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:06:11.531 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.406752+0000", 2024-09-21T10:06:11.532 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.532 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.532 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.532 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:06:11.532 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.406752+0000", 2024-09-21T10:06:11.532 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.532 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.532 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.532 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:06:11.532 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.406750+0000", 2024-09-21T10:06:11.532 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:06:11.532 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.532 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.532 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:06:11.533 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.406783+0000", 2024-09-21T10:06:11.533 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2200999999999999e-05 2024-09-21T10:06:11.533 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.533 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.533 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:06:11.533 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.406793+0000", 2024-09-21T10:06:11.533 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0349999999999999e-05 2024-09-21T10:06:11.533 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.533 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.533 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:06:11.534 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.406799+0000", 2024-09-21T10:06:11.534 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.784e-06 2024-09-21T10:06:11.534 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.534 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.534 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.534 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.408181+0000", 2024-09-21T10:06:11.534 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001381902 2024-09-21T10:06:11.534 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.534 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.534 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:06:11.534 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.408186+0000", 2024-09-21T10:06:11.535 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5890000000000002e-06 2024-09-21T10:06:11.535 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.535 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.535 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.535 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.413748+0000", 2024-09-21T10:06:11.535 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.005562045 2024-09-21T10:06:11.535 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.535 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.535 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:06:11.535 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.413853+0000", 2024-09-21T10:06:11.535 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000105066 2024-09-21T10:06:11.536 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.536 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.536 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 4", 2024-09-21T10:06:11.536 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.413969+0000", 2024-09-21T10:06:11.536 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000116049 2024-09-21T10:06:11.536 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.536 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.536 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:06:11.536 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.421082+0000", 2024-09-21T10:06:11.536 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0071127689999999997 2024-09-21T10:06:11.536 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.537 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.537 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.537 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.421997+0000", 2024-09-21T10:06:11.537 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00091438399999999997 2024-09-21T10:06:11.537 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.537 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.537 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:06:11.537 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.422058+0000", 2024-09-21T10:06:11.537 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1338000000000001e-05 2024-09-21T10:06:11.537 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.538 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.538 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:06:11.538 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.422140+0000", 2024-09-21T10:06:11.538 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.1667000000000003e-05 2024-09-21T10:06:11.538 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.538 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.538 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.538 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.538 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.538 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95110 10.b 10:de41cee3:::benchmark_data_smithi107_184135_object95109:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-21T10:06:11.538 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.407299+0000", 2024-09-21T10:06:11.539 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 5.0009364649999997, 2024-09-21T10:06:11.539 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0202640590000001, 2024-09-21T10:06:11.539 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:06:11.539 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:11.539 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:06:11.539 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:06:11.539 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:11.539 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95110 2024-09-21T10:06:11.539 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.540 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:06:11.540 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.540 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:06:11.540 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407299+0000", 2024-09-21T10:06:11.540 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.540 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.540 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.540 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:06:11.540 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407299+0000", 2024-09-21T10:06:11.540 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:06:11.540 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.540 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.540 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:06:11.541 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407298+0000", 2024-09-21T10:06:11.541 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:06:11.541 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.541 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.541 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:06:11.541 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407389+0000", 2024-09-21T10:06:11.541 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.1442999999999998e-05 2024-09-21T10:06:11.541 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.541 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.542 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:06:11.542 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407400+0000", 2024-09-21T10:06:11.542 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0757999999999999e-05 2024-09-21T10:06:11.542 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.542 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.542 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:06:11.542 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407417+0000", 2024-09-21T10:06:11.542 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.7297999999999999e-05 2024-09-21T10:06:11.542 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.542 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.542 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.542 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407552+0000", 2024-09-21T10:06:11.542 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013406 2024-09-21T10:06:11.542 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.543 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.543 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:06:11.543 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407558+0000", 2024-09-21T10:06:11.543 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8229999999999997e-06 2024-09-21T10:06:11.543 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.543 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.543 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:11.544 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:06:11.544 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.422095+0000", 2024-09-21T10:06:11.544 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0145365420000001 2024-09-21T10:06:11.545 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.545 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.545 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:06:11.545 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.422194+0000", 2024-09-21T10:06:11.545 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.9501999999999997e-05 2024-09-21T10:06:11.545 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.545 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.545 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1", 2024-09-21T10:06:11.545 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.422335+0000", 2024-09-21T10:06:11.545 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014081 2024-09-21T10:06:11.546 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.546 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.546 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:06:11.546 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.426372+0000", 2024-09-21T10:06:11.546 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0040367700000000003 2024-09-21T10:06:11.546 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.546 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.546 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:11.546 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.427451+0000", 2024-09-21T10:06:11.547 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0010793350000000001 2024-09-21T10:06:11.547 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.547 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.547 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:06:11.547 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.427490+0000", 2024-09-21T10:06:11.547 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8217999999999999e-05 2024-09-21T10:06:11.547 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:06:11.547 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:06:11.547 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:06:11.548 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.427563+0000", 2024-09-21T10:06:11.548 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3359e-05 2024-09-21T10:06:11.548 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.548 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.548 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.548 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:06:11.548 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:06:11.548 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:06:11.548 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:11 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:11.548 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:11 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:06:11.549 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:11 smithi129 ceph-mon[88471]: osdmap e204: 8 total, 7 up, 8 in 2024-09-21T10:06:11.549 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:11.549 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:11.593 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:11.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:11.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:11.604 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:11 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:11.604 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:11 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:06:11.605 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:11 smithi107 ceph-mon[102221]: osdmap e204: 8 total, 7 up, 8 in 2024-09-21T10:06:11.605 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:11.605 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:11.605 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:11 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:11.605 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:11 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:06:11.605 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:11 smithi107 ceph-mon[105022]: osdmap e204: 8 total, 7 up, 8 in 2024-09-21T10:06:11.605 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:11.606 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:11.658 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 62 16 100045 100029 100.824 62.75 0.0186127 0.00991305 2024-09-21T10:06:11.695 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:11.735 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:11.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:11.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:11.836 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:11.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:06:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:06:11] "GET /metrics HTTP/1.1" 200 36785 "" "Prometheus/2.51.0" 2024-09-21T10:06:11.879 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:11.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:11.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:11.964 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:06:11.981 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:12.030 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:12.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:12.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:12.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:12.158 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:12.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:12.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:12.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:12.285 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:12.296 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:12.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:12.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:12.398 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:12.442 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:12.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:12.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:12.475 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:06:12.544 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:12.581 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:12.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:12.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:12.658 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 63 16 101056 101040 100.226 63.1875 0.0128931 0.00997245 2024-09-21T10:06:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:12 smithi129 ceph-mon[88471]: pgmap v724: 109 pgs: 13 active+undersized, 19 peering, 10 active+undersized+degraded, 67 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 5.7 KiB/s rd, 50 MiB/s wr, 806 op/s; 23998/287622 objects degraded (8.344%); 8.7 KiB/s, 0 objects/s recovering 2024-09-21T10:06:12.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:12.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:12.714 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:12.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:12.815 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:12.855 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:12.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:12.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:12.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:12 smithi107 ceph-mon[105022]: pgmap v724: 109 pgs: 13 active+undersized, 19 peering, 10 active+undersized+degraded, 67 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 5.7 KiB/s rd, 50 MiB/s wr, 806 op/s; 23998/287622 objects degraded (8.344%); 8.7 KiB/s, 0 objects/s recovering 2024-09-21T10:06:12.859 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:06:12 smithi107 podman[191392]: 2024-09-21 10:06:12.602382683 +0000 UTC m=+2.400003848 container died 17402d49bcdf631376ce51847f37c5297ba4b5810e7eab3a660a5064dd73469c (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-21T10:06:12.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:12 smithi107 ceph-mon[102221]: pgmap v724: 109 pgs: 13 active+undersized, 19 peering, 10 active+undersized+degraded, 67 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 5.7 KiB/s rd, 50 MiB/s wr, 806 op/s; 23998/287622 objects degraded (8.344%); 8.7 KiB/s, 0 objects/s recovering 2024-09-21T10:06:12.957 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:12.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:12.993 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:12.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:13.094 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:13.136 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:13.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:13.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:13.238 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:13.277 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:13.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:13.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:13.378 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:13.414 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:13.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:13.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:13.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.0.service' 2024-09-21T10:06:13.490 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:06:13.490 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-21T10:06:13.515 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:13.554 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:13.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:13.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:13.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:06:13.656 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:13.658 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 64 16 102255 102239 99.8312 74.9375 0.00463259 0.010012 2024-09-21T10:06:13.664 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:13 smithi107 ceph-mon[102221]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-09-21T10:06:13.664 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:13 smithi107 ceph-mon[105022]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-09-21T10:06:13.690 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:13.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:13.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:13.792 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:13.828 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:13.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:13.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:13 smithi129 ceph-mon[88471]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-09-21T10:06:13.930 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:13.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:13.967 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:13.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:14.068 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:14.100 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:14.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:14.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:14.201 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:14.244 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:14.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:14.346 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:14.376 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:14.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:14.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:14.478 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:14.507 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:14.528 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:14.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:14.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:14.531 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:14.630 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:14.658 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 65 16 103482 103466 99.4751 76.6875 0.00844309 0.0100487 2024-09-21T10:06:14.674 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:14.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:14.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:14.776 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:14.788 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:14 smithi107 ceph-mon[102221]: pgmap v725: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 5.0 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 62122/300153 objects degraded (20.697%); 7.7 KiB/s, 0 objects/s recovering 2024-09-21T10:06:14.788 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:14 smithi107 ceph-mon[105022]: pgmap v725: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 5.0 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 62122/300153 objects degraded (20.697%); 7.7 KiB/s, 0 objects/s recovering 2024-09-21T10:06:14.811 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:14.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:14.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:14.913 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:14 smithi129 ceph-mon[88471]: pgmap v725: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.1 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 5.0 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 62122/300153 objects degraded (20.697%); 7.7 KiB/s, 0 objects/s recovering 2024-09-21T10:06:14.954 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:14.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:14.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:15.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:15.081 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:15.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:15.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:15.183 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:15.223 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:15.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:15.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:15.325 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:15.349 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:15.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:15.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:15.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:15.480 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:15.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:15.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:15.570 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:06:15.571 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:06:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T10:06:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:06:15.571 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:06:15.571 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:06:15.572 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:06:15.582 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:15.593 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:06:15.593 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:06:15.615 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:06:15.615 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:06:15.624 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:15.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:15.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:15.639 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:06:15.640 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:06:15.658 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 66 16 104683 104667 99.1051 75.0625 0.0113002 0.0100852 2024-09-21T10:06:15.670 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:06:15.671 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:06:15.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:15.750 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:15.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:15.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:15.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:15.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:15 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 63189/304506 objects degraded (20.751%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:15.861 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:15 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/77327696' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:15.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:15 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 63189/304506 objects degraded (20.751%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:15.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:15 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/77327696' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:15.880 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:15.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:15.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:15 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 63189/304506 objects degraded (20.751%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:15 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/77327696' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:15.981 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:16.003 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:16.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:16.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:16.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:16.130 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:16.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:16.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:16.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:16.259 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:16.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:16.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:16.360 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:16.387 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:16.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:16.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:16.490 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:16.523 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:16.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:16.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:16.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:16.650 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:16.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:16.658 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 67 16 105830 105814 98.6958 71.6875 0.017861 0.0101275 2024-09-21T10:06:16.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:16.781 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:16.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:16.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:16 smithi107 ceph-mon[102221]: pgmap v726: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 4.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 63189/304506 objects degraded (20.751%); 6.5 KiB/s, 0 objects/s recovering 2024-09-21T10:06:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:16 smithi107 ceph-mon[105022]: pgmap v726: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 4.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 63189/304506 objects degraded (20.751%); 6.5 KiB/s, 0 objects/s recovering 2024-09-21T10:06:16.882 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:16.914 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:06:16.914 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:06:16.915 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:16.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:16.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:16.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:16 smithi129 ceph-mon[88471]: pgmap v726: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 4.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s; 63189/304506 objects degraded (20.751%); 6.5 KiB/s, 0 objects/s recovering 2024-09-21T10:06:17.017 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:17.052 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:17.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:17.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:17.155 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:17.191 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:17.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:17.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:17.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:17.321 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:17.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:17.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:17.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:17.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:17.457 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:17.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:17.558 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:17.610 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:17.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:17.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:17.658 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 68 16 107023 107007 98.3408 74.5625 0.00978557 0.0101644 2024-09-21T10:06:17.712 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:17.753 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:17.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:17.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:17.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:17.882 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:17.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:17.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:17.984 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:18.021 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:18.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:18.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:18.123 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:18.159 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:18.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:18.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:18.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:18.288 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:18.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:18.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:18.390 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:18.429 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:18.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:18.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:18.491 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:06:18.491 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:06:18.491 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-21T10:06:18.491 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:06:18.492 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:06:18.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-21T10:06:18.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:18.565 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:18.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:18.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:18.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:18 smithi107 ceph-mon[102221]: pgmap v727: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 3.5 KiB/s rd, 50 MiB/s wr, 803 op/s; 63528/305523 objects degraded (20.793%); 5.4 KiB/s, 0 objects/s recovering 2024-09-21T10:06:18.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:18 smithi107 ceph-mon[105022]: pgmap v727: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 3.5 KiB/s rd, 50 MiB/s wr, 803 op/s; 63528/305523 objects degraded (20.793%); 5.4 KiB/s, 0 objects/s recovering 2024-09-21T10:06:18.659 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 69 16 108219 108203 97.9984 74.75 0.0168814 0.0101992 2024-09-21T10:06:18.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:18.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:18.700 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:18.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:18.800 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:18.801 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:06:18.801 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:06:18.831 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:18.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:18.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:18.915 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:06:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:18 smithi129 ceph-mon[88471]: pgmap v727: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 3.5 KiB/s rd, 50 MiB/s wr, 803 op/s; 63528/305523 objects degraded (20.793%); 5.4 KiB/s, 0 objects/s recovering 2024-09-21T10:06:18.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:18.985 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:18.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:18.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:19.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:19.118 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:19.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:19.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:19.220 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:19.258 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:19.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:19.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:19.361 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:19.397 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:19.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:19.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:19.499 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:19.528 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:19.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:19.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:19.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:06:19.630 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:19.659 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 70 16 109422 109406 97.6725 75.1875 0.00778995 0.010234 2024-09-21T10:06:19.669 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:19.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:19.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:19.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:19.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:19.811 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:19.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:19.912 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:19.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:19.943 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:19.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:19.945 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:19.974 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:20.044 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:20.069 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:20.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:20.171 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:20.206 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:20.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:20.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:20.307 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:20.348 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:20.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:20.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:20.450 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:20.484 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:20.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:20.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:20.511 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:06:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:06:20.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:06:20.586 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:20.613 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:20.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:20.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:20.659 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 71 16 110564 110548 97.302 71.375 0.0233807 0.0102732 2024-09-21T10:06:20.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:20.744 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:20.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:20.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:20.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:20.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:20 smithi107 ceph-mon[102221]: pgmap v728: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s; 65738/317664 objects degraded (20.694%) 2024-09-21T10:06:20.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:20 smithi107 ceph-mon[105022]: pgmap v728: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s; 65738/317664 objects degraded (20.694%) 2024-09-21T10:06:20.875 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:20.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:20.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:20 smithi129 ceph-mon[88471]: pgmap v728: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s; 65738/317664 objects degraded (20.694%) 2024-09-21T10:06:20.976 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:21.014 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:21.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:21.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:21.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:21.147 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:21.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:21.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:21.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:21.280 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:21.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:21.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:21.382 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:21.414 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:21.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:21.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:21.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:21.544 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:21.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:21.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:21.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:21.659 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 72 16 111718 111702 96.9523 72.125 0.0155979 0.0103095 2024-09-21T10:06:21.675 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:21.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:21.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:21.777 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:21.815 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:21.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:21.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:21.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:21 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 66867/322245 objects degraded (20.750%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:21 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 66867/322245 objects degraded (20.750%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:21.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:06:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:06:21] "GET /metrics HTTP/1.1" 200 36785 "" "Prometheus/2.51.0" 2024-09-21T10:06:21.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:21.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:21 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 66867/322245 objects degraded (20.750%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:21.942 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:21.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:21.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:22.044 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:22.079 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:22.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:22.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:22.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:22.207 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:22.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:22.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:22.309 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:22.346 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:22.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:22.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:22.448 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:22.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:22.485 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:22.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:22.586 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:22.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:22.623 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:22.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:22.659 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 73 16 112834 112818 96.5796 69.75 0.0058794 0.010349 2024-09-21T10:06:22.724 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:22.770 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:22.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:22.853 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:22 smithi129 ceph-mon[88471]: pgmap v729: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 66867/322245 objects degraded (20.750%) 2024-09-21T10:06:22.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:22 smithi107 ceph-mon[102221]: pgmap v729: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 66867/322245 objects degraded (20.750%) 2024-09-21T10:06:22.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:22 smithi107 ceph-mon[105022]: pgmap v729: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 66867/322245 objects degraded (20.750%) 2024-09-21T10:06:22.872 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:22.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:22.907 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:22.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:23.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:23.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:23.052 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:23.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:23.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:23.178 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:23.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:23.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:23.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:23.307 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:23.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:23.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:23.409 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:23.448 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:23.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:23.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:23.550 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:23.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:23.589 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:23.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:23.659 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 74 16 113967 113951 96.2312 70.8125 0.00349837 0.010383 2024-09-21T10:06:23.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:23.720 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:23.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:23.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:23.821 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:23.853 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:23.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:23.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:23.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:23.980 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:23.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:23.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:24.082 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:24.110 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:24.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:24.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:24.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:24.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:24.244 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:24.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:24.345 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:24.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:24.383 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:24.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:24.484 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:24.521 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:24.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:24.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:24.533 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:24.623 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:24.660 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 75 16 115121 115105 95.9098 72.125 0.0244174 0.0104223 2024-09-21T10:06:24.664 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:24.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:24.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:24.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:24 smithi107 ceph-mon[102221]: pgmap v730: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s; 69422/335409 objects degraded (20.698%) 2024-09-21T10:06:24.674 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:24 smithi107 ceph-mon[105022]: pgmap v730: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s; 69422/335409 objects degraded (20.698%) 2024-09-21T10:06:24.767 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:24.807 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:24.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:24.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:24.909 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:24.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:24 smithi129 ceph-mon[88471]: pgmap v730: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s; 69422/335409 objects degraded (20.698%) 2024-09-21T10:06:24.954 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:24.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:24.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:25.056 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:25.090 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:25.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:25.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:25.192 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:25.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:25.231 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:25.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:25.332 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:25.368 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:25.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:25.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:25.470 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:25.509 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:25.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:25.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:25.586 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:06:25.610 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:25.649 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:25.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:25.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:25.660 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 76 16 116263 116247 95.5868 71.375 0.0138014 0.0104572 2024-09-21T10:06:25.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:25.778 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:25.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:25.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:25 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/607020370' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:06:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:25 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:06:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:25 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/607020370' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:06:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:25 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:06:25.879 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:25.906 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:25.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:25.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:25 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/607020370' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:06:25.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:25 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:06:26.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:26.038 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:26.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:26.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:26.140 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:26.169 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:26.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:26.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:26.271 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:26.312 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:26.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:26.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:26.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:26.438 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:26.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:26.539 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:26.576 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:26.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:26.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:26.660 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 77 16 117407 117391 95.274 71.5 0.0171216 0.0104914 2024-09-21T10:06:26.678 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:26.689 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:26 smithi107 ceph-mon[102221]: pgmap v731: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 70496/339669 objects degraded (20.754%) 2024-09-21T10:06:26.689 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:26 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:26.689 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:26 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:06:26.690 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:26 smithi107 ceph-mon[102221]: osdmap e205: 8 total, 7 up, 8 in 2024-09-21T10:06:26.690 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:26.690 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:26.690 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:26 smithi107 ceph-mon[105022]: pgmap v731: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 70496/339669 objects degraded (20.754%) 2024-09-21T10:06:26.691 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:26 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:26.691 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:26 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:06:26.691 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:26 smithi107 ceph-mon[105022]: osdmap e205: 8 total, 7 up, 8 in 2024-09-21T10:06:26.691 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:26.691 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:26.713 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:26.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:26.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:26.815 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:26.862 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:26.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:26.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:26 smithi129 ceph-mon[88471]: pgmap v731: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 70496/339669 objects degraded (20.754%) 2024-09-21T10:06:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:26 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:26 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:06:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:26 smithi129 ceph-mon[88471]: osdmap e205: 8 total, 7 up, 8 in 2024-09-21T10:06:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:26.964 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:26.990 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:26.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:26.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:27.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:27.118 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:27.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:27.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:27.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:27.252 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:27.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:27.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:27.264 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T10:06:27.285 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:06:27.286 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:06:27.311 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:06:27.311 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:06:27.329 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:06:27.330 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:06:27.354 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:27.362 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:06:27.362 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:06:27.389 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:06:27.390 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:06:27.392 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:27.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:27.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:27.416 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:06:27.417 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:06:27.435 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:06:27.436 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:06:27.493 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:27.529 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:27.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:27.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:27.630 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:27.660 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 78 16 118532 118516 94.9539 70.3125 0.0159023 0.0105265 2024-09-21T10:06:27.670 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:27.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:27.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:27.772 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:27.816 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:27.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:27.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:27.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:27 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2560949149' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:27.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:27 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2560949149' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:27.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:27 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2560949149' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:27.945 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:27.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:27.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:28.047 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:28.071 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:28.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:28.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:28.173 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:28.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:28.213 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:28.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:28.314 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:28.349 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:28.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:28.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:28.450 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:28.487 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:28.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:28.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:28.588 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:28.620 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:28.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:28.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:28.660 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 79 16 119667 119651 94.6498 70.9375 0.0159235 0.0105607 2024-09-21T10:06:28.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:28.750 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:28.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:28.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:28.852 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:28.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:28 smithi107 ceph-mon[102221]: pgmap v733: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 70879/340818 objects degraded (20.797%) 2024-09-21T10:06:28.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:28 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 70879/340818 objects degraded (20.797%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:28.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:28 smithi107 ceph-mon[105022]: pgmap v733: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 70879/340818 objects degraded (20.797%) 2024-09-21T10:06:28.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:28 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 70879/340818 objects degraded (20.797%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:28.899 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:28.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:28.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:28.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:28 smithi129 ceph-mon[88471]: pgmap v733: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 70879/340818 objects degraded (20.797%) 2024-09-21T10:06:28.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:28 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 70879/340818 objects degraded (20.797%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:28.950 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:06:28.950 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:06:29.001 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:29.025 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:29.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:29.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:29.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:06:29.127 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:29.158 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:29.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:29.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:29.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:29.284 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:29.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:29.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:29.386 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:29.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:29.427 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:29.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:29.528 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:29.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:29.562 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:29.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:29.660 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:06:29.659202+0000 min lat: 0.000982552 max lat: 1.05751 avg lat: 0.0105937 2024-09-21T10:06:29.660 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:06:29.660 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 80 16 120806 120790 94.3565 71.1875 0.00542341 0.0105937 2024-09-21T10:06:29.663 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:29.698 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:29.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:29.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:29.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:29.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:29.828 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:29.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:29.929 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:29.961 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:29.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:29.962 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:29.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:30.063 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:30.109 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:30.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:30.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:30.211 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:30.245 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:30.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:30.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:30.347 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:30.378 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:30.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:30.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:30.480 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:30.506 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:30.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:30.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:30.525 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:06:30.525 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:06:30.526 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:06:30.591 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:06:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:06:30.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:06:30.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:30.644 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:30.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:30.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:30.660 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 81 16 121916 121900 94.048 69.375 0.0121138 0.0106279 2024-09-21T10:06:30.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:30.776 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:30.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:30.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:30.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:30 smithi107 ceph-mon[102221]: pgmap v734: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s; 73581/355080 objects degraded (20.722%) 2024-09-21T10:06:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:30 smithi107 ceph-mon[105022]: pgmap v734: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s; 73581/355080 objects degraded (20.722%) 2024-09-21T10:06:30.877 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:30.903 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:30.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:30.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:30.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:30 smithi129 ceph-mon[88471]: pgmap v734: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s; 73581/355080 objects degraded (20.722%) 2024-09-21T10:06:30.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:06:31.005 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:31.049 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:31.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:31.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:31.151 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:31.185 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:31.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:31.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:31.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:31.317 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:31.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:31.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:31.419 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:31.455 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:31.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:31.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:31.530 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:06:31.530 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:06:31.530 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-09-21T10:06:31.530 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-09-21T10:06:31.530 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:06:31.557 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:31.596 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:31.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:31.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:31.660 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 82 16 123060 123044 93.773 71.5 0.0066074 0.0106592 2024-09-21T10:06:31.698 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:31.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:31.740 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:31.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:31.782 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:06:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:06:31] "GET /metrics HTTP/1.1" 200 36778 "" "Prometheus/2.51.0" 2024-09-21T10:06:31.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:31.869 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:31.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:31.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:31.971 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:32.006 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:32.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:32.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:32.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:32.136 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:32.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:32.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:32.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:32.267 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:32.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:32.367 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:32.392 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:32.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:32.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:32.494 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:32.532 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:32.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:32.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:32.568 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:06:32.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:32.660 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 83 16 124187 124171 93.4918 70.4375 0.0290324 0.010692 2024-09-21T10:06:32.663 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:32.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:32.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:32.765 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:32.793 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:32 smithi107 ceph-mon[105022]: pgmap v735: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s; 73932/356133 objects degraded (20.760%) 2024-09-21T10:06:32.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:32 smithi107 ceph-mon[102221]: pgmap v735: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s; 73932/356133 objects degraded (20.760%) 2024-09-21T10:06:32.805 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:32.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:32.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:32.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:32 smithi129 ceph-mon[88471]: pgmap v735: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s; 73932/356133 objects degraded (20.760%) 2024-09-21T10:06:32.930 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:32.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:32.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:33.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:33.076 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:33.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:33.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:33.178 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:33.203 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:33.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:33.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:33.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:33.330 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:33.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:33.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:33.432 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:33.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:33.471 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:33.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:33.512 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:33.572 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:33.604 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:33.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:33.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:33.660 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 84 16 125306 125290 93.2113 69.9375 0.0167257 0.0107233 2024-09-21T10:06:33.706 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:33.749 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:33.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:33.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:33.815 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:33 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 76364/368769 objects degraded (20.708%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:33.816 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:33 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 76364/368769 objects degraded (20.708%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:33.850 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:33.895 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:33.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:33.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:33 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 76364/368769 objects degraded (20.708%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:33.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:34.028 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:34.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:34.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:34.130 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:34.164 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:34.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:34.265 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:34.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:34.304 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:34.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:34.339 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:34.405 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:34.437 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:34.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:34.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:34.539 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:34.573 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:34.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:34.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:34.651 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:06:34.660 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 85 16 126402 126386 92.9205 68.5 0.00819225 0.0107571 2024-09-21T10:06:34.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:34.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:34.712 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:34.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:34.813 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:34.851 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:34.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:34.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:34 smithi129 ceph-mon[88471]: pgmap v736: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 76364/368769 objects degraded (20.708%) 2024-09-21T10:06:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:34 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3517171337' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:34.952 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:34.989 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:34.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:34.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:35.075 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:34 smithi107 ceph-mon[102221]: pgmap v736: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 76364/368769 objects degraded (20.708%) 2024-09-21T10:06:35.075 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:34 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3517171337' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:35.076 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:34 smithi107 ceph-mon[105022]: pgmap v736: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 76364/368769 objects degraded (20.708%) 2024-09-21T10:06:35.076 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:34 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3517171337' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:35.092 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:35.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:35.136 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:35.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:35.237 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:35.276 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:35.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:35.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:35.378 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:35.403 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:35.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:35.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:35.505 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:35.543 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:35.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:35.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:35.644 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:35.660 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 86 16 127505 127489 92.6416 68.9375 0.0250904 0.0107896 2024-09-21T10:06:35.682 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:35.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:35.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:35.741 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:35 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:06:35.742 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:35 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3517171337' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:06:35.742 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:35 smithi107 ceph-mon[102221]: osdmap e206: 8 total, 7 up, 8 in 2024-09-21T10:06:35.742 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:35 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:06:35.742 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:35 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3517171337' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:06:35.742 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:35 smithi107 ceph-mon[105022]: osdmap e206: 8 total, 7 up, 8 in 2024-09-21T10:06:35.784 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:35.822 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:35.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:35.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:35.924 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:35.965 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:35.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:35.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:36.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:36.098 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:36.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:36.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:36.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:35 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:06:36.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:35 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3517171337' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:06:36.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:35 smithi129 ceph-mon[88471]: osdmap e206: 8 total, 7 up, 8 in 2024-09-21T10:06:36.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:36.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:36.231 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:36.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:36.304 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:06:36.305 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:06:36.305 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-21T10:06:36.305 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:06:36.305 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:06:36.305 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-21T10:06:36.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:36.360 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:36.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:36.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:36.462 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:36.510 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:36.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:36.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:36.612 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:36.655 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:36.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:36.660 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 87 16 128580 128564 92.3489 67.1875 0.00812083 0.0108239 2024-09-21T10:06:36.756 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:36.775 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:06:36.800 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:36.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:36.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:36 smithi107 ceph-mon[102221]: pgmap v738: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s; 77441/373026 objects degraded (20.760%) 2024-09-21T10:06:36.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:36 smithi107 ceph-mon[105022]: pgmap v738: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s; 77441/373026 objects degraded (20.760%) 2024-09-21T10:06:36.902 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:36.943 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:36.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:36.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:36.952 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:36 smithi129 ceph-mon[88471]: pgmap v738: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s; 77441/373026 objects degraded (20.760%) 2024-09-21T10:06:37.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:37.069 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:37.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:37.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:37.171 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:37.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:37.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:37.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:37.325 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:37.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:37.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:37.426 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:37.465 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:37.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:37.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:37.566 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:37.610 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:37.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:37.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:37.661 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 88 16 129637 129621 92.0502 66.0625 0.0274148 0.0108593 2024-09-21T10:06:37.711 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:37.735 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:37.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:37.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:37.837 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:37.871 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:37.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:37.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:37.973 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:38.016 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:38.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:38.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:38.117 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:38.147 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:38.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:38.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:38.249 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T10:06:38.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:38.281 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:06:38.281 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:06:38.282 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:38.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:38.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:38.308 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:06:38.308 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:06:38.335 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:06:38.335 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:06:38.362 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:06:38.363 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:06:38.384 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:38.392 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:06:38.392 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:06:38.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:38.419 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:38.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:38.433 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:06:38.433 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:06:38.460 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:06:38.460 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:06:38.520 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:38.557 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:38.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:38.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:38.659 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:38.661 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 89 16 130699 130683 91.7617 66.375 0.0112095 0.0108933 2024-09-21T10:06:38.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:38.693 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:38.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:38.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:38 smithi107 ceph-mon[102221]: pgmap v739: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 77782/374049 objects degraded (20.795%) 2024-09-21T10:06:38.774 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:38 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1672010303' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:38.775 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:38 smithi107 ceph-mon[105022]: pgmap v739: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 77782/374049 objects degraded (20.795%) 2024-09-21T10:06:38.775 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:38 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1672010303' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:38.794 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:38.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:38.817 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:38.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:38.869 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:06:38.869 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:06:38.918 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:38.951 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:38.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:38.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:39.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:06:39.053 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:39.093 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:39.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:39.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:39.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:38 smithi129 ceph-mon[88471]: pgmap v739: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 77782/374049 objects degraded (20.795%) 2024-09-21T10:06:39.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:38 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1672010303' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:39.195 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:39.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:39.233 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:39.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:39.334 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:39.364 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:39.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:39.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:39.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:39.501 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:39.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:39.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:39.603 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:39.631 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:39.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:39.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:39.717 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:06:39.718 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:06:39.718 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 90 4 131563 131559 91.3504 54.75 0.0383358 0.0109419 2024-09-21T10:06:39.818 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:39.840 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:39.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:39.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:39.942 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:39.967 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:39.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:40.032 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:39 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 77782/374049 objects degraded (20.795%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:40.033 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:39 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 77782/374049 objects degraded (20.795%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:40.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:06:40.068 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:40.070 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:39 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 77782/374049 objects degraded (20.795%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:40.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:40.092 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:40.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:40.168 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:40.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:40.215 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:40.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:40.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:40.317 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:40.338 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:40.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:40.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:40.340 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:06:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:06:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:06:40.440 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:40.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:40.463 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:40.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:40.563 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:40.584 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:40.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:40.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:40.661 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total time run: 90.0114 2024-09-21T10:06:40.661 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total writes made: 131563 2024-09-21T10:06:40.661 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Write size: 65536 2024-09-21T10:06:40.661 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object size: 65536 2024-09-21T10:06:40.661 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Bandwidth (MB/sec): 91.3516 2024-09-21T10:06:40.661 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Bandwidth: 39.5574 2024-09-21T10:06:40.662 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max bandwidth (MB/sec): 243.75 2024-09-21T10:06:40.662 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min bandwidth (MB/sec): 19.9375 2024-09-21T10:06:40.662 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average IOPS: 1461 2024-09-21T10:06:40.662 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev IOPS: 632.919 2024-09-21T10:06:40.662 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max IOPS: 3900 2024-09-21T10:06:40.662 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min IOPS: 319 2024-09-21T10:06:40.662 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average Latency(s): 0.0109426 2024-09-21T10:06:40.662 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Latency(s): 0.0152213 2024-09-21T10:06:40.662 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max latency(s): 1.05751 2024-09-21T10:06:40.662 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min latency(s): 0.000982552 2024-09-21T10:06:40.669 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:06:40.686 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:40.710 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:40.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:40.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:40.812 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:40.838 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:40.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:40.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:40 smithi107 ceph-mon[102221]: pgmap v740: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s; 80518/388548 objects degraded (20.723%) 2024-09-21T10:06:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:40 smithi107 ceph-mon[105022]: pgmap v740: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s; 80518/388548 objects degraded (20.723%) 2024-09-21T10:06:40.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:40.967 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:40.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:40.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:40.969 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:41.069 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:41.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:41.111 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:41.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:40 smithi129 ceph-mon[88471]: pgmap v740: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s; 80518/388548 objects degraded (20.723%) 2024-09-21T10:06:41.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:41.237 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:41.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:41.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:41.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:41.361 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:41.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:41.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:41.463 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:41.497 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:41.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:41.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:41.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:41.623 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:41.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:41.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:41.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:06:41.725 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:41.762 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:41.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:41.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:41.863 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:41.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:41.897 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:41.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:41.997 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:42.033 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:42.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:42.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:42.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:06:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:06:41] "GET /metrics HTTP/1.1" 200 36781 "" "Prometheus/2.51.0" 2024-09-21T10:06:42.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:42.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:42.162 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:42.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:42.263 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:42.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:42.298 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:42.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:42.399 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:42 smithi129 ceph-mon[88471]: pgmap v741: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 80818/389448 objects degraded (20.752%) 2024-09-21T10:06:42.425 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:42.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:42.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:42.444 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:42.444 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:42.444 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:42 smithi107 ceph-mon[105022]: pgmap v741: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 80818/389448 objects degraded (20.752%) 2024-09-21T10:06:42.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:42.444 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:42.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:42 smithi107 ceph-mon[102221]: pgmap v741: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s; 80818/389448 objects degraded (20.752%) 2024-09-21T10:06:42.527 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:42.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:42.558 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:42.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:42.659 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:42.692 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:42.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:42.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:42.794 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:42.817 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:42.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:42.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:42.919 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:42.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:42.952 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:42.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:43.053 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:43.086 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:43.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:43.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:43.188 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:43.209 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:43.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:43.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:43.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:43.342 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:43.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:43.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:43.423 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:06:43.423 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:06:43.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:43.469 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:43.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:43.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:43.571 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:43.613 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:43.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:43.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:43.716 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:43.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:43.767 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:43.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:43.857 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:06:43.857 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "" 2024-09-21T10:06:43.858 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:06:43.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:43.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:43.901 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:43.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:44.002 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:44.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:44.042 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:44.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:44.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:44.168 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:44.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:44.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:44.234 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:06:44.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:44.296 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:44.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:44.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:44.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:44 smithi107 ceph-mon[102221]: pgmap v742: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 53 MiB/s wr, 981 op/s; 81062/390402 objects degraded (20.764%) 2024-09-21T10:06:44.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:44 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4276449045' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:06:44.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:44 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:06:44.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:44 smithi107 ceph-mon[105022]: pgmap v742: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 53 MiB/s wr, 981 op/s; 81062/390402 objects degraded (20.764%) 2024-09-21T10:06:44.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:44 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4276449045' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:06:44.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:44 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:06:44.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:06:44.399 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:44.429 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:44.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:44.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:44.509 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:44 smithi129 ceph-mon[88471]: pgmap v742: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 53 MiB/s wr, 981 op/s; 81062/390402 objects degraded (20.764%) 2024-09-21T10:06:44.510 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:44 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4276449045' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:06:44.510 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:44 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:06:44.531 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:44.577 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:44.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:44.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:44.589 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:44.680 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:44.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:44.711 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:44.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:44.812 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:44.847 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:44.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:44.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:44.948 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:44.984 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:44.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:44.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:45.086 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:45.126 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:45.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:45.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:45.228 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:45.254 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:45.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:45.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:45.312 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:45.356 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:45.391 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:45.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:45.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:45.493 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:45.519 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:45.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:45.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:45.579 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:45 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:45.579 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:45 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:06:45.579 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:45 smithi107 ceph-mon[102221]: osdmap e207: 8 total, 7 up, 8 in 2024-09-21T10:06:45.579 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:45 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 81062/390402 objects degraded (20.764%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:45.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:45 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:45.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:45 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:06:45.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:45 smithi107 ceph-mon[105022]: osdmap e207: 8 total, 7 up, 8 in 2024-09-21T10:06:45.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:45 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 81062/390402 objects degraded (20.764%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:45.621 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:45.666 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:45.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:45.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:45 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:45 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:06:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:45 smithi129 ceph-mon[88471]: osdmap e207: 8 total, 7 up, 8 in 2024-09-21T10:06:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:45 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 81062/390402 objects degraded (20.764%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:45.768 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:45.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:45.807 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:45.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:45.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:45.931 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:45.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:45.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:46.033 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:46.072 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:46.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:46.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:46.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:46.205 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:46.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:46.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:46.306 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:46.342 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:46.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:46.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:46.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:46.473 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:46.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:46.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:46.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:46.598 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:46.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:46.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:46.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:46 smithi107 ceph-mon[102221]: pgmap v744: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 45 MiB/s wr, 930 op/s; 80563/388587 objects degraded (20.732%) 2024-09-21T10:06:46.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:46 smithi107 ceph-mon[105022]: pgmap v744: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 45 MiB/s wr, 930 op/s; 80563/388587 objects degraded (20.732%) 2024-09-21T10:06:46.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:46 smithi129 ceph-mon[88471]: pgmap v744: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 45 MiB/s wr, 930 op/s; 80563/388587 objects degraded (20.732%) 2024-09-21T10:06:46.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:46.726 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:46.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:46.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:46.828 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:46.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:46.866 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:46.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:46.967 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:46.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:47.000 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:47.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:47.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:06:47.100 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:47.143 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:47.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:47.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:47.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:47.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:47.276 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:47.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:47.377 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:47.414 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:47.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:47.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:47.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:47.543 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:47.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:47.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:47.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:47.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:47.670 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:47.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:47.723 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:06:47.723 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:06:47.772 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:47.804 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:47.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:47.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:47.905 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:47.944 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:47.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:47.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:48.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:48.072 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:48.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:48.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:48.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:48.209 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:48.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:48.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:48.310 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:48.347 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:48.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:48.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:48.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:48 smithi107 ceph-mon[102221]: pgmap v745: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 43 MiB/s wr, 927 op/s; 80255/387663 objects degraded (20.702%) 2024-09-21T10:06:48.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:48 smithi107 ceph-mon[105022]: pgmap v745: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 43 MiB/s wr, 927 op/s; 80255/387663 objects degraded (20.702%) 2024-09-21T10:06:48.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:48.483 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:48.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:48.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:48.493 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:06:48.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:48.617 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:48.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:48.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:48.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:48 smithi129 ceph-mon[88471]: pgmap v745: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 43 MiB/s wr, 927 op/s; 80255/387663 objects degraded (20.702%) 2024-09-21T10:06:48.718 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:48.752 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:48.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:48.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:48.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:48.880 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:48.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:48.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:48.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:49.008 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:49.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:49.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:49.110 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:49.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:49.141 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:49.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:49.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:49.243 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:49.280 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:49.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:49.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:49.382 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:49.414 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:49.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:49.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:49.516 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:49.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:49.548 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:49.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:49.552 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:49.556 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:06:49.556 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:06:49.556 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-21T10:06:49.556 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-09-21T10:06:49.649 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:49.684 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:49.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:49.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:49.786 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:49.817 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:49.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:49.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:49.919 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:49.955 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:49.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:49.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:50.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:50.091 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:50.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:50.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:50.134 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:06:50.137 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:06:50.138 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T10:06:50.138 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:06:50.138 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:06:50.140 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:06:50.141 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:06:50.162 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:06:50.163 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:06:50.188 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:06:50.190 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:06:50.193 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:50.217 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:06:50.218 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:06:50.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:50.238 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:50.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:50.242 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:06:50.243 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:06:50.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:50.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:50.370 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:50.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:50.472 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:50.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:50.506 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:50.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:50.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:50 smithi107 ceph-mon[105022]: pgmap v746: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 13 MiB/s wr, 913 op/s; 77622/373578 objects degraded (20.778%) 2024-09-21T10:06:50.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:50 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 77622/373578 objects degraded (20.778%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:50.539 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:50 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3847402289' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:50.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:50 smithi107 ceph-mon[102221]: pgmap v746: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 13 MiB/s wr, 913 op/s; 77622/373578 objects degraded (20.778%) 2024-09-21T10:06:50.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:50 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 77622/373578 objects degraded (20.778%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:50.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:50 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3847402289' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:50.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:50.648 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:50.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:50.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:50 smithi129 ceph-mon[88471]: pgmap v746: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 13 MiB/s wr, 913 op/s; 77622/373578 objects degraded (20.778%) 2024-09-21T10:06:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:50 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 77622/373578 objects degraded (20.778%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:50 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3847402289' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:06:50.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:06:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:06:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:06:50.750 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:50.786 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:50.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:50.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:50.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:50.918 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:50.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:50.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:51.020 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:51.047 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:51.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:51.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:51.149 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:51.188 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:51.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:51.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:51.290 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:51.322 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:51.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:51.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:51.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:51.449 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:51.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:51.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:51.550 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:51.583 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:51.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:51.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:51.684 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:51.716 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:51.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:51.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:51.806 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:06:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:06:51] "GET /metrics HTTP/1.1" 200 36781 "" "Prometheus/2.51.0" 2024-09-21T10:06:51.817 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:51.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:51.852 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:51.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:51.953 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:51.990 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:51.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:52.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:52.118 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:52.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:52.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:52.220 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:52.254 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:52.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:52.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:52.356 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:52.392 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:52.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:52.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:52.494 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:52.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:52.534 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:52.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:52.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:52 smithi107 ceph-mon[102221]: pgmap v747: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 11 MiB/s wr, 923 op/s; 77225/372387 objects degraded (20.738%) 2024-09-21T10:06:52.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:52 smithi107 ceph-mon[105022]: pgmap v747: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 11 MiB/s wr, 923 op/s; 77225/372387 objects degraded (20.738%) 2024-09-21T10:06:52.635 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:52.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:52 smithi129 ceph-mon[88471]: pgmap v747: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 11 MiB/s wr, 923 op/s; 77225/372387 objects degraded (20.738%) 2024-09-21T10:06:52.671 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:52.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:52.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:52.773 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:52.820 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:52.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:52.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:52.922 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:52.957 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:52.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:52.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:53.058 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:53.094 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:53.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:53.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:53.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:53.224 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:53.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:53.325 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:53.351 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:53.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:53.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:53.453 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:53.488 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:53.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:53.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:53.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:53.619 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:53.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:53.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:53.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:53.746 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:53.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:53.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:53.848 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:53.885 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:53.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:53.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:53.987 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:54.026 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:54.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:54.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:54.128 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:54.169 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:54.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:54.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:54.257 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:06:54.257 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:06:54.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:54.299 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:54.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:54.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:54.401 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:54.436 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:54.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:54.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:54.470 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:54 smithi107 ceph-mon[102221]: pgmap v748: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 960 KiB/s wr, 1.06k op/s; 74765/359370 objects degraded (20.804%) 2024-09-21T10:06:54.470 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:54 smithi107 ceph-mon[105022]: pgmap v748: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 960 KiB/s wr, 1.06k op/s; 74765/359370 objects degraded (20.804%) 2024-09-21T10:06:54.538 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:54.573 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:54.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:54.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:54 smithi129 ceph-mon[88471]: pgmap v748: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 960 KiB/s wr, 1.06k op/s; 74765/359370 objects degraded (20.804%) 2024-09-21T10:06:54.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:54.709 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:54.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:54.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:54.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:54.843 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:54.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:54.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:54.945 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:54.982 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:54.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:54.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:55.084 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:55.117 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:55.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:55.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:55.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:55.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:55.252 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:55.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:55.260 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:06:55.284 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:06:55.353 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:55.387 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:55.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:55.489 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:55.524 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:55.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:55.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:55.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:55 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 74765/359370 objects degraded (20.804%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:55.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:55 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/314662418' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:55.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:55 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:55.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:55 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 74765/359370 objects degraded (20.804%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:55.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:55 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/314662418' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:55.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:55 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:55.626 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:55.665 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:55.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:55.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:55 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 74765/359370 objects degraded (20.804%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:55 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/314662418' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:55 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:06:55.767 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:55.809 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:55.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:55.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:55.911 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:55.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:55.955 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:55.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:56.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:56.082 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:56.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:56.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:56.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:56.209 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:56.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:56.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:56.257 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:06:56.310 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:56.356 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:56.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:56.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:56.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:56 smithi107 ceph-mon[102221]: pgmap v749: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 73768/355356 objects degraded (20.759%) 2024-09-21T10:06:56.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:56 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:56.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:56 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:06:56.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:56 smithi107 ceph-mon[102221]: osdmap e208: 8 total, 7 up, 8 in 2024-09-21T10:06:56.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:56.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:56.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:56 smithi107 ceph-mon[105022]: pgmap v749: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 73768/355356 objects degraded (20.759%) 2024-09-21T10:06:56.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:56 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:56.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:56 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:06:56.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:56 smithi107 ceph-mon[105022]: osdmap e208: 8 total, 7 up, 8 in 2024-09-21T10:06:56.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:56.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:56.458 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:56.495 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:56.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:56.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:56.597 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:56.635 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:56.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:56.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:56 smithi129 ceph-mon[88471]: pgmap v749: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 73768/355356 objects degraded (20.759%) 2024-09-21T10:06:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:56 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:06:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:56 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:06:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:56 smithi129 ceph-mon[88471]: osdmap e208: 8 total, 7 up, 8 in 2024-09-21T10:06:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:06:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:56.737 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:56.777 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:56.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:56.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:56.879 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:56.921 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:56.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:56.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:57.023 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:57.057 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:57.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:57.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:57.158 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:57.197 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:57.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:57.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:57.299 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:57.319 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:57.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:57.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:57.421 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:57.458 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:57.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:57.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:57.560 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:57.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:57.597 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:57.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:57.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:57.729 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:57.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:57.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:57.830 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:57.860 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:57.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:57.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:57.962 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:57.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:57.999 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:58.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:58.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:58.125 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:58.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:58.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:58.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:06:58.262 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:58.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:58.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:06:58.364 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:58.397 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:58.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:58.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:58.499 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:58.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:58.537 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:58.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:58.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:06:58 smithi107 ceph-mon[102221]: pgmap v751: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.2 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 73415/354297 objects degraded (20.721%) 2024-09-21T10:06:58.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:06:58 smithi107 ceph-mon[105022]: pgmap v751: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.2 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 73415/354297 objects degraded (20.721%) 2024-09-21T10:06:58.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:58.661 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:58.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:58.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:58.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:06:58 smithi129 ceph-mon[88471]: pgmap v751: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 7.2 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 73415/354297 objects degraded (20.721%) 2024-09-21T10:06:58.713 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:06:58.713 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-09-21T10:06:58.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-09-21T10:06:58.713 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:06:58.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.713 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:35543 10.5 10:ad273feb:::benchmark_data_smithi107_184135_object35542:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-21T10:06:58.713 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:05:22.702490+0000", 2024-09-21T10:06:58.713 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 95.574571574000004, 2024-09-21T10:06:58.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.084464328000000005, 2024-09-21T10:06:58.713 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:06:58.713 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:06:58.714 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:58.714 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:06:58.714 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:06:58.714 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:58.714 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 35543 2024-09-21T10:06:58.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.714 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:06:58.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:06:58.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.702490+0000", 2024-09-21T10:06:58.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:06:58.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.702490+0000", 2024-09-21T10:06:58.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:06:58.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.702492+0000", 2024-09-21T10:06:58.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.624e-06 2024-09-21T10:06:58.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:06:58.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.702526+0000", 2024-09-21T10:06:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4499999999999998e-05 2024-09-21T10:06:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:06:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.702537+0000", 2024-09-21T10:06:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1077000000000001e-05 2024-09-21T10:06:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:06:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.702544+0000", 2024-09-21T10:06:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8480000000000003e-06 2024-09-21T10:06:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.702603+0000", 2024-09-21T10:06:58.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9206000000000002e-05 2024-09-21T10:06:58.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:06:58.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.702731+0000", 2024-09-21T10:06:58.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012756200000000001 2024-09-21T10:06:58.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:06:58.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.702882+0000", 2024-09-21T10:06:58.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000151232 2024-09-21T10:06:58.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:06:58.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.705191+0000", 2024-09-21T10:06:58.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023084260000000001 2024-09-21T10:06:58.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:58.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.705313+0000", 2024-09-21T10:06:58.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012266800000000001 2024-09-21T10:06:58.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:58.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.786865+0000", 2024-09-21T10:06:58.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.081552032999999996 2024-09-21T10:06:58.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:06:58.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.786920+0000", 2024-09-21T10:06:58.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4673999999999999e-05 2024-09-21T10:06:58.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:06:58.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.786955+0000", 2024-09-21T10:06:58.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4477999999999998e-05 2024-09-21T10:06:58.719 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.719 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.719 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.719 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.36210.0:35544 10.e e197/190 10:7e37d48c:::benchmark_data_smithi107_184135_object35543:head v 197'2314, mlcod=197'2309)", 2024-09-21T10:06:58.720 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:05:22.705409+0000", 2024-09-21T10:06:58.720 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 95.571652631000006, 2024-09-21T10:06:58.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.081447715000000004, 2024-09-21T10:06:58.720 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:06:58.720 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:06:58.720 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:58.720 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:06:58.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:06:58.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.705409+0000", 2024-09-21T10:06:58.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:06:58.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.705409+0000", 2024-09-21T10:06:58.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:06:58.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.705410+0000", 2024-09-21T10:06:58.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1999999999999999e-07 2024-09-21T10:06:58.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:06:58.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.705446+0000", 2024-09-21T10:06:58.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6164999999999997e-05 2024-09-21T10:06:58.721 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.721 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:06:58.721 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.705456+0000", 2024-09-21T10:06:58.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0298e-05 2024-09-21T10:06:58.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:06:58.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.705463+0000", 2024-09-21T10:06:58.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6359999999999997e-06 2024-09-21T10:06:58.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.705753+0000", 2024-09-21T10:06:58.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00029013200000000002 2024-09-21T10:06:58.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:06:58.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.705770+0000", 2024-09-21T10:06:58.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6923999999999999e-05 2024-09-21T10:06:58.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:06:58.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.786802+0000", 2024-09-21T10:06:58.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.081031719000000002 2024-09-21T10:06:58.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:06:58.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.786857+0000", 2024-09-21T10:06:58.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5121000000000003e-05 2024-09-21T10:06:58.723 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.723 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.723 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.724 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:35552 10.3 10:c666d5e6:::benchmark_data_smithi107_184135_object35551:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-21T10:06:58.724 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:05:22.706941+0000", 2024-09-21T10:06:58.724 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 95.570121157000003, 2024-09-21T10:06:58.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.082481501999999998, 2024-09-21T10:06:58.724 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:06:58.724 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:06:58.724 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:58.724 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:06:58.724 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:06:58.725 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:58.725 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 35552 2024-09-21T10:06:58.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.725 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:06:58.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:06:58.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.706941+0000", 2024-09-21T10:06:58.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:06:58.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.706941+0000", 2024-09-21T10:06:58.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:06:58.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.706942+0000", 2024-09-21T10:06:58.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.516e-06 2024-09-21T10:06:58.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:06:58.726 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.706978+0000", 2024-09-21T10:06:58.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6140000000000003e-05 2024-09-21T10:06:58.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:06:58.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.706998+0000", 2024-09-21T10:06:58.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0000000000000002e-05 2024-09-21T10:06:58.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:06:58.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.707006+0000", 2024-09-21T10:06:58.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7689999999999996e-06 2024-09-21T10:06:58.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.707066+0000", 2024-09-21T10:06:58.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9929999999999997e-05 2024-09-21T10:06:58.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:06:58.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.787355+0000", 2024-09-21T10:06:58.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.080288998 2024-09-21T10:06:58.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:06:58.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.787462+0000", 2024-09-21T10:06:58.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000106828 2024-09-21T10:06:58.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:06:58.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.788792+0000", 2024-09-21T10:06:58.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001330524 2024-09-21T10:06:58.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:58.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.788991+0000", 2024-09-21T10:06:58.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000198624 2024-09-21T10:06:58.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:58.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.789349+0000", 2024-09-21T10:06:58.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035774299999999999 2024-09-21T10:06:58.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:06:58.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.789382+0000", 2024-09-21T10:06:58.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2867999999999998e-05 2024-09-21T10:06:58.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:06:58.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.789422+0000", 2024-09-21T10:06:58.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0562000000000002e-05 2024-09-21T10:06:58.730 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.730 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.730 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.730 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.36210.0:35543 10.5 e197/190 ondisk, result = 0)", 2024-09-21T10:06:58.730 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:05:22.707367+0000", 2024-09-21T10:06:58.731 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 95.569694353000003, 2024-09-21T10:06:58.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.079592463000000002, 2024-09-21T10:06:58.731 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:06:58.731 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:06:58.731 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:06:58.731 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:06:58.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:06:58.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.707367+0000", 2024-09-21T10:06:58.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:06:58.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.707367+0000", 2024-09-21T10:06:58.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:06:58.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.707368+0000", 2024-09-21T10:06:58.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0399999999999995e-07 2024-09-21T10:06:58.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:06:58.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.707372+0000", 2024-09-21T10:06:58.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0539999999999996e-06 2024-09-21T10:06:58.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:06:58.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.786771+0000", 2024-09-21T10:06:58.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.079398450999999995 2024-09-21T10:06:58.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:06:58.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.786789+0000", 2024-09-21T10:06:58.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8148e-05 2024-09-21T10:06:58.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.786849+0000", 2024-09-21T10:06:58.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0092e-05 2024-09-21T10:06:58.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:06:58.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.786859+0000", 2024-09-21T10:06:58.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8649999999999996e-06 2024-09-21T10:06:58.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:06:58.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.786960+0000", 2024-09-21T10:06:58.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101149 2024-09-21T10:06:58.734 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.734 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.734 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.735 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.36210.0:35545 10.e e197/190 10:7d7c0c94:::benchmark_data_smithi107_184135_object35544:head v 197'2315, mlcod=197'2312)", 2024-09-21T10:06:58.735 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:05:22.707842+0000", 2024-09-21T10:06:58.735 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 95.569219439999998, 2024-09-21T10:06:58.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.082567309000000005, 2024-09-21T10:06:58.735 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:06:58.735 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:06:58.735 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:58.735 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:06:58.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:06:58.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.707842+0000", 2024-09-21T10:06:58.735 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.735 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.735 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.735 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:06:58.735 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.707842+0000", 2024-09-21T10:06:58.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:06:58.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.707843+0000", 2024-09-21T10:06:58.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1399999999999997e-07 2024-09-21T10:06:58.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:06:58.736 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.707919+0000", 2024-09-21T10:06:58.736 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6163999999999998e-05 2024-09-21T10:06:58.736 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.736 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.736 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:06:58.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.786817+0000", 2024-09-21T10:06:58.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.078898223000000003 2024-09-21T10:06:58.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:06:58.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.786838+0000", 2024-09-21T10:06:58.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0939999999999999e-05 2024-09-21T10:06:58.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.786881+0000", 2024-09-21T10:06:58.737 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2833999999999997e-05 2024-09-21T10:06:58.737 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.737 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.737 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:06:58.737 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.786902+0000", 2024-09-21T10:06:58.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0945999999999999e-05 2024-09-21T10:06:58.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:06:58.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.790380+0000", 2024-09-21T10:06:58.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0034777739999999999 2024-09-21T10:06:58.738 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.738 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.738 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:06:58.738 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:22.790410+0000", 2024-09-21T10:06:58.738 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9813999999999999e-05 2024-09-21T10:06:58.738 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.738 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.738 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.739 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.739 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:69443 10.3 10:c67cb1dc:::benchmark_data_smithi107_184135_object69442:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-21T10:06:58.739 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:05:44.198237+0000", 2024-09-21T10:06:58.739 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.078824925999996, 2024-09-21T10:06:58.739 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.15296036299999999, 2024-09-21T10:06:58.739 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:06:58.739 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:06:58.739 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:58.739 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:06:58.739 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:06:58.739 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:58.739 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 69443 2024-09-21T10:06:58.739 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.740 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:06:58.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.740 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:06:58.740 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.198237+0000", 2024-09-21T10:06:58.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.740 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:06:58.740 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.198237+0000", 2024-09-21T10:06:58.740 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.740 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.740 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.740 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:06:58.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.198237+0000", 2024-09-21T10:06:58.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.75e-07 2024-09-21T10:06:58.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:06:58.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.198283+0000", 2024-09-21T10:06:58.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5874999999999997e-05 2024-09-21T10:06:58.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:06:58.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.198290+0000", 2024-09-21T10:06:58.741 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7490000000000001e-06 2024-09-21T10:06:58.741 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.741 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.741 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:06:58.741 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.198295+0000", 2024-09-21T10:06:58.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9710000000000003e-06 2024-09-21T10:06:58.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.204927+0000", 2024-09-21T10:06:58.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066318599999999998 2024-09-21T10:06:58.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:06:58.742 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.205054+0000", 2024-09-21T10:06:58.742 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012731999999999999 2024-09-21T10:06:58.742 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.742 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.742 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:06:58.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.205186+0000", 2024-09-21T10:06:58.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131979 2024-09-21T10:06:58.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:06:58.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.210458+0000", 2024-09-21T10:06:58.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052722979999999999 2024-09-21T10:06:58.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.743 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.743 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:58.743 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.210514+0000", 2024-09-21T10:06:58.743 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5269000000000002e-05 2024-09-21T10:06:58.743 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:58.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.351106+0000", 2024-09-21T10:06:58.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.14059222199999999 2024-09-21T10:06:58.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:06:58.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.351162+0000", 2024-09-21T10:06:58.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5662999999999999e-05 2024-09-21T10:06:58.744 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.744 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.744 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:06:58.744 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.351197+0000", 2024-09-21T10:06:58.744 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5682000000000002e-05 2024-09-21T10:06:58.744 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.745 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.745 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.745 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.745 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.745 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:69445 10.3 10:cfade645:::benchmark_data_smithi107_184135_object69444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-21T10:06:58.745 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:05:44.202856+0000", 2024-09-21T10:06:58.745 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.074205778000007, 2024-09-21T10:06:58.745 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.149205429, 2024-09-21T10:06:58.745 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:06:58.745 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:06:58.745 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:58.745 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:06:58.745 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:06:58.746 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:58.746 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 69445 2024-09-21T10:06:58.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.746 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:06:58.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.746 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:06:58.746 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.202856+0000", 2024-09-21T10:06:58.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.746 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.746 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.746 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:06:58.746 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.202856+0000", 2024-09-21T10:06:58.746 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:06:58.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.202857+0000", 2024-09-21T10:06:58.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.474e-06 2024-09-21T10:06:58.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:06:58.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.202904+0000", 2024-09-21T10:06:58.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6817999999999998e-05 2024-09-21T10:06:58.747 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.747 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.747 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:06:58.747 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.202919+0000", 2024-09-21T10:06:58.747 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5172e-05 2024-09-21T10:06:58.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:06:58.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.202928+0000", 2024-09-21T10:06:58.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6740000000000003e-06 2024-09-21T10:06:58.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.207711+0000", 2024-09-21T10:06:58.748 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0047825239999999998 2024-09-21T10:06:58.748 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.748 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.748 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:06:58.748 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.207805+0000", 2024-09-21T10:06:58.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4636000000000007e-05 2024-09-21T10:06:58.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:06:58.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.207951+0000", 2024-09-21T10:06:58.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014554300000000001 2024-09-21T10:06:58.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:06:58.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.213022+0000", 2024-09-21T10:06:58.749 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005070873 2024-09-21T10:06:58.749 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.749 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.749 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:58.749 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.213537+0000", 2024-09-21T10:06:58.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000515775 2024-09-21T10:06:58.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:58.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.351908+0000", 2024-09-21T10:06:58.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.13837032599999999 2024-09-21T10:06:58.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:06:58.750 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.351961+0000", 2024-09-21T10:06:58.750 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3208999999999998e-05 2024-09-21T10:06:58.750 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.750 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.750 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:06:58.751 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.352061+0000", 2024-09-21T10:06:58.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000100405 2024-09-21T10:06:58.751 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.751 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.751 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.751 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.751 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.751 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:69451 10.5 10:adb02b41:::benchmark_data_smithi107_184135_object69450:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-21T10:06:58.751 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:05:44.207929+0000", 2024-09-21T10:06:58.751 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.069133120000004, 2024-09-21T10:06:58.751 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.14025181, 2024-09-21T10:06:58.751 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:06:58.751 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:06:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:06:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:06:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 69451 2024-09-21T10:06:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:06:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:06:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.207929+0000", 2024-09-21T10:06:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:06:58.752 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.207929+0000", 2024-09-21T10:06:58.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:06:58.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.207930+0000", 2024-09-21T10:06:58.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7579999999999999e-06 2024-09-21T10:06:58.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:06:58.753 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.207955+0000", 2024-09-21T10:06:58.753 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4786e-05 2024-09-21T10:06:58.753 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.753 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.753 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:06:58.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.207970+0000", 2024-09-21T10:06:58.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4498000000000001e-05 2024-09-21T10:06:58.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:06:58.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.207977+0000", 2024-09-21T10:06:58.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7719999999999994e-06 2024-09-21T10:06:58.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.754 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.754 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.208054+0000", 2024-09-21T10:06:58.754 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6545999999999995e-05 2024-09-21T10:06:58.754 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.754 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:06:58.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.208175+0000", 2024-09-21T10:06:58.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012087999999999999 2024-09-21T10:06:58.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:06:58.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.208315+0000", 2024-09-21T10:06:58.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013979199999999999 2024-09-21T10:06:58.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.755 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.755 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:06:58.755 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.210916+0000", 2024-09-21T10:06:58.755 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002601786 2024-09-21T10:06:58.755 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:58.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.213701+0000", 2024-09-21T10:06:58.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027848629999999998 2024-09-21T10:06:58.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:58.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.348080+0000", 2024-09-21T10:06:58.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.13437848399999999 2024-09-21T10:06:58.756 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.756 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.756 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:06:58.756 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.348131+0000", 2024-09-21T10:06:58.756 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1415999999999997e-05 2024-09-21T10:06:58.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.757 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:06:58.757 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.348180+0000", 2024-09-21T10:06:58.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9228999999999999e-05 2024-09-21T10:06:58.757 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.757 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.757 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.757 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.757 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.757 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:69452 10.3 10:c5f87082:::benchmark_data_smithi107_184135_object69451:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-21T10:06:58.757 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:05:44.208001+0000", 2024-09-21T10:06:58.757 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.069060918999995, 2024-09-21T10:06:58.757 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.14423381800000001, 2024-09-21T10:06:58.758 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:06:58.758 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:06:58.758 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:58.758 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:06:58.758 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:06:58.758 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:58.758 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 69452 2024-09-21T10:06:58.758 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.758 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:06:58.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.758 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:06:58.758 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.208001+0000", 2024-09-21T10:06:58.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.759 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:06:58.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.208001+0000", 2024-09-21T10:06:58.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.759 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:06:58.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.208001+0000", 2024-09-21T10:06:58.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7800000000000001e-07 2024-09-21T10:06:58.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:06:58.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.208025+0000", 2024-09-21T10:06:58.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3898999999999999e-05 2024-09-21T10:06:58.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:06:58.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.208033+0000", 2024-09-21T10:06:58.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7929999999999998e-06 2024-09-21T10:06:58.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:06:58.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.208039+0000", 2024-09-21T10:06:58.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6420000000000004e-06 2024-09-21T10:06:58.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.210537+0000", 2024-09-21T10:06:58.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002498522 2024-09-21T10:06:58.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:06:58.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.210665+0000", 2024-09-21T10:06:58.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012771599999999999 2024-09-21T10:06:58.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:06:58.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.210802+0000", 2024-09-21T10:06:58.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000136876 2024-09-21T10:06:58.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:06:58.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.213297+0000", 2024-09-21T10:06:58.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024951650000000001 2024-09-21T10:06:58.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:58.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.213580+0000", 2024-09-21T10:06:58.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00028244799999999998 2024-09-21T10:06:58.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:58.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.352156+0000", 2024-09-21T10:06:58.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.13857657300000001 2024-09-21T10:06:58.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:06:58.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.352199+0000", 2024-09-21T10:06:58.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2892999999999997e-05 2024-09-21T10:06:58.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:06:58.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.352235+0000", 2024-09-21T10:06:58.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5713000000000002e-05 2024-09-21T10:06:58.763 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.763 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.764 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.764 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:69457 10.3 10:c0b6d67f:::benchmark_data_smithi107_184135_object69456:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-21T10:06:58.764 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:05:44.213693+0000", 2024-09-21T10:06:58.764 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.063368819999994, 2024-09-21T10:06:58.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.1387592, 2024-09-21T10:06:58.764 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:06:58.764 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:06:58.764 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:58.764 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:06:58.764 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:06:58.764 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:58.764 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 69457 2024-09-21T10:06:58.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.765 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:06:58.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:06:58.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.213693+0000", 2024-09-21T10:06:58.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:06:58.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.213693+0000", 2024-09-21T10:06:58.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:06:58.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.213694+0000", 2024-09-21T10:06:58.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3510000000000001e-06 2024-09-21T10:06:58.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:06:58.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.213715+0000", 2024-09-21T10:06:58.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0404e-05 2024-09-21T10:06:58.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:06:58.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.213732+0000", 2024-09-21T10:06:58.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7736000000000001e-05 2024-09-21T10:06:58.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:06:58.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.213738+0000", 2024-09-21T10:06:58.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8359999999999997e-06 2024-09-21T10:06:58.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.213787+0000", 2024-09-21T10:06:58.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9196000000000001e-05 2024-09-21T10:06:58.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:06:58.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.213931+0000", 2024-09-21T10:06:58.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000143545 2024-09-21T10:06:58.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:06:58.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.214072+0000", 2024-09-21T10:06:58.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014064399999999999 2024-09-21T10:06:58.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:06:58.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.351056+0000", 2024-09-21T10:06:58.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.136984679 2024-09-21T10:06:58.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:58.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.352110+0000", 2024-09-21T10:06:58.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010535849999999999 2024-09-21T10:06:58.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:58.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.352332+0000", 2024-09-21T10:06:58.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022231400000000001 2024-09-21T10:06:58.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:06:58.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.352402+0000", 2024-09-21T10:06:58.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9760999999999998e-05 2024-09-21T10:06:58.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:06:58.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.352452+0000", 2024-09-21T10:06:58.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0148999999999997e-05 2024-09-21T10:06:58.769 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.769 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.770 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.770 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:69458 10.3 10:cc079acd:::benchmark_data_smithi107_184135_object69457:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-21T10:06:58.770 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:05:44.213752+0000", 2024-09-21T10:06:58.770 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.063309746000002, 2024-09-21T10:06:58.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.13883994299999999, 2024-09-21T10:06:58.770 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:06:58.770 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:06:58.770 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:58.770 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:06:58.770 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:06:58.770 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:58.770 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 69458 2024-09-21T10:06:58.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.771 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:06:58.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:06:58.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.213752+0000", 2024-09-21T10:06:58.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:06:58.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.213752+0000", 2024-09-21T10:06:58.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:06:58.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.213752+0000", 2024-09-21T10:06:58.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.27e-07 2024-09-21T10:06:58.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:06:58.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.213768+0000", 2024-09-21T10:06:58.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5979000000000001e-05 2024-09-21T10:06:58.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:06:58.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.213773+0000", 2024-09-21T10:06:58.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8600000000000001e-06 2024-09-21T10:06:58.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:06:58.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.213776+0000", 2024-09-21T10:06:58.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2499999999999998e-06 2024-09-21T10:06:58.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.348214+0000", 2024-09-21T10:06:58.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.13443759299999999 2024-09-21T10:06:58.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:06:58.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.348359+0000", 2024-09-21T10:06:58.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000145303 2024-09-21T10:06:58.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:06:58.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.348484+0000", 2024-09-21T10:06:58.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000124153 2024-09-21T10:06:58.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:06:58.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.352246+0000", 2024-09-21T10:06:58.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003762521 2024-09-21T10:06:58.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:58.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.352291+0000", 2024-09-21T10:06:58.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4743000000000002e-05 2024-09-21T10:06:58.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:58.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.352501+0000", 2024-09-21T10:06:58.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000210196 2024-09-21T10:06:58.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:06:58.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.352558+0000", 2024-09-21T10:06:58.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7046000000000002e-05 2024-09-21T10:06:58.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:06:58.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.352592+0000", 2024-09-21T10:06:58.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3971999999999999e-05 2024-09-21T10:06:58.776 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.776 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.776 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.776 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.36210.0:69460 10.8 e198/190 10:1553172b:::benchmark_data_smithi107_184135_object69459:head v 198'4214, mlcod=198'4212)", 2024-09-21T10:06:58.776 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:05:44.214171+0000", 2024-09-21T10:06:58.776 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.062890945999996, 2024-09-21T10:06:58.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.13660981, 2024-09-21T10:06:58.776 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:06:58.776 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:06:58.776 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:58.777 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:06:58.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:06:58.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.214171+0000", 2024-09-21T10:06:58.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:06:58.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.214171+0000", 2024-09-21T10:06:58.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:06:58.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.214171+0000", 2024-09-21T10:06:58.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0399999999999996e-07 2024-09-21T10:06:58.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:06:58.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.214193+0000", 2024-09-21T10:06:58.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1644000000000001e-05 2024-09-21T10:06:58.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:06:58.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.214204+0000", 2024-09-21T10:06:58.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0732e-05 2024-09-21T10:06:58.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:06:58.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.214210+0000", 2024-09-21T10:06:58.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2280000000000003e-06 2024-09-21T10:06:58.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.214326+0000", 2024-09-21T10:06:58.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000116284 2024-09-21T10:06:58.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:06:58.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.214352+0000", 2024-09-21T10:06:58.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5843e-05 2024-09-21T10:06:58.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:06:58.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.350702+0000", 2024-09-21T10:06:58.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.136349417 2024-09-21T10:06:58.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:06:58.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.350781+0000", 2024-09-21T10:06:58.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9058000000000001e-05 2024-09-21T10:06:58.780 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.780 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.780 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.780 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.36210.0:69451 10.5 e198/190 ondisk, result = 0)", 2024-09-21T10:06:58.780 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:05:44.214960+0000", 2024-09-21T10:06:58.780 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.062101996999999, 2024-09-21T10:06:58.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.13322734, 2024-09-21T10:06:58.781 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:06:58.781 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:06:58.781 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:06:58.781 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:06:58.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:06:58.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.214960+0000", 2024-09-21T10:06:58.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:06:58.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.214960+0000", 2024-09-21T10:06:58.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:06:58.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.214960+0000", 2024-09-21T10:06:58.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8999999999999996e-07 2024-09-21T10:06:58.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:06:58.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.214965+0000", 2024-09-21T10:06:58.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1450000000000001e-06 2024-09-21T10:06:58.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:06:58.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.214975+0000", 2024-09-21T10:06:58.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8879999999999992e-06 2024-09-21T10:06:58.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:06:58.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.214980+0000", 2024-09-21T10:06:58.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9569999999999999e-06 2024-09-21T10:06:58.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.348063+0000", 2024-09-21T10:06:58.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.133082069 2024-09-21T10:06:58.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:06:58.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.348073+0000", 2024-09-21T10:06:58.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0390999999999999e-05 2024-09-21T10:06:58.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:06:58.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.348187+0000", 2024-09-21T10:06:58.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0001142 2024-09-21T10:06:58.784 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.784 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.784 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.784 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.36210.0:69457 10.3 e198/190 ondisk, result = 0)", 2024-09-21T10:06:58.784 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:05:44.216956+0000", 2024-09-21T10:06:58.784 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.060105436000001, 2024-09-21T10:06:58.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.13516557300000001, 2024-09-21T10:06:58.784 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:06:58.784 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:06:58.784 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:06:58.784 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:06:58.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:06:58.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.216956+0000", 2024-09-21T10:06:58.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:06:58.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.216956+0000", 2024-09-21T10:06:58.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:06:58.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.216957+0000", 2024-09-21T10:06:58.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7600000000000006e-07 2024-09-21T10:06:58.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:06:58.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.216962+0000", 2024-09-21T10:06:58.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8640000000000004e-06 2024-09-21T10:06:58.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:06:58.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.348117+0000", 2024-09-21T10:06:58.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.13115496700000001 2024-09-21T10:06:58.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:06:58.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.348122+0000", 2024-09-21T10:06:58.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.288e-06 2024-09-21T10:06:58.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.352098+0000", 2024-09-21T10:06:58.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0039758969999999999 2024-09-21T10:06:58.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:06:58.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.352105+0000", 2024-09-21T10:06:58.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1770000000000003e-06 2024-09-21T10:06:58.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:06:58.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.352122+0000", 2024-09-21T10:06:58.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7404000000000001e-05 2024-09-21T10:06:58.788 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.788 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.788 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.788 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.36210.0:69443 10.3 e198/190 ondisk, result = 0)", 2024-09-21T10:06:58.788 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:05:44.217238+0000", 2024-09-21T10:06:58.788 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.059823656999995, 2024-09-21T10:06:58.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.13403491000000001, 2024-09-21T10:06:58.788 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:06:58.788 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:06:58.788 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:06:58.788 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:06:58.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:06:58.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.217238+0000", 2024-09-21T10:06:58.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:06:58.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.217238+0000", 2024-09-21T10:06:58.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:06:58.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.217239+0000", 2024-09-21T10:06:58.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8999999999999996e-07 2024-09-21T10:06:58.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:06:58.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.217243+0000", 2024-09-21T10:06:58.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9960000000000004e-06 2024-09-21T10:06:58.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:06:58.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.348028+0000", 2024-09-21T10:06:58.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.130785443 2024-09-21T10:06:58.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:06:58.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.348043+0000", 2024-09-21T10:06:58.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4739999999999999e-05 2024-09-21T10:06:58.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.351092+0000", 2024-09-21T10:06:58.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003048968 2024-09-21T10:06:58.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:06:58.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.351100+0000", 2024-09-21T10:06:58.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9219999999999997e-06 2024-09-21T10:06:58.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:06:58.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.351273+0000", 2024-09-21T10:06:58.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000173251 2024-09-21T10:06:58.791 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.792 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.792 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.792 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.36210.0:69468 10.e e198/190 10:70df37c0:::benchmark_data_smithi107_184135_object69467:head v 198'4385, mlcod=198'4383)", 2024-09-21T10:06:58.792 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:05:44.220196+0000", 2024-09-21T10:06:58.792 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 74.056865959000007, 2024-09-21T10:06:58.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.13656089399999999, 2024-09-21T10:06:58.792 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:06:58.792 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:06:58.792 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:58.792 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:06:58.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:06:58.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.220196+0000", 2024-09-21T10:06:58.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:06:58.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.220196+0000", 2024-09-21T10:06:58.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:06:58.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.220197+0000", 2024-09-21T10:06:58.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6000000000000003e-07 2024-09-21T10:06:58.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:06:58.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.220270+0000", 2024-09-21T10:06:58.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2922000000000004e-05 2024-09-21T10:06:58.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:06:58.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.348049+0000", 2024-09-21T10:06:58.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.127779848 2024-09-21T10:06:58.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:06:58.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.348058+0000", 2024-09-21T10:06:58.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.619e-06 2024-09-21T10:06:58.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.348122+0000", 2024-09-21T10:06:58.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4244000000000004e-05 2024-09-21T10:06:58.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:06:58.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.348144+0000", 2024-09-21T10:06:58.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2255e-05 2024-09-21T10:06:58.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:06:58.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.356694+0000", 2024-09-21T10:06:58.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085494660000000004 2024-09-21T10:06:58.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:06:58.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:05:44.356757+0000", 2024-09-21T10:06:58.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2780000000000005e-05 2024-09-21T10:06:58.796 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.796 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.796 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.796 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:95095 10.5 10:a62481d1:::benchmark_data_smithi107_184135_object95094:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T10:06:58.796 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:06:06.381486+0000", 2024-09-21T10:06:58.796 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 51.895575970000003, 2024-09-21T10:06:58.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.022675132, 2024-09-21T10:06:58.797 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:06:58.797 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:06:58.797 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:06:58.797 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:06:58.797 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:06:58.797 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:58.797 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 95095 2024-09-21T10:06:58.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.797 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:06:58.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:06:58.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.381486+0000", 2024-09-21T10:06:58.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:06:58.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.381486+0000", 2024-09-21T10:06:58.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:06:58.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.381488+0000", 2024-09-21T10:06:58.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7400000000000001e-06 2024-09-21T10:06:58.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:06:58.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.381511+0000", 2024-09-21T10:06:58.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3374e-05 2024-09-21T10:06:58.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:06:58.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.381527+0000", 2024-09-21T10:06:58.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5928000000000001e-05 2024-09-21T10:06:58.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:06:58.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.381534+0000", 2024-09-21T10:06:58.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1369999999999996e-06 2024-09-21T10:06:58.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.381603+0000", 2024-09-21T10:06:58.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9395999999999998e-05 2024-09-21T10:06:58.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:06:58.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.381747+0000", 2024-09-21T10:06:58.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014325399999999999 2024-09-21T10:06:58.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:06:58.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.382052+0000", 2024-09-21T10:06:58.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000305183 2024-09-21T10:06:58.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:06:58.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.385248+0000", 2024-09-21T10:06:58.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031962660000000001 2024-09-21T10:06:58.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:58.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.386212+0000", 2024-09-21T10:06:58.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00096411100000000003 2024-09-21T10:06:58.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.406246+0000", 2024-09-21T10:06:58.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020034209000000001 2024-09-21T10:06:58.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:06:58.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.406257+0000", 2024-09-21T10:06:58.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0839000000000001e-05 2024-09-21T10:06:58.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.404063+0000", 2024-09-21T10:06:58.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.99780565700000001 2024-09-21T10:06:58.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:06:58.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.404161+0000", 2024-09-21T10:06:58.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8037999999999999e-05 2024-09-21T10:06:58.802 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.802 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.803 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.803 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:95097 10.5 10:a3df757c:::benchmark_data_smithi107_184135_object95096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T10:06:58.803 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:06:06.388020+0000", 2024-09-21T10:06:58.803 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 51.889041474000003, 2024-09-21T10:06:58.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0166121699999999, 2024-09-21T10:06:58.803 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:06:58.803 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:06:58.803 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:06:58.803 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:06:58.803 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:06:58.803 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:58.803 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 95097 2024-09-21T10:06:58.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.804 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:06:58.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:06:58.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.388020+0000", 2024-09-21T10:06:58.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:06:58.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.388020+0000", 2024-09-21T10:06:58.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:06:58.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.388022+0000", 2024-09-21T10:06:58.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.61e-06 2024-09-21T10:06:58.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:06:58.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.388045+0000", 2024-09-21T10:06:58.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2631999999999999e-05 2024-09-21T10:06:58.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:06:58.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.388061+0000", 2024-09-21T10:06:58.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6738000000000001e-05 2024-09-21T10:06:58.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:06:58.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.388069+0000", 2024-09-21T10:06:58.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4259999999999997e-06 2024-09-21T10:06:58.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.388121+0000", 2024-09-21T10:06:58.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2444999999999997e-05 2024-09-21T10:06:58.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:06:58.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.388286+0000", 2024-09-21T10:06:58.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000164537 2024-09-21T10:06:58.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:06:58.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.388564+0000", 2024-09-21T10:06:58.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00027838899999999998 2024-09-21T10:06:58.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:06:58.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.392309+0000", 2024-09-21T10:06:58.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003744813 2024-09-21T10:06:58.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:58.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.392717+0000", 2024-09-21T10:06:58.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040770299999999998 2024-09-21T10:06:58.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.406330+0000", 2024-09-21T10:06:58.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013613484 2024-09-21T10:06:58.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:06:58.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.406337+0000", 2024-09-21T10:06:58.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9569999999999996e-06 2024-09-21T10:06:58.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.404560+0000", 2024-09-21T10:06:58.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.99822255900000001 2024-09-21T10:06:58.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:06:58.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.404632+0000", 2024-09-21T10:06:58.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2877e-05 2024-09-21T10:06:58.809 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.809 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.809 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.809 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:95106 10.8 10:128143e9:::benchmark_data_smithi107_184135_object95105:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-21T10:06:58.809 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:06:06.400796+0000", 2024-09-21T10:06:58.809 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 51.876265629000002, 2024-09-21T10:06:58.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0161196219999999, 2024-09-21T10:06:58.809 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:06:58.809 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:06:58.809 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:58.809 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:06:58.809 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:06:58.810 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:58.810 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 95106 2024-09-21T10:06:58.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.810 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:06:58.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:06:58.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.400796+0000", 2024-09-21T10:06:58.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:06:58.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.400796+0000", 2024-09-21T10:06:58.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:06:58.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.400798+0000", 2024-09-21T10:06:58.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.813e-06 2024-09-21T10:06:58.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:06:58.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.400834+0000", 2024-09-21T10:06:58.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5772000000000002e-05 2024-09-21T10:06:58.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:06:58.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.400845+0000", 2024-09-21T10:06:58.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1749000000000001e-05 2024-09-21T10:06:58.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:06:58.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.400855+0000", 2024-09-21T10:06:58.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8689999999999999e-06 2024-09-21T10:06:58.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.402893+0000", 2024-09-21T10:06:58.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020376090000000001 2024-09-21T10:06:58.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:06:58.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.402902+0000", 2024-09-21T10:06:58.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1209999999999996e-06 2024-09-21T10:06:58.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.404968+0000", 2024-09-21T10:06:58.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0020656670000001 2024-09-21T10:06:58.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:06:58.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.407877+0000", 2024-09-21T10:06:58.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0029096069999999998 2024-09-21T10:06:58.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 7", 2024-09-21T10:06:58.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.407981+0000", 2024-09-21T10:06:58.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010329 2024-09-21T10:06:58.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:06:58.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.414245+0000", 2024-09-21T10:06:58.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0062643009999999999 2024-09-21T10:06:58.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:58.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.416792+0000", 2024-09-21T10:06:58.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002546815 2024-09-21T10:06:58.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:06:58.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.416837+0000", 2024-09-21T10:06:58.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5547000000000003e-05 2024-09-21T10:06:58.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:06:58.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.416916+0000", 2024-09-21T10:06:58.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8461999999999995e-05 2024-09-21T10:06:58.815 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.815 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.815 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.815 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:95108 10.8 10:166cc55a:::benchmark_data_smithi107_184135_object95107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-21T10:06:58.815 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:06:06.406993+0000", 2024-09-21T10:06:58.816 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 51.870069182999998, 2024-09-21T10:06:58.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0100435590000001, 2024-09-21T10:06:58.816 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:06:58.816 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:06:58.816 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:06:58.816 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:06:58.816 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:06:58.816 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:06:58.816 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 95108 2024-09-21T10:06:58.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.816 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:06:58.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:06:58.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.406993+0000", 2024-09-21T10:06:58.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:06:58.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.406993+0000", 2024-09-21T10:06:58.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:06:58.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:06:58.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.406994+0000", 2024-09-21T10:06:58.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4249999999999999e-06 2024-09-21T10:06:58.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:06:58.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.407032+0000", 2024-09-21T10:06:58.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8442000000000002e-05 2024-09-21T10:06:58.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:06:58.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.407044+0000", 2024-09-21T10:06:58.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1411e-05 2024-09-21T10:06:58.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:06:58.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.407051+0000", 2024-09-21T10:06:58.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0400000000000004e-06 2024-09-21T10:06:58.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.407100+0000", 2024-09-21T10:06:58.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9364999999999999e-05 2024-09-21T10:06:58.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:06:58.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.407109+0000", 2024-09-21T10:06:58.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5159999999999994e-06 2024-09-21T10:06:58.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:06:58.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.411341+0000", 2024-09-21T10:06:58.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.004232566 2024-09-21T10:06:58.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:06:58.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.411481+0000", 2024-09-21T10:06:58.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014006399999999999 2024-09-21T10:06:58.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 7", 2024-09-21T10:06:58.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.411641+0000", 2024-09-21T10:06:58.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015907200000000001 2024-09-21T10:06:58.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:06:58.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.414430+0000", 2024-09-21T10:06:58.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027895160000000001 2024-09-21T10:06:58.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:06:58.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.416956+0000", 2024-09-21T10:06:58.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002526272 2024-09-21T10:06:58.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:06:58.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.416988+0000", 2024-09-21T10:06:58.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.15e-05 2024-09-21T10:06:58.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:06:58.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:06:58.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:06:58.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.417036+0000", 2024-09-21T10:06:58.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8369999999999999e-05 2024-09-21T10:06:58.821 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.821 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.822 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.822 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:06:58.822 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:06:58.822 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:06:58.822 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:06:58.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:58.855 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:58.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:06:58.956 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:59.000 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:59.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:59.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:06:59.102 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:59.138 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:59.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:59.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:59.240 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:06:59.274 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:59.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:59.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:06:59.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:59.405 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:59.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:59.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:59.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:06:59.535 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:59.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:59.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:06:59.637 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:06:59.669 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:59.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:59.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:06:59.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:06:59.794 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:06:59.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:59.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:06:59.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:06:59.896 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:06:59.913 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:06:59.931 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:06:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:06:59.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:00.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:00.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:00.079 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:00.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:00.180 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:00.215 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:00.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:00.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:00.317 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:00.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:00.353 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:00.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:00.419 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:00.454 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:00.505 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:00.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:00.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:00.557 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:00 smithi107 ceph-mon[102221]: pgmap v752: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s; 70638/339690 objects degraded (20.795%) 2024-09-21T10:07:00.557 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:00 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 70638/339690 objects degraded (20.795%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:00.558 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:00 smithi107 ceph-mon[105022]: pgmap v752: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s; 70638/339690 objects degraded (20.795%) 2024-09-21T10:07:00.558 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:00 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 70638/339690 objects degraded (20.795%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:00.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:00.640 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:00.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:00.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:00 smithi129 ceph-mon[88471]: pgmap v752: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s; 70638/339690 objects degraded (20.795%) 2024-09-21T10:07:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:00 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 70638/339690 objects degraded (20.795%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:07:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:07:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:07:00.743 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:00.786 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:00.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:00.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:00.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:00.913 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:00.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:00.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:01.014 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:01.052 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:01.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:01.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:01.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:01.181 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:01.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:01.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:01.283 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:01.316 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:01.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:01.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:01.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:01.452 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:01.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:01.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:01.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:01.583 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:01.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:01.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:01.631 INFO:teuthology.orchestra.run.smithi107.stdout:moved 0 / 327 (0%) 2024-09-21T10:07:01.631 INFO:teuthology.orchestra.run.smithi107.stdout:avg 46.7143 2024-09-21T10:07:01.631 INFO:teuthology.orchestra.run.smithi107.stdout:stddev 8.22391 -> 8.22391 (expected baseline 6.32778) 2024-09-21T10:07:01.631 INFO:teuthology.orchestra.run.smithi107.stdout:min osd.2 with 31 -> 31 pgs (0.663609 -> 0.663609 * mean) 2024-09-21T10:07:01.631 INFO:teuthology.orchestra.run.smithi107.stdout:max osd.4 with 49 -> 49 pgs (1.04893 -> 1.04893 * mean) 2024-09-21T10:07:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:07:01.631 INFO:teuthology.orchestra.run.smithi107.stdout:oload 1000 2024-09-21T10:07:01.631 INFO:teuthology.orchestra.run.smithi107.stdout:max_change 3 2024-09-21T10:07:01.631 INFO:teuthology.orchestra.run.smithi107.stdout:max_change_osds 4 2024-09-21T10:07:01.631 INFO:teuthology.orchestra.run.smithi107.stdout:average_utilization 0.0349 2024-09-21T10:07:01.632 INFO:teuthology.orchestra.run.smithi107.stdout:overload_utilization 0.3487 2024-09-21T10:07:01.632 INFO:teuthology.orchestra.run.smithi107.stderr:no change 2024-09-21T10:07:01.685 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:01.721 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:01.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:01.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:01.823 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:01.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:07:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:07:01] "GET /metrics HTTP/1.1" 200 36780 "" "Prometheus/2.51.0" 2024-09-21T10:07:01.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:01.862 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:01.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:01.963 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:01.998 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:01.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:01.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:02.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:02.128 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:02.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:02.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:02.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:02.257 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:02.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:02.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:02.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:02.390 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:02.390 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:02.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:02.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:02.493 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:02.525 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:02.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:02.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:02.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:02 smithi107 ceph-mon[102221]: pgmap v753: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 70296/338664 objects degraded (20.757%) 2024-09-21T10:07:02.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:02 smithi107 ceph-mon[102221]: from='client.46097 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:02.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:02 smithi107 ceph-mon[105022]: pgmap v753: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 70296/338664 objects degraded (20.757%) 2024-09-21T10:07:02.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:02 smithi107 ceph-mon[105022]: from='client.46097 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:02.627 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:02.661 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:02.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:02.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:02 smithi129 ceph-mon[88471]: pgmap v753: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 70296/338664 objects degraded (20.757%) 2024-09-21T10:07:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:02 smithi129 ceph-mon[88471]: from='client.46097 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:02.766 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:02.809 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:02.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:02.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:02.865 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:02.889 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T10:07:02.889 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 1000 1.0 2024-09-21T10:07:02.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:02.963 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:02.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:02.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:03.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:03.087 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:03.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:03.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:03.188 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:03.226 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:03.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:03.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:03.328 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:03.362 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:03.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:03.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:03.464 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:03.497 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:03.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:03.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:03.598 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:03.637 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:03.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:03.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:03.739 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:03.779 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:03.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:03.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:03.880 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:03.919 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:03.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:03.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:04.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:04.050 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:04.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:04.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:04.152 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:04.187 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:04.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:04.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:04.289 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:04.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:04.331 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:04.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:04.432 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:04.472 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:04.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:04.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:04.574 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:04.586 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:04 smithi107 ceph-mon[102221]: pgmap v754: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 67801/325773 objects degraded (20.812%) 2024-09-21T10:07:04.587 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:04 smithi107 ceph-mon[105022]: pgmap v754: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 67801/325773 objects degraded (20.812%) 2024-09-21T10:07:04.613 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:04.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:04.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:04 smithi129 ceph-mon[88471]: pgmap v754: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 67801/325773 objects degraded (20.812%) 2024-09-21T10:07:04.714 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:04.748 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:04.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:04.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:04.850 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:04.861 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T10:07:04.885 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:07:04.886 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:07:04.890 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:04.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:04.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:04.907 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:07:04.907 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:07:04.929 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:07:04.930 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:07:04.957 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:07:04.957 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:07:04.981 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:07:04.981 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:07:04.992 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:05.001 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:07:05.001 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:07:05.029 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:07:05.029 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:07:05.031 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:05.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:05.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:05.133 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:05.168 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:05.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:05.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:05.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:05.299 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:05.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:05.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:05.401 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:05.439 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:05.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:05.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:05.541 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:05.574 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:05.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:05.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:05.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:05 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 67801/325773 objects degraded (20.812%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:05.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:05 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2662419237' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:05.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:05 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 67801/325773 objects degraded (20.812%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:05.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:05 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2662419237' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:05.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:05 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 67801/325773 objects degraded (20.812%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:05.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:05 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2662419237' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:05.676 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:05.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:05.717 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:05.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:05.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:05.844 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:05.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:05.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:05.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:05.971 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:05.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:05.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:06.072 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:06.112 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:06.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:06.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:06.214 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:06.253 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:06.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:06.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:06.355 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:06.392 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:06.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:06.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:06.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:06.516 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:06.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:06.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:06.608 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:07:06.608 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:07:06.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:06.646 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:06.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:06.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:06.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:06 smithi129 ceph-mon[88471]: pgmap v755: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 66765/321621 objects degraded (20.759%) 2024-09-21T10:07:06.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:06.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:06 smithi107 ceph-mon[102221]: pgmap v755: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 66765/321621 objects degraded (20.759%) 2024-09-21T10:07:06.766 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:06 smithi107 ceph-mon[105022]: pgmap v755: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 66765/321621 objects degraded (20.759%) 2024-09-21T10:07:06.772 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:06.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:06.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:06.874 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:06.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:06.913 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:06.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:07.014 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:07.049 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:07.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:07.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:07.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:07.191 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:07.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:07.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:07.293 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:07.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:07.338 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:07.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:07.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:07.463 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:07.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:07.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:07.565 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:07.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:07.609 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:07.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:07.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:07 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:07.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:07 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:07.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:07 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:07.690 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:07:07.691 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:07:07.710 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:07.739 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:07.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:07.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:07.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:07.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:07.874 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:07.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:07.975 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:08.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:08.026 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:08.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:08.127 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:08.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:08.172 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:08.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:08.273 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:08.299 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:08.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:08.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:08.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:07:08.368 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:08.401 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:08.434 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:08.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:08.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:08.536 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:08.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:08.585 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:08.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:08.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:08 smithi107 ceph-mon[105022]: pgmap v756: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 974 op/s; 66413/320565 objects degraded (20.717%) 2024-09-21T10:07:08.604 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:08 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:08.604 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:08 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:08.604 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:08 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:08.604 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:08 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:08.604 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:08 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3489496744' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:07:08.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:07:08.668 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:07:08.686 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:08.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:08.724 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:08.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:08.825 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:08 smithi107 ceph-mon[102221]: pgmap v756: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 974 op/s; 66413/320565 objects degraded (20.717%) 2024-09-21T10:07:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:08 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:08 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:08 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:08 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:08 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3489496744' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:07:08.873 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:08.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:08.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:08 smithi129 ceph-mon[88471]: pgmap v756: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 974 op/s; 66413/320565 objects degraded (20.717%) 2024-09-21T10:07:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:08 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:08 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:08 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:08 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:08 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3489496744' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:07:08.975 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:09.001 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:09.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:09.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:09.102 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:09.142 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:09.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:09.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:09.243 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:09.272 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:09.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:09.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:09.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:09.400 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:09.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:09.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:09.502 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:09.552 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:09.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:09.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:09.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:09.682 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:09.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:09.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:09.784 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:09 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:09 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3489496744' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:07:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:09 smithi107 ceph-mon[102221]: osdmap e209: 8 total, 7 up, 8 in 2024-09-21T10:07:09.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:09 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 66413/320565 objects degraded (20.717%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:09.793 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:09 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:09.793 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:09 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3489496744' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:07:09.793 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:09 smithi107 ceph-mon[105022]: osdmap e209: 8 total, 7 up, 8 in 2024-09-21T10:07:09.793 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:09 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 66413/320565 objects degraded (20.717%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:09.817 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:09.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:09.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:09.918 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:09 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:09 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3489496744' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:07:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:09 smithi129 ceph-mon[88471]: osdmap e209: 8 total, 7 up, 8 in 2024-09-21T10:07:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:09 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 66413/320565 objects degraded (20.717%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:09.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:09.983 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:09.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:10.083 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:10.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:10.136 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:10.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:10.237 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:10.262 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:10.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:10.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:10.364 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:10.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:10.405 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:10.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:10.500 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:10.501 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:10.506 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:10.539 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:10.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:10.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:10.546 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:07:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:07:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:07:10.641 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:10.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:10.677 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:10.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:10.778 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:10.807 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:10.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:10.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:10.878 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:10 smithi129 ceph-mon[88471]: pgmap v758: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 63611/306201 objects degraded (20.774%) 2024-09-21T10:07:11.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:11.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:10 smithi107 ceph-mon[102221]: pgmap v758: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 63611/306201 objects degraded (20.774%) 2024-09-21T10:07:11.294 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:10 smithi107 ceph-mon[105022]: pgmap v758: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 63611/306201 objects degraded (20.774%) 2024-09-21T10:07:11.321 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:11.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:11.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:11.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:11.461 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:11.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:11.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:11.474 INFO:teuthology.orchestra.run.smithi107.stdout:moved 0 / 327 (0%) 2024-09-21T10:07:11.474 INFO:teuthology.orchestra.run.smithi107.stdout:avg 46.7143 2024-09-21T10:07:11.474 INFO:teuthology.orchestra.run.smithi107.stdout:stddev 8.22391 -> 8.22391 (expected baseline 6.32778) 2024-09-21T10:07:11.474 INFO:teuthology.orchestra.run.smithi107.stdout:min osd.2 with 31 -> 31 pgs (0.663609 -> 0.663609 * mean) 2024-09-21T10:07:11.474 INFO:teuthology.orchestra.run.smithi107.stdout:max osd.4 with 49 -> 49 pgs (1.04893 -> 1.04893 * mean) 2024-09-21T10:07:11.474 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:07:11.474 INFO:teuthology.orchestra.run.smithi107.stdout:oload 1000 2024-09-21T10:07:11.474 INFO:teuthology.orchestra.run.smithi107.stdout:max_change 1 2024-09-21T10:07:11.474 INFO:teuthology.orchestra.run.smithi107.stdout:max_change_osds 4 2024-09-21T10:07:11.474 INFO:teuthology.orchestra.run.smithi107.stdout:average_utilization 0.0326 2024-09-21T10:07:11.475 INFO:teuthology.orchestra.run.smithi107.stdout:overload_utilization 0.3256 2024-09-21T10:07:11.475 INFO:teuthology.orchestra.run.smithi107.stderr:no change 2024-09-21T10:07:11.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:11.585 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:11.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:11.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:11.687 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:11.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:11.743 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:11.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:11.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:07:11.844 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:11.893 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:11.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:11.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:11.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:11.995 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:12.026 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:12.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:12.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:12.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:12.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:07:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:07:11] "GET /metrics HTTP/1.1" 200 36783 "" "Prometheus/2.51.0" 2024-09-21T10:07:12.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:12.127 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:12.161 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:12.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:12.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:12.263 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:12.328 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:12.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:12.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:12.429 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:12.494 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:12.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:12.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:12.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:12.622 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:12.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:12.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:12.700 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-21T10:07:12.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 110 3.0 2024-09-21T10:07:12.724 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:12.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:12.775 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:12.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:12.862 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:12 smithi129 ceph-mon[88471]: pgmap v759: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 63300/305268 objects degraded (20.736%) 2024-09-21T10:07:12.862 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:12 smithi129 ceph-mon[88471]: from='client.46112 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:12.876 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:12.916 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:12.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:12.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:12.927 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:12 smithi107 ceph-mon[102221]: pgmap v759: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 63300/305268 objects degraded (20.736%) 2024-09-21T10:07:12.927 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:12 smithi107 ceph-mon[102221]: from='client.46112 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:12.927 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:12 smithi107 ceph-mon[105022]: pgmap v759: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 63300/305268 objects degraded (20.736%) 2024-09-21T10:07:12.927 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:12 smithi107 ceph-mon[105022]: from='client.46112 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:13.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:13.044 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:13.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:13.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:13.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:13.180 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:13.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:13.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:13.282 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:13.310 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:13.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:13.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:13.412 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:13.452 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:13.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:13.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:13.554 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:13.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:13.594 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:13.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:13.695 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:13.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:13.737 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:13.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:13.838 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:13.865 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:13.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:13.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:13.967 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:13.999 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:14.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:14.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:14.104 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:14.148 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:14.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:14.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:14.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:14.275 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:14.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:14.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:14.377 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:14.413 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:14.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:14.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:14.515 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:14.557 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:14.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:14.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:14 smithi107 ceph-mon[102221]: pgmap v760: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 61151/294234 objects degraded (20.783%) 2024-09-21T10:07:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:14 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:14 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:14 smithi107 ceph-mon[105022]: pgmap v760: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 61151/294234 objects degraded (20.783%) 2024-09-21T10:07:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:14 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:14 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:14.659 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:14 smithi129 ceph-mon[88471]: pgmap v760: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 61151/294234 objects degraded (20.783%) 2024-09-21T10:07:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:14 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:14 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:14.693 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:14.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:14.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:14.795 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:14.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:14.827 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:14.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:14.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:14.953 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:14.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:14.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:15.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:15.079 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:15.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:15.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:15.180 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:15.207 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:15.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:15.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:15.309 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:15.333 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:07:15.333 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:07:15.345 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:15.348 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:15.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:15.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:15.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:15.477 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:15.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:15.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:15.579 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:15 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 61151/294234 objects degraded (20.783%), 18 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:15.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:15 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 61151/294234 objects degraded (20.783%), 18 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:15.616 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:15.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:15.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:15.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:15 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 61151/294234 objects degraded (20.783%), 18 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:15.718 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:15.757 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:15.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:15.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:15.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:15.886 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:15.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:15.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:15.987 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:16.023 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:16.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:16.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:16.125 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:16.163 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:16.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:16.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:16.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:07:16.265 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:16.303 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:16.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:16.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:16.404 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:16.440 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:16.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:16.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:16.542 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:16.576 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:16.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:16.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:16.579 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:16 smithi107 ceph-mon[102221]: pgmap v761: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 60221/290481 objects degraded (20.731%) 2024-09-21T10:07:16.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:16 smithi107 ceph-mon[105022]: pgmap v761: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 60221/290481 objects degraded (20.731%) 2024-09-21T10:07:16.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:16 smithi129 ceph-mon[88471]: pgmap v761: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 5.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 60221/290481 objects degraded (20.731%) 2024-09-21T10:07:16.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:16.705 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:16.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:16.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:16.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:16.839 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:16.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:16.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:16.869 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:16.941 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:16.977 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:16.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:16.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:17.079 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:17.111 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:17.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:17.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:17.213 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:17.251 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:17.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:17.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:17.353 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:17.386 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:17.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:17.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:17.487 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:17.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:17.521 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:17.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:17.622 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:17.664 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:17.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:17.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:17.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:17.791 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:17.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:17.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:17.893 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:17.933 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:17.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:17.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:18.035 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:18.070 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:18.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:18.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:18.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:18.199 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:18.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:18.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:18.300 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:18.344 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:18.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:18.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:18.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:18 smithi107 ceph-mon[102221]: pgmap v762: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 59904/289530 objects degraded (20.690%) 2024-09-21T10:07:18.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:18 smithi107 ceph-mon[105022]: pgmap v762: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 59904/289530 objects degraded (20.690%) 2024-09-21T10:07:18.445 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:18.483 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:18.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:18.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:18.585 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:18.625 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:18.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:18.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:18 smithi129 ceph-mon[88471]: pgmap v762: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 59904/289530 objects degraded (20.690%) 2024-09-21T10:07:18.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:18.755 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:18.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:18.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:18.857 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:18.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:18.889 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:18.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:18.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:19.015 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:19.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:19.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:19.117 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:19.148 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:19.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:19.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:19.250 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:19.282 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:19.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:19.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:19.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:19 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3749018029' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:19.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:19 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3749018029' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:19.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:19.411 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:19.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:19.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:19.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:19.540 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:07:19.540 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:07:19.540 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:07:19.540 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:07:19.540 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:07:19.541 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:07:19.541 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:07:19.541 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T10:07:19.541 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:07:19.541 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:07:19.541 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:07:19.541 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:07:19.541 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:07:19.541 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:07:19.541 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:07:19.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:19.542 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:19.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:19.643 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:19 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3749018029' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:19.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:19.684 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:19.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:19.785 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:19.821 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:19.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:19.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:19.923 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:19.949 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:19.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:19.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:20.050 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:20.083 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:20.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:20.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:20.185 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:20.211 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:20.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:20.313 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:20.350 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:20.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:20.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:20.452 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:20.475 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:20.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:20.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:20.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:20.601 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:20.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:20.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:20.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:20 smithi107 ceph-mon[102221]: pgmap v763: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 57110/274755 objects degraded (20.786%) 2024-09-21T10:07:20.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:20 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 57110/274755 objects degraded (20.786%), 18 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:20.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:20 smithi107 ceph-mon[105022]: pgmap v763: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 57110/274755 objects degraded (20.786%) 2024-09-21T10:07:20.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:20 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 57110/274755 objects degraded (20.786%), 18 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:20.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:20 smithi129 ceph-mon[88471]: pgmap v763: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 57110/274755 objects degraded (20.786%) 2024-09-21T10:07:20.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:20 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 57110/274755 objects degraded (20.786%), 18 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:07:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:07:20.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:07:20.703 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:20.740 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:20.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:20.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:20.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:20.868 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:20.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:20.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:20.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:20.998 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:20.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:20.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:21.100 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:21.122 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:21.138 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:21.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:21.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:21.240 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:21.277 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:21.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:21.378 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:21.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:21.418 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:21.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:21.519 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:21.544 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:21.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:21.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:21.646 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:21.668 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:07:21.668 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:07:21.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:21.680 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:21.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:21.781 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:21.820 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:21.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:21.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:21.892 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:07:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:07:21] "GET /metrics HTTP/1.1" 200 36783 "" "Prometheus/2.51.0" 2024-09-21T10:07:21.921 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:21.953 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:21.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:21.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:22.054 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:22.087 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:22.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:22.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:22.189 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:22.225 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:22.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:22.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:22.327 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:22.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:22.369 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:22.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:22.461 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:22 smithi107 ceph-mon[102221]: pgmap v764: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 905 op/s; 56718/273579 objects degraded (20.732%) 2024-09-21T10:07:22.461 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:22 smithi107 ceph-mon[105022]: pgmap v764: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 905 op/s; 56718/273579 objects degraded (20.732%) 2024-09-21T10:07:22.470 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:22.503 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:22.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:22.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:22.605 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:22.644 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:22.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:22.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:22.648 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:22 smithi129 ceph-mon[88471]: pgmap v764: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 905 op/s; 56718/273579 objects degraded (20.732%) 2024-09-21T10:07:22.746 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:22.781 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:22.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:22.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:22.883 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:22.918 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:22.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:22.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:23.020 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:23.041 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:23.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:23.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:23.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:23.171 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:23.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:23.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:23.273 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:23.315 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:23.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:23.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:23.337 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:07:23.416 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:23.437 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:23.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:23.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:23.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:23.560 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:23.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:23.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:23.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:23 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/367928660' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:23.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:23 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:23.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:23 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'44182','4':'48608','6':'53940'}"} : dispatch 2024-09-21T10:07:23.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:23 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/367928660' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:23.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:23 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:23.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:23 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'44182','4':'48608','6':'53940'}"} : dispatch 2024-09-21T10:07:23.661 INFO:teuthology.orchestra.run.smithi107.stdout:moved 13 / 327 (3.97554%) 2024-09-21T10:07:23.661 INFO:teuthology.orchestra.run.smithi107.stdout:avg 46.7143 2024-09-21T10:07:23.661 INFO:teuthology.orchestra.run.smithi107.stdout:stddev 8.22391 -> 6.2548 (expected baseline 6.32778) 2024-09-21T10:07:23.661 INFO:teuthology.orchestra.run.smithi107.stdout:min osd.2 with 31 -> 37 pgs (0.663609 -> 0.792049 * mean) 2024-09-21T10:07:23.661 INFO:teuthology.orchestra.run.smithi107.stdout:max osd.4 with 49 -> 37 pgs (1.04893 -> 0.792049 * mean) 2024-09-21T10:07:23.661 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:07:23.661 INFO:teuthology.orchestra.run.smithi107.stdout:oload 110 2024-09-21T10:07:23.661 INFO:teuthology.orchestra.run.smithi107.stdout:max_change 3 2024-09-21T10:07:23.662 INFO:teuthology.orchestra.run.smithi107.stdout:max_change_osds 4 2024-09-21T10:07:23.662 INFO:teuthology.orchestra.run.smithi107.stdout:average_utilization 0.0305 2024-09-21T10:07:23.662 INFO:teuthology.orchestra.run.smithi107.stdout:overload_utilization 0.0335 2024-09-21T10:07:23.662 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 weight 1.0000 -> 0.6742 2024-09-21T10:07:23.662 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4 weight 1.0000 -> 0.7417 2024-09-21T10:07:23.662 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6 weight 1.0000 -> 0.8231 2024-09-21T10:07:23.662 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:23.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:07:23.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:23 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/367928660' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:23.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:23 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:23.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:23 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'44182','4':'48608','6':'53940'}"} : dispatch 2024-09-21T10:07:23.686 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:23.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:23.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:23.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:23.809 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:23.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:23.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:23.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:23.932 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:23.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:23.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:24.033 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:24.059 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:24.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:24.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:24.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:24.187 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:24.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:24.289 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:24.314 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:24.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:24.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:24.417 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:24.441 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:24.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:24.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:24.543 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:24.566 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:24.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:24.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:24.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:24 smithi107 ceph-mon[102221]: from='client.46127 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:24.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:24 smithi107 ceph-mon[102221]: pgmap v765: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s; 54276/261150 objects degraded (20.783%) 2024-09-21T10:07:24.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:24 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:07:24.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:24 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:07:24.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:24 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'44182','4':'48608','6':'53940'}"}]': finished 2024-09-21T10:07:24.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:24 smithi107 ceph-mon[102221]: osdmap e210: 8 total, 7 up, 8 in 2024-09-21T10:07:24.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:24 smithi107 ceph-mon[105022]: from='client.46127 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:24.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:24 smithi107 ceph-mon[105022]: pgmap v765: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s; 54276/261150 objects degraded (20.783%) 2024-09-21T10:07:24.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:24 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:07:24.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:24 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:07:24.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:24 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'44182','4':'48608','6':'53940'}"}]': finished 2024-09-21T10:07:24.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:24 smithi107 ceph-mon[105022]: osdmap e210: 8 total, 7 up, 8 in 2024-09-21T10:07:24.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:24.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:24 smithi129 ceph-mon[88471]: from='client.46127 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:24.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:24 smithi129 ceph-mon[88471]: pgmap v765: 109 pgs: 24 active+undersized, 18 active+undersized+degraded, 67 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s; 54276/261150 objects degraded (20.783%) 2024-09-21T10:07:24.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:24 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:07:24.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:24 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:07:24.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:24 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'44182','4':'48608','6':'53940'}"}]': finished 2024-09-21T10:07:24.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:24 smithi129 ceph-mon[88471]: osdmap e210: 8 total, 7 up, 8 in 2024-09-21T10:07:24.690 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:24.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:24.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:24.792 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:24.814 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:24.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:24.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:24.916 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:24.938 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:24.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:24.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:25.040 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:25.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:25.062 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:25.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:25.163 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:25.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:25.186 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:25.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:25.287 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:25.309 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:25.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:25.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:25.411 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:25.432 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:25.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:25.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:25.534 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:25.567 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:25.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:25.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:25.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:25 smithi107 ceph-mon[102221]: osdmap e211: 8 total, 7 up, 8 in 2024-09-21T10:07:25.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:25 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 54276/261150 objects degraded (20.783%), 18 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:25.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:25 smithi107 ceph-mon[105022]: osdmap e211: 8 total, 7 up, 8 in 2024-09-21T10:07:25.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:25 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 54276/261150 objects degraded (20.783%), 18 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:25.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:25.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:25 smithi129 ceph-mon[88471]: osdmap e211: 8 total, 7 up, 8 in 2024-09-21T10:07:25.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:25 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 54276/261150 objects degraded (20.783%), 18 pgs degraded, 42 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:25.705 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:25.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:25.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:25.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:25.832 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:25.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:25.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:25.901 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-21T10:07:25.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 110 0.05 2024-09-21T10:07:25.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:25.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:25.992 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:25.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:26.093 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:26.125 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:26.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:26.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:26.227 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:26.256 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:26.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:26.358 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:26.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:26.394 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:26.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:26.495 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:26.527 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:26.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:26.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:26.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:26 smithi107 ceph-mon[102221]: pgmap v768: 109 pgs: 2 remapped+peering, 3 peering, 22 active+undersized, 17 active+undersized+degraded, 65 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.33k op/s; 53394/257520 objects degraded (20.734%); 0 B/s, 0 objects/s recovering 2024-09-21T10:07:26.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:26 smithi107 ceph-mon[102221]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:07:26.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:26 smithi107 ceph-mon[102221]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 209'11313 2024-09-21T10:07:26.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:26 smithi107 ceph-mon[102221]: 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 209'11121 2024-09-21T10:07:26.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:26 smithi107 ceph-mon[102221]: osdmap e212: 8 total, 7 up, 8 in 2024-09-21T10:07:26.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:26 smithi107 ceph-mon[102221]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 209'11221 2024-09-21T10:07:26.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:26 smithi107 ceph-mon[102221]: 1.11 scrub starts 2024-09-21T10:07:26.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:26 smithi107 ceph-mon[102221]: 1.11 scrub ok 2024-09-21T10:07:26.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:26.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:26.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:26 smithi107 ceph-mon[105022]: pgmap v768: 109 pgs: 2 remapped+peering, 3 peering, 22 active+undersized, 17 active+undersized+degraded, 65 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.33k op/s; 53394/257520 objects degraded (20.734%); 0 B/s, 0 objects/s recovering 2024-09-21T10:07:26.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:26 smithi107 ceph-mon[105022]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:07:26.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:26 smithi107 ceph-mon[105022]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 209'11313 2024-09-21T10:07:26.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:26 smithi107 ceph-mon[105022]: 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 209'11121 2024-09-21T10:07:26.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:26 smithi107 ceph-mon[105022]: osdmap e212: 8 total, 7 up, 8 in 2024-09-21T10:07:26.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:26 smithi107 ceph-mon[105022]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 209'11221 2024-09-21T10:07:26.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:26 smithi107 ceph-mon[105022]: 1.11 scrub starts 2024-09-21T10:07:26.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:26 smithi107 ceph-mon[105022]: 1.11 scrub ok 2024-09-21T10:07:26.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:26.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:26.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:26.662 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:26.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:26.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:26 smithi129 ceph-mon[88471]: pgmap v768: 109 pgs: 2 remapped+peering, 3 peering, 22 active+undersized, 17 active+undersized+degraded, 65 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.33k op/s; 53394/257520 objects degraded (20.734%); 0 B/s, 0 objects/s recovering 2024-09-21T10:07:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:26 smithi129 ceph-mon[88471]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:07:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:26 smithi129 ceph-mon[88471]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 209'11313 2024-09-21T10:07:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:26 smithi129 ceph-mon[88471]: 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 209'11121 2024-09-21T10:07:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:26 smithi129 ceph-mon[88471]: osdmap e212: 8 total, 7 up, 8 in 2024-09-21T10:07:26.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:26 smithi129 ceph-mon[88471]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 209'11221 2024-09-21T10:07:26.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:26 smithi129 ceph-mon[88471]: 1.11 scrub starts 2024-09-21T10:07:26.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:26 smithi129 ceph-mon[88471]: 1.11 scrub ok 2024-09-21T10:07:26.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:26.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:26.763 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:26.790 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:26.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:26.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:26.892 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:26.923 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:26.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:26.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:27.025 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:27.058 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:27.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:27.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:27.159 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:27.193 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:27.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:27.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:27.294 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:27.337 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:27.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:27.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:27.439 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:27.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:27.474 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:27.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:27.480 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:27.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:27.608 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:27.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:27 smithi107 ceph-mon[102221]: 1.e scrub starts 2024-09-21T10:07:27.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:27 smithi107 ceph-mon[102221]: 1.e scrub ok 2024-09-21T10:07:27.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:27 smithi107 ceph-mon[105022]: 1.e scrub starts 2024-09-21T10:07:27.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:27 smithi107 ceph-mon[105022]: 1.e scrub ok 2024-09-21T10:07:27.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:27.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:27.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:27 smithi129 ceph-mon[88471]: 1.e scrub starts 2024-09-21T10:07:27.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:27 smithi129 ceph-mon[88471]: 1.e scrub ok 2024-09-21T10:07:27.711 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:27.743 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:27.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:27.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:27.845 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:27.883 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:27.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:27.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:27.984 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:28.022 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:28.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:28.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:28.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:28.151 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:28.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:28.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:28.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:28.281 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:28.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:28.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:28.382 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:28.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:28.425 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:28.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:28.456 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:28 smithi107 ceph-mon[102221]: pgmap v770: 109 pgs: 3 active+clean+remapped, 2 remapped+peering, 3 peering, 15 active+undersized, 15 active+undersized+degraded, 71 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 925 op/s; 53190/256914 objects degraded (20.703%); 3 B/s, 0 objects/s recovering 2024-09-21T10:07:28.456 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:28 smithi107 ceph-mon[105022]: pgmap v770: 109 pgs: 3 active+clean+remapped, 2 remapped+peering, 3 peering, 15 active+undersized, 15 active+undersized+degraded, 71 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 925 op/s; 53190/256914 objects degraded (20.703%); 3 B/s, 0 objects/s recovering 2024-09-21T10:07:28.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:28.552 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:28.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:28.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:28.653 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:28.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:28 smithi129 ceph-mon[88471]: pgmap v770: 109 pgs: 3 active+clean+remapped, 2 remapped+peering, 3 peering, 15 active+undersized, 15 active+undersized+degraded, 71 active+clean; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 925 op/s; 53190/256914 objects degraded (20.703%); 3 B/s, 0 objects/s recovering 2024-09-21T10:07:28.691 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:28.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:28.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:28.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:28.819 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:28.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:28.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:28.921 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:28.957 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:28.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:28.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:29.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:29.088 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:29.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:29.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:29.190 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:29.226 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:29.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:29.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:29.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:29.355 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:29.355 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:07:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:29.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:29.457 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:29.496 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:29.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:29.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:29.565 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:29 smithi107 ceph-mon[102221]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:07:29.565 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:29 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 51096/248658 objects degraded (20.549%), 12 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:29.568 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:29 smithi107 ceph-mon[105022]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:07:29.568 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:29 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 51096/248658 objects degraded (20.549%), 12 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:29.598 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:29.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:29.637 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:29.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:29.738 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:29.776 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:29.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:29.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:29.817 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:29.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:29.910 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:29.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:29.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:29 smithi129 ceph-mon[88471]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:07:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:29 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 51096/248658 objects degraded (20.549%), 12 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:30.012 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:30.049 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:30.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:30.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:30.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:30.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:30.183 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:30.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:30.284 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:30.317 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:30.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:30.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:30.420 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:30.455 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:30.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:30.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:30.557 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:30.590 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:30.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:30.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:30.615 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T10:07:30.648 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:07:30.648 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:07:30.649 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:30 smithi107 ceph-mon[102221]: pgmap v771: 109 pgs: 2 active+undersized+degraded+remapped+backfilling, 3 active+clean+remapped, 2 remapped+peering, 1 active+remapped+backfill_wait, 14 active+undersized, 10 active+undersized+degraded, 77 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 51096/248658 objects degraded (20.549%); 5163/248658 objects misplaced (2.076%); 5.7 MiB/s, 93 objects/s recovering 2024-09-21T10:07:30.649 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:30 smithi107 ceph-mon[105022]: pgmap v771: 109 pgs: 2 active+undersized+degraded+remapped+backfilling, 3 active+clean+remapped, 2 remapped+peering, 1 active+remapped+backfill_wait, 14 active+undersized, 10 active+undersized+degraded, 77 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 51096/248658 objects degraded (20.549%); 5163/248658 objects misplaced (2.076%); 5.7 MiB/s, 93 objects/s recovering 2024-09-21T10:07:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:30 smithi129 ceph-mon[88471]: pgmap v771: 109 pgs: 2 active+undersized+degraded+remapped+backfilling, 3 active+clean+remapped, 2 remapped+peering, 1 active+remapped+backfill_wait, 14 active+undersized, 10 active+undersized+degraded, 77 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 51096/248658 objects degraded (20.549%); 5163/248658 objects misplaced (2.076%); 5.7 MiB/s, 93 objects/s recovering 2024-09-21T10:07:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:07:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:07:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:07:30.676 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:07:30.677 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:07:30.695 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:30.701 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:07:30.701 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:07:30.737 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:30.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:30.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:30.745 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:07:30.745 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:07:30.771 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:07:30.771 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:07:30.806 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:07:30.807 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:07:30.832 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:07:30.832 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:07:30.839 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:30.875 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:30.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:30.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:30.977 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:31.015 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:31.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:31.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:31.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:31.145 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:31.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:31.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:31.247 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:31.289 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:31.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:31.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:31.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:31.419 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:31.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:31.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:31.488 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:07:31.488 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:07:31.520 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:31.558 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:31.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:31.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:31.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:31 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/54422787' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:31.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:31 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/54422787' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:31.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:31.685 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:31.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:31.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:31.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:31.816 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:31.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:31.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:31.918 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:31 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/54422787' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:31.953 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:31.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:31.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:32.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:32.079 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:32.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:32.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:07:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:07:31] "GET /metrics HTTP/1.1" 200 36821 "" "Prometheus/2.51.0" 2024-09-21T10:07:32.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:32.206 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:32.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:32.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:32.268 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:07:32.269 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:07:32.308 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:32.344 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:32.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:32.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:32.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:32.474 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:32.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:32.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:32.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:32.600 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:32.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:32.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:32.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:32 smithi107 ceph-mon[102221]: pgmap v772: 109 pgs: 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 14 active+undersized, 10 active+undersized+degraded, 77 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 565 op/s; 50803/247779 objects degraded (20.503%); 5163/247779 objects misplaced (2.084%); 4.4 MiB/s, 71 objects/s recovering 2024-09-21T10:07:32.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:32 smithi107 ceph-mon[105022]: pgmap v772: 109 pgs: 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 14 active+undersized, 10 active+undersized+degraded, 77 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 565 op/s; 50803/247779 objects degraded (20.503%); 5163/247779 objects misplaced (2.084%); 4.4 MiB/s, 71 objects/s recovering 2024-09-21T10:07:32.702 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:32.736 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:32.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:32.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:32.838 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:32.870 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:32.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:32.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:32 smithi129 ceph-mon[88471]: pgmap v772: 109 pgs: 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 14 active+undersized, 10 active+undersized+degraded, 77 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 565 op/s; 50803/247779 objects degraded (20.503%); 5163/247779 objects misplaced (2.084%); 4.4 MiB/s, 71 objects/s recovering 2024-09-21T10:07:32.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:33.001 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:33.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:33.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:33.103 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:33.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:33.146 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:33.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:33.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:33.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:33.288 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:33.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:33.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:33.418 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:33.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:33.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:33.520 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:33.561 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:33.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:33.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:33.663 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:33.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:33.703 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:33.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:33.730 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:07:33.804 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:33.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:33.874 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:33.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:33.975 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:34.005 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:34.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:34.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:34.107 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:34.131 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:34.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:34.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:34.178 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:07:34.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:34.265 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:34.269 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:07:34.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:34.272 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:34.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:34.373 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:34.404 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:34.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:34.506 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:34.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:34.546 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:34.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:34.586 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:34 smithi107 ceph-mon[105022]: pgmap v773: 109 pgs: 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 14 active+undersized, 10 active+undersized+degraded, 77 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 852 op/s; 47193/235584 objects degraded (20.032%); 4812/235584 objects misplaced (2.043%); 13 MiB/s, 213 objects/s recovering 2024-09-21T10:07:34.586 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:34 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 47193/235584 objects degraded (20.032%), 12 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:34.586 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:34 smithi107 ceph-mon[102221]: pgmap v773: 109 pgs: 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 14 active+undersized, 10 active+undersized+degraded, 77 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 852 op/s; 47193/235584 objects degraded (20.032%); 4812/235584 objects misplaced (2.043%); 13 MiB/s, 213 objects/s recovering 2024-09-21T10:07:34.586 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:34 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 47193/235584 objects degraded (20.032%), 12 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:34.647 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:34.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:34 smithi129 ceph-mon[88471]: pgmap v773: 109 pgs: 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 14 active+undersized, 10 active+undersized+degraded, 77 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 852 op/s; 47193/235584 objects degraded (20.032%); 4812/235584 objects misplaced (2.043%); 13 MiB/s, 213 objects/s recovering 2024-09-21T10:07:34.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:34 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 47193/235584 objects degraded (20.032%), 12 pgs degraded, 24 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:34.686 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:34.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:34.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:34.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:34.832 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:34.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:34.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:34.934 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:34.963 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:34.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:34.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:35.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:35.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:35.103 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:35.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:35.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:35.232 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:35.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:35.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:35.334 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:35.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:35.376 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:35.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:35.452 INFO:teuthology.orchestra.run.smithi107.stdout:moved 9 / 327 (2.75229%) 2024-09-21T10:07:35.452 INFO:teuthology.orchestra.run.smithi107.stdout:avg 46.7143 2024-09-21T10:07:35.452 INFO:teuthology.orchestra.run.smithi107.stdout:stddev 6.2548 -> 6.53562 (expected baseline 6.32778) 2024-09-21T10:07:35.452 INFO:teuthology.orchestra.run.smithi107.stdout:min osd.2 with 37 -> 38 pgs (0.792049 -> 0.813456 * mean) 2024-09-21T10:07:35.452 INFO:teuthology.orchestra.run.smithi107.stdout:max osd.1 with 49 -> 51 pgs (1.04893 -> 1.09174 * mean) 2024-09-21T10:07:35.452 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:07:35.452 INFO:teuthology.orchestra.run.smithi107.stdout:oload 110 2024-09-21T10:07:35.452 INFO:teuthology.orchestra.run.smithi107.stdout:max_change 0.05 2024-09-21T10:07:35.452 INFO:teuthology.orchestra.run.smithi107.stdout:max_change_osds 4 2024-09-21T10:07:35.452 INFO:teuthology.orchestra.run.smithi107.stdout:average_utilization 0.0281 2024-09-21T10:07:35.452 INFO:teuthology.orchestra.run.smithi107.stdout:overload_utilization 0.0309 2024-09-21T10:07:35.453 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 weight 0.6742 -> 0.6242 2024-09-21T10:07:35.453 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4 weight 0.7417 -> 0.6917 2024-09-21T10:07:35.453 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6 weight 0.8231 -> 0.7731 2024-09-21T10:07:35.476 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:35.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:35.510 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:35.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:35.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:35 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'40906','4':'45332','6':'50664'}"} : dispatch 2024-09-21T10:07:35.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:35 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'40906','4':'45332','6':'50664'}"} : dispatch 2024-09-21T10:07:35.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:35.635 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:35.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:35.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:35.736 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:35.762 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:35.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:35.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:35.795 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:35 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'40906','4':'45332','6':'50664'}"} : dispatch 2024-09-21T10:07:35.863 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:35.886 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:35.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:35.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:35.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:36.011 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:36.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:36.103 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-21T10:07:36.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd test-reweight-by-utilization 110 1.0 2024-09-21T10:07:36.112 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:36.136 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:36.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:36.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:36.238 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:36.261 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:36.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:36.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:36.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:36.385 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:36.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:36.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:36.488 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:36.516 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:36.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:36.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:36.618 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:36.644 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:36.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:36.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:36.745 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:36.773 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:36.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:36.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:36.781 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:36 smithi129 ceph-mon[88471]: pgmap v774: 109 pgs: 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 14 active+undersized, 10 active+undersized+degraded, 77 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 866 op/s; 46204/231486 objects degraded (19.960%); 4812/231486 objects misplaced (2.079%); 12 MiB/s, 188 objects/s recovering 2024-09-21T10:07:36.781 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:36 smithi129 ceph-mon[88471]: from='client.46139 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:36.781 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:36 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'40906','4':'45332','6':'50664'}"}]': finished 2024-09-21T10:07:36.781 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:36 smithi129 ceph-mon[88471]: osdmap e213: 8 total, 7 up, 8 in 2024-09-21T10:07:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:36 smithi107 ceph-mon[102221]: pgmap v774: 109 pgs: 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 14 active+undersized, 10 active+undersized+degraded, 77 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 866 op/s; 46204/231486 objects degraded (19.960%); 4812/231486 objects misplaced (2.079%); 12 MiB/s, 188 objects/s recovering 2024-09-21T10:07:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:36 smithi107 ceph-mon[102221]: from='client.46139 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:36 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'40906','4':'45332','6':'50664'}"}]': finished 2024-09-21T10:07:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:36 smithi107 ceph-mon[102221]: osdmap e213: 8 total, 7 up, 8 in 2024-09-21T10:07:36.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:36 smithi107 ceph-mon[105022]: pgmap v774: 109 pgs: 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 14 active+undersized, 10 active+undersized+degraded, 77 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 866 op/s; 46204/231486 objects degraded (19.960%); 4812/231486 objects misplaced (2.079%); 12 MiB/s, 188 objects/s recovering 2024-09-21T10:07:36.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:36 smithi107 ceph-mon[105022]: from='client.46139 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:36.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:36 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'40906','4':'45332','6':'50664'}"}]': finished 2024-09-21T10:07:36.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:36 smithi107 ceph-mon[105022]: osdmap e213: 8 total, 7 up, 8 in 2024-09-21T10:07:36.875 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:36.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:36.904 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:36.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:37.005 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:37.031 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:37.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:37.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:37.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:37.156 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:37.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:37.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:37.258 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:37.280 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:37.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:37.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:37.382 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:37.407 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:37.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:37.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:37.509 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:37.538 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:37.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:37.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:37.607 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:37 smithi107 ceph-mon[102221]: osdmap e214: 8 total, 7 up, 8 in 2024-09-21T10:07:37.607 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:37 smithi107 ceph-mon[105022]: osdmap e214: 8 total, 7 up, 8 in 2024-09-21T10:07:37.632 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:37.640 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:37.660 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:37.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:37.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:37.762 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:37.800 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:37.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:37.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:37.902 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:37.918 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:37 smithi129 ceph-mon[88471]: osdmap e214: 8 total, 7 up, 8 in 2024-09-21T10:07:37.926 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:37.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:37.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:38.027 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:38.069 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:38.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:38.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:38.171 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:38.215 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:38.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:38.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:38.317 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:38.347 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:38.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:38.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:38.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:38.480 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:07:38.487 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:38.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:38.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:38.589 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:38.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:38.617 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:38.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:38.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:38 smithi129 ceph-mon[88471]: pgmap v777: 109 pgs: 2 peering, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 10 active+undersized+degraded, 76 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 745 op/s; 45963/230763 objects degraded (19.918%); 4812/230763 objects misplaced (2.085%); 10 MiB/s, 165 objects/s recovering 2024-09-21T10:07:38.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:38 smithi129 ceph-mon[88471]: osdmap e215: 8 total, 7 up, 8 in 2024-09-21T10:07:38.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:38 smithi129 ceph-mon[88471]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 212'11779 2024-09-21T10:07:38.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:38 smithi129 ceph-mon[88471]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 212'11946 2024-09-21T10:07:38.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:38 smithi129 ceph-mon[88471]: 10.d starting backfill to osd.7 from (0'0,0'0] MAX to 212'11779 2024-09-21T10:07:38.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:38 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1640207496' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:07:38.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:38.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:38.719 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:38.758 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:38.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:38.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:38 smithi107 ceph-mon[102221]: pgmap v777: 109 pgs: 2 peering, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 10 active+undersized+degraded, 76 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 745 op/s; 45963/230763 objects degraded (19.918%); 4812/230763 objects misplaced (2.085%); 10 MiB/s, 165 objects/s recovering 2024-09-21T10:07:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:38 smithi107 ceph-mon[102221]: osdmap e215: 8 total, 7 up, 8 in 2024-09-21T10:07:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:38 smithi107 ceph-mon[102221]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 212'11779 2024-09-21T10:07:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:38 smithi107 ceph-mon[102221]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 212'11946 2024-09-21T10:07:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:38 smithi107 ceph-mon[102221]: 10.d starting backfill to osd.7 from (0'0,0'0] MAX to 212'11779 2024-09-21T10:07:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:38 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1640207496' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:07:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:38.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:38.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:38 smithi107 ceph-mon[105022]: pgmap v777: 109 pgs: 2 peering, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 10 active+undersized+degraded, 76 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 745 op/s; 45963/230763 objects degraded (19.918%); 4812/230763 objects misplaced (2.085%); 10 MiB/s, 165 objects/s recovering 2024-09-21T10:07:38.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:38 smithi107 ceph-mon[105022]: osdmap e215: 8 total, 7 up, 8 in 2024-09-21T10:07:38.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:38 smithi107 ceph-mon[105022]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 212'11779 2024-09-21T10:07:38.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:38 smithi107 ceph-mon[105022]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 212'11946 2024-09-21T10:07:38.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:38 smithi107 ceph-mon[105022]: 10.d starting backfill to osd.7 from (0'0,0'0] MAX to 212'11779 2024-09-21T10:07:38.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:38 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1640207496' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:07:38.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:38.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:38.860 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:38.898 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:38.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:38.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:39.000 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:39.027 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:39.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:39.128 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:39.169 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:39.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:39.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:39.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:39.299 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:39.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:39.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:39.401 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:39.440 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:39.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:39.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:39.542 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:39.577 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:39.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:39.678 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:39.716 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:39.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:39.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:39.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:39 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:39.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:39 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:39.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:39 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:07:39.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:39 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1640207496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:07:39.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:39 smithi107 ceph-mon[102221]: osdmap e216: 8 total, 7 up, 8 in 2024-09-21T10:07:39.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:39 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:39.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:39 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 45963/230763 objects degraded (19.918%), 12 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:39.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:39 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:39.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:39 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:39.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:39 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:07:39.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:39 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1640207496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:07:39.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:39 smithi107 ceph-mon[105022]: osdmap e216: 8 total, 7 up, 8 in 2024-09-21T10:07:39.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:39 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:39.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:39 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 45963/230763 objects degraded (19.918%), 12 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:39.817 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:39.857 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:39.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:39.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:39.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:39 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:39.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:39 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:39.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:39 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:07:39.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:39 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1640207496' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:07:39.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:39 smithi129 ceph-mon[88471]: osdmap e216: 8 total, 7 up, 8 in 2024-09-21T10:07:39.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:39 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:39.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:39 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 45963/230763 objects degraded (19.918%), 12 pgs degraded, 23 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:39.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:39.987 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:39.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:39.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:40.089 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:40.132 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:40.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:40.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:40.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:40.260 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:40.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:40.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:40.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:40.389 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:40.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:40.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:40.442 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:07:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:07:40.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:07:40.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:40.520 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:40.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:40.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:40.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:40.646 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:40.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:40.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:40.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:40 smithi107 ceph-mon[102221]: pgmap v780: 109 pgs: 1 activating+remapped, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 peering, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 75 active+clean; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 675 op/s; 41818/223380 objects degraded (18.721%); 13554/223380 objects misplaced (6.068%); 53 MiB/s, 841 objects/s recovering 2024-09-21T10:07:40.705 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:40 smithi107 ceph-mon[105022]: pgmap v780: 109 pgs: 1 activating+remapped, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 peering, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 75 active+clean; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 675 op/s; 41818/223380 objects degraded (18.721%); 13554/223380 objects misplaced (6.068%); 53 MiB/s, 841 objects/s recovering 2024-09-21T10:07:40.715 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:40 smithi129 ceph-mon[88471]: pgmap v780: 109 pgs: 1 activating+remapped, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 peering, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 75 active+clean; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 675 op/s; 41818/223380 objects degraded (18.721%); 13554/223380 objects misplaced (6.068%); 53 MiB/s, 841 objects/s recovering 2024-09-21T10:07:40.749 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:40.776 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:40.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:40.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:40.806 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T10:07:40.813 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:40.827 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:07:40.827 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:07:40.847 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:07:40.847 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:07:40.872 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:07:40.872 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:07:40.879 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:40.883 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:07:40.883 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:07:40.895 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:07:40.895 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:07:40.918 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:07:40.918 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:07:40.919 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:40.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:40.949 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:07:40.950 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:07:40.986 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:07:40.986 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:07:41.022 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:41.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:41.067 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:41.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:41.169 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:41.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:41.202 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:41.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:41.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:41.328 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:41.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:41.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:41.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:41.457 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:41.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:41.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:41.558 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:41.604 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:07:41.619 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:41.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:41.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:41.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:41 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/790086171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:41.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:41.668 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:41.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:41.748 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:41.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:41.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:41.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:41 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/790086171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:41.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:41.770 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:41.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:41 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/790086171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:41.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:07:41.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:41.850 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:41.878 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:07:41.879 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:07:41.903 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:41.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:41.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:42.005 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:42.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:42.052 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:42.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:42.085 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:07:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:07:41] "GET /metrics HTTP/1.1" 200 36835 "" "Prometheus/2.51.0" 2024-09-21T10:07:42.095 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:07:42.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:07:42.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:42.200 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:42.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:42.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:42.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:42.331 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:42.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:42.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:42.434 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:42.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:42.482 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:42.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:42.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:42.607 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:42.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:42.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:42.709 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:42.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:42.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:42.782 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:42 smithi107 ceph-mon[102221]: pgmap v781: 109 pgs: 1 activating+remapped, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 500 op/s; 41648/222870 objects degraded (18.687%); 13554/222870 objects misplaced (6.082%); 37 MiB/s, 587 objects/s recovering 2024-09-21T10:07:42.782 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:42 smithi107 ceph-mon[105022]: pgmap v781: 109 pgs: 1 activating+remapped, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 500 op/s; 41648/222870 objects degraded (18.687%); 13554/222870 objects misplaced (6.082%); 37 MiB/s, 587 objects/s recovering 2024-09-21T10:07:42.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:42.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:42.895 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:42.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:42 smithi129 ceph-mon[88471]: pgmap v781: 109 pgs: 1 activating+remapped, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 4.5 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 500 op/s; 41648/222870 objects degraded (18.687%); 13554/222870 objects misplaced (6.082%); 37 MiB/s, 587 objects/s recovering 2024-09-21T10:07:42.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:43.021 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:43.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:43.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:43.123 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:43.156 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:43.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:43.258 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:43.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:43.299 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:43.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:43.400 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:43.443 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:43.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:43.448 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:43.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:43.549 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:43.575 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:43.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:43.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:43.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:43.706 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:43.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:43.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:43.716 INFO:teuthology.orchestra.run.smithi107.stdout:moved 14 / 327 (4.28135%) 2024-09-21T10:07:43.716 INFO:teuthology.orchestra.run.smithi107.stdout:avg 46.7143 2024-09-21T10:07:43.716 INFO:teuthology.orchestra.run.smithi107.stdout:stddev 6.53562 -> 9.26922 (expected baseline 6.32778) 2024-09-21T10:07:43.716 INFO:teuthology.orchestra.run.smithi107.stdout:min osd.6 with 37 -> 34 pgs (0.792049 -> 0.727829 * mean) 2024-09-21T10:07:43.716 INFO:teuthology.orchestra.run.smithi107.stdout:max osd.1 with 51 -> 57 pgs (1.09174 -> 1.22018 * mean) 2024-09-21T10:07:43.716 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:07:43.717 INFO:teuthology.orchestra.run.smithi107.stdout:oload 110 2024-09-21T10:07:43.717 INFO:teuthology.orchestra.run.smithi107.stdout:max_change 1 2024-09-21T10:07:43.717 INFO:teuthology.orchestra.run.smithi107.stdout:max_change_osds 4 2024-09-21T10:07:43.717 INFO:teuthology.orchestra.run.smithi107.stdout:average_utilization 0.0271 2024-09-21T10:07:43.717 INFO:teuthology.orchestra.run.smithi107.stdout:overload_utilization 0.0298 2024-09-21T10:07:43.717 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 weight 0.6242 -> 0.3743 2024-09-21T10:07:43.717 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4 weight 0.6917 -> 0.5569 2024-09-21T10:07:43.717 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6 weight 0.7731 -> 0.6748 2024-09-21T10:07:43.717 INFO:teuthology.orchestra.run.smithi107.stderr:no change 2024-09-21T10:07:43.808 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:43.844 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:43.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:43.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:43.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:07:43 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T10:07:43.713+0000 7efdb5a17640 -1 mgr.server reply reply (3) No such process no change 2024-09-21T10:07:43.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:07:43.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:43.978 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:43.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:43.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:44.080 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:44.133 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:44.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:44.235 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:44.240 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:07:44.240 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:07:44.240 INFO:teuthology.orchestra.run.smithi129.stdout: "complaint_time": 30, 2024-09-21T10:07:44.240 INFO:teuthology.orchestra.run.smithi129.stdout: "num_blocked_ops": 0 2024-09-21T10:07:44.240 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:07:44.282 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:44.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:44.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:44.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:44.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:44.427 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:44.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:44.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:44.556 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:44.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:44.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:44.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:44.683 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:44.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:44.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:44.784 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:44.821 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:44.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:44.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:44.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:44 smithi107 ceph-mon[102221]: pgmap v782: 109 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 881 op/s; 38896/213675 objects degraded (18.203%); 12421/213675 objects misplaced (5.813%); 47 MiB/s, 754 objects/s recovering 2024-09-21T10:07:44.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:44 smithi107 ceph-mon[102221]: from='client.46157 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:44.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:44 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 38896/213675 objects degraded (18.203%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:44.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:44 smithi107 ceph-mon[105022]: pgmap v782: 109 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 881 op/s; 38896/213675 objects degraded (18.203%); 12421/213675 objects misplaced (5.813%); 47 MiB/s, 754 objects/s recovering 2024-09-21T10:07:44.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:44 smithi107 ceph-mon[105022]: from='client.46157 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:44.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:44 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 38896/213675 objects degraded (18.203%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:44 smithi129 ceph-mon[88471]: pgmap v782: 109 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 881 op/s; 38896/213675 objects degraded (18.203%); 12421/213675 objects misplaced (5.813%); 47 MiB/s, 754 objects/s recovering 2024-09-21T10:07:44.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:44 smithi129 ceph-mon[88471]: from='client.46157 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:44.922 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:44 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 38896/213675 objects degraded (18.203%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:44.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:44.967 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:44.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:44.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:45.068 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:45.109 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:45.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:45.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:45.211 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:45.243 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:45.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:45.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:45.345 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:45.382 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:45.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:45.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:45.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:45.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:45.509 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:45.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:45.610 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:45.644 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:45.655 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:45.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:45.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:45.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:45.784 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:45.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:45.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:45.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:45.907 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:45.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:45.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:46.009 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:46.034 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:46.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:46.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:46.136 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:46.180 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:46.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:46.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:46.282 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:46.307 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:46.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:46.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:46.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:46.437 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:46.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:46.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:46.539 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:46.574 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:46.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:46.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:46.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:46.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:46.704 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:46.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:46.805 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:46.813 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:46 smithi107 ceph-mon[102221]: pgmap v783: 109 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 847 op/s; 38087/210426 objects degraded (18.100%); 12421/210426 objects misplaced (5.903%); 40 MiB/s, 634 objects/s recovering 2024-09-21T10:07:46.813 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:46 smithi107 ceph-mon[105022]: pgmap v783: 109 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 847 op/s; 38087/210426 objects degraded (18.100%); 12421/210426 objects misplaced (5.903%); 40 MiB/s, 634 objects/s recovering 2024-09-21T10:07:46.843 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:46.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:46.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:46.854 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:46 smithi129 ceph-mon[88471]: pgmap v783: 109 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 847 op/s; 38087/210426 objects degraded (18.100%); 12421/210426 objects misplaced (5.903%); 40 MiB/s, 634 objects/s recovering 2024-09-21T10:07:46.883 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:07:46.945 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:47.011 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:47.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:47.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:47.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:47.138 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:47.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:47.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:47.240 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:47.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:47.294 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:47.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:47.390 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:07:47.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:47.420 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:47.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:47.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:47.521 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:47.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:47.569 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:47.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:47.670 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:47.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:47.711 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:47.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:47.812 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:47.842 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:47.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:47.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:47.947 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:47.988 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:47.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:47.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:48.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:48.119 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:48.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:48.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:48.221 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:48.245 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:48.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:48.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:48.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:48.376 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:48.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:48.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:48.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:48.506 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:48.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:48.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:48.521 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:07:48.608 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:48.639 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:48.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:48.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:48.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:48 smithi129 ceph-mon[88471]: pgmap v784: 109 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 723 op/s; 37825/209640 objects degraded (18.043%); 12421/209640 objects misplaced (5.925%); 33 MiB/s, 521 objects/s recovering 2024-09-21T10:07:48.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:48 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1276039259' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:48.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:48 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:48.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:48.771 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:48.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:48.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:48.857 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:48 smithi107 ceph-mon[102221]: pgmap v784: 109 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 723 op/s; 37825/209640 objects degraded (18.043%); 12421/209640 objects misplaced (5.925%); 33 MiB/s, 521 objects/s recovering 2024-09-21T10:07:48.857 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:48 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1276039259' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:48.857 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:48 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:48.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:48 smithi107 ceph-mon[105022]: pgmap v784: 109 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 723 op/s; 37825/209640 objects degraded (18.043%); 12421/209640 objects misplaced (5.925%); 33 MiB/s, 521 objects/s recovering 2024-09-21T10:07:48.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:48 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1276039259' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:48.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:48 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:07:48.873 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:48.906 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:48.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:48.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:49.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:49.033 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:49.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:49.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:49.135 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:49.171 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:49.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:49.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:49.273 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:49.309 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:49.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:49.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:49.411 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:49.456 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:49.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:49.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:49.517 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:07:49.518 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 20, 2024-09-21T10:07:49.518 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 600, 2024-09-21T10:07:49.518 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:07:49.518 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.518 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:59377 9.e 9:78629d4b:::benchmark_data_smithi107_152010_object59376:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-21T10:07:49.518 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:28.166811+0000", 2024-09-21T10:07:49.518 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 381.34550990999998, 2024-09-21T10:07:49.518 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.20589206400000001, 2024-09-21T10:07:49.518 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:07:49.518 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:49.518 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:07:49.518 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:07:49.518 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:07:49.519 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 59377 2024-09-21T10:07:49.519 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.519 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:07:49.519 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.519 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:07:49.519 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.166811+0000", 2024-09-21T10:07:49.519 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.519 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.519 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.519 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:07:49.519 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.166811+0000", 2024-09-21T10:07:49.519 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.519 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.519 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.520 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:07:49.520 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.166810+0000", 2024-09-21T10:07:49.520 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:07:49.520 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.520 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.520 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:07:49.520 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.166861+0000", 2024-09-21T10:07:49.520 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0497000000000001e-05 2024-09-21T10:07:49.520 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.520 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.520 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:07:49.520 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.166873+0000", 2024-09-21T10:07:49.520 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2191000000000001e-05 2024-09-21T10:07:49.520 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.521 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.521 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:07:49.521 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.166879+0000", 2024-09-21T10:07:49.521 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0290000000000002e-06 2024-09-21T10:07:49.521 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.521 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.521 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.521 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.166916+0000", 2024-09-21T10:07:49.521 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6696000000000002e-05 2024-09-21T10:07:49.521 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.521 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.521 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:07:49.521 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.167010+0000", 2024-09-21T10:07:49.521 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.4344000000000003e-05 2024-09-21T10:07:49.522 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.522 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.522 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:07:49.522 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.167108+0000", 2024-09-21T10:07:49.522 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.7974999999999996e-05 2024-09-21T10:07:49.522 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.522 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.522 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:07:49.522 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.168751+0000", 2024-09-21T10:07:49.522 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0016428359999999999 2024-09-21T10:07:49.522 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.522 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.522 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.522 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.176900+0000", 2024-09-21T10:07:49.523 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0081491519999999998 2024-09-21T10:07:49.523 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.523 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.523 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.523 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.372549+0000", 2024-09-21T10:07:49.523 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.195649398 2024-09-21T10:07:49.523 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.523 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.523 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:07:49.523 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.372605+0000", 2024-09-21T10:07:49.523 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6098999999999999e-05 2024-09-21T10:07:49.523 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.523 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.523 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:07:49.524 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.372704+0000", 2024-09-21T10:07:49.524 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.8083000000000003e-05 2024-09-21T10:07:49.524 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.524 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.524 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.524 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.524 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.524 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:59390 9.e 9:742efc15:::benchmark_data_smithi107_152010_object59389:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e154)", 2024-09-21T10:07:49.524 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:28.180915+0000", 2024-09-21T10:07:49.524 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 381.33140640400001, 2024-09-21T10:07:49.524 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.192036757, 2024-09-21T10:07:49.524 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:07:49.524 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:49.524 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:07:49.525 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:07:49.525 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:07:49.525 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 59390 2024-09-21T10:07:49.525 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.525 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:07:49.525 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.525 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:07:49.525 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.180915+0000", 2024-09-21T10:07:49.525 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.525 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.525 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.525 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:07:49.525 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.180915+0000", 2024-09-21T10:07:49.525 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.525 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.526 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.526 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:07:49.526 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.180910+0000", 2024-09-21T10:07:49.526 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999948 2024-09-21T10:07:49.526 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.526 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.526 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:07:49.526 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.180980+0000", 2024-09-21T10:07:49.526 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.0110000000000005e-05 2024-09-21T10:07:49.526 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.526 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.526 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:07:49.526 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.180991+0000", 2024-09-21T10:07:49.526 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0947999999999999e-05 2024-09-21T10:07:49.527 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.527 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.527 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:07:49.527 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.180996+0000", 2024-09-21T10:07:49.527 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1109999999999999e-06 2024-09-21T10:07:49.527 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.527 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.527 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.527 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.181029+0000", 2024-09-21T10:07:49.527 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3013999999999999e-05 2024-09-21T10:07:49.527 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.527 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.527 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:07:49.527 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.181120+0000", 2024-09-21T10:07:49.528 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.1198000000000003e-05 2024-09-21T10:07:49.528 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.528 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.528 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:07:49.528 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.181216+0000", 2024-09-21T10:07:49.528 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.6146999999999999e-05 2024-09-21T10:07:49.528 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.528 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.528 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:07:49.528 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.182799+0000", 2024-09-21T10:07:49.528 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0015824260000000001 2024-09-21T10:07:49.528 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.528 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.528 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.528 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.184037+0000", 2024-09-21T10:07:49.529 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0012377480000000001 2024-09-21T10:07:49.529 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.529 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.529 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.529 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.372820+0000", 2024-09-21T10:07:49.529 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.18878344599999999 2024-09-21T10:07:49.529 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.529 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.529 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:07:49.529 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.372863+0000", 2024-09-21T10:07:49.529 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3506e-05 2024-09-21T10:07:49.529 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.529 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.529 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:07:49.529 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:28.372952+0000", 2024-09-21T10:07:49.530 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.8238000000000005e-05 2024-09-21T10:07:49.530 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.530 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.530 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.530 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.530 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.530 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:74508 9.e 9:73ae9622:::benchmark_data_smithi107_152010_object74507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-21T10:07:49.530 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:41.860213+0000", 2024-09-21T10:07:49.530 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 367.65210869999999, 2024-09-21T10:07:49.530 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.19656214699999999, 2024-09-21T10:07:49.530 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:07:49.530 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:49.530 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:07:49.530 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:07:49.531 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:07:49.531 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 74508 2024-09-21T10:07:49.531 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.531 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:07:49.531 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.531 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:07:49.531 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.860213+0000", 2024-09-21T10:07:49.531 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.531 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.531 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.531 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:07:49.531 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.860213+0000", 2024-09-21T10:07:49.531 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.532 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.532 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.532 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:07:49.532 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.860211+0000", 2024-09-21T10:07:49.532 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:07:49.532 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.532 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.532 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:07:49.532 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.860266+0000", 2024-09-21T10:07:49.532 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4657000000000002e-05 2024-09-21T10:07:49.532 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.532 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.532 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:07:49.532 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.860277+0000", 2024-09-21T10:07:49.533 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1646e-05 2024-09-21T10:07:49.533 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.533 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.533 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:07:49.533 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.860283+0000", 2024-09-21T10:07:49.533 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2460000000000003e-06 2024-09-21T10:07:49.533 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.533 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.533 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.533 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.860334+0000", 2024-09-21T10:07:49.533 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1134999999999997e-05 2024-09-21T10:07:49.533 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.533 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.533 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:07:49.534 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.860474+0000", 2024-09-21T10:07:49.534 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014038599999999999 2024-09-21T10:07:49.534 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.534 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.534 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:07:49.534 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.860628+0000", 2024-09-21T10:07:49.534 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015359299999999999 2024-09-21T10:07:49.534 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.534 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.534 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:07:49.534 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.862650+0000", 2024-09-21T10:07:49.534 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0020217939999999999 2024-09-21T10:07:49.534 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.535 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.535 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.535 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:41.863490+0000", 2024-09-21T10:07:49.535 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00084041499999999996 2024-09-21T10:07:49.535 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.535 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.535 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.535 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:42.056663+0000", 2024-09-21T10:07:49.535 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.19317345899999999 2024-09-21T10:07:49.535 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.535 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.535 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:07:49.535 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:42.056709+0000", 2024-09-21T10:07:49.535 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5275000000000003e-05 2024-09-21T10:07:49.536 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.536 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.536 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:07:49.536 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:42.056775+0000", 2024-09-21T10:07:49.536 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6149999999999995e-05 2024-09-21T10:07:49.536 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.536 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.536 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.536 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.536 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.536 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:101224 9.e 9:7283d620:::benchmark_data_smithi107_152010_object101223:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-21T10:07:49.536 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:09.474070+0000", 2024-09-21T10:07:49.536 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 340.03825178400001, 2024-09-21T10:07:49.537 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28291996200000002, 2024-09-21T10:07:49.537 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:07:49.537 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:49.537 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:07:49.537 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:07:49.537 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:07:49.537 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 101224 2024-09-21T10:07:49.537 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.537 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:07:49.537 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.537 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:07:49.537 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474070+0000", 2024-09-21T10:07:49.537 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.537 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.538 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.538 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:07:49.538 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474070+0000", 2024-09-21T10:07:49.538 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.538 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.538 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.538 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:07:49.538 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474068+0000", 2024-09-21T10:07:49.538 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:07:49.538 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.538 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.538 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:07:49.538 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474170+0000", 2024-09-21T10:07:49.538 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000102308 2024-09-21T10:07:49.539 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.539 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.539 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:07:49.539 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474185+0000", 2024-09-21T10:07:49.539 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.448e-05 2024-09-21T10:07:49.539 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.539 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.539 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:07:49.539 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474190+0000", 2024-09-21T10:07:49.539 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5969999999999999e-06 2024-09-21T10:07:49.539 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.539 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.539 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.539 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474235+0000", 2024-09-21T10:07:49.539 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4521000000000003e-05 2024-09-21T10:07:49.540 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.540 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.540 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:07:49.540 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474327+0000", 2024-09-21T10:07:49.540 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.2227999999999999e-05 2024-09-21T10:07:49.540 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.540 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.540 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:07:49.540 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474424+0000", 2024-09-21T10:07:49.540 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.6824999999999995e-05 2024-09-21T10:07:49.540 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.540 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.540 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:07:49.540 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.477173+0000", 2024-09-21T10:07:49.540 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0027488759999999999 2024-09-21T10:07:49.541 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.541 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.541 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.541 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.481128+0000", 2024-09-21T10:07:49.541 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003955592 2024-09-21T10:07:49.541 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.541 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.541 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.541 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.756811+0000", 2024-09-21T10:07:49.541 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.27568240500000002 2024-09-21T10:07:49.541 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.541 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.541 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:07:49.541 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.756879+0000", 2024-09-21T10:07:49.542 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8109999999999997e-05 2024-09-21T10:07:49.542 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.542 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.542 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:07:49.542 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.756990+0000", 2024-09-21T10:07:49.542 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000110536 2024-09-21T10:07:49.542 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.542 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.542 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.542 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.542 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.542 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:101232 9.e 9:7d6e3112:::benchmark_data_smithi107_152010_object101231:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-21T10:07:49.542 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:09.490031+0000", 2024-09-21T10:07:49.542 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 340.02229066000001, 2024-09-21T10:07:49.543 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.27560621600000001, 2024-09-21T10:07:49.543 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:07:49.543 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:49.543 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:07:49.543 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:07:49.543 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:07:49.543 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 101232 2024-09-21T10:07:49.543 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.543 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:07:49.543 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.543 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:07:49.543 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.490031+0000", 2024-09-21T10:07:49.543 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.543 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.544 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.544 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:07:49.544 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.490031+0000", 2024-09-21T10:07:49.544 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.544 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.544 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.544 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:07:49.545 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.490029+0000", 2024-09-21T10:07:49.545 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:07:49.545 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.545 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.545 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:07:49.545 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.490091+0000", 2024-09-21T10:07:49.545 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1692999999999994e-05 2024-09-21T10:07:49.545 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.545 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.545 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:07:49.545 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.490105+0000", 2024-09-21T10:07:49.546 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3743999999999999e-05 2024-09-21T10:07:49.546 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.546 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.546 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:07:49.546 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.490110+0000", 2024-09-21T10:07:49.546 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1510000000000003e-06 2024-09-21T10:07:49.546 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.546 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.546 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.546 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.490155+0000", 2024-09-21T10:07:49.546 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5104e-05 2024-09-21T10:07:49.546 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.546 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.546 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:07:49.547 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.490253+0000", 2024-09-21T10:07:49.547 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.8528000000000003e-05 2024-09-21T10:07:49.547 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.547 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.547 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:07:49.547 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.490342+0000", 2024-09-21T10:07:49.547 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.8349000000000004e-05 2024-09-21T10:07:49.547 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.547 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.547 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:07:49.547 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.492828+0000", 2024-09-21T10:07:49.547 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002486055 2024-09-21T10:07:49.547 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.547 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.548 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.548 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.493332+0000", 2024-09-21T10:07:49.548 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00050429700000000001 2024-09-21T10:07:49.548 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.548 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.548 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.548 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.765486+0000", 2024-09-21T10:07:49.548 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.27215410099999998 2024-09-21T10:07:49.548 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.548 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.548 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:07:49.548 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.765551+0000", 2024-09-21T10:07:49.548 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4460000000000003e-05 2024-09-21T10:07:49.548 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.548 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.549 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:07:49.549 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.765637+0000", 2024-09-21T10:07:49.549 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6267999999999995e-05 2024-09-21T10:07:49.549 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.549 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.549 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.549 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.549 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.549 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:32049 10.1 10:842495b3:::benchmark_data_smithi107_184135_object32048:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-21T10:07:49.549 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:20.697016+0000", 2024-09-21T10:07:49.549 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 148.81530556300001, 2024-09-21T10:07:49.549 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.186558366, 2024-09-21T10:07:49.549 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:07:49.549 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:49.550 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:07:49.550 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:07:49.550 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:07:49.550 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32049 2024-09-21T10:07:49.550 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.550 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:07:49.550 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.550 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:07:49.550 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.697016+0000", 2024-09-21T10:07:49.550 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.550 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.550 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.550 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:07:49.550 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.697016+0000", 2024-09-21T10:07:49.551 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.551 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.551 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.551 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:07:49.551 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.697015+0000", 2024-09-21T10:07:49.551 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:07:49.551 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.551 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.551 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:07:49.551 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.697209+0000", 2024-09-21T10:07:49.551 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000194336 2024-09-21T10:07:49.551 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.551 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.551 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:07:49.551 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.697223+0000", 2024-09-21T10:07:49.552 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3603e-05 2024-09-21T10:07:49.552 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.552 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.552 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:07:49.552 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.697229+0000", 2024-09-21T10:07:49.552 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.618e-06 2024-09-21T10:07:49.552 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.552 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.552 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.552 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.697859+0000", 2024-09-21T10:07:49.552 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00062961199999999999 2024-09-21T10:07:49.552 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.552 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.552 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:07:49.553 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.697957+0000", 2024-09-21T10:07:49.553 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.7594000000000001e-05 2024-09-21T10:07:49.553 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.553 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.553 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:07:49.553 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.698074+0000", 2024-09-21T10:07:49.553 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000117632 2024-09-21T10:07:49.553 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.553 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.553 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:07:49.553 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.699441+0000", 2024-09-21T10:07:49.553 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0013663359999999999 2024-09-21T10:07:49.553 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.553 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.553 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.553 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.701303+0000", 2024-09-21T10:07:49.554 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001862753 2024-09-21T10:07:49.554 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.554 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.554 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.554 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.883355+0000", 2024-09-21T10:07:49.554 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.18205179699999999 2024-09-21T10:07:49.554 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.554 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.554 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:07:49.554 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.883456+0000", 2024-09-21T10:07:49.554 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000101346 2024-09-21T10:07:49.554 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.554 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.554 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:07:49.554 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:20.883574+0000", 2024-09-21T10:07:49.555 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000117674 2024-09-21T10:07:49.555 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.555 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.555 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.555 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.555 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.555 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95096 10.1 10:8e5d57bd:::benchmark_data_smithi107_184135_object95095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T10:07:49.555 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.382152+0000", 2024-09-21T10:07:49.555 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 103.130169674, 2024-09-21T10:07:49.555 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0260022529999999, 2024-09-21T10:07:49.555 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:07:49.555 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:07:49.556 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:07:49.556 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:07:49.556 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:07:49.556 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95096 2024-09-21T10:07:49.556 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.556 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:07:49.556 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.556 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:07:49.556 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.382152+0000", 2024-09-21T10:07:49.556 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.556 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.556 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.556 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:07:49.556 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.382152+0000", 2024-09-21T10:07:49.557 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.557 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.557 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.557 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:07:49.557 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.382151+0000", 2024-09-21T10:07:49.557 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:07:49.557 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.557 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.557 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:07:49.557 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.382242+0000", 2024-09-21T10:07:49.557 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.0941999999999998e-05 2024-09-21T10:07:49.557 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.557 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.557 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:07:49.558 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.382256+0000", 2024-09-21T10:07:49.558 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3987e-05 2024-09-21T10:07:49.558 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.558 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.558 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:07:49.558 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.382261+0000", 2024-09-21T10:07:49.558 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5350000000000004e-06 2024-09-21T10:07:49.558 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.558 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.558 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.558 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.384419+0000", 2024-09-21T10:07:49.558 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002158185 2024-09-21T10:07:49.558 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.558 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.559 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:07:49.559 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.384566+0000", 2024-09-21T10:07:49.559 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000146565 2024-09-21T10:07:49.559 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.559 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.559 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:07:49.559 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.384723+0000", 2024-09-21T10:07:49.559 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000156904 2024-09-21T10:07:49.559 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.559 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.559 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.559 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.387646+0000", 2024-09-21T10:07:49.559 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0029228180000000002 2024-09-21T10:07:49.559 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.560 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.560 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:07:49.560 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388645+0000", 2024-09-21T10:07:49.560 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00099896800000000003 2024-09-21T10:07:49.560 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.560 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.560 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.560 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.406885+0000", 2024-09-21T10:07:49.560 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.018240084 2024-09-21T10:07:49.560 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.560 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.560 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.560 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.406893+0000", 2024-09-21T10:07:49.560 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.5879999999999998e-06 2024-09-21T10:07:49.561 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.561 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.561 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.561 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.408020+0000", 2024-09-21T10:07:49.561 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0011265600000001 2024-09-21T10:07:49.561 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.561 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.561 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:07:49.561 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.408154+0000", 2024-09-21T10:07:49.561 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013410900000000001 2024-09-21T10:07:49.561 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.561 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.561 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.561 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.561 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.562 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95084 10.4 10:27466553:::benchmark_data_smithi107_184135_object95083:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-21T10:07:49.562 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.399729+0000", 2024-09-21T10:07:49.562 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 103.112592025, 2024-09-21T10:07:49.562 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0066420840000001, 2024-09-21T10:07:49.562 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:07:49.562 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:07:49.562 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:07:49.562 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:07:49.562 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:07:49.562 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95084 2024-09-21T10:07:49.562 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.562 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:07:49.562 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.563 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:07:49.563 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.399729+0000", 2024-09-21T10:07:49.563 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.563 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.563 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.563 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:07:49.563 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.399729+0000", 2024-09-21T10:07:49.563 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.563 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.563 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.563 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:07:49.563 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.399728+0000", 2024-09-21T10:07:49.563 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:07:49.563 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.563 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.563 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:07:49.563 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.399764+0000", 2024-09-21T10:07:49.564 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5880000000000002e-05 2024-09-21T10:07:49.564 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.564 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.564 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:07:49.564 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.399773+0000", 2024-09-21T10:07:49.564 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.9340000000000001e-06 2024-09-21T10:07:49.564 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.564 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.564 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:07:49.564 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.399779+0000", 2024-09-21T10:07:49.564 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7049999999999996e-06 2024-09-21T10:07:49.564 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.564 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.564 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.564 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.399958+0000", 2024-09-21T10:07:49.565 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000179166 2024-09-21T10:07:49.565 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.565 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.565 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op must wait for map", 2024-09-21T10:07:49.565 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.399965+0000", 2024-09-21T10:07:49.565 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1539999999999999e-06 2024-09-21T10:07:49.565 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.565 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.565 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.565 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.409511+0000", 2024-09-21T10:07:49.565 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0095469490000000008 2024-09-21T10:07:49.565 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.565 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.565 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.565 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.409518+0000", 2024-09-21T10:07:49.565 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.99e-06 2024-09-21T10:07:49.566 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.566 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.566 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.566 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.404037+0000", 2024-09-21T10:07:49.566 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.99451875499999998 2024-09-21T10:07:49.566 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.566 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.566 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:07:49.566 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.406371+0000", 2024-09-21T10:07:49.566 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002334207 2024-09-21T10:07:49.566 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.566 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.566 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.567 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.567 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.567 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95091 10.4 10:21690f8a:::benchmark_data_smithi107_184135_object95090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-21T10:07:49.567 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.399795+0000", 2024-09-21T10:07:49.567 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 103.112526094, 2024-09-21T10:07:49.567 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0066720069999999, 2024-09-21T10:07:49.567 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:07:49.567 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:07:49.567 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:07:49.567 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:07:49.567 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:07:49.567 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95091 2024-09-21T10:07:49.568 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.568 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:07:49.568 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.568 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:07:49.568 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.399795+0000", 2024-09-21T10:07:49.568 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.568 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.568 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.568 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:07:49.568 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.399795+0000", 2024-09-21T10:07:49.568 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.568 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.568 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.568 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:07:49.568 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.399795+0000", 2024-09-21T10:07:49.569 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:07:49.569 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.569 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.569 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:07:49.569 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.399831+0000", 2024-09-21T10:07:49.569 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6671999999999997e-05 2024-09-21T10:07:49.569 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.569 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.569 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:07:49.569 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.399839+0000", 2024-09-21T10:07:49.569 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.7919999999999993e-06 2024-09-21T10:07:49.569 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.569 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.569 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:07:49.569 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.399844+0000", 2024-09-21T10:07:49.570 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4490000000000001e-06 2024-09-21T10:07:49.570 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.570 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.570 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.570 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.400042+0000", 2024-09-21T10:07:49.570 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000198247 2024-09-21T10:07:49.570 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.570 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.570 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting_for_map not empty", 2024-09-21T10:07:49.570 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.400047+0000", 2024-09-21T10:07:49.570 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9740000000000001e-06 2024-09-21T10:07:49.571 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.571 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.571 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.571 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.409547+0000", 2024-09-21T10:07:49.571 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0094998619999999995 2024-09-21T10:07:49.571 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.571 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.571 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.571 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.409552+0000", 2024-09-21T10:07:49.571 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0209999999999999e-06 2024-09-21T10:07:49.571 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.571 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.571 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.571 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.406399+0000", 2024-09-21T10:07:49.572 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.99684761200000005 2024-09-21T10:07:49.572 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.572 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.572 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:07:49.572 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.406467+0000", 2024-09-21T10:07:49.572 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7922000000000005e-05 2024-09-21T10:07:49.572 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.572 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.572 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.572 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.572 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.572 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95098 10.4 10:250057e1:::benchmark_data_smithi107_184135_object95097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-21T10:07:49.572 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.399873+0000", 2024-09-21T10:07:49.572 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 103.112448089, 2024-09-21T10:07:49.572 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0152940859999999, 2024-09-21T10:07:49.573 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:07:49.573 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:49.573 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:07:49.573 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:07:49.573 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:07:49.573 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95098 2024-09-21T10:07:49.573 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.573 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:07:49.573 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.573 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:07:49.573 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.399873+0000", 2024-09-21T10:07:49.573 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.573 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.573 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.574 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:07:49.574 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.399873+0000", 2024-09-21T10:07:49.574 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.574 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.574 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.574 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:07:49.574 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.399872+0000", 2024-09-21T10:07:49.574 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:07:49.574 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.574 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.574 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:07:49.574 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.399902+0000", 2024-09-21T10:07:49.574 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0006999999999999e-05 2024-09-21T10:07:49.574 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.575 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.575 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:07:49.575 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.399912+0000", 2024-09-21T10:07:49.575 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.4599999999999992e-06 2024-09-21T10:07:49.575 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.575 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.575 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:07:49.575 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.399917+0000", 2024-09-21T10:07:49.575 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3329999999999996e-06 2024-09-21T10:07:49.575 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.575 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.575 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.575 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.400062+0000", 2024-09-21T10:07:49.575 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014520200000000001 2024-09-21T10:07:49.576 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.576 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.576 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting_for_map not empty", 2024-09-21T10:07:49.576 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.400067+0000", 2024-09-21T10:07:49.576 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0790000000000002e-06 2024-09-21T10:07:49.576 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.576 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.576 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.576 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.409570+0000", 2024-09-21T10:07:49.576 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0095037439999999997 2024-09-21T10:07:49.576 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.576 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.576 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.576 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.409574+0000", 2024-09-21T10:07:49.576 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0210000000000001e-06 2024-09-21T10:07:49.576 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.577 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.577 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.577 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.406506+0000", 2024-09-21T10:07:49.577 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.99693126899999995 2024-09-21T10:07:49.577 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.577 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.577 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:07:49.577 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.407214+0000", 2024-09-21T10:07:49.577 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00070824600000000005 2024-09-21T10:07:49.577 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.577 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.577 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3", 2024-09-21T10:07:49.577 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.407305+0000", 2024-09-21T10:07:49.577 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.1019000000000004e-05 2024-09-21T10:07:49.578 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.578 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.578 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.578 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.413686+0000", 2024-09-21T10:07:49.578 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0063808049999999998 2024-09-21T10:07:49.578 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.578 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.578 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:07:49.578 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.415049+0000", 2024-09-21T10:07:49.578 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001363202 2024-09-21T10:07:49.578 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.578 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.578 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:07:49.578 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.415094+0000", 2024-09-21T10:07:49.578 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5284999999999998e-05 2024-09-21T10:07:49.579 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.579 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.579 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:07:49.579 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.415167+0000", 2024-09-21T10:07:49.579 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3189999999999996e-05 2024-09-21T10:07:49.579 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.579 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.579 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.579 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.579 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.579 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95109 10.1 10:8af1aad8:::benchmark_data_smithi107_184135_object95108:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-21T10:07:49.579 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.407359+0000", 2024-09-21T10:07:49.579 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 103.104962415, 2024-09-21T10:07:49.579 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.014383362, 2024-09-21T10:07:49.579 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:07:49.580 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:49.580 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:07:49.580 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:07:49.580 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:07:49.580 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95109 2024-09-21T10:07:49.580 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.580 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:07:49.580 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.580 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:07:49.580 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407359+0000", 2024-09-21T10:07:49.580 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.580 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.580 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.580 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:07:49.580 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407359+0000", 2024-09-21T10:07:49.581 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.581 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.581 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.581 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:07:49.581 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407357+0000", 2024-09-21T10:07:49.581 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:07:49.581 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.581 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.581 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:07:49.581 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407388+0000", 2024-09-21T10:07:49.581 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1201000000000002e-05 2024-09-21T10:07:49.581 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.581 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.581 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:07:49.581 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407399+0000", 2024-09-21T10:07:49.581 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0905e-05 2024-09-21T10:07:49.582 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.582 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.582 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:07:49.582 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407412+0000", 2024-09-21T10:07:49.582 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2320000000000001e-05 2024-09-21T10:07:49.582 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.582 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.582 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.582 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.408247+0000", 2024-09-21T10:07:49.582 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00083585899999999999 2024-09-21T10:07:49.582 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.582 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.582 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.582 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.408253+0000", 2024-09-21T10:07:49.583 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6570000000000002e-06 2024-09-21T10:07:49.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.583 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.583 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.583 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.408175+0000", 2024-09-21T10:07:49.583 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.99992146800000004 2024-09-21T10:07:49.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.583 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.583 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:07:49.583 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.408273+0000", 2024-09-21T10:07:49.583 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.8091000000000007e-05 2024-09-21T10:07:49.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.584 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.584 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 7", 2024-09-21T10:07:49.584 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.408400+0000", 2024-09-21T10:07:49.584 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000127463 2024-09-21T10:07:49.584 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.584 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.584 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:07:49.584 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.415020+0000", 2024-09-21T10:07:49.584 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0066195630000000002 2024-09-21T10:07:49.584 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.584 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:49.585 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.585 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.421574+0000", 2024-09-21T10:07:49.586 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0065543340000000002 2024-09-21T10:07:49.586 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.586 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.586 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:07:49.586 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.421657+0000", 2024-09-21T10:07:49.586 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.2819999999999996e-05 2024-09-21T10:07:49.586 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.586 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.586 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:07:49.586 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.421742+0000", 2024-09-21T10:07:49.586 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.5421999999999996e-05 2024-09-21T10:07:49.586 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.587 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.587 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.587 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.587 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.587 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:177921 10.4 10:28af2ff1:::benchmark_data_smithi107_184135_object46355:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-09-21T10:07:49.587 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:07:23.327078+0000", 2024-09-21T10:07:49.587 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 26.185243032999999, 2024-09-21T10:07:49.587 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1501687870000001, 2024-09-21T10:07:49.587 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:07:49.587 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:07:49.587 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:07:49.587 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:07:49.587 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:07:49.587 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 177921 2024-09-21T10:07:49.587 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.588 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:07:49.588 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.588 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:07:49.588 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.327078+0000", 2024-09-21T10:07:49.588 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.588 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.588 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.588 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:07:49.588 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.327078+0000", 2024-09-21T10:07:49.588 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.588 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.588 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.588 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:07:49.589 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.327077+0000", 2024-09-21T10:07:49.589 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:07:49.589 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.589 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.589 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:07:49.589 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.327083+0000", 2024-09-21T10:07:49.589 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4570000000000002e-06 2024-09-21T10:07:49.589 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.589 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.589 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:07:49.589 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.327095+0000", 2024-09-21T10:07:49.589 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1279e-05 2024-09-21T10:07:49.589 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.590 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.590 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:07:49.590 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.327100+0000", 2024-09-21T10:07:49.590 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4190000000000001e-06 2024-09-21T10:07:49.590 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.590 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.590 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.590 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.333800+0000", 2024-09-21T10:07:49.590 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0066994589999999996 2024-09-21T10:07:49.590 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.590 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.590 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:07:49.590 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.333926+0000", 2024-09-21T10:07:49.591 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012686500000000001 2024-09-21T10:07:49.591 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.591 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.591 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3", 2024-09-21T10:07:49.591 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.334048+0000", 2024-09-21T10:07:49.591 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000121412 2024-09-21T10:07:49.591 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.591 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.591 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.591 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.342002+0000", 2024-09-21T10:07:49.591 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0079540270000000007 2024-09-21T10:07:49.591 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.591 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.591 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.591 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.342008+0000", 2024-09-21T10:07:49.592 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4139999999999999e-06 2024-09-21T10:07:49.592 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.592 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.592 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.592 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.345535+0000", 2024-09-21T10:07:49.592 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0035272129999999 2024-09-21T10:07:49.592 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.592 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.592 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.592 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.345545+0000", 2024-09-21T10:07:49.592 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.1919999999999994e-06 2024-09-21T10:07:49.592 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.592 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.592 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.592 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.447819+0000", 2024-09-21T10:07:49.593 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1022742270000001 2024-09-21T10:07:49.593 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.593 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.593 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for missing object", 2024-09-21T10:07:49.593 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.447942+0000", 2024-09-21T10:07:49.593 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012334299999999999 2024-09-21T10:07:49.593 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.593 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.593 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.593 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.477168+0000", 2024-09-21T10:07:49.593 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.029226202 2024-09-21T10:07:49.593 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.593 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.593 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:07:49.593 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.477247+0000", 2024-09-21T10:07:49.594 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.8647999999999996e-05 2024-09-21T10:07:49.594 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.594 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.594 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.594 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.594 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.594 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:177936 10.4 10:2dcd0ed4:::benchmark_data_smithi107_184135_object46370:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-21T10:07:49.594 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:07:23.351510+0000", 2024-09-21T10:07:49.594 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 26.160811450000001, 2024-09-21T10:07:49.594 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1246232909999998, 2024-09-21T10:07:49.594 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:07:49.594 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:49.594 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:07:49.594 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:07:49.595 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:07:49.595 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 177936 2024-09-21T10:07:49.595 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.595 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:07:49.595 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.595 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:07:49.595 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.351510+0000", 2024-09-21T10:07:49.595 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.595 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.595 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.595 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:07:49.595 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.351510+0000", 2024-09-21T10:07:49.595 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.595 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.595 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.596 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:07:49.596 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.351508+0000", 2024-09-21T10:07:49.596 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:07:49.596 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.596 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.596 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:07:49.596 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.351518+0000", 2024-09-21T10:07:49.596 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0051000000000001e-05 2024-09-21T10:07:49.596 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.596 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.596 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:07:49.596 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.351528+0000", 2024-09-21T10:07:49.596 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.9720000000000004e-06 2024-09-21T10:07:49.596 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.597 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.597 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:07:49.597 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.351534+0000", 2024-09-21T10:07:49.597 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4589999999999997e-06 2024-09-21T10:07:49.597 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.597 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.597 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.597 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.351584+0000", 2024-09-21T10:07:49.597 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9904999999999997e-05 2024-09-21T10:07:49.597 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.597 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.597 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.597 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.351591+0000", 2024-09-21T10:07:49.597 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.1879999999999996e-06 2024-09-21T10:07:49.598 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.598 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.598 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.598 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.345571+0000", 2024-09-21T10:07:49.598 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.99397972499999998 2024-09-21T10:07:49.598 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.598 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.598 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.598 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.345577+0000", 2024-09-21T10:07:49.598 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4729999999999997e-06 2024-09-21T10:07:49.598 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.598 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.598 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.598 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.447985+0000", 2024-09-21T10:07:49.598 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.102407922 2024-09-21T10:07:49.598 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.599 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.599 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:07:49.599 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.448078+0000", 2024-09-21T10:07:49.599 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.3292000000000001e-05 2024-09-21T10:07:49.599 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.599 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.599 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:07:49.599 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.448141+0000", 2024-09-21T10:07:49.599 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2675000000000006e-05 2024-09-21T10:07:49.599 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.599 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.599 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:07:49.599 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.455537+0000", 2024-09-21T10:07:49.599 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0073957750000000003 2024-09-21T10:07:49.599 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.599 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.600 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.600 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.475957+0000", 2024-09-21T10:07:49.600 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.020419637000000001 2024-09-21T10:07:49.600 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.600 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.600 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.600 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.475989+0000", 2024-09-21T10:07:49.600 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2391999999999998e-05 2024-09-21T10:07:49.600 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.600 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.600 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:07:49.600 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.476039+0000", 2024-09-21T10:07:49.600 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9672999999999997e-05 2024-09-21T10:07:49.600 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.601 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.601 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:07:49.601 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.476133+0000", 2024-09-21T10:07:49.601 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.4615000000000001e-05 2024-09-21T10:07:49.601 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.601 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.601 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.601 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.601 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.601 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:177941 10.4 10:280948d2:::benchmark_data_smithi107_184135_object46375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-21T10:07:49.601 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:07:23.358586+0000", 2024-09-21T10:07:49.601 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 26.153735394999998, 2024-09-21T10:07:49.601 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1189177579999998, 2024-09-21T10:07:49.601 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:07:49.601 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:49.602 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:07:49.602 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:07:49.602 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:07:49.602 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 177941 2024-09-21T10:07:49.602 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.602 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:07:49.602 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.602 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:07:49.602 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.358586+0000", 2024-09-21T10:07:49.602 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.602 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.602 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.602 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:07:49.602 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.358586+0000", 2024-09-21T10:07:49.603 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.603 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.603 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:07:49.603 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.358584+0000", 2024-09-21T10:07:49.603 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:07:49.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.603 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.603 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:07:49.603 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.358592+0000", 2024-09-21T10:07:49.603 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2840000000000001e-06 2024-09-21T10:07:49.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.603 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.603 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:07:49.603 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.358605+0000", 2024-09-21T10:07:49.604 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2955999999999999e-05 2024-09-21T10:07:49.604 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.604 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.604 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:07:49.604 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.358611+0000", 2024-09-21T10:07:49.604 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0769999999999996e-06 2024-09-21T10:07:49.604 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.604 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.604 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.604 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.358652+0000", 2024-09-21T10:07:49.604 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0933000000000003e-05 2024-09-21T10:07:49.604 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.604 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.604 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.604 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.358660+0000", 2024-09-21T10:07:49.604 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.0220000000000004e-06 2024-09-21T10:07:49.605 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.605 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.605 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.605 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.345718+0000", 2024-09-21T10:07:49.605 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.98705806100000004 2024-09-21T10:07:49.605 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.605 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.605 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.605 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.345728+0000", 2024-09-21T10:07:49.605 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0164e-05 2024-09-21T10:07:49.605 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.605 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.605 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.605 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.450590+0000", 2024-09-21T10:07:49.605 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.104861895 2024-09-21T10:07:49.605 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.606 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.606 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:07:49.606 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.450697+0000", 2024-09-21T10:07:49.606 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000107453 2024-09-21T10:07:49.606 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.606 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.606 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:07:49.606 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.450805+0000", 2024-09-21T10:07:49.606 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000107618 2024-09-21T10:07:49.606 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.606 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.606 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:07:49.606 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.471590+0000", 2024-09-21T10:07:49.606 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.020785716999999999 2024-09-21T10:07:49.606 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.607 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.607 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.607 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.476838+0000", 2024-09-21T10:07:49.607 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0052479909999999996 2024-09-21T10:07:49.607 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.607 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.607 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.607 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.477342+0000", 2024-09-21T10:07:49.607 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00050362499999999995 2024-09-21T10:07:49.607 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.607 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.607 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:07:49.607 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.477389+0000", 2024-09-21T10:07:49.607 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6987999999999998e-05 2024-09-21T10:07:49.607 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.607 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.608 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:07:49.608 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.477504+0000", 2024-09-21T10:07:49.608 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000114633 2024-09-21T10:07:49.608 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.608 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.608 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.608 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.608 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.608 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:177945 10.4 10:2bbb8c40:::benchmark_data_smithi107_184135_object46379:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-21T10:07:49.608 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:07:23.360459+0000", 2024-09-21T10:07:49.608 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 26.151862439999999, 2024-09-21T10:07:49.608 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.11721775, 2024-09-21T10:07:49.608 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:07:49.608 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:49.609 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:07:49.609 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:07:49.609 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:07:49.609 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 177945 2024-09-21T10:07:49.609 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.609 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:07:49.609 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.609 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:07:49.609 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.360459+0000", 2024-09-21T10:07:49.609 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.609 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.609 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.609 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:07:49.609 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.360459+0000", 2024-09-21T10:07:49.610 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.610 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.610 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.610 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:07:49.610 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.360458+0000", 2024-09-21T10:07:49.610 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:07:49.610 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.610 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.610 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:07:49.610 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.360464+0000", 2024-09-21T10:07:49.610 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1589999999999998e-06 2024-09-21T10:07:49.610 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.610 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.611 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:07:49.611 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.360478+0000", 2024-09-21T10:07:49.611 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.454e-05 2024-09-21T10:07:49.611 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.611 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.611 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:07:49.611 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.360484+0000", 2024-09-21T10:07:49.611 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7710000000000003e-06 2024-09-21T10:07:49.611 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.611 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.611 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.611 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.360523+0000", 2024-09-21T10:07:49.611 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8896000000000002e-05 2024-09-21T10:07:49.611 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.611 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.612 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.612 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.360530+0000", 2024-09-21T10:07:49.612 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9240000000000001e-06 2024-09-21T10:07:49.612 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.612 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.612 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.612 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.345988+0000", 2024-09-21T10:07:49.612 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.98545798699999998 2024-09-21T10:07:49.612 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.612 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.612 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.612 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.345996+0000", 2024-09-21T10:07:49.612 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.3459999999999994e-06 2024-09-21T10:07:49.612 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.612 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.613 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.613 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.453053+0000", 2024-09-21T10:07:49.613 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.107056298 2024-09-21T10:07:49.613 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.613 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.613 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:07:49.613 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.453141+0000", 2024-09-21T10:07:49.613 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.8280000000000002e-05 2024-09-21T10:07:49.613 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.613 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.613 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:07:49.613 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.453240+0000", 2024-09-21T10:07:49.613 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.9548000000000003e-05 2024-09-21T10:07:49.613 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.613 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.614 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:07:49.614 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.476848+0000", 2024-09-21T10:07:49.614 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.023607408999999999 2024-09-21T10:07:49.614 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.614 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.614 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.614 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.477281+0000", 2024-09-21T10:07:49.614 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00043283500000000001 2024-09-21T10:07:49.614 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.614 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.614 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.614 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.477537+0000", 2024-09-21T10:07:49.614 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00025659199999999999 2024-09-21T10:07:49.614 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.614 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.615 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:07:49.615 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.477582+0000", 2024-09-21T10:07:49.615 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4458e-05 2024-09-21T10:07:49.615 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.615 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.615 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:07:49.615 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.477677+0000", 2024-09-21T10:07:49.615 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.4897999999999998e-05 2024-09-21T10:07:49.615 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.615 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.615 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.615 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.615 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.615 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:177957 10.4 10:2fd1f4d8:::benchmark_data_smithi107_184135_object46391:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-21T10:07:49.615 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:07:23.371677+0000", 2024-09-21T10:07:49.616 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 26.140644073000001, 2024-09-21T10:07:49.616 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1061974530000001, 2024-09-21T10:07:49.616 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:07:49.616 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:49.616 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:07:49.616 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:07:49.616 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:07:49.616 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 177957 2024-09-21T10:07:49.616 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.616 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:07:49.616 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.616 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:07:49.616 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.371677+0000", 2024-09-21T10:07:49.616 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.616 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.616 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.617 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:07:49.617 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.371677+0000", 2024-09-21T10:07:49.617 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.617 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.617 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.617 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:07:49.617 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.371675+0000", 2024-09-21T10:07:49.617 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:07:49.617 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.617 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.617 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:07:49.617 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.371683+0000", 2024-09-21T10:07:49.617 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.6229999999999997e-06 2024-09-21T10:07:49.617 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.618 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.618 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:07:49.618 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.371696+0000", 2024-09-21T10:07:49.618 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2948e-05 2024-09-21T10:07:49.618 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.618 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.618 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:07:49.618 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.371702+0000", 2024-09-21T10:07:49.618 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.976e-06 2024-09-21T10:07:49.618 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.618 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.618 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.618 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.371746+0000", 2024-09-21T10:07:49.618 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3729000000000001e-05 2024-09-21T10:07:49.619 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.619 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.619 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.619 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.371752+0000", 2024-09-21T10:07:49.619 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7440000000000001e-06 2024-09-21T10:07:49.619 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.619 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.619 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.619 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.346029+0000", 2024-09-21T10:07:49.619 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.97427632799999997 2024-09-21T10:07:49.619 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.619 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.619 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.619 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.346034+0000", 2024-09-21T10:07:49.619 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5319999999999997e-06 2024-09-21T10:07:49.620 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.620 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.620 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.620 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.455578+0000", 2024-09-21T10:07:49.620 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1095437560000001 2024-09-21T10:07:49.620 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.620 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.620 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:07:49.620 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.455710+0000", 2024-09-21T10:07:49.620 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013155799999999999 2024-09-21T10:07:49.620 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.620 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.620 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:07:49.621 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.455835+0000", 2024-09-21T10:07:49.621 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012517600000000001 2024-09-21T10:07:49.621 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.621 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.621 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:07:49.621 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.476854+0000", 2024-09-21T10:07:49.621 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.021019065999999999 2024-09-21T10:07:49.621 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.621 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.621 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.621 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.477312+0000", 2024-09-21T10:07:49.621 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00045857800000000001 2024-09-21T10:07:49.621 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.621 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.621 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.621 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.477738+0000", 2024-09-21T10:07:49.622 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00042566499999999998 2024-09-21T10:07:49.622 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.622 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.622 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:07:49.622 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.477783+0000", 2024-09-21T10:07:49.622 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4586999999999998e-05 2024-09-21T10:07:49.622 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.622 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.622 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:07:49.622 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.477875+0000", 2024-09-21T10:07:49.622 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.1997999999999996e-05 2024-09-21T10:07:49.622 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.622 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.622 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.622 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.623 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.623 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:177965 10.4 10:2f0b43a2:::benchmark_data_smithi107_184135_object46399:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-21T10:07:49.623 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:07:23.377521+0000", 2024-09-21T10:07:49.623 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 26.134800709, 2024-09-21T10:07:49.623 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.100558473, 2024-09-21T10:07:49.623 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:07:49.623 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:49.623 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:07:49.623 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:07:49.623 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:07:49.623 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 177965 2024-09-21T10:07:49.623 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.623 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:07:49.623 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.623 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:07:49.624 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.377521+0000", 2024-09-21T10:07:49.624 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.624 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.624 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.624 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:07:49.624 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.377521+0000", 2024-09-21T10:07:49.624 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.624 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.624 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.624 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:07:49.625 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.377520+0000", 2024-09-21T10:07:49.625 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:07:49.625 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.625 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.625 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:07:49.626 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.377524+0000", 2024-09-21T10:07:49.626 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6940000000000003e-06 2024-09-21T10:07:49.626 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.626 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.626 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:07:49.626 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.377535+0000", 2024-09-21T10:07:49.626 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0291e-05 2024-09-21T10:07:49.626 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.626 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.626 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:07:49.626 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.377541+0000", 2024-09-21T10:07:49.626 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3389999999999998e-06 2024-09-21T10:07:49.627 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.627 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.627 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.627 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.377589+0000", 2024-09-21T10:07:49.627 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8090999999999997e-05 2024-09-21T10:07:49.627 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.627 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.627 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.627 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.377598+0000", 2024-09-21T10:07:49.627 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.7630000000000006e-06 2024-09-21T10:07:49.627 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.627 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.627 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.627 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.346057+0000", 2024-09-21T10:07:49.627 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.96845874200000004 2024-09-21T10:07:49.628 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.628 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.628 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.628 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.346064+0000", 2024-09-21T10:07:49.628 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9639999999999997e-06 2024-09-21T10:07:49.628 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.628 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.628 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.628 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.459567+0000", 2024-09-21T10:07:49.628 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.113503922 2024-09-21T10:07:49.628 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.628 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.628 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:07:49.628 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.459682+0000", 2024-09-21T10:07:49.629 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011469200000000001 2024-09-21T10:07:49.629 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.629 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.629 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:07:49.629 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.459795+0000", 2024-09-21T10:07:49.629 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011319699999999999 2024-09-21T10:07:49.629 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.629 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.629 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:07:49.629 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.476860+0000", 2024-09-21T10:07:49.629 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.017064717 2024-09-21T10:07:49.629 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.629 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.629 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.630 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.477709+0000", 2024-09-21T10:07:49.630 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00084885600000000005 2024-09-21T10:07:49.630 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.630 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.630 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.630 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.477915+0000", 2024-09-21T10:07:49.630 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00020594400000000001 2024-09-21T10:07:49.630 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.630 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.630 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:07:49.630 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.477990+0000", 2024-09-21T10:07:49.630 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.5174999999999998e-05 2024-09-21T10:07:49.630 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.631 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.631 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:07:49.631 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.478079+0000", 2024-09-21T10:07:49.631 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.9079999999999994e-05 2024-09-21T10:07:49.631 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.631 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.631 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.631 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.631 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.631 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:177988 10.4 10:23ee0563:::benchmark_data_smithi107_184135_object46422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-21T10:07:49.631 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:07:23.401801+0000", 2024-09-21T10:07:49.631 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 26.110520659999999, 2024-09-21T10:07:49.631 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.0765722590000002, 2024-09-21T10:07:49.632 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:07:49.632 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:49.632 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:07:49.632 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:07:49.632 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:07:49.632 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 177988 2024-09-21T10:07:49.632 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.632 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:07:49.632 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.632 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:07:49.632 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.401801+0000", 2024-09-21T10:07:49.632 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.632 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.632 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.632 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:07:49.632 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.401801+0000", 2024-09-21T10:07:49.633 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.633 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.633 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.633 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:07:49.633 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.401799+0000", 2024-09-21T10:07:49.633 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:07:49.633 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.633 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.633 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:07:49.633 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.401805+0000", 2024-09-21T10:07:49.633 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6840000000000001e-06 2024-09-21T10:07:49.633 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.633 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.633 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:07:49.633 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.401817+0000", 2024-09-21T10:07:49.633 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2281e-05 2024-09-21T10:07:49.634 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.634 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.634 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:07:49.634 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.401821+0000", 2024-09-21T10:07:49.634 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3080000000000001e-06 2024-09-21T10:07:49.634 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.634 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.634 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.634 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.401855+0000", 2024-09-21T10:07:49.634 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3525000000000002e-05 2024-09-21T10:07:49.634 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.634 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.634 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.635 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.401860+0000", 2024-09-21T10:07:49.635 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9300000000000002e-06 2024-09-21T10:07:49.635 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.635 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.635 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.635 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.346117+0000", 2024-09-21T10:07:49.635 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.944257122 2024-09-21T10:07:49.635 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.635 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.635 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.635 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.346123+0000", 2024-09-21T10:07:49.635 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7039999999999999e-06 2024-09-21T10:07:49.635 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.635 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.635 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.636 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.462847+0000", 2024-09-21T10:07:49.636 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1167242019999999 2024-09-21T10:07:49.636 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.636 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.636 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:07:49.636 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.462991+0000", 2024-09-21T10:07:49.636 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014423999999999999 2024-09-21T10:07:49.636 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.636 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.636 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:07:49.636 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.463134+0000", 2024-09-21T10:07:49.636 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000143287 2024-09-21T10:07:49.636 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.636 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.636 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:07:49.637 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.476866+0000", 2024-09-21T10:07:49.637 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.013731205 2024-09-21T10:07:49.637 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.637 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.637 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.637 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.478113+0000", 2024-09-21T10:07:49.637 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0012477180000000001 2024-09-21T10:07:49.637 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.637 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.637 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.637 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.478141+0000", 2024-09-21T10:07:49.637 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7141e-05 2024-09-21T10:07:49.637 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.637 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.637 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:07:49.637 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.478215+0000", 2024-09-21T10:07:49.638 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.4349000000000003e-05 2024-09-21T10:07:49.638 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.638 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.638 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:07:49.638 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.478373+0000", 2024-09-21T10:07:49.638 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015809299999999999 2024-09-21T10:07:49.638 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.638 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.638 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.638 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.638 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.638 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:177995 10.4 10:2cda8116:::benchmark_data_smithi107_184135_object46429:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-21T10:07:49.638 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:07:23.416432+0000", 2024-09-21T10:07:49.638 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 26.095889201999999, 2024-09-21T10:07:49.639 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.0621494290000002, 2024-09-21T10:07:49.639 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:07:49.639 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:49.639 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:07:49.639 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:07:49.639 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:07:49.639 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 177995 2024-09-21T10:07:49.639 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.639 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:07:49.639 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.639 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:07:49.639 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.416432+0000", 2024-09-21T10:07:49.639 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.639 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.639 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.639 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:07:49.640 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.416432+0000", 2024-09-21T10:07:49.640 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.640 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.640 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.640 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:07:49.640 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.416431+0000", 2024-09-21T10:07:49.640 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:07:49.640 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.640 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.640 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:07:49.640 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.416436+0000", 2024-09-21T10:07:49.640 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9289999999999997e-06 2024-09-21T10:07:49.640 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.640 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.641 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:07:49.641 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.416445+0000", 2024-09-21T10:07:49.641 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.4490000000000005e-06 2024-09-21T10:07:49.641 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.641 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.641 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:07:49.641 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.416449+0000", 2024-09-21T10:07:49.641 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.9060000000000004e-06 2024-09-21T10:07:49.641 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.641 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.641 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.641 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.416481+0000", 2024-09-21T10:07:49.641 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1678999999999999e-05 2024-09-21T10:07:49.642 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.642 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.642 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.642 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.416486+0000", 2024-09-21T10:07:49.642 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.566e-06 2024-09-21T10:07:49.642 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.642 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.642 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.642 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.346149+0000", 2024-09-21T10:07:49.642 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.92966353599999996 2024-09-21T10:07:49.642 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.643 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.643 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.643 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.346154+0000", 2024-09-21T10:07:49.643 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2120000000000002e-06 2024-09-21T10:07:49.643 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.643 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.643 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.643 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.466108+0000", 2024-09-21T10:07:49.643 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.119953677 2024-09-21T10:07:49.643 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.643 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.643 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:07:49.643 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.466238+0000", 2024-09-21T10:07:49.643 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013039900000000001 2024-09-21T10:07:49.644 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.644 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.644 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:07:49.644 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.466375+0000", 2024-09-21T10:07:49.644 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000136258 2024-09-21T10:07:49.644 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.644 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.644 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:07:49.644 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.477139+0000", 2024-09-21T10:07:49.644 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.010764245 2024-09-21T10:07:49.644 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.644 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.644 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.644 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.478416+0000", 2024-09-21T10:07:49.644 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0012772250000000001 2024-09-21T10:07:49.645 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.645 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.645 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.645 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.478446+0000", 2024-09-21T10:07:49.645 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0082000000000001e-05 2024-09-21T10:07:49.645 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.645 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.645 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:07:49.645 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.478489+0000", 2024-09-21T10:07:49.645 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2515999999999997e-05 2024-09-21T10:07:49.645 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.645 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.645 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:07:49.645 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.478582+0000", 2024-09-21T10:07:49.645 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.2810000000000001e-05 2024-09-21T10:07:49.645 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.646 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.646 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.646 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.646 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.646 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:177996 10.4 10:2c30373c:::benchmark_data_smithi107_184135_object46430:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-21T10:07:49.646 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:07:23.417445+0000", 2024-09-21T10:07:49.646 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 26.094875982000001, 2024-09-21T10:07:49.646 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.0613115999999998, 2024-09-21T10:07:49.646 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:07:49.646 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:07:49.646 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:07:49.646 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:07:49.646 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:07:49.646 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 177996 2024-09-21T10:07:49.646 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.647 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:07:49.647 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.647 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:07:49.647 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.417445+0000", 2024-09-21T10:07:49.647 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.647 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.647 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.647 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:07:49.647 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.417445+0000", 2024-09-21T10:07:49.647 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:07:49.647 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.647 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.647 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:07:49.647 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.417445+0000", 2024-09-21T10:07:49.647 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:07:49.648 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.648 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.648 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:07:49.648 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.417449+0000", 2024-09-21T10:07:49.648 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7480000000000001e-06 2024-09-21T10:07:49.648 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.648 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.648 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:07:49.649 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.417461+0000", 2024-09-21T10:07:49.649 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1997999999999999e-05 2024-09-21T10:07:49.650 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.650 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.650 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:07:49.650 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.417467+0000", 2024-09-21T10:07:49.650 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6169999999999998e-06 2024-09-21T10:07:49.650 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.650 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.650 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.650 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.417508+0000", 2024-09-21T10:07:49.650 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1106999999999998e-05 2024-09-21T10:07:49.650 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.650 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.650 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.650 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.417513+0000", 2024-09-21T10:07:49.651 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9609999999999996e-06 2024-09-21T10:07:49.651 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.651 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.651 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.651 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.346581+0000", 2024-09-21T10:07:49.651 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.92906845999999998 2024-09-21T10:07:49.651 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.651 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.651 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:07:49.651 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.346588+0000", 2024-09-21T10:07:49.651 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8569999999999997e-06 2024-09-21T10:07:49.651 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.651 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.651 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:07:49.652 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.471960+0000", 2024-09-21T10:07:49.652 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1253714340000001 2024-09-21T10:07:49.652 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.652 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.652 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:07:49.652 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.472090+0000", 2024-09-21T10:07:49.652 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013015499999999999 2024-09-21T10:07:49.652 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.652 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.652 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:07:49.652 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.472246+0000", 2024-09-21T10:07:49.652 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000156501 2024-09-21T10:07:49.652 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.652 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.652 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:07:49.653 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.477144+0000", 2024-09-21T10:07:49.653 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.004897892 2024-09-21T10:07:49.653 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.653 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.653 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.653 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.478614+0000", 2024-09-21T10:07:49.653 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0014694969999999999 2024-09-21T10:07:49.653 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.653 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.653 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:07:49.653 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.478639+0000", 2024-09-21T10:07:49.653 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.5003000000000001e-05 2024-09-21T10:07:49.653 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.653 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.653 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:07:49.654 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.478676+0000", 2024-09-21T10:07:49.654 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7156999999999997e-05 2024-09-21T10:07:49.654 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:07:49.654 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:07:49.654 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:07:49.654 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.478757+0000", 2024-09-21T10:07:49.654 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.0995000000000006e-05 2024-09-21T10:07:49.654 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.654 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.654 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.654 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:07:49.654 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:07:49.654 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:07:49.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:49 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:49.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:49 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:07:49.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:49 smithi129 ceph-mon[88471]: osdmap e217: 8 total, 7 up, 8 in 2024-09-21T10:07:49.655 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:49 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 37825/209640 objects degraded (18.043%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:49.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:49.656 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:49.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:49.756 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:49 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:49.757 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:49 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:07:49.757 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:49 smithi107 ceph-mon[102221]: osdmap e217: 8 total, 7 up, 8 in 2024-09-21T10:07:49.757 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:49 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 37825/209640 objects degraded (18.043%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:49.757 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:49 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:49.757 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:49 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:07:49.757 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:49 smithi107 ceph-mon[105022]: osdmap e217: 8 total, 7 up, 8 in 2024-09-21T10:07:49.757 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:49 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 37825/209640 objects degraded (18.043%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:49.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:49.767 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T10:07:49.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:49.789 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:49.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:49.798 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:07:49.798 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:07:49.827 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:07:49.827 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:07:49.858 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:07:49.858 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:07:49.883 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:07:49.884 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:07:49.890 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:49.906 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:07:49.907 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:07:49.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:49.927 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:49.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:49.939 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:07:49.939 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:07:49.963 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:07:49.963 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:07:49.963 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:07:49.963 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.485678 2024-09-21T10:07:49.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 5 0.48567776442088795 2024-09-21T10:07:49.970 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:07:49.970 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:07:50.028 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:50.064 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:50.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:50.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:50.165 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:50.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:50.209 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:50.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:50.310 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:50.331 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:50.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:50.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:50.433 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:50.476 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:50.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:50.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:50.523 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:07:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:07:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:07:50.578 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:50.603 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:50.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:50.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:50.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:50 smithi107 ceph-mon[102221]: pgmap v786: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 859 op/s; 34734/197583 objects degraded (17.579%); 11246/197583 objects misplaced (5.692%); 20 MiB/s, 323 objects/s recovering 2024-09-21T10:07:50.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:50 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/710067713' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:50.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:50 smithi107 ceph-mon[105022]: pgmap v786: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 859 op/s; 34734/197583 objects degraded (17.579%); 11246/197583 objects misplaced (5.692%); 20 MiB/s, 323 objects/s recovering 2024-09-21T10:07:50.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:50 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/710067713' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:07:50.704 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:07:50.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:50.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:50.767 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:50.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:50.835 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:50.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:50.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:50.895 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:50.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:50.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:50 smithi129 ceph-mon[88471]: pgmap v786: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 4.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 859 op/s; 34734/197583 objects degraded (17.579%); 11246/197583 objects misplaced (5.692%); 20 MiB/s, 323 objects/s recovering 2024-09-21T10:07:50.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:50 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/710067713' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:07:50.996 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:51.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:51.038 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:51.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:51.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:51.166 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:51.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:51.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:51.185 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:07:51.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:51.294 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:51.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:51.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:51.396 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:51.432 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:51.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:51.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:51.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:51.557 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:51.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:51.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:51.658 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:51.684 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:51.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:51.785 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:51.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:51.829 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:51.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:51.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:51.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:51.957 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:51.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:52.058 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:52.065 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:07:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:07:51] "GET /metrics HTTP/1.1" 200 36835 "" "Prometheus/2.51.0" 2024-09-21T10:07:52.096 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:52.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:52.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:52.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:52.219 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:52.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:52.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:52.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:52.346 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:52.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:52.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:52.447 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:52.471 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:52.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:52.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:52.573 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:52.610 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:52.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:52.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:52.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:07:52.712 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:52.733 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:52.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:52.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:52.835 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:52.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:52 smithi107 ceph-mon[102221]: pgmap v787: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 4.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 870 op/s; 34457/196752 objects degraded (17.513%); 11246/196752 objects misplaced (5.716%); 20 MiB/s, 323 objects/s recovering 2024-09-21T10:07:52.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:52 smithi107 ceph-mon[105022]: pgmap v787: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 4.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 870 op/s; 34457/196752 objects degraded (17.513%); 11246/196752 objects misplaced (5.716%); 20 MiB/s, 323 objects/s recovering 2024-09-21T10:07:52.871 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:52.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:52.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:52.873 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:52 smithi129 ceph-mon[88471]: pgmap v787: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 4.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 870 op/s; 34457/196752 objects degraded (17.513%); 11246/196752 objects misplaced (5.716%); 20 MiB/s, 323 objects/s recovering 2024-09-21T10:07:52.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:52.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:52.996 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:52.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:53.097 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:53.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:53.138 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:53.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:53.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:53.261 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:53.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:53.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:53.363 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:53.399 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:53.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:53.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:53.501 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:53.541 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:53.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:53.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:53.643 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:53.676 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:53.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:53.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:53.778 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:53.818 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:53.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:53.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:53.920 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:53.959 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:53.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:53.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:54.060 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:54.082 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:54.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:54.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:54.139 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:54.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:54.229 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:54.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:54.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:54.279 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:07:54.330 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:54.373 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:54.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:54.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:54.475 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:54.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:54.525 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:54.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:54.626 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:54.678 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:54.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:54.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:54.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:54 smithi107 ceph-mon[105022]: pgmap v788: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 888 op/s; 31864/187017 objects degraded (17.038%); 10133/187017 objects misplaced (5.418%); 19 MiB/s, 306 objects/s recovering 2024-09-21T10:07:54.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:54 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 31864/187017 objects degraded (17.038%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:54.700 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:54 smithi107 ceph-mon[102221]: pgmap v788: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 888 op/s; 31864/187017 objects degraded (17.038%); 10133/187017 objects misplaced (5.418%); 19 MiB/s, 306 objects/s recovering 2024-09-21T10:07:54.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:54 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 31864/187017 objects degraded (17.038%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:54.750 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:07:54.780 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:54.820 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:54.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:54.901 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:54 smithi129 ceph-mon[88471]: pgmap v788: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 888 op/s; 31864/187017 objects degraded (17.038%); 10133/187017 objects misplaced (5.418%); 19 MiB/s, 306 objects/s recovering 2024-09-21T10:07:54.901 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:54 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 31864/187017 objects degraded (17.038%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:54.922 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:54.964 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:54.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:54.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:55.066 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:55.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:55.124 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:55.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:55.225 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:55.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:55.267 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:55.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:55.366 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:07:55.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:55.426 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:55.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:55.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:55.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:55.566 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:55.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:55.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:55.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:55.698 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:55.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:55.799 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:55.825 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:55.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:55.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:55.927 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:55.955 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:55.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:55.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:56.057 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:56.082 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:56.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:56.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:56.183 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:56.205 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:56.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:56.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:56.307 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:56.330 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:56.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:56.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:56.432 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:56.454 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:56.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:56.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:56.545 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.5 primary-affinity to 0.485678 (8318292) 2024-09-21T10:07:56.556 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:56.584 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:56.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:56.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:56.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:56.710 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:56.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:56.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:56.798 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:56 smithi129 ceph-mon[88471]: pgmap v789: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 888 op/s; 31083/183747 objects degraded (16.916%); 10133/183747 objects misplaced (5.515%); 19 MiB/s, 306 objects/s recovering 2024-09-21T10:07:56.798 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:56 smithi129 ceph-mon[88471]: osdmap e218: 8 total, 7 up, 8 in 2024-09-21T10:07:56.799 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:56 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/788591597' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.48567776442088795} : dispatch 2024-09-21T10:07:56.799 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:56 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.48567776442088795} : dispatch 2024-09-21T10:07:56.799 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:56.802 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:07:56.802 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:07:56.812 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:07:56.836 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:56.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:56.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:07:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:56 smithi107 ceph-mon[102221]: pgmap v789: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 888 op/s; 31083/183747 objects degraded (16.916%); 10133/183747 objects misplaced (5.515%); 19 MiB/s, 306 objects/s recovering 2024-09-21T10:07:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:56 smithi107 ceph-mon[102221]: osdmap e218: 8 total, 7 up, 8 in 2024-09-21T10:07:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:56 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/788591597' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.48567776442088795} : dispatch 2024-09-21T10:07:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:56 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.48567776442088795} : dispatch 2024-09-21T10:07:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:56 smithi107 ceph-mon[105022]: pgmap v789: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 888 op/s; 31083/183747 objects degraded (16.916%); 10133/183747 objects misplaced (5.515%); 19 MiB/s, 306 objects/s recovering 2024-09-21T10:07:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:56 smithi107 ceph-mon[105022]: osdmap e218: 8 total, 7 up, 8 in 2024-09-21T10:07:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:56 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/788591597' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.48567776442088795} : dispatch 2024-09-21T10:07:56.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:56 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.48567776442088795} : dispatch 2024-09-21T10:07:56.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:56.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:56.958 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:56.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:56.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:57.060 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:57.083 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:57.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:57.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:57.185 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:57.226 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:57.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:57.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:57.328 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:57.350 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:57.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:57.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:57.452 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:57.474 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:57.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:57.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:57.540 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:07:57.576 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:57.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:57.607 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:57.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:57.707 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:57.740 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:57.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:57.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:57.841 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:57 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.48567776442088795}]': finished 2024-09-21T10:07:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:57 smithi107 ceph-mon[102221]: osdmap e219: 8 total, 7 up, 8 in 2024-09-21T10:07:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:57 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3084221289' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:07:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:57 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:57 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3084221289' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:07:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:57 smithi107 ceph-mon[102221]: osdmap e220: 8 total, 7 up, 8 in 2024-09-21T10:07:57.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:57 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.48567776442088795}]': finished 2024-09-21T10:07:57.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:57 smithi107 ceph-mon[105022]: osdmap e219: 8 total, 7 up, 8 in 2024-09-21T10:07:57.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:57 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3084221289' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:07:57.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:57 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:57.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:57 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3084221289' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:07:57.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:57 smithi107 ceph-mon[105022]: osdmap e220: 8 total, 7 up, 8 in 2024-09-21T10:07:57.877 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:57.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:57.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:57.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:57 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.48567776442088795}]': finished 2024-09-21T10:07:57.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:57 smithi129 ceph-mon[88471]: osdmap e219: 8 total, 7 up, 8 in 2024-09-21T10:07:57.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:57 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3084221289' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:07:57.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:57 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:07:57.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:57 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3084221289' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:07:57.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:57 smithi129 ceph-mon[88471]: osdmap e220: 8 total, 7 up, 8 in 2024-09-21T10:07:57.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:58.004 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:58.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:58.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:58.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:07:58.131 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:58.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:58.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:07:58.233 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:58.285 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:58.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:58.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:58.387 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:58.431 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:58.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:58.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:58.521 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:07:58.533 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:58.607 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:58.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:58.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:58.709 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:58.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:58.750 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:58.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:58.851 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:58 smithi107 ceph-mon[102221]: pgmap v792: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 peering, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 75 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 607 op/s; 30855/183063 objects degraded (16.855%); 10133/183063 objects misplaced (5.535%); 12 MiB/s, 191 objects/s recovering 2024-09-21T10:07:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:58 smithi107 ceph-mon[102221]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:07:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:58 smithi107 ceph-mon[105022]: pgmap v792: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 peering, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 75 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 607 op/s; 30855/183063 objects degraded (16.855%); 10133/183063 objects misplaced (5.535%); 12 MiB/s, 191 objects/s recovering 2024-09-21T10:07:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:58 smithi107 ceph-mon[105022]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:07:58.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:58.912 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:58.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:58 smithi129 ceph-mon[88471]: pgmap v792: 109 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfilling, 2 peering, 2 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 1 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 75 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 4.2 KiB/s rd, 9.2 KiB/s wr, 607 op/s; 30855/183063 objects degraded (16.855%); 10133/183063 objects misplaced (5.535%); 12 MiB/s, 191 objects/s recovering 2024-09-21T10:07:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:58 smithi129 ceph-mon[88471]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:07:58.966 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:07:59.013 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:07:59.069 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:59.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:59.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:07:59.170 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:07:59.216 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:59.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:59.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:07:59.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:07:59.345 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:59.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:59.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:07:59.447 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:59.489 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:59.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:59.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:59.591 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:59.633 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:59.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:59.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:59.735 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:07:59.761 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:07:59.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:59.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:07:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:07:59 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 30855/183063 objects degraded (16.855%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:07:59 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 30855/183063 objects degraded (16.855%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-21T10:07:59.863 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:07:59.903 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:07:59.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:07:59.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:07:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:07:59 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 30855/183063 objects degraded (16.855%), 12 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:00.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:00.029 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:00.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:00.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:00.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:00.160 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:00.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:00.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:00.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:00.286 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:00.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:00.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:00.389 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:00.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:00.434 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:00.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:00.535 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:00.548 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:08:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:08:00.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:08:00.566 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:00.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:00.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:00.649 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T10:08:00.650 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:00 smithi107 ceph-mon[102221]: pgmap v794: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 peering, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 3.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 34 KiB/s rd, 74 KiB/s wr, 665 op/s; 28350/175377 objects degraded (16.165%); 8551/175377 objects misplaced (4.876%); 19 MiB/s, 304 objects/s recovering 2024-09-21T10:08:00.650 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:00 smithi107 ceph-mon[105022]: pgmap v794: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 peering, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 3.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 34 KiB/s rd, 74 KiB/s wr, 665 op/s; 28350/175377 objects degraded (16.165%); 8551/175377 objects misplaced (4.876%); 19 MiB/s, 304 objects/s recovering 2024-09-21T10:08:00.668 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:00.676 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:08:00.677 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:08:00.701 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:08:00.701 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:08:00.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:00.712 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:00.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:00.725 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:08:00.725 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:08:00.757 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:08:00.758 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:08:00.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:08:00.786 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:08:00.786 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:08:00.813 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:00.816 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:08:00.816 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:08:00.850 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:00.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:00.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:00.853 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:08:00.853 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:08:00.886 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:00 smithi129 ceph-mon[88471]: pgmap v794: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 peering, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 3.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 34 KiB/s rd, 74 KiB/s wr, 665 op/s; 28350/175377 objects degraded (16.165%); 8551/175377 objects misplaced (4.876%); 19 MiB/s, 304 objects/s recovering 2024-09-21T10:08:00.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:00.977 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:00.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:00.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:01.047 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:08:01.047 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:08:01.047 INFO:teuthology.orchestra.run.smithi129.stdout: "complaint_time": 30, 2024-09-21T10:08:01.047 INFO:teuthology.orchestra.run.smithi129.stdout: "num_blocked_ops": 0 2024-09-21T10:08:01.047 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:08:01.079 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:01.126 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:01.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:01.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:01.228 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:01.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:01.277 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:01.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:01.378 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:01.416 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:01.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:01.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:01.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:01.546 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:01.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:01.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:01.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:01 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2905723688' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:01.677 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:08:01.678 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:08:01.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:01.682 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:01.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:01.710 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:01.741 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:08:01.783 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:01.833 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:01.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:01.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:01.840 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:01 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2905723688' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:01.840 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:01 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2905723688' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:01.840 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:08:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:08:01] "GET /metrics HTTP/1.1" 200 36833 "" "Prometheus/2.51.0" 2024-09-21T10:08:01.935 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:01.976 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:01.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:01.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:02.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:02.103 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:02.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:02.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:02.188 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:08:02.205 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:02.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:02.265 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:02.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:02.366 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:02.401 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:02.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:02.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:02.503 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:02.549 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:02.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:02.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:02.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:02.676 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:08:02.677 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:08:02.677 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-21T10:08:02.677 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:08:02.677 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:08:02.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-21T10:08:02.680 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:02.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:02.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:02.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:02.814 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:02.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:02.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:02 smithi107 ceph-mon[102221]: pgmap v795: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 peering, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 3.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 34 KiB/s rd, 74 KiB/s wr, 520 op/s; 28134/174729 objects degraded (16.102%); 8551/174729 objects misplaced (4.894%); 19 MiB/s, 304 objects/s recovering 2024-09-21T10:08:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:02 smithi107 ceph-mon[102221]: 1.8 scrub starts 2024-09-21T10:08:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:02 smithi107 ceph-mon[102221]: 1.8 scrub ok 2024-09-21T10:08:02.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:02 smithi107 ceph-mon[105022]: pgmap v795: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 peering, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 3.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 34 KiB/s rd, 74 KiB/s wr, 520 op/s; 28134/174729 objects degraded (16.102%); 8551/174729 objects misplaced (4.894%); 19 MiB/s, 304 objects/s recovering 2024-09-21T10:08:02.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:02 smithi107 ceph-mon[105022]: 1.8 scrub starts 2024-09-21T10:08:02.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:02 smithi107 ceph-mon[105022]: 1.8 scrub ok 2024-09-21T10:08:02.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:02.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:02 smithi129 ceph-mon[88471]: pgmap v795: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 peering, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 3.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 34 KiB/s rd, 74 KiB/s wr, 520 op/s; 28134/174729 objects degraded (16.102%); 8551/174729 objects misplaced (4.894%); 19 MiB/s, 304 objects/s recovering 2024-09-21T10:08:02.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:02 smithi129 ceph-mon[88471]: 1.8 scrub starts 2024-09-21T10:08:02.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:02 smithi129 ceph-mon[88471]: 1.8 scrub ok 2024-09-21T10:08:02.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:02.968 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:02.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:03.021 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:03.069 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:03.109 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:03.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:03.210 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:03.251 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:03.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:03.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:03.352 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:03.392 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:03.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:03.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:03.494 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:03.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:03.528 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:03.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:03.629 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:03.668 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:03.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:03.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:08:03.769 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:03.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:03.814 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:03.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:03.915 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:03.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:03 smithi129 ceph-mon[88471]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T10:08:03.936 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:03 smithi107 ceph-mon[102221]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T10:08:03.936 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:03 smithi107 ceph-mon[105022]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T10:08:03.957 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:03.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:03.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:04.059 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:04.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:04.096 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:04.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:04.196 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:04.232 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:04.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:04.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:04.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:04.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:04.358 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:04.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:04.460 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:04.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:04.501 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:04.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:04.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:04.627 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:04.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:04.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:04.642 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:08:04.728 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:04.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:04.768 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:04.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:04 smithi107 ceph-mon[102221]: pgmap v796: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 27 KiB/s rd, 58 KiB/s wr, 822 op/s; 26375/165195 objects degraded (15.966%); 7514/165195 objects misplaced (4.549%); 18 MiB/s, 292 objects/s recovering 2024-09-21T10:08:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:04 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1453371053' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:04 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 26375/165195 objects degraded (15.966%), 10 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:04 smithi107 ceph-mon[105022]: pgmap v796: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 27 KiB/s rd, 58 KiB/s wr, 822 op/s; 26375/165195 objects degraded (15.966%); 7514/165195 objects misplaced (4.549%); 18 MiB/s, 292 objects/s recovering 2024-09-21T10:08:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:04 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1453371053' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:04 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 26375/165195 objects degraded (15.966%), 10 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:04.869 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:04.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:04.913 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:04.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:04 smithi129 ceph-mon[88471]: pgmap v796: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 27 KiB/s rd, 58 KiB/s wr, 822 op/s; 26375/165195 objects degraded (15.966%); 7514/165195 objects misplaced (4.549%); 18 MiB/s, 292 objects/s recovering 2024-09-21T10:08:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:04 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1453371053' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:04 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 26375/165195 objects degraded (15.966%), 10 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:05.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:05.044 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:05.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:05.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:05.145 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:05.190 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:05.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:05.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:05.291 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:05.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:05.331 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:05.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:05.423 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:08:05.423 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 20, 2024-09-21T10:08:05.423 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 600, 2024-09-21T10:08:05.424 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:08:05.424 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.424 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32231 9.7 9:ecfb968d:::benchmark_data_smithi107_152010_object32230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:08:05.424 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.559414+0000", 2024-09-21T10:08:05.424 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 420.81886756599999, 2024-09-21T10:08:05.424 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23909994900000001, 2024-09-21T10:08:05.424 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:05.424 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:05.424 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:05.424 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:08:05.425 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:08:05.425 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32231 2024-09-21T10:08:05.425 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.425 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:05.425 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.425 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:05.425 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.559414+0000", 2024-09-21T10:08:05.425 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.426 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.426 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.426 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:05.426 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.559414+0000", 2024-09-21T10:08:05.426 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.426 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.426 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.426 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:05.426 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.559412+0000", 2024-09-21T10:08:05.426 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:08:05.426 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.426 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.426 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:05.427 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.560092+0000", 2024-09-21T10:08:05.427 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00067992900000000002 2024-09-21T10:08:05.427 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.427 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.427 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:05.427 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.560104+0000", 2024-09-21T10:08:05.427 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1887e-05 2024-09-21T10:08:05.428 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.428 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.428 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:05.428 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.560110+0000", 2024-09-21T10:08:05.428 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5740000000000003e-06 2024-09-21T10:08:05.428 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.428 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.428 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.429 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.560155+0000", 2024-09-21T10:08:05.429 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4913000000000002e-05 2024-09-21T10:08:05.429 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.429 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.429 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:05.429 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.560278+0000", 2024-09-21T10:08:05.429 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012316399999999999 2024-09-21T10:08:05.429 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.429 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.429 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:08:05.429 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.560401+0000", 2024-09-21T10:08:05.429 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012349299999999999 2024-09-21T10:08:05.429 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.430 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.430 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:05.430 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.562243+0000", 2024-09-21T10:08:05.430 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0018417520000000001 2024-09-21T10:08:05.430 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.430 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.430 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.430 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.563134+0000", 2024-09-21T10:08:05.430 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00089091400000000003 2024-09-21T10:08:05.430 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.430 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.431 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.431 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.798444+0000", 2024-09-21T10:08:05.431 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23531032700000001 2024-09-21T10:08:05.431 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.431 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.431 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:05.431 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.798486+0000", 2024-09-21T10:08:05.431 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1326999999999999e-05 2024-09-21T10:08:05.431 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.431 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.431 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:05.432 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.798514+0000", 2024-09-21T10:08:05.432 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.809e-05 2024-09-21T10:08:05.432 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.432 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.432 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.432 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.432 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.432 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32237 9.6 9:67022ea0:::benchmark_data_smithi107_152010_object32236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:08:05.432 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.561242+0000", 2024-09-21T10:08:05.432 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 420.81703976900002, 2024-09-21T10:08:05.432 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23891710999999999, 2024-09-21T10:08:05.432 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:05.432 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:05.433 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:05.433 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:08:05.433 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:08:05.433 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32237 2024-09-21T10:08:05.433 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.433 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:05.433 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.433 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:05.433 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561242+0000", 2024-09-21T10:08:05.433 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.433 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.433 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.433 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:05.433 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561242+0000", 2024-09-21T10:08:05.433 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.434 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.434 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.434 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:05.434 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561233+0000", 2024-09-21T10:08:05.434 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999914 2024-09-21T10:08:05.434 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.434 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.434 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:05.434 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561365+0000", 2024-09-21T10:08:05.434 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013210300000000001 2024-09-21T10:08:05.434 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.435 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.435 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:05.435 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561377+0000", 2024-09-21T10:08:05.435 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1426e-05 2024-09-21T10:08:05.435 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.435 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.435 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:05.435 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561382+0000", 2024-09-21T10:08:05.435 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4480000000000002e-06 2024-09-21T10:08:05.435 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.436 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.436 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.436 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561429+0000", 2024-09-21T10:08:05.436 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6761000000000002e-05 2024-09-21T10:08:05.436 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.436 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.436 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:05.436 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561536+0000", 2024-09-21T10:08:05.436 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000107021 2024-09-21T10:08:05.436 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.436 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.436 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:08:05.436 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561674+0000", 2024-09-21T10:08:05.437 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013777700000000001 2024-09-21T10:08:05.437 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.437 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.437 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:05.437 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.562828+0000", 2024-09-21T10:08:05.437 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0011543860000000001 2024-09-21T10:08:05.437 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.437 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.437 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.437 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564200+0000", 2024-09-21T10:08:05.437 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001372465 2024-09-21T10:08:05.437 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.437 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.438 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.438 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800061+0000", 2024-09-21T10:08:05.438 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.2358606 2024-09-21T10:08:05.438 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.438 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.438 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:05.438 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800127+0000", 2024-09-21T10:08:05.438 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6154000000000004e-05 2024-09-21T10:08:05.438 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.438 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.438 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:05.439 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800159+0000", 2024-09-21T10:08:05.439 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1534e-05 2024-09-21T10:08:05.439 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.439 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.439 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.439 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.439 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.439 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32243 9.6 9:68ed822c:::benchmark_data_smithi107_152010_object32242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:08:05.439 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.564332+0000", 2024-09-21T10:08:05.439 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 420.813949686, 2024-09-21T10:08:05.439 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23596514499999999, 2024-09-21T10:08:05.439 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:05.439 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:05.439 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:05.440 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:08:05.440 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:08:05.440 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32243 2024-09-21T10:08:05.440 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.440 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:05.440 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.440 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:05.440 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564332+0000", 2024-09-21T10:08:05.440 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.440 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.440 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.441 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:05.441 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564332+0000", 2024-09-21T10:08:05.441 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.441 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.441 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.441 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:05.441 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564331+0000", 2024-09-21T10:08:05.441 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:08:05.441 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.441 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.441 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:05.441 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564391+0000", 2024-09-21T10:08:05.442 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0105999999999997e-05 2024-09-21T10:08:05.442 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.442 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.442 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:05.442 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564401+0000", 2024-09-21T10:08:05.442 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.8099999999999992e-06 2024-09-21T10:08:05.442 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.442 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.442 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:05.442 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564413+0000", 2024-09-21T10:08:05.442 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2163e-05 2024-09-21T10:08:05.442 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.442 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.443 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.443 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564449+0000", 2024-09-21T10:08:05.443 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5775999999999998e-05 2024-09-21T10:08:05.443 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.443 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.443 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:05.443 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564557+0000", 2024-09-21T10:08:05.443 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010838199999999999 2024-09-21T10:08:05.443 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.444 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.444 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:08:05.444 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564694+0000", 2024-09-21T10:08:05.444 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013716299999999999 2024-09-21T10:08:05.444 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.444 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.445 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:05.445 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.565898+0000", 2024-09-21T10:08:05.445 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0012039570000000001 2024-09-21T10:08:05.445 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.445 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.445 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.445 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.567502+0000", 2024-09-21T10:08:05.445 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001604149 2024-09-21T10:08:05.445 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.445 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.445 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.445 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800209+0000", 2024-09-21T10:08:05.445 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.232706624 2024-09-21T10:08:05.445 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.445 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.446 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:05.446 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800255+0000", 2024-09-21T10:08:05.446 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6452000000000002e-05 2024-09-21T10:08:05.446 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.446 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.446 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:05.446 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800297+0000", 2024-09-21T10:08:05.446 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1508999999999998e-05 2024-09-21T10:08:05.446 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.446 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.446 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.446 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.446 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.446 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32262 9.6 9:64cc6811:::benchmark_data_smithi107_152010_object32261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:08:05.446 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.575084+0000", 2024-09-21T10:08:05.447 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 420.80319752299999, 2024-09-21T10:08:05.447 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.22529529000000001, 2024-09-21T10:08:05.447 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:05.447 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:05.447 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:05.447 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:08:05.447 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:08:05.447 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32262 2024-09-21T10:08:05.447 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.447 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:05.447 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.447 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:05.447 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575084+0000", 2024-09-21T10:08:05.448 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.448 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.448 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.448 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:05.448 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575084+0000", 2024-09-21T10:08:05.448 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.448 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.448 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.448 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:05.448 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575083+0000", 2024-09-21T10:08:05.448 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:08:05.448 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.448 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.448 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:05.448 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575130+0000", 2024-09-21T10:08:05.448 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7709e-05 2024-09-21T10:08:05.449 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.449 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.449 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:05.449 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575141+0000", 2024-09-21T10:08:05.449 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0971e-05 2024-09-21T10:08:05.449 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.449 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.449 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:05.449 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575146+0000", 2024-09-21T10:08:05.449 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.262e-06 2024-09-21T10:08:05.449 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.449 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.449 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.450 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575189+0000", 2024-09-21T10:08:05.450 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3216000000000001e-05 2024-09-21T10:08:05.450 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.450 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.450 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:05.450 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575313+0000", 2024-09-21T10:08:05.450 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012379099999999999 2024-09-21T10:08:05.450 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.450 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.450 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:08:05.450 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575477+0000", 2024-09-21T10:08:05.450 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00016417499999999999 2024-09-21T10:08:05.450 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.451 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.451 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:05.451 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.577444+0000", 2024-09-21T10:08:05.451 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0019669790000000002 2024-09-21T10:08:05.451 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.451 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.451 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.451 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.577574+0000", 2024-09-21T10:08:05.451 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012998900000000001 2024-09-21T10:08:05.451 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.451 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.451 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.451 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800326+0000", 2024-09-21T10:08:05.451 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.22275213099999999 2024-09-21T10:08:05.451 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.452 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.452 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:05.452 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800353+0000", 2024-09-21T10:08:05.452 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7056e-05 2024-09-21T10:08:05.452 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.452 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.452 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:05.452 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800379+0000", 2024-09-21T10:08:05.452 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.6203e-05 2024-09-21T10:08:05.452 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.452 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.452 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.452 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.452 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.453 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32272 9.6 9:68a66308:::benchmark_data_smithi107_152010_object32271:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:08:05.453 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.579482+0000", 2024-09-21T10:08:05.453 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 420.79879929200001, 2024-09-21T10:08:05.453 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.22308472700000001, 2024-09-21T10:08:05.453 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:05.453 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:05.453 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:05.453 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:08:05.453 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:08:05.453 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32272 2024-09-21T10:08:05.453 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.453 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:05.453 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.454 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:05.454 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579482+0000", 2024-09-21T10:08:05.454 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.454 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.454 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.454 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:05.454 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579482+0000", 2024-09-21T10:08:05.454 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.454 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.454 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.454 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:05.454 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579481+0000", 2024-09-21T10:08:05.454 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:08:05.454 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.455 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.455 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:05.455 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579545+0000", 2024-09-21T10:08:05.455 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4506999999999998e-05 2024-09-21T10:08:05.455 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.455 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.455 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:05.455 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579557+0000", 2024-09-21T10:08:05.455 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1610000000000001e-05 2024-09-21T10:08:05.455 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.455 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.455 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:05.455 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579562+0000", 2024-09-21T10:08:05.456 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4970000000000001e-06 2024-09-21T10:08:05.456 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.456 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.456 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.456 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579606+0000", 2024-09-21T10:08:05.456 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3180000000000003e-05 2024-09-21T10:08:05.456 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.456 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.456 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:05.456 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579720+0000", 2024-09-21T10:08:05.456 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000114689 2024-09-21T10:08:05.456 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.456 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.456 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:08:05.457 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579854+0000", 2024-09-21T10:08:05.457 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000134144 2024-09-21T10:08:05.457 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.457 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.457 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:05.457 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.580922+0000", 2024-09-21T10:08:05.457 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001067744 2024-09-21T10:08:05.457 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.457 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.457 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.457 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581827+0000", 2024-09-21T10:08:05.457 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00090439700000000004 2024-09-21T10:08:05.457 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.457 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.458 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.458 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.802500+0000", 2024-09-21T10:08:05.458 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.22067335699999999 2024-09-21T10:08:05.458 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.458 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.458 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:05.458 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.802531+0000", 2024-09-21T10:08:05.458 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1469e-05 2024-09-21T10:08:05.458 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.458 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.458 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:05.458 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.802567+0000", 2024-09-21T10:08:05.458 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5410000000000001e-05 2024-09-21T10:08:05.459 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.459 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.459 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.459 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.459 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.459 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32275 9.7 9:ef23b11d:::benchmark_data_smithi107_152010_object32274:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:08:05.459 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.581115+0000", 2024-09-21T10:08:05.459 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 420.79716592099999, 2024-09-21T10:08:05.459 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21812805800000001, 2024-09-21T10:08:05.459 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:05.459 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:05.460 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:05.460 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:08:05.460 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:08:05.460 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32275 2024-09-21T10:08:05.460 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.460 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:05.460 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.460 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:05.460 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581115+0000", 2024-09-21T10:08:05.460 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.460 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.460 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.460 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:05.460 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581115+0000", 2024-09-21T10:08:05.461 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.461 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.461 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.461 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:05.461 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581115+0000", 2024-09-21T10:08:05.461 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:08:05.461 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.461 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.461 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:05.461 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581168+0000", 2024-09-21T10:08:05.461 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3013000000000002e-05 2024-09-21T10:08:05.461 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.462 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.462 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:05.462 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581179+0000", 2024-09-21T10:08:05.462 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1970000000000001e-05 2024-09-21T10:08:05.462 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.462 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.462 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:05.462 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581184+0000", 2024-09-21T10:08:05.462 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7949999999999998e-06 2024-09-21T10:08:05.462 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.463 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.463 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.463 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581209+0000", 2024-09-21T10:08:05.463 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.4389999999999999e-05 2024-09-21T10:08:05.463 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.463 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.463 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:05.463 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581282+0000", 2024-09-21T10:08:05.463 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3590000000000005e-05 2024-09-21T10:08:05.463 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.463 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.463 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:08:05.463 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581399+0000", 2024-09-21T10:08:05.463 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000117137 2024-09-21T10:08:05.464 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.464 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.464 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:05.464 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.582724+0000", 2024-09-21T10:08:05.464 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0013241220000000001 2024-09-21T10:08:05.464 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.464 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.464 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.464 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.582977+0000", 2024-09-21T10:08:05.464 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00025303400000000002 2024-09-21T10:08:05.464 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.464 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.464 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.464 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.799174+0000", 2024-09-21T10:08:05.464 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21619723299999999 2024-09-21T10:08:05.465 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.465 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.465 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:05.465 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.799206+0000", 2024-09-21T10:08:05.465 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2026999999999997e-05 2024-09-21T10:08:05.465 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.465 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.465 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:05.465 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.799244+0000", 2024-09-21T10:08:05.465 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7721e-05 2024-09-21T10:08:05.465 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.465 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.466 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.466 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.466 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.466 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32283 9.7 9:ee5c92df:::benchmark_data_smithi107_152010_object32282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:08:05.466 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.589070+0000", 2024-09-21T10:08:05.466 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 420.789211841, 2024-09-21T10:08:05.466 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21034788500000001, 2024-09-21T10:08:05.466 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:05.466 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:05.466 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:05.466 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:08:05.467 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:08:05.467 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32283 2024-09-21T10:08:05.467 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.467 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:05.467 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.467 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:05.467 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589070+0000", 2024-09-21T10:08:05.467 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.467 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.467 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.467 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:05.467 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589070+0000", 2024-09-21T10:08:05.467 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.468 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.468 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.468 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:05.468 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589068+0000", 2024-09-21T10:08:05.468 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:08:05.468 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.468 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.468 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:05.468 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589113+0000", 2024-09-21T10:08:05.468 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4545000000000001e-05 2024-09-21T10:08:05.468 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.468 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.468 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:05.469 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589126+0000", 2024-09-21T10:08:05.469 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2738999999999999e-05 2024-09-21T10:08:05.469 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.469 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.469 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:05.469 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589130+0000", 2024-09-21T10:08:05.469 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3510000000000004e-06 2024-09-21T10:08:05.469 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.469 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.469 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.469 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589162+0000", 2024-09-21T10:08:05.469 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1612999999999997e-05 2024-09-21T10:08:05.469 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.470 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.470 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:05.470 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589244+0000", 2024-09-21T10:08:05.470 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.2725000000000005e-05 2024-09-21T10:08:05.470 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.470 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.470 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:08:05.470 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589327+0000", 2024-09-21T10:08:05.470 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.3182000000000004e-05 2024-09-21T10:08:05.470 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.470 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.470 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:05.470 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.590365+0000", 2024-09-21T10:08:05.471 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0010372059999999999 2024-09-21T10:08:05.471 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.471 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.471 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.471 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.590700+0000", 2024-09-21T10:08:05.471 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00033515699999999999 2024-09-21T10:08:05.471 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.471 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.471 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.471 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.799339+0000", 2024-09-21T10:08:05.471 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.20863885800000001 2024-09-21T10:08:05.471 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.471 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.472 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:05.472 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.799377+0000", 2024-09-21T10:08:05.472 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8281000000000002e-05 2024-09-21T10:08:05.472 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.472 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.472 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:05.472 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.799417+0000", 2024-09-21T10:08:05.472 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0466999999999997e-05 2024-09-21T10:08:05.472 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.472 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.472 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.472 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.472 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.472 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:101219 9.f 9:fbbb13dd:::benchmark_data_smithi107_152010_object101218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-21T10:08:05.473 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:09.472644+0000", 2024-09-21T10:08:05.473 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 354.90563711200002, 2024-09-21T10:08:05.473 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.27593879999999998, 2024-09-21T10:08:05.473 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:05.473 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:05.473 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:05.473 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:08:05.473 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:08:05.473 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 101219 2024-09-21T10:08:05.473 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.473 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:05.473 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.473 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:05.474 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472644+0000", 2024-09-21T10:08:05.474 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.474 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.474 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.474 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:05.474 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472644+0000", 2024-09-21T10:08:05.474 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.474 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.474 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.474 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:05.474 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472643+0000", 2024-09-21T10:08:05.474 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:08:05.474 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.474 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.475 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:05.475 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472693+0000", 2024-09-21T10:08:05.475 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9314999999999998e-05 2024-09-21T10:08:05.475 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.475 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.475 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:05.475 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472707+0000", 2024-09-21T10:08:05.475 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4048e-05 2024-09-21T10:08:05.475 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.475 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.475 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:05.475 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472712+0000", 2024-09-21T10:08:05.475 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.524e-06 2024-09-21T10:08:05.475 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.475 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.475 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.476 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472747+0000", 2024-09-21T10:08:05.476 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5207999999999999e-05 2024-09-21T10:08:05.476 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.476 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.476 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:05.476 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472830+0000", 2024-09-21T10:08:05.476 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.3107000000000002e-05 2024-09-21T10:08:05.476 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.476 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.476 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:08:05.476 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472921+0000", 2024-09-21T10:08:05.476 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.0845000000000002e-05 2024-09-21T10:08:05.476 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.476 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.476 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:05.476 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.475574+0000", 2024-09-21T10:08:05.477 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002653121 2024-09-21T10:08:05.477 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.477 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.477 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.477 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.484356+0000", 2024-09-21T10:08:05.477 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0087821459999999994 2024-09-21T10:08:05.477 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.477 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.477 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.477 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.748351+0000", 2024-09-21T10:08:05.477 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.26399478999999998 2024-09-21T10:08:05.477 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.477 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.477 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:05.477 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.748541+0000", 2024-09-21T10:08:05.478 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00018932500000000001 2024-09-21T10:08:05.478 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.478 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.478 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:05.478 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.748583+0000", 2024-09-21T10:08:05.478 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2478999999999998e-05 2024-09-21T10:08:05.478 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.478 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.478 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.478 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.478 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.478 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:101221 9.6 9:6e7b28d1:::benchmark_data_smithi107_152010_object101220:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-21T10:08:05.478 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:09.473703+0000", 2024-09-21T10:08:05.478 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 354.90457819599999, 2024-09-21T10:08:05.478 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.27788642400000002, 2024-09-21T10:08:05.478 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:05.479 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:05.479 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:05.479 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:08:05.479 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:08:05.479 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 101221 2024-09-21T10:08:05.479 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.479 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:05.479 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.479 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:05.479 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473703+0000", 2024-09-21T10:08:05.479 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.479 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.479 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.479 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:05.479 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473703+0000", 2024-09-21T10:08:05.479 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.480 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.480 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.480 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:05.480 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473703+0000", 2024-09-21T10:08:05.480 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:05.480 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.480 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.480 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:05.480 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473759+0000", 2024-09-21T10:08:05.480 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5989999999999998e-05 2024-09-21T10:08:05.480 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.480 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.480 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:05.480 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473768+0000", 2024-09-21T10:08:05.481 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.0119999999999994e-06 2024-09-21T10:08:05.481 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.481 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.481 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:05.481 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473771+0000", 2024-09-21T10:08:05.481 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6440000000000002e-06 2024-09-21T10:08:05.481 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.481 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.481 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.481 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473801+0000", 2024-09-21T10:08:05.481 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.9403e-05 2024-09-21T10:08:05.481 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.481 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.481 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:05.481 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473906+0000", 2024-09-21T10:08:05.482 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000105516 2024-09-21T10:08:05.482 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.482 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.482 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:08:05.482 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473986+0000", 2024-09-21T10:08:05.482 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9802000000000006e-05 2024-09-21T10:08:05.482 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.482 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.482 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:05.482 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.480993+0000", 2024-09-21T10:08:05.482 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.007007231 2024-09-21T10:08:05.482 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.482 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.482 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.482 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.481030+0000", 2024-09-21T10:08:05.483 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6848999999999999e-05 2024-09-21T10:08:05.483 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.483 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.483 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.483 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.751491+0000", 2024-09-21T10:08:05.483 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.27046129299999999 2024-09-21T10:08:05.483 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.483 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.483 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:05.483 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.751548+0000", 2024-09-21T10:08:05.483 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6422999999999998e-05 2024-09-21T10:08:05.483 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.483 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.483 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:05.484 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.751590+0000", 2024-09-21T10:08:05.484 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1850000000000001e-05 2024-09-21T10:08:05.484 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.484 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.484 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.484 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.484 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.484 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:101223 9.6 9:66631bd2:::benchmark_data_smithi107_152010_object101222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-21T10:08:05.484 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:09.473791+0000", 2024-09-21T10:08:05.484 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 354.904490426, 2024-09-21T10:08:05.484 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.283198595, 2024-09-21T10:08:05.484 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:05.484 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:05.484 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:05.484 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:08:05.484 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:08:05.485 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 101223 2024-09-21T10:08:05.485 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.485 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:05.485 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.485 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:05.485 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473791+0000", 2024-09-21T10:08:05.485 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.485 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.485 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.485 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:05.485 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473791+0000", 2024-09-21T10:08:05.485 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.485 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.485 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.486 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:05.486 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473790+0000", 2024-09-21T10:08:05.486 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:05.486 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.486 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.486 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:05.486 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473870+0000", 2024-09-21T10:08:05.486 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9717999999999998e-05 2024-09-21T10:08:05.486 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.486 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.486 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:05.486 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473878+0000", 2024-09-21T10:08:05.486 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9589999999999995e-06 2024-09-21T10:08:05.486 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.487 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.487 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:05.487 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473883+0000", 2024-09-21T10:08:05.487 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8960000000000002e-06 2024-09-21T10:08:05.487 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.487 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.487 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.487 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.476473+0000", 2024-09-21T10:08:05.487 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0025897509999999999 2024-09-21T10:08:05.487 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.487 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.487 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:05.487 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.476556+0000", 2024-09-21T10:08:05.487 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.3242e-05 2024-09-21T10:08:05.487 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.487 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.488 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:08:05.488 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.476636+0000", 2024-09-21T10:08:05.488 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.0174999999999997e-05 2024-09-21T10:08:05.488 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.488 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.488 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:05.488 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.480996+0000", 2024-09-21T10:08:05.488 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0043602459999999999 2024-09-21T10:08:05.488 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.488 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.488 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.488 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.488327+0000", 2024-09-21T10:08:05.488 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0073310729999999996 2024-09-21T10:08:05.489 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.489 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.489 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.489 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.756861+0000", 2024-09-21T10:08:05.489 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.26853359700000001 2024-09-21T10:08:05.489 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.489 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.489 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:05.489 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.756947+0000", 2024-09-21T10:08:05.489 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6316999999999994e-05 2024-09-21T10:08:05.489 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.489 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.490 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:05.490 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.756990+0000", 2024-09-21T10:08:05.490 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2259000000000003e-05 2024-09-21T10:08:05.490 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.490 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.490 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.490 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.490 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.490 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:101225 9.6 9:6bf868dc:::benchmark_data_smithi107_152010_object101224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-21T10:08:05.490 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:09.474216+0000", 2024-09-21T10:08:05.490 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 354.90406576800001, 2024-09-21T10:08:05.490 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28288837500000003, 2024-09-21T10:08:05.491 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:05.491 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:05.491 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:05.491 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:08:05.491 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:08:05.491 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 101225 2024-09-21T10:08:05.491 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.491 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:05.491 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.491 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:05.491 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474216+0000", 2024-09-21T10:08:05.491 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.491 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.491 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.492 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:05.492 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474216+0000", 2024-09-21T10:08:05.492 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.492 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.492 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.492 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:05.492 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474215+0000", 2024-09-21T10:08:05.492 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:08:05.492 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.492 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.492 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:05.492 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474301+0000", 2024-09-21T10:08:05.492 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6462e-05 2024-09-21T10:08:05.492 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.492 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.493 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:05.493 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474313+0000", 2024-09-21T10:08:05.493 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1685000000000001e-05 2024-09-21T10:08:05.493 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.493 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.493 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:05.493 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474318+0000", 2024-09-21T10:08:05.493 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2190000000000003e-06 2024-09-21T10:08:05.493 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.493 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.493 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.493 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.478980+0000", 2024-09-21T10:08:05.493 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0046614480000000003 2024-09-21T10:08:05.493 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.493 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.494 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:05.494 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.479090+0000", 2024-09-21T10:08:05.494 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000110468 2024-09-21T10:08:05.494 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.494 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.494 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:08:05.494 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.479168+0000", 2024-09-21T10:08:05.494 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.7837999999999996e-05 2024-09-21T10:08:05.494 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.494 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.494 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:05.494 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.481151+0000", 2024-09-21T10:08:05.494 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0019829729999999999 2024-09-21T10:08:05.495 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.495 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.495 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.495 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.488352+0000", 2024-09-21T10:08:05.495 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0072014410000000003 2024-09-21T10:08:05.495 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.495 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.495 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.495 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.757027+0000", 2024-09-21T10:08:05.495 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.26867438300000002 2024-09-21T10:08:05.495 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.495 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.495 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:05.495 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.757069+0000", 2024-09-21T10:08:05.496 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1835000000000002e-05 2024-09-21T10:08:05.496 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.496 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.496 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:05.496 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.757104+0000", 2024-09-21T10:08:05.496 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5410000000000001e-05 2024-09-21T10:08:05.496 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.496 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.496 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.496 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.496 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.496 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:126108 9.6 9:6f7b6afe:::benchmark_data_smithi107_152010_object16171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-21T10:08:05.497 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:38.616365+0000", 2024-09-21T10:08:05.497 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 325.76191620600002, 2024-09-21T10:08:05.497 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.213592333, 2024-09-21T10:08:05.497 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:05.497 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:08:05.497 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:05.497 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:08:05.497 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:08:05.497 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 126108 2024-09-21T10:08:05.497 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.497 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:05.497 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.497 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:05.497 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616365+0000", 2024-09-21T10:08:05.498 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.498 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.498 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.498 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:05.498 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616365+0000", 2024-09-21T10:08:05.498 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.498 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.498 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.498 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:05.498 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616364+0000", 2024-09-21T10:08:05.498 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:08:05.499 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.499 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.499 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:05.499 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616371+0000", 2024-09-21T10:08:05.499 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9840000000000004e-06 2024-09-21T10:08:05.499 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.499 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.499 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:05.499 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616390+0000", 2024-09-21T10:08:05.499 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.9476000000000001e-05 2024-09-21T10:08:05.499 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.499 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.499 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:05.499 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616397+0000", 2024-09-21T10:08:05.500 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4130000000000003e-06 2024-09-21T10:08:05.500 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.500 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.500 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.500 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616446+0000", 2024-09-21T10:08:05.500 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8998000000000001e-05 2024-09-21T10:08:05.500 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.500 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.500 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:05.500 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616569+0000", 2024-09-21T10:08:05.500 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000123318 2024-09-21T10:08:05.500 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.501 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.501 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:08:05.501 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616671+0000", 2024-09-21T10:08:05.501 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010222599999999999 2024-09-21T10:08:05.501 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.501 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.501 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:05.501 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.620447+0000", 2024-09-21T10:08:05.501 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0037759579999999998 2024-09-21T10:08:05.501 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.501 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.501 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.502 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.829942+0000", 2024-09-21T10:08:05.502 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.20949464800000001 2024-09-21T10:08:05.502 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.502 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.502 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:05.502 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.829958+0000", 2024-09-21T10:08:05.502 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5863999999999999e-05 2024-09-21T10:08:05.502 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.502 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.502 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.502 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.502 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.502 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop_reply(client.45761.0:126108 9.6 e162/141)", 2024-09-21T10:08:05.502 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:38.629127+0000", 2024-09-21T10:08:05.503 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 325.74915435000003, 2024-09-21T10:08:05.503 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.20087063799999999, 2024-09-21T10:08:05.503 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:05.503 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:08:05.503 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:05.503 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.503 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:05.503 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.629127+0000", 2024-09-21T10:08:05.503 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.503 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.503 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.503 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:05.503 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.629127+0000", 2024-09-21T10:08:05.503 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.503 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.504 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.504 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:05.504 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.629126+0000", 2024-09-21T10:08:05.504 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:08:05.504 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.504 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.504 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:05.504 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.629132+0000", 2024-09-21T10:08:05.504 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.781e-06 2024-09-21T10:08:05.504 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.504 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.504 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:05.504 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.629148+0000", 2024-09-21T10:08:05.504 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5446000000000001e-05 2024-09-21T10:08:05.504 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.505 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.505 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:05.505 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.629154+0000", 2024-09-21T10:08:05.505 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7989999999999999e-06 2024-09-21T10:08:05.505 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.505 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.505 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.505 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.829983+0000", 2024-09-21T10:08:05.505 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.20082974300000001 2024-09-21T10:08:05.505 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.505 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.505 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:05.505 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.829998+0000", 2024-09-21T10:08:05.505 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4307999999999999e-05 2024-09-21T10:08:05.506 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.506 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.506 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.506 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.506 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.506 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:126108 9.16 9:6f7b6afe:::benchmark_data_smithi107_152010_object16171:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163)", 2024-09-21T10:08:05.506 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:38.641118+0000", 2024-09-21T10:08:05.506 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 325.73716353499998, 2024-09-21T10:08:05.506 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.03992945, 2024-09-21T10:08:05.506 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:05.506 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:08:05.506 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:05.506 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:08:05.506 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:08:05.506 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 126108 2024-09-21T10:08:05.507 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.507 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:05.507 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.507 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:05.507 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.641118+0000", 2024-09-21T10:08:05.507 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.507 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.507 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.507 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:05.507 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.641118+0000", 2024-09-21T10:08:05.507 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.507 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.507 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.507 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:05.507 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.641117+0000", 2024-09-21T10:08:05.508 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:08:05.508 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.508 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.508 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:05.508 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.641124+0000", 2024-09-21T10:08:05.508 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5200000000000003e-06 2024-09-21T10:08:05.508 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.508 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.508 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:05.508 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.641133+0000", 2024-09-21T10:08:05.508 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.7790000000000007e-06 2024-09-21T10:08:05.508 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.508 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.508 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:05.509 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.641139+0000", 2024-09-21T10:08:05.509 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8510000000000004e-06 2024-09-21T10:08:05.509 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.509 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.509 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.509 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.830197+0000", 2024-09-21T10:08:05.509 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.18905825400000001 2024-09-21T10:08:05.509 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.509 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.509 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:08:05.509 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.830204+0000", 2024-09-21T10:08:05.509 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9940000000000003e-06 2024-09-21T10:08:05.509 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.509 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.509 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.510 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:39.680990+0000", 2024-09-21T10:08:05.510 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.85078537099999996 2024-09-21T10:08:05.510 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.510 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.510 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:05.510 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:39.681047+0000", 2024-09-21T10:08:05.510 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7512000000000001e-05 2024-09-21T10:08:05.510 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.510 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.510 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.510 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.511 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.511 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:126119 9.a 9:5778ec59:::benchmark_data_smithi107_152010_object16182:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-21T10:08:05.511 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:38.652541+0000", 2024-09-21T10:08:05.511 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 325.72574046099999, 2024-09-21T10:08:05.511 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.413684987, 2024-09-21T10:08:05.511 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:05.511 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:05.511 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:05.511 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:08:05.511 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:08:05.511 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 126119 2024-09-21T10:08:05.511 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.511 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:05.511 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.512 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:05.512 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652541+0000", 2024-09-21T10:08:05.512 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.512 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.512 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.512 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:05.512 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652541+0000", 2024-09-21T10:08:05.512 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.512 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.512 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.512 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:05.512 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652539+0000", 2024-09-21T10:08:05.512 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999976 2024-09-21T10:08:05.513 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.513 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.513 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:05.513 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652548+0000", 2024-09-21T10:08:05.513 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.5810000000000003e-06 2024-09-21T10:08:05.513 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.513 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.513 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:05.513 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652568+0000", 2024-09-21T10:08:05.513 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.9656999999999998e-05 2024-09-21T10:08:05.513 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.513 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.513 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:05.514 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652575+0000", 2024-09-21T10:08:05.514 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2960000000000001e-06 2024-09-21T10:08:05.514 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.514 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.514 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.514 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652636+0000", 2024-09-21T10:08:05.514 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0293e-05 2024-09-21T10:08:05.514 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.514 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.514 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:05.514 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652783+0000", 2024-09-21T10:08:05.514 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014715 2024-09-21T10:08:05.514 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.514 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.515 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:08:05.515 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652898+0000", 2024-09-21T10:08:05.515 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011513700000000001 2024-09-21T10:08:05.515 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.515 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.515 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:05.515 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.657580+0000", 2024-09-21T10:08:05.515 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0046820580000000002 2024-09-21T10:08:05.515 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.515 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.515 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.515 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.895867+0000", 2024-09-21T10:08:05.515 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23828700799999999 2024-09-21T10:08:05.515 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.516 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.516 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.516 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:39.066119+0000", 2024-09-21T10:08:05.516 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.170251927 2024-09-21T10:08:05.516 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.516 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.516 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:05.516 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:39.066160+0000", 2024-09-21T10:08:05.516 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.074e-05 2024-09-21T10:08:05.516 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.516 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.516 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:05.516 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:39.066226+0000", 2024-09-21T10:08:05.516 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6333000000000003e-05 2024-09-21T10:08:05.516 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.517 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.517 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.517 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.517 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.517 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95099 10.6 10:64ab5024:::benchmark_data_smithi107_184135_object95098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T10:08:05.517 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.388487+0000", 2024-09-21T10:08:05.517 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 117.98979480600001, 2024-09-21T10:08:05.517 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.019649673, 2024-09-21T10:08:05.517 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:05.517 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:08:05.517 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:05.517 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:05.517 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:05.517 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95099 2024-09-21T10:08:05.517 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.518 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:05.518 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.518 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:05.518 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388487+0000", 2024-09-21T10:08:05.518 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.518 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.518 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.518 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:05.518 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388487+0000", 2024-09-21T10:08:05.518 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.518 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.518 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.518 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:05.519 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388485+0000", 2024-09-21T10:08:05.519 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:08:05.519 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.519 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.519 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:05.519 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388545+0000", 2024-09-21T10:08:05.519 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0622999999999999e-05 2024-09-21T10:08:05.519 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.519 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.519 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:05.519 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388561+0000", 2024-09-21T10:08:05.519 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5667000000000001e-05 2024-09-21T10:08:05.519 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.519 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.520 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:05.520 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388567+0000", 2024-09-21T10:08:05.520 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8640000000000001e-06 2024-09-21T10:08:05.520 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.520 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.520 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.520 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388616+0000", 2024-09-21T10:08:05.520 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9168000000000001e-05 2024-09-21T10:08:05.520 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.520 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.520 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:05.520 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388773+0000", 2024-09-21T10:08:05.520 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000157465 2024-09-21T10:08:05.520 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.520 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.521 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:08:05.521 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388903+0000", 2024-09-21T10:08:05.521 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012996399999999999 2024-09-21T10:08:05.521 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.521 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.521 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:05.521 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.400728+0000", 2024-09-21T10:08:05.521 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.01182456 2024-09-21T10:08:05.521 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.521 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.521 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.521 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407770+0000", 2024-09-21T10:08:05.521 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0070416530000000001 2024-09-21T10:08:05.522 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.522 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.522 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:08:05.522 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407778+0000", 2024-09-21T10:08:05.522 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9549999999999992e-06 2024-09-21T10:08:05.522 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.522 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.522 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.522 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.408044+0000", 2024-09-21T10:08:05.522 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.000266069 2024-09-21T10:08:05.522 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.522 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.522 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:05.523 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.408136+0000", 2024-09-21T10:08:05.523 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.2651000000000005e-05 2024-09-21T10:08:05.523 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.523 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.523 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.523 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.523 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.523 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95101 10.6 10:6a273935:::benchmark_data_smithi107_184135_object95100:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T10:08:05.523 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.389198+0000", 2024-09-21T10:08:05.523 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 117.989083214, 2024-09-21T10:08:05.523 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0191866650000001, 2024-09-21T10:08:05.523 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:05.523 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:08:05.524 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:05.524 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:05.524 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:05.524 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95101 2024-09-21T10:08:05.524 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.524 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:05.524 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.524 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:05.524 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.389198+0000", 2024-09-21T10:08:05.524 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.524 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.524 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.524 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:05.524 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.389198+0000", 2024-09-21T10:08:05.525 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.525 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.525 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.525 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:05.525 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.389197+0000", 2024-09-21T10:08:05.525 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:08:05.525 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.525 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.525 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:05.525 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.389230+0000", 2024-09-21T10:08:05.525 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3234000000000001e-05 2024-09-21T10:08:05.525 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.526 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.526 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:05.526 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.389246+0000", 2024-09-21T10:08:05.526 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5705999999999998e-05 2024-09-21T10:08:05.526 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.526 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.526 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:05.526 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.389253+0000", 2024-09-21T10:08:05.526 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8990000000000004e-06 2024-09-21T10:08:05.526 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.526 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.526 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.526 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.393452+0000", 2024-09-21T10:08:05.527 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.004198757 2024-09-21T10:08:05.527 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.527 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.527 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:05.527 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.393568+0000", 2024-09-21T10:08:05.527 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011586699999999999 2024-09-21T10:08:05.527 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.527 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.527 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:08:05.527 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.393687+0000", 2024-09-21T10:08:05.527 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000119133 2024-09-21T10:08:05.527 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.527 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.527 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.527 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.408391+0000", 2024-09-21T10:08:05.528 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.014703842 2024-09-21T10:08:05.528 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.528 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.528 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:08:05.528 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.408398+0000", 2024-09-21T10:08:05.528 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9349999999999997e-06 2024-09-21T10:08:05.528 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.528 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.528 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.528 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.408337+0000", 2024-09-21T10:08:05.528 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.999939103 2024-09-21T10:08:05.528 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.528 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.529 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:05.529 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.408385+0000", 2024-09-21T10:08:05.529 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8242000000000003e-05 2024-09-21T10:08:05.529 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.529 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.529 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.529 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.529 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.529 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:156069 10.9 10:9f2ca64f:::benchmark_data_smithi107_184135_object24503:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-09-21T10:08:05.529 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:07:02.948366+0000", 2024-09-21T10:08:05.529 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 61.429915592999997, 2024-09-21T10:08:05.529 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.17986073899999999, 2024-09-21T10:08:05.529 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:05.529 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:05.530 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:05.530 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:05.530 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:05.530 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 156069 2024-09-21T10:08:05.530 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.530 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:05.530 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.530 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:05.530 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.948366+0000", 2024-09-21T10:08:05.530 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.530 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.530 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.530 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:05.530 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.948366+0000", 2024-09-21T10:08:05.530 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.531 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.531 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.531 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:05.531 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.948364+0000", 2024-09-21T10:08:05.531 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:08:05.531 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.531 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.531 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:05.531 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.948371+0000", 2024-09-21T10:08:05.531 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7800000000000003e-06 2024-09-21T10:08:05.531 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.531 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.531 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:05.532 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.948382+0000", 2024-09-21T10:08:05.532 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1009000000000001e-05 2024-09-21T10:08:05.532 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.532 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.532 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:05.532 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.948388+0000", 2024-09-21T10:08:05.532 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5310000000000001e-06 2024-09-21T10:08:05.532 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.532 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.532 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.532 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.951399+0000", 2024-09-21T10:08:05.532 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003010895 2024-09-21T10:08:05.532 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.532 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.532 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:05.533 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.951516+0000", 2024-09-21T10:08:05.533 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011707800000000001 2024-09-21T10:08:05.533 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.533 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.533 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:08:05.533 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.951622+0000", 2024-09-21T10:08:05.533 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000106242 2024-09-21T10:08:05.533 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.533 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.533 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.533 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.957821+0000", 2024-09-21T10:08:05.533 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0061991119999999997 2024-09-21T10:08:05.533 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.533 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.534 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:05.534 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.957830+0000", 2024-09-21T10:08:05.534 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.0250000000000008e-06 2024-09-21T10:08:05.534 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.534 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.534 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.534 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:03.128154+0000", 2024-09-21T10:08:05.534 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.17032346400000001 2024-09-21T10:08:05.534 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.534 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.534 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:05.534 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:03.128202+0000", 2024-09-21T10:08:05.534 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8936e-05 2024-09-21T10:08:05.535 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.535 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.535 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:05.535 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:03.128227+0000", 2024-09-21T10:08:05.535 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.4066e-05 2024-09-21T10:08:05.535 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.535 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.535 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.535 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.535 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.535 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36210.0:156071 10.3 e208/190 10:cafb58df:::benchmark_data_smithi107_184135_object24505:head v 208'9901, mlcod=208'9898)", 2024-09-21T10:08:05.535 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:07:02.954900+0000", 2024-09-21T10:08:05.535 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 61.42338101, 2024-09-21T10:08:05.535 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.18390453500000001, 2024-09-21T10:08:05.535 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:05.535 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:05.536 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:05.536 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.536 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:05.536 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.954900+0000", 2024-09-21T10:08:05.536 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.536 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.536 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.536 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:05.536 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.954900+0000", 2024-09-21T10:08:05.536 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.536 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.536 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.536 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:05.536 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.954900+0000", 2024-09-21T10:08:05.537 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:08:05.537 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.537 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.537 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:05.537 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.954906+0000", 2024-09-21T10:08:05.537 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6449999999999999e-06 2024-09-21T10:08:05.537 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.537 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.537 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:05.537 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.954917+0000", 2024-09-21T10:08:05.537 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1202e-05 2024-09-21T10:08:05.537 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.538 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.538 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:05.538 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.954923+0000", 2024-09-21T10:08:05.538 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2870000000000004e-06 2024-09-21T10:08:05.538 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.538 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.538 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.538 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.955030+0000", 2024-09-21T10:08:05.538 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000107686 2024-09-21T10:08:05.538 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.538 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.538 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:05.538 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.955086+0000", 2024-09-21T10:08:05.538 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5968999999999999e-05 2024-09-21T10:08:05.539 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.539 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.539 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:05.539 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:03.138625+0000", 2024-09-21T10:08:05.539 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.183538389 2024-09-21T10:08:05.539 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.539 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.539 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:05.539 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:03.138805+0000", 2024-09-21T10:08:05.539 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00018018 2024-09-21T10:08:05.539 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.539 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.539 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.540 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.540 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.540 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:156101 10.9 10:9a5fc5a4:::benchmark_data_smithi107_184135_object24535:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-09-21T10:08:05.540 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:07:02.975432+0000", 2024-09-21T10:08:05.540 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 61.402849775999996, 2024-09-21T10:08:05.540 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.17988677, 2024-09-21T10:08:05.540 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:05.540 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:05.540 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:05.540 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:05.540 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:05.540 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 156101 2024-09-21T10:08:05.541 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.541 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:05.541 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.541 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:05.541 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.975432+0000", 2024-09-21T10:08:05.541 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.541 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.541 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.541 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:05.541 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.975432+0000", 2024-09-21T10:08:05.541 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:05.541 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.541 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.541 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:05.541 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.975429+0000", 2024-09-21T10:08:05.542 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:08:05.542 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.542 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.542 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:05.542 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:02.975440+0000", 2024-09-21T10:08:05.542 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0716e-05 2024-09-21T10:08:05.542 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.542 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.542 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:05.542 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:03.128139+0000", 2024-09-21T10:08:05.542 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.152698677 2024-09-21T10:08:05.542 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.542 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.542 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:05.542 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:03.128151+0000", 2024-09-21T10:08:05.543 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1785e-05 2024-09-21T10:08:05.543 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.543 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.543 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:05.543 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:03.128251+0000", 2024-09-21T10:08:05.543 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.9982999999999995e-05 2024-09-21T10:08:05.543 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.543 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.543 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:05.543 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:03.128365+0000", 2024-09-21T10:08:05.543 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000114805 2024-09-21T10:08:05.543 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.543 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.544 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:08:05.544 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:03.128482+0000", 2024-09-21T10:08:05.544 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000116586 2024-09-21T10:08:05.544 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.544 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.544 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.544 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:03.143890+0000", 2024-09-21T10:08:05.544 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.015407641999999999 2024-09-21T10:08:05.544 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.544 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.544 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:05.544 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:03.143918+0000", 2024-09-21T10:08:05.544 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.8705e-05 2024-09-21T10:08:05.544 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.544 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.545 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:05.545 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:03.155225+0000", 2024-09-21T10:08:05.545 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.011306884 2024-09-21T10:08:05.545 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.545 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.545 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:05.545 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:03.155273+0000", 2024-09-21T10:08:05.545 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7639000000000002e-05 2024-09-21T10:08:05.545 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:05.545 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:05.545 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:05.545 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:03.155318+0000", 2024-09-21T10:08:05.545 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5478000000000001e-05 2024-09-21T10:08:05.545 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.546 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.546 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.546 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:05.546 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:05.546 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:08:05.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:05.615 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:05.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:05.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:05.716 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:05.727 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:05 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:08:05.727 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:05 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1453371053' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:08:05.727 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:05 smithi107 ceph-mon[102221]: osdmap e221: 8 total, 7 up, 8 in 2024-09-21T10:08:05.727 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:05 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:08:05.727 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:05 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1453371053' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:08:05.728 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:05 smithi107 ceph-mon[105022]: osdmap e221: 8 total, 7 up, 8 in 2024-09-21T10:08:05.745 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:05.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:05.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:05.847 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:05.884 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:05.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:05.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:05 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:08:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:05 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1453371053' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:08:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:05 smithi129 ceph-mon[88471]: osdmap e221: 8 total, 7 up, 8 in 2024-09-21T10:08:05.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:06.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:06.016 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:06.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:06.019 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:08:06.019 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "" 2024-09-21T10:08:06.019 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:08:06.118 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:06.161 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:06.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:06.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:06.263 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:06.272 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:06.300 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:06.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:06.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:06.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:06.434 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:06.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:06.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:06.538 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:06.577 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:06.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:06.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:06.678 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:06.714 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:06.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:06.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:06.816 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:06.860 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:06.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:06.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:06.865 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:08:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:06 smithi129 ceph-mon[88471]: pgmap v798: 109 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 78 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 13 KiB/s rd, 28 KiB/s wr, 943 op/s; 25495/160578 objects degraded (15.877%); 7514/160578 objects misplaced (4.679%); 18 MiB/s, 279 objects/s recovering 2024-09-21T10:08:06.961 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:06.967 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:06 smithi107 ceph-mon[102221]: pgmap v798: 109 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 78 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 13 KiB/s rd, 28 KiB/s wr, 943 op/s; 25495/160578 objects degraded (15.877%); 7514/160578 objects misplaced (4.679%); 18 MiB/s, 279 objects/s recovering 2024-09-21T10:08:06.967 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:06 smithi107 ceph-mon[105022]: pgmap v798: 109 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 78 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 13 KiB/s rd, 28 KiB/s wr, 943 op/s; 25495/160578 objects degraded (15.877%); 7514/160578 objects misplaced (4.679%); 18 MiB/s, 279 objects/s recovering 2024-09-21T10:08:07.017 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:07.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:07.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:07.119 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:07.144 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:07.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:07.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:07.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:07.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:07.317 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:07.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:07.364 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:08:07.418 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:07.444 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:07.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:07.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:07.546 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:07.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:08:07.588 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:07.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:07.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:07.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:07.716 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:07.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:07.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:07.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:07.845 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:07.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:07.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:07.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:07.977 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:07.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:07.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:08.079 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:08.120 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:08.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:08.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:08.221 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:08.248 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:08.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:08.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:08.350 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:08.389 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:08.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:08.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:08.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:08.518 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:08.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:08.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:08.619 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:08.659 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:08.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:08.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:08.761 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:08.810 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:08.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:08.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:08.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:08.914 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:08 smithi107 ceph-mon[102221]: pgmap v799: 109 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 78 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 11 KiB/s rd, 23 KiB/s wr, 816 op/s; 25164/159585 objects degraded (15.768%); 7514/159585 objects misplaced (4.708%); 15 MiB/s, 231 objects/s recovering 2024-09-21T10:08:08.915 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:08 smithi129 ceph-mon[88471]: pgmap v799: 109 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 78 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 11 KiB/s rd, 23 KiB/s wr, 816 op/s; 25164/159585 objects degraded (15.768%); 7514/159585 objects misplaced (4.708%); 15 MiB/s, 231 objects/s recovering 2024-09-21T10:08:08.917 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:08 smithi107 ceph-mon[105022]: pgmap v799: 109 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 78 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 11 KiB/s rd, 23 KiB/s wr, 816 op/s; 25164/159585 objects degraded (15.768%); 7514/159585 objects misplaced (4.708%); 15 MiB/s, 231 objects/s recovering 2024-09-21T10:08:08.921 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:08.926 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T10:08:08.950 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:08.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:08.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:08.956 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:08:08.956 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:08:08.986 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:08:08.986 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:08:09.012 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:08:09.012 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:08:09.038 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:08:09.038 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:08:09.053 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:09.071 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:08:09.071 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:08:09.105 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:08:09.105 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:08:09.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:09.106 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:09.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:09.131 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:08:09.132 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:08:09.206 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:09.241 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:09.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:09.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:09.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:09.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:09.371 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:09.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:09.389 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:08:09.389 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:08:09.472 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:09.500 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:09.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:09.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:09.602 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:09.641 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:09.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:09.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:09.743 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:09.786 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:09.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:09.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:09.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:09 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3153229508' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:09.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:09 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 25164/159585 objects degraded (15.768%), 10 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:09.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:09 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3153229508' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:09.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:09 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 25164/159585 objects degraded (15.768%), 10 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:09.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:09.913 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:09.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:09.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:09.994 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:08:09.995 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:08:10.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:10.043 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:10.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:10.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:10.101 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:09 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3153229508' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:10.102 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:09 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 25164/159585 objects degraded (15.768%), 10 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:10.145 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:10.185 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:10.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:10.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:10.286 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:10.321 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:10.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:10.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:10.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:08:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:08:10.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:08:10.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:10.464 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:10.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:10.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:10.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:10.593 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:10.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:10.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:10.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:10.716 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:10.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:10.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:10.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:10.840 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:10.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:10.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:10.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:10 smithi107 ceph-mon[102221]: pgmap v800: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 3.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 928 op/s; 23113/147480 objects degraded (15.672%); 6478/147480 objects misplaced (4.392%); 5.3 MiB/s, 84 objects/s recovering 2024-09-21T10:08:10.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:10 smithi107 ceph-mon[105022]: pgmap v800: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 3.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 928 op/s; 23113/147480 objects degraded (15.672%); 6478/147480 objects misplaced (4.392%); 5.3 MiB/s, 84 objects/s recovering 2024-09-21T10:08:10.942 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:10.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:10.968 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:10.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:11.069 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:11.085 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:10 smithi129 ceph-mon[88471]: pgmap v800: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 3.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 928 op/s; 23113/147480 objects degraded (15.672%); 6478/147480 objects misplaced (4.392%); 5.3 MiB/s, 84 objects/s recovering 2024-09-21T10:08:11.111 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:11.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:11.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:11.213 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:11.249 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:11.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:11.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:11.352 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:11.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:11.374 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:11.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:11.475 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:11.500 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:11.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:11.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:11.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:11.629 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:11.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:11.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:11.731 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:11.756 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:11.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:11.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:11.851 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:08:11.852 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:08:11.852 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-21T10:08:11.852 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:08:11.852 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:08:11.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-21T10:08:11.858 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:11.878 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:11.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:11.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:11.980 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:11.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:08:12.017 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:12.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:12.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:12.021 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:08:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:08:11] "GET /metrics HTTP/1.1" 200 36840 "" "Prometheus/2.51.0" 2024-09-21T10:08:12.119 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:12.144 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:12.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:12.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:12.166 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:08:12.174 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:12.247 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:12.288 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:12.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:12.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:08:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:12 smithi107 ceph-mon[102221]: pgmap v801: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 934 op/s; 22832/146637 objects degraded (15.570%); 6478/146637 objects misplaced (4.418%); 5.3 MiB/s, 84 objects/s recovering 2024-09-21T10:08:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:12 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4023232230' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:08:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:12 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:08:12.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:08:12.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:12.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:12 smithi107 ceph-mon[105022]: pgmap v801: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 934 op/s; 22832/146637 objects degraded (15.570%); 6478/146637 objects misplaced (4.418%); 5.3 MiB/s, 84 objects/s recovering 2024-09-21T10:08:12.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:12 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4023232230' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:08:12.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:12 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:08:12.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:08:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:12 smithi129 ceph-mon[88471]: pgmap v801: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 3.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 934 op/s; 22832/146637 objects degraded (15.570%); 6478/146637 objects misplaced (4.418%); 5.3 MiB/s, 84 objects/s recovering 2024-09-21T10:08:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:12 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4023232230' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:08:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:12 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:08:12.421 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:12.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:12.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:12.523 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:12.573 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:12.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:12.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:12.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:12.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:12.696 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:12.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:12.798 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:12.822 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:12.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:12.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:12.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:12.952 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:12.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:12.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:13.054 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:13.091 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:13.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:13.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:13.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:13.218 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:13.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:13.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:13.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:13 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:08:13.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:13 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:08:13.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:13 smithi107 ceph-mon[102221]: osdmap e222: 8 total, 7 up, 8 in 2024-09-21T10:08:13.301 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:13 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:08:13.301 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:13 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:08:13.301 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:13 smithi107 ceph-mon[105022]: osdmap e222: 8 total, 7 up, 8 in 2024-09-21T10:08:13.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:13.350 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:13.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:13.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:13.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:13 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:08:13.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:13 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:08:13.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:13 smithi129 ceph-mon[88471]: osdmap e222: 8 total, 7 up, 8 in 2024-09-21T10:08:13.452 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:13.491 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:13.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:13.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:13.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:13.615 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:13.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:13.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:13.717 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:13.739 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:13.753 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:13.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:13.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:13.855 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:13.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:13.891 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:13.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:13.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:14.019 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:14.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:14.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:14.078 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:08:14.121 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:14.191 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:14.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:14.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:14.293 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:14.338 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:14.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:14.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:14.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:14 smithi129 ceph-mon[88471]: pgmap v803: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 913 op/s; 21244/137976 objects degraded (15.397%); 5569/137976 objects misplaced (4.036%); 5.7 MiB/s, 92 objects/s recovering 2024-09-21T10:08:14.440 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:14.496 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:14 smithi107 ceph-mon[102221]: pgmap v803: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 913 op/s; 21244/137976 objects degraded (15.397%); 5569/137976 objects misplaced (4.036%); 5.7 MiB/s, 92 objects/s recovering 2024-09-21T10:08:14.497 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:14 smithi107 ceph-mon[105022]: pgmap v803: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 913 op/s; 21244/137976 objects degraded (15.397%); 5569/137976 objects misplaced (4.036%); 5.7 MiB/s, 92 objects/s recovering 2024-09-21T10:08:14.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:14.499 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:14.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:14.514 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:08:14.601 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:14.649 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:14.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:14.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:14.751 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:14.785 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:14.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:14.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:14.886 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:14.923 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:14.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:14.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:14.979 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:08:14.979 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:08:14.979 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:08:15.024 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:15.069 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:15.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:15.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:15.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:08:15.171 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:15.206 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:15.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:15.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:15.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:15.342 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:15.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:15.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:15.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:15 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 21244/137976 objects degraded (15.397%), 10 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:15.427 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:15 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 21244/137976 objects degraded (15.397%), 10 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:15.428 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:15 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 21244/137976 objects degraded (15.397%), 10 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:15.444 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:15.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:15.474 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:15.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:15.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:15.600 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:15.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:15.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:15.702 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:15.728 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:15.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:15.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:15.829 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:15.868 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:15.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:15.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:15.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:15.995 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:15.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:15.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:16.097 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:16.148 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:16.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:16.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:16.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:16.280 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:16.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:16.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:16.367 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:16 smithi129 ceph-mon[88471]: pgmap v804: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 20498/134193 objects degraded (15.275%); 5569/134193 objects misplaced (4.150%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T10:08:16.381 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:16.382 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:16.426 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:16.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:16.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:16.501 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:16 smithi107 ceph-mon[102221]: pgmap v804: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 20498/134193 objects degraded (15.275%); 5569/134193 objects misplaced (4.150%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T10:08:16.501 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:16 smithi107 ceph-mon[105022]: pgmap v804: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 20498/134193 objects degraded (15.275%); 5569/134193 objects misplaced (4.150%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T10:08:16.527 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:16.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:16.571 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:16.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:16.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:16.698 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:08:16.698 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:08:16.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:16.699 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:16.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:16.800 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:16.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:16.839 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:16.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:16.940 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:16.974 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:16.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:16.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:17.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:17.106 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:17.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:17.207 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:17.236 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:17.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:17.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:17.338 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:17.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:17.384 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:17.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:17.485 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:17.515 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:17.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:17.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:17.619 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:17.659 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:17.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:17.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:17.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:17.788 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:17.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:17.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:17.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:17.917 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:17.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:17.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:17.991 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:08:18.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:18.041 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:18.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:18.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:18.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:18.170 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:18.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:18.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:18.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:18.300 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:18.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:18.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:18.402 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:18.421 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:08:18.450 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:18.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:18.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:18.552 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:18.566 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:18 smithi129 ceph-mon[88471]: pgmap v805: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 20219/133356 objects degraded (15.162%); 5569/133356 objects misplaced (4.176%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T10:08:18.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:18.608 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:18.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:18.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:18 smithi107 ceph-mon[102221]: pgmap v805: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 20219/133356 objects degraded (15.162%); 5569/133356 objects misplaced (4.176%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T10:08:18.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:18 smithi107 ceph-mon[105022]: pgmap v805: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 873 op/s; 20219/133356 objects degraded (15.162%); 5569/133356 objects misplaced (4.176%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T10:08:18.708 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:18.756 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:18.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:18.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:18.859 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:18.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:18.905 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:18.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:19.006 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:19.044 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:19.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:19.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:19.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:19.168 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:19.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:19.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:19.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:19.302 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:19.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:19.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:19.404 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:19.431 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:19.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:19.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:19.532 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:19.568 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:19.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:19.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:19.670 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:19.698 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:19.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:19.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:19.800 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:19.833 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:19.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:19.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:19.926 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T10:08:19.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:19.955 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:08:19.955 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:08:19.964 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:19.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:19.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:19.987 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:08:19.987 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:08:20.006 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:08:20.006 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:08:20.036 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:08:20.036 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:08:20.064 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:08:20.065 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:08:20.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:20.095 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:20.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:20.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:20.096 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:08:20.096 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:08:20.125 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:08:20.125 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:08:20.197 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:20.226 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:20.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:20.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:20.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:20.353 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:20.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:20.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:20.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:20 smithi107 ceph-mon[102221]: pgmap v806: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 888 op/s; 18229/120807 objects degraded (15.089%); 4619/120807 objects misplaced (3.823%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T10:08:20.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:20 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 18229/120807 objects degraded (15.089%), 10 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:20.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:20 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3676529556' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:20.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:20 smithi107 ceph-mon[105022]: pgmap v806: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 888 op/s; 18229/120807 objects degraded (15.089%); 4619/120807 objects misplaced (3.823%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T10:08:20.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:20 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 18229/120807 objects degraded (15.089%), 10 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:20.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:20 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3676529556' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:20.455 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:20.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:20.482 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:20.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:20.583 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:20.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:20.626 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:20.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:20.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:20 smithi129 ceph-mon[88471]: pgmap v806: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 888 op/s; 18229/120807 objects degraded (15.089%); 4619/120807 objects misplaced (3.823%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T10:08:20.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:20 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 18229/120807 objects degraded (15.089%), 10 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:20.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:20 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3676529556' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:20.671 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:08:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:08:20.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:08:20.728 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:20.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:20.770 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:20.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:20.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:20.900 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:20.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:20.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:20.912 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:08:20.912 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:08:20.912 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:08:20.912 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.000000 2024-09-21T10:08:20.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 1 0 2024-09-21T10:08:21.001 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:21.040 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:21.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:21.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:21.142 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:21.194 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:21.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:21.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:21.278 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:08:21.278 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:08:21.286 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:08:21.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:21.332 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:21.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:21.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:21.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:21 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2505917918' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:21.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:21 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:21.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:21.467 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:21.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:21.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:21.540 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:21.568 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:21.603 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:21.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:21.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:21 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2505917918' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:21.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:21 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:21.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:21 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2505917918' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:21.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:21 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:21.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:21.733 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:21.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:21.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:21.796 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:08:21.796 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 20, 2024-09-21T10:08:21.796 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 600, 2024-09-21T10:08:21.796 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:08:21.796 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.796 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75905 10.b 10:df33c370:::benchmark_data_smithi107_184135_object75904:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:08:21.796 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.065777+0000", 2024-09-21T10:08:21.796 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 152.72548500100001, 2024-09-21T10:08:21.796 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.253965828, 2024-09-21T10:08:21.796 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:21.796 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:21.796 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:21.797 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:21.797 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:21.797 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75905 2024-09-21T10:08:21.797 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.797 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:21.797 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.797 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:21.797 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.065777+0000", 2024-09-21T10:08:21.797 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.797 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.797 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.797 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:21.797 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.065777+0000", 2024-09-21T10:08:21.797 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.797 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.798 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.798 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:21.798 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.065776+0000", 2024-09-21T10:08:21.798 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:08:21.798 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.798 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.798 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:21.798 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.065834+0000", 2024-09-21T10:08:21.798 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7809000000000002e-05 2024-09-21T10:08:21.798 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.798 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.799 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:21.799 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.065842+0000", 2024-09-21T10:08:21.799 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.1480000000000006e-06 2024-09-21T10:08:21.799 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.799 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.799 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:21.799 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.065847+0000", 2024-09-21T10:08:21.799 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8230000000000002e-06 2024-09-21T10:08:21.799 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.799 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.799 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.799 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.069759+0000", 2024-09-21T10:08:21.799 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0039118210000000002 2024-09-21T10:08:21.799 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.799 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.800 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:21.800 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.069860+0000", 2024-09-21T10:08:21.800 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010105199999999999 2024-09-21T10:08:21.800 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.800 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.800 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:08:21.800 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.069976+0000", 2024-09-21T10:08:21.800 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011605899999999999 2024-09-21T10:08:21.800 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.800 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.800 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.800 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.076580+0000", 2024-09-21T10:08:21.800 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.006603996 2024-09-21T10:08:21.800 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.800 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.801 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:21.801 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.080422+0000", 2024-09-21T10:08:21.801 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00384287 2024-09-21T10:08:21.801 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.801 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.801 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.801 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.319673+0000", 2024-09-21T10:08:21.801 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.239250085 2024-09-21T10:08:21.801 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.801 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.801 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:21.801 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.319715+0000", 2024-09-21T10:08:21.801 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2398999999999998e-05 2024-09-21T10:08:21.801 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.801 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.801 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:21.802 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.319743+0000", 2024-09-21T10:08:21.802 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7674999999999998e-05 2024-09-21T10:08:21.802 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.802 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.802 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.802 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.802 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.802 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75913 10.b 10:d690a9d4:::benchmark_data_smithi107_184135_object75912:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:08:21.802 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.071867+0000", 2024-09-21T10:08:21.802 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 152.71939489100001, 2024-09-21T10:08:21.802 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.247963933, 2024-09-21T10:08:21.802 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:21.802 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:21.803 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:21.803 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:21.803 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:21.803 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75913 2024-09-21T10:08:21.803 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.803 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:21.803 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.803 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:21.803 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.071867+0000", 2024-09-21T10:08:21.803 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.803 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.803 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.804 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:21.804 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.071867+0000", 2024-09-21T10:08:21.804 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.804 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.804 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.804 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:21.804 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.071866+0000", 2024-09-21T10:08:21.804 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:08:21.804 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.804 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.804 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:21.804 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.072104+0000", 2024-09-21T10:08:21.804 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00023810199999999999 2024-09-21T10:08:21.804 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.804 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.805 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:21.805 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.072116+0000", 2024-09-21T10:08:21.805 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.171e-05 2024-09-21T10:08:21.805 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.805 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.805 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:21.805 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.072120+0000", 2024-09-21T10:08:21.805 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3089999999999997e-06 2024-09-21T10:08:21.805 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.805 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.805 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.805 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.072705+0000", 2024-09-21T10:08:21.805 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00058508799999999999 2024-09-21T10:08:21.805 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.805 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.805 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:21.805 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.072824+0000", 2024-09-21T10:08:21.806 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000119162 2024-09-21T10:08:21.806 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.806 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.806 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:08:21.806 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.072952+0000", 2024-09-21T10:08:21.806 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012725599999999999 2024-09-21T10:08:21.806 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.806 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.807 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:21.807 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.080427+0000", 2024-09-21T10:08:21.807 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.007475359 2024-09-21T10:08:21.807 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.807 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.807 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.807 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.081859+0000", 2024-09-21T10:08:21.807 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001432399 2024-09-21T10:08:21.807 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.807 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.807 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.807 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.319776+0000", 2024-09-21T10:08:21.807 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23791665200000001 2024-09-21T10:08:21.808 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.808 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.808 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:21.808 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.319809+0000", 2024-09-21T10:08:21.808 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2802000000000002e-05 2024-09-21T10:08:21.808 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.808 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.808 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:21.808 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.319831+0000", 2024-09-21T10:08:21.808 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1970000000000001e-05 2024-09-21T10:08:21.808 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.808 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.808 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.808 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.808 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.809 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75917 10.0 10:04450359:::benchmark_data_smithi107_184135_object75916:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:08:21.809 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.074183+0000", 2024-09-21T10:08:21.809 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 152.71707865299999, 2024-09-21T10:08:21.809 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25040411400000001, 2024-09-21T10:08:21.809 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:21.809 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:21.809 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:21.809 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:21.810 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:21.810 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75917 2024-09-21T10:08:21.810 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.810 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:21.810 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.810 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:21.810 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074183+0000", 2024-09-21T10:08:21.810 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.811 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.811 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.811 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:21.811 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074183+0000", 2024-09-21T10:08:21.811 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.811 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.811 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.811 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:21.811 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074182+0000", 2024-09-21T10:08:21.811 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:08:21.811 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.811 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.811 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:21.811 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074235+0000", 2024-09-21T10:08:21.811 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2439999999999999e-05 2024-09-21T10:08:21.812 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.812 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.812 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:21.812 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074245+0000", 2024-09-21T10:08:21.812 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0042e-05 2024-09-21T10:08:21.812 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.812 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.812 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:21.812 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074260+0000", 2024-09-21T10:08:21.812 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5486999999999999e-05 2024-09-21T10:08:21.812 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.812 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.812 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.812 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074390+0000", 2024-09-21T10:08:21.812 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012976099999999999 2024-09-21T10:08:21.812 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.813 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.813 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:21.813 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074502+0000", 2024-09-21T10:08:21.813 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011167000000000001 2024-09-21T10:08:21.813 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.813 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.813 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:08:21.813 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074616+0000", 2024-09-21T10:08:21.813 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000114481 2024-09-21T10:08:21.813 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.813 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.813 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:21.813 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.078114+0000", 2024-09-21T10:08:21.813 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003497413 2024-09-21T10:08:21.813 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.814 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.814 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.814 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.078322+0000", 2024-09-21T10:08:21.814 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00020845799999999999 2024-09-21T10:08:21.814 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.814 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.814 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.814 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.324518+0000", 2024-09-21T10:08:21.814 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.246196096 2024-09-21T10:08:21.814 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.814 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.814 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:21.815 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.324553+0000", 2024-09-21T10:08:21.815 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4459999999999999e-05 2024-09-21T10:08:21.815 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.815 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.815 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:21.815 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.324587+0000", 2024-09-21T10:08:21.815 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4591999999999997e-05 2024-09-21T10:08:21.815 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.815 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.815 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.815 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.815 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.815 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75920 10.b 10:da23dad2:::benchmark_data_smithi107_184135_object75919:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:08:21.816 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.074352+0000", 2024-09-21T10:08:21.816 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 152.716910302, 2024-09-21T10:08:21.816 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.24755237099999999, 2024-09-21T10:08:21.816 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:21.816 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:21.816 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:21.816 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:21.816 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:21.816 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75920 2024-09-21T10:08:21.816 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.816 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:21.816 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.816 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:21.816 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074352+0000", 2024-09-21T10:08:21.817 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.817 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.817 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.817 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:21.817 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074352+0000", 2024-09-21T10:08:21.817 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.817 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.817 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.817 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:21.817 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074351+0000", 2024-09-21T10:08:21.817 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:21.817 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.817 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.817 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:21.817 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074380+0000", 2024-09-21T10:08:21.818 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.8773999999999999e-05 2024-09-21T10:08:21.818 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.818 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.818 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:21.818 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074387+0000", 2024-09-21T10:08:21.818 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7440000000000001e-06 2024-09-21T10:08:21.818 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.818 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.818 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:21.818 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.074390+0000", 2024-09-21T10:08:21.818 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8539999999999999e-06 2024-09-21T10:08:21.818 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.818 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.818 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.819 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.077335+0000", 2024-09-21T10:08:21.819 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0029443210000000002 2024-09-21T10:08:21.819 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.819 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.819 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:21.819 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.077456+0000", 2024-09-21T10:08:21.819 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000121178 2024-09-21T10:08:21.819 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.819 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.819 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:08:21.819 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.077583+0000", 2024-09-21T10:08:21.819 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012657599999999999 2024-09-21T10:08:21.819 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.819 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.819 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:21.820 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.080555+0000", 2024-09-21T10:08:21.820 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0029719970000000001 2024-09-21T10:08:21.820 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.820 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.820 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.820 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.081977+0000", 2024-09-21T10:08:21.820 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0014220159999999999 2024-09-21T10:08:21.820 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.820 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.820 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.820 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.321814+0000", 2024-09-21T10:08:21.821 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23983771600000001 2024-09-21T10:08:21.821 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.821 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.821 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:21.821 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.321867+0000", 2024-09-21T10:08:21.821 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2587999999999999e-05 2024-09-21T10:08:21.821 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.821 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.821 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:21.822 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.321904+0000", 2024-09-21T10:08:21.822 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7024999999999999e-05 2024-09-21T10:08:21.822 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.822 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.822 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.822 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.822 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.822 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75931 10.0 10:02f906db:::benchmark_data_smithi107_184135_object75930:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:08:21.822 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.084617+0000", 2024-09-21T10:08:21.822 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 152.70664503699999, 2024-09-21T10:08:21.822 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.24220968000000001, 2024-09-21T10:08:21.823 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:21.823 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:21.823 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:21.823 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:21.823 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:21.823 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75931 2024-09-21T10:08:21.823 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.823 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:21.823 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.823 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:21.823 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.084617+0000", 2024-09-21T10:08:21.823 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.823 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.823 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.823 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:21.824 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.084617+0000", 2024-09-21T10:08:21.824 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.824 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.824 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.824 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:21.824 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.084616+0000", 2024-09-21T10:08:21.824 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:08:21.824 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.824 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.824 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:21.824 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.084703+0000", 2024-09-21T10:08:21.824 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6710000000000002e-05 2024-09-21T10:08:21.824 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.825 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.825 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:21.825 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.084715+0000", 2024-09-21T10:08:21.825 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2319e-05 2024-09-21T10:08:21.825 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.825 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.825 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:21.825 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.084721+0000", 2024-09-21T10:08:21.825 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2339999999999999e-06 2024-09-21T10:08:21.825 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.825 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.825 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.825 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.084768+0000", 2024-09-21T10:08:21.825 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6458999999999997e-05 2024-09-21T10:08:21.825 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.826 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.826 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:21.826 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.084925+0000", 2024-09-21T10:08:21.826 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015770700000000001 2024-09-21T10:08:21.826 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.826 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.826 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:08:21.826 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.085032+0000", 2024-09-21T10:08:21.826 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000106436 2024-09-21T10:08:21.826 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.826 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.826 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.826 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.090962+0000", 2024-09-21T10:08:21.827 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0059300890000000004 2024-09-21T10:08:21.827 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.827 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.827 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:21.827 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.091650+0000", 2024-09-21T10:08:21.827 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00068853600000000001 2024-09-21T10:08:21.827 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.827 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.827 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.827 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.326760+0000", 2024-09-21T10:08:21.827 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23510992 2024-09-21T10:08:21.827 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.827 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.827 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:21.827 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.326794+0000", 2024-09-21T10:08:21.828 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3200000000000001e-05 2024-09-21T10:08:21.828 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.828 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.828 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:21.828 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.326826+0000", 2024-09-21T10:08:21.828 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.294e-05 2024-09-21T10:08:21.828 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.828 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.828 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.828 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.828 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.828 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75937 10.0 10:0c60d6b6:::benchmark_data_smithi107_184135_object75936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:08:21.828 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.087439+0000", 2024-09-21T10:08:21.828 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 152.70382287000001, 2024-09-21T10:08:21.828 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.239497922, 2024-09-21T10:08:21.829 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:21.829 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:21.829 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:21.829 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:21.829 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:21.829 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75937 2024-09-21T10:08:21.829 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.829 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:21.829 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.829 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:21.829 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.087439+0000", 2024-09-21T10:08:21.829 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.829 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.829 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.830 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:21.830 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.087439+0000", 2024-09-21T10:08:21.830 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.830 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.830 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.830 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:21.830 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.087439+0000", 2024-09-21T10:08:21.830 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:21.830 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.830 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.830 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:21.830 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.087460+0000", 2024-09-21T10:08:21.831 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1469999999999999e-05 2024-09-21T10:08:21.831 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.831 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.831 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:21.831 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.087464+0000", 2024-09-21T10:08:21.831 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4270000000000001e-06 2024-09-21T10:08:21.831 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.831 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.831 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:21.831 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.087467+0000", 2024-09-21T10:08:21.831 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.553e-06 2024-09-21T10:08:21.831 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.832 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.832 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.832 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.087675+0000", 2024-09-21T10:08:21.832 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000207906 2024-09-21T10:08:21.832 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.832 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.832 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:21.832 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.087772+0000", 2024-09-21T10:08:21.832 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.6756999999999995e-05 2024-09-21T10:08:21.832 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.832 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.832 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:08:21.832 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.087892+0000", 2024-09-21T10:08:21.832 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000120674 2024-09-21T10:08:21.832 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.833 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.833 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.833 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.091273+0000", 2024-09-21T10:08:21.833 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003380802 2024-09-21T10:08:21.833 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.833 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.833 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:21.833 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.091668+0000", 2024-09-21T10:08:21.833 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000394931 2024-09-21T10:08:21.833 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.833 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.833 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.833 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.326870+0000", 2024-09-21T10:08:21.833 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.235202349 2024-09-21T10:08:21.833 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.834 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.834 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:21.834 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.326900+0000", 2024-09-21T10:08:21.834 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.9692e-05 2024-09-21T10:08:21.834 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.834 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.834 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:21.834 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.326937+0000", 2024-09-21T10:08:21.834 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6739000000000002e-05 2024-09-21T10:08:21.834 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.834 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.834 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.834 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.835 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.835 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75939 10.b 10:d4e3f307:::benchmark_data_smithi107_184135_object75938:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:08:21.835 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.089180+0000", 2024-09-21T10:08:21.835 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 152.702081612, 2024-09-21T10:08:21.835 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23286099399999999, 2024-09-21T10:08:21.835 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:21.835 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:21.835 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:21.835 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:21.835 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:21.835 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75939 2024-09-21T10:08:21.835 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.835 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:21.835 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.835 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:21.836 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.089180+0000", 2024-09-21T10:08:21.836 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.836 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.836 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.836 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:21.836 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.089180+0000", 2024-09-21T10:08:21.836 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.836 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.836 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.836 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:21.836 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.089179+0000", 2024-09-21T10:08:21.836 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:08:21.836 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.836 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.836 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:21.837 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.089230+0000", 2024-09-21T10:08:21.837 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1076999999999999e-05 2024-09-21T10:08:21.837 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.837 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.837 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:21.837 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.089247+0000", 2024-09-21T10:08:21.837 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.6375000000000002e-05 2024-09-21T10:08:21.837 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.837 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.837 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:21.837 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.089252+0000", 2024-09-21T10:08:21.837 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3900000000000001e-06 2024-09-21T10:08:21.837 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.837 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.838 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.838 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.090734+0000", 2024-09-21T10:08:21.838 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0014821789999999999 2024-09-21T10:08:21.838 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.838 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.838 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:21.838 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.090992+0000", 2024-09-21T10:08:21.838 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000258266 2024-09-21T10:08:21.838 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.838 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.838 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:08:21.839 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.091136+0000", 2024-09-21T10:08:21.839 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014327400000000001 2024-09-21T10:08:21.839 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.839 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.839 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:21.839 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.096462+0000", 2024-09-21T10:08:21.839 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0053259020000000004 2024-09-21T10:08:21.839 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.839 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.839 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.839 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.096500+0000", 2024-09-21T10:08:21.839 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8269000000000002e-05 2024-09-21T10:08:21.839 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.839 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.840 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.840 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.321962+0000", 2024-09-21T10:08:21.840 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.225462052 2024-09-21T10:08:21.840 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.840 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.840 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:21.840 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.322007+0000", 2024-09-21T10:08:21.840 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5615000000000003e-05 2024-09-21T10:08:21.841 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.841 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.841 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:21.841 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.322041+0000", 2024-09-21T10:08:21.841 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3738999999999997e-05 2024-09-21T10:08:21.841 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.841 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.842 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.842 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.842 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.842 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75942 10.b 10:dbcbfc02:::benchmark_data_smithi107_184135_object75941:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:08:21.842 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.092155+0000", 2024-09-21T10:08:21.842 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 152.69910632400001, 2024-09-21T10:08:21.842 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23667438599999999, 2024-09-21T10:08:21.843 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:21.843 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:21.843 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:21.843 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:21.843 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:21.843 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75942 2024-09-21T10:08:21.844 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.844 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:21.844 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.844 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:21.844 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.092155+0000", 2024-09-21T10:08:21.844 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.844 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.844 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.845 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:21.845 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.092155+0000", 2024-09-21T10:08:21.845 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.845 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.845 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.845 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:21.846 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.092154+0000", 2024-09-21T10:08:21.846 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:08:21.846 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.846 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.846 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:21.846 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.092207+0000", 2024-09-21T10:08:21.846 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3140999999999998e-05 2024-09-21T10:08:21.846 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.846 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.846 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:21.846 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.092220+0000", 2024-09-21T10:08:21.846 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2267e-05 2024-09-21T10:08:21.847 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.847 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.847 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:21.847 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.092226+0000", 2024-09-21T10:08:21.847 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2049999999999999e-06 2024-09-21T10:08:21.847 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.847 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.847 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.847 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.093742+0000", 2024-09-21T10:08:21.847 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0015158979999999999 2024-09-21T10:08:21.847 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.847 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.847 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:21.847 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.093852+0000", 2024-09-21T10:08:21.847 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011042700000000001 2024-09-21T10:08:21.848 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.848 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.848 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:08:21.848 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.093994+0000", 2024-09-21T10:08:21.848 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000141877 2024-09-21T10:08:21.848 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.848 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.848 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:21.848 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.096506+0000", 2024-09-21T10:08:21.848 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0025119230000000001 2024-09-21T10:08:21.848 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.848 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.848 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.848 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.096533+0000", 2024-09-21T10:08:21.848 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7387000000000001e-05 2024-09-21T10:08:21.849 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.849 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.849 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.849 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.328717+0000", 2024-09-21T10:08:21.849 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.232183787 2024-09-21T10:08:21.849 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.849 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.849 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:21.849 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.328793+0000", 2024-09-21T10:08:21.849 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.5401000000000005e-05 2024-09-21T10:08:21.849 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.849 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.849 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:21.849 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.328830+0000", 2024-09-21T10:08:21.849 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7215000000000002e-05 2024-09-21T10:08:21.850 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.850 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.850 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.850 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.850 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.850 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75961 10.0 10:0cdaf837:::benchmark_data_smithi107_184135_object75960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:08:21.850 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.112151+0000", 2024-09-21T10:08:21.850 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 152.679111167, 2024-09-21T10:08:21.850 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21492383800000001, 2024-09-21T10:08:21.850 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:21.850 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:21.850 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:21.850 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:21.851 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:21.851 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75961 2024-09-21T10:08:21.851 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.851 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:21.851 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.851 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:21.851 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112151+0000", 2024-09-21T10:08:21.851 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.851 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.851 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.851 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:21.851 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112151+0000", 2024-09-21T10:08:21.851 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.851 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.851 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.852 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:21.852 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112150+0000", 2024-09-21T10:08:21.852 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:08:21.852 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.852 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.852 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:21.852 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112206+0000", 2024-09-21T10:08:21.852 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6582000000000001e-05 2024-09-21T10:08:21.852 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.852 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.852 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:21.852 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112216+0000", 2024-09-21T10:08:21.852 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.7939999999999997e-06 2024-09-21T10:08:21.852 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.852 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.853 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:21.853 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112221+0000", 2024-09-21T10:08:21.853 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7709999999999997e-06 2024-09-21T10:08:21.853 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.853 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.853 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.853 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112281+0000", 2024-09-21T10:08:21.853 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0659000000000003e-05 2024-09-21T10:08:21.853 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.853 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.853 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:21.853 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112395+0000", 2024-09-21T10:08:21.853 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000113105 2024-09-21T10:08:21.853 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.853 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.854 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:08:21.854 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112490+0000", 2024-09-21T10:08:21.854 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.5764e-05 2024-09-21T10:08:21.854 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.854 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:21.856 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:21.856 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.116884+0000", 2024-09-21T10:08:21.856 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0043935170000000004 2024-09-21T10:08:21.856 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.856 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.856 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.856 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.116920+0000", 2024-09-21T10:08:21.856 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6304999999999999e-05 2024-09-21T10:08:21.856 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.856 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.856 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.856 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.326981+0000", 2024-09-21T10:08:21.856 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21006082300000001 2024-09-21T10:08:21.857 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.857 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.857 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:21.857 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.327031+0000", 2024-09-21T10:08:21.857 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9876000000000002e-05 2024-09-21T10:08:21.857 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.857 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.857 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:21.857 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.327074+0000", 2024-09-21T10:08:21.857 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3628000000000003e-05 2024-09-21T10:08:21.857 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.857 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.857 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.857 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.857 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.858 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:75962 10.0 10:098753a3:::benchmark_data_smithi107_184135_object75961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-21T10:08:21.858 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:05:49.112237+0000", 2024-09-21T10:08:21.858 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 152.67902433699999, 2024-09-21T10:08:21.858 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.218636311, 2024-09-21T10:08:21.858 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:21.858 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:21.858 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:21.858 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:21.858 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:21.858 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 75962 2024-09-21T10:08:21.858 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.858 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:21.858 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.858 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:21.859 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112237+0000", 2024-09-21T10:08:21.859 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.859 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.859 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.859 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:21.859 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112237+0000", 2024-09-21T10:08:21.859 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.859 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.859 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.859 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:21.859 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112237+0000", 2024-09-21T10:08:21.859 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:21.859 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.859 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.860 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:21.860 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112298+0000", 2024-09-21T10:08:21.860 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1126999999999999e-05 2024-09-21T10:08:21.860 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.860 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.860 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:21.860 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112309+0000", 2024-09-21T10:08:21.860 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1430000000000001e-05 2024-09-21T10:08:21.860 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.860 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.860 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:21.860 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.112316+0000", 2024-09-21T10:08:21.860 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4099999999999996e-06 2024-09-21T10:08:21.860 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.861 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.861 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.861 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.114729+0000", 2024-09-21T10:08:21.861 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0024129080000000001 2024-09-21T10:08:21.861 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.861 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.861 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:21.861 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.114803+0000", 2024-09-21T10:08:21.861 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.4084000000000004e-05 2024-09-21T10:08:21.861 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.861 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.861 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:08:21.862 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.114936+0000", 2024-09-21T10:08:21.862 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013269900000000001 2024-09-21T10:08:21.862 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.862 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.862 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.862 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.117014+0000", 2024-09-21T10:08:21.862 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002078608 2024-09-21T10:08:21.863 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.863 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.863 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:21.863 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.117082+0000", 2024-09-21T10:08:21.863 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7507000000000003e-05 2024-09-21T10:08:21.863 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.863 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.864 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.864 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.330702+0000", 2024-09-21T10:08:21.864 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21362015200000001 2024-09-21T10:08:21.864 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.864 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.864 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:21.864 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.330774+0000", 2024-09-21T10:08:21.864 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2231999999999996e-05 2024-09-21T10:08:21.864 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.864 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.864 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:21.864 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:05:49.330874+0000", 2024-09-21T10:08:21.864 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.9710000000000006e-05 2024-09-21T10:08:21.864 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.865 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.865 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.865 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.865 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.865 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95103 10.d 10:b3c31b59:::benchmark_data_smithi107_184135_object95102:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T10:08:21.865 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.391296+0000", 2024-09-21T10:08:21.865 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 135.39996539800001, 2024-09-21T10:08:21.865 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0224283670000001, 2024-09-21T10:08:21.865 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:21.865 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:08:21.865 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:21.865 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:21.865 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:21.865 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95103 2024-09-21T10:08:21.866 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.866 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:21.866 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.866 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:21.866 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.391296+0000", 2024-09-21T10:08:21.866 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.866 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.866 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.866 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:21.866 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.391296+0000", 2024-09-21T10:08:21.866 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.866 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.866 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.866 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:21.866 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.391295+0000", 2024-09-21T10:08:21.867 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:08:21.867 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.867 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.867 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:21.867 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.391325+0000", 2024-09-21T10:08:21.867 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0151e-05 2024-09-21T10:08:21.867 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.867 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.867 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:21.867 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.391336+0000", 2024-09-21T10:08:21.867 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0859000000000001e-05 2024-09-21T10:08:21.867 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.867 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.867 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:21.867 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.391342+0000", 2024-09-21T10:08:21.868 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7860000000000002e-06 2024-09-21T10:08:21.868 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.868 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.868 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.868 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.391596+0000", 2024-09-21T10:08:21.868 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00025450400000000001 2024-09-21T10:08:21.868 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.868 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.868 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:21.868 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.391763+0000", 2024-09-21T10:08:21.868 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00016683600000000001 2024-09-21T10:08:21.868 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.868 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.868 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:08:21.869 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.392522+0000", 2024-09-21T10:08:21.869 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00075902900000000004 2024-09-21T10:08:21.869 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.869 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.869 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.869 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407862+0000", 2024-09-21T10:08:21.869 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.015340354 2024-09-21T10:08:21.869 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.869 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.869 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:08:21.869 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407869+0000", 2024-09-21T10:08:21.869 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7900000000000002e-06 2024-09-21T10:08:21.869 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.870 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.870 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.870 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.413651+0000", 2024-09-21T10:08:21.870 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0057817849999999 2024-09-21T10:08:21.870 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.870 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.870 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:21.870 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.413725+0000", 2024-09-21T10:08:21.870 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3898000000000003e-05 2024-09-21T10:08:21.870 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.870 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.870 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.870 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95107 10.d 10:b5c30b02:::benchmark_data_smithi107_184135_object95106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-21T10:08:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.406752+0000", 2024-09-21T10:08:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 135.384510143, 2024-09-21T10:08:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0153879509999999, 2024-09-21T10:08:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95107 2024-09-21T10:08:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.871 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:21.872 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.406752+0000", 2024-09-21T10:08:21.872 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.872 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.872 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.872 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:21.872 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.406752+0000", 2024-09-21T10:08:21.872 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.872 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.872 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.872 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:21.872 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.406750+0000", 2024-09-21T10:08:21.872 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:08:21.872 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.872 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.873 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:21.873 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.406783+0000", 2024-09-21T10:08:21.873 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2200999999999999e-05 2024-09-21T10:08:21.873 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.873 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.873 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:21.873 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.406793+0000", 2024-09-21T10:08:21.873 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0349999999999999e-05 2024-09-21T10:08:21.873 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.873 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.873 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:21.873 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.406799+0000", 2024-09-21T10:08:21.873 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.784e-06 2024-09-21T10:08:21.873 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.408181+0000", 2024-09-21T10:08:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001381902 2024-09-21T10:08:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:08:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.408186+0000", 2024-09-21T10:08:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5890000000000002e-06 2024-09-21T10:08:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.413748+0000", 2024-09-21T10:08:21.874 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.005562045 2024-09-21T10:08:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.413853+0000", 2024-09-21T10:08:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000105066 2024-09-21T10:08:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 4", 2024-09-21T10:08:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.413969+0000", 2024-09-21T10:08:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000116049 2024-09-21T10:08:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.421082+0000", 2024-09-21T10:08:21.875 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0071127689999999997 2024-09-21T10:08:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.421997+0000", 2024-09-21T10:08:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00091438399999999997 2024-09-21T10:08:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.422058+0000", 2024-09-21T10:08:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1338000000000001e-05 2024-09-21T10:08:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:21.876 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.422140+0000", 2024-09-21T10:08:21.877 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.1667000000000003e-05 2024-09-21T10:08:21.877 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.877 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.877 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.877 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.877 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.877 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95110 10.b 10:de41cee3:::benchmark_data_smithi107_184135_object95109:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-21T10:08:21.877 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.407299+0000", 2024-09-21T10:08:21.877 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 135.38396296600001, 2024-09-21T10:08:21.877 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0202640590000001, 2024-09-21T10:08:21.877 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:21.877 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:21.877 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:21.877 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:21.878 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:21.878 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95110 2024-09-21T10:08:21.878 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.878 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:21.878 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.878 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:21.878 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407299+0000", 2024-09-21T10:08:21.878 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.878 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.878 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.878 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:21.878 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407299+0000", 2024-09-21T10:08:21.878 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.878 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.879 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.879 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:21.879 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407298+0000", 2024-09-21T10:08:21.879 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:08:21.879 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.879 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.879 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:21.879 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407389+0000", 2024-09-21T10:08:21.879 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.1442999999999998e-05 2024-09-21T10:08:21.879 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.879 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.879 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:21.879 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407400+0000", 2024-09-21T10:08:21.879 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0757999999999999e-05 2024-09-21T10:08:21.880 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.880 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.880 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:21.880 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407417+0000", 2024-09-21T10:08:21.880 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.7297999999999999e-05 2024-09-21T10:08:21.880 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.880 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.880 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.880 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407552+0000", 2024-09-21T10:08:21.880 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013406 2024-09-21T10:08:21.880 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.880 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.880 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:08:21.880 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407558+0000", 2024-09-21T10:08:21.881 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8229999999999997e-06 2024-09-21T10:08:21.881 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.881 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.881 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.881 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.422095+0000", 2024-09-21T10:08:21.881 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0145365420000001 2024-09-21T10:08:21.881 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.881 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.881 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:21.881 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.422194+0000", 2024-09-21T10:08:21.881 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.9501999999999997e-05 2024-09-21T10:08:21.881 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.881 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.881 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1", 2024-09-21T10:08:21.882 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.422335+0000", 2024-09-21T10:08:21.882 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014081 2024-09-21T10:08:21.882 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.882 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.882 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:21.882 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.426372+0000", 2024-09-21T10:08:21.882 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0040367700000000003 2024-09-21T10:08:21.882 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.882 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.882 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.882 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.427451+0000", 2024-09-21T10:08:21.882 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0010793350000000001 2024-09-21T10:08:21.882 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.882 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.883 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:21.883 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.427490+0000", 2024-09-21T10:08:21.883 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8217999999999999e-05 2024-09-21T10:08:21.883 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.883 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.883 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:21.883 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.427563+0000", 2024-09-21T10:08:21.883 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3359e-05 2024-09-21T10:08:21.883 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.883 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.883 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.883 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.883 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.883 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:177951 10.a 10:531f048b:::benchmark_data_smithi107_184135_object46385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-21T10:08:21.883 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:07:23.365499+0000", 2024-09-21T10:08:21.884 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 58.425763175999997, 2024-09-21T10:08:21.884 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.2598969740000001, 2024-09-21T10:08:21.884 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:21.884 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:21.884 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:21.884 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:21.884 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:21.884 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 177951 2024-09-21T10:08:21.884 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.884 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:21.884 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.884 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:21.885 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.365499+0000", 2024-09-21T10:08:21.885 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.885 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.885 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.885 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:21.885 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.365499+0000", 2024-09-21T10:08:21.885 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.885 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.885 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.885 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:21.885 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.365498+0000", 2024-09-21T10:08:21.885 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:21.885 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.885 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.885 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:21.886 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.365501+0000", 2024-09-21T10:08:21.886 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4450000000000001e-06 2024-09-21T10:08:21.886 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.886 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.886 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:21.886 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.365513+0000", 2024-09-21T10:08:21.886 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1524e-05 2024-09-21T10:08:21.886 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.886 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.886 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:21.886 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.365517+0000", 2024-09-21T10:08:21.886 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4240000000000003e-06 2024-09-21T10:08:21.886 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.886 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.887 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.887 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.365560+0000", 2024-09-21T10:08:21.887 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2314000000000002e-05 2024-09-21T10:08:21.887 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.887 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.887 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:08:21.887 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.365567+0000", 2024-09-21T10:08:21.887 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3270000000000003e-06 2024-09-21T10:08:21.887 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.887 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.887 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.887 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.339589+0000", 2024-09-21T10:08:21.887 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.97402238900000004 2024-09-21T10:08:21.887 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.888 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.888 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:08:21.888 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.339601+0000", 2024-09-21T10:08:21.888 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1168e-05 2024-09-21T10:08:21.888 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.888 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.888 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.888 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.592147+0000", 2024-09-21T10:08:21.888 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.252545923 2024-09-21T10:08:21.888 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.888 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.888 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:21.888 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.592213+0000", 2024-09-21T10:08:21.888 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6124999999999995e-05 2024-09-21T10:08:21.888 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.889 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.889 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:08:21.889 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.592733+0000", 2024-09-21T10:08:21.889 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00052057399999999999 2024-09-21T10:08:21.889 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.889 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.889 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.889 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.597899+0000", 2024-09-21T10:08:21.889 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0051660539999999998 2024-09-21T10:08:21.889 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.889 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.889 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.889 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.617608+0000", 2024-09-21T10:08:21.889 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.019708473000000001 2024-09-21T10:08:21.890 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.890 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.890 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:21.890 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.625211+0000", 2024-09-21T10:08:21.890 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0076037220000000003 2024-09-21T10:08:21.890 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.890 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.890 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:21.890 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.625305+0000", 2024-09-21T10:08:21.890 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.3041999999999995e-05 2024-09-21T10:08:21.890 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.890 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.890 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:21.890 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.625396+0000", 2024-09-21T10:08:21.891 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.1100000000000005e-05 2024-09-21T10:08:21.891 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.891 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.891 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.891 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.891 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.891 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:177963 10.0 10:0c263a19:::benchmark_data_smithi107_184135_object46397:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-21T10:08:21.891 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:07:23.374621+0000", 2024-09-21T10:08:21.891 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 58.416640809999997, 2024-09-21T10:08:21.891 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.081622039, 2024-09-21T10:08:21.891 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:21.891 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:21.891 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:21.891 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:21.892 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:21.892 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 177963 2024-09-21T10:08:21.892 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.892 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:21.892 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.892 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:21.892 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.374621+0000", 2024-09-21T10:08:21.892 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.892 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.892 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.892 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:21.892 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.374621+0000", 2024-09-21T10:08:21.892 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.892 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.892 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.893 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:21.893 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.374619+0000", 2024-09-21T10:08:21.893 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:08:21.893 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.893 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.893 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:21.893 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.374626+0000", 2024-09-21T10:08:21.893 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.1570000000000003e-06 2024-09-21T10:08:21.893 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.893 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.893 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:21.893 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.374643+0000", 2024-09-21T10:08:21.893 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.6968000000000001e-05 2024-09-21T10:08:21.893 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.893 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.894 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:21.894 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.374650+0000", 2024-09-21T10:08:21.894 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2960000000000004e-06 2024-09-21T10:08:21.894 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.894 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.894 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.894 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.374706+0000", 2024-09-21T10:08:21.894 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6403999999999997e-05 2024-09-21T10:08:21.894 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.894 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.894 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:08:21.894 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.374715+0000", 2024-09-21T10:08:21.894 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.7919999999999998e-06 2024-09-21T10:08:21.894 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.895 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.895 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.895 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.341881+0000", 2024-09-21T10:08:21.895 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.96716574099999997 2024-09-21T10:08:21.895 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.895 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.895 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:08:21.895 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.341892+0000", 2024-09-21T10:08:21.895 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.114e-05 2024-09-21T10:08:21.895 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.895 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.895 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.895 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.441908+0000", 2024-09-21T10:08:21.896 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1000164349999999 2024-09-21T10:08:21.896 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.896 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.896 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:21.896 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.442024+0000", 2024-09-21T10:08:21.896 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000116069 2024-09-21T10:08:21.896 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.896 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.896 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4,7", 2024-09-21T10:08:21.896 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.442520+0000", 2024-09-21T10:08:21.896 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00049540900000000004 2024-09-21T10:08:21.896 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.896 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.896 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.896 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.455974+0000", 2024-09-21T10:08:21.897 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.013454847000000001 2024-09-21T10:08:21.897 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.897 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.897 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.897 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.456054+0000", 2024-09-21T10:08:21.897 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9553000000000002e-05 2024-09-21T10:08:21.897 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.897 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.897 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:21.897 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.456064+0000", 2024-09-21T10:08:21.897 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.5210000000000008e-06 2024-09-21T10:08:21.897 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.897 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.897 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.898 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.456127+0000", 2024-09-21T10:08:21.898 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3332999999999998e-05 2024-09-21T10:08:21.898 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.898 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.898 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:21.898 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.456182+0000", 2024-09-21T10:08:21.898 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4796999999999997e-05 2024-09-21T10:08:21.898 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.898 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.898 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:21.898 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.456243+0000", 2024-09-21T10:08:21.898 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1357000000000002e-05 2024-09-21T10:08:21.898 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.898 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.899 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.899 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.899 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.899 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:177964 10.0 10:08fd14d6:::benchmark_data_smithi107_184135_object46398:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-21T10:08:21.899 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:07:23.377057+0000", 2024-09-21T10:08:21.899 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 58.414204818000002, 2024-09-21T10:08:21.899 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.0794880450000002, 2024-09-21T10:08:21.899 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:21.899 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:21.899 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:21.899 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:21.899 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:21.899 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 177964 2024-09-21T10:08:21.900 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.900 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:21.900 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.900 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:21.900 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.377057+0000", 2024-09-21T10:08:21.900 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.900 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.900 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.900 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:21.900 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.377057+0000", 2024-09-21T10:08:21.900 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.900 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.900 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.900 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:21.900 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.377056+0000", 2024-09-21T10:08:21.901 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:08:21.901 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.901 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.901 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:21.901 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.377062+0000", 2024-09-21T10:08:21.901 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1099999999999999e-06 2024-09-21T10:08:21.901 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.901 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.901 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:21.901 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.377076+0000", 2024-09-21T10:08:21.901 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4103e-05 2024-09-21T10:08:21.901 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.901 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.901 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:21.902 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.377082+0000", 2024-09-21T10:08:21.902 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7509999999999997e-06 2024-09-21T10:08:21.902 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.902 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.902 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.902 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.377142+0000", 2024-09-21T10:08:21.902 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0705999999999998e-05 2024-09-21T10:08:21.902 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.902 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.902 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:08:21.902 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.377153+0000", 2024-09-21T10:08:21.902 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0603e-05 2024-09-21T10:08:21.902 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.902 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.903 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.903 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.342088+0000", 2024-09-21T10:08:21.903 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.96493549099999998 2024-09-21T10:08:21.903 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.903 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.903 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:08:21.903 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.342097+0000", 2024-09-21T10:08:21.903 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.2179999999999999e-06 2024-09-21T10:08:21.903 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.903 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.903 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.903 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.447352+0000", 2024-09-21T10:08:21.903 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1052548609999999 2024-09-21T10:08:21.903 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.904 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.904 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:21.904 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.447448+0000", 2024-09-21T10:08:21.904 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.6930999999999997e-05 2024-09-21T10:08:21.904 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.904 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.904 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4,7", 2024-09-21T10:08:21.904 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.447530+0000", 2024-09-21T10:08:21.904 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.1079000000000001e-05 2024-09-21T10:08:21.904 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.904 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.904 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.904 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.456279+0000", 2024-09-21T10:08:21.904 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0087491770000000003 2024-09-21T10:08:21.905 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.905 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.905 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:21.905 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.456289+0000", 2024-09-21T10:08:21.905 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.8919999999999995e-06 2024-09-21T10:08:21.905 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.905 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.905 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.905 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.456415+0000", 2024-09-21T10:08:21.905 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012677599999999999 2024-09-21T10:08:21.905 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.905 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.905 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.905 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.456443+0000", 2024-09-21T10:08:21.906 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7211000000000001e-05 2024-09-21T10:08:21.906 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.906 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.906 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:21.906 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.456471+0000", 2024-09-21T10:08:21.906 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.8387000000000001e-05 2024-09-21T10:08:21.906 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.906 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.906 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:21.906 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.456545+0000", 2024-09-21T10:08:21.906 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.4062999999999999e-05 2024-09-21T10:08:21.906 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.906 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.906 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.907 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.907 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.907 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:177972 10.0 10:0acc188d:::benchmark_data_smithi107_184135_object46406:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-21T10:08:21.907 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:07:23.387867+0000", 2024-09-21T10:08:21.907 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 58.403395242999999, 2024-09-21T10:08:21.907 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.0793451429999998, 2024-09-21T10:08:21.907 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:21.907 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:21.907 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:21.907 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:21.907 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:21.907 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 177972 2024-09-21T10:08:21.907 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.907 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:21.908 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.908 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:21.908 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.387867+0000", 2024-09-21T10:08:21.908 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.908 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.908 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.908 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:21.908 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.387867+0000", 2024-09-21T10:08:21.908 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.908 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.908 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.908 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:21.908 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.387865+0000", 2024-09-21T10:08:21.908 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:08:21.908 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.909 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.909 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:21.909 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.387873+0000", 2024-09-21T10:08:21.909 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.4069999999999993e-06 2024-09-21T10:08:21.909 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.909 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.909 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:21.909 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.387888+0000", 2024-09-21T10:08:21.909 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4559000000000001e-05 2024-09-21T10:08:21.909 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.909 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.909 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:21.909 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.387895+0000", 2024-09-21T10:08:21.909 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7630000000000003e-06 2024-09-21T10:08:21.910 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.910 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.910 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.910 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.387937+0000", 2024-09-21T10:08:21.910 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2459999999999997e-05 2024-09-21T10:08:21.910 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.910 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.911 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:08:21.912 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.387944+0000", 2024-09-21T10:08:21.912 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2180000000000002e-06 2024-09-21T10:08:21.912 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.912 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.912 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.912 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.342167+0000", 2024-09-21T10:08:21.912 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.95422224499999997 2024-09-21T10:08:21.912 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.912 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.912 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:08:21.912 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.342180+0000", 2024-09-21T10:08:21.912 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3273e-05 2024-09-21T10:08:21.913 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.913 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.913 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.913 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.450301+0000", 2024-09-21T10:08:21.913 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1081206969999999 2024-09-21T10:08:21.913 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.913 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.913 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:21.913 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.450386+0000", 2024-09-21T10:08:21.913 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.5177000000000001e-05 2024-09-21T10:08:21.913 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.913 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.913 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4,7", 2024-09-21T10:08:21.913 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.450486+0000", 2024-09-21T10:08:21.914 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000100382 2024-09-21T10:08:21.914 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.914 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.914 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:21.914 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.456295+0000", 2024-09-21T10:08:21.914 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0058084779999999997 2024-09-21T10:08:21.914 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.914 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.914 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.914 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.456574+0000", 2024-09-21T10:08:21.914 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00027891800000000002 2024-09-21T10:08:21.914 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.914 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.914 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.915 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.456607+0000", 2024-09-21T10:08:21.915 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3692999999999997e-05 2024-09-21T10:08:21.915 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.915 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.915 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.915 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.467046+0000", 2024-09-21T10:08:21.915 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.010438439000000001 2024-09-21T10:08:21.915 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.915 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.915 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:21.915 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.467094+0000", 2024-09-21T10:08:21.915 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8727999999999998e-05 2024-09-21T10:08:21.915 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.915 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.916 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:21.916 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.467212+0000", 2024-09-21T10:08:21.916 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000117346 2024-09-21T10:08:21.916 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.916 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.916 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.916 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.916 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.916 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:177974 10.a 10:5d67d956:::benchmark_data_smithi107_184135_object46408:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-21T10:08:21.916 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:07:23.388668+0000", 2024-09-21T10:08:21.916 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 58.402593422999999, 2024-09-21T10:08:21.916 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.2368817050000001, 2024-09-21T10:08:21.916 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:21.916 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:21.916 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:21.917 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:21.917 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:21.917 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 177974 2024-09-21T10:08:21.917 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.917 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:21.917 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.917 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:21.917 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.388668+0000", 2024-09-21T10:08:21.917 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.917 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.917 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.917 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:21.917 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.388668+0000", 2024-09-21T10:08:21.917 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.918 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.918 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.918 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:21.918 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.388668+0000", 2024-09-21T10:08:21.918 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:08:21.918 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.918 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.918 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:21.918 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.388672+0000", 2024-09-21T10:08:21.918 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4599999999999996e-06 2024-09-21T10:08:21.918 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.918 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.918 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:21.919 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.388682+0000", 2024-09-21T10:08:21.919 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.9029999999999999e-06 2024-09-21T10:08:21.919 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.919 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.919 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:21.919 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.388686+0000", 2024-09-21T10:08:21.919 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.318e-06 2024-09-21T10:08:21.919 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.919 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.919 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.919 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.388734+0000", 2024-09-21T10:08:21.919 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7611000000000002e-05 2024-09-21T10:08:21.919 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.919 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.919 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:08:21.920 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.388743+0000", 2024-09-21T10:08:21.920 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.8859999999999993e-06 2024-09-21T10:08:21.920 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.920 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.920 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.920 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.339712+0000", 2024-09-21T10:08:21.920 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.95096870899999997 2024-09-21T10:08:21.920 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.920 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.920 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:08:21.920 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.339719+0000", 2024-09-21T10:08:21.920 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.6340000000000001e-06 2024-09-21T10:08:21.920 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.920 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.921 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.921 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.597928+0000", 2024-09-21T10:08:21.921 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2582087120000001 2024-09-21T10:08:21.921 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.921 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.921 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:21.921 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.598102+0000", 2024-09-21T10:08:21.921 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000173869 2024-09-21T10:08:21.921 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.921 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.921 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:08:21.921 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.598245+0000", 2024-09-21T10:08:21.921 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000143246 2024-09-21T10:08:21.921 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.921 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.922 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.922 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.608763+0000", 2024-09-21T10:08:21.922 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.010517535999999999 2024-09-21T10:08:21.922 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.922 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.922 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.922 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.617667+0000", 2024-09-21T10:08:21.922 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0089041299999999997 2024-09-21T10:08:21.922 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.922 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.922 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:21.922 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.625413+0000", 2024-09-21T10:08:21.922 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0077462060000000003 2024-09-21T10:08:21.922 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.923 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.923 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:21.923 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.625450+0000", 2024-09-21T10:08:21.923 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.752e-05 2024-09-21T10:08:21.923 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.923 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.923 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:21.923 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.625550+0000", 2024-09-21T10:08:21.923 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.9705999999999997e-05 2024-09-21T10:08:21.923 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.923 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.923 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.923 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.923 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.923 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:177981 10.0 10:03c424e3:::benchmark_data_smithi107_184135_object46415:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-21T10:08:21.924 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:07:23.395272+0000", 2024-09-21T10:08:21.924 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 58.395990089999998, 2024-09-21T10:08:21.924 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.072288854, 2024-09-21T10:08:21.924 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:21.924 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:21.924 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:21.924 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:21.924 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:21.924 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 177981 2024-09-21T10:08:21.924 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.924 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:21.924 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.924 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:21.924 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.395272+0000", 2024-09-21T10:08:21.924 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.925 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.925 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.925 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:21.925 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.395272+0000", 2024-09-21T10:08:21.925 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.925 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.925 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.925 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:21.925 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.395271+0000", 2024-09-21T10:08:21.925 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:08:21.925 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.925 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.925 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:21.926 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.395274+0000", 2024-09-21T10:08:21.926 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7300000000000001e-06 2024-09-21T10:08:21.926 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.926 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.926 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:21.926 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.395280+0000", 2024-09-21T10:08:21.926 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1149999999999999e-06 2024-09-21T10:08:21.926 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.926 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.926 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:21.926 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.395284+0000", 2024-09-21T10:08:21.926 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0239999999999999e-06 2024-09-21T10:08:21.926 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.926 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.927 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.927 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.395332+0000", 2024-09-21T10:08:21.927 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7630000000000003e-05 2024-09-21T10:08:21.927 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.927 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.927 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:08:21.927 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.395339+0000", 2024-09-21T10:08:21.927 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.5809999999999999e-06 2024-09-21T10:08:21.927 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.927 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.927 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.927 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.342211+0000", 2024-09-21T10:08:21.927 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.94687176399999995 2024-09-21T10:08:21.927 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.928 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.928 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:08:21.928 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.342216+0000", 2024-09-21T10:08:21.928 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4560000000000002e-06 2024-09-21T10:08:21.928 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.928 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.928 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.928 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.453011+0000", 2024-09-21T10:08:21.928 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1107954680000001 2024-09-21T10:08:21.928 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.928 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.928 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:21.928 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.453101+0000", 2024-09-21T10:08:21.928 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.0131999999999997e-05 2024-09-21T10:08:21.928 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.929 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.929 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4,7", 2024-09-21T10:08:21.929 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.453209+0000", 2024-09-21T10:08:21.929 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000107933 2024-09-21T10:08:21.929 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.929 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.929 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:21.929 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.456302+0000", 2024-09-21T10:08:21.929 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0030926059999999999 2024-09-21T10:08:21.929 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.929 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.929 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.929 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.456641+0000", 2024-09-21T10:08:21.929 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00033896300000000001 2024-09-21T10:08:21.930 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.930 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.930 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.930 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.466993+0000", 2024-09-21T10:08:21.930 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.010352755999999999 2024-09-21T10:08:21.930 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.930 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.930 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.930 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.467362+0000", 2024-09-21T10:08:21.930 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00036841800000000002 2024-09-21T10:08:21.930 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.930 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.930 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:21.930 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.467454+0000", 2024-09-21T10:08:21.931 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.2171999999999998e-05 2024-09-21T10:08:21.931 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.931 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.931 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:21.931 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.467561+0000", 2024-09-21T10:08:21.931 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000106606 2024-09-21T10:08:21.931 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.931 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.931 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.931 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.931 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.931 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:177987 10.0 10:02ebdac6:::benchmark_data_smithi107_184135_object46421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-09-21T10:08:21.931 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:07:23.400591+0000", 2024-09-21T10:08:21.931 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 58.390670350000001, 2024-09-21T10:08:21.932 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.0671607719999998, 2024-09-21T10:08:21.932 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:08:21.932 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:08:21.932 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:08:21.932 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:08:21.932 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:08:21.932 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 177987 2024-09-21T10:08:21.932 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.932 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:08:21.932 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.932 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:08:21.932 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.400591+0000", 2024-09-21T10:08:21.932 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.932 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.933 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.933 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:08:21.933 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.400591+0000", 2024-09-21T10:08:21.933 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:08:21.933 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.933 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.933 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:08:21.933 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.400590+0000", 2024-09-21T10:08:21.933 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:08:21.933 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.933 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.933 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:08:21.933 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.400595+0000", 2024-09-21T10:08:21.933 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8239999999999999e-06 2024-09-21T10:08:21.933 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.934 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.934 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:08:21.934 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.400604+0000", 2024-09-21T10:08:21.934 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.8230000000000001e-06 2024-09-21T10:08:21.934 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.934 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.934 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:08:21.934 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.400608+0000", 2024-09-21T10:08:21.934 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.9009999999999996e-06 2024-09-21T10:08:21.934 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.934 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.934 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.934 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.400656+0000", 2024-09-21T10:08:21.934 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7697000000000001e-05 2024-09-21T10:08:21.935 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.935 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.935 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:08:21.935 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:23.400662+0000", 2024-09-21T10:08:21.935 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4309999999999999e-06 2024-09-21T10:08:21.935 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.935 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.935 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.935 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.342357+0000", 2024-09-21T10:08:21.935 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.941694955 2024-09-21T10:08:21.935 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.935 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.935 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:08:21.935 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:24.342362+0000", 2024-09-21T10:08:21.936 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5639999999999996e-06 2024-09-21T10:08:21.936 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.936 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.936 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:08:21.936 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.456663+0000", 2024-09-21T10:08:21.936 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.114300377 2024-09-21T10:08:21.936 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.936 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.936 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:08:21.936 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.456791+0000", 2024-09-21T10:08:21.936 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000128532 2024-09-21T10:08:21.936 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.936 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.936 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4,7", 2024-09-21T10:08:21.936 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.456924+0000", 2024-09-21T10:08:21.937 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013217599999999999 2024-09-21T10:08:21.937 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.937 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.937 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.937 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.467269+0000", 2024-09-21T10:08:21.937 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.010345721 2024-09-21T10:08:21.937 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.937 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.937 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.937 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.467319+0000", 2024-09-21T10:08:21.937 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0197000000000001e-05 2024-09-21T10:08:21.937 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.937 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.937 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:08:21.938 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.467331+0000", 2024-09-21T10:08:21.938 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1494e-05 2024-09-21T10:08:21.938 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.938 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.938 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:08:21.938 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.467601+0000", 2024-09-21T10:08:21.938 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000270375 2024-09-21T10:08:21.938 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.938 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.938 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:08:21.938 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.467653+0000", 2024-09-21T10:08:21.938 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2129999999999997e-05 2024-09-21T10:08:21.938 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:08:21.938 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:08:21.939 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:08:21.939 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:07:25.467752+0000", 2024-09-21T10:08:21.939 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.8740999999999995e-05 2024-09-21T10:08:21.939 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.939 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.939 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.939 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:08:21.939 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:08:21.939 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:08:21.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:21.940 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:21.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:22.041 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:22.068 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:22.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:22.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:22.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:08:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:08:21] "GET /metrics HTTP/1.1" 200 36840 "" "Prometheus/2.51.0" 2024-09-21T10:08:22.170 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:22.209 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:22.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:22.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:22.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:22.344 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:22.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:22.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:22.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:22 smithi129 ceph-mon[88471]: pgmap v807: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 889 op/s; 17938/119934 objects degraded (14.957%); 4619/119934 objects misplaced (3.851%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T10:08:22.421 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:22 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:08:22.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:22 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:08:22.422 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:22 smithi129 ceph-mon[88471]: osdmap e223: 8 total, 7 up, 8 in 2024-09-21T10:08:22.446 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:22.487 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:22.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:22.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:22.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:22 smithi107 ceph-mon[102221]: pgmap v807: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 889 op/s; 17938/119934 objects degraded (14.957%); 4619/119934 objects misplaced (3.851%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T10:08:22.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:22 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:08:22.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:22 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:08:22.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:22 smithi107 ceph-mon[102221]: osdmap e223: 8 total, 7 up, 8 in 2024-09-21T10:08:22.530 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:22 smithi107 ceph-mon[105022]: pgmap v807: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 889 op/s; 17938/119934 objects degraded (14.957%); 4619/119934 objects misplaced (3.851%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T10:08:22.530 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:22 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:08:22.530 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:22 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:08:22.530 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:22 smithi107 ceph-mon[105022]: osdmap e223: 8 total, 7 up, 8 in 2024-09-21T10:08:22.589 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:22.621 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:22.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:22.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:22.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:22.727 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:08:22.750 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:22.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:22.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:22.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:22.877 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:22.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:22.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:22.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:23.004 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:23.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:23.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:23.105 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:23.175 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:23.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:23.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:23.176 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:08:23.277 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:23.279 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:08:23.304 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:23.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:23.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:23.406 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:23.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:23.454 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:23.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:23.555 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:23.599 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:23.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:23.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:23.701 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:23.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:23.742 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:23.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:23.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:23.872 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:23.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:23.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:23.974 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:24.009 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:24.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:24.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:24.111 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:24.154 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:24.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:24.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:24.256 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:24.279 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:08:24.289 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:24.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:24.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:24.391 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:24.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:24 smithi129 ceph-mon[88471]: pgmap v809: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 16302/110664 objects degraded (14.731%); 3601/110664 objects misplaced (3.254%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T10:08:24.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:24.424 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:24.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:24.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:24.577 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:24.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:24.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:24.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:24 smithi107 ceph-mon[102221]: pgmap v809: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 16302/110664 objects degraded (14.731%); 3601/110664 objects misplaced (3.254%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T10:08:24.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:24 smithi107 ceph-mon[105022]: pgmap v809: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 16302/110664 objects degraded (14.731%); 3601/110664 objects misplaced (3.254%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T10:08:24.652 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:24.664 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:24.679 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:24.718 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:24.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:24.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:24.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:24.855 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:24.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:24.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:24.958 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:24.994 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:24.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:24.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:25.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:25.127 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:25.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:25.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:25.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:25.256 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:25.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:25.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:25.298 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.1 primary-affinity to 0 (802) 2024-09-21T10:08:25.358 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:25.391 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:25.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:25.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:25.493 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:25.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:25.517 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:25.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:25.596 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:25 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3729195863' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.0} : dispatch 2024-09-21T10:08:25.596 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:25 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 16302/110664 objects degraded (14.731%), 10 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:25.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:25 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3729195863' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.0} : dispatch 2024-09-21T10:08:25.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:25 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 16302/110664 objects degraded (14.731%), 10 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:25.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:25 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3729195863' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.0} : dispatch 2024-09-21T10:08:25.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:25 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 16302/110664 objects degraded (14.731%), 10 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:25.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:25.640 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:25.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:25.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:25.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:25.763 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:25.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:25.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:25.864 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:25.891 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:25.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:25.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:25.993 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:08:25.993 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:08:25.993 INFO:teuthology.orchestra.run.smithi129.stdout: "num_ops": 0 2024-09-21T10:08:25.993 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:08:25.993 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:26.015 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:26.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:26.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:26.117 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:26.144 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:26.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:26.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:26.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:26.270 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:26.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:26.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:26.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:26.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:26.398 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:26.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:26.499 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:26.528 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:26.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:26.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:26.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:26 smithi107 ceph-mon[102221]: pgmap v810: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 920 op/s; 15506/106581 objects degraded (14.549%); 3601/106581 objects misplaced (3.379%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T10:08:26.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:26 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3729195863' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]': finished 2024-09-21T10:08:26.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:26 smithi107 ceph-mon[102221]: osdmap e224: 8 total, 7 up, 8 in 2024-09-21T10:08:26.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:08:26.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:26.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:26 smithi107 ceph-mon[105022]: pgmap v810: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 920 op/s; 15506/106581 objects degraded (14.549%); 3601/106581 objects misplaced (3.379%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T10:08:26.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:26 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3729195863' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]': finished 2024-09-21T10:08:26.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:26 smithi107 ceph-mon[105022]: osdmap e224: 8 total, 7 up, 8 in 2024-09-21T10:08:26.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:08:26.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:26.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:26.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:26.653 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:26.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:26 smithi129 ceph-mon[88471]: pgmap v810: 109 pgs: 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 79 active+clean; 2.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 920 op/s; 15506/106581 objects degraded (14.549%); 3601/106581 objects misplaced (3.379%); 5.6 MiB/s, 89 objects/s recovering 2024-09-21T10:08:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:26 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3729195863' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]': finished 2024-09-21T10:08:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:26 smithi129 ceph-mon[88471]: osdmap e224: 8 total, 7 up, 8 in 2024-09-21T10:08:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:08:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:26.754 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:26.779 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:26.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:26.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:26.828 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:08:26.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:26.902 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:26.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:26.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:27.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:27.025 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:27.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:27.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:27.127 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:27.147 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:08:27.154 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:27.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:27.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:27.256 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:27.281 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:27.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:27.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:27.383 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:27.412 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:27.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:27.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:27.514 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:27.551 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:27.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:27.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:27.597 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:27 smithi107 ceph-mon[102221]: osdmap e225: 8 total, 7 up, 8 in 2024-09-21T10:08:27.597 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:27 smithi107 ceph-mon[105022]: osdmap e225: 8 total, 7 up, 8 in 2024-09-21T10:08:27.610 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:27 smithi129 ceph-mon[88471]: osdmap e225: 8 total, 7 up, 8 in 2024-09-21T10:08:27.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:27.682 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:27.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:27.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:27.784 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:27.820 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:27.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:27.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:27.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:27.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:27.960 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:27.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:28.061 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:28.104 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:28.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:28.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:28.205 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:28.243 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:28.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:28.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:28.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:28.371 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:28.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:28.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:28.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:28 smithi129 ceph-mon[88471]: pgmap v813: 109 pgs: 2 peering, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 779 op/s; 15281/105906 objects degraded (14.429%); 3601/105906 objects misplaced (3.400%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:08:28.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:28 smithi129 ceph-mon[88471]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:08:28.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:28 smithi129 ceph-mon[88471]: osdmap e226: 8 total, 7 up, 8 in 2024-09-21T10:08:28.472 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:28.498 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:28.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:28.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:28.562 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:28 smithi107 ceph-mon[102221]: pgmap v813: 109 pgs: 2 peering, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 779 op/s; 15281/105906 objects degraded (14.429%); 3601/105906 objects misplaced (3.400%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:08:28.562 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:28 smithi107 ceph-mon[102221]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:08:28.562 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:28 smithi107 ceph-mon[102221]: osdmap e226: 8 total, 7 up, 8 in 2024-09-21T10:08:28.562 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:28 smithi107 ceph-mon[105022]: pgmap v813: 109 pgs: 2 peering, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 13 active+undersized, 9 active+undersized+degraded, 77 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 779 op/s; 15281/105906 objects degraded (14.429%); 3601/105906 objects misplaced (3.400%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:08:28.562 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:28 smithi107 ceph-mon[105022]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:08:28.562 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:28 smithi107 ceph-mon[105022]: osdmap e226: 8 total, 7 up, 8 in 2024-09-21T10:08:28.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:28.621 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:28.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:28.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:28.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:28.744 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:28.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:28.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:28.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:28.867 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:28.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:28.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:28.969 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:28.994 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:28.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:29.079 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:08:29.079 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:08:29.079 INFO:teuthology.orchestra.run.smithi129.stdout: "complaint_time": 30, 2024-09-21T10:08:29.079 INFO:teuthology.orchestra.run.smithi129.stdout: "num_blocked_ops": 0 2024-09-21T10:08:29.079 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:08:29.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:29.117 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:29.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:29.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:29.219 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:29.242 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:29.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:29.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:29.343 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:29.368 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:29.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:29.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:29.470 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:29.492 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:29.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:29.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:29.557 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:29 smithi107 ceph-mon[102221]: osdmap e227: 8 total, 7 up, 8 in 2024-09-21T10:08:29.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:29 smithi107 ceph-mon[105022]: osdmap e227: 8 total, 7 up, 8 in 2024-09-21T10:08:29.594 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:29.617 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:29.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:29.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:29 smithi129 ceph-mon[88471]: osdmap e227: 8 total, 7 up, 8 in 2024-09-21T10:08:29.719 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:29.742 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:29.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:29.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:29.806 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T10:08:29.821 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:08:29.822 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:08:29.838 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:08:29.838 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:08:29.844 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:29.854 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:08:29.854 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:08:29.872 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:29.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:29.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:29.875 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:08:29.875 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:08:29.897 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:08:29.897 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:08:29.957 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:08:29.958 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:08:29.974 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:29.980 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:08:29.980 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:08:30.007 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:30.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:30.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:30.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:30.130 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:30.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:30.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:30.232 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:30.253 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:30.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:30.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:30.355 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:30.393 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:30.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:30.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:30.494 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:30.515 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:30.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:30.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:30.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:30 smithi107 ceph-mon[102221]: pgmap v816: 109 pgs: 1 active+remapped+backfilling, 2 peering, 6 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 78 active+clean; 2.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 11 KiB/s rd, 25 KiB/s wr, 757 op/s; 14061/97566 objects degraded (14.412%); 1114/97566 objects misplaced (1.142%); 34 MiB/s, 551 objects/s recovering 2024-09-21T10:08:30.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:30 smithi107 ceph-mon[102221]: osdmap e228: 8 total, 7 up, 8 in 2024-09-21T10:08:30.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:30 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 14061/97566 objects degraded (14.412%), 9 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:30.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:30 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2608456758' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:30.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:30 smithi107 ceph-mon[105022]: pgmap v816: 109 pgs: 1 active+remapped+backfilling, 2 peering, 6 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 78 active+clean; 2.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 11 KiB/s rd, 25 KiB/s wr, 757 op/s; 14061/97566 objects degraded (14.412%); 1114/97566 objects misplaced (1.142%); 34 MiB/s, 551 objects/s recovering 2024-09-21T10:08:30.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:30 smithi107 ceph-mon[105022]: osdmap e228: 8 total, 7 up, 8 in 2024-09-21T10:08:30.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:30 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 14061/97566 objects degraded (14.412%), 9 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:30.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:30 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2608456758' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:30.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:30.647 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:30.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:30.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:30 smithi129 ceph-mon[88471]: pgmap v816: 109 pgs: 1 active+remapped+backfilling, 2 peering, 6 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 78 active+clean; 2.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 11 KiB/s rd, 25 KiB/s wr, 757 op/s; 14061/97566 objects degraded (14.412%); 1114/97566 objects misplaced (1.142%); 34 MiB/s, 551 objects/s recovering 2024-09-21T10:08:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:30 smithi129 ceph-mon[88471]: osdmap e228: 8 total, 7 up, 8 in 2024-09-21T10:08:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:30 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 14061/97566 objects degraded (14.412%), 9 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:30 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2608456758' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:08:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:08:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:08:30.749 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:30.782 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:30.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:30.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:30.884 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:30.917 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:30.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:30.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:31.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:31.043 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:31.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:31.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:31.145 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:31.180 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:31.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:31.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:31.282 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:31.318 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:31.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:31.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:31.417 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:08:31.417 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:08:31.419 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:31.457 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:31.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:31.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:31.559 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:31.582 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:31.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:31.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:31.600 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:08:31.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:31.706 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:31.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:31.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:31.808 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:31.830 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:31.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:31.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:31.882 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:08:31.932 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:31.954 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:31.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:31.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:32.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:32.077 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:32.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:32.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:08:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:08:31] "GET /metrics HTTP/1.1" 200 36839 "" "Prometheus/2.51.0" 2024-09-21T10:08:32.179 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:32.201 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:32.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:32.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:32.303 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:32.324 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:32.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:32.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:32.335 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:08:32.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:32.455 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:32.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:32.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:32.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:32.584 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:32.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:32.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:32.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:32 smithi107 ceph-mon[102221]: pgmap v818: 109 pgs: 1 active+remapped+backfilling, 6 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 80 active+clean; 2.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 14 KiB/s rd, 30 KiB/s wr, 620 op/s; 14061/97566 objects degraded (14.412%); 1114/97566 objects misplaced (1.142%); 28 MiB/s, 448 objects/s recovering 2024-09-21T10:08:32.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:32 smithi107 ceph-mon[102221]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T10:08:32.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:32 smithi107 ceph-mon[102221]: osdmap e229: 8 total, 7 up, 8 in 2024-09-21T10:08:32.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:32 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1101428040' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:08:32.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:32 smithi107 ceph-mon[105022]: pgmap v818: 109 pgs: 1 active+remapped+backfilling, 6 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 80 active+clean; 2.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 14 KiB/s rd, 30 KiB/s wr, 620 op/s; 14061/97566 objects degraded (14.412%); 1114/97566 objects misplaced (1.142%); 28 MiB/s, 448 objects/s recovering 2024-09-21T10:08:32.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:32 smithi107 ceph-mon[105022]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T10:08:32.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:32 smithi107 ceph-mon[105022]: osdmap e229: 8 total, 7 up, 8 in 2024-09-21T10:08:32.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:32 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1101428040' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:08:32.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:32 smithi129 ceph-mon[88471]: pgmap v818: 109 pgs: 1 active+remapped+backfilling, 6 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 80 active+clean; 2.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 14 KiB/s rd, 30 KiB/s wr, 620 op/s; 14061/97566 objects degraded (14.412%); 1114/97566 objects misplaced (1.142%); 28 MiB/s, 448 objects/s recovering 2024-09-21T10:08:32.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:32 smithi129 ceph-mon[88471]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T10:08:32.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:32 smithi129 ceph-mon[88471]: osdmap e229: 8 total, 7 up, 8 in 2024-09-21T10:08:32.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:32 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1101428040' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:08:32.686 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:32.718 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:32.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:32.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:32.820 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:32.857 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:32.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:32.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:32.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:32.991 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:32.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:32.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:33.022 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:08:33.023 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:08:33.023 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:08:33.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd out 4 2024-09-21T10:08:33.094 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:33.124 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:33.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:33.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:33.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:33.255 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:33.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:33.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:33.357 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:33.377 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:33.394 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:33.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:33.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:33.418 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:08:33.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:33.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:33.528 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:33.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:33.600 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:33 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:08:33.601 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:33 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1101428040' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:08:33.601 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:33 smithi129 ceph-mon[88471]: osdmap e230: 8 total, 7 up, 8 in 2024-09-21T10:08:33.601 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:33 smithi129 ceph-mon[88471]: 1.10 scrub starts 2024-09-21T10:08:33.601 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:33 smithi129 ceph-mon[88471]: 1.10 scrub ok 2024-09-21T10:08:33.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:33 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:08:33.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:33 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1101428040' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:08:33.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:33 smithi107 ceph-mon[102221]: osdmap e230: 8 total, 7 up, 8 in 2024-09-21T10:08:33.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:33 smithi107 ceph-mon[102221]: 1.10 scrub starts 2024-09-21T10:08:33.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:33 smithi107 ceph-mon[102221]: 1.10 scrub ok 2024-09-21T10:08:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:33 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:08:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:33 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1101428040' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:08:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:33 smithi107 ceph-mon[105022]: osdmap e230: 8 total, 7 up, 8 in 2024-09-21T10:08:33.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:33 smithi107 ceph-mon[105022]: 1.10 scrub starts 2024-09-21T10:08:33.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:33 smithi107 ceph-mon[105022]: 1.10 scrub ok 2024-09-21T10:08:33.630 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:33.665 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:33.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:33.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:33.757 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:08:33.757 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:08:33.766 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:33.805 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:33.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:33.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:33.907 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:33.940 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:33.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:33.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:34.042 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:34.077 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:34.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:34.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:34.179 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:34.209 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:34.209 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:34.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:34.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:34.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:34.349 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:34.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:34.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:34.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:34 smithi129 ceph-mon[88471]: 1.3 scrub starts 2024-09-21T10:08:34.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:34 smithi129 ceph-mon[88471]: 1.3 scrub ok 2024-09-21T10:08:34.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:34 smithi129 ceph-mon[88471]: pgmap v821: 109 pgs: 5 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 82 active+clean; 1.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 555 op/s; 13276/91719 objects degraded (14.475%); 14 MiB/s, 222 objects/s recovering 2024-09-21T10:08:34.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:34.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:08:34.481 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:34.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:34.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:34.583 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:34.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:34 smithi107 ceph-mon[102221]: 1.3 scrub starts 2024-09-21T10:08:34.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:34 smithi107 ceph-mon[102221]: 1.3 scrub ok 2024-09-21T10:08:34.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:34 smithi107 ceph-mon[102221]: pgmap v821: 109 pgs: 5 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 82 active+clean; 1.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 555 op/s; 13276/91719 objects degraded (14.475%); 14 MiB/s, 222 objects/s recovering 2024-09-21T10:08:34.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:34 smithi107 ceph-mon[105022]: 1.3 scrub starts 2024-09-21T10:08:34.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:34 smithi107 ceph-mon[105022]: 1.3 scrub ok 2024-09-21T10:08:34.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:34 smithi107 ceph-mon[105022]: pgmap v821: 109 pgs: 5 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 82 active+clean; 1.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 555 op/s; 13276/91719 objects degraded (14.475%); 14 MiB/s, 222 objects/s recovering 2024-09-21T10:08:34.617 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:34.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:34.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:34.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:34.750 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:34.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:34.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:34.853 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:34.888 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:34.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:34.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:34.990 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:34.997 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:08:35.027 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:35.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:35.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:35.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:35.162 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:35.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:35.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:35.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:35.290 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:35.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:35.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:35.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:35.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:35.436 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:35.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:35.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:35.571 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:35.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:35.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:35.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:35 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 13276/91719 objects degraded (14.475%), 9 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:35.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:35 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 13276/91719 objects degraded (14.475%), 9 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:35 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 13276/91719 objects degraded (14.475%), 9 pgs degraded, 16 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:35.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:35.705 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:35.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:35.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:35.806 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:35.842 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:35.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:35.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:35.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:35.993 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:35.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:35.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:36.094 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:36.128 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:36.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:36.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:36.230 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:36.266 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:36.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:36.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:36.368 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:36.408 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:36.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:36.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:36.510 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:36.552 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:36.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:36.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:36.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:36 smithi107 ceph-mon[102221]: pgmap v822: 109 pgs: 5 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 82 active+clean; 1.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 441 op/s; 12921/89613 objects degraded (14.419%); 11 MiB/s, 181 objects/s recovering 2024-09-21T10:08:36.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:36 smithi107 ceph-mon[105022]: pgmap v822: 109 pgs: 5 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 82 active+clean; 1.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 441 op/s; 12921/89613 objects degraded (14.419%); 11 MiB/s, 181 objects/s recovering 2024-09-21T10:08:36.654 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:36.658 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:36.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:36 smithi129 ceph-mon[88471]: pgmap v822: 109 pgs: 5 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 82 active+clean; 1.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 441 op/s; 12921/89613 objects degraded (14.419%); 11 MiB/s, 181 objects/s recovering 2024-09-21T10:08:36.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:36.700 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:36.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:36.801 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:36.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:36.839 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:36.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:36.940 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:36.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:36.978 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:36.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:37.079 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:37.118 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:37.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:37.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:37.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:37.236 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:36 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a[102198]: 2024-09-21T10:08:36.968+0000 7f3189124640 -1 mon.a@0(leader).osd e230 definitely_dead 0 2024-09-21T10:08:37.251 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:37.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:37.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:37.253 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:37.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:37.371 INFO:teuthology.orchestra.run.smithi107.stderr:marked out osd.4. 2024-09-21T10:08:37.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:37.392 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:37.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:37.493 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:37.494 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:37 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3426603424' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:08:37.494 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:37 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:08:37.494 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:37 smithi107 ceph-mon[102221]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T10:08:37.494 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:37 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3426603424' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:08:37.494 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:37 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:08:37.494 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:37 smithi107 ceph-mon[105022]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T10:08:37.515 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:37.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:37.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:37.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:37.638 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:37.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:37.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:37.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:37 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3426603424' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:08:37.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:37 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:08:37.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:37 smithi129 ceph-mon[88471]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T10:08:37.739 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:37.760 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:37.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:37.862 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:37.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:37.884 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:37.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:37.985 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:38.006 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:38.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:38.108 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:38.129 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:38.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:38.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:38.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:38.255 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:38.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:38.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:38.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:38.393 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:38.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:38.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:38.495 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:38.517 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:38.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:38.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:38 smithi107 ceph-mon[105022]: pgmap v823: 109 pgs: 5 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 82 active+clean; 1.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 371 op/s; 12640/88770 objects degraded (14.239%); 8.6 MiB/s, 137 objects/s recovering 2024-09-21T10:08:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:38 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T10:08:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:38 smithi107 ceph-mon[105022]: osdmap e231: 8 total, 7 up, 7 in 2024-09-21T10:08:38.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:38 smithi107 ceph-mon[102221]: pgmap v823: 109 pgs: 5 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 82 active+clean; 1.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 371 op/s; 12640/88770 objects degraded (14.239%); 8.6 MiB/s, 137 objects/s recovering 2024-09-21T10:08:38.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:38 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T10:08:38.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:38 smithi107 ceph-mon[102221]: osdmap e231: 8 total, 7 up, 7 in 2024-09-21T10:08:38.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:38.640 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:38.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:38.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:38.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:38 smithi129 ceph-mon[88471]: pgmap v823: 109 pgs: 5 active+clean+remapped, 13 active+undersized, 9 active+undersized+degraded, 82 active+clean; 1.8 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 371 op/s; 12640/88770 objects degraded (14.239%); 8.6 MiB/s, 137 objects/s recovering 2024-09-21T10:08:38.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:38 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T10:08:38.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:38 smithi129 ceph-mon[88471]: osdmap e231: 8 total, 7 up, 7 in 2024-09-21T10:08:38.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:38.764 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:38.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:38.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:38.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:38.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:38.889 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:38.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:38.990 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:39.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:39.012 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:39.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:39.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:39.135 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:39.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:39.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:39.237 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:39.258 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:39.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:39.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:39.360 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:39.392 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:39.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:39.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:39.494 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:39.516 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:39.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:39.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:39.618 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:39.639 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:39.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:39.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:39 smithi129 ceph-mon[88471]: osdmap e232: 8 total, 7 up, 7 in 2024-09-21T10:08:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:39 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:39 smithi129 ceph-mon[88471]: osdmap e233: 8 total, 7 up, 7 in 2024-09-21T10:08:39.741 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:39.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:39.791 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:39.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:39.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:39 smithi107 ceph-mon[105022]: osdmap e232: 8 total, 7 up, 7 in 2024-09-21T10:08:39.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:39 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:39.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:39 smithi107 ceph-mon[105022]: osdmap e233: 8 total, 7 up, 7 in 2024-09-21T10:08:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:39 smithi107 ceph-mon[102221]: osdmap e232: 8 total, 7 up, 7 in 2024-09-21T10:08:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:39 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:39 smithi107 ceph-mon[102221]: osdmap e233: 8 total, 7 up, 7 in 2024-09-21T10:08:39.892 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:39.924 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:39.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:39.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:40.026 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:40.066 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:40.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:40.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:40.168 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:40.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:40.214 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:40.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:40.315 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:40.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:40.349 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:40.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:40.351 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:08:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:08:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:08:40.451 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:40.491 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:40.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:40.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:40.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:40 smithi107 ceph-mon[102221]: pgmap v826: 109 pgs: 2 remapped+peering, 12 peering, 5 active+clean+remapped, 12 active+undersized, 8 active+undersized+degraded, 70 active+clean; 1.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 881 op/s; 10009/78918 objects degraded (12.683%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:08:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:40 smithi107 ceph-mon[102221]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 230'15046 2024-09-21T10:08:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:40 smithi107 ceph-mon[102221]: 10.d starting backfill to osd.2 from (0'0,0'0] MAX to 230'14830 2024-09-21T10:08:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:40 smithi107 ceph-mon[102221]: 10.4 starting backfill to osd.2 from (0'0,0'0] MAX to 230'15072 2024-09-21T10:08:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:40 smithi107 ceph-mon[102221]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 230'14575 2024-09-21T10:08:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:40 smithi107 ceph-mon[102221]: 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 230'14750 2024-09-21T10:08:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:40 smithi107 ceph-mon[102221]: 10.e starting backfill to osd.3 from (0'0,0'0] MAX to 230'15121 2024-09-21T10:08:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:40 smithi107 ceph-mon[102221]: 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 230'14712 2024-09-21T10:08:40.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:40 smithi107 ceph-mon[102221]: Health check failed: Reduced data availability: 5 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-21T10:08:40.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:40 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 10009/78918 objects degraded (12.683%), 8 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:40 smithi107 ceph-mon[105022]: pgmap v826: 109 pgs: 2 remapped+peering, 12 peering, 5 active+clean+remapped, 12 active+undersized, 8 active+undersized+degraded, 70 active+clean; 1.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 881 op/s; 10009/78918 objects degraded (12.683%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:08:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:40 smithi107 ceph-mon[105022]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 230'15046 2024-09-21T10:08:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:40 smithi107 ceph-mon[105022]: 10.d starting backfill to osd.2 from (0'0,0'0] MAX to 230'14830 2024-09-21T10:08:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:40 smithi107 ceph-mon[105022]: 10.4 starting backfill to osd.2 from (0'0,0'0] MAX to 230'15072 2024-09-21T10:08:40.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:40 smithi107 ceph-mon[105022]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 230'14575 2024-09-21T10:08:40.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:40 smithi107 ceph-mon[105022]: 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 230'14750 2024-09-21T10:08:40.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:40 smithi107 ceph-mon[105022]: 10.e starting backfill to osd.3 from (0'0,0'0] MAX to 230'15121 2024-09-21T10:08:40.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:40 smithi107 ceph-mon[105022]: 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 230'14712 2024-09-21T10:08:40.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:40 smithi107 ceph-mon[105022]: Health check failed: Reduced data availability: 5 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-21T10:08:40.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:40 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 10009/78918 objects degraded (12.683%), 8 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:40.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:40.624 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:40.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:40.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:40 smithi129 ceph-mon[88471]: pgmap v826: 109 pgs: 2 remapped+peering, 12 peering, 5 active+clean+remapped, 12 active+undersized, 8 active+undersized+degraded, 70 active+clean; 1.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 881 op/s; 10009/78918 objects degraded (12.683%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:08:40.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:40 smithi129 ceph-mon[88471]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 230'15046 2024-09-21T10:08:40.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:40 smithi129 ceph-mon[88471]: 10.d starting backfill to osd.2 from (0'0,0'0] MAX to 230'14830 2024-09-21T10:08:40.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:40 smithi129 ceph-mon[88471]: 10.4 starting backfill to osd.2 from (0'0,0'0] MAX to 230'15072 2024-09-21T10:08:40.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:40 smithi129 ceph-mon[88471]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 230'14575 2024-09-21T10:08:40.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:40 smithi129 ceph-mon[88471]: 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 230'14750 2024-09-21T10:08:40.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:40 smithi129 ceph-mon[88471]: 10.e starting backfill to osd.3 from (0'0,0'0] MAX to 230'15121 2024-09-21T10:08:40.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:40 smithi129 ceph-mon[88471]: 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 230'14712 2024-09-21T10:08:40.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:40 smithi129 ceph-mon[88471]: Health check failed: Reduced data availability: 5 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-21T10:08:40.673 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:40 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 10009/78918 objects degraded (12.683%), 8 pgs degraded, 15 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:40.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:40.750 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:40.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:40.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:40.852 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:40.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:40.906 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:40.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:41.006 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:41.027 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:41.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:41.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:41.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:41.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:41.151 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:41.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:41.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:41.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:41.300 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:41.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:41.401 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:41.435 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:41.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:41.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:41.537 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:41.568 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:41.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:41.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:41.620 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T10:08:41.652 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:08:41.653 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:08:41.669 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:41.683 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:08:41.684 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:08:41.713 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:08:41.714 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:08:41.734 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:41.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:41.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:41.752 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:08:41.752 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:08:41.797 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:08:41.797 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:08:41.835 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:41.836 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:08:41.836 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:08:41.854 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:08:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:08:41] "GET /metrics HTTP/1.1" 200 36802 "" "Prometheus/2.51.0" 2024-09-21T10:08:41.870 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:08:41.870 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:08:41.875 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:41.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:41.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:41.977 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:42.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:42.014 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:42.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:42.115 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:42.150 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:42.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:42.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:42.252 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:42.320 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:42.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:42.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:08:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:42 smithi129 ceph-mon[88471]: pgmap v828: 109 pgs: 2 remapped+peering, 1 active+undersized+degraded+remapped+backfilling, 12 peering, 5 active+clean+remapped, 12 active+undersized, 7 active+undersized+degraded, 70 active+clean; 1.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 651 op/s; 9561/77865 objects degraded (12.279%); 1505/77865 objects misplaced (1.933%); 1.1 MiB/s, 16 objects/s recovering 2024-09-21T10:08:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:42 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3846171552' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:42.423 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:42.461 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:42.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:42.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:42.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:42.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:08:42.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:42.603 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:42 smithi107 ceph-mon[102221]: pgmap v828: 109 pgs: 2 remapped+peering, 1 active+undersized+degraded+remapped+backfilling, 12 peering, 5 active+clean+remapped, 12 active+undersized, 7 active+undersized+degraded, 70 active+clean; 1.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 651 op/s; 9561/77865 objects degraded (12.279%); 1505/77865 objects misplaced (1.933%); 1.1 MiB/s, 16 objects/s recovering 2024-09-21T10:08:42.603 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:42 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3846171552' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:42.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:08:42.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:42.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:42 smithi107 ceph-mon[105022]: pgmap v828: 109 pgs: 2 remapped+peering, 1 active+undersized+degraded+remapped+backfilling, 12 peering, 5 active+clean+remapped, 12 active+undersized, 7 active+undersized+degraded, 70 active+clean; 1.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 651 op/s; 9561/77865 objects degraded (12.279%); 1505/77865 objects misplaced (1.933%); 1.1 MiB/s, 16 objects/s recovering 2024-09-21T10:08:42.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:42 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3846171552' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:08:42.606 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:42.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:42.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:42.708 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:42.748 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:42.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:42.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:42.850 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:42.881 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:42.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:42.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:42.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:43.025 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:43.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:43.127 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:43.159 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:43.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:43.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:43.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:43.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:43.300 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:43.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:43.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:43.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:43.438 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:43.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:43.540 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:43.574 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:43.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:43.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:43.661 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:08:43.662 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:08:43.676 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:43.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:43.719 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:43.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:43.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:43.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:43.857 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:43.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:43.933 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:08:43.933 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:08:43.933 INFO:tasks.thrashosds.thrasher:Reweighting osd 5 to 0.3507234027166677 2024-09-21T10:08:43.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 5 0.3507234027166677 2024-09-21T10:08:43.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:44.046 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:44.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:44.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:44.193 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:08:44.193 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:08:44.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.193 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.36210.0:242288 10.b e233/202 10:dfbf6e92:::benchmark_data_smithi107_184135_object110722:head v 233'15251, mlcod=233'15249)", 2024-09-21T10:08:44.193 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:44.006694+0000", 2024-09-21T10:08:44.193 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.011051225, 2024-09-21T10:08:44.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011077475, 2024-09-21T10:08:44.193 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:08:44.193 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:08:44.193 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:08:44.193 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:08:44.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:08:44.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.006694+0000", 2024-09-21T10:08:44.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:08:44.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:08:44.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.006694+0000", 2024-09-21T10:08:44.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:08:44.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:08:44.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.006694+0000", 2024-09-21T10:08:44.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.84e-07 2024-09-21T10:08:44.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:08:44.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.006703+0000", 2024-09-21T10:08:44.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.67e-06 2024-09-21T10:08:44.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:08:44.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.006709+0000", 2024-09-21T10:08:44.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7910000000000001e-06 2024-09-21T10:08:44.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:08:44.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.006713+0000", 2024-09-21T10:08:44.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1779999999999996e-06 2024-09-21T10:08:44.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:08:44.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.008958+0000", 2024-09-21T10:08:44.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022456440000000002 2024-09-21T10:08:44.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:08:44.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.008984+0000", 2024-09-21T10:08:44.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5389e-05 2024-09-21T10:08:44.197 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:08:44.197 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:08:44.197 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:08:44.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.197 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.36210.0:242292 10.0 e233/227 10:000b5881:::benchmark_data_smithi107_184135_object110726:head v 233'15182, mlcod=233'15180)", 2024-09-21T10:08:44.197 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:44.013995+0000", 2024-09-21T10:08:44.197 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0037499460000000001, 2024-09-21T10:08:44.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038486789999999998, 2024-09-21T10:08:44.197 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:08:44.197 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:08:44.197 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:08:44.197 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:08:44.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.198 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:08:44.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.013995+0000", 2024-09-21T10:08:44.198 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:08:44.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.198 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.198 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:08:44.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.013995+0000", 2024-09-21T10:08:44.198 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:08:44.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.198 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.198 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:08:44.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.013996+0000", 2024-09-21T10:08:44.198 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0400000000000005e-07 2024-09-21T10:08:44.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:08:44.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.014001+0000", 2024-09-21T10:08:44.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4999999999999999e-06 2024-09-21T10:08:44.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:08:44.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.014012+0000", 2024-09-21T10:08:44.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0448000000000001e-05 2024-09-21T10:08:44.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:08:44.200 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.014019+0000", 2024-09-21T10:08:44.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0859999999999996e-06 2024-09-21T10:08:44.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:08:44.200 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.014082+0000", 2024-09-21T10:08:44.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2940000000000004e-05 2024-09-21T10:08:44.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:08:44.200 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.014102+0000", 2024-09-21T10:08:44.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9899999999999999e-05 2024-09-21T10:08:44.200 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:08:44.201 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:08:44.201 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:08:44.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.201 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.36210.0:242297 10.b e233/202 10:db2bcc68:::benchmark_data_smithi107_184135_object110731:head v 233'15252, mlcod=233'15249)", 2024-09-21T10:08:44.201 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:44.015717+0000", 2024-09-21T10:08:44.201 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0020281100000000001, 2024-09-21T10:08:44.201 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021768870000000002, 2024-09-21T10:08:44.201 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:08:44.201 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:08:44.201 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:08:44.201 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:08:44.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.202 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:08:44.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.015717+0000", 2024-09-21T10:08:44.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:08:44.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.202 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:08:44.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.015717+0000", 2024-09-21T10:08:44.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:08:44.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.202 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:08:44.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.015718+0000", 2024-09-21T10:08:44.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6899999999999996e-07 2024-09-21T10:08:44.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.203 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:08:44.203 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.015723+0000", 2024-09-21T10:08:44.203 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9989999999999999e-06 2024-09-21T10:08:44.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.203 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:08:44.203 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.015734+0000", 2024-09-21T10:08:44.203 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0803e-05 2024-09-21T10:08:44.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.203 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:08:44.203 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.015740+0000", 2024-09-21T10:08:44.203 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8619999999999997e-06 2024-09-21T10:08:44.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:08:44.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.017004+0000", 2024-09-21T10:08:44.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001263611 2024-09-21T10:08:44.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:08:44.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:08:44.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:08:44.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:44.017022+0000", 2024-09-21T10:08:44.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8512000000000002e-05 2024-09-21T10:08:44.204 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:08:44.204 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:08:44.204 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:08:44.204 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:08:44.204 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-09-21T10:08:44.205 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 3 2024-09-21T10:08:44.205 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:08:44.210 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:44.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:44.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:44.312 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:44.363 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:44.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:44.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:44.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:44.503 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:44.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:44.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:44.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:44 smithi107 ceph-mon[102221]: pgmap v829: 109 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 12 active+clean+remapped, 1 active+remapped+backfill_wait, 11 active+undersized, 5 active+undersized+degraded, 73 active+clean; 1.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 5.7 KiB/s rd, 12 KiB/s wr, 1.01k op/s; 9827/70593 objects degraded (13.921%); 8507/70593 objects misplaced (12.051%); 7.2 MiB/s, 1 keys/s, 116 objects/s recovering 2024-09-21T10:08:44.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:44 smithi107 ceph-mon[105022]: pgmap v829: 109 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 12 active+clean+remapped, 1 active+remapped+backfill_wait, 11 active+undersized, 5 active+undersized+degraded, 73 active+clean; 1.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 5.7 KiB/s rd, 12 KiB/s wr, 1.01k op/s; 9827/70593 objects degraded (13.921%); 8507/70593 objects misplaced (12.051%); 7.2 MiB/s, 1 keys/s, 116 objects/s recovering 2024-09-21T10:08:44.645 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:44.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:44.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:44 smithi129 ceph-mon[88471]: pgmap v829: 109 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 2 remapped+peering, 2 active+undersized+degraded+remapped+backfilling, 12 active+clean+remapped, 1 active+remapped+backfill_wait, 11 active+undersized, 5 active+undersized+degraded, 73 active+clean; 1.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 5.7 KiB/s rd, 12 KiB/s wr, 1.01k op/s; 9827/70593 objects degraded (13.921%); 8507/70593 objects misplaced (12.051%); 7.2 MiB/s, 1 keys/s, 116 objects/s recovering 2024-09-21T10:08:44.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:44.778 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:44.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:44.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:44.880 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:44.917 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:44.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:44.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:45.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:45.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:45.057 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:45.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:45.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:45.189 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:45.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:45.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:45.224 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:08:45.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:45.365 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:08:45.377 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:45.385 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:45.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:45.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:45.490 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:45.527 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:45.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:45.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:45.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:45 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:08:45.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:45 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:08:45.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:45 smithi107 ceph-mon[102221]: Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:08:45.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:45 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 9827/70593 objects degraded (13.921%), 9 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:45.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:45 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3918619893' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:45.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:45 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:45.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:45 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:08:45.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:45 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:08:45.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:45 smithi107 ceph-mon[105022]: Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:08:45.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:45 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 9827/70593 objects degraded (13.921%), 9 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:45.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:45 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3918619893' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:45.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:45 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:45.628 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:45.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:08:45.664 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:45.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:45.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:45 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:08:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:45 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:08:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:45 smithi129 ceph-mon[88471]: Health check update: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:08:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:45 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 9827/70593 objects degraded (13.921%), 9 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:45 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3918619893' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:45 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:08:45.767 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:45.812 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:45.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:45.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:45.914 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:45.952 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:45.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:45.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:46.054 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:46.084 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:46.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:46.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:46.185 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:46.212 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:46.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:46.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:46.314 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:46.351 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:46.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:46.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:46.452 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:46.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:46 smithi107 ceph-mon[102221]: pgmap v830: 109 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 13 active+clean+remapped, 2 active+remapped+backfill_wait, 11 active+undersized, 5 active+undersized+degraded, 73 active+clean; 1.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 4.3 KiB/s rd, 9.4 KiB/s wr, 967 op/s; 8346/66027 objects degraded (12.640%); 9534/66027 objects misplaced (14.440%); 12 MiB/s, 1 keys/s, 190 objects/s recovering 2024-09-21T10:08:46.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:46 smithi107 ceph-mon[102221]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:08:46.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:46 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:08:46.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:46 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:08:46.478 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:46 smithi107 ceph-mon[102221]: osdmap e234: 8 total, 7 up, 7 in 2024-09-21T10:08:46.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:46 smithi107 ceph-mon[105022]: pgmap v830: 109 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 13 active+clean+remapped, 2 active+remapped+backfill_wait, 11 active+undersized, 5 active+undersized+degraded, 73 active+clean; 1.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 4.3 KiB/s rd, 9.4 KiB/s wr, 967 op/s; 8346/66027 objects degraded (12.640%); 9534/66027 objects misplaced (14.440%); 12 MiB/s, 1 keys/s, 190 objects/s recovering 2024-09-21T10:08:46.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:46 smithi107 ceph-mon[105022]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:08:46.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:46 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:08:46.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:46 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:08:46.479 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:46 smithi107 ceph-mon[105022]: osdmap e234: 8 total, 7 up, 7 in 2024-09-21T10:08:46.494 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:46.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:46.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:46.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:46.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:46.640 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:46.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:46.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:46 smithi129 ceph-mon[88471]: pgmap v830: 109 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 13 active+clean+remapped, 2 active+remapped+backfill_wait, 11 active+undersized, 5 active+undersized+degraded, 73 active+clean; 1.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 4.3 KiB/s rd, 9.4 KiB/s wr, 967 op/s; 8346/66027 objects degraded (12.640%); 9534/66027 objects misplaced (14.440%); 12 MiB/s, 1 keys/s, 190 objects/s recovering 2024-09-21T10:08:46.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:46 smithi129 ceph-mon[88471]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:08:46.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:46 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:08:46.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:46 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:08:46.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:46 smithi129 ceph-mon[88471]: osdmap e234: 8 total, 7 up, 7 in 2024-09-21T10:08:46.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:46.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:46.792 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:46.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:46.893 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:46.947 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:46.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:46.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:47.049 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:47.084 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:47.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:47.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:47.186 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:47.215 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:47.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:47.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:47.317 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:47.360 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:47.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:47.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:47.462 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:47.496 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:47.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:47.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:47.598 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:47.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:47.633 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:47.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:47.656 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:47.735 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:47.768 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:47.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:47.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:47.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:47.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:47.910 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:47.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:48.010 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:48.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:48.046 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:48.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:48.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:48.160 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:48.195 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:08:48.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:48.232 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:48.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:48.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:48.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:48 smithi107 ceph-mon[105022]: pgmap v832: 109 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 13 active+clean+remapped, 2 active+remapped+backfill_wait, 11 active+undersized, 5 active+undersized+degraded, 73 active+clean; 1.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 702 op/s; 7172/62055 objects degraded (11.557%); 7637/62055 objects misplaced (12.307%); 22 MiB/s, 1 keys/s, 360 objects/s recovering 2024-09-21T10:08:48.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:48 smithi107 ceph-mon[102221]: pgmap v832: 109 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 13 active+clean+remapped, 2 active+remapped+backfill_wait, 11 active+undersized, 5 active+undersized+degraded, 73 active+clean; 1.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 702 op/s; 7172/62055 objects degraded (11.557%); 7637/62055 objects misplaced (12.307%); 22 MiB/s, 1 keys/s, 360 objects/s recovering 2024-09-21T10:08:48.383 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:48.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:48.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:48.517 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:48.545 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:48.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:48.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:48.646 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:48.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:48 smithi129 ceph-mon[88471]: pgmap v832: 109 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 13 active+clean+remapped, 2 active+remapped+backfill_wait, 11 active+undersized, 5 active+undersized+degraded, 73 active+clean; 1.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 702 op/s; 7172/62055 objects degraded (11.557%); 7637/62055 objects misplaced (12.307%); 22 MiB/s, 1 keys/s, 360 objects/s recovering 2024-09-21T10:08:48.672 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:48.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:48.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:48.773 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:48.799 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:48.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:48.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:48.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:48.927 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:48.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:48.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:49.029 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:49.053 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:49.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:49.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:49.154 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:49.177 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:49.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:49.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:49.279 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:49.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:49.310 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:49.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:49.411 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:49.438 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:49.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:49.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:49.539 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:49.569 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:49.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:49.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:49 smithi107 ceph-mon[102221]: osdmap e235: 8 total, 7 up, 7 in 2024-09-21T10:08:49.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:49 smithi107 ceph-mon[105022]: osdmap e235: 8 total, 7 up, 7 in 2024-09-21T10:08:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:49 smithi129 ceph-mon[88471]: osdmap e235: 8 total, 7 up, 7 in 2024-09-21T10:08:49.671 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:49.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:49.695 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:49.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:49.796 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:49.819 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:49.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:49.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:49.921 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:49.944 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:49.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:49.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:50.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:50.070 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:50.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:50.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:50.172 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:50.194 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:50.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:50.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:50.296 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:50.321 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:50.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:50.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:50.359 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.5 to 0.350723 (59c9) 2024-09-21T10:08:50.422 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:50.448 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:50.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:50.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:50.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:50.575 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:50.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:50.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:50.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:50 smithi107 ceph-mon[102221]: pgmap v834: 109 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 13 active+clean+remapped, 2 active+remapped+backfill_wait, 11 active+undersized, 5 active+undersized+degraded, 73 active+clean; 1.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 5865/51531 objects degraded (11.381%); 6931/51531 objects misplaced (13.450%); 22 MiB/s, 1 keys/s, 353 objects/s recovering 2024-09-21T10:08:50.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:50 smithi107 ceph-mon[102221]: osdmap e236: 8 total, 7 up, 7 in 2024-09-21T10:08:50.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:50 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 5865/51531 objects degraded (11.381%), 9 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:50.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:50 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3824051968' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 0.3507234027166677} : dispatch 2024-09-21T10:08:50.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:50 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 0.3507234027166677} : dispatch 2024-09-21T10:08:50.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:50 smithi107 ceph-mon[105022]: pgmap v834: 109 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 13 active+clean+remapped, 2 active+remapped+backfill_wait, 11 active+undersized, 5 active+undersized+degraded, 73 active+clean; 1.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 5865/51531 objects degraded (11.381%); 6931/51531 objects misplaced (13.450%); 22 MiB/s, 1 keys/s, 353 objects/s recovering 2024-09-21T10:08:50.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:50 smithi107 ceph-mon[105022]: osdmap e236: 8 total, 7 up, 7 in 2024-09-21T10:08:50.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:50 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 5865/51531 objects degraded (11.381%), 9 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:50.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:50 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3824051968' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 0.3507234027166677} : dispatch 2024-09-21T10:08:50.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:50 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 0.3507234027166677} : dispatch 2024-09-21T10:08:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:50 smithi129 ceph-mon[88471]: pgmap v834: 109 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 13 active+clean+remapped, 2 active+remapped+backfill_wait, 11 active+undersized, 5 active+undersized+degraded, 73 active+clean; 1.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 5865/51531 objects degraded (11.381%); 6931/51531 objects misplaced (13.450%); 22 MiB/s, 1 keys/s, 353 objects/s recovering 2024-09-21T10:08:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:50 smithi129 ceph-mon[88471]: osdmap e236: 8 total, 7 up, 7 in 2024-09-21T10:08:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:50 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 5865/51531 objects degraded (11.381%), 9 pgs degraded, 13 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:50 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3824051968' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 0.3507234027166677} : dispatch 2024-09-21T10:08:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:50 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 0.3507234027166677} : dispatch 2024-09-21T10:08:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:08:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:08:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:08:50.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:50.701 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:50.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:50.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:50.803 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:50.825 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:50.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:50.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:50.927 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:50.949 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:50.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:50.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:51.051 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:51.072 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:51.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:51.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:51.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:51.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:51.196 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:51.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:51.297 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:51.320 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:51.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:51.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:51.422 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:51.447 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:51.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:51.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:51.549 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:51.570 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:51.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:51.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:51.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:51 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 0.3507234027166677}]': finished 2024-09-21T10:08:51.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:51 smithi129 ceph-mon[88471]: osdmap e237: 8 total, 7 up, 7 in 2024-09-21T10:08:51.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:51 smithi129 ceph-mon[88471]: osdmap e238: 8 total, 7 up, 7 in 2024-09-21T10:08:51.672 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:51.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:51.695 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:51.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:51.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:51 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 0.3507234027166677}]': finished 2024-09-21T10:08:51.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:51 smithi107 ceph-mon[102221]: osdmap e237: 8 total, 7 up, 7 in 2024-09-21T10:08:51.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:51 smithi107 ceph-mon[102221]: osdmap e238: 8 total, 7 up, 7 in 2024-09-21T10:08:51.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:51 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 0.3507234027166677}]': finished 2024-09-21T10:08:51.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:51 smithi107 ceph-mon[105022]: osdmap e237: 8 total, 7 up, 7 in 2024-09-21T10:08:51.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:51 smithi107 ceph-mon[105022]: osdmap e238: 8 total, 7 up, 7 in 2024-09-21T10:08:51.796 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:51.817 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:51.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:51.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:51.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:51.940 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:51.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:51.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:52.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:52.053 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:08:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:08:51] "GET /metrics HTTP/1.1" 200 36802 "" "Prometheus/2.51.0" 2024-09-21T10:08:52.063 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:52.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:52.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:52.164 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:52.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:52.187 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:52.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:52.288 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:52.309 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:52.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:52.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:52.315 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:08:52.315 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:08:52.315 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-09-21T10:08:52.315 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-09-21T10:08:52.316 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:08:52.411 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:52.432 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:52.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:52.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:52.534 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:52.559 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:52.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:52.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:52.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:52 smithi107 ceph-mon[102221]: pgmap v837: 109 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 13 active+clean+remapped, 2 active+remapped+backfill_wait, 11 active+undersized, 5 active+undersized+degraded, 73 active+clean; 1.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 825 op/s; 5865/51531 objects degraded (11.381%); 6931/51531 objects misplaced (13.450%); 15 MiB/s, 245 objects/s recovering 2024-09-21T10:08:52.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:52 smithi107 ceph-mon[102221]: osdmap e239: 8 total, 7 up, 7 in 2024-09-21T10:08:52.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:52 smithi107 ceph-mon[105022]: pgmap v837: 109 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 13 active+clean+remapped, 2 active+remapped+backfill_wait, 11 active+undersized, 5 active+undersized+degraded, 73 active+clean; 1.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 825 op/s; 5865/51531 objects degraded (11.381%); 6931/51531 objects misplaced (13.450%); 15 MiB/s, 245 objects/s recovering 2024-09-21T10:08:52.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:52 smithi107 ceph-mon[105022]: osdmap e239: 8 total, 7 up, 7 in 2024-09-21T10:08:52.661 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:52.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:52 smithi129 ceph-mon[88471]: pgmap v837: 109 pgs: 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfilling, 13 active+clean+remapped, 2 active+remapped+backfill_wait, 11 active+undersized, 5 active+undersized+degraded, 73 active+clean; 1.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 825 op/s; 5865/51531 objects degraded (11.381%); 6931/51531 objects misplaced (13.450%); 15 MiB/s, 245 objects/s recovering 2024-09-21T10:08:52.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:52 smithi129 ceph-mon[88471]: osdmap e239: 8 total, 7 up, 7 in 2024-09-21T10:08:52.687 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:52.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:52.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:52.789 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:52.812 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:52.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:52.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:52.914 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:52.936 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:52.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:52.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:53.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:53.065 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:53.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:53.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:53.166 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:53.188 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:53.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:53.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:53.289 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:53.311 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:53.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:53.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:53.412 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:53.434 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:53.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:53.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:53.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:53.563 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:53.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:53.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:53.665 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:53.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:53 smithi107 ceph-mon[102221]: 10.9 starting backfill to osd.3 from (0'0,0'0] MAX to 236'15394 2024-09-21T10:08:53.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:53 smithi107 ceph-mon[102221]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 236'15464 2024-09-21T10:08:53.666 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:53 smithi107 ceph-mon[105022]: 10.9 starting backfill to osd.3 from (0'0,0'0] MAX to 236'15394 2024-09-21T10:08:53.666 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:53 smithi107 ceph-mon[105022]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 236'15464 2024-09-21T10:08:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:53 smithi129 ceph-mon[88471]: 10.9 starting backfill to osd.3 from (0'0,0'0] MAX to 236'15394 2024-09-21T10:08:53.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:53 smithi129 ceph-mon[88471]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 236'15464 2024-09-21T10:08:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:53.692 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:53.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:53.793 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:53.815 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:53.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:53.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:53.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:53.955 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:53.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:53.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:54.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:54.092 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:54.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:54.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:54.194 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:54.215 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:54.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:54.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:54.317 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:54.355 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:54.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:54.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:54.457 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:54.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:54.487 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:54.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:54.588 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:54.609 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:54.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:54 smithi129 ceph-mon[88471]: pgmap v840: 109 pgs: 1 active+remapped+backfilling, 2 remapped+peering, 1 unknown, 6 peering, 17 active+clean+remapped, 1 active+remapped+backfill_wait, 9 active+undersized, 5 active+undersized+degraded, 67 active+clean; 990 MiB data, 10 GiB used, 615 GiB / 626 GiB avail; 17 KiB/s rd, 37 KiB/s wr, 340 op/s; 2981/47571 objects degraded (6.266%); 5736/47571 objects misplaced (12.058%); 62 MiB/s, 995 objects/s recovering 2024-09-21T10:08:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:54 smithi129 ceph-mon[88471]: Health check failed: Reduced data availability: 4 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:08:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:54 smithi129 ceph-mon[88471]: osdmap e240: 8 total, 7 up, 7 in 2024-09-21T10:08:54.711 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:54.740 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:54.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:54.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:54.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:08:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:54 smithi107 ceph-mon[102221]: pgmap v840: 109 pgs: 1 active+remapped+backfilling, 2 remapped+peering, 1 unknown, 6 peering, 17 active+clean+remapped, 1 active+remapped+backfill_wait, 9 active+undersized, 5 active+undersized+degraded, 67 active+clean; 990 MiB data, 10 GiB used, 615 GiB / 626 GiB avail; 17 KiB/s rd, 37 KiB/s wr, 340 op/s; 2981/47571 objects degraded (6.266%); 5736/47571 objects misplaced (12.058%); 62 MiB/s, 995 objects/s recovering 2024-09-21T10:08:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:54 smithi107 ceph-mon[102221]: Health check failed: Reduced data availability: 4 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:08:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:54 smithi107 ceph-mon[102221]: osdmap e240: 8 total, 7 up, 7 in 2024-09-21T10:08:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:54 smithi107 ceph-mon[105022]: pgmap v840: 109 pgs: 1 active+remapped+backfilling, 2 remapped+peering, 1 unknown, 6 peering, 17 active+clean+remapped, 1 active+remapped+backfill_wait, 9 active+undersized, 5 active+undersized+degraded, 67 active+clean; 990 MiB data, 10 GiB used, 615 GiB / 626 GiB avail; 17 KiB/s rd, 37 KiB/s wr, 340 op/s; 2981/47571 objects degraded (6.266%); 5736/47571 objects misplaced (12.058%); 62 MiB/s, 995 objects/s recovering 2024-09-21T10:08:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:54 smithi107 ceph-mon[105022]: Health check failed: Reduced data availability: 4 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:08:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:54 smithi107 ceph-mon[105022]: osdmap e240: 8 total, 7 up, 7 in 2024-09-21T10:08:54.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:54.877 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:54.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:08:54.978 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:55.000 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:55.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:55.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:55.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:55.133 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:55.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:55.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:55.234 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:55.267 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:55.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:55.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:55.369 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:55.408 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:55.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:55.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:55.509 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:55.535 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:55.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:55.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:55.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:55 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2981/47571 objects degraded (6.266%), 5 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:55.672 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:55.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:55.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:55.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:55.807 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:55.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:55.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:55 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2981/47571 objects degraded (6.266%), 5 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:55.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:55 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2981/47571 objects degraded (6.266%), 5 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:55.910 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:55.947 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:55.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:55.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:56.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:56.070 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:56.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:56.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:56.172 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:56.200 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:56.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:56.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:56.302 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:56.333 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:56.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:56.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:56.435 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:56.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:56.471 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:56.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:56.572 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:56.602 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:56.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:56.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:56 smithi129 ceph-mon[88471]: pgmap v842: 109 pgs: 2 active+remapped+backfilling, 2 remapped+peering, 1 unknown, 6 peering, 19 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 67 active+clean; 973 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 9.2 KiB/s rd, 20 KiB/s wr, 333 op/s; 2877/46755 objects degraded (6.153%); 5360/46755 objects misplaced (11.464%); 68 MiB/s, 1.09k objects/s recovering 2024-09-21T10:08:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:08:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:56.704 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:56.735 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:56.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:56.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:56.837 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:56 smithi107 ceph-mon[102221]: pgmap v842: 109 pgs: 2 active+remapped+backfilling, 2 remapped+peering, 1 unknown, 6 peering, 19 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 67 active+clean; 973 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 9.2 KiB/s rd, 20 KiB/s wr, 333 op/s; 2877/46755 objects degraded (6.153%); 5360/46755 objects misplaced (11.464%); 68 MiB/s, 1.09k objects/s recovering 2024-09-21T10:08:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:08:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:56 smithi107 ceph-mon[105022]: pgmap v842: 109 pgs: 2 active+remapped+backfilling, 2 remapped+peering, 1 unknown, 6 peering, 19 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 67 active+clean; 973 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 9.2 KiB/s rd, 20 KiB/s wr, 333 op/s; 2877/46755 objects degraded (6.153%); 5360/46755 objects misplaced (11.464%); 68 MiB/s, 1.09k objects/s recovering 2024-09-21T10:08:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:08:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:56.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:56.867 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:56.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:56.896 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:08:56.896 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:08:56.896 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-09-21T10:08:56.896 INFO:tasks.cephadm.osd.0:Restarting osd.0 (starting--it wasn't running)... 2024-09-21T10:08:56.896 DEBUG:teuthology.orchestra.run.smithi107:osd.0> sudo journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.0.service 2024-09-21T10:08:56.899 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl start ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.0 2024-09-21T10:08:56.968 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:56.994 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:56.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:56.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:57.096 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:57.125 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:57.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:57.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:57.227 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:57.259 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:57.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:57.359 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:08:56 smithi107 systemd[1]: Starting Ceph osd.0 for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:08:57.360 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:57.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:57.404 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:57.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:57.505 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:57.537 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:57.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:57.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:57.639 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:57.662 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:57.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:57.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:57.763 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:57.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:57.801 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:57.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:57 smithi107 ceph-mon[102221]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 3 pgs peering) 2024-09-21T10:08:57.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:57 smithi107 ceph-mon[105022]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 3 pgs peering) 2024-09-21T10:08:57.902 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:57.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:57 smithi129 ceph-mon[88471]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 3 pgs peering) 2024-09-21T10:08:57.932 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:57.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:57.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:58.034 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:58.070 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:58.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:58.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:58.172 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:58.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:58.202 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:58.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:58.303 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:58.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:58.329 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:58.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:58.429 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:58.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:58.463 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:58.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:58.564 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:58.589 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:58.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:58.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:58.690 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:58.719 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:58.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:58.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:58.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:58.845 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:58.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:58.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:58 smithi107 ceph-mon[102221]: pgmap v843: 109 pgs: 2 active+remapped+backfilling, 21 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 74 active+clean; 911 MiB data, 10 GiB used, 615 GiB / 626 GiB avail; 7.5 KiB/s rd, 16 KiB/s wr, 434 op/s; 2877/43800 objects degraded (6.568%); 4751/43800 objects misplaced (10.847%); 55 MiB/s, 881 objects/s recovering 2024-09-21T10:08:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:58 smithi107 ceph-mon[105022]: pgmap v843: 109 pgs: 2 active+remapped+backfilling, 21 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 74 active+clean; 911 MiB data, 10 GiB used, 615 GiB / 626 GiB avail; 7.5 KiB/s rd, 16 KiB/s wr, 434 op/s; 2877/43800 objects degraded (6.568%); 4751/43800 objects misplaced (10.847%); 55 MiB/s, 881 objects/s recovering 2024-09-21T10:08:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:58 smithi129 ceph-mon[88471]: pgmap v843: 109 pgs: 2 active+remapped+backfilling, 21 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 74 active+clean; 911 MiB data, 10 GiB used, 615 GiB / 626 GiB avail; 7.5 KiB/s rd, 16 KiB/s wr, 434 op/s; 2877/43800 objects degraded (6.568%); 4751/43800 objects misplaced (10.847%); 55 MiB/s, 881 objects/s recovering 2024-09-21T10:08:58.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:58.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:58.986 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:58.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:59.087 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:08:59.117 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:59.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:59.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:08:59.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:59.239 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:59.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:59.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:08:59.341 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:59.375 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:59.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:59.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:59.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:08:59.503 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:59.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:59.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:08:59.605 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:08:59.626 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:59.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:08:59.728 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:08:59.749 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:59.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:59.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:08:59.851 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:08:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:59 smithi107 ceph-mon[105022]: 1.5 scrub starts 2024-09-21T10:08:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:59 smithi107 ceph-mon[105022]: 1.5 scrub ok 2024-09-21T10:08:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:59 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2877/43800 objects degraded (6.568%), 5 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:08:59 smithi107 ceph-mon[105022]: osdmap e241: 8 total, 7 up, 7 in 2024-09-21T10:08:59.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:59 smithi107 ceph-mon[102221]: 1.5 scrub starts 2024-09-21T10:08:59.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:59 smithi107 ceph-mon[102221]: 1.5 scrub ok 2024-09-21T10:08:59.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:59 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2877/43800 objects degraded (6.568%), 5 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:59.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:08:59 smithi107 ceph-mon[102221]: osdmap e241: 8 total, 7 up, 7 in 2024-09-21T10:08:59.872 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:08:59.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:08:59.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:08:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:59 smithi129 ceph-mon[88471]: 1.5 scrub starts 2024-09-21T10:08:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:59 smithi129 ceph-mon[88471]: 1.5 scrub ok 2024-09-21T10:08:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:59 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2877/43800 objects degraded (6.568%), 5 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-21T10:08:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:08:59 smithi129 ceph-mon[88471]: osdmap e241: 8 total, 7 up, 7 in 2024-09-21T10:08:59.974 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:08:59.998 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:08:59.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:00.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:00.101 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:00.123 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:00.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:00.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:00.135 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:00.224 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:00.246 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:00.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:00.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:00.348 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:00.369 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:00.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:00.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:00.471 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:00.493 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:00.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:00.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:00.595 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:00.622 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:00.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:00.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:00.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:09:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:00 smithi129 ceph-mon[88471]: pgmap v844: 109 pgs: 2 active+remapped+backfilling, 21 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 74 active+clean; 701 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.4 KiB/s rd, 3.0 KiB/s wr, 682 op/s; 1950/33714 objects degraded (5.784%); 3060/33714 objects misplaced (9.076%); 37 MiB/s, 598 objects/s recovering 2024-09-21T10:09:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:00 smithi129 ceph-mon[88471]: osdmap e242: 8 total, 7 up, 7 in 2024-09-21T10:09:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:09:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:09:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:09:00.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:00.742 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:00 smithi107 ceph-mon[102221]: pgmap v844: 109 pgs: 2 active+remapped+backfilling, 21 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 74 active+clean; 701 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.4 KiB/s rd, 3.0 KiB/s wr, 682 op/s; 1950/33714 objects degraded (5.784%); 3060/33714 objects misplaced (9.076%); 37 MiB/s, 598 objects/s recovering 2024-09-21T10:09:00.742 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:00 smithi107 ceph-mon[102221]: osdmap e242: 8 total, 7 up, 7 in 2024-09-21T10:09:00.742 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:00 smithi107 ceph-mon[105022]: pgmap v844: 109 pgs: 2 active+remapped+backfilling, 21 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 74 active+clean; 701 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.4 KiB/s rd, 3.0 KiB/s wr, 682 op/s; 1950/33714 objects degraded (5.784%); 3060/33714 objects misplaced (9.076%); 37 MiB/s, 598 objects/s recovering 2024-09-21T10:09:00.743 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:00 smithi107 ceph-mon[105022]: osdmap e242: 8 total, 7 up, 7 in 2024-09-21T10:09:00.770 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:00.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:00.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:00.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:00.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:00.920 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:00.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:01.020 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:01.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:01.049 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:01.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:01.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:01.200 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:01.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:01.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:01.301 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:01.329 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:01.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:01.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:01.431 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:01.458 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:01.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:01.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:01.490 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:09:01.490 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:09:01.490 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:09:01.490 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:09:01.490 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:09:01.490 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T10:09:01.491 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:09:01.491 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:09:01.491 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:09:01.491 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:09:01.491 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:09:01.522 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:09:01.522 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:09:01.554 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:09:01.554 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:09:01.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:01.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:01.599 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:01.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:01.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:01.733 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:01.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:01.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:01.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:01 smithi107 ceph-mon[102221]: 1.12 scrub starts 2024-09-21T10:09:01.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:01 smithi107 ceph-mon[102221]: 1.12 scrub ok 2024-09-21T10:09:01.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:01 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/211127710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:01.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:01 smithi107 ceph-mon[105022]: 1.12 scrub starts 2024-09-21T10:09:01.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:01 smithi107 ceph-mon[105022]: 1.12 scrub ok 2024-09-21T10:09:01.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:01 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/211127710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:01.835 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:01.865 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:01.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:01.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:01.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:01 smithi129 ceph-mon[88471]: 1.12 scrub starts 2024-09-21T10:09:01.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:01 smithi129 ceph-mon[88471]: 1.12 scrub ok 2024-09-21T10:09:01.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:01 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/211127710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:01.966 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:01.997 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:01.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:01.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:02.077 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:09:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:09:01] "GET /metrics HTTP/1.1" 200 36857 "" "Prometheus/2.51.0" 2024-09-21T10:09:02.099 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:02.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:02.129 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:02.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:02.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:02.266 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:02.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:02.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:02.368 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:02.395 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:02.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:02.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:02.496 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:02.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:02.527 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:02.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:02.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:02.664 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:02.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:02.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:02.765 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:02.766 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:02 smithi107 ceph-mon[102221]: pgmap v847: 109 pgs: 2 active+remapped+backfilling, 21 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 74 active+clean; 701 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 588 op/s; 1950/33714 objects degraded (5.784%); 3060/33714 objects misplaced (9.076%); 20 MiB/s, 317 objects/s recovering 2024-09-21T10:09:02.766 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:02 smithi107 ceph-mon[105022]: pgmap v847: 109 pgs: 2 active+remapped+backfilling, 21 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 74 active+clean; 701 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 588 op/s; 1950/33714 objects degraded (5.784%); 3060/33714 objects misplaced (9.076%); 20 MiB/s, 317 objects/s recovering 2024-09-21T10:09:02.793 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:02.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:02.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:02.851 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:09:02.851 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:09:02.895 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:02.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:02 smithi129 ceph-mon[88471]: pgmap v847: 109 pgs: 2 active+remapped+backfilling, 21 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 74 active+clean; 701 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 588 op/s; 1950/33714 objects degraded (5.784%); 3060/33714 objects misplaced (9.076%); 20 MiB/s, 317 objects/s recovering 2024-09-21T10:09:02.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:02.931 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:02.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:03.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:03.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:03.068 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:03.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:03.109 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:02 smithi107 podman[211320]: 2024-09-21 10:09:02.67191548 +0000 UTC m=+1.277995188 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:09:03.109 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:02 smithi107 podman[211320]: 2024-09-21 10:09:02.976062771 +0000 UTC m=+1.582142481 container create 8ec53327dc257cd9d8ec5176249f47b6e13e1d1d991a0493270deaef03f968ad (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b) 2024-09-21T10:09:03.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:03.213 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:03.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:03.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:03.315 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:03.349 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:03.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:03.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:03.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:03.491 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:03.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:03.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:03.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:03.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:03.627 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:03.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:03.728 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:03.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:03.755 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:03.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:03.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:03.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:03.893 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:03.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:03.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:04.031 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:04.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:04.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:04.132 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:04.158 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:04.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:04.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:04.178 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:04 smithi107 podman[211320]: 2024-09-21 10:09:04.068241071 +0000 UTC m=+2.674320778 container init 8ec53327dc257cd9d8ec5176249f47b6e13e1d1d991a0493270deaef03f968ad (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-21T10:09:04.178 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:04 smithi107 podman[211320]: 2024-09-21 10:09:04.083958157 +0000 UTC m=+2.690037863 container start 8ec53327dc257cd9d8ec5176249f47b6e13e1d1d991a0493270deaef03f968ad (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-62da54b, io.buildah.version=1.37.2) 2024-09-21T10:09:04.178 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:04 smithi107 podman[211320]: 2024-09-21 10:09:04.176640272 +0000 UTC m=+2.782719992 container attach 8ec53327dc257cd9d8ec5176249f47b6e13e1d1d991a0493270deaef03f968ad (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-21T10:09:04.260 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:04.293 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:04.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:04.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:04.395 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:04.430 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:04.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:04.460 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:04 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate[211503]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:09:04.460 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:04 smithi107 bash[211320]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:09:04.460 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:04 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate[211503]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:09:04.460 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:04 smithi107 bash[211320]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:09:04.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:04.571 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:04.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:04.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:04.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:04.709 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:04.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:04.810 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:04.837 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:04.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:04.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:04.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:04 smithi107 ceph-mon[102221]: pgmap v848: 109 pgs: 1 active+remapped+backfilling, 21 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 75 active+clean; 500 MiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 943 op/s; 1654/24072 objects degraded (6.871%); 1832/24072 objects misplaced (7.611%); 9.8 MiB/s, 157 objects/s recovering 2024-09-21T10:09:04.843 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:04 smithi107 ceph-mon[102221]: 1.1 scrub starts 2024-09-21T10:09:04.843 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:04 smithi107 ceph-mon[102221]: 1.1 scrub ok 2024-09-21T10:09:04.843 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:04 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 1654/24072 objects degraded (6.871%), 5 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-21T10:09:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:04 smithi107 ceph-mon[105022]: pgmap v848: 109 pgs: 1 active+remapped+backfilling, 21 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 75 active+clean; 500 MiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 943 op/s; 1654/24072 objects degraded (6.871%); 1832/24072 objects misplaced (7.611%); 9.8 MiB/s, 157 objects/s recovering 2024-09-21T10:09:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:04 smithi107 ceph-mon[105022]: 1.1 scrub starts 2024-09-21T10:09:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:04 smithi107 ceph-mon[105022]: 1.1 scrub ok 2024-09-21T10:09:04.843 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:04 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 1654/24072 objects degraded (6.871%), 5 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-21T10:09:04.853 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:09:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:04 smithi129 ceph-mon[88471]: pgmap v848: 109 pgs: 1 active+remapped+backfilling, 21 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 75 active+clean; 500 MiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 943 op/s; 1654/24072 objects degraded (6.871%); 1832/24072 objects misplaced (7.611%); 9.8 MiB/s, 157 objects/s recovering 2024-09-21T10:09:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:04 smithi129 ceph-mon[88471]: 1.1 scrub starts 2024-09-21T10:09:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:04 smithi129 ceph-mon[88471]: 1.1 scrub ok 2024-09-21T10:09:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:04 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 1654/24072 objects degraded (6.871%), 5 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-21T10:09:04.938 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:04.961 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:04.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:04.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:05.063 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:05.092 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:05.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:05.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:05.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:05.235 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:05.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:05.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:05.337 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:05.358 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:05.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:05.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:05.460 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:05.493 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:05.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:05.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:05.595 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:05.620 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:05.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:05.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:05.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:05.775 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:05.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:05.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:05.857 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:05 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate[211503]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T10:09:05.857 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:05 smithi107 bash[211320]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T10:09:05.857 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:05 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate[211503]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-21T10:09:05.857 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:05 smithi107 bash[211320]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-21T10:09:05.877 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:05.918 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:05.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:05.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:06.020 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:06.051 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:06.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:06.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:06.109 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:05 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate[211503]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-21T10:09:06.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:05 smithi107 bash[211320]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-21T10:09:06.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:05 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate[211503]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-21T10:09:06.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:05 smithi107 bash[211320]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-21T10:09:06.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:05 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate[211503]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-21T10:09:06.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:05 smithi107 bash[211320]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-21T10:09:06.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:05 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate[211503]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T10:09:06.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:05 smithi107 bash[211320]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T10:09:06.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:05 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate[211503]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-21T10:09:06.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:05 smithi107 bash[211320]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-21T10:09:06.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:06 smithi107 podman[211320]: 2024-09-21 10:09:06.08567861 +0000 UTC m=+4.691758317 container died 8ec53327dc257cd9d8ec5176249f47b6e13e1d1d991a0493270deaef03f968ad (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, RELEASE=main-62da54b, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-21T10:09:06.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:06.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:06.196 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:06.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:06.297 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:06.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:06.340 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:06.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:06.441 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:06.474 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:06.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:06.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:06.513 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:09:06.576 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:06.609 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:06.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:06.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:06.710 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:06.748 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:06.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:06.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:06.850 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:06.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:06 smithi107 ceph-mon[102221]: pgmap v849: 109 pgs: 21 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 76 active+clean; 403 MiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 1018/19407 objects degraded (5.246%); 1820/19407 objects misplaced (9.378%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:09:06.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:06 smithi107 ceph-mon[102221]: 1.11 scrub starts 2024-09-21T10:09:06.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:06 smithi107 ceph-mon[102221]: 1.11 scrub ok 2024-09-21T10:09:06.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:06 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/977640767' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:09:06.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:06 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:09:06.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:06 smithi107 ceph-mon[105022]: pgmap v849: 109 pgs: 21 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 76 active+clean; 403 MiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 1018/19407 objects degraded (5.246%); 1820/19407 objects misplaced (9.378%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:09:06.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:06 smithi107 ceph-mon[105022]: 1.11 scrub starts 2024-09-21T10:09:06.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:06 smithi107 ceph-mon[105022]: 1.11 scrub ok 2024-09-21T10:09:06.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:06 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/977640767' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:09:06.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:06 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:09:06.879 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:06.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:06.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:06 smithi129 ceph-mon[88471]: pgmap v849: 109 pgs: 21 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 76 active+clean; 403 MiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s; 1018/19407 objects degraded (5.246%); 1820/19407 objects misplaced (9.378%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:09:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:06 smithi129 ceph-mon[88471]: 1.11 scrub starts 2024-09-21T10:09:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:06 smithi129 ceph-mon[88471]: 1.11 scrub ok 2024-09-21T10:09:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:06 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/977640767' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:09:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:06 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:09:06.981 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:07.016 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:07.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:07.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:07.118 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:07.144 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:06 smithi107 podman[212029]: 2024-09-21 10:09:06.883884598 +0000 UTC m=+0.921748302 container remove 8ec53327dc257cd9d8ec5176249f47b6e13e1d1d991a0493270deaef03f968ad (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-activate, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T10:09:07.146 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:07.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:07.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:07.248 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:07.277 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:07.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:07.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:07.378 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:07.406 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:07.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:07.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:07.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:07.556 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:07.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:07.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:07.658 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:07.687 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:07.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:07.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:07.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:07.829 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:07.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:07.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:07.836 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Removed 131563 objects 2024-09-21T10:09:07.836 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Clean up completed and total clean up time :147.167 2024-09-21T10:09:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:07 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:09:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:07 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:09:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:07 smithi107 ceph-mon[102221]: osdmap e243: 8 total, 7 up, 7 in 2024-09-21T10:09:07.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:07 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:09:07.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:07 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:09:07.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:07 smithi107 ceph-mon[105022]: osdmap e243: 8 total, 7 up, 7 in 2024-09-21T10:09:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:07 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:09:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:07 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:09:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:07 smithi129 ceph-mon[88471]: osdmap e243: 8 total, 7 up, 7 in 2024-09-21T10:09:07.931 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:07.953 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:07.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:07.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:08.054 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:08.077 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:08.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:08.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:08.080 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:08.123 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:08.179 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:08.203 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:08.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:08.305 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:08.328 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:08.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:08.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:08.429 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:08.453 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:08.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:08.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:08.555 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:08.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:08.580 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:08.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:08.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:08 smithi107 ceph-mon[102221]: pgmap v851: 109 pgs: 21 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 76 active+clean; 282 MiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 856 op/s; 1018/13596 objects degraded (7.487%); 705/13596 objects misplaced (5.185%); 730 KiB/s, 11 objects/s recovering 2024-09-21T10:09:08.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:08 smithi107 ceph-mon[105022]: pgmap v851: 109 pgs: 21 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 76 active+clean; 282 MiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 856 op/s; 1018/13596 objects degraded (7.487%); 705/13596 objects misplaced (5.185%); 730 KiB/s, 11 objects/s recovering 2024-09-21T10:09:08.682 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:08.705 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:08.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:08.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:08.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:08.829 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:08.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:08.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:08 smithi129 ceph-mon[88471]: pgmap v851: 109 pgs: 21 active+clean+remapped, 7 active+undersized, 5 active+undersized+degraded, 76 active+clean; 282 MiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 856 op/s; 1018/13596 objects degraded (7.487%); 705/13596 objects misplaced (5.185%); 730 KiB/s, 11 objects/s recovering 2024-09-21T10:09:08.930 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:08.954 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:08.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:08.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:09.056 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:09.080 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:09.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:09.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:09.182 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:09.206 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:09.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:09.308 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:09.331 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:09.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:09.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:09.433 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:09.456 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:09.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:09.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:09.557 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:09.580 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:09.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:09.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:09.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:09 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 1018/13596 objects degraded (7.487%), 5 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-21T10:09:09.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:09 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 1018/13596 objects degraded (7.487%), 5 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-21T10:09:09.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:09.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:09.706 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:09.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:09.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:09.830 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:09.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:09.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:09 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 1018/13596 objects degraded (7.487%), 5 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-21T10:09:09.931 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:09.957 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:09.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:09.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:10.059 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:10.084 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:10.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:10.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:09:10.186 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:10.208 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:10.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:10.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:10.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:10.333 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:10.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:10.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:10.342 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:10 smithi107 podman[212455]: 2024-09-21 10:09:10.137273681 +0000 UTC m=+0.021902043 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:09:10.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:10.458 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:10.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:10.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:10.517 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:09:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:09:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:09:10.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:10.580 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:10.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:10.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:10.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:10 smithi107 ceph-mon[105022]: pgmap v852: 109 pgs: 21 active+clean+remapped, 8 active+undersized, 4 active+undersized+degraded, 76 active+clean; 39 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.20k op/s; 8/1935 objects degraded (0.413%); 349/1935 objects misplaced (18.036%); 641 KiB/s, 10 objects/s recovering 2024-09-21T10:09:10.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:10 smithi107 ceph-mon[102221]: pgmap v852: 109 pgs: 21 active+clean+remapped, 8 active+undersized, 4 active+undersized+degraded, 76 active+clean; 39 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.20k op/s; 8/1935 objects degraded (0.413%); 349/1935 objects misplaced (18.036%); 641 KiB/s, 10 objects/s recovering 2024-09-21T10:09:10.683 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:10.704 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:10.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:10.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:10.780 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:10 smithi129 ceph-mon[88471]: pgmap v852: 109 pgs: 21 active+clean+remapped, 8 active+undersized, 4 active+undersized+degraded, 76 active+clean; 39 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.20k op/s; 8/1935 objects degraded (0.413%); 349/1935 objects misplaced (18.036%); 641 KiB/s, 10 objects/s recovering 2024-09-21T10:09:10.806 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:10.827 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:10.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:10.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:10.929 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:10.950 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:10.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:10.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:11.052 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:11.073 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:11.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:11.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:11.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:11.196 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:11.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:11.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:11.298 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:11.319 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:11.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:11.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:11.359 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:11 smithi107 podman[212455]: 2024-09-21 10:09:11.131891004 +0000 UTC m=+1.016519398 container create 621c0348f44478d35411c06235dd833382407e8c0e4767c3222485f1bed4f670 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:09:11.421 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:11.441 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:11.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:11.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:11.543 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:11.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:11.565 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:11.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:11.646 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:11 smithi107 podman[212455]: 2024-09-21 10:09:11.485406567 +0000 UTC m=+1.370034924 container init 621c0348f44478d35411c06235dd833382407e8c0e4767c3222485f1bed4f670 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b) 2024-09-21T10:09:11.646 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:11 smithi107 podman[212455]: 2024-09-21 10:09:11.49033588 +0000 UTC m=+1.374964233 container start 621c0348f44478d35411c06235dd833382407e8c0e4767c3222485f1bed4f670 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-21T10:09:11.646 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:11 smithi107 ceph-osd[212726]: -- 172.21.15.107:0/1381520788 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558c8bc3ba40 con 0x558c8bc19c00 2024-09-21T10:09:11.646 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:11 smithi107 bash[212455]: 621c0348f44478d35411c06235dd833382407e8c0e4767c3222485f1bed4f670 2024-09-21T10:09:11.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:09:11.666 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:11.687 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:11.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:11.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:11.789 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:11.811 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:11.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:11.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:11.896 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:09:11.897 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:09:11.897 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:11 smithi107 systemd[1]: Started Ceph osd.0 for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T10:09:11.897 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:09:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:09:11] "GET /metrics HTTP/1.1" 200 36828 "" "Prometheus/2.51.0" 2024-09-21T10:09:11.914 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:11.935 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:11.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:11.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:12.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:12.060 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:12.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:12.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:12.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:12.183 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:12.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:12.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:12.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:12.307 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:12.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:12.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:12.408 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:12 smithi129 ceph-mon[88471]: pgmap v853: 109 pgs: 21 active+clean+remapped, 8 active+undersized, 4 active+undersized+degraded, 76 active+clean; 39 MiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s; 8/1935 objects degraded (0.413%); 349/1935 objects misplaced (18.036%); 564 KiB/s, 8 objects/s recovering 2024-09-21T10:09:12.431 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:12.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:12.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:12.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:12.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:12.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:12 smithi107 ceph-mon[102221]: pgmap v853: 109 pgs: 21 active+clean+remapped, 8 active+undersized, 4 active+undersized+degraded, 76 active+clean; 39 MiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s; 8/1935 objects degraded (0.413%); 349/1935 objects misplaced (18.036%); 564 KiB/s, 8 objects/s recovering 2024-09-21T10:09:12.445 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:12.445 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:12.446 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:12 smithi107 ceph-mon[105022]: pgmap v853: 109 pgs: 21 active+clean+remapped, 8 active+undersized, 4 active+undersized+degraded, 76 active+clean; 39 MiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s; 8/1935 objects degraded (0.413%); 349/1935 objects misplaced (18.036%); 564 KiB/s, 8 objects/s recovering 2024-09-21T10:09:12.533 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:12.555 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:12.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:12.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:12.656 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:12.679 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:12.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:12.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:12.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:12.804 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:12.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:12.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:12.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:12.928 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:12.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:13.028 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:13.050 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:13.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:13.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:13.151 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:13.176 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:13.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:13.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:13.278 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:13.301 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:13.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:13.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:13.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:09:13.402 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:13.425 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:13.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:13.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:13.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:13.549 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:13.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:13.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:13.567 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:13 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0[212700]: 2024-09-21T10:09:13.527+0000 7f5982d70640 -1 Falling back to public interface 2024-09-21T10:09:13.608 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:09:13.608 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:09:13.608 INFO:teuthology.orchestra.run.smithi107.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T10:09:13.608 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:09:13.608 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:09:13.614 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:13.617 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:09:13.617 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:09:13.637 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:09:13.637 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:09:13.651 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:13.653 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-09-21T10:09:13.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-09-21T10:09:13.657 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:13.660 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:09:13.660 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:09:13.673 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:13.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:13.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:13.677 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:09:13.678 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:09:13.693 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:09:13.693 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:09:13.774 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:13.796 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:13.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:13.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:13.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:13.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:13.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:13.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:13.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:13.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:13 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3995773198' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:13.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:13.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:13.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:13.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:13.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:13.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:13 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3995773198' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:13.898 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:13.921 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:13.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:13.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:14.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:14.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:14.047 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:14.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:14.148 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:14.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:13 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:14.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:13 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:14.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:13 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:14.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:13 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:14.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:13 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:14.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:13 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3995773198' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:14.171 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:14.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:14.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:14.273 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:14.297 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:14.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:14.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:14.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:14.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:14.421 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:14.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:14.522 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:14.548 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:14.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:14.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:14.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:14.671 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:14.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:14.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:14.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:14.795 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:14.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:14.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:14.897 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:14.923 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:14.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:14.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:14.995 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:14 smithi107 ceph-mon[102221]: pgmap v854: 109 pgs: 21 active+clean+remapped, 9 active+undersized, 3 active+undersized+degraded, 76 active+clean; 652 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 797 op/s; 2/78 objects degraded (2.564%); 3/78 objects misplaced (3.846%); 61 KiB/s, 1 objects/s recovering 2024-09-21T10:09:14.996 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:14 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2/78 objects degraded (2.564%), 3 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-21T10:09:14.996 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:14 smithi107 ceph-mon[105022]: pgmap v854: 109 pgs: 21 active+clean+remapped, 9 active+undersized, 3 active+undersized+degraded, 76 active+clean; 652 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 797 op/s; 2/78 objects degraded (2.564%); 3/78 objects misplaced (3.846%); 61 KiB/s, 1 objects/s recovering 2024-09-21T10:09:14.996 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:14 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2/78 objects degraded (2.564%), 3 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-21T10:09:15.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:15.045 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:15.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:15.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:15.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:15.168 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:15.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:15.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:15.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:14 smithi129 ceph-mon[88471]: pgmap v854: 109 pgs: 21 active+clean+remapped, 9 active+undersized, 3 active+undersized+degraded, 76 active+clean; 652 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 797 op/s; 2/78 objects degraded (2.564%); 3/78 objects misplaced (3.846%); 61 KiB/s, 1 objects/s recovering 2024-09-21T10:09:15.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:14 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2/78 objects degraded (2.564%), 3 pgs degraded, 11 pgs undersized (PG_DEGRADED) 2024-09-21T10:09:15.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:15.292 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:15.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:15.393 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:15.419 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:15.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:15.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:15.521 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:15.545 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:15.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:15.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:15.669 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:15.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:15.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:15.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:15.793 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:15.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:15.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:15.895 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:15.921 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:15.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:15.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:16.023 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:16.048 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:16.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:16.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:16.150 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:16.174 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:16.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:16.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:16.275 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:16.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:16.301 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:16.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:16.370 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:09:16.370 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:09:16.402 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:16.426 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:16.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:16.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:16.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:16.551 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:16.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:16.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:16.653 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:16.680 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:16.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:16.781 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:16.806 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:16.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:16.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:16.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:16.930 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:16.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:17.032 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:17.058 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:17.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:17.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:17.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:16 smithi107 ceph-mon[102221]: pgmap v855: 109 pgs: 21 active+clean+remapped, 9 active+undersized, 3 active+undersized+degraded, 76 active+clean; 652 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 642 op/s; 2/78 objects degraded (2.564%); 3/78 objects misplaced (3.846%) 2024-09-21T10:09:17.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:16 smithi107 ceph-mon[105022]: pgmap v855: 109 pgs: 21 active+clean+remapped, 9 active+undersized, 3 active+undersized+degraded, 76 active+clean; 652 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 642 op/s; 2/78 objects degraded (2.564%); 3/78 objects misplaced (3.846%) 2024-09-21T10:09:17.160 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:16 smithi129 ceph-mon[88471]: pgmap v855: 109 pgs: 21 active+clean+remapped, 9 active+undersized, 3 active+undersized+degraded, 76 active+clean; 652 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 642 op/s; 2/78 objects degraded (2.564%); 3/78 objects misplaced (3.846%) 2024-09-21T10:09:17.185 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:17.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:17.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:17.287 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:17.312 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:17.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:17.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:17.414 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:17.440 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:17.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:17.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:17.541 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:17.567 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:17.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:17.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:17.668 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:17.692 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:17.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:17.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:17.794 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:17.815 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:17.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:17.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:17.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:17.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:17.939 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:17.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:18.040 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:18.066 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:18.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:18.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:18.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:18.190 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:18.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:18.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:18.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:18.312 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:18.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:18.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:18.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:09:18.414 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:18.440 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:18.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:18.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:18.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:18.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:18.567 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:18.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:18.659 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-09-21T10:09:18.660 INFO:teuthology.orchestra.run.smithi107.stderr:0 2024-09-21T10:09:18.660 INFO:teuthology.orchestra.run.smithi107.stderr:1 2024-09-21T10:09:18.660 INFO:teuthology.orchestra.run.smithi107.stderr:2 2024-09-21T10:09:18.660 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2024-09-21T10:09:18.660 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2024-09-21T10:09:18.660 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:09:18.660 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2024-09-21T10:09:18.660 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2024-09-21T10:09:18.660 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2024-09-21T10:09:18.660 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator fragmentation block 2024-09-21T10:09:18.660 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: invalid command 2024-09-21T10:09:18.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:18.691 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:18.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:18.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:18.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:18.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:18.816 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:18.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:18.917 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:18.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:18.946 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:18.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:19.032 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:18 smithi107 ceph-mon[102221]: pgmap v856: 109 pgs: 21 active+clean+remapped, 9 active+undersized, 3 active+undersized+degraded, 76 active+clean; 652 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 489 op/s; 2/78 objects degraded (2.564%); 3/78 objects misplaced (3.846%) 2024-09-21T10:09:19.032 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:18 smithi107 ceph-mon[105022]: pgmap v856: 109 pgs: 21 active+clean+remapped, 9 active+undersized, 3 active+undersized+degraded, 76 active+clean; 652 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 489 op/s; 2/78 objects degraded (2.564%); 3/78 objects misplaced (3.846%) 2024-09-21T10:09:19.047 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:19.072 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:19.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:19.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:19.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:18 smithi129 ceph-mon[88471]: pgmap v856: 109 pgs: 21 active+clean+remapped, 9 active+undersized, 3 active+undersized+degraded, 76 active+clean; 652 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 489 op/s; 2/78 objects degraded (2.564%); 3/78 objects misplaced (3.846%) 2024-09-21T10:09:19.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:19.194 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:19.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:19.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:19.296 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:19.317 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:19.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:19.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:19.419 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:19.444 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:19.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:19.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:19.448 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:19.448 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:19.547 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:19.571 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:19.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:19.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:19.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:19.694 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:19.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:19.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:19.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:19.817 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:19.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:19.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:19.918 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:19.944 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:19.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:19.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:20.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:20.068 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:20.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:20.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:20.170 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:20.198 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:20.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:20.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:20.300 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:20.326 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:20.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:20.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:20.428 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:20.453 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:20.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:20.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:20.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:20.576 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:20.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:20.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:09:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:09:20.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:09:20.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:20.705 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:20.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:20.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:20.806 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:20.832 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:20.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:20.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:20.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:20.955 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:20.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:20.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:21.058 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:21.084 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:21.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:21.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:20 smithi107 ceph-mon[102221]: pgmap v857: 109 pgs: 21 active+clean+remapped, 9 active+undersized, 3 active+undersized+degraded, 76 active+clean; 652 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 374 op/s; 2/78 objects degraded (2.564%); 3/78 objects misplaced (3.846%) 2024-09-21T10:09:21.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:20 smithi107 ceph-mon[105022]: pgmap v857: 109 pgs: 21 active+clean+remapped, 9 active+undersized, 3 active+undersized+degraded, 76 active+clean; 652 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 374 op/s; 2/78 objects degraded (2.564%); 3/78 objects misplaced (3.846%) 2024-09-21T10:09:21.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:20 smithi129 ceph-mon[88471]: pgmap v857: 109 pgs: 21 active+clean+remapped, 9 active+undersized, 3 active+undersized+degraded, 76 active+clean; 652 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 374 op/s; 2/78 objects degraded (2.564%); 3/78 objects misplaced (3.846%) 2024-09-21T10:09:21.185 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:21.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:21.211 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:21.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:21.312 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:21.335 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:21.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:21.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:21.437 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:21.460 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:21.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:21.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:21.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:21.584 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:21.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:21.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:21.686 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:21.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:21.710 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:21.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:21.717 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:09:21.718 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-21T10:09:21.799 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:09:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:09:21] "GET /metrics HTTP/1.1" 200 36828 "" "Prometheus/2.51.0" 2024-09-21T10:09:21.799 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0[212700]: 2024-09-21T10:09:21.528+0000 7f5982d70640 -1 osd.0 201 log_to_monitors true 2024-09-21T10:09:21.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:21.841 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:21.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:21.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:21.942 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:21.965 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:21.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:21.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:22.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:22.088 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:22.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:22.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:22.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:21 smithi107 ceph-mon[102221]: from='osd.0 [v2:172.21.15.107:6802/1818143286,v1:172.21.15.107:6803/1818143286]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T10:09:22.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:21 smithi107 ceph-mon[102221]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T10:09:22.110 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:09:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0[212700]: 2024-09-21T10:09:21.860+0000 7f5976b12640 -1 osd.0 244 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T10:09:22.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:21 smithi107 ceph-mon[105022]: from='osd.0 [v2:172.21.15.107:6802/1818143286,v1:172.21.15.107:6803/1818143286]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T10:09:22.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:21 smithi107 ceph-mon[105022]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T10:09:22.129 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:22.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:21 smithi129 ceph-mon[88471]: from='osd.0 [v2:172.21.15.107:6802/1818143286,v1:172.21.15.107:6803/1818143286]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T10:09:22.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:21 smithi129 ceph-mon[88471]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T10:09:22.190 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:22.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:22.215 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:22.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:22.316 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:22.342 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:22.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:22.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:22.444 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:22.469 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:22.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:22.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:22.571 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:22.594 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:22.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:22.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:22.696 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:22.719 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:22.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:22.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:22.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:22.854 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:22.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:22.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:22.956 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:22.979 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:22.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:22.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:23.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:23.102 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:23.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:23.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:23.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:22 smithi107 ceph-mon[105022]: pgmap v858: 109 pgs: 21 active+clean+remapped, 9 active+undersized, 3 active+undersized+degraded, 76 active+clean; 652 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 51 op/s; 2/78 objects degraded (2.564%); 3/78 objects misplaced (3.846%) 2024-09-21T10:09:23.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:22 smithi107 ceph-mon[105022]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T10:09:23.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:22 smithi107 ceph-mon[105022]: osdmap e244: 8 total, 7 up, 7 in 2024-09-21T10:09:23.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:22 smithi107 ceph-mon[105022]: from='osd.0 [v2:172.21.15.107:6802/1818143286,v1:172.21.15.107:6803/1818143286]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T10:09:23.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:22 smithi107 ceph-mon[105022]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T10:09:23.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:22 smithi107 ceph-mon[102221]: pgmap v858: 109 pgs: 21 active+clean+remapped, 9 active+undersized, 3 active+undersized+degraded, 76 active+clean; 652 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 51 op/s; 2/78 objects degraded (2.564%); 3/78 objects misplaced (3.846%) 2024-09-21T10:09:23.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:22 smithi107 ceph-mon[102221]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T10:09:23.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:22 smithi107 ceph-mon[102221]: osdmap e244: 8 total, 7 up, 7 in 2024-09-21T10:09:23.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:22 smithi107 ceph-mon[102221]: from='osd.0 [v2:172.21.15.107:6802/1818143286,v1:172.21.15.107:6803/1818143286]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T10:09:23.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:22 smithi107 ceph-mon[102221]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T10:09:23.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:22 smithi129 ceph-mon[88471]: pgmap v858: 109 pgs: 21 active+clean+remapped, 9 active+undersized, 3 active+undersized+degraded, 76 active+clean; 652 KiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 51 op/s; 2/78 objects degraded (2.564%); 3/78 objects misplaced (3.846%) 2024-09-21T10:09:23.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:22 smithi129 ceph-mon[88471]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T10:09:23.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:22 smithi129 ceph-mon[88471]: osdmap e244: 8 total, 7 up, 7 in 2024-09-21T10:09:23.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:22 smithi129 ceph-mon[88471]: from='osd.0 [v2:172.21.15.107:6802/1818143286,v1:172.21.15.107:6803/1818143286]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T10:09:23.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:22 smithi129 ceph-mon[88471]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-09-21T10:09:23.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:23.225 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:23.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:23.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:23.327 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:23.353 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:23.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:23.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:23.454 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:23.479 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:23.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:23.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:23.582 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:23.607 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:23.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:23.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:23.709 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:23.733 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:23.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:23.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:23.825 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:09:23.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:23.868 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:23.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:23.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:23.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:23.993 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:23.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:23.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:24.094 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:24.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:23 smithi107 ceph-mon[105022]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:09:24.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:23 smithi107 ceph-mon[105022]: osd.0 [v2:172.21.15.107:6802/1818143286,v1:172.21.15.107:6803/1818143286] boot 2024-09-21T10:09:24.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:23 smithi107 ceph-mon[105022]: osdmap e245: 8 total, 8 up, 7 in 2024-09-21T10:09:24.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:23 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T10:09:24.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:23 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1999671694' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:09:24.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:23 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:09:24.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:23 smithi107 ceph-mon[102221]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:09:24.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:23 smithi107 ceph-mon[102221]: osd.0 [v2:172.21.15.107:6802/1818143286,v1:172.21.15.107:6803/1818143286] boot 2024-09-21T10:09:24.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:23 smithi107 ceph-mon[102221]: osdmap e245: 8 total, 8 up, 7 in 2024-09-21T10:09:24.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:23 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T10:09:24.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:23 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1999671694' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:09:24.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:23 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:09:24.116 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:24.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:24.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:24.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:23 smithi129 ceph-mon[88471]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:09:24.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:23 smithi129 ceph-mon[88471]: osd.0 [v2:172.21.15.107:6802/1818143286,v1:172.21.15.107:6803/1818143286] boot 2024-09-21T10:09:24.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:23 smithi129 ceph-mon[88471]: osdmap e245: 8 total, 8 up, 7 in 2024-09-21T10:09:24.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:23 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T10:09:24.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:23 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1999671694' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:09:24.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:23 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:09:24.218 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:24.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:24.241 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:24.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:24.342 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:24.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:24.365 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:24.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:24.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:24.486 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:24.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:24.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:24.588 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:24.608 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:24.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:24.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:24.711 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:24.733 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:24.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:24.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:24.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:24.864 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:24.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:24.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:24.966 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:24.988 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:24.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:24.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:25.091 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:25.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:24 smithi107 ceph-mon[102221]: pgmap v861: 109 pgs: 2 remapped+peering, 19 active+clean+remapped, 9 active+undersized, 3 active+undersized+degraded, 76 active+clean; 652 KiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 2/78 objects degraded (2.564%); 3/78 objects misplaced (3.846%) 2024-09-21T10:09:25.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:24 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:09:25.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:24 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:09:25.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:24 smithi107 ceph-mon[102221]: 10.2 starting backfill to osd.0 from (0'0,201'6045] MAX to 243'16580 2024-09-21T10:09:25.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:24 smithi107 ceph-mon[102221]: 10.6 starting backfill to osd.0 from (0'0,201'5837] MAX to 243'16092 2024-09-21T10:09:25.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:24 smithi107 ceph-mon[102221]: osdmap e246: 8 total, 8 up, 7 in 2024-09-21T10:09:25.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:24 smithi107 ceph-mon[102221]: 10.1 starting backfill to osd.0 from (0'0,201'5900] MAX to 243'16288 2024-09-21T10:09:25.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:24 smithi107 ceph-mon[102221]: 10.c starting backfill to osd.0 from (0'0,201'5909] MAX to 243'16228 2024-09-21T10:09:25.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:24 smithi107 ceph-mon[105022]: pgmap v861: 109 pgs: 2 remapped+peering, 19 active+clean+remapped, 9 active+undersized, 3 active+undersized+degraded, 76 active+clean; 652 KiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 2/78 objects degraded (2.564%); 3/78 objects misplaced (3.846%) 2024-09-21T10:09:25.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:24 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:09:25.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:24 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:09:25.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:24 smithi107 ceph-mon[105022]: 10.2 starting backfill to osd.0 from (0'0,201'6045] MAX to 243'16580 2024-09-21T10:09:25.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:24 smithi107 ceph-mon[105022]: 10.6 starting backfill to osd.0 from (0'0,201'5837] MAX to 243'16092 2024-09-21T10:09:25.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:24 smithi107 ceph-mon[105022]: osdmap e246: 8 total, 8 up, 7 in 2024-09-21T10:09:25.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:24 smithi107 ceph-mon[105022]: 10.1 starting backfill to osd.0 from (0'0,201'5900] MAX to 243'16288 2024-09-21T10:09:25.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:24 smithi107 ceph-mon[105022]: 10.c starting backfill to osd.0 from (0'0,201'5909] MAX to 243'16228 2024-09-21T10:09:25.112 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:25.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:25.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:24 smithi129 ceph-mon[88471]: pgmap v861: 109 pgs: 2 remapped+peering, 19 active+clean+remapped, 9 active+undersized, 3 active+undersized+degraded, 76 active+clean; 652 KiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 2/78 objects degraded (2.564%); 3/78 objects misplaced (3.846%) 2024-09-21T10:09:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:24 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:09:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:24 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:09:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:24 smithi129 ceph-mon[88471]: 10.2 starting backfill to osd.0 from (0'0,201'6045] MAX to 243'16580 2024-09-21T10:09:25.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:24 smithi129 ceph-mon[88471]: 10.6 starting backfill to osd.0 from (0'0,201'5837] MAX to 243'16092 2024-09-21T10:09:25.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:24 smithi129 ceph-mon[88471]: osdmap e246: 8 total, 8 up, 7 in 2024-09-21T10:09:25.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:24 smithi129 ceph-mon[88471]: 10.1 starting backfill to osd.0 from (0'0,201'5900] MAX to 243'16288 2024-09-21T10:09:25.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:24 smithi129 ceph-mon[88471]: 10.c starting backfill to osd.0 from (0'0,201'5909] MAX to 243'16228 2024-09-21T10:09:25.214 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:25.234 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:25.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:25.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:25.336 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:25.359 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:25.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:25.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:25.461 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:25.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:25.484 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:25.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:25.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:25.606 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:25.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:25.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:25.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:25.729 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:25.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:25.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:25.830 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:25.852 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:25.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:25.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:25.954 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:25.975 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:25.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:25.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:26.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:25 smithi107 ceph-mon[102221]: osdmap e247: 8 total, 8 up, 7 in 2024-09-21T10:09:26.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:25 smithi107 ceph-mon[102221]: 10.b starting backfill to osd.0 from (0'0,201'6006] MAX to 243'16572 2024-09-21T10:09:26.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:25 smithi107 ceph-mon[102221]: 10.8 starting backfill to osd.0 from (0'0,201'5764] MAX to 243'15944 2024-09-21T10:09:26.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:25 smithi107 ceph-mon[102221]: 10.5 starting backfill to osd.0 from (0'0,201'5811] MAX to 243'16350 2024-09-21T10:09:26.036 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:25 smithi107 ceph-mon[105022]: osdmap e247: 8 total, 8 up, 7 in 2024-09-21T10:09:26.036 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:25 smithi107 ceph-mon[105022]: 10.b starting backfill to osd.0 from (0'0,201'6006] MAX to 243'16572 2024-09-21T10:09:26.036 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:25 smithi107 ceph-mon[105022]: 10.8 starting backfill to osd.0 from (0'0,201'5764] MAX to 243'15944 2024-09-21T10:09:26.036 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:25 smithi107 ceph-mon[105022]: 10.5 starting backfill to osd.0 from (0'0,201'5811] MAX to 243'16350 2024-09-21T10:09:26.076 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:26.097 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:26.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:26.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:25 smithi129 ceph-mon[88471]: osdmap e247: 8 total, 8 up, 7 in 2024-09-21T10:09:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:25 smithi129 ceph-mon[88471]: 10.b starting backfill to osd.0 from (0'0,201'6006] MAX to 243'16572 2024-09-21T10:09:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:25 smithi129 ceph-mon[88471]: 10.8 starting backfill to osd.0 from (0'0,201'5764] MAX to 243'15944 2024-09-21T10:09:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:25 smithi129 ceph-mon[88471]: 10.5 starting backfill to osd.0 from (0'0,201'5811] MAX to 243'16350 2024-09-21T10:09:26.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:26.220 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:26.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:26.322 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:26.343 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:26.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:26.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:26.444 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:26.466 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:26.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:26.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:26.568 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:26.590 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:26.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:26.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:26.691 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:26.712 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:26.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:26.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:26.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:09:26.814 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:26.835 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:26.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:26.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:26.937 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:26.958 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:26.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:26.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:27.060 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:27.081 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:27.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:27.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:27.098 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:26 smithi107 ceph-mon[102221]: pgmap v864: 109 pgs: 2 peering, 4 remapped+peering, 17 active+clean+remapped, 7 active+undersized, 2 active+undersized+degraded, 77 active+clean; 652 KiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 2/78 objects degraded (2.564%); 2/78 objects misplaced (2.564%) 2024-09-21T10:09:27.098 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:26 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2/78 objects degraded (2.564%), 2 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-21T10:09:27.098 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:26 smithi107 ceph-mon[102221]: osdmap e248: 8 total, 8 up, 7 in 2024-09-21T10:09:27.098 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:27.098 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:27.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:26 smithi107 ceph-mon[105022]: pgmap v864: 109 pgs: 2 peering, 4 remapped+peering, 17 active+clean+remapped, 7 active+undersized, 2 active+undersized+degraded, 77 active+clean; 652 KiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 2/78 objects degraded (2.564%); 2/78 objects misplaced (2.564%) 2024-09-21T10:09:27.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:26 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2/78 objects degraded (2.564%), 2 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-21T10:09:27.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:26 smithi107 ceph-mon[105022]: osdmap e248: 8 total, 8 up, 7 in 2024-09-21T10:09:27.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:27.098 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:26 smithi129 ceph-mon[88471]: pgmap v864: 109 pgs: 2 peering, 4 remapped+peering, 17 active+clean+remapped, 7 active+undersized, 2 active+undersized+degraded, 77 active+clean; 652 KiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 2/78 objects degraded (2.564%); 2/78 objects misplaced (2.564%) 2024-09-21T10:09:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:26 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2/78 objects degraded (2.564%), 2 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-09-21T10:09:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:26 smithi129 ceph-mon[88471]: osdmap e248: 8 total, 8 up, 7 in 2024-09-21T10:09:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:27.183 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:27.204 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:27.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:27.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:27.306 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:27.328 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:27.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:27.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:27.430 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:27.451 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:27.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:27.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:27.553 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:27.575 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:27.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:27.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:27.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:27.698 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:27.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:27.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:27.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:27.821 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:27.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:27.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:27.882 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:09:27.882 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:09:27.882 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-09-21T10:09:27.882 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:09:27.923 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:27.945 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:27.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:27.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:27.986 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:27 smithi107 ceph-mon[102221]: osdmap e249: 8 total, 8 up, 7 in 2024-09-21T10:09:27.986 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:27 smithi107 ceph-mon[105022]: osdmap e249: 8 total, 8 up, 7 in 2024-09-21T10:09:28.047 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:28.068 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:28.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:28.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:27 smithi129 ceph-mon[88471]: osdmap e249: 8 total, 8 up, 7 in 2024-09-21T10:09:28.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:28.191 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:28.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:28.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:28.293 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:28.314 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:28.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:28.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:28.416 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:28.437 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:28.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:28.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:28.539 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:28.561 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:28.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:28.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:28.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:28.684 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:28.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:28.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:28.786 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:28.807 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:28.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:28.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:28.845 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_5' removed 2024-09-21T10:09:28.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:28.944 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:28.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:28.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:29.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:29.068 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:29.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:29.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:29.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:28 smithi107 ceph-mon[102221]: pgmap v867: 109 pgs: 3 peering, 4 remapped+peering, 1 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+undersized, 97 active+clean; 652 KiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 22 B/s, 0 objects/s recovering 2024-09-21T10:09:29.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:28 smithi107 ceph-mon[102221]: osdmap e250: 8 total, 8 up, 7 in 2024-09-21T10:09:29.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:28 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3746059348' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:09:29.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:28 smithi107 ceph-mon[105022]: pgmap v867: 109 pgs: 3 peering, 4 remapped+peering, 1 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+undersized, 97 active+clean; 652 KiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 22 B/s, 0 objects/s recovering 2024-09-21T10:09:29.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:28 smithi107 ceph-mon[105022]: osdmap e250: 8 total, 8 up, 7 in 2024-09-21T10:09:29.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:28 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3746059348' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:09:29.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:28 smithi129 ceph-mon[88471]: pgmap v867: 109 pgs: 3 peering, 4 remapped+peering, 1 active+clean+remapped, 2 active+remapped+backfill_wait, 2 active+undersized, 97 active+clean; 652 KiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 22 B/s, 0 objects/s recovering 2024-09-21T10:09:29.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:28 smithi129 ceph-mon[88471]: osdmap e250: 8 total, 8 up, 7 in 2024-09-21T10:09:29.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:28 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3746059348' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:09:29.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:29.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:29.191 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:29.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:29.292 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:29.314 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:29.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:29.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:29.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:09:29.416 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:29.442 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:29.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:29.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:29.544 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:29.565 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:29.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:29.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:29.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:29.688 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:29.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:29.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:29.789 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:29.811 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:29.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:29.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:29.913 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:29.935 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:29.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:29.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:30.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:30.059 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:30.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:30.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:30.068 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:29 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3746059348' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:09:30.068 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:29 smithi107 ceph-mon[102221]: osdmap e251: 8 total, 8 up, 7 in 2024-09-21T10:09:30.069 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:29 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3746059348' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:09:30.069 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:29 smithi107 ceph-mon[105022]: osdmap e251: 8 total, 8 up, 7 in 2024-09-21T10:09:30.091 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:09:30.092 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:09:30.092 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_6 2024-09-21T10:09:30.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_6 16 2024-09-21T10:09:30.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:30.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:29 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3746059348' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:09:30.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:29 smithi129 ceph-mon[88471]: osdmap e251: 8 total, 8 up, 7 in 2024-09-21T10:09:30.182 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:30.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:30.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:30.277 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:09:30.278 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-21T10:09:30.284 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:30.286 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:30.288 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:09:30.288 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:09:30.306 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:09:30.306 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:09:30.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:30.307 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:30.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:30.325 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:30.326 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:09:30.326 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:09:30.343 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:09:30.344 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:09:30.359 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:09:30.359 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:09:30.374 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:09:30.375 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:09:30.390 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:09:30.391 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:09:30.408 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:30.429 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:30.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:30.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:30.531 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:30.552 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:30.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:30.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:30.654 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:09:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:09:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:09:30.676 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:30.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:30.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:30.714 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:09:30.777 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:30.799 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:30.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:30.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:30.902 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:30.923 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:30.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:30.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:31.025 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:31.046 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:31.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:31.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:31.148 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:31.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:30 smithi129 ceph-mon[88471]: pgmap v870: 93 pgs: 93 active+clean; 652 KiB data, 5.6 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:09:31.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:30 smithi129 ceph-mon[88471]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs undersized) 2024-09-21T10:09:31.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:30 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/543939313' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:31.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:31.171 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:31.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:31.272 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:31.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:30 smithi107 ceph-mon[102221]: pgmap v870: 93 pgs: 93 active+clean; 652 KiB data, 5.6 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:09:31.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:30 smithi107 ceph-mon[102221]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs undersized) 2024-09-21T10:09:31.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:30 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/543939313' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:31.295 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:30 smithi107 ceph-mon[105022]: pgmap v870: 93 pgs: 93 active+clean; 652 KiB data, 5.6 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:09:31.295 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:30 smithi107 ceph-mon[105022]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2 pgs undersized) 2024-09-21T10:09:31.295 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:30 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/543939313' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:31.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:31.296 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:31.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:31.397 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:31.419 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:31.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:31.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:31.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:31.544 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:31.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:31.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:31.646 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:31.668 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:31.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:31.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:31.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:31.792 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:31.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:31.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:31.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:09:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:09:31] "GET /metrics HTTP/1.1" 200 36662 "" "Prometheus/2.51.0" 2024-09-21T10:09:31.893 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:31.916 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:31.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:31.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:32.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:32.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:32.041 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:32.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:32.141 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:32.165 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:32.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:32.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:32.267 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:32.290 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:32.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:32.391 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:32.413 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:32.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:32.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:32.515 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:32.538 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:32.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:32.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:32.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:32.661 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:32.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:32.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:32.763 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:32.785 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:32.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:32.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:32.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:32.908 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:32.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:32.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:33.010 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:33.033 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:33.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:33.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:33.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:33.159 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:33.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:33.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:32 smithi129 ceph-mon[88471]: pgmap v871: 93 pgs: 93 active+clean; 652 KiB data, 5.6 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:09:33.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:33.287 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:33.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:33.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:32 smithi107 ceph-mon[102221]: pgmap v871: 93 pgs: 93 active+clean; 652 KiB data, 5.6 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:09:33.306 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:32 smithi107 ceph-mon[105022]: pgmap v871: 93 pgs: 93 active+clean; 652 KiB data, 5.6 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:09:33.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:09:33.388 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:33.410 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:33.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:33.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:33.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:33.534 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:33.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:33.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:33.635 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:33.657 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:33.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:33.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:33.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:33.781 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:33.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:33.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:33.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:33.904 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:33.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:34.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:34.027 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:34.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:34.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:34.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:34.151 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:34.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:34.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:34.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:34.275 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:34.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:34.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:34.377 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:34.398 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:34.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:34.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:34.500 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:34.526 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:34.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:34.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:34.615 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:34.627 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:34.652 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:34.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:34.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:34.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:34.775 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:34.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:34.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:34.877 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:34.903 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:34.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:34.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:35.005 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:35.031 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:35.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:35.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:35.073 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:34 smithi107 ceph-mon[102221]: pgmap v872: 93 pgs: 93 active+clean; 652 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:35.073 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:34 smithi107 ceph-mon[105022]: pgmap v872: 93 pgs: 93 active+clean; 652 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:35.133 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:35.159 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:35.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:35.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:35.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:34 smithi129 ceph-mon[88471]: pgmap v872: 93 pgs: 93 active+clean; 652 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:35.261 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:35.287 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:35.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:35.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:35.389 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:35.414 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:35.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:35.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:35.481 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:35.515 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:35.538 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:35.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:35.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:35.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:35.662 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:35.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:35.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:35.763 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:35.789 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:35.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:35.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:35.891 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:35.917 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:35.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:35.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:35.979 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:09:35.979 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:09:35.985 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:36.018 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:36.043 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:36.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:36.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:36.145 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:36.170 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:36.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:36.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:36.272 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:36.294 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:36.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:36.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:36.396 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:36.422 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:36.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:36.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:36.523 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:36.545 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:36.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:36.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:36.647 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:36.674 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:36.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:36.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:36.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:36.797 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:36.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:36.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:36.899 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:36.923 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:36 smithi107 ceph-mon[102221]: pgmap v873: 93 pgs: 93 active+clean; 652 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:36.923 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:36 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:36.924 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:36 smithi107 ceph-mon[105022]: pgmap v873: 93 pgs: 93 active+clean; 652 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:36.924 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:36 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:36.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:36.925 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:36.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:37.026 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:37.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:37.050 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:37.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:37.151 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:37.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:36 smithi129 ceph-mon[88471]: pgmap v873: 93 pgs: 93 active+clean; 652 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:37.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:36 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:37.175 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:37.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:37.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:37.192 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:09:37.192 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:09:37.277 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:37.300 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:37.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:37.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:37.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:37.424 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:37.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:37.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:37.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:37.547 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:37.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:37.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:37.649 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:37.676 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:37.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:37.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:37.777 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:37.799 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:37.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:37.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:37.875 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:09:37.901 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:37.927 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:37.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:37.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:38.029 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:38.052 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:38.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:38.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:38.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:38.176 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:38.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:38.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:38.278 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:38.304 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:38.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:38.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:38.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:38.427 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:38.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:38.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:38.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:38.551 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:38.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:38.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:38.653 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:38.679 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:38.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:38.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:38.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:38.802 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:38.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:38.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:38.904 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:38.930 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:38.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:38.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:39.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:39.053 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:39.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:39.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:39.128 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:38 smithi107 ceph-mon[102221]: pgmap v874: 93 pgs: 93 active+clean; 652 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:39.129 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:38 smithi107 ceph-mon[105022]: pgmap v874: 93 pgs: 93 active+clean; 652 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:39.155 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:39.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:38 smithi129 ceph-mon[88471]: pgmap v874: 93 pgs: 93 active+clean; 652 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:39.181 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:39.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:39.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:39.282 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:39.307 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:39.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:39.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:39.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:39.431 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:39.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:39.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:39.533 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:39.557 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:39.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:39.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:39.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:39.681 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:39.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:39.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:39.783 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:39.805 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:39.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:39.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:39.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:39.928 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:39.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:39.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:40.029 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:40.056 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:40.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:40.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:40.157 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:40.183 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:40.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:40.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:40.285 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:40.310 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:40.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:40.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:40.412 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:40.437 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:40.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:40.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:40.539 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:40.564 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:40.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:40.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:40.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:40.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:09:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:09:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:09:40.687 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:40.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:40.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:40.788 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:40.803 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:40.814 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:40.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:40.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:40.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:40.932 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:40 smithi107 ceph-mon[102221]: pgmap v875: 93 pgs: 93 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 25 KiB/s, 0 objects/s recovering 2024-09-21T10:09:40.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:40 smithi107 ceph-mon[105022]: pgmap v875: 93 pgs: 93 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 25 KiB/s, 0 objects/s recovering 2024-09-21T10:09:40.939 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:40.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:40.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:41.040 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:41.055 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:40 smithi129 ceph-mon[88471]: pgmap v875: 93 pgs: 93 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 25 KiB/s, 0 objects/s recovering 2024-09-21T10:09:41.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:41.066 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:41.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:41.167 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:41.189 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:41.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:41.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:41.291 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:41.312 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:41.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:41.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:41.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:41.435 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:41.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:41.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:41.537 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:41.558 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:41.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:41.660 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:41.668 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:09:41.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T10:09:41.681 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:41.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:41.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:41.783 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:41.804 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:41.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:41.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:41.839 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:09:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:09:41] "GET /metrics HTTP/1.1" 200 34408 "" "Prometheus/2.51.0" 2024-09-21T10:09:41.886 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_6' created 2024-09-21T10:09:41.906 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:41.927 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:41.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:41.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:42.029 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:42.050 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:42.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:42.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:42.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:41 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2795778345' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-21T10:09:42.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:42.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:42.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:41 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2795778345' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-21T10:09:42.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:42.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:42.152 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:42.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:41 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2795778345' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-21T10:09:42.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:42.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:42.173 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:42.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:42.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:42.274 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:42.295 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:42.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:42.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:42.397 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:42.418 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:42.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:42.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:42.487 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:09:42.488 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:09:42.494 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:09:42.494 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:09:42.510 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:09:42.510 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:09:42.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:42.526 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:09:42.526 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:09:42.544 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:09:42.544 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:09:42.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:42.545 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:42.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:42.559 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:09:42.559 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:09:42.575 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:09:42.575 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:09:42.590 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:09:42.590 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:09:42.646 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:42.668 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:42.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:42.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:42.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:42.793 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:42.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:42.892 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:09:42.894 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:42.918 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:42.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:42.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:43.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:43.041 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:43.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:43.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:43.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:43.164 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:43.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:43.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:43.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:42 smithi129 ceph-mon[88471]: pgmap v876: 93 pgs: 93 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:43.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:42 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2795778345' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-09-21T10:09:43.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:42 smithi129 ceph-mon[88471]: osdmap e252: 8 total, 8 up, 7 in 2024-09-21T10:09:43.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:42 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1210340042' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:09:43.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:42 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:09:43.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:42 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2753227139' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:43.266 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:43.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:43.288 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:43.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:43.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:42 smithi107 ceph-mon[105022]: pgmap v876: 93 pgs: 93 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:43.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:42 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2795778345' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-09-21T10:09:43.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:42 smithi107 ceph-mon[105022]: osdmap e252: 8 total, 8 up, 7 in 2024-09-21T10:09:43.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:42 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1210340042' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:09:43.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:42 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:09:43.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:42 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2753227139' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:43.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:42 smithi107 ceph-mon[102221]: pgmap v876: 93 pgs: 93 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:43.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:42 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2795778345' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-09-21T10:09:43.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:42 smithi107 ceph-mon[102221]: osdmap e252: 8 total, 8 up, 7 in 2024-09-21T10:09:43.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:42 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1210340042' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:09:43.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:42 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:09:43.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:42 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2753227139' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:43.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:43.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:43.412 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:43.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:43.513 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:43.535 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:43.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:43.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:43.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:43.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:43.659 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:43.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:43.760 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:43.783 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:43.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:43.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:43.884 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:43.908 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:43.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:43.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:44.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:44.031 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:44.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:44.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:44.133 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:44.155 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:44.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:44.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:44.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:43 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:09:44.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:43 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:09:44.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:43 smithi129 ceph-mon[88471]: osdmap e253: 8 total, 8 up, 7 in 2024-09-21T10:09:44.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:44.259 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:43 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:09:44.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:43 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:09:44.260 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:43 smithi107 ceph-mon[102221]: osdmap e253: 8 total, 8 up, 7 in 2024-09-21T10:09:44.260 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:43 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:09:44.260 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:43 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:09:44.260 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:43 smithi107 ceph-mon[105022]: osdmap e253: 8 total, 8 up, 7 in 2024-09-21T10:09:44.279 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:44.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:44.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:44.382 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:44.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-09-21T10:09:44.405 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:44.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:44.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:44.506 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:44.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:44.530 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:44.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:44.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:44.653 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:44.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:44.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:44.754 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:44.777 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:44.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:44.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:44.879 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:44.902 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:44.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:44.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:45.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:45.025 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:45.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:45.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:44 smithi107 ceph-mon[102221]: pgmap v879: 109 pgs: 2 creating+peering, 14 unknown, 93 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:44 smithi107 ceph-mon[102221]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:09:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:44 smithi107 ceph-mon[102221]: osdmap e254: 8 total, 8 up, 7 in 2024-09-21T10:09:45.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:44 smithi107 ceph-mon[105022]: pgmap v879: 109 pgs: 2 creating+peering, 14 unknown, 93 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:45.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:44 smithi107 ceph-mon[105022]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:09:45.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:44 smithi107 ceph-mon[105022]: osdmap e254: 8 total, 8 up, 7 in 2024-09-21T10:09:45.127 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:45.149 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:45.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:45.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:44 smithi129 ceph-mon[88471]: pgmap v879: 109 pgs: 2 creating+peering, 14 unknown, 93 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:45.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:44 smithi129 ceph-mon[88471]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:09:45.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:44 smithi129 ceph-mon[88471]: osdmap e254: 8 total, 8 up, 7 in 2024-09-21T10:09:45.251 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:45.257 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:45.272 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:45.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:45.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:45.374 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:45.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:45.398 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:45.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:45.499 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:45.522 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:45.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:45.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:45.646 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:45.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:45.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:45.748 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:45.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:45.772 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:45.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:45.873 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:45.895 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:45.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:45.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:45.997 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:46.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:46.024 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:46.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:46.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:46.147 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:46.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:46.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:46.248 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:46.271 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:46.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:46.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:46.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:09:46.368 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:09:46.368 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-09-21T10:09:46.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-09-21T10:09:46.368 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:09:46.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.368 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:95095 10.5 10:a62481d1:::benchmark_data_smithi107_184135_object95094:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T10:09:46.368 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:06:06.381486+0000", 2024-09-21T10:09:46.368 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 219.97284196699999, 2024-09-21T10:09:46.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.022675132, 2024-09-21T10:09:46.369 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:09:46.369 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:09:46.369 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:09:46.369 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:09:46.369 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:09:46.369 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:09:46.369 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 95095 2024-09-21T10:09:46.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.369 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:09:46.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:09:46.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.381486+0000", 2024-09-21T10:09:46.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:09:46.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.381486+0000", 2024-09-21T10:09:46.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:09:46.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.381488+0000", 2024-09-21T10:09:46.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7400000000000001e-06 2024-09-21T10:09:46.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:09:46.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.381511+0000", 2024-09-21T10:09:46.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3374e-05 2024-09-21T10:09:46.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:09:46.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.381527+0000", 2024-09-21T10:09:46.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5928000000000001e-05 2024-09-21T10:09:46.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:09:46.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.381534+0000", 2024-09-21T10:09:46.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1369999999999996e-06 2024-09-21T10:09:46.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.381603+0000", 2024-09-21T10:09:46.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9395999999999998e-05 2024-09-21T10:09:46.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:09:46.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.381747+0000", 2024-09-21T10:09:46.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014325399999999999 2024-09-21T10:09:46.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:09:46.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.382052+0000", 2024-09-21T10:09:46.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000305183 2024-09-21T10:09:46.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:09:46.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.385248+0000", 2024-09-21T10:09:46.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031962660000000001 2024-09-21T10:09:46.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.386212+0000", 2024-09-21T10:09:46.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00096411100000000003 2024-09-21T10:09:46.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.374 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.406246+0000", 2024-09-21T10:09:46.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020034209000000001 2024-09-21T10:09:46.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.374 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.406257+0000", 2024-09-21T10:09:46.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0839000000000001e-05 2024-09-21T10:09:46.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.404063+0000", 2024-09-21T10:09:46.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.99780565700000001 2024-09-21T10:09:46.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:09:46.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.404161+0000", 2024-09-21T10:09:46.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8037999999999999e-05 2024-09-21T10:09:46.375 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.375 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.375 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.376 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:95097 10.5 10:a3df757c:::benchmark_data_smithi107_184135_object95096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T10:09:46.376 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:06:06.388020+0000", 2024-09-21T10:09:46.376 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 219.96630747099999, 2024-09-21T10:09:46.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0166121699999999, 2024-09-21T10:09:46.376 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:09:46.376 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:09:46.376 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:09:46.376 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:09:46.376 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:09:46.376 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:09:46.376 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 95097 2024-09-21T10:09:46.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.376 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:09:46.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:09:46.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.388020+0000", 2024-09-21T10:09:46.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:09:46.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.388020+0000", 2024-09-21T10:09:46.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:09:46.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.388022+0000", 2024-09-21T10:09:46.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.61e-06 2024-09-21T10:09:46.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.378 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.378 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:09:46.378 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.388045+0000", 2024-09-21T10:09:46.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2631999999999999e-05 2024-09-21T10:09:46.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.378 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.378 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:09:46.378 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.388061+0000", 2024-09-21T10:09:46.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6738000000000001e-05 2024-09-21T10:09:46.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.378 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.378 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:09:46.378 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.388069+0000", 2024-09-21T10:09:46.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4259999999999997e-06 2024-09-21T10:09:46.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.379 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.388121+0000", 2024-09-21T10:09:46.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2444999999999997e-05 2024-09-21T10:09:46.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:09:46.379 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.388286+0000", 2024-09-21T10:09:46.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000164537 2024-09-21T10:09:46.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:09:46.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.388564+0000", 2024-09-21T10:09:46.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00027838899999999998 2024-09-21T10:09:46.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:09:46.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.392309+0000", 2024-09-21T10:09:46.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003744813 2024-09-21T10:09:46.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.392717+0000", 2024-09-21T10:09:46.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040770299999999998 2024-09-21T10:09:46.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.406330+0000", 2024-09-21T10:09:46.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013613484 2024-09-21T10:09:46.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.406337+0000", 2024-09-21T10:09:46.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9569999999999996e-06 2024-09-21T10:09:46.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.404560+0000", 2024-09-21T10:09:46.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.99822255900000001 2024-09-21T10:09:46.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.382 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:09:46.382 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.404632+0000", 2024-09-21T10:09:46.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2877e-05 2024-09-21T10:09:46.382 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.382 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.382 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.382 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:95106 10.8 10:128143e9:::benchmark_data_smithi107_184135_object95105:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e202)", 2024-09-21T10:09:46.382 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:06:06.400796+0000", 2024-09-21T10:09:46.383 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 219.953531626, 2024-09-21T10:09:46.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0161196219999999, 2024-09-21T10:09:46.383 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:09:46.383 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:09:46.383 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:46.383 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:09:46.383 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:09:46.383 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:09:46.383 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 95106 2024-09-21T10:09:46.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.383 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:09:46.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:09:46.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.400796+0000", 2024-09-21T10:09:46.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.384 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:09:46.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.400796+0000", 2024-09-21T10:09:46.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.384 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:09:46.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.400798+0000", 2024-09-21T10:09:46.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.813e-06 2024-09-21T10:09:46.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:09:46.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.400834+0000", 2024-09-21T10:09:46.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5772000000000002e-05 2024-09-21T10:09:46.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:09:46.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.400845+0000", 2024-09-21T10:09:46.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1749000000000001e-05 2024-09-21T10:09:46.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:09:46.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.400855+0000", 2024-09-21T10:09:46.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8689999999999999e-06 2024-09-21T10:09:46.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.402893+0000", 2024-09-21T10:09:46.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020376090000000001 2024-09-21T10:09:46.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.402902+0000", 2024-09-21T10:09:46.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1209999999999996e-06 2024-09-21T10:09:46.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.404968+0000", 2024-09-21T10:09:46.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0020656670000001 2024-09-21T10:09:46.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:09:46.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.407877+0000", 2024-09-21T10:09:46.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0029096069999999998 2024-09-21T10:09:46.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 7", 2024-09-21T10:09:46.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.407981+0000", 2024-09-21T10:09:46.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010329 2024-09-21T10:09:46.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:09:46.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.414245+0000", 2024-09-21T10:09:46.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0062643009999999999 2024-09-21T10:09:46.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.416792+0000", 2024-09-21T10:09:46.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002546815 2024-09-21T10:09:46.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:09:46.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.416837+0000", 2024-09-21T10:09:46.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5547000000000003e-05 2024-09-21T10:09:46.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.389 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:09:46.389 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.416916+0000", 2024-09-21T10:09:46.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8461999999999995e-05 2024-09-21T10:09:46.389 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.389 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.389 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.389 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:95108 10.8 10:166cc55a:::benchmark_data_smithi107_184135_object95107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-21T10:09:46.389 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:06:06.406993+0000", 2024-09-21T10:09:46.389 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 219.94733518000001, 2024-09-21T10:09:46.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0100435590000001, 2024-09-21T10:09:46.390 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:09:46.390 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:09:46.390 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:46.390 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:09:46.390 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:09:46.390 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:09:46.390 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 95108 2024-09-21T10:09:46.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.390 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:09:46.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:09:46.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.406993+0000", 2024-09-21T10:09:46.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:09:46.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.406993+0000", 2024-09-21T10:09:46.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:09:46.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.406994+0000", 2024-09-21T10:09:46.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4249999999999999e-06 2024-09-21T10:09:46.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:09:46.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.407032+0000", 2024-09-21T10:09:46.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8442000000000002e-05 2024-09-21T10:09:46.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:09:46.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.407044+0000", 2024-09-21T10:09:46.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1411e-05 2024-09-21T10:09:46.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:09:46.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.407051+0000", 2024-09-21T10:09:46.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0400000000000004e-06 2024-09-21T10:09:46.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.407100+0000", 2024-09-21T10:09:46.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9364999999999999e-05 2024-09-21T10:09:46.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:06.407109+0000", 2024-09-21T10:09:46.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5159999999999994e-06 2024-09-21T10:09:46.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.411341+0000", 2024-09-21T10:09:46.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.004232566 2024-09-21T10:09:46.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:09:46.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.411481+0000", 2024-09-21T10:09:46.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014006399999999999 2024-09-21T10:09:46.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 7", 2024-09-21T10:09:46.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.411641+0000", 2024-09-21T10:09:46.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015907200000000001 2024-09-21T10:09:46.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:09:46.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.414430+0000", 2024-09-21T10:09:46.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027895160000000001 2024-09-21T10:09:46.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.416956+0000", 2024-09-21T10:09:46.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002526272 2024-09-21T10:09:46.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:09:46.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.416988+0000", 2024-09-21T10:09:46.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.15e-05 2024-09-21T10:09:46.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:09:46.396 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:06:07.417036+0000", 2024-09-21T10:09:46.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8369999999999999e-05 2024-09-21T10:09:46.396 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.396 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.396 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.396 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:247231 10.9 10:9266f709:::benchmark_data_smithi107_184135_object115665:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:09:46.396 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:50.408737+0000", 2024-09-21T10:09:46.396 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 55.945591256999997, 2024-09-21T10:09:46.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1251118249999998, 2024-09-21T10:09:46.397 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:09:46.397 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:09:46.397 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:46.397 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:09:46.397 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:09:46.397 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:09:46.397 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 247231 2024-09-21T10:09:46.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.397 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:09:46.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:09:46.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.408737+0000", 2024-09-21T10:09:46.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:09:46.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.408737+0000", 2024-09-21T10:09:46.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:09:46.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.408739+0000", 2024-09-21T10:09:46.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.029e-06 2024-09-21T10:09:46.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:09:46.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.408745+0000", 2024-09-21T10:09:46.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.959e-06 2024-09-21T10:09:46.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:09:46.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.408761+0000", 2024-09-21T10:09:46.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6353000000000001e-05 2024-09-21T10:09:46.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:09:46.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.408768+0000", 2024-09-21T10:09:46.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6159999999999996e-06 2024-09-21T10:09:46.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.408848+0000", 2024-09-21T10:09:46.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9703999999999994e-05 2024-09-21T10:09:46.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.408857+0000", 2024-09-21T10:09:46.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9700000000000005e-06 2024-09-21T10:09:46.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.361748+0000", 2024-09-21T10:09:46.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.95289130799999999 2024-09-21T10:09:46.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.361760+0000", 2024-09-21T10:09:46.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1688999999999999e-05 2024-09-21T10:09:46.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.517499+0000", 2024-09-21T10:09:46.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1557386039999999 2024-09-21T10:09:46.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:09:46.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.517613+0000", 2024-09-21T10:09:46.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000113805 2024-09-21T10:09:46.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5,7", 2024-09-21T10:09:46.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.518061+0000", 2024-09-21T10:09:46.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00044853200000000002 2024-09-21T10:09:46.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.529487+0000", 2024-09-21T10:09:46.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011425519 2024-09-21T10:09:46.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.529535+0000", 2024-09-21T10:09:46.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8028999999999997e-05 2024-09-21T10:09:46.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.529598+0000", 2024-09-21T10:09:46.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3150000000000004e-05 2024-09-21T10:09:46.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.404 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:09:46.404 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.533661+0000", 2024-09-21T10:09:46.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0040628859999999999 2024-09-21T10:09:46.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.404 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:09:46.404 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.533718+0000", 2024-09-21T10:09:46.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7045e-05 2024-09-21T10:09:46.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.404 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:09:46.404 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.533848+0000", 2024-09-21T10:09:46.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013062699999999999 2024-09-21T10:09:46.405 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.405 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.405 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.405 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:247253 10.9 10:9c4255ed:::benchmark_data_smithi107_184135_object115687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:09:46.405 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:50.431813+0000", 2024-09-21T10:09:46.405 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 55.922514970000002, 2024-09-21T10:09:46.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1023250619999998, 2024-09-21T10:09:46.405 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:09:46.405 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:09:46.405 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:46.405 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:09:46.405 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:09:46.406 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:09:46.406 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 247253 2024-09-21T10:09:46.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.406 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:09:46.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.406 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:09:46.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.431813+0000", 2024-09-21T10:09:46.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.406 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:09:46.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.431813+0000", 2024-09-21T10:09:46.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.407 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:09:46.407 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.431814+0000", 2024-09-21T10:09:46.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.553e-06 2024-09-21T10:09:46.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.407 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:09:46.408 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.431827+0000", 2024-09-21T10:09:46.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3107e-05 2024-09-21T10:09:46.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.408 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:09:46.408 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.431840+0000", 2024-09-21T10:09:46.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2081e-05 2024-09-21T10:09:46.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:09:46.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.431847+0000", 2024-09-21T10:09:46.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4150000000000002e-06 2024-09-21T10:09:46.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.431906+0000", 2024-09-21T10:09:46.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9261000000000001e-05 2024-09-21T10:09:46.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.431915+0000", 2024-09-21T10:09:46.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8740000000000001e-06 2024-09-21T10:09:46.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.411 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.361786+0000", 2024-09-21T10:09:46.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.92987135300000001 2024-09-21T10:09:46.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.361794+0000", 2024-09-21T10:09:46.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.357e-06 2024-09-21T10:09:46.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.529630+0000", 2024-09-21T10:09:46.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1678363460000001 2024-09-21T10:09:46.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:09:46.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.529718+0000", 2024-09-21T10:09:46.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8238000000000005e-05 2024-09-21T10:09:46.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5,7", 2024-09-21T10:09:46.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.529793+0000", 2024-09-21T10:09:46.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4548999999999994e-05 2024-09-21T10:09:46.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.533896+0000", 2024-09-21T10:09:46.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041030909999999997 2024-09-21T10:09:46.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:46.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:09:46.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.533909+0000", 2024-09-21T10:09:46.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2986999999999999e-05 2024-09-21T10:09:46.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.533946+0000", 2024-09-21T10:09:46.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7449e-05 2024-09-21T10:09:46.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.418 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.418 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.533997+0000", 2024-09-21T10:09:46.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0659999999999999e-05 2024-09-21T10:09:46.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.418 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:09:46.418 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.534039+0000", 2024-09-21T10:09:46.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2358999999999999e-05 2024-09-21T10:09:46.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.418 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:09:46.418 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.534138+0000", 2024-09-21T10:09:46.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8381999999999994e-05 2024-09-21T10:09:46.419 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.419 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.419 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.419 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:247270 10.9 10:9c37c643:::benchmark_data_smithi107_184135_object115704:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:09:46.419 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:50.444792+0000", 2024-09-21T10:09:46.419 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 55.909535587000001, 2024-09-21T10:09:46.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.117923904, 2024-09-21T10:09:46.419 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:09:46.419 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:09:46.419 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:46.419 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:09:46.420 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:09:46.420 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:09:46.420 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 247270 2024-09-21T10:09:46.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.420 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:09:46.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.420 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:09:46.420 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.444792+0000", 2024-09-21T10:09:46.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.420 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:09:46.420 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.444792+0000", 2024-09-21T10:09:46.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:09:46.421 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.444793+0000", 2024-09-21T10:09:46.421 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0100000000000004e-07 2024-09-21T10:09:46.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:09:46.421 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.444797+0000", 2024-09-21T10:09:46.421 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.721e-06 2024-09-21T10:09:46.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:09:46.421 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.444804+0000", 2024-09-21T10:09:46.422 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5669999999999997e-06 2024-09-21T10:09:46.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.422 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:09:46.422 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.444811+0000", 2024-09-21T10:09:46.422 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3330000000000002e-06 2024-09-21T10:09:46.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.422 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.422 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.444879+0000", 2024-09-21T10:09:46.422 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8695000000000006e-05 2024-09-21T10:09:46.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.422 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.444888+0000", 2024-09-21T10:09:46.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5890000000000003e-06 2024-09-21T10:09:46.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.423 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.362063+0000", 2024-09-21T10:09:46.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.91717548500000001 2024-09-21T10:09:46.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.423 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.362072+0000", 2024-09-21T10:09:46.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8999999999999995e-06 2024-09-21T10:09:46.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.424 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.424 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.534171+0000", 2024-09-21T10:09:46.424 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1720986099999999 2024-09-21T10:09:46.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.424 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:09:46.424 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.534273+0000", 2024-09-21T10:09:46.424 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010196600000000001 2024-09-21T10:09:46.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.424 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5,7", 2024-09-21T10:09:46.424 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.534374+0000", 2024-09-21T10:09:46.424 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010144700000000001 2024-09-21T10:09:46.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.542641+0000", 2024-09-21T10:09:46.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0082664399999999999 2024-09-21T10:09:46.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.542683+0000", 2024-09-21T10:09:46.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1971000000000002e-05 2024-09-21T10:09:46.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.553376+0000", 2024-09-21T10:09:46.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010693362 2024-09-21T10:09:46.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.426 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:09:46.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.562549+0000", 2024-09-21T10:09:46.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0091732740000000004 2024-09-21T10:09:46.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.426 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:09:46.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.562602+0000", 2024-09-21T10:09:46.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3167000000000001e-05 2024-09-21T10:09:46.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:09:46.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.562716+0000", 2024-09-21T10:09:46.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000113676 2024-09-21T10:09:46.427 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.427 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.427 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.427 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:247273 10.9 10:99c5a2f7:::benchmark_data_smithi107_184135_object115707:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:09:46.428 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:50.447745+0000", 2024-09-21T10:09:46.428 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 55.906582718999999, 2024-09-21T10:09:46.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1151268839999999, 2024-09-21T10:09:46.428 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:09:46.428 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:09:46.428 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:46.428 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:09:46.428 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:09:46.428 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:09:46.428 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 247273 2024-09-21T10:09:46.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.428 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:09:46.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.428 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:09:46.429 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.447745+0000", 2024-09-21T10:09:46.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.429 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:09:46.429 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.447745+0000", 2024-09-21T10:09:46.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.429 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:09:46.429 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.447746+0000", 2024-09-21T10:09:46.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.141e-06 2024-09-21T10:09:46.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.429 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:09:46.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.447751+0000", 2024-09-21T10:09:46.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4700000000000004e-06 2024-09-21T10:09:46.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:09:46.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.447761+0000", 2024-09-21T10:09:46.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0368e-05 2024-09-21T10:09:46.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:09:46.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.447775+0000", 2024-09-21T10:09:46.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4177e-05 2024-09-21T10:09:46.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.447848+0000", 2024-09-21T10:09:46.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2761000000000004e-05 2024-09-21T10:09:46.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:50.447860+0000", 2024-09-21T10:09:46.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1753000000000001e-05 2024-09-21T10:09:46.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.362115+0000", 2024-09-21T10:09:46.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.91425533599999997 2024-09-21T10:09:46.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.432 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.432 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.362124+0000", 2024-09-21T10:09:46.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9779999999999994e-06 2024-09-21T10:09:46.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.432 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.432 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.542711+0000", 2024-09-21T10:09:46.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.180587061 2024-09-21T10:09:46.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.432 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:09:46.432 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.542903+0000", 2024-09-21T10:09:46.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019238 2024-09-21T10:09:46.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5,7", 2024-09-21T10:09:46.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.543034+0000", 2024-09-21T10:09:46.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013075000000000001 2024-09-21T10:09:46.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.553417+0000", 2024-09-21T10:09:46.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010382786999999999 2024-09-21T10:09:46.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.553470+0000", 2024-09-21T10:09:46.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3457e-05 2024-09-21T10:09:46.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.553512+0000", 2024-09-21T10:09:46.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1579000000000003e-05 2024-09-21T10:09:46.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:09:46.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.562724+0000", 2024-09-21T10:09:46.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0092116730000000001 2024-09-21T10:09:46.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:09:46.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.562764+0000", 2024-09-21T10:09:46.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0263999999999999e-05 2024-09-21T10:09:46.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:09:46.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.562872+0000", 2024-09-21T10:09:46.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010794900000000001 2024-09-21T10:09:46.435 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.435 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.436 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.436 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.436 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:247293 10.9 10:9cc60b29:::benchmark_data_smithi107_184135_object115727:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T10:09:46.436 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:51.460805+0000", 2024-09-21T10:09:46.436 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 54.893522990999998, 2024-09-21T10:09:46.436 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.102965384, 2024-09-21T10:09:46.436 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:09:46.436 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:09:46.436 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:46.436 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:09:46.436 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:09:46.436 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:09:46.436 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 247293 2024-09-21T10:09:46.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.437 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:09:46.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:09:46.437 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.460805+0000", 2024-09-21T10:09:46.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:09:46.437 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.460805+0000", 2024-09-21T10:09:46.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:09:46.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.460807+0000", 2024-09-21T10:09:46.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7290000000000001e-06 2024-09-21T10:09:46.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.438 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:09:46.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.460812+0000", 2024-09-21T10:09:46.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4600000000000002e-06 2024-09-21T10:09:46.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.438 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:09:46.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.460880+0000", 2024-09-21T10:09:46.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8387999999999996e-05 2024-09-21T10:09:46.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.439 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:09:46.439 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.460888+0000", 2024-09-21T10:09:46.439 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9880000000000004e-06 2024-09-21T10:09:46.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.439 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.439 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.460952+0000", 2024-09-21T10:09:46.439 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3468999999999998e-05 2024-09-21T10:09:46.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.439 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.439 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.460963+0000", 2024-09-21T10:09:46.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0909e-05 2024-09-21T10:09:46.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.549952+0000", 2024-09-21T10:09:46.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.088988928 2024-09-21T10:09:46.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:09:46.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.550088+0000", 2024-09-21T10:09:46.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013624000000000001 2024-09-21T10:09:46.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5,7", 2024-09-21T10:09:46.441 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.550214+0000", 2024-09-21T10:09:46.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012568 2024-09-21T10:09:46.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.441 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.441 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.562491+0000", 2024-09-21T10:09:46.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012277523 2024-09-21T10:09:46.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.441 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:09:46.441 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.562879+0000", 2024-09-21T10:09:46.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000388031 2024-09-21T10:09:46.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.442 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.562922+0000", 2024-09-21T10:09:46.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2750000000000002e-05 2024-09-21T10:09:46.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.443 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.562959+0000", 2024-09-21T10:09:46.443 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7574000000000003e-05 2024-09-21T10:09:46.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.443 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:09:46.443 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.563020+0000", 2024-09-21T10:09:46.443 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0534e-05 2024-09-21T10:09:46.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:09:46.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.563770+0000", 2024-09-21T10:09:46.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00075018100000000005 2024-09-21T10:09:46.444 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.444 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.444 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.445 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:247300 10.9 10:91d148b8:::benchmark_data_smithi107_184135_object115734:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T10:09:46.445 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:51.469177+0000", 2024-09-21T10:09:46.445 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 54.885150467000003, 2024-09-21T10:09:46.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0948672749999999, 2024-09-21T10:09:46.445 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:09:46.445 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:09:46.445 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:46.445 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:09:46.445 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:09:46.446 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:09:46.446 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 247300 2024-09-21T10:09:46.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.446 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:09:46.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:09:46.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.469177+0000", 2024-09-21T10:09:46.446 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.447 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:09:46.447 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.469177+0000", 2024-09-21T10:09:46.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.447 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:09:46.447 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.469179+0000", 2024-09-21T10:09:46.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.409e-06 2024-09-21T10:09:46.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.448 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:09:46.448 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.469184+0000", 2024-09-21T10:09:46.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8960000000000002e-06 2024-09-21T10:09:46.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:09:46.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.469197+0000", 2024-09-21T10:09:46.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3190000000000001e-05 2024-09-21T10:09:46.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:09:46.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.469204+0000", 2024-09-21T10:09:46.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3490000000000003e-06 2024-09-21T10:09:46.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.469267+0000", 2024-09-21T10:09:46.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3077999999999995e-05 2024-09-21T10:09:46.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.469277+0000", 2024-09-21T10:09:46.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.008e-05 2024-09-21T10:09:46.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.451 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.553540+0000", 2024-09-21T10:09:46.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.084263073 2024-09-21T10:09:46.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:09:46.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.553673+0000", 2024-09-21T10:09:46.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000132244 2024-09-21T10:09:46.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5,7", 2024-09-21T10:09:46.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.553986+0000", 2024-09-21T10:09:46.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00031366199999999998 2024-09-21T10:09:46.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.453 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.562534+0000", 2024-09-21T10:09:46.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085473440000000001 2024-09-21T10:09:46.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.453 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:09:46.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.562884+0000", 2024-09-21T10:09:46.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035075699999999999 2024-09-21T10:09:46.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.563816+0000", 2024-09-21T10:09:46.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00093124899999999999 2024-09-21T10:09:46.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.455 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.563898+0000", 2024-09-21T10:09:46.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2615999999999996e-05 2024-09-21T10:09:46.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.455 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:09:46.455 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.563947+0000", 2024-09-21T10:09:46.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8541000000000002e-05 2024-09-21T10:09:46.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:09:46.456 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.564045+0000", 2024-09-21T10:09:46.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7787000000000004e-05 2024-09-21T10:09:46.456 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.456 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.457 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.457 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:247314 10.9 10:93ecd3c5:::benchmark_data_smithi107_184135_object115748:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T10:09:46.457 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:51.493319+0000", 2024-09-21T10:09:46.457 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 54.861008673000001, 2024-09-21T10:09:46.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0709433610000001, 2024-09-21T10:09:46.457 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:09:46.457 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:09:46.457 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:46.458 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:09:46.458 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:09:46.458 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:09:46.458 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 247314 2024-09-21T10:09:46.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.458 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:09:46.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:09:46.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.493319+0000", 2024-09-21T10:09:46.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:09:46.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.493319+0000", 2024-09-21T10:09:46.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:09:46.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.493321+0000", 2024-09-21T10:09:46.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.547e-06 2024-09-21T10:09:46.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:09:46.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.493326+0000", 2024-09-21T10:09:46.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9219999999999996e-06 2024-09-21T10:09:46.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:09:46.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.493338+0000", 2024-09-21T10:09:46.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2461e-05 2024-09-21T10:09:46.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:09:46.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.493347+0000", 2024-09-21T10:09:46.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3319999999999998e-06 2024-09-21T10:09:46.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.493406+0000", 2024-09-21T10:09:46.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8919000000000003e-05 2024-09-21T10:09:46.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:51.493418+0000", 2024-09-21T10:09:46.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1664e-05 2024-09-21T10:09:46.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.558426+0000", 2024-09-21T10:09:46.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.065007772 2024-09-21T10:09:46.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:09:46.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.558635+0000", 2024-09-21T10:09:46.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00020949200000000001 2024-09-21T10:09:46.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5,7", 2024-09-21T10:09:46.462 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.558752+0000", 2024-09-21T10:09:46.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000117253 2024-09-21T10:09:46.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:09:46.462 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.562889+0000", 2024-09-21T10:09:46.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041364130000000002 2024-09-21T10:09:46.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.462 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.563863+0000", 2024-09-21T10:09:46.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00097411800000000005 2024-09-21T10:09:46.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.564083+0000", 2024-09-21T10:09:46.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021977499999999999 2024-09-21T10:09:46.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.564122+0000", 2024-09-21T10:09:46.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8810000000000003e-05 2024-09-21T10:09:46.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:09:46.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.564171+0000", 2024-09-21T10:09:46.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9175999999999998e-05 2024-09-21T10:09:46.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:09:46.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:52.564262+0000", 2024-09-21T10:09:46.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1706999999999995e-05 2024-09-21T10:09:46.464 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.464 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.464 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.464 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:253860 10.9 10:9012a306:::benchmark_data_smithi107_184135_object122294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T10:09:46.464 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:59.425482+0000", 2024-09-21T10:09:46.464 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.928845332999998, 2024-09-21T10:09:46.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0634980300000001, 2024-09-21T10:09:46.464 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:09:46.465 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:09:46.465 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:09:46.465 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:09:46.465 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:09:46.465 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:09:46.465 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 253860 2024-09-21T10:09:46.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.465 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:09:46.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:09:46.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.425482+0000", 2024-09-21T10:09:46.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:09:46.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.425482+0000", 2024-09-21T10:09:46.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:09:46.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.425484+0000", 2024-09-21T10:09:46.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.652e-06 2024-09-21T10:09:46.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:09:46.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.425490+0000", 2024-09-21T10:09:46.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0950000000000001e-06 2024-09-21T10:09:46.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:09:46.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.425568+0000", 2024-09-21T10:09:46.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7862999999999996e-05 2024-09-21T10:09:46.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:09:46.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.425576+0000", 2024-09-21T10:09:46.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2490000000000001e-06 2024-09-21T10:09:46.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.427906+0000", 2024-09-21T10:09:46.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002330033 2024-09-21T10:09:46.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:09:46.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.428040+0000", 2024-09-21T10:09:46.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013359999999999999 2024-09-21T10:09:46.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,5,7", 2024-09-21T10:09:46.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.428218+0000", 2024-09-21T10:09:46.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017765599999999999 2024-09-21T10:09:46.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.438937+0000", 2024-09-21T10:09:46.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010719279 2024-09-21T10:09:46.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.438947+0000", 2024-09-21T10:09:46.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6970000000000005e-06 2024-09-21T10:09:46.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.479628+0000", 2024-09-21T10:09:46.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0406818870000001 2024-09-21T10:09:46.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.470 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for missing object", 2024-09-21T10:09:46.470 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.479740+0000", 2024-09-21T10:09:46.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011163 2024-09-21T10:09:46.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.470 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.470 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.488916+0000", 2024-09-21T10:09:46.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0091756619999999994 2024-09-21T10:09:46.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.470 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:09:46.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.488980+0000", 2024-09-21T10:09:46.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4727000000000006e-05 2024-09-21T10:09:46.471 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.471 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.471 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.471 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:253861 10.9 10:9fb5f9b1:::benchmark_data_smithi107_184135_object122295:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T10:09:46.471 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:59.425596+0000", 2024-09-21T10:09:46.471 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.928731945999999, 2024-09-21T10:09:46.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0635803660000001, 2024-09-21T10:09:46.471 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:09:46.471 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:09:46.471 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:46.472 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:09:46.472 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:09:46.472 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:09:46.472 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 253861 2024-09-21T10:09:46.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.472 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:09:46.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:09:46.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.425596+0000", 2024-09-21T10:09:46.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:09:46.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.425596+0000", 2024-09-21T10:09:46.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:09:46.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.425596+0000", 2024-09-21T10:09:46.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.06e-07 2024-09-21T10:09:46.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:09:46.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.425600+0000", 2024-09-21T10:09:46.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0989999999999999e-06 2024-09-21T10:09:46.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:09:46.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.425607+0000", 2024-09-21T10:09:46.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9759999999999998e-06 2024-09-21T10:09:46.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:09:46.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.425612+0000", 2024-09-21T10:09:46.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0819999999999998e-06 2024-09-21T10:09:46.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.439121+0000", 2024-09-21T10:09:46.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013509022000000001 2024-09-21T10:09:46.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.439128+0000", 2024-09-21T10:09:46.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6629999999999993e-06 2024-09-21T10:09:46.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.480045+0000", 2024-09-21T10:09:46.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0409162320000001 2024-09-21T10:09:46.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.475 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:09:46.475 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.480121+0000", 2024-09-21T10:09:46.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6558999999999997e-05 2024-09-21T10:09:46.475 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.475 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.476 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:09:46.476 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.480227+0000", 2024-09-21T10:09:46.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000106198 2024-09-21T10:09:46.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.476 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.476 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.488536+0000", 2024-09-21T10:09:46.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0083087109999999999 2024-09-21T10:09:46.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.476 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:09:46.476 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.488547+0000", 2024-09-21T10:09:46.476 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1328999999999999e-05 2024-09-21T10:09:46.476 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.476 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.489017+0000", 2024-09-21T10:09:46.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046984499999999998 2024-09-21T10:09:46.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:09:46.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.489074+0000", 2024-09-21T10:09:46.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6736e-05 2024-09-21T10:09:46.477 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.477 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.477 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:09:46.477 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.489176+0000", 2024-09-21T10:09:46.477 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102308 2024-09-21T10:09:46.477 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.478 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.478 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.478 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.478 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.478 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:253887 10.9 10:90c249a6:::benchmark_data_smithi107_184135_object122321:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-21T10:09:46.478 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:59.450927+0000", 2024-09-21T10:09:46.478 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.903401150999997, 2024-09-21T10:09:46.478 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0418112639999999, 2024-09-21T10:09:46.478 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:09:46.478 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:09:46.478 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:46.478 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:09:46.478 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:09:46.479 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:09:46.479 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 253887 2024-09-21T10:09:46.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.479 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:09:46.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:09:46.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.450927+0000", 2024-09-21T10:09:46.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.479 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.479 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:09:46.479 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.450927+0000", 2024-09-21T10:09:46.479 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.479 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:09:46.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.450928+0000", 2024-09-21T10:09:46.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1039999999999999e-06 2024-09-21T10:09:46.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:09:46.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.450931+0000", 2024-09-21T10:09:46.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2729999999999998e-06 2024-09-21T10:09:46.480 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.480 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.480 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:09:46.480 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.450940+0000", 2024-09-21T10:09:46.480 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3340000000000008e-06 2024-09-21T10:09:46.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:09:46.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.450946+0000", 2024-09-21T10:09:46.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9699999999999996e-06 2024-09-21T10:09:46.481 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.481 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.481 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.481 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.451486+0000", 2024-09-21T10:09:46.481 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00053944099999999999 2024-09-21T10:09:46.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.482 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.482 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.482 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.451494+0000", 2024-09-21T10:09:46.482 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9119999999999998e-06 2024-09-21T10:09:46.482 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.483 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.483340+0000", 2024-09-21T10:09:46.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.031846595 2024-09-21T10:09:46.483 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.483 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:09:46.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.483458+0000", 2024-09-21T10:09:46.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000117601 2024-09-21T10:09:46.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.484 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:09:46.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.483561+0000", 2024-09-21T10:09:46.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103179 2024-09-21T10:09:46.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:09:46.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.488551+0000", 2024-09-21T10:09:46.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0049900020000000003 2024-09-21T10:09:46.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.489234+0000", 2024-09-21T10:09:46.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000683054 2024-09-21T10:09:46.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.486 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.486 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.492557+0000", 2024-09-21T10:09:46.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003323281 2024-09-21T10:09:46.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:09:46.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.492640+0000", 2024-09-21T10:09:46.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2268999999999994e-05 2024-09-21T10:09:46.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.488 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:09:46.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.492738+0000", 2024-09-21T10:09:46.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8249000000000001e-05 2024-09-21T10:09:46.488 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.488 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.488 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.488 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:253892 10.9 10:9aef61b1:::benchmark_data_smithi107_184135_object122326:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-21T10:09:46.488 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:59.452005+0000", 2024-09-21T10:09:46.489 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.902322709000003, 2024-09-21T10:09:46.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.041055778, 2024-09-21T10:09:46.489 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:09:46.489 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:09:46.489 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:46.489 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:09:46.489 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:09:46.489 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:09:46.489 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 253892 2024-09-21T10:09:46.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.489 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:09:46.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:09:46.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.452005+0000", 2024-09-21T10:09:46.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:09:46.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.452005+0000", 2024-09-21T10:09:46.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:09:46.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.452006+0000", 2024-09-21T10:09:46.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6400000000000001e-07 2024-09-21T10:09:46.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:09:46.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.452010+0000", 2024-09-21T10:09:46.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1679999999999997e-06 2024-09-21T10:09:46.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:09:46.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.452019+0000", 2024-09-21T10:09:46.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1980000000000007e-06 2024-09-21T10:09:46.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.491 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:09:46.491 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.452026+0000", 2024-09-21T10:09:46.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5710000000000003e-06 2024-09-21T10:09:46.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.492 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.452836+0000", 2024-09-21T10:09:46.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00080971000000000005 2024-09-21T10:09:46.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.492 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.452844+0000", 2024-09-21T10:09:46.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3920000000000003e-06 2024-09-21T10:09:46.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.492 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.485928+0000", 2024-09-21T10:09:46.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0330839249999999 2024-09-21T10:09:46.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:09:46.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.486018+0000", 2024-09-21T10:09:46.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0388000000000003e-05 2024-09-21T10:09:46.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:09:46.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.486093+0000", 2024-09-21T10:09:46.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4763000000000002e-05 2024-09-21T10:09:46.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:09:46.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.488979+0000", 2024-09-21T10:09:46.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028857380000000001 2024-09-21T10:09:46.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.489279+0000", 2024-09-21T10:09:46.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030031099999999999 2024-09-21T10:09:46.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.492920+0000", 2024-09-21T10:09:46.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0036412219999999999 2024-09-21T10:09:46.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:09:46.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.492975+0000", 2024-09-21T10:09:46.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4438000000000003e-05 2024-09-21T10:09:46.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:09:46.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.493061+0000", 2024-09-21T10:09:46.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6089999999999997e-05 2024-09-21T10:09:46.495 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.495 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.495 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.496 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:253912 10.9 10:922e3efd:::benchmark_data_smithi107_184135_object122346:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-21T10:09:46.496 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:59.470996+0000", 2024-09-21T10:09:46.496 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.883331941000002, 2024-09-21T10:09:46.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.029757786, 2024-09-21T10:09:46.496 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:09:46.496 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:09:46.496 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:46.496 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:09:46.496 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:09:46.496 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:09:46.496 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 253912 2024-09-21T10:09:46.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.496 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:09:46.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:09:46.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.470996+0000", 2024-09-21T10:09:46.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:09:46.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.470996+0000", 2024-09-21T10:09:46.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:09:46.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.470997+0000", 2024-09-21T10:09:46.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.049e-06 2024-09-21T10:09:46.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:09:46.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.471001+0000", 2024-09-21T10:09:46.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0330000000000002e-06 2024-09-21T10:09:46.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:09:46.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.471009+0000", 2024-09-21T10:09:46.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8509999999999999e-06 2024-09-21T10:09:46.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.498 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:09:46.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.471013+0000", 2024-09-21T10:09:46.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6879999999999998e-06 2024-09-21T10:09:46.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.471068+0000", 2024-09-21T10:09:46.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4268000000000003e-05 2024-09-21T10:09:46.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.471073+0000", 2024-09-21T10:09:46.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5779999999999998e-06 2024-09-21T10:09:46.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.489298+0000", 2024-09-21T10:09:46.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0182247250000001 2024-09-21T10:09:46.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:09:46.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.489361+0000", 2024-09-21T10:09:46.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2946000000000003e-05 2024-09-21T10:09:46.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:09:46.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.489436+0000", 2024-09-21T10:09:46.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4901000000000007e-05 2024-09-21T10:09:46.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:09:46.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.493073+0000", 2024-09-21T10:09:46.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0036370959999999998 2024-09-21T10:09:46.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.493111+0000", 2024-09-21T10:09:46.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8296000000000001e-05 2024-09-21T10:09:46.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.500662+0000", 2024-09-21T10:09:46.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0075507400000000002 2024-09-21T10:09:46.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.502 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:09:46.502 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.500715+0000", 2024-09-21T10:09:46.502 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2846999999999997e-05 2024-09-21T10:09:46.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.502 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:09:46.502 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.500754+0000", 2024-09-21T10:09:46.502 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8767999999999999e-05 2024-09-21T10:09:46.502 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.502 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.502 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.503 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:253922 10.9 10:96f04a1c:::benchmark_data_smithi107_184135_object122356:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-21T10:09:46.503 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:59.476761+0000", 2024-09-21T10:09:46.503 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.877566502999997, 2024-09-21T10:09:46.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.027350548, 2024-09-21T10:09:46.503 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:09:46.503 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:09:46.503 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:46.503 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:09:46.503 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:09:46.503 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:09:46.503 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 253922 2024-09-21T10:09:46.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.503 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:09:46.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.504 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:09:46.504 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.476761+0000", 2024-09-21T10:09:46.504 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.504 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:09:46.504 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.476761+0000", 2024-09-21T10:09:46.504 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.504 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:09:46.504 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.476762+0000", 2024-09-21T10:09:46.504 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7599999999999996e-07 2024-09-21T10:09:46.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:09:46.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.476766+0000", 2024-09-21T10:09:46.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.053e-06 2024-09-21T10:09:46.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:09:46.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.476781+0000", 2024-09-21T10:09:46.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4328000000000001e-05 2024-09-21T10:09:46.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:09:46.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.476788+0000", 2024-09-21T10:09:46.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9119999999999998e-06 2024-09-21T10:09:46.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.506 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.506 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.476847+0000", 2024-09-21T10:09:46.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8635999999999999e-05 2024-09-21T10:09:46.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.506 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.506 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.476859+0000", 2024-09-21T10:09:46.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.185e-05 2024-09-21T10:09:46.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.506 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.506 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.493160+0000", 2024-09-21T10:09:46.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.01630113 2024-09-21T10:09:46.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.507 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:09:46.507 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.493329+0000", 2024-09-21T10:09:46.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016861900000000001 2024-09-21T10:09:46.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.507 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:09:46.507 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.493442+0000", 2024-09-21T10:09:46.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011341999999999999 2024-09-21T10:09:46.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.507 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:09:46.507 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.500135+0000", 2024-09-21T10:09:46.508 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066929770000000001 2024-09-21T10:09:46.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.508 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.508 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.508 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.500170+0000", 2024-09-21T10:09:46.508 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4758000000000002e-05 2024-09-21T10:09:46.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.508 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.508 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.508 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.504031+0000", 2024-09-21T10:09:46.508 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038607590000000001 2024-09-21T10:09:46.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.509 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.509 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:09:46.509 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.504077+0000", 2024-09-21T10:09:46.509 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6168999999999998e-05 2024-09-21T10:09:46.509 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.509 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.509 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:09:46.509 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.504112+0000", 2024-09-21T10:09:46.509 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5061000000000002e-05 2024-09-21T10:09:46.510 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.510 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.510 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.510 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:253928 10.9 10:9e9713cf:::benchmark_data_smithi107_184135_object122362:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-21T10:09:46.510 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:59.485562+0000", 2024-09-21T10:09:46.510 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.868765574999998, 2024-09-21T10:09:46.510 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0288828189999999, 2024-09-21T10:09:46.511 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:09:46.511 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:09:46.511 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:46.511 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:09:46.511 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:09:46.511 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:09:46.511 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 253928 2024-09-21T10:09:46.511 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.512 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:09:46.512 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.512 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:09:46.512 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.485562+0000", 2024-09-21T10:09:46.512 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.512 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.512 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.512 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:09:46.512 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.485562+0000", 2024-09-21T10:09:46.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:09:46.513 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.485564+0000", 2024-09-21T10:09:46.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.345e-06 2024-09-21T10:09:46.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:09:46.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.485568+0000", 2024-09-21T10:09:46.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7319999999999998e-06 2024-09-21T10:09:46.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:09:46.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.485578+0000", 2024-09-21T10:09:46.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.516e-06 2024-09-21T10:09:46.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:09:46.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.485585+0000", 2024-09-21T10:09:46.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9310000000000002e-06 2024-09-21T10:09:46.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.485635+0000", 2024-09-21T10:09:46.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0525999999999997e-05 2024-09-21T10:09:46.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.516 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.516 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.485645+0000", 2024-09-21T10:09:46.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8390000000000002e-06 2024-09-21T10:09:46.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.516 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.516 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.496538+0000", 2024-09-21T10:09:46.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0108925929999999 2024-09-21T10:09:46.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:09:46.517 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.496788+0000", 2024-09-21T10:09:46.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00025016600000000002 2024-09-21T10:09:46.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:09:46.518 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.496928+0000", 2024-09-21T10:09:46.518 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000139839 2024-09-21T10:09:46.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.518 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:09:46.518 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.503979+0000", 2024-09-21T10:09:46.518 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.007050873 2024-09-21T10:09:46.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.519 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.519 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.504243+0000", 2024-09-21T10:09:46.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00026448900000000002 2024-09-21T10:09:46.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.519 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.519 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.514356+0000", 2024-09-21T10:09:46.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010113303000000001 2024-09-21T10:09:46.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:09:46.520 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.514406+0000", 2024-09-21T10:09:46.520 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9323000000000002e-05 2024-09-21T10:09:46.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:09:46.521 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.514445+0000", 2024-09-21T10:09:46.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9344000000000001e-05 2024-09-21T10:09:46.521 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.521 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.521 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.521 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:253951 10.9 10:9c869e79:::benchmark_data_smithi107_184135_object122385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-21T10:09:46.521 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:59.517491+0000", 2024-09-21T10:09:46.521 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.836836335999998, 2024-09-21T10:09:46.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.99712503299999999, 2024-09-21T10:09:46.521 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:09:46.521 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:09:46.521 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:46.522 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:09:46.522 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:09:46.522 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:09:46.522 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 253951 2024-09-21T10:09:46.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.522 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:09:46.522 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.522 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:09:46.522 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.517491+0000", 2024-09-21T10:09:46.522 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.522 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.522 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:09:46.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.517491+0000", 2024-09-21T10:09:46.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.523 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:09:46.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.517493+0000", 2024-09-21T10:09:46.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.621e-06 2024-09-21T10:09:46.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.523 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:09:46.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.517497+0000", 2024-09-21T10:09:46.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8999999999999999e-06 2024-09-21T10:09:46.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.524 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:09:46.524 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.517506+0000", 2024-09-21T10:09:46.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0969999999999995e-06 2024-09-21T10:09:46.524 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.524 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:09:46.524 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.517511+0000", 2024-09-21T10:09:46.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1549999999999998e-06 2024-09-21T10:09:46.524 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.524 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.524 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.517563+0000", 2024-09-21T10:09:46.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1329999999999998e-05 2024-09-21T10:09:46.524 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.517569+0000", 2024-09-21T10:09:46.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5769999999999999e-06 2024-09-21T10:09:46.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.500797+0000", 2024-09-21T10:09:46.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.98322806100000004 2024-09-21T10:09:46.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:09:46.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.500901+0000", 2024-09-21T10:09:46.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010365999999999999 2024-09-21T10:09:46.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.526 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:09:46.526 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.500998+0000", 2024-09-21T10:09:46.526 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6874999999999997e-05 2024-09-21T10:09:46.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.526 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:09:46.526 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.514319+0000", 2024-09-21T10:09:46.526 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013321308 2024-09-21T10:09:46.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.526 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.527 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.514489+0000", 2024-09-21T10:09:46.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000169879 2024-09-21T10:09:46.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.527 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.527 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.514542+0000", 2024-09-21T10:09:46.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2732000000000003e-05 2024-09-21T10:09:46.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.527 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:09:46.527 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.514585+0000", 2024-09-21T10:09:46.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2910000000000001e-05 2024-09-21T10:09:46.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.528 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:09:46.528 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.514616+0000", 2024-09-21T10:09:46.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1928000000000003e-05 2024-09-21T10:09:46.528 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.528 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.528 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.528 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:253974 10.9 10:92c2e253:::benchmark_data_smithi107_184135_object122408:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-09-21T10:09:46.528 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:59.587453+0000", 2024-09-21T10:09:46.528 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 46.766874778999998, 2024-09-21T10:09:46.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.92737824700000004, 2024-09-21T10:09:46.529 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:09:46.529 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:09:46.529 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:09:46.529 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:09:46.529 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:09:46.529 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:09:46.529 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 253974 2024-09-21T10:09:46.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.529 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:09:46.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:09:46.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.587453+0000", 2024-09-21T10:09:46.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:09:46.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.587453+0000", 2024-09-21T10:09:46.530 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:09:46.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:09:46.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.587455+0000", 2024-09-21T10:09:46.530 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1459999999999999e-06 2024-09-21T10:09:46.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:09:46.531 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.587465+0000", 2024-09-21T10:09:46.531 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7000000000000003e-06 2024-09-21T10:09:46.531 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.531 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.531 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:09:46.531 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.587490+0000", 2024-09-21T10:09:46.531 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4919e-05 2024-09-21T10:09:46.531 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.531 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.531 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:09:46.531 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.587503+0000", 2024-09-21T10:09:46.531 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3254e-05 2024-09-21T10:09:46.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.532 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.532 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.532 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.587603+0000", 2024-09-21T10:09:46.532 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9955000000000001e-05 2024-09-21T10:09:46.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.532 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.532 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:09:46.532 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:59.587617+0000", 2024-09-21T10:09:46.532 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4323e-05 2024-09-21T10:09:46.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.532 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.532 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:09:46.532 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.504271+0000", 2024-09-21T10:09:46.533 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.916654148 2024-09-21T10:09:46.533 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.533 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.533 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:09:46.533 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.504362+0000", 2024-09-21T10:09:46.533 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0561000000000002e-05 2024-09-21T10:09:46.533 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.533 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.533 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:09:46.533 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.504466+0000", 2024-09-21T10:09:46.533 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103568 2024-09-21T10:09:46.533 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.533 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.533 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:09:46.533 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.514454+0000", 2024-09-21T10:09:46.534 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0099884910000000004 2024-09-21T10:09:46.534 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.534 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.534 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.534 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.514667+0000", 2024-09-21T10:09:46.534 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000213282 2024-09-21T10:09:46.534 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.534 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.534 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:09:46.534 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.514704+0000", 2024-09-21T10:09:46.534 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6464999999999997e-05 2024-09-21T10:09:46.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.535 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:09:46.535 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.514792+0000", 2024-09-21T10:09:46.535 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8508e-05 2024-09-21T10:09:46.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:09:46.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:09:46.535 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:09:46.535 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:09:00.514831+0000", 2024-09-21T10:09:46.535 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8927000000000002e-05 2024-09-21T10:09:46.535 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.535 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.535 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.536 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:09:46.536 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:09:46.536 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:09:46.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:46.559 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:46.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:46.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:46.681 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:46.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:46.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:46.782 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:46.808 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:46.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:46.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:46.909 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:46.935 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:46.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:46.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:47.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:47.059 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:47.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:47.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:47.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:47.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:46 smithi129 ceph-mon[88471]: pgmap v881: 109 pgs: 2 creating+peering, 10 unknown, 97 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:47.183 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:46 smithi107 ceph-mon[102221]: pgmap v881: 109 pgs: 2 creating+peering, 10 unknown, 97 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:47.183 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:46 smithi107 ceph-mon[105022]: pgmap v881: 109 pgs: 2 creating+peering, 10 unknown, 97 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:47.187 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:47.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:47.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:47.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:09:47.289 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:47.314 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:47.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:47.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:47.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:47.438 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:47.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:47.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:47.517 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:47.540 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:47.562 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:47.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:47.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:47.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:47.686 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:47.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:47.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:47.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:47.810 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:47.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:47.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:47.912 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:47.937 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:47.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:47.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:48.039 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:48.063 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:48.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:48.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:48.165 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:48.188 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:48.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:48.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:48.290 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:48.311 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:48.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:48.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:48.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:48.433 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:48.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:48.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:48.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:48.558 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:48.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:48.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:48.659 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:48.686 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:48.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:48.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:48.808 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:48.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:48.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:48.909 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:48.910 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T10:09:48.935 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:48.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:48.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:49.036 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:49.059 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:49.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:49.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:49.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:48 smithi107 ceph-mon[105022]: pgmap v882: 109 pgs: 2 creating+peering, 107 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:49.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:48 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/780604119' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:09:49.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:48 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:09:49.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:48 smithi107 ceph-mon[102221]: pgmap v882: 109 pgs: 2 creating+peering, 107 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:49.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:48 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/780604119' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:09:49.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:48 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:09:49.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:49.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:48 smithi129 ceph-mon[88471]: pgmap v882: 109 pgs: 2 creating+peering, 107 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:49.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:48 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/780604119' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:09:49.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:48 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:09:49.183 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:49.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:49.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:49.284 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:49.306 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:49.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:49.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:49.408 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:49.434 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:49.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:49.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:49.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:49.557 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:49.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:49.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:49.658 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:49.684 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:49.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:49.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:49.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:09:49.785 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:49.807 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:49.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:49.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:49.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:49.943 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:49.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:49.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:49.990 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:49 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:09:49.990 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:49 smithi107 ceph-mon[102221]: osdmap e255: 8 total, 8 up, 7 in 2024-09-21T10:09:49.991 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:49 smithi107 ceph-mon[102221]: osdmap e256: 8 total, 8 up, 7 in 2024-09-21T10:09:49.991 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:49 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:09:49.991 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:49 smithi107 ceph-mon[105022]: osdmap e255: 8 total, 8 up, 7 in 2024-09-21T10:09:49.991 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:49 smithi107 ceph-mon[105022]: osdmap e256: 8 total, 8 up, 7 in 2024-09-21T10:09:50.045 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:50.072 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:50.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:50.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:50.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:50.196 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:50.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:50.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:50.215 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:49 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:09:50.215 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:49 smithi129 ceph-mon[88471]: osdmap e255: 8 total, 8 up, 7 in 2024-09-21T10:09:50.215 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:49 smithi129 ceph-mon[88471]: osdmap e256: 8 total, 8 up, 7 in 2024-09-21T10:09:50.297 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:50.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:50.324 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:50.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:50.425 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:50.451 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:50.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:50.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:50.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:50.574 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:50.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:50.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:09:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:09:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:09:50.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:50.696 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:50.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:50.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:50.798 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:50.824 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:50.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:50.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:50.926 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:50.935 INFO:teuthology.orchestra.run.smithi107.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-09-21T10:09:50.952 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:50.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:50.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:50.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T10:09:51.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:51.075 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:51.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:51.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:51.177 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:51.179 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:51.202 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:51.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:51.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:51.306 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:51.317 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:50 smithi107 ceph-mon[102221]: pgmap v884: 109 pgs: 4 peering, 105 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:51.317 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:50 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2761863383' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:09:51.317 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:50 smithi107 ceph-mon[105022]: pgmap v884: 109 pgs: 4 peering, 105 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:51.318 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:50 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2761863383' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:09:51.331 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:51.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:51.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:51.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:50 smithi129 ceph-mon[88471]: pgmap v884: 109 pgs: 4 peering, 105 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:51.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:50 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2761863383' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:09:51.432 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:51.456 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:51.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:51.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:51.558 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:51.580 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:51.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:51.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:51.666 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:51.666 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:51.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:51.704 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:51.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:51.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:51.805 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:51.828 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:51.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:51.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:51.904 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:09:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:09:51] "GET /metrics HTTP/1.1" 200 34408 "" "Prometheus/2.51.0" 2024-09-21T10:09:51.930 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:51.953 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:51.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:51.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:52.054 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:52.078 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:52.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:52.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:52.162 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:51 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2761863383' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:09:52.162 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:51 smithi107 ceph-mon[102221]: osdmap e257: 8 total, 8 up, 7 in 2024-09-21T10:09:52.162 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:51 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2761863383' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:09:52.162 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:51 smithi107 ceph-mon[105022]: osdmap e257: 8 total, 8 up, 7 in 2024-09-21T10:09:52.180 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:52.201 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:52.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:52.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:52.303 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:52.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:52.325 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:52.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:51 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2761863383' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:09:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:51 smithi129 ceph-mon[88471]: osdmap e257: 8 total, 8 up, 7 in 2024-09-21T10:09:52.426 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:52.448 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:52.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:52.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:52.550 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:52.571 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:52.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:52.643 DEBUG:teuthology.orchestra.run.smithi107:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-09-21T10:09:52.645 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:09:52.645 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:09:52.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:52.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:52.696 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:52.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:52.714 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:hints = 1 2024-09-21T10:09:52.714 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:09:52.714 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object prefix: benchmark_data_smithi107_218812 2024-09-21T10:09:52.714 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:09:52.714 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:09:52.796 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:52.825 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:52.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:52.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:52.927 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:52.956 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:52.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:52.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:53.057 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:53.085 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:53.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:53.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:53.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:53.223 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:53.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:53.325 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:53.353 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:53.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:53.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:53.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:52 smithi107 ceph-mon[102221]: pgmap v887: 109 pgs: 4 peering, 105 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:53.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:52 smithi107 ceph-mon[102221]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:09:53.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:52 smithi107 ceph-mon[105022]: pgmap v887: 109 pgs: 4 peering, 105 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:53.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:52 smithi107 ceph-mon[105022]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:09:53.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:52 smithi129 ceph-mon[88471]: pgmap v887: 109 pgs: 4 peering, 105 active+clean; 652 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail 2024-09-21T10:09:53.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:52 smithi129 ceph-mon[88471]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:09:53.456 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:53.482 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:53.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:53.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:53.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:53.622 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:53.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:53.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:53.714 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1 16 3516 3500 218.738 218.75 0.00809264 0.00455588 2024-09-21T10:09:53.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:53.763 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:53.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:53.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:53.865 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:53.892 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:53.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:53.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:53.993 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:54.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:54.033 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:54.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:54.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:54.172 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:54.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:54.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:54.274 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:54.300 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:54.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:54.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:54.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:54.445 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:54.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:54.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:54.547 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:54.574 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:54.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:54.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:54.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:54.701 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:54.713 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:54.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:54.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:54.715 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2 16 6702 6686 208.909 199.125 0.00648641 0.0047731 2024-09-21T10:09:54.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:54.856 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:54.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:54.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:54.958 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:54.986 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:54.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:54.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:55.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:55.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:54 smithi107 ceph-mon[102221]: pgmap v888: 109 pgs: 4 peering, 105 active+clean; 1.7 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 5.7 KiB/s rd, 194 KiB/s wr, 6 op/s; 8.8 KiB/s, 1 objects/s recovering 2024-09-21T10:09:55.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:54 smithi107 ceph-mon[102221]: 1.11 scrub starts 2024-09-21T10:09:55.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:54 smithi107 ceph-mon[102221]: 1.11 scrub ok 2024-09-21T10:09:55.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:54 smithi107 ceph-mon[105022]: pgmap v888: 109 pgs: 4 peering, 105 active+clean; 1.7 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 5.7 KiB/s rd, 194 KiB/s wr, 6 op/s; 8.8 KiB/s, 1 objects/s recovering 2024-09-21T10:09:55.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:54 smithi107 ceph-mon[105022]: 1.11 scrub starts 2024-09-21T10:09:55.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:54 smithi107 ceph-mon[105022]: 1.11 scrub ok 2024-09-21T10:09:55.124 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:55.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:55.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:55.226 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:55.249 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:55.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:55.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:55.351 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:55.379 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:55.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:55.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:55.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:54 smithi129 ceph-mon[88471]: pgmap v888: 109 pgs: 4 peering, 105 active+clean; 1.7 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 5.7 KiB/s rd, 194 KiB/s wr, 6 op/s; 8.8 KiB/s, 1 objects/s recovering 2024-09-21T10:09:55.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:54 smithi129 ceph-mon[88471]: 1.11 scrub starts 2024-09-21T10:09:55.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:54 smithi129 ceph-mon[88471]: 1.11 scrub ok 2024-09-21T10:09:55.481 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:55.518 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:55.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:55.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:55.620 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:55.647 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:55.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:55.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:55.714 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3 16 9626 9610 200.184 182.75 0.00714973 0.0049853 2024-09-21T10:09:55.748 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:55.777 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:55.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:55.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:55.878 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:55.905 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:55.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:55.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:56.006 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:56.034 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:56.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:56.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:56.136 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:56.162 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:56.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:56.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:56.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:56.303 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:56.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:56.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:56.405 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:56.429 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:56.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:56.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:56.531 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:56.586 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:56.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:56.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:56.687 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:56.712 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:56.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:56.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:56.714 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 4 16 12314 12298 192.135 168 0.00842011 0.0051973 2024-09-21T10:09:56.814 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:56.838 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:56.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:56.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:56.940 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:56.965 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:56.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:56.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:56.989 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:56 smithi107 ceph-mon[102221]: pgmap v889: 109 pgs: 109 active+clean; 53 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 5.4 KiB/s rd, 8.3 MiB/s wr, 136 op/s; 8.4 KiB/s, 1 objects/s recovering 2024-09-21T10:09:56.989 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:56.989 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:56.989 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:56 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1878071831' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:56.990 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:56 smithi107 ceph-mon[105022]: pgmap v889: 109 pgs: 109 active+clean; 53 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 5.4 KiB/s rd, 8.3 MiB/s wr, 136 op/s; 8.4 KiB/s, 1 objects/s recovering 2024-09-21T10:09:56.990 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:56.990 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:56.990 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:56 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1878071831' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:57.067 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:57.094 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:57.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:57.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:57.196 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:57.220 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:57.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:57.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:57.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:57.368 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:57.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:57.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:57.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:56 smithi129 ceph-mon[88471]: pgmap v889: 109 pgs: 109 active+clean; 53 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 5.4 KiB/s rd, 8.3 MiB/s wr, 136 op/s; 8.4 KiB/s, 1 objects/s recovering 2024-09-21T10:09:57.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:57.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:09:57.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:56 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1878071831' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:09:57.470 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:57.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:57.492 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:57.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:57.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:57.630 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:57.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:57.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:57.715 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 5 16 14383 14367 179.568 129.312 0.00640349 0.00556206 2024-09-21T10:09:57.731 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:57.767 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:57.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:57.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:57.869 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:09:57.896 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:57.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:57.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:09:57.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:09:58.027 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:58.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:58.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:09:58.128 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:58.154 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:58.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:58.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:58.256 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:09:58.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:58.282 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:58.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:09:58.383 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:58.411 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:58.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:58.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:58.513 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:58.540 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:58.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:58.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:58.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:09:58.678 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:58.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:58.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:09:58.715 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 6 16 16766 16750 174.454 148.938 0.00504623 0.00572581 2024-09-21T10:09:58.781 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:58.807 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:58.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:58.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:58.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:58.953 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:58.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:58.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:58.981 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:09:58.981 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:09:59.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:09:59.094 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:59.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:59.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:09:59.171 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:09:59.171 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:09:59.171 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:09:59.172 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:09:59.172 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:09:59.172 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:09:59.172 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:09:59.172 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:09:59.172 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:09:59.172 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:09:59.172 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:09:59.172 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:09:59.172 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:09:59.172 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:09:59.172 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:09:59.173 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:09:59.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:59.233 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:59.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:59.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:59.335 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:09:59.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:09:58 smithi107 ceph-mon[102221]: pgmap v890: 109 pgs: 109 active+clean; 561 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 73 B/s, 1 objects/s recovering 2024-09-21T10:09:59.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:09:58 smithi107 ceph-mon[105022]: pgmap v890: 109 pgs: 109 active+clean; 561 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 73 B/s, 1 objects/s recovering 2024-09-21T10:09:59.363 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:59.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:59.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:09:59.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:09:58 smithi129 ceph-mon[88471]: pgmap v890: 109 pgs: 109 active+clean; 561 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 73 B/s, 1 objects/s recovering 2024-09-21T10:09:59.465 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:09:59.491 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:09:59.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:59.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:09:59.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:59.631 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:59.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:59.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:59.715 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 7 15 19196 19181 171.236 151.938 0.00719392 0.00583353 2024-09-21T10:09:59.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:59.769 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:59.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:59.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:09:59.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:09:59.907 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:09:59.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:09:59.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:00.009 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:00.046 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:00.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:00.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:00.149 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:00.175 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:00.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:00.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:00.277 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:00.313 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:00.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:00.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:00.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:00.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:00.452 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:00.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:00.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:00.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:00.591 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:00.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:10:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:10:00.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:10:00.693 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:00.715 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 8 16 21504 21488 167.853 144.188 0.0021567 0.00595088 2024-09-21T10:10:00.717 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:00.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:00.819 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:00.845 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:00.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:00.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:00.947 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:00.972 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:00.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:00.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:00.981 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:10:01.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:01.129 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:01.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:01.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:01.231 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:01.257 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:01.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:01.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:01.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:00 smithi107 ceph-mon[102221]: pgmap v891: 109 pgs: 109 active+clean; 831 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 89 MiB/s wr, 1.42k op/s; 63 B/s, 0 objects/s recovering 2024-09-21T10:10:01.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:00 smithi107 ceph-mon[102221]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-21T10:10:01.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:00 smithi107 ceph-mon[102221]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-21T10:10:01.300 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:00 smithi107 ceph-mon[105022]: pgmap v891: 109 pgs: 109 active+clean; 831 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 89 MiB/s wr, 1.42k op/s; 63 B/s, 0 objects/s recovering 2024-09-21T10:10:01.300 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:00 smithi107 ceph-mon[105022]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-21T10:10:01.300 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:00 smithi107 ceph-mon[105022]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-21T10:10:01.359 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:01.384 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:01.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:01.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:01.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:00 smithi129 ceph-mon[88471]: pgmap v891: 109 pgs: 109 active+clean; 831 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 89 MiB/s wr, 1.42k op/s; 63 B/s, 0 objects/s recovering 2024-09-21T10:10:01.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:00 smithi129 ceph-mon[88471]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-21T10:10:01.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:00 smithi129 ceph-mon[88471]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-21T10:10:01.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:01.525 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:01.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:01.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:01.627 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:01.663 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:01.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:01.715 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 9 15 23467 23452 162.841 122.75 0.00340202 0.00613591 2024-09-21T10:10:01.765 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:01.791 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:01.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:01.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:01.893 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:01.915 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:01.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:01.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:01.970 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T10:10:02.016 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:02.044 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:02.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:02.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:02.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:10:02.084 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:01 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2463413956' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:10:02.084 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:01 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2463413956' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:10:02.085 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:10:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:10:01] "GET /metrics HTTP/1.1" 200 36794 "" "Prometheus/2.51.0" 2024-09-21T10:10:02.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:02.188 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:02.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:02.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:02.289 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:02.310 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:02.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:02.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:02.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:02.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:01 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2463413956' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:10:02.449 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:02.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:02.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:02.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:02.587 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:02.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:02.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:02.689 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:02.710 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:02.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:02.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:02.715 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 10 16 25190 25174 157.319 107.625 0.00137207 0.00633396 2024-09-21T10:10:02.812 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:02.850 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:02.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:02.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:02.952 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:02.979 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:02.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:03.081 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:03.106 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:03.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:03.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:03.208 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:03.230 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:03.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:03.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:03.332 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:03.357 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:03.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:03.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:03.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:02 smithi107 ceph-mon[102221]: pgmap v892: 109 pgs: 109 active+clean; 831 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 81 MiB/s wr, 1.29k op/s; 57 B/s, 0 objects/s recovering 2024-09-21T10:10:03.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:02 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2463413956' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:10:03.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:02 smithi107 ceph-mon[102221]: osdmap e258: 8 total, 8 up, 7 in 2024-09-21T10:10:03.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:02 smithi107 ceph-mon[105022]: pgmap v892: 109 pgs: 109 active+clean; 831 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 81 MiB/s wr, 1.29k op/s; 57 B/s, 0 objects/s recovering 2024-09-21T10:10:03.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:02 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2463413956' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:10:03.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:02 smithi107 ceph-mon[105022]: osdmap e258: 8 total, 8 up, 7 in 2024-09-21T10:10:03.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:02 smithi129 ceph-mon[88471]: pgmap v892: 109 pgs: 109 active+clean; 831 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 81 MiB/s wr, 1.29k op/s; 57 B/s, 0 objects/s recovering 2024-09-21T10:10:03.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:02 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2463413956' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:10:03.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:02 smithi129 ceph-mon[88471]: osdmap e258: 8 total, 8 up, 7 in 2024-09-21T10:10:03.459 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:03.485 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:03.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:03.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:03.587 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:03.609 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:03.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:03.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:03.710 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:03.715 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 11 16 26862 26846 152.516 104.5 0.00447077 0.00655123 2024-09-21T10:10:03.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:03.749 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:03.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:03.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:03.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:03.889 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:03.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:03.989 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:04.013 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:04.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:04.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:04.115 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:04.142 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:04.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:04.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:04.244 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:04.279 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:04.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:04.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:04.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:03 smithi107 ceph-mon[105022]: pgmap v894: 109 pgs: 109 active+clean; 1.4 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-09-21T10:10:04.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:03 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2741135811' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:10:04.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:03 smithi107 ceph-mon[102221]: pgmap v894: 109 pgs: 109 active+clean; 1.4 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-09-21T10:10:04.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:03 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2741135811' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:10:04.381 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:04.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:04.403 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:04.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:04.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:03 smithi129 ceph-mon[88471]: pgmap v894: 109 pgs: 109 active+clean; 1.4 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 139 MiB/s wr, 2.23k op/s 2024-09-21T10:10:04.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:03 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2741135811' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:10:04.504 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:04.529 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:04.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:04.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:04.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:04.670 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:04.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:04.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:04.711 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:10:04.715 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 12 16 28687 28671 149.311 114.062 0.00703227 0.00669196 2024-09-21T10:10:04.772 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:04.795 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:04.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:04.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:04.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:04.935 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:04.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:04.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:05.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:05.071 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:05.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:05.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:05.172 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:05.199 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:05.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:05.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:05.300 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:05.328 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:05.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:05.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:05.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:04 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:10:05.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:04 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:10:05.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:04 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2741135811' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:10:05.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:04 smithi107 ceph-mon[102221]: osdmap e259: 8 total, 8 up, 7 in 2024-09-21T10:10:05.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:04 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:10:05.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:04 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:10:05.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:04 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2741135811' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:10:05.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:04 smithi107 ceph-mon[105022]: osdmap e259: 8 total, 8 up, 7 in 2024-09-21T10:10:05.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:04 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:10:05.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:04 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:10:05.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:04 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2741135811' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:10:05.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:04 smithi129 ceph-mon[88471]: osdmap e259: 8 total, 8 up, 7 in 2024-09-21T10:10:05.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:05.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:05.468 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:05.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:05.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:05.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:05.591 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:05.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:05.692 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:05.716 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 13 16 30203 30187 145.113 94.75 0.00476101 0.00688521 2024-09-21T10:10:05.719 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:05.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:05.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:05.821 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:05.848 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:05.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:05.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:05.951 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:05.974 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:05.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:05.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:06.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:06.114 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:06.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:06.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:06.216 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:06.237 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:06.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:06.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:06.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:06.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:06 smithi107 ceph-mon[102221]: pgmap v896: 109 pgs: 109 active+clean; 1.5 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-09-21T10:10:06.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:06 smithi107 ceph-mon[105022]: pgmap v896: 109 pgs: 109 active+clean; 1.5 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-09-21T10:10:06.372 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:06.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:06.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:06.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:06 smithi129 ceph-mon[88471]: pgmap v896: 109 pgs: 109 active+clean; 1.5 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-09-21T10:10:06.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:06.501 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:06.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:06.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:06.603 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:06.631 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:06.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:06.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:06.716 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 14 16 31883 31867 142.247 105 0.00854619 0.00702471 2024-09-21T10:10:06.733 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:06.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:06.760 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:06.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:06.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:06.882 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:06.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:06.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:06.984 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:07.013 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:07.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:07.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:07.115 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:07.137 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:07.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:07.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:07.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:07.277 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:07.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:07.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:07.379 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:07.415 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:07.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:07.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:07.518 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:07.540 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:07.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:07.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:07.641 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:07.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:07.676 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:07.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:07.716 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 15 16 33234 33218 138.393 84.4375 0.0110147 0.00722027 2024-09-21T10:10:07.777 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:07.798 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:07.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:07.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:07.899 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:07.937 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:07.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:07.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:08.038 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:08.065 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:08.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:08.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:08.203 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:08.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:08.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:08.335 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:08.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:08.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:08.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:08.459 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:08.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:08.560 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:08.585 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:08.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:08.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:08.599 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:08 smithi107 ceph-mon[102221]: pgmap v897: 109 pgs: 109 active+clean; 1.8 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-09-21T10:10:08.600 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:08 smithi107 ceph-mon[105022]: pgmap v897: 109 pgs: 109 active+clean; 1.8 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-09-21T10:10:08.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:08 smithi129 ceph-mon[88471]: pgmap v897: 109 pgs: 109 active+clean; 1.8 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 131 MiB/s wr, 2.09k op/s 2024-09-21T10:10:08.686 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:08.711 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:08.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:08.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:08.716 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 16 16 34323 34307 133.997 68.0625 0.00555069 0.00745722 2024-09-21T10:10:08.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:08.850 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:08.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:08.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:08.952 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:08.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:08.988 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:08.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:09.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:09.125 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:09.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:09.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:09.227 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:09.252 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:09.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:09.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:09.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:09.394 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:09.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:09.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:09.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:09.517 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:09.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:09.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:09.619 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:09.643 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:09.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:09.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:09.716 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 17 16 35861 35845 131.769 96.125 0.0768432 0.00758239 2024-09-21T10:10:09.744 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:09.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:09.782 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:09.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:09.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:09.921 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:09.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:10.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:10.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:10.056 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:10.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:10.157 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:10.183 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:10.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:10.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:10.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:10.325 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:10.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:10.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:10.427 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:10.454 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:10.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:10.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:10.471 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:10 smithi129 ceph-mon[88471]: pgmap v898: 109 pgs: 109 active+clean; 2.0 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-09-21T10:10:10.471 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:10:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:10:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:10:10.556 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:10.593 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:10.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:10.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:10.694 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:10.716 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 18 16 37617 37601 130.545 109.75 0.00207384 0.00765159 2024-09-21T10:10:10.722 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:10.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:10.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:10.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:10.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:10 smithi107 ceph-mon[102221]: pgmap v898: 109 pgs: 109 active+clean; 2.0 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-09-21T10:10:10.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:10 smithi107 ceph-mon[105022]: pgmap v898: 109 pgs: 109 active+clean; 2.0 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 149 MiB/s wr, 2.39k op/s 2024-09-21T10:10:10.862 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:10.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:10.964 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:11.008 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:11.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:11.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:11.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:11.149 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:11.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:11.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:11.250 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:11.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:11.276 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:11.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:11.377 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:11.415 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:11.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:11.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:11.517 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:11.559 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:11.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:11.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:11.661 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:11.715 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:11.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:11.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:11.718 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 19 16 39225 39209 128.946 100.5 0.0070522 0.00775017 2024-09-21T10:10:11.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:11.855 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:11.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:11.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:11.957 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:12.005 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:12.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:12.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:12.044 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:10:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:10:11] "GET /metrics HTTP/1.1" 200 36800 "" "Prometheus/2.51.0" 2024-09-21T10:10:12.106 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:12.134 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:12.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:12.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:12.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:12.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:12.276 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:12.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:12 smithi107 ceph-mon[102221]: pgmap v899: 109 pgs: 109 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T10:10:12.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:12.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:12.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:12 smithi107 ceph-mon[105022]: pgmap v899: 109 pgs: 109 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T10:10:12.377 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:12.405 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:12.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:12.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:12.506 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:12.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T10:10:12.531 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:12.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:12.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:12.537 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:12.541 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:12.632 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:12.657 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:12.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:12.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:12 smithi129 ceph-mon[88471]: pgmap v899: 109 pgs: 109 active+clean; 2.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T10:10:12.720 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:10:12.718671+0000 min lat: 0.00109786 max lat: 0.204001 avg lat: 0.00782053 2024-09-21T10:10:12.720 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:10:12.720 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 20 16 40915 40899 127.775 105.625 0.0121836 0.00782053 2024-09-21T10:10:12.759 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:12.785 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:12.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:12.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:12.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:12.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:12.943 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:12.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:13.044 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:13.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:13.080 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:13.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:13.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:13.224 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:13.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:13.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:13.326 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:13.351 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:13.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:13.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:13.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:13.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:13.496 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:13.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:13.597 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:13.628 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:13.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:13.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:13.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:10:13.720 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 21 16 42383 42367 126.057 91.75 0.00603925 0.00792645 2024-09-21T10:10:13.729 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:13.765 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:13.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:13.867 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:13.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:13.913 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:13.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:14.013 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:14.041 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:14.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:14.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:14.143 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:14.171 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:14.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:14.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:14.273 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:14.302 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:14.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:14.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:14.404 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:14.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:14 smithi129 ceph-mon[88471]: pgmap v900: 109 pgs: 109 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:10:14.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:14 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:10:14.429 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:14.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:14.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:14.531 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:14.557 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:14.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:14.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:14 smithi107 ceph-mon[105022]: pgmap v900: 109 pgs: 109 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:10:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:14 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:10:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:14 smithi107 ceph-mon[102221]: pgmap v900: 109 pgs: 109 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 101 MiB/s wr, 1.62k op/s 2024-09-21T10:10:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:14 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:10:14.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:14.680 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:14.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:14.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:14.720 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 22 16 43564 43548 123.682 73.8125 0.15573 0.0080464 2024-09-21T10:10:14.782 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:14.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:14.806 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:14.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:14.907 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:14.931 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:14.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:14.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:15.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:15.071 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:15.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:15.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:15.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:15.200 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:15.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:15.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:15.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:15.340 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:15.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:15.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:15.443 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:15.474 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:15.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:15.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:15.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:15.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:15.617 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:15.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:15.718 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:15.720 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 23 16 45076 45060 122.413 94.5 0.0075058 0.00816352 2024-09-21T10:10:15.746 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:15.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:15.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:15.847 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:15.873 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:15.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:15.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:15.975 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:16.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:16.005 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:16.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:16.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:16.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:16.143 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:16.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:16.244 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:16.271 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:16.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:16.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:16.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:16.422 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:16.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:16.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:16.524 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:16.551 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:16.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:16.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:16.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:16 smithi107 ceph-mon[102221]: pgmap v901: 109 pgs: 109 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-21T10:10:16.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:16 smithi107 ceph-mon[105022]: pgmap v901: 109 pgs: 109 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-21T10:10:16.612 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:16 smithi129 ceph-mon[88471]: pgmap v901: 109 pgs: 109 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-21T10:10:16.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:16.696 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:16.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:16.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:16.720 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 24 16 46695 46679 121.528 101.188 0.0163739 0.00822282 2024-09-21T10:10:16.799 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:16.837 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:16.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:16.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:16.878 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:16.939 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:16.964 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:16.969 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:16.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:16.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:17.071 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:17.098 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:17.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:17.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:17.200 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:17.223 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:17.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:17.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:17.325 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:17.368 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:17.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:17.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:17.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:17.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:17.511 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:17.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:17.611 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:17.638 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:17.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:17.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:17.720 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 25 16 48099 48083 120.177 87.75 0.0081694 0.00831532 2024-09-21T10:10:17.740 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:17.768 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:17.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:17.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:17.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:17.891 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:17.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:17.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:17.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:18.026 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:18.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:18.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:18.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:18.164 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:18.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:18.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:18.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:18.307 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:18.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:18.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:18.409 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:18.417 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:18 smithi129 ceph-mon[88471]: pgmap v902: 109 pgs: 109 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-21T10:10:18.435 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:18.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:18.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:18.537 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:18.563 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:18.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:18.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:18.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:18 smithi107 ceph-mon[102221]: pgmap v902: 109 pgs: 109 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-21T10:10:18.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:18 smithi107 ceph-mon[105022]: pgmap v902: 109 pgs: 109 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-21T10:10:18.665 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:18.695 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:18.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:18.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:18.722 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 26 16 49538 49522 119.014 89.9375 0.0106675 0.00839729 2024-09-21T10:10:18.741 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:10:18.741 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:10:18.776 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:10:18.777 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:10:18.796 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:18.809 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:10:18.809 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:10:18.822 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:18.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:18.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:18.849 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:10:18.849 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:10:18.880 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:10:18.880 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:10:18.910 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:10:18.910 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:10:18.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:18.945 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:10:18.945 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:10:18.963 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:18.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:18.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:18.990 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:10:18.990 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:10:19.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:19.102 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:19.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:19.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:19.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:19.253 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:19.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:19.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:19.355 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:19.384 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:19.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:19.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:19.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:19.510 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:19.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:19.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:19.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:19 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3767694334' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:19.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:19 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:19.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:19 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:19.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:19 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3767694334' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:19.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:19 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:19.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:19 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:19.612 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:19.632 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:19.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:19.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:19 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3767694334' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:19 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:19 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:19.721 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 27 16 50710 50694 117.319 73.25 0.00431977 0.00847918 2024-09-21T10:10:19.734 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:19.756 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:19.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:19.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:19.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:19.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:19.896 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:19.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:19.997 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:20.022 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:20.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:20.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:20.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:20.159 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:20.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:20.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:20.261 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:20.288 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:20.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:20.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:20.390 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:20.416 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:20.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:20.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:20.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:20.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:20.556 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:20.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:20.657 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:10:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:10:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:10:20.682 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:20.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:20.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:20.721 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 28 16 52066 52050 116.155 84.75 0.00979046 0.0086041 2024-09-21T10:10:20.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:20.820 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:20.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:20.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:20.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:20 smithi107 ceph-mon[102221]: pgmap v903: 109 pgs: 109 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T10:10:20.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:20 smithi107 ceph-mon[105022]: pgmap v903: 109 pgs: 109 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T10:10:20.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:20.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:20.959 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:20.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:21.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:21.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:21.098 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:21.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:21.099 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:10:21.099 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:10:21.099 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-09-21T10:10:21.099 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-09-21T10:10:21.100 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:10:21.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:20 smithi129 ceph-mon[88471]: pgmap v903: 109 pgs: 109 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T10:10:21.199 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:21.224 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:21.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:21.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:21.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:21.361 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:21.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:21.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:21.463 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:21.489 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:21.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:21.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:21.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:21.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:21.634 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:21.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:21.721 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 29 16 53487 53471 115.212 88.8125 0.0098165 0.00867391 2024-09-21T10:10:21.735 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:21.763 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:21.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:21.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:21.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:21.904 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:21.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:21.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:22.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:22.046 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:22.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:22.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:22.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:10:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:10:21] "GET /metrics HTTP/1.1" 200 36800 "" "Prometheus/2.51.0" 2024-09-21T10:10:22.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:22.186 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:22.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:22.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:22.288 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:22.313 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:22.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:22.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:22.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:22.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:22.456 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:22.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:22.557 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:22.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:22.583 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:22.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:22.684 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:22.711 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:22.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:22.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:22.721 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 30 16 54915 54899 114.347 89.25 0.00402514 0.00873922 2024-09-21T10:10:22.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:22.854 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:22.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:22.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:22.956 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:22.979 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:22.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:22.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:23.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:10:23.074 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:22 smithi107 ceph-mon[102221]: pgmap v904: 109 pgs: 109 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-21T10:10:23.074 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:22 smithi107 ceph-mon[105022]: pgmap v904: 109 pgs: 109 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-21T10:10:23.080 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:23.106 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:23.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:23.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:23.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:22 smithi129 ceph-mon[88471]: pgmap v904: 109 pgs: 109 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-21T10:10:23.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:23.256 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:23.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:23.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:23.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:23.397 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:23.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:23.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:23.499 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:23.524 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:23.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:23.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:23.626 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:23.652 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:23.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:23.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:23.721 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 31 15 56176 56161 113.202 78.875 0.011272 0.00882873 2024-09-21T10:10:23.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:23.791 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:23.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:23.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:23.893 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:23.917 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:23.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:23.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:24.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:24.062 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:24.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:24.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:24.163 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:24.190 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:24.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:24.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:24.293 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:24.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:24.332 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:24.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:24.434 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:24.457 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:24.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:24.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:24.559 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:24.582 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:24.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:24.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:24.684 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:24.708 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:24.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:24.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:24.722 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 32 16 57456 57440 112.163 79.9375 0.0221304 0.00891064 2024-09-21T10:10:24.810 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:24.835 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:24.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:24.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:24.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:24 smithi107 ceph-mon[102221]: pgmap v905: 109 pgs: 109 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-21T10:10:24.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:24 smithi107 ceph-mon[105022]: pgmap v905: 109 pgs: 109 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-21T10:10:24.936 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:24.961 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:24.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:24.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:25.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:25.101 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:25.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:25.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:24 smithi129 ceph-mon[88471]: pgmap v905: 109 pgs: 109 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-21T10:10:25.203 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:25.227 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:25.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:25.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:25.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:25.367 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:25.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:25.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:25.469 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:25.493 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:25.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:25.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:25.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:25.635 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:25.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:25.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:25.721 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 33 16 58651 58635 111.027 74.6875 0.0121045 0.00900102 2024-09-21T10:10:25.737 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:25.776 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:25.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:25.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:25.878 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:25.899 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:25.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:25.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:26.001 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:26.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:26.038 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:26.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:26.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:26.183 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:26.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:26.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:26.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:26.306 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:26.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:26.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:26.407 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:26.430 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:26.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:26.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:26.517 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:10:26.532 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:26.557 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:26.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:26.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:26.658 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:26.683 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:26.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:26.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:26.721 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 34 16 59792 59776 109.859 71.3125 0.0112712 0.00909715 2024-09-21T10:10:26.785 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:26.820 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:26 smithi107 ceph-mon[102221]: pgmap v906: 109 pgs: 109 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-21T10:10:26.820 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [1, 7]} : dispatch 2024-09-21T10:10:26.820 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [6, 5]} : dispatch 2024-09-21T10:10:26.820 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [6, 5]} : dispatch 2024-09-21T10:10:26.820 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [7, 2]} : dispatch 2024-09-21T10:10:26.820 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [1, 7]} : dispatch 2024-09-21T10:10:26.820 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]} : dispatch 2024-09-21T10:10:26.820 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-21T10:10:26.820 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:26.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:26 smithi107 ceph-mon[105022]: pgmap v906: 109 pgs: 109 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-21T10:10:26.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [1, 7]} : dispatch 2024-09-21T10:10:26.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [6, 5]} : dispatch 2024-09-21T10:10:26.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [6, 5]} : dispatch 2024-09-21T10:10:26.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [7, 2]} : dispatch 2024-09-21T10:10:26.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [1, 7]} : dispatch 2024-09-21T10:10:26.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]} : dispatch 2024-09-21T10:10:26.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-21T10:10:26.821 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:26.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:26.850 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:26.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:26.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:26.994 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:26.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:26.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:27.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:27.134 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:27.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:27.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:26 smithi129 ceph-mon[88471]: pgmap v906: 109 pgs: 109 active+clean; 3.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-21T10:10:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [1, 7]} : dispatch 2024-09-21T10:10:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [6, 5]} : dispatch 2024-09-21T10:10:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [6, 5]} : dispatch 2024-09-21T10:10:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [7, 2]} : dispatch 2024-09-21T10:10:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [1, 7]} : dispatch 2024-09-21T10:10:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]} : dispatch 2024-09-21T10:10:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-21T10:10:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:27.236 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:27.264 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:27.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:27.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:27.365 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:27.392 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:27.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:27.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:27.494 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:27.523 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:27.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:27.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:27.625 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:27.650 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:27.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:27.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:27.721 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 35 16 61116 61100 109.084 82.75 0.00748999 0.00916259 2024-09-21T10:10:27.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:27.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:27.831 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:27.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:27.932 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:27.957 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:27.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:27.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:28.041 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:28.058 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:28.082 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:28.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:28.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:28.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [1, 7]}]': finished 2024-09-21T10:10:28.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [6, 5]}]': finished 2024-09-21T10:10:28.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [6, 5]}]': finished 2024-09-21T10:10:28.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [7, 2]}]': finished 2024-09-21T10:10:28.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [1, 7]}]': finished 2024-09-21T10:10:28.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]': finished 2024-09-21T10:10:28.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-09-21T10:10:28.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:27 smithi107 ceph-mon[102221]: osdmap e260: 8 total, 8 up, 7 in 2024-09-21T10:10:28.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [1, 7]}]': finished 2024-09-21T10:10:28.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [6, 5]}]': finished 2024-09-21T10:10:28.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [6, 5]}]': finished 2024-09-21T10:10:28.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [7, 2]}]': finished 2024-09-21T10:10:28.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [1, 7]}]': finished 2024-09-21T10:10:28.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]': finished 2024-09-21T10:10:28.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-09-21T10:10:28.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:27 smithi107 ceph-mon[105022]: osdmap e260: 8 total, 8 up, 7 in 2024-09-21T10:10:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [1, 7]}]': finished 2024-09-21T10:10:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [6, 5]}]': finished 2024-09-21T10:10:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [6, 5]}]': finished 2024-09-21T10:10:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [7, 2]}]': finished 2024-09-21T10:10:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [1, 7]}]': finished 2024-09-21T10:10:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]': finished 2024-09-21T10:10:28.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-09-21T10:10:28.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:27 smithi129 ceph-mon[88471]: osdmap e260: 8 total, 8 up, 7 in 2024-09-21T10:10:28.184 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:28.207 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:28.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:28.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:28.309 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:28.374 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:28.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:28.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:28.475 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:28.527 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:28.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:28.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:28.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:28.673 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:28.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:28.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:28.721 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 36 16 62216 62200 107.964 68.75 0.00957126 0.00925603 2024-09-21T10:10:28.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:28.818 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:10:28.828 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:28.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:28.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:28.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:28.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:28 smithi107 ceph-mon[102221]: pgmap v908: 109 pgs: 2 peering, 107 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-21T10:10:28.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:28 smithi107 ceph-mon[102221]: osdmap e261: 8 total, 8 up, 7 in 2024-09-21T10:10:28.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:28 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/33598342' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:10:28.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:28 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:10:28.971 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:28 smithi107 ceph-mon[105022]: pgmap v908: 109 pgs: 2 peering, 107 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-21T10:10:28.971 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:28 smithi107 ceph-mon[105022]: osdmap e261: 8 total, 8 up, 7 in 2024-09-21T10:10:28.971 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:28 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/33598342' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:10:28.971 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:28 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:10:28.978 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:28.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:28.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:29.080 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:29.105 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:29.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:29.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:29.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:28 smithi129 ceph-mon[88471]: pgmap v908: 109 pgs: 2 peering, 107 active+clean; 3.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-21T10:10:29.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:28 smithi129 ceph-mon[88471]: osdmap e261: 8 total, 8 up, 7 in 2024-09-21T10:10:29.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:28 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/33598342' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:10:29.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:28 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:10:29.207 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:29.247 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:29.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:29.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:29.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:29.382 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:29.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:29.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:29.484 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:29.507 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:29.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:29.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:29.524 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:29.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:29.660 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:29.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:29.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:29.722 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 37 16 63383 63367 107.017 72.9375 0.00729274 0.00933925 2024-09-21T10:10:29.762 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:29.785 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:29.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:29.836 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T10:10:29.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:29.932 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:29.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:29.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:30.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:30.071 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:30.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:30.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:30.073 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:29 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:10:30.073 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:29 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:10:30.073 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:29 smithi107 ceph-mon[102221]: osdmap e262: 8 total, 8 up, 7 in 2024-09-21T10:10:30.073 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:29 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2621864634' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:10:30.073 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:29 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:10:30.073 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:29 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:10:30.073 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:29 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:10:30.073 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:29 smithi107 ceph-mon[105022]: osdmap e262: 8 total, 8 up, 7 in 2024-09-21T10:10:30.074 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:29 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2621864634' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:10:30.074 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:29 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:10:30.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:29 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:10:30.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:29 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:10:30.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:29 smithi129 ceph-mon[88471]: osdmap e262: 8 total, 8 up, 7 in 2024-09-21T10:10:30.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:29 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2621864634' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:10:30.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:29 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:10:30.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:30.209 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:30.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:30.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:30.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:30.335 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:30.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:30.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:30.436 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:30.472 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:30.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:30.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:30.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:30.614 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:30.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:30.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:10:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:10:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:10:30.716 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:30.722 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 38 16 64647 64631 106.279 79 0.00868262 0.00940331 2024-09-21T10:10:30.755 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:30.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:30.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:30.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:30.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:30.901 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:30.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:31.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:31.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:31.046 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:31.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:31.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:30 smithi107 ceph-mon[105022]: pgmap v911: 109 pgs: 2 peering, 107 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T10:10:31.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:30 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:10:31.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:30 smithi107 ceph-mon[105022]: osdmap e263: 8 total, 8 up, 7 in 2024-09-21T10:10:31.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:30 smithi107 ceph-mon[102221]: pgmap v911: 109 pgs: 2 peering, 107 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T10:10:31.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:30 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:10:31.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:30 smithi107 ceph-mon[102221]: osdmap e263: 8 total, 8 up, 7 in 2024-09-21T10:10:31.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:31.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:30 smithi129 ceph-mon[88471]: pgmap v911: 109 pgs: 2 peering, 107 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T10:10:31.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:30 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:10:31.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:30 smithi129 ceph-mon[88471]: osdmap e263: 8 total, 8 up, 7 in 2024-09-21T10:10:31.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:31.186 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:31.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:31.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:31.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:31.323 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:31.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:31.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:31.478 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:31.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:31.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:31.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:31.621 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:31.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:31.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:31.722 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 39 16 65786 65770 105.379 71.1875 0.00203559 0.00947332 2024-09-21T10:10:31.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:31.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:31.752 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:31.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:31.853 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:31.889 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:31.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:31.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:31.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:32.028 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:32.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:32.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:10:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:10:31] "GET /metrics HTTP/1.1" 200 36800 "" "Prometheus/2.51.0" 2024-09-21T10:10:32.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:10:32.130 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:32.155 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:32.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:32.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:32.256 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:32.283 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:32.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:32.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:32.385 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:32.408 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:32.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:32.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:32.509 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:32.531 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:32.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:32.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:32.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:32.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:32.680 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:32.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:32.722 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:10:32.721441+0000 min lat: 0.00109786 max lat: 0.260771 avg lat: 0.00955732 2024-09-21T10:10:32.722 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:10:32.722 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 40 16 66962 66946 104.582 73.5 0.0196946 0.00955732 2024-09-21T10:10:32.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:32.817 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:32.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:32.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:32.959 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:32.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:32.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:33.061 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:33.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T10:10:33.085 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:33.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:33.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:33.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:32 smithi107 ceph-mon[105022]: pgmap v913: 109 pgs: 2 peering, 107 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 38 MiB/s wr, 604 op/s 2024-09-21T10:10:33.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:32 smithi107 ceph-mon[102221]: pgmap v913: 109 pgs: 2 peering, 107 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 38 MiB/s wr, 604 op/s 2024-09-21T10:10:33.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:32 smithi129 ceph-mon[88471]: pgmap v913: 109 pgs: 2 peering, 107 active+clean; 3.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 38 MiB/s wr, 604 op/s 2024-09-21T10:10:33.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:33.247 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:33.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:33.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:33.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:33.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:33.393 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:33.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:33.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:33.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:33.538 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:33.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:33.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:33.690 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:33.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:33.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:33.722 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 41 16 68131 68115 103.813 73.0625 0.00566218 0.00962806 2024-09-21T10:10:33.792 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:33.816 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:33.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:33.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:33.918 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:33.943 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:33.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:33.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:34.045 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:34.069 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:34.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:34.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:34.171 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:34.197 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:34.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:34.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:34.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:34.354 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:34.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:34.455 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:34.479 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:34.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:34.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:34.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:34.582 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:10:34.582 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:10:34.616 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:10:34.616 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:10:34.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:34.639 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:34.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:34.650 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:10:34.650 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:10:34.681 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:10:34.682 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:10:34.714 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:10:34.714 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:10:34.723 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 42 16 69255 69239 103.014 70.25 0.011113 0.00970234 2024-09-21T10:10:34.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:34.744 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:10:34.744 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:10:34.776 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:10:34.776 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:10:34.799 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:34.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:34.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:34.817 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:10:34.817 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:10:34.900 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:34.926 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:34.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:34.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:35.027 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:35.047 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:34 smithi107 ceph-mon[102221]: pgmap v914: 109 pgs: 109 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-21T10:10:35.047 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:34 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3719393509' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:35.048 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:34 smithi107 ceph-mon[105022]: pgmap v914: 109 pgs: 109 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-21T10:10:35.048 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:34 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3719393509' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:35.051 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:35.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:35.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:35.096 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:10:35.096 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:10:35.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:35.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:34 smithi129 ceph-mon[88471]: pgmap v914: 109 pgs: 109 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-21T10:10:35.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:34 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3719393509' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:35.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:35.203 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:35.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:35.304 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:35.328 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:35.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:35.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:35.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:35.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:35.467 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:35.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:35.568 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:35.593 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:35.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:35.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:35.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:35.724 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 43 16 70371 70355 102.237 69.75 0.00953443 0.00977513 2024-09-21T10:10:35.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:35.734 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:35.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:35.835 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:35.860 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:35.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:35.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:35.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:35.994 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:35.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:35.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:36.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:36.133 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:36.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:36.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:36.234 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:36.260 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:36.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:36.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:36.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:36.406 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:36.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:36.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:36.507 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:36.531 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:36.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:36.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:36.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:36.667 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:36.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:36.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:36.724 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 44 16 71530 71514 101.559 72.4375 0.00961331 0.00984107 2024-09-21T10:10:36.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:36.807 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:36.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:36.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:36.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:36.952 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:36.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:37.046 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:37.046 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:37.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:37.092 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:10:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:36 smithi107 ceph-mon[102221]: pgmap v915: 109 pgs: 109 active+clean; 4.0 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 59 MiB/s wr, 950 op/s 2024-09-21T10:10:37.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:36 smithi107 ceph-mon[105022]: pgmap v915: 109 pgs: 109 active+clean; 4.0 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 59 MiB/s wr, 950 op/s 2024-09-21T10:10:37.117 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:36 smithi129 ceph-mon[88471]: pgmap v915: 109 pgs: 109 active+clean; 4.0 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 59 MiB/s wr, 950 op/s 2024-09-21T10:10:37.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:37.131 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:37.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:37.232 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:37.261 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:37.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:37.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:37.364 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:37.392 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:37.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:37.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:37.419 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:10:37.493 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:37.519 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:37.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:37.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:37.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:37.669 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:37.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:37.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:37.725 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 45 16 72702 72686 100.929 73.25 0.00623121 0.00990322 2024-09-21T10:10:37.771 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:37.802 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:37.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:37.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:37.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:10:37.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:37.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:37.984 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:37.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:38.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:38.129 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:38.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:38.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:38.231 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:38.256 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:38.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:38.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:38.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:38.401 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:38.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:38.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:38.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:38.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:38.563 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:38.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:38.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:38.708 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:38.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:38.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:38.725 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 46 16 73764 73748 100.177 66.375 0.0169895 0.00997694 2024-09-21T10:10:38.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:38.841 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:38.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:38.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:38.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:38.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:38.981 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:38.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:39.082 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:39.105 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:39.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:39.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:39.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:38 smithi107 ceph-mon[102221]: pgmap v916: 109 pgs: 109 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-21T10:10:39.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:38 smithi107 ceph-mon[105022]: pgmap v916: 109 pgs: 109 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-21T10:10:39.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:38 smithi129 ceph-mon[88471]: pgmap v916: 109 pgs: 109 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-21T10:10:39.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:39.245 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:39.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:39.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:39.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:39.386 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:39.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:39.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:39.450 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:10:39.450 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:10:39.488 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:39.518 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:39.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:39.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:39.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:39.656 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:39.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:39.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:39.725 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 47 16 74956 74940 99.631 74.5 0.0149792 0.0100317 2024-09-21T10:10:39.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:39.800 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:39.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:39.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:39.902 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:39.939 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:39.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:39.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:40.040 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:40.064 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:40.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:40.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:40.127 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:10:40.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:40.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:40.211 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:40.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:40.312 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:40.340 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:40.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:40.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:40.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:10:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:10:40.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:10:40.442 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:40.449 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:10:40.470 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:40.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:40.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:40.572 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:40.600 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:40.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:40.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:40.702 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:40.726 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:40.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:40.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:40.728 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 48 16 76077 76061 99.0107 70.0625 0.0157886 0.0100935 2024-09-21T10:10:40.829 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:40.854 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:40.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:40.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:40.956 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:40.981 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:40.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:40.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:41.082 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:40 smithi107 ceph-mon[102221]: pgmap v917: 109 pgs: 109 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-21T10:10:41.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:40 smithi107 ceph-mon[105022]: pgmap v917: 109 pgs: 109 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-21T10:10:41.110 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:41.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:41.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:40 smithi129 ceph-mon[88471]: pgmap v917: 109 pgs: 109 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-21T10:10:41.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:41.241 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:41.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:41.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:41.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:41.385 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:41.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:41.487 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:41.516 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:41.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:41.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:41.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:41.655 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:41.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:41.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:41.727 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 49 16 77082 77066 98.272 62.8125 0.00922746 0.0101712 2024-09-21T10:10:41.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:41.793 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:41.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:41.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:41.894 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:41.921 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:41.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:41.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:42.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:42.034 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:10:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:10:41] "GET /metrics HTTP/1.1" 200 36803 "" "Prometheus/2.51.0" 2024-09-21T10:10:42.071 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:42.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:42.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:42.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:42.204 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:42.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:42.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:42.306 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:42.331 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:42.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:42.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:42.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:42.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:42.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:42 smithi107 ceph-mon[102221]: pgmap v918: 109 pgs: 109 active+clean; 4.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:10:42.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:42.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:42.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:42.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:42 smithi107 ceph-mon[105022]: pgmap v918: 109 pgs: 109 active+clean; 4.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:10:42.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:42 smithi129 ceph-mon[88471]: pgmap v918: 109 pgs: 109 active+clean; 4.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:10:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:42.433 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:42.458 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:42.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:42.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:42.559 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:42.592 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:42.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:42.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:42.694 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:42.725 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:42.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:42.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:42.727 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 50 16 78094 78078 97.5715 63.25 0.0292555 0.0102434 2024-09-21T10:10:42.827 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:42.860 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:42.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:42.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:42.864 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:10:42.864 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:10:42.961 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:42.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:43.011 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:43.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:43.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:43.112 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:43.138 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:43.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:43.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:43.193 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T10:10:43.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:43.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:43.298 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:43.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:43.399 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:43.426 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:43.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:43.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:43.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:43.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:43 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4002045307' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:10:43.573 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:43 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:10:43.574 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:43.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:43.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:43.597 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:43 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4002045307' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:10:43.597 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:43 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:10:43.598 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:43 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4002045307' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:10:43.598 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:43 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:10:43.605 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:10:43.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:43.722 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:43.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:43.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:43.728 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 51 16 79127 79111 96.9239 64.5625 0.0153067 0.0103126 2024-09-21T10:10:43.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:43.859 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:43.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:43.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:43.925 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:10:43.961 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:43.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:43.990 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:43.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:44.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T10:10:44.091 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:44.116 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:44.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:44.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:44.217 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:44.243 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:44.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:44.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:44.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:44 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:10:44.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:44 smithi107 ceph-mon[102221]: osdmap e264: 8 total, 8 up, 7 in 2024-09-21T10:10:44.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:44 smithi107 ceph-mon[102221]: pgmap v920: 109 pgs: 109 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:10:44.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:44 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:44.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:44 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:44.286 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:44 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:10:44.286 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:44 smithi107 ceph-mon[105022]: osdmap e264: 8 total, 8 up, 7 in 2024-09-21T10:10:44.286 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:44 smithi107 ceph-mon[105022]: pgmap v920: 109 pgs: 109 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:10:44.286 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:44 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:44.286 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:44 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:44.344 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:44.352 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:44 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:10:44.352 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:44 smithi129 ceph-mon[88471]: osdmap e264: 8 total, 8 up, 7 in 2024-09-21T10:10:44.352 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:44 smithi129 ceph-mon[88471]: pgmap v920: 109 pgs: 109 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-21T10:10:44.352 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:44 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:44.353 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:44 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:44.371 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:44.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:44.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:44.473 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:44.497 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:44.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:44.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:44.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:44.636 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:44.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:44.728 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 52 16 80098 80082 96.2271 60.6875 0.00951702 0.0103868 2024-09-21T10:10:44.738 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:44.763 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:44.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:44.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:44.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:44.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:44.910 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:44.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:45.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:45.036 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:45.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:45.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:45.137 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:45.162 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:45.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:45.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:45.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:45.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:45.303 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:45.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:45.404 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:45 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:10:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:45 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:10:45.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:45 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:45.446 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:45.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:45.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:45.547 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:45.575 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:45.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:45.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:45 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:10:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:45 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:10:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:45 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:45.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:45 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:10:45.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:45 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:10:45.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:45 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:10:45.678 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:45.704 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:45.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:45.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:45.728 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 53 16 81185 81169 95.6933 67.9375 0.013164 0.0104449 2024-09-21T10:10:45.747 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:10:45.747 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:10:45.806 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:45.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:45.834 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:45.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:45.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:45.974 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:45.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:45.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:46.075 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:46.104 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:46.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:46.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:46.206 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:46.231 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:46.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:46.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:46.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:46.376 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:46.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:46.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:46.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:46 smithi129 ceph-mon[88471]: pgmap v921: 109 pgs: 109 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T10:10:46.478 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:46.507 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:46.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:46.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:46.514 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:10:46.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:46 smithi107 ceph-mon[102221]: pgmap v921: 109 pgs: 109 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T10:10:46.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:46 smithi107 ceph-mon[105022]: pgmap v921: 109 pgs: 109 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T10:10:46.609 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:46.634 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:46.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:46.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:46.728 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 54 16 82235 82219 95.1364 65.625 0.0142004 0.0105058 2024-09-21T10:10:46.735 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:46.773 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:46.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:46.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:46.875 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:46.880 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:10:46.906 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:46.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:46.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:47.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:47.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:47.035 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:47.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:47.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:47.178 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:47.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:47.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:47.279 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:47.304 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:47.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:47.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:47.405 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:47.427 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:47.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:47.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:47.529 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:47.555 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:47.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:47.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:47.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:47.678 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:47.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:47.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:47.728 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 55 16 83081 83065 94.368 52.875 0.00273167 0.0105681 2024-09-21T10:10:47.780 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:47.804 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:47.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:47.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:47.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:47.953 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:47.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:47.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:48.054 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:48.075 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:48.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:48.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:48.176 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:48.205 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:48.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:48.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:48.307 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:48.333 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:48.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:48.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:48 smithi129 ceph-mon[88471]: pgmap v922: 109 pgs: 109 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-21T10:10:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:48 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3201936315' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:10:48.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:48 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:10:48.435 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:48.460 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:48.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:48.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:48.562 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:48.588 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:48.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:48.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:48 smithi107 ceph-mon[102221]: pgmap v922: 109 pgs: 109 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-21T10:10:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:48 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3201936315' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:10:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:48 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:10:48.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:48 smithi107 ceph-mon[105022]: pgmap v922: 109 pgs: 109 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-21T10:10:48.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:48 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3201936315' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:10:48.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:48 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:10:48.690 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:48.715 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:48.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:48.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:48.728 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 56 16 83908 83892 93.6059 51.6875 0.0120654 0.0106782 2024-09-21T10:10:48.816 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:48.842 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:48.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:48.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:48.848 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:10:48.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:48.976 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:48.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:48.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:49.078 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:49.105 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:49.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:49.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:49.207 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:49.251 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:49.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:49.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:49.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:49.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:49.388 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:49.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:49.489 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:49.515 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:49.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:49.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:49 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:10:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:49 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:10:49.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:49 smithi107 ceph-mon[102221]: osdmap e265: 8 total, 8 up, 7 in 2024-09-21T10:10:49.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:49 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:10:49.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:49 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:10:49.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:49 smithi107 ceph-mon[105022]: osdmap e265: 8 total, 8 up, 7 in 2024-09-21T10:10:49.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:49.657 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:49.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:49.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:49 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:10:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:49 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:10:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:49 smithi129 ceph-mon[88471]: osdmap e265: 8 total, 8 up, 7 in 2024-09-21T10:10:49.728 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 57 16 84937 84921 93.092 64.3125 0.0122423 0.0107363 2024-09-21T10:10:49.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:49.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:49.795 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:49.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:49.896 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:49.924 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:49.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:50.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:50.063 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:50.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:50.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:50.165 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:50.189 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:50.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:50.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:50.291 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:50.318 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:50.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:50.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:50.420 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:50.442 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:50.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:50.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:50.544 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:50.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:50.578 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:50.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:50.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:50 smithi107 ceph-mon[105022]: pgmap v924: 109 pgs: 109 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-09-21T10:10:50.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:50 smithi107 ceph-mon[102221]: pgmap v924: 109 pgs: 109 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-09-21T10:10:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:50 smithi129 ceph-mon[88471]: pgmap v924: 109 pgs: 109 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-09-21T10:10:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:10:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:10:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:10:50.678 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:50.702 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:50.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:50.728 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 58 16 85882 85866 92.5053 59.0625 0.0123296 0.0108051 2024-09-21T10:10:50.770 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:50.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:50.880 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:50.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:50.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:50.982 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:51.006 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:51.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:51.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:51.108 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:51.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:51.134 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:51.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:51.234 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:51.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:51.271 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:51.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:51.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:51.413 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:51.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:51.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:51.515 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:51.546 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:51.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:51.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:51.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:51.690 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:51.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:51.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:51.728 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 59 16 86803 86787 91.913 57.5625 0.0102488 0.0108737 2024-09-21T10:10:51.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:51.829 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:51.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:51.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:51.931 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:51.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:51.966 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:51.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:52.028 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:10:52.034 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:10:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:10:51] "GET /metrics HTTP/1.1" 200 36803 "" "Prometheus/2.51.0" 2024-09-21T10:10:52.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:52.121 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:52.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:52.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:52.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:52.243 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:10:52.244 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:10:52.263 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:52.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:52.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:52.279 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:10:52.279 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:10:52.316 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:10:52.316 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:10:52.347 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:10:52.347 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:10:52.360 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:10:52.360 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:10:52.365 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:52.389 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:10:52.389 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:10:52.392 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:52.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:52.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:52 smithi129 ceph-mon[88471]: pgmap v925: 109 pgs: 109 active+clean; 5.0 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 46 MiB/s wr, 741 op/s 2024-09-21T10:10:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:52 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4108607344' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:52.421 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:10:52.421 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:10:52.449 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:10:52.450 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:10:52.479 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:10:52.479 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:10:52.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:52.531 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:52.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:52.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:52.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:52 smithi107 ceph-mon[102221]: pgmap v925: 109 pgs: 109 active+clean; 5.0 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 46 MiB/s wr, 741 op/s 2024-09-21T10:10:52.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:52 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4108607344' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:52.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:52 smithi107 ceph-mon[105022]: pgmap v925: 109 pgs: 109 active+clean; 5.0 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 46 MiB/s wr, 741 op/s 2024-09-21T10:10:52.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:52 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4108607344' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:10:52.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:52.668 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:52.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:52.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:52.728 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:10:52.727669+0000 min lat: 0.00109786 max lat: 0.260771 avg lat: 0.0109419 2024-09-21T10:10:52.728 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:10:52.729 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 60 16 87733 87717 91.3499 58.125 0.00535351 0.0109419 2024-09-21T10:10:52.769 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:52.795 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:52.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:52.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:52.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:52.927 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:52.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:52.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:53.029 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:53.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:53.055 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:53.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:53.156 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:53.184 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:53.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:53.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:53.191 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:10:53.286 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:53.322 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:53.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:53.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:53.424 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:53.451 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:53.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:53.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:53.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:53.555 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:10:53.587 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:53.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:53.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:53.689 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:53.731 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 61 16 88743 88727 90.884 63.125 0.030062 0.0109975 2024-09-21T10:10:53.743 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:53.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:53.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:53.845 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:53.889 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:53.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:53.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:53.991 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:54.019 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:54.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:54.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:54.121 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:54.145 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:54.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:54.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:54.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:54.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:54.292 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:54.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:54.393 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:54.417 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:54.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:54.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:54.519 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:54.541 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:54.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:54.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:54.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:54 smithi129 ceph-mon[88471]: pgmap v926: 109 pgs: 109 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 61 MiB/s wr, 972 op/s 2024-09-21T10:10:54.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:54.689 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:54.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:54.731 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 62 16 89679 89663 90.3617 58.5 0.0162459 0.0110617 2024-09-21T10:10:54.791 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:54.832 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:54.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:54.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:54.839 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:54 smithi107 ceph-mon[102221]: pgmap v926: 109 pgs: 109 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 61 MiB/s wr, 972 op/s 2024-09-21T10:10:54.840 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:54 smithi107 ceph-mon[105022]: pgmap v926: 109 pgs: 109 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 61 MiB/s wr, 972 op/s 2024-09-21T10:10:54.934 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:54.958 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:54.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:54.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:55.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:55.103 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:55.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:55.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:55.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:55.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:55.255 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:55.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:55.356 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:55.374 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:10:55.375 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:10:55.375 INFO:teuthology.orchestra.run.smithi129.stdout: "complaint_time": 30, 2024-09-21T10:10:55.375 INFO:teuthology.orchestra.run.smithi129.stdout: "num_blocked_ops": 0 2024-09-21T10:10:55.375 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:10:55.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:55.392 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:55.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:55.493 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:55.517 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:55.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:55.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:55.527 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:55.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:55.663 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:55.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:55.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:55.731 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 63 16 90480 90464 89.7221 50.0625 0.0055194 0.0111399 2024-09-21T10:10:55.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:55.807 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:55.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:55.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:55.909 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:55.934 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:55.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:55.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:56.036 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:56.059 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:56.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:56.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:56.130 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:10:56.160 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:56.198 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:56.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:56.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:56.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:56.335 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:56.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:56.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:56.383 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T10:10:56.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:56.448 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:10:56.461 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:56.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:56.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:56 smithi107 ceph-mon[102221]: pgmap v927: 109 pgs: 109 active+clean; 5.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 61 MiB/s wr, 969 op/s 2024-09-21T10:10:56.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:56 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3535209389' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:10:56.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:56 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:10:56.537 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:56.537 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:56 smithi107 ceph-mon[105022]: pgmap v927: 109 pgs: 109 active+clean; 5.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 61 MiB/s wr, 969 op/s 2024-09-21T10:10:56.538 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:56 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3535209389' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:10:56.538 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:56 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:10:56.538 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:56.562 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:56.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:56.584 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:56.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:56 smithi129 ceph-mon[88471]: pgmap v927: 109 pgs: 109 active+clean; 5.3 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 61 MiB/s wr, 969 op/s 2024-09-21T10:10:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:56 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3535209389' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:10:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:56 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:10:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:56.686 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:56.707 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:56.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:56.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:56.731 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 64 16 91129 91113 88.954 40.5625 0.00299278 0.0111831 2024-09-21T10:10:56.810 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:56.832 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:56.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:56.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:56.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:56.955 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:56.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:56.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:57.056 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:57.078 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:57.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:57.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:57.180 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:57.203 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:57.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:57.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:57.305 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:57.327 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:57.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:57.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:57.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:57.450 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:57.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:57.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:57.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:10:57.513 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:57 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:10:57.513 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:57 smithi107 ceph-mon[102221]: osdmap e266: 8 total, 8 up, 7 in 2024-09-21T10:10:57.513 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:57 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:10:57.513 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:57 smithi107 ceph-mon[105022]: osdmap e266: 8 total, 8 up, 7 in 2024-09-21T10:10:57.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:57.573 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:57.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:57.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:57.625 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T10:10:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:57 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:10:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:57 smithi129 ceph-mon[88471]: osdmap e266: 8 total, 8 up, 7 in 2024-09-21T10:10:57.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:57.697 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:57.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:57.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:57.731 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 65 16 91129 91113 87.5857 0 - 0.0111831 2024-09-21T10:10:57.799 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:57.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:57.821 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:57.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:57.922 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:10:57.943 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:57.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:57.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:10:58.045 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:10:58.066 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:58.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:58.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:10:58.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:58.177 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:58.192 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:58.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:58.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:58.222 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:10:58.294 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:10:58.315 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:58.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:58.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:10:58.390 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:10:58.417 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:58.441 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:58.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:58.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:58.543 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:58.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:58.566 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:58.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:58.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:58.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:58 smithi129 ceph-mon[88471]: pgmap v929: 109 pgs: 17 peering, 92 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 962 op/s 2024-09-21T10:10:58.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:58 smithi129 ceph-mon[88471]: osdmap e267: 8 total, 8 up, 7 in 2024-09-21T10:10:58.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:58 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3884878449' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:10:58.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:58 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:10:58.703 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:58.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:58.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:58.731 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 66 16 91166 91150 86.2939 1.15625 0.0059076 0.0114562 2024-09-21T10:10:58.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:58 smithi107 ceph-mon[102221]: pgmap v929: 109 pgs: 17 peering, 92 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 962 op/s 2024-09-21T10:10:58.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:58 smithi107 ceph-mon[102221]: osdmap e267: 8 total, 8 up, 7 in 2024-09-21T10:10:58.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:58 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3884878449' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:10:58.781 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:58 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:10:58.781 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:58 smithi107 ceph-mon[105022]: pgmap v929: 109 pgs: 17 peering, 92 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 962 op/s 2024-09-21T10:10:58.781 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:58 smithi107 ceph-mon[105022]: osdmap e267: 8 total, 8 up, 7 in 2024-09-21T10:10:58.782 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:58 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3884878449' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:10:58.782 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:58 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:10:58.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:58.838 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:58.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:58.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:58.941 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:58.971 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:58.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:58.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:59.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:10:59.100 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:10:59.100 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 20, 2024-09-21T10:10:59.100 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 600, 2024-09-21T10:10:59.100 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:10:59.101 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.101 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32231 9.7 9:ecfb968d:::benchmark_data_smithi107_152010_object32230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:10:59.101 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.559414+0000", 2024-09-21T10:10:59.101 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 595.15216564399998, 2024-09-21T10:10:59.101 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23909994900000001, 2024-09-21T10:10:59.101 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:10:59.101 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:10:59.101 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:10:59.101 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:10:59.101 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:10:59.101 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32231 2024-09-21T10:10:59.101 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.101 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:10:59.102 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.102 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:10:59.102 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.559414+0000", 2024-09-21T10:10:59.102 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.102 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.102 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.102 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:10:59.102 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.559414+0000", 2024-09-21T10:10:59.102 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.102 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.102 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.102 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:10:59.102 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.559412+0000", 2024-09-21T10:10:59.102 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:10:59.102 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.103 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.103 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:10:59.103 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.560092+0000", 2024-09-21T10:10:59.103 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00067992900000000002 2024-09-21T10:10:59.103 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.103 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.103 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:10:59.103 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.560104+0000", 2024-09-21T10:10:59.103 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1887e-05 2024-09-21T10:10:59.103 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.103 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.103 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:10:59.103 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.560110+0000", 2024-09-21T10:10:59.103 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5740000000000003e-06 2024-09-21T10:10:59.104 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.104 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.104 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.104 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.560155+0000", 2024-09-21T10:10:59.104 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4913000000000002e-05 2024-09-21T10:10:59.104 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.104 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.104 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:10:59.104 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.560278+0000", 2024-09-21T10:10:59.104 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012316399999999999 2024-09-21T10:10:59.104 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.104 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.104 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:10:59.104 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.560401+0000", 2024-09-21T10:10:59.104 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012349299999999999 2024-09-21T10:10:59.105 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.105 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.105 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:10:59.105 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.562243+0000", 2024-09-21T10:10:59.105 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0018417520000000001 2024-09-21T10:10:59.105 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.105 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.105 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.105 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.563134+0000", 2024-09-21T10:10:59.105 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00089091400000000003 2024-09-21T10:10:59.105 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.105 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.105 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.105 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.798444+0000", 2024-09-21T10:10:59.105 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23531032700000001 2024-09-21T10:10:59.105 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.106 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.106 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:10:59.106 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.798486+0000", 2024-09-21T10:10:59.106 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1326999999999999e-05 2024-09-21T10:10:59.106 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.106 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.106 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:10:59.106 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.798514+0000", 2024-09-21T10:10:59.106 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.809e-05 2024-09-21T10:10:59.106 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.106 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.106 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.106 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.107 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.107 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32237 9.6 9:67022ea0:::benchmark_data_smithi107_152010_object32236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:10:59.107 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.561242+0000", 2024-09-21T10:10:59.107 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 595.15033784699995, 2024-09-21T10:10:59.107 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23891710999999999, 2024-09-21T10:10:59.107 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:10:59.107 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:10:59.107 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:10:59.107 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:10:59.107 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:10:59.107 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32237 2024-09-21T10:10:59.107 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.107 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:10:59.107 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.107 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:10:59.108 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561242+0000", 2024-09-21T10:10:59.108 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.108 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.108 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.108 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:10:59.108 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561242+0000", 2024-09-21T10:10:59.108 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.108 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.108 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.108 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:10:59.108 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561233+0000", 2024-09-21T10:10:59.109 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999914 2024-09-21T10:10:59.109 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.109 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.109 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:10:59.109 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561365+0000", 2024-09-21T10:10:59.109 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013210300000000001 2024-09-21T10:10:59.109 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.109 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.109 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:10:59.109 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561377+0000", 2024-09-21T10:10:59.109 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1426e-05 2024-09-21T10:10:59.109 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.109 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.110 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:10:59.110 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561382+0000", 2024-09-21T10:10:59.110 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4480000000000002e-06 2024-09-21T10:10:59.110 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.110 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.110 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.110 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561429+0000", 2024-09-21T10:10:59.110 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6761000000000002e-05 2024-09-21T10:10:59.110 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.110 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.110 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:10:59.110 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561536+0000", 2024-09-21T10:10:59.110 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000107021 2024-09-21T10:10:59.111 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.111 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.111 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:10:59.111 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.561674+0000", 2024-09-21T10:10:59.111 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013777700000000001 2024-09-21T10:10:59.111 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.111 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.111 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:10:59.111 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.562828+0000", 2024-09-21T10:10:59.111 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0011543860000000001 2024-09-21T10:10:59.111 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.111 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.111 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.111 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564200+0000", 2024-09-21T10:10:59.111 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001372465 2024-09-21T10:10:59.112 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.112 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.112 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.112 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800061+0000", 2024-09-21T10:10:59.112 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.2358606 2024-09-21T10:10:59.112 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.112 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.112 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:10:59.112 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800127+0000", 2024-09-21T10:10:59.112 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6154000000000004e-05 2024-09-21T10:10:59.112 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.112 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.112 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:10:59.112 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800159+0000", 2024-09-21T10:10:59.112 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1534e-05 2024-09-21T10:10:59.113 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.113 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.113 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.113 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.113 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.113 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32243 9.6 9:68ed822c:::benchmark_data_smithi107_152010_object32242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:10:59.113 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.564332+0000", 2024-09-21T10:10:59.113 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 595.14724776399999, 2024-09-21T10:10:59.113 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23596514499999999, 2024-09-21T10:10:59.113 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:10:59.113 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:10:59.113 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:10:59.113 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:10:59.113 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:10:59.114 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32243 2024-09-21T10:10:59.114 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.114 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:10:59.114 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.114 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:10:59.114 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564332+0000", 2024-09-21T10:10:59.114 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.114 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.114 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.114 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:10:59.114 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564332+0000", 2024-09-21T10:10:59.114 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.114 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.114 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.115 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:10:59.115 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564331+0000", 2024-09-21T10:10:59.115 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:10:59.115 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.115 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.115 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:10:59.115 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564391+0000", 2024-09-21T10:10:59.115 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0105999999999997e-05 2024-09-21T10:10:59.115 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.115 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.115 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:10:59.115 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564401+0000", 2024-09-21T10:10:59.115 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.8099999999999992e-06 2024-09-21T10:10:59.115 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.115 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.116 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:10:59.116 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564413+0000", 2024-09-21T10:10:59.116 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2163e-05 2024-09-21T10:10:59.116 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.116 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.116 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.116 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564449+0000", 2024-09-21T10:10:59.116 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5775999999999998e-05 2024-09-21T10:10:59.116 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.116 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.116 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:10:59.116 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564557+0000", 2024-09-21T10:10:59.116 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010838199999999999 2024-09-21T10:10:59.116 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.116 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.117 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:10:59.117 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.564694+0000", 2024-09-21T10:10:59.117 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013716299999999999 2024-09-21T10:10:59.117 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.117 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.117 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:10:59.117 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.565898+0000", 2024-09-21T10:10:59.117 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0012039570000000001 2024-09-21T10:10:59.117 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.117 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.117 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.117 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.567502+0000", 2024-09-21T10:10:59.117 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001604149 2024-09-21T10:10:59.117 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.117 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.118 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.118 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800209+0000", 2024-09-21T10:10:59.118 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.232706624 2024-09-21T10:10:59.118 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.118 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.118 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:10:59.118 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800255+0000", 2024-09-21T10:10:59.118 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6452000000000002e-05 2024-09-21T10:10:59.118 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.118 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.118 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:10:59.118 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800297+0000", 2024-09-21T10:10:59.119 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1508999999999998e-05 2024-09-21T10:10:59.119 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.119 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.119 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.119 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.119 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.119 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32262 9.6 9:64cc6811:::benchmark_data_smithi107_152010_object32261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:10:59.119 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.575084+0000", 2024-09-21T10:10:59.119 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 595.13649560099998, 2024-09-21T10:10:59.119 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.22529529000000001, 2024-09-21T10:10:59.119 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:10:59.119 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:10:59.119 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:10:59.119 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:10:59.120 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:10:59.120 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32262 2024-09-21T10:10:59.120 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.120 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:10:59.120 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.120 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:10:59.120 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575084+0000", 2024-09-21T10:10:59.120 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.120 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.120 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.120 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:10:59.120 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575084+0000", 2024-09-21T10:10:59.120 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.121 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.121 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.121 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:10:59.121 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575083+0000", 2024-09-21T10:10:59.121 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:10:59.121 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.121 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.121 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:10:59.121 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575130+0000", 2024-09-21T10:10:59.121 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7709e-05 2024-09-21T10:10:59.121 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.121 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.121 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:10:59.121 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575141+0000", 2024-09-21T10:10:59.121 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0971e-05 2024-09-21T10:10:59.122 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.122 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.122 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:10:59.122 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575146+0000", 2024-09-21T10:10:59.122 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.262e-06 2024-09-21T10:10:59.122 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.122 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.122 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.122 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575189+0000", 2024-09-21T10:10:59.122 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3216000000000001e-05 2024-09-21T10:10:59.122 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.122 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.122 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:10:59.123 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575313+0000", 2024-09-21T10:10:59.123 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012379099999999999 2024-09-21T10:10:59.123 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.123 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.123 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:10:59.123 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.575477+0000", 2024-09-21T10:10:59.123 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00016417499999999999 2024-09-21T10:10:59.123 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.123 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.123 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:10:59.123 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.577444+0000", 2024-09-21T10:10:59.123 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0019669790000000002 2024-09-21T10:10:59.123 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.123 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.123 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.124 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.577574+0000", 2024-09-21T10:10:59.124 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012998900000000001 2024-09-21T10:10:59.124 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.124 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.124 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.124 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800326+0000", 2024-09-21T10:10:59.124 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.22275213099999999 2024-09-21T10:10:59.124 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.124 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.124 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:10:59.124 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800353+0000", 2024-09-21T10:10:59.124 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7056e-05 2024-09-21T10:10:59.124 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.124 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.125 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:10:59.125 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.800379+0000", 2024-09-21T10:10:59.125 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.6203e-05 2024-09-21T10:10:59.125 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.125 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.125 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.125 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.125 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.125 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32272 9.6 9:68a66308:::benchmark_data_smithi107_152010_object32271:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:10:59.125 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.579482+0000", 2024-09-21T10:10:59.125 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 595.13209737, 2024-09-21T10:10:59.125 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.22308472700000001, 2024-09-21T10:10:59.125 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:10:59.125 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:10:59.125 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:10:59.125 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:10:59.126 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:10:59.126 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32272 2024-09-21T10:10:59.126 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.126 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:10:59.126 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.126 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:10:59.126 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579482+0000", 2024-09-21T10:10:59.126 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.126 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.126 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.126 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:10:59.126 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579482+0000", 2024-09-21T10:10:59.126 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.126 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.126 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.127 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:10:59.127 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579481+0000", 2024-09-21T10:10:59.127 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:10:59.127 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.127 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.127 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:10:59.127 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579545+0000", 2024-09-21T10:10:59.127 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4506999999999998e-05 2024-09-21T10:10:59.127 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.127 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.127 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:10:59.127 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579557+0000", 2024-09-21T10:10:59.127 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1610000000000001e-05 2024-09-21T10:10:59.127 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.127 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.128 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:10:59.128 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579562+0000", 2024-09-21T10:10:59.128 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4970000000000001e-06 2024-09-21T10:10:59.128 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.128 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.128 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.128 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579606+0000", 2024-09-21T10:10:59.128 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3180000000000003e-05 2024-09-21T10:10:59.128 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.128 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.128 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:10:59.128 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579720+0000", 2024-09-21T10:10:59.128 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000114689 2024-09-21T10:10:59.128 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.128 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.129 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:10:59.129 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.579854+0000", 2024-09-21T10:10:59.129 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000134144 2024-09-21T10:10:59.129 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.129 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.129 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:10:59.129 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.580922+0000", 2024-09-21T10:10:59.129 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001067744 2024-09-21T10:10:59.129 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.129 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.129 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.129 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581827+0000", 2024-09-21T10:10:59.129 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00090439700000000004 2024-09-21T10:10:59.129 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.130 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.130 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.130 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.802500+0000", 2024-09-21T10:10:59.130 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.22067335699999999 2024-09-21T10:10:59.130 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.130 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.130 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:10:59.130 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.802531+0000", 2024-09-21T10:10:59.130 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1469e-05 2024-09-21T10:10:59.130 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.131 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.131 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:10:59.131 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.802567+0000", 2024-09-21T10:10:59.131 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5410000000000001e-05 2024-09-21T10:10:59.131 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.131 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.131 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.131 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.131 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.131 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32275 9.7 9:ef23b11d:::benchmark_data_smithi107_152010_object32274:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:10:59.131 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.581115+0000", 2024-09-21T10:10:59.131 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 595.13046399899997, 2024-09-21T10:10:59.131 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21812805800000001, 2024-09-21T10:10:59.132 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:10:59.132 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:10:59.132 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:10:59.132 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:10:59.132 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:10:59.132 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32275 2024-09-21T10:10:59.132 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.132 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:10:59.132 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.132 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:10:59.132 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581115+0000", 2024-09-21T10:10:59.132 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.132 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.132 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.133 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:10:59.133 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581115+0000", 2024-09-21T10:10:59.133 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.133 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.133 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.133 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:10:59.133 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581115+0000", 2024-09-21T10:10:59.133 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:10:59.133 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.133 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.133 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:10:59.133 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581168+0000", 2024-09-21T10:10:59.133 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3013000000000002e-05 2024-09-21T10:10:59.133 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.133 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.134 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:10:59.134 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581179+0000", 2024-09-21T10:10:59.134 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1970000000000001e-05 2024-09-21T10:10:59.134 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.134 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.134 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:10:59.134 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581184+0000", 2024-09-21T10:10:59.134 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7949999999999998e-06 2024-09-21T10:10:59.134 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.134 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.134 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.134 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581209+0000", 2024-09-21T10:10:59.135 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.4389999999999999e-05 2024-09-21T10:10:59.135 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.135 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.135 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:10:59.135 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581282+0000", 2024-09-21T10:10:59.135 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3590000000000005e-05 2024-09-21T10:10:59.135 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.135 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.135 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:10:59.135 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.581399+0000", 2024-09-21T10:10:59.135 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000117137 2024-09-21T10:10:59.135 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.135 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.136 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:10:59.136 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.582724+0000", 2024-09-21T10:10:59.136 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0013241220000000001 2024-09-21T10:10:59.136 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.136 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.136 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.136 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.582977+0000", 2024-09-21T10:10:59.136 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00025303400000000002 2024-09-21T10:10:59.136 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.136 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.136 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.136 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.799174+0000", 2024-09-21T10:10:59.136 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21619723299999999 2024-09-21T10:10:59.136 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.136 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.137 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:10:59.137 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.799206+0000", 2024-09-21T10:10:59.137 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2026999999999997e-05 2024-09-21T10:10:59.137 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.137 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.137 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:10:59.137 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.799244+0000", 2024-09-21T10:10:59.137 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7721e-05 2024-09-21T10:10:59.137 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.137 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.137 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.137 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.137 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.137 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:32283 9.7 9:ee5c92df:::benchmark_data_smithi107_152010_object32282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T10:10:59.138 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:01:03.589070+0000", 2024-09-21T10:10:59.138 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 595.12250991899998, 2024-09-21T10:10:59.138 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21034788500000001, 2024-09-21T10:10:59.138 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:10:59.138 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:10:59.138 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:10:59.138 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:10:59.138 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:10:59.138 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32283 2024-09-21T10:10:59.138 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.138 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:10:59.138 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.138 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:10:59.139 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589070+0000", 2024-09-21T10:10:59.139 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.139 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.139 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.139 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:10:59.139 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589070+0000", 2024-09-21T10:10:59.139 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.139 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.139 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.139 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:10:59.139 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589068+0000", 2024-09-21T10:10:59.139 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:10:59.139 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.140 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.140 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:10:59.140 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589113+0000", 2024-09-21T10:10:59.140 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4545000000000001e-05 2024-09-21T10:10:59.140 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.140 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.140 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:10:59.140 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589126+0000", 2024-09-21T10:10:59.140 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2738999999999999e-05 2024-09-21T10:10:59.140 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.140 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.140 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:10:59.140 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589130+0000", 2024-09-21T10:10:59.140 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3510000000000004e-06 2024-09-21T10:10:59.140 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.140 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.141 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.141 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589162+0000", 2024-09-21T10:10:59.141 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1612999999999997e-05 2024-09-21T10:10:59.141 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.141 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.141 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:10:59.141 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589244+0000", 2024-09-21T10:10:59.141 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.2725000000000005e-05 2024-09-21T10:10:59.141 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.141 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.141 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:10:59.141 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.589327+0000", 2024-09-21T10:10:59.141 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.3182000000000004e-05 2024-09-21T10:10:59.141 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.141 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:10:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.590365+0000", 2024-09-21T10:10:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0010372059999999999 2024-09-21T10:10:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.590700+0000", 2024-09-21T10:10:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00033515699999999999 2024-09-21T10:10:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.799339+0000", 2024-09-21T10:10:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.20863885800000001 2024-09-21T10:10:59.142 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.143 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.143 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:10:59.143 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.799377+0000", 2024-09-21T10:10:59.143 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8281000000000002e-05 2024-09-21T10:10:59.143 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.143 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.143 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:10:59.143 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:01:03.799417+0000", 2024-09-21T10:10:59.143 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0466999999999997e-05 2024-09-21T10:10:59.143 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.143 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.143 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.143 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.143 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.143 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:101219 9.f 9:fbbb13dd:::benchmark_data_smithi107_152010_object101218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-21T10:10:59.144 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:09.472644+0000", 2024-09-21T10:10:59.144 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 529.23893519000001, 2024-09-21T10:10:59.144 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.27593879999999998, 2024-09-21T10:10:59.144 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:10:59.144 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:10:59.144 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:10:59.144 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:10:59.144 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:10:59.144 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 101219 2024-09-21T10:10:59.144 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.144 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:10:59.144 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.144 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:10:59.144 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472644+0000", 2024-09-21T10:10:59.144 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.144 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.145 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.145 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:10:59.145 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472644+0000", 2024-09-21T10:10:59.145 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.145 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.145 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.145 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:10:59.145 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472643+0000", 2024-09-21T10:10:59.145 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:10:59.145 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.145 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.145 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:10:59.145 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472693+0000", 2024-09-21T10:10:59.145 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9314999999999998e-05 2024-09-21T10:10:59.145 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.146 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.146 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:10:59.146 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472707+0000", 2024-09-21T10:10:59.146 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4048e-05 2024-09-21T10:10:59.146 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.146 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.146 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:10:59.146 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472712+0000", 2024-09-21T10:10:59.146 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.524e-06 2024-09-21T10:10:59.146 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.146 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.146 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.146 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472747+0000", 2024-09-21T10:10:59.146 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5207999999999999e-05 2024-09-21T10:10:59.146 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.147 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.147 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:10:59.147 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472830+0000", 2024-09-21T10:10:59.147 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.3107000000000002e-05 2024-09-21T10:10:59.147 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.147 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.147 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:10:59.147 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.472921+0000", 2024-09-21T10:10:59.147 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.0845000000000002e-05 2024-09-21T10:10:59.147 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.147 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.147 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:10:59.147 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.475574+0000", 2024-09-21T10:10:59.147 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002653121 2024-09-21T10:10:59.148 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.148 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.148 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.148 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.484356+0000", 2024-09-21T10:10:59.148 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0087821459999999994 2024-09-21T10:10:59.148 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.148 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.148 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.148 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.748351+0000", 2024-09-21T10:10:59.148 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.26399478999999998 2024-09-21T10:10:59.148 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.148 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.148 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:10:59.148 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.748541+0000", 2024-09-21T10:10:59.148 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00018932500000000001 2024-09-21T10:10:59.149 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.149 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.149 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:10:59.149 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.748583+0000", 2024-09-21T10:10:59.149 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2478999999999998e-05 2024-09-21T10:10:59.149 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.149 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.149 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.149 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.149 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.149 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:101221 9.6 9:6e7b28d1:::benchmark_data_smithi107_152010_object101220:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-21T10:10:59.149 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:09.473703+0000", 2024-09-21T10:10:59.149 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 529.23787627399997, 2024-09-21T10:10:59.149 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.27788642400000002, 2024-09-21T10:10:59.150 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:10:59.150 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:10:59.150 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:10:59.150 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:10:59.150 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:10:59.150 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 101221 2024-09-21T10:10:59.150 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.150 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:10:59.150 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.150 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:10:59.150 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473703+0000", 2024-09-21T10:10:59.150 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.150 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.150 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.151 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:10:59.151 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473703+0000", 2024-09-21T10:10:59.151 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.151 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.151 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.151 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:10:59.151 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473703+0000", 2024-09-21T10:10:59.151 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:10:59.151 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.151 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.151 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:10:59.151 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473759+0000", 2024-09-21T10:10:59.151 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5989999999999998e-05 2024-09-21T10:10:59.151 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.151 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.152 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:10:59.152 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473768+0000", 2024-09-21T10:10:59.152 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.0119999999999994e-06 2024-09-21T10:10:59.152 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.152 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.152 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:10:59.152 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473771+0000", 2024-09-21T10:10:59.152 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6440000000000002e-06 2024-09-21T10:10:59.152 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.152 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.152 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.152 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473801+0000", 2024-09-21T10:10:59.152 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.9403e-05 2024-09-21T10:10:59.152 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.152 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.152 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:10:59.153 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473906+0000", 2024-09-21T10:10:59.153 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000105516 2024-09-21T10:10:59.153 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.153 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.153 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:10:59.153 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473986+0000", 2024-09-21T10:10:59.153 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9802000000000006e-05 2024-09-21T10:10:59.153 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.153 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.153 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:10:59.153 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.480993+0000", 2024-09-21T10:10:59.153 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.007007231 2024-09-21T10:10:59.153 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.153 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.154 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.154 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.481030+0000", 2024-09-21T10:10:59.154 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6848999999999999e-05 2024-09-21T10:10:59.154 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.154 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.154 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.154 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.751491+0000", 2024-09-21T10:10:59.154 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.27046129299999999 2024-09-21T10:10:59.154 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.154 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.154 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:10:59.154 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.751548+0000", 2024-09-21T10:10:59.155 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6422999999999998e-05 2024-09-21T10:10:59.155 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.155 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.155 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:10:59.155 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.751590+0000", 2024-09-21T10:10:59.155 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1850000000000001e-05 2024-09-21T10:10:59.155 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.155 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.155 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.155 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.155 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.155 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:101223 9.6 9:66631bd2:::benchmark_data_smithi107_152010_object101222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-21T10:10:59.155 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:09.473791+0000", 2024-09-21T10:10:59.156 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 529.23778850400004, 2024-09-21T10:10:59.156 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.283198595, 2024-09-21T10:10:59.156 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:10:59.156 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:10:59.156 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:10:59.156 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:10:59.156 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:10:59.156 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 101223 2024-09-21T10:10:59.157 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.157 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:10:59.158 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.158 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:10:59.158 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473791+0000", 2024-09-21T10:10:59.158 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.158 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.158 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.158 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:10:59.158 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473791+0000", 2024-09-21T10:10:59.158 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.158 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.158 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.158 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:10:59.158 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473790+0000", 2024-09-21T10:10:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:10:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:10:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473870+0000", 2024-09-21T10:10:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9717999999999998e-05 2024-09-21T10:10:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:10:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473878+0000", 2024-09-21T10:10:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9589999999999995e-06 2024-09-21T10:10:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.159 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:10:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.473883+0000", 2024-09-21T10:10:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8960000000000002e-06 2024-09-21T10:10:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.476473+0000", 2024-09-21T10:10:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0025897509999999999 2024-09-21T10:10:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:10:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.476556+0000", 2024-09-21T10:10:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.3242e-05 2024-09-21T10:10:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.160 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:10:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.476636+0000", 2024-09-21T10:10:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.0174999999999997e-05 2024-09-21T10:10:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:10:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.480996+0000", 2024-09-21T10:10:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0043602459999999999 2024-09-21T10:10:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.488327+0000", 2024-09-21T10:10:59.161 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0073310729999999996 2024-09-21T10:10:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.756861+0000", 2024-09-21T10:10:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.26853359700000001 2024-09-21T10:10:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:10:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.756947+0000", 2024-09-21T10:10:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6316999999999994e-05 2024-09-21T10:10:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:10:59.162 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.756990+0000", 2024-09-21T10:10:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2259000000000003e-05 2024-09-21T10:10:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:101225 9.6 9:6bf868dc:::benchmark_data_smithi107_152010_object101224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-21T10:10:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:09.474216+0000", 2024-09-21T10:10:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 529.23736384599999, 2024-09-21T10:10:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28288837500000003, 2024-09-21T10:10:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:10:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:10:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:10:59.163 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:10:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:10:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 101225 2024-09-21T10:10:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:10:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:10:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474216+0000", 2024-09-21T10:10:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:10:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474216+0000", 2024-09-21T10:10:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.164 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:10:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474215+0000", 2024-09-21T10:10:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:10:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:10:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474301+0000", 2024-09-21T10:10:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6462e-05 2024-09-21T10:10:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:10:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474313+0000", 2024-09-21T10:10:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1685000000000001e-05 2024-09-21T10:10:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.165 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:10:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.474318+0000", 2024-09-21T10:10:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2190000000000003e-06 2024-09-21T10:10:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.478980+0000", 2024-09-21T10:10:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0046614480000000003 2024-09-21T10:10:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:10:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.479090+0000", 2024-09-21T10:10:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000110468 2024-09-21T10:10:59.166 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.167 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.167 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:10:59.167 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.479168+0000", 2024-09-21T10:10:59.167 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.7837999999999996e-05 2024-09-21T10:10:59.167 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.167 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.167 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:10:59.167 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.481151+0000", 2024-09-21T10:10:59.167 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0019829729999999999 2024-09-21T10:10:59.167 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.167 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.167 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.167 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.488352+0000", 2024-09-21T10:10:59.167 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0072014410000000003 2024-09-21T10:10:59.167 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.168 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.168 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.168 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.757027+0000", 2024-09-21T10:10:59.168 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.26867438300000002 2024-09-21T10:10:59.168 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.168 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.168 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:10:59.168 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.757069+0000", 2024-09-21T10:10:59.168 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1835000000000002e-05 2024-09-21T10:10:59.168 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.168 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.168 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:10:59.168 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:09.757104+0000", 2024-09-21T10:10:59.168 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5410000000000001e-05 2024-09-21T10:10:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:126108 9.6 9:6f7b6afe:::benchmark_data_smithi107_152010_object16171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-09-21T10:10:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:38.616365+0000", 2024-09-21T10:10:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 500.09521428400001, 2024-09-21T10:10:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.213592333, 2024-09-21T10:10:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:10:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:10:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:10:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:10:59.169 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:10:59.170 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 126108 2024-09-21T10:10:59.170 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.170 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:10:59.170 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.170 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:10:59.170 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616365+0000", 2024-09-21T10:10:59.170 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.170 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.170 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.170 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:10:59.170 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616365+0000", 2024-09-21T10:10:59.170 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.170 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.171 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.171 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:10:59.171 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616364+0000", 2024-09-21T10:10:59.171 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:10:59.171 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.171 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.171 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:10:59.171 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616371+0000", 2024-09-21T10:10:59.171 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9840000000000004e-06 2024-09-21T10:10:59.171 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.171 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.171 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:10:59.171 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616390+0000", 2024-09-21T10:10:59.171 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.9476000000000001e-05 2024-09-21T10:10:59.171 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.172 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.172 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:10:59.172 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616397+0000", 2024-09-21T10:10:59.172 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4130000000000003e-06 2024-09-21T10:10:59.172 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.172 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.172 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.172 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616446+0000", 2024-09-21T10:10:59.172 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8998000000000001e-05 2024-09-21T10:10:59.172 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.172 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.172 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:10:59.172 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616569+0000", 2024-09-21T10:10:59.172 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000123318 2024-09-21T10:10:59.172 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.173 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.173 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:10:59.173 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.616671+0000", 2024-09-21T10:10:59.173 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010222599999999999 2024-09-21T10:10:59.173 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.173 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.173 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:10:59.173 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.620447+0000", 2024-09-21T10:10:59.173 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0037759579999999998 2024-09-21T10:10:59.173 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.173 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.173 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.173 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.829942+0000", 2024-09-21T10:10:59.173 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.20949464800000001 2024-09-21T10:10:59.174 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.174 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.174 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:10:59.174 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.829958+0000", 2024-09-21T10:10:59.174 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5863999999999999e-05 2024-09-21T10:10:59.174 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.174 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.174 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.174 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.174 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.174 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:126108 9.16 9:6f7b6afe:::benchmark_data_smithi107_152010_object16171:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163)", 2024-09-21T10:10:59.174 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:38.641118+0000", 2024-09-21T10:10:59.174 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 500.07046161300002, 2024-09-21T10:10:59.174 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.03992945, 2024-09-21T10:10:59.174 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:10:59.175 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:10:59.175 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:10:59.175 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:10:59.175 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:10:59.175 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 126108 2024-09-21T10:10:59.175 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.175 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:10:59.175 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.175 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:10:59.175 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.641118+0000", 2024-09-21T10:10:59.175 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.175 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.175 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.175 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:10:59.175 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.641118+0000", 2024-09-21T10:10:59.176 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.176 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.176 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.176 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:10:59.176 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.641117+0000", 2024-09-21T10:10:59.176 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:10:59.176 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.176 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.176 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:10:59.176 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.641124+0000", 2024-09-21T10:10:59.176 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5200000000000003e-06 2024-09-21T10:10:59.176 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.176 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.176 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:10:59.176 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.641133+0000", 2024-09-21T10:10:59.177 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.7790000000000007e-06 2024-09-21T10:10:59.177 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.177 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.177 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:10:59.177 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.641139+0000", 2024-09-21T10:10:59.177 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8510000000000004e-06 2024-09-21T10:10:59.177 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.177 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.177 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.177 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.830197+0000", 2024-09-21T10:10:59.177 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.18905825400000001 2024-09-21T10:10:59.177 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.177 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.177 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:10:59.177 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.830204+0000", 2024-09-21T10:10:59.177 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9940000000000003e-06 2024-09-21T10:10:59.177 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.178 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.178 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.178 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:39.680990+0000", 2024-09-21T10:10:59.178 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.85078537099999996 2024-09-21T10:10:59.178 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.178 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.178 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:10:59.178 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:39.681047+0000", 2024-09-21T10:10:59.178 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7512000000000001e-05 2024-09-21T10:10:59.178 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.178 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.178 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.178 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.179 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.179 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.45761.0:126119 9.a 9:5778ec59:::benchmark_data_smithi107_152010_object16182:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-09-21T10:10:59.179 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:02:38.652541+0000", 2024-09-21T10:10:59.179 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 500.05903853900003, 2024-09-21T10:10:59.179 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.413684987, 2024-09-21T10:10:59.179 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:10:59.179 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:10:59.179 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:10:59.179 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.45761", 2024-09-21T10:10:59.179 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/142323775", 2024-09-21T10:10:59.179 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 126119 2024-09-21T10:10:59.179 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.179 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:10:59.179 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.180 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:10:59.180 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652541+0000", 2024-09-21T10:10:59.180 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.180 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.180 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.180 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:10:59.180 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652541+0000", 2024-09-21T10:10:59.180 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.180 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.180 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.180 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:10:59.180 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652539+0000", 2024-09-21T10:10:59.180 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999976 2024-09-21T10:10:59.181 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.181 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.181 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:10:59.181 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652548+0000", 2024-09-21T10:10:59.181 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.5810000000000003e-06 2024-09-21T10:10:59.181 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.181 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.181 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:10:59.181 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652568+0000", 2024-09-21T10:10:59.181 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.9656999999999998e-05 2024-09-21T10:10:59.181 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.181 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.181 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:10:59.181 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652575+0000", 2024-09-21T10:10:59.181 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2960000000000001e-06 2024-09-21T10:10:59.182 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.182 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.182 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.182 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652636+0000", 2024-09-21T10:10:59.182 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0293e-05 2024-09-21T10:10:59.182 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.182 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.182 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:10:59.182 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652783+0000", 2024-09-21T10:10:59.182 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014715 2024-09-21T10:10:59.182 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.182 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.182 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:10:59.182 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.652898+0000", 2024-09-21T10:10:59.183 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011513700000000001 2024-09-21T10:10:59.183 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.183 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.183 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:10:59.183 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.657580+0000", 2024-09-21T10:10:59.183 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0046820580000000002 2024-09-21T10:10:59.183 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.183 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.183 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.183 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:38.895867+0000", 2024-09-21T10:10:59.183 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23828700799999999 2024-09-21T10:10:59.183 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.183 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.183 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.183 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:39.066119+0000", 2024-09-21T10:10:59.184 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.170251927 2024-09-21T10:10:59.184 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.184 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.184 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:10:59.184 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:39.066160+0000", 2024-09-21T10:10:59.184 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.074e-05 2024-09-21T10:10:59.184 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.184 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.184 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:10:59.184 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:02:39.066226+0000", 2024-09-21T10:10:59.184 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6333000000000003e-05 2024-09-21T10:10:59.184 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.184 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.184 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.184 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.185 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.185 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95099 10.6 10:64ab5024:::benchmark_data_smithi107_184135_object95098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T10:10:59.185 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.388487+0000", 2024-09-21T10:10:59.185 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 292.323092884, 2024-09-21T10:10:59.185 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.019649673, 2024-09-21T10:10:59.185 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:10:59.185 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:10:59.185 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:10:59.185 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:10:59.185 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:10:59.185 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95099 2024-09-21T10:10:59.185 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.185 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:10:59.186 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.186 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:10:59.186 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388487+0000", 2024-09-21T10:10:59.186 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.186 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.186 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.186 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:10:59.186 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388487+0000", 2024-09-21T10:10:59.186 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.186 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.186 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.186 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:10:59.186 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388485+0000", 2024-09-21T10:10:59.186 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:10:59.187 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.187 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.187 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:10:59.187 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388545+0000", 2024-09-21T10:10:59.187 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0622999999999999e-05 2024-09-21T10:10:59.187 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.187 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.187 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:10:59.187 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388561+0000", 2024-09-21T10:10:59.187 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5667000000000001e-05 2024-09-21T10:10:59.187 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.187 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.187 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:10:59.187 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388567+0000", 2024-09-21T10:10:59.187 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8640000000000001e-06 2024-09-21T10:10:59.187 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.188 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.188 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.188 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388616+0000", 2024-09-21T10:10:59.188 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9168000000000001e-05 2024-09-21T10:10:59.188 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.188 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.188 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:10:59.188 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388773+0000", 2024-09-21T10:10:59.188 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000157465 2024-09-21T10:10:59.188 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.188 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.188 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:10:59.188 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388903+0000", 2024-09-21T10:10:59.188 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012996399999999999 2024-09-21T10:10:59.189 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.189 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.189 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:10:59.189 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.400728+0000", 2024-09-21T10:10:59.189 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.01182456 2024-09-21T10:10:59.189 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.189 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.189 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.189 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407770+0000", 2024-09-21T10:10:59.189 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0070416530000000001 2024-09-21T10:10:59.189 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.189 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.189 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:10:59.189 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407778+0000", 2024-09-21T10:10:59.190 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9549999999999992e-06 2024-09-21T10:10:59.190 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.190 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.190 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.190 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.408044+0000", 2024-09-21T10:10:59.190 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.000266069 2024-09-21T10:10:59.190 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.190 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.190 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:10:59.190 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.408136+0000", 2024-09-21T10:10:59.190 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.2651000000000005e-05 2024-09-21T10:10:59.190 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.190 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.191 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.191 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.191 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.191 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95101 10.6 10:6a273935:::benchmark_data_smithi107_184135_object95100:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T10:10:59.191 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.389198+0000", 2024-09-21T10:10:59.191 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 292.32238129199999, 2024-09-21T10:10:59.191 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0191866650000001, 2024-09-21T10:10:59.191 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:10:59.191 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:10:59.191 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:10:59.191 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:10:59.191 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:10:59.191 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95101 2024-09-21T10:10:59.191 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.191 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:10:59.192 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.192 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:10:59.192 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.389198+0000", 2024-09-21T10:10:59.192 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.192 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.192 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.192 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:10:59.192 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.389198+0000", 2024-09-21T10:10:59.192 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.192 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.192 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.192 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:10:59.192 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.389197+0000", 2024-09-21T10:10:59.192 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:10:59.192 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.192 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.193 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:10:59.193 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.389230+0000", 2024-09-21T10:10:59.193 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3234000000000001e-05 2024-09-21T10:10:59.193 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.193 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.193 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:10:59.193 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.389246+0000", 2024-09-21T10:10:59.193 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5705999999999998e-05 2024-09-21T10:10:59.193 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.193 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.193 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:10:59.193 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.389253+0000", 2024-09-21T10:10:59.193 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8990000000000004e-06 2024-09-21T10:10:59.194 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.194 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.194 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.194 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.393452+0000", 2024-09-21T10:10:59.194 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.004198757 2024-09-21T10:10:59.194 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.194 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.194 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:10:59.194 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.393568+0000", 2024-09-21T10:10:59.194 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011586699999999999 2024-09-21T10:10:59.194 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.194 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.194 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:10:59.194 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.393687+0000", 2024-09-21T10:10:59.195 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000119133 2024-09-21T10:10:59.195 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.195 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.195 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.195 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.408391+0000", 2024-09-21T10:10:59.195 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.014703842 2024-09-21T10:10:59.195 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.195 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.195 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:10:59.195 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.408398+0000", 2024-09-21T10:10:59.195 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9349999999999997e-06 2024-09-21T10:10:59.195 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.195 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.195 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.196 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.408337+0000", 2024-09-21T10:10:59.196 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.999939103 2024-09-21T10:10:59.196 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.196 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.196 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:10:59.196 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.408385+0000", 2024-09-21T10:10:59.196 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8242000000000003e-05 2024-09-21T10:10:59.196 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.196 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.196 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.196 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.196 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.196 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247256 10.6 10:6298720f:::benchmark_data_smithi107_184135_object115690:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:10:59.196 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:50.433070+0000", 2024-09-21T10:10:59.197 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 128.278509431, 2024-09-21T10:10:59.197 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.99909955800000005, 2024-09-21T10:10:59.197 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:10:59.197 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:10:59.197 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:10:59.197 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:10:59.197 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:10:59.197 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247256 2024-09-21T10:10:59.197 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.197 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:10:59.197 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.197 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:10:59.197 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433070+0000", 2024-09-21T10:10:59.197 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.197 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.198 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.198 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:10:59.198 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433070+0000", 2024-09-21T10:10:59.198 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.198 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.198 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.198 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:10:59.198 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433067+0000", 2024-09-21T10:10:59.198 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999971 2024-09-21T10:10:59.198 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.198 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.198 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:10:59.198 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433079+0000", 2024-09-21T10:10:59.199 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1345000000000001e-05 2024-09-21T10:10:59.199 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.199 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.199 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:10:59.199 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433099+0000", 2024-09-21T10:10:59.199 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.0852e-05 2024-09-21T10:10:59.199 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.199 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.199 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:10:59.199 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433108+0000", 2024-09-21T10:10:59.199 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.8270000000000004e-06 2024-09-21T10:10:59.199 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.199 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.200 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.200 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433168+0000", 2024-09-21T10:10:59.200 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9737000000000001e-05 2024-09-21T10:10:59.200 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.200 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.200 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:10:59.200 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433179+0000", 2024-09-21T10:10:59.200 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0577000000000001e-05 2024-09-21T10:10:59.200 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.200 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.200 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.200 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.417734+0000", 2024-09-21T10:10:59.200 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.98455554300000003 2024-09-21T10:10:59.201 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.201 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.201 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:10:59.201 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.417856+0000", 2024-09-21T10:10:59.201 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000121657 2024-09-21T10:10:59.201 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.201 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.201 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:10:59.201 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.417974+0000", 2024-09-21T10:10:59.201 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000118348 2024-09-21T10:10:59.201 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.201 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.201 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.202 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.421532+0000", 2024-09-21T10:10:59.202 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003558205 2024-09-21T10:10:59.202 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.202 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.202 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.202 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432010+0000", 2024-09-21T10:10:59.202 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.010477765999999999 2024-09-21T10:10:59.202 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.202 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.202 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:10:59.202 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432019+0000", 2024-09-21T10:10:59.202 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.1379999999999996e-06 2024-09-21T10:10:59.202 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.202 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.203 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:10:59.203 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432070+0000", 2024-09-21T10:10:59.203 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0719e-05 2024-09-21T10:10:59.203 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.203 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.203 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:10:59.203 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432170+0000", 2024-09-21T10:10:59.203 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.9630000000000007e-05 2024-09-21T10:10:59.203 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.203 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.203 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.203 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.203 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.203 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247262 10.6 10:67e2f4ef:::benchmark_data_smithi107_184135_object115696:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:10:59.203 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:50.439088+0000", 2024-09-21T10:10:59.204 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 128.27249154, 2024-09-21T10:10:59.204 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.99353488599999995, 2024-09-21T10:10:59.204 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:10:59.204 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:10:59.204 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:10:59.204 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:10:59.204 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:10:59.204 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247262 2024-09-21T10:10:59.204 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.204 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:10:59.204 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.204 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:10:59.204 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439088+0000", 2024-09-21T10:10:59.204 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.204 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.204 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.204 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:10:59.205 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439088+0000", 2024-09-21T10:10:59.205 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.205 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.205 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.205 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:10:59.205 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439086+0000", 2024-09-21T10:10:59.205 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:10:59.205 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.205 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.205 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:10:59.205 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439094+0000", 2024-09-21T10:10:59.205 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.7840000000000004e-06 2024-09-21T10:10:59.205 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.205 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.206 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:10:59.206 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439111+0000", 2024-09-21T10:10:59.206 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.7059e-05 2024-09-21T10:10:59.206 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.206 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.206 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:10:59.206 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439118+0000", 2024-09-21T10:10:59.206 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5239999999999997e-06 2024-09-21T10:10:59.206 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.206 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.206 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.206 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439172+0000", 2024-09-21T10:10:59.206 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4327999999999999e-05 2024-09-21T10:10:59.207 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.207 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.207 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:10:59.207 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439185+0000", 2024-09-21T10:10:59.207 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3502999999999999e-05 2024-09-21T10:10:59.207 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.207 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.207 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.207 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.421553+0000", 2024-09-21T10:10:59.207 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.98236765199999998 2024-09-21T10:10:59.207 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.207 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.207 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:10:59.207 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.421626+0000", 2024-09-21T10:10:59.207 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3146999999999996e-05 2024-09-21T10:10:59.208 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.208 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.208 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:10:59.208 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.421685+0000", 2024-09-21T10:10:59.208 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8621e-05 2024-09-21T10:10:59.208 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.208 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.208 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:10:59.208 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432293+0000", 2024-09-21T10:10:59.208 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.010608332 2024-09-21T10:10:59.208 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.208 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.208 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.209 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432344+0000", 2024-09-21T10:10:59.209 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1279999999999997e-05 2024-09-21T10:10:59.209 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.209 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.209 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.209 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432459+0000", 2024-09-21T10:10:59.209 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000114418 2024-09-21T10:10:59.209 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.209 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.209 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:10:59.209 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432518+0000", 2024-09-21T10:10:59.209 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9299000000000003e-05 2024-09-21T10:10:59.209 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.209 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.210 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:10:59.210 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432623+0000", 2024-09-21T10:10:59.210 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010468599999999999 2024-09-21T10:10:59.210 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.210 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.210 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.210 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.210 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.210 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247267 10.6 10:615d1e96:::benchmark_data_smithi107_184135_object115701:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:10:59.210 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:50.444059+0000", 2024-09-21T10:10:59.210 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 128.267520671, 2024-09-21T10:10:59.210 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.98879305900000003, 2024-09-21T10:10:59.210 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:10:59.211 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:10:59.211 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:10:59.211 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:10:59.211 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:10:59.211 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247267 2024-09-21T10:10:59.211 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.211 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:10:59.211 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.211 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:10:59.211 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444059+0000", 2024-09-21T10:10:59.211 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.211 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.211 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.211 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:10:59.212 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444059+0000", 2024-09-21T10:10:59.212 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.212 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.212 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.212 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:10:59.212 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444057+0000", 2024-09-21T10:10:59.212 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:10:59.212 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.212 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.212 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:10:59.212 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444064+0000", 2024-09-21T10:10:59.212 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.5349999999999999e-06 2024-09-21T10:10:59.212 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.212 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.213 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:10:59.213 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444080+0000", 2024-09-21T10:10:59.213 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5478999999999999e-05 2024-09-21T10:10:59.213 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.213 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.213 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:10:59.213 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444086+0000", 2024-09-21T10:10:59.213 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.631e-06 2024-09-21T10:10:59.213 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.213 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.213 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.213 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444247+0000", 2024-09-21T10:10:59.213 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00016138 2024-09-21T10:10:59.213 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.214 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.214 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:10:59.214 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444256+0000", 2024-09-21T10:10:59.214 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.3510000000000008e-06 2024-09-21T10:10:59.214 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.214 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.214 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.214 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.424581+0000", 2024-09-21T10:10:59.214 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.98032498800000001 2024-09-21T10:10:59.214 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.214 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.214 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:10:59.214 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.424692+0000", 2024-09-21T10:10:59.215 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000111113 2024-09-21T10:10:59.215 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.215 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.215 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:10:59.215 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.424815+0000", 2024-09-21T10:10:59.215 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012274900000000001 2024-09-21T10:10:59.215 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.215 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.215 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:10:59.215 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432299+0000", 2024-09-21T10:10:59.215 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0074834740000000004 2024-09-21T10:10:59.215 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.215 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.216 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.216 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432681+0000", 2024-09-21T10:10:59.216 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00038213 2024-09-21T10:10:59.216 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.216 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.216 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.216 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432722+0000", 2024-09-21T10:10:59.216 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.138e-05 2024-09-21T10:10:59.216 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.216 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.216 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:10:59.216 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432765+0000", 2024-09-21T10:10:59.216 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2769000000000003e-05 2024-09-21T10:10:59.216 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.217 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.217 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:10:59.217 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432852+0000", 2024-09-21T10:10:59.217 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.7026000000000003e-05 2024-09-21T10:10:59.217 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.217 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.217 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.217 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.217 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.217 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247272 10.6 10:699abf74:::benchmark_data_smithi107_184135_object115706:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:10:59.217 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:50.447705+0000", 2024-09-21T10:10:59.217 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 128.263874832, 2024-09-21T10:10:59.217 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.985396304, 2024-09-21T10:10:59.218 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:10:59.218 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:10:59.218 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:10:59.218 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:10:59.218 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:10:59.218 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247272 2024-09-21T10:10:59.218 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.218 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:10:59.218 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.218 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:10:59.218 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447705+0000", 2024-09-21T10:10:59.218 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.218 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.219 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.219 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:10:59.219 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447705+0000", 2024-09-21T10:10:59.219 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:10:59.219 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.219 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.219 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:10:59.219 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447703+0000", 2024-09-21T10:10:59.219 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:10:59.219 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.219 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.219 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:10:59.219 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447711+0000", 2024-09-21T10:10:59.220 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.5229999999999998e-06 2024-09-21T10:10:59.220 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.220 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.220 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:10:59.220 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447721+0000", 2024-09-21T10:10:59.220 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0040999999999999e-05 2024-09-21T10:10:59.220 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.220 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.220 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:10:59.220 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447725+0000", 2024-09-21T10:10:59.220 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2679999999999996e-06 2024-09-21T10:10:59.220 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.220 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.220 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.221 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447767+0000", 2024-09-21T10:10:59.221 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1897999999999998e-05 2024-09-21T10:10:59.221 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.221 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.221 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:10:59.221 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447776+0000", 2024-09-21T10:10:59.221 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.2089999999999994e-06 2024-09-21T10:10:59.221 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.221 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.221 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:10:59.221 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.427698+0000", 2024-09-21T10:10:59.221 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.97992116600000001 2024-09-21T10:10:59.221 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.221 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.221 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:10:59.222 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.427813+0000", 2024-09-21T10:10:59.222 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000115301 2024-09-21T10:10:59.222 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.222 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.222 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:10:59.222 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.427925+0000", 2024-09-21T10:10:59.222 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000112324 2024-09-21T10:10:59.222 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.222 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.222 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:10:59.222 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432357+0000", 2024-09-21T10:10:59.222 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0044316570000000003 2024-09-21T10:10:59.222 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.223 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.223 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.223 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432891+0000", 2024-09-21T10:10:59.223 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00053454999999999996 2024-09-21T10:10:59.223 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.223 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.223 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:10:59.223 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432945+0000", 2024-09-21T10:10:59.223 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3307999999999999e-05 2024-09-21T10:10:59.223 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.223 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.223 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:10:59.223 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.433014+0000", 2024-09-21T10:10:59.223 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8949999999999995e-05 2024-09-21T10:10:59.224 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:10:59.224 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:10:59.224 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:10:59.224 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.433101+0000", 2024-09-21T10:10:59.224 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.7322000000000002e-05 2024-09-21T10:10:59.224 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.224 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.224 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.224 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:10:59.224 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:10:59.224 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:10:59.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:59.226 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:59.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:10:59.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:10:59.366 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:59.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:59.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:10:59.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:59.502 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:59.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:59.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:59.603 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:10:59.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:59 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:10:59.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:59 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:10:59.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:59 smithi107 ceph-mon[102221]: 11.7 starting backfill to osd.5 from (0'0,0'0] MAX to 265'5669 2024-09-21T10:10:59.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:59 smithi107 ceph-mon[102221]: osdmap e268: 8 total, 8 up, 7 in 2024-09-21T10:10:59.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:59 smithi107 ceph-mon[102221]: 11.8 starting backfill to osd.5 from (0'0,0'0] MAX to 265'5794 2024-09-21T10:10:59.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:10:59 smithi107 ceph-mon[102221]: 11.a starting backfill to osd.5 from (0'0,0'0] MAX to 265'5667 2024-09-21T10:10:59.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:59 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:10:59.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:59 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:10:59.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:59 smithi107 ceph-mon[105022]: 11.7 starting backfill to osd.5 from (0'0,0'0] MAX to 265'5669 2024-09-21T10:10:59.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:59 smithi107 ceph-mon[105022]: osdmap e268: 8 total, 8 up, 7 in 2024-09-21T10:10:59.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:59 smithi107 ceph-mon[105022]: 11.8 starting backfill to osd.5 from (0'0,0'0] MAX to 265'5794 2024-09-21T10:10:59.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:10:59 smithi107 ceph-mon[105022]: 11.a starting backfill to osd.5 from (0'0,0'0] MAX to 265'5667 2024-09-21T10:10:59.635 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:10:59.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:59.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:10:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:59 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:10:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:59 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:10:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:59 smithi129 ceph-mon[88471]: 11.7 starting backfill to osd.5 from (0'0,0'0] MAX to 265'5669 2024-09-21T10:10:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:59 smithi129 ceph-mon[88471]: osdmap e268: 8 total, 8 up, 7 in 2024-09-21T10:10:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:59 smithi129 ceph-mon[88471]: 11.8 starting backfill to osd.5 from (0'0,0'0] MAX to 265'5794 2024-09-21T10:10:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:10:59 smithi129 ceph-mon[88471]: 11.a starting backfill to osd.5 from (0'0,0'0] MAX to 265'5667 2024-09-21T10:10:59.731 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 67 16 92194 92178 85.9649 64.25 0.00930282 0.0116276 2024-09-21T10:10:59.738 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:10:59.767 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:59.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:59.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:10:59.869 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:10:59.892 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:10:59.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:10:59.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:10:59.993 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:00.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:00.026 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:00.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:00.127 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:00.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:00.160 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:00.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:00.261 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:00.313 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:11:00.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:00.315 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:00.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:00.416 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:00.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:00.444 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:00.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:00.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:00.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:00.579 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:00.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:11:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:11:00.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:11:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:00 smithi129 ceph-mon[88471]: pgmap v932: 109 pgs: 17 peering, 92 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 5.7 KiB/s rd, 51 MiB/s wr, 811 op/s; 51 KiB/s, 1 objects/s recovering 2024-09-21T10:11:00.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:00.717 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:00.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:00.721 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:11:00.731 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 68 16 93182 93166 85.6088 61.75 0.0180065 0.0116761 2024-09-21T10:11:00.819 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:00.846 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:00.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:00.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:00.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:00 smithi107 ceph-mon[102221]: pgmap v932: 109 pgs: 17 peering, 92 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 5.7 KiB/s rd, 51 MiB/s wr, 811 op/s; 51 KiB/s, 1 objects/s recovering 2024-09-21T10:11:00.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:00 smithi107 ceph-mon[105022]: pgmap v932: 109 pgs: 17 peering, 92 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 5.7 KiB/s rd, 51 MiB/s wr, 811 op/s; 51 KiB/s, 1 objects/s recovering 2024-09-21T10:11:00.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:00.981 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:00.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:00.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:01.083 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:01.113 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:01.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:01.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:01.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:01.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:11:01.245 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:01.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:01.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:01.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:01.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:01.400 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:01.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:01.501 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:01.593 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:01.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:01.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:01.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:01.732 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 69 16 94060 94044 85.1632 54.875 0.017582 0.0117366 2024-09-21T10:11:01.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:01.751 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:01.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:01.821 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:11:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:11:01] "GET /metrics HTTP/1.1" 200 36802 "" "Prometheus/2.51.0" 2024-09-21T10:11:01.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:01.893 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:01.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:01.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:01.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:02.031 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:02.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:02.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:02.133 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:02.161 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:02.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:02.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:02.263 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:02.296 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:02.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:02.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:02.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:02.437 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:02.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:02.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:02.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:02 smithi107 ceph-mon[102221]: pgmap v933: 109 pgs: 17 peering, 92 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 5.7 KiB/s rd, 41 MiB/s wr, 657 op/s; 51 KiB/s, 1 objects/s recovering 2024-09-21T10:11:02.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:02 smithi107 ceph-mon[105022]: pgmap v933: 109 pgs: 17 peering, 92 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 5.7 KiB/s rd, 41 MiB/s wr, 657 op/s; 51 KiB/s, 1 objects/s recovering 2024-09-21T10:11:02.539 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:02.545 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:02 smithi129 ceph-mon[88471]: pgmap v933: 109 pgs: 17 peering, 92 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 5.7 KiB/s rd, 41 MiB/s wr, 657 op/s; 51 KiB/s, 1 objects/s recovering 2024-09-21T10:11:02.574 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:02.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:02.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:02.584 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:11:02.584 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:11:02.584 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.584 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36561.0:94827 11.1 11:878a7e9e:::benchmark_data_smithi107_218812_object94826:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-21T10:11:02.584 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:11:02.575365+0000", 2024-09-21T10:11:02.584 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.005311199, 2024-09-21T10:11:02.584 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0053438720000000004, 2024-09-21T10:11:02.584 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:02.584 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:11:02.584 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:02.584 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36561", 2024-09-21T10:11:02.585 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:11:02.585 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 94827 2024-09-21T10:11:02.585 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.585 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:02.585 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.585 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:02.585 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.575365+0000", 2024-09-21T10:11:02.585 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:02.585 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.585 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.585 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:02.585 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.575365+0000", 2024-09-21T10:11:02.585 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:02.585 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.585 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.586 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:02.586 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.575364+0000", 2024-09-21T10:11:02.586 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:11:02.586 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.586 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.586 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:02.586 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.575461+0000", 2024-09-21T10:11:02.586 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.7281000000000006e-05 2024-09-21T10:11:02.586 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.586 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.586 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:02.586 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.575474+0000", 2024-09-21T10:11:02.586 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3327e-05 2024-09-21T10:11:02.586 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.587 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.587 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:02.587 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.575480+0000", 2024-09-21T10:11:02.587 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6590000000000004e-06 2024-09-21T10:11:02.587 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.587 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.587 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:02.587 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.575618+0000", 2024-09-21T10:11:02.587 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013833300000000001 2024-09-21T10:11:02.587 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.587 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.587 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:02.587 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.575745+0000", 2024-09-21T10:11:02.587 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012715200000000001 2024-09-21T10:11:02.588 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.588 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.588 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:11:02.588 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.575847+0000", 2024-09-21T10:11:02.588 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000101962 2024-09-21T10:11:02.588 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.588 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.588 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:02.588 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.579000+0000", 2024-09-21T10:11:02.588 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003152307 2024-09-21T10:11:02.588 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:02.588 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:02.588 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:02.589 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.589 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.589 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:94829 11.a e268/267 11:59c82b51:::benchmark_data_smithi107_218812_object94828:head v 268'5900, mlcod=268'5898)", 2024-09-21T10:11:02.589 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:11:02.579329+0000", 2024-09-21T10:11:02.589 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.0013478330000000001, 2024-09-21T10:11:02.589 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001459087, 2024-09-21T10:11:02.589 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:02.589 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "started", 2024-09-21T10:11:02.589 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:02.589 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.589 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:02.589 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.579329+0000", 2024-09-21T10:11:02.589 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:02.590 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.590 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.590 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:02.590 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.579329+0000", 2024-09-21T10:11:02.590 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:02.590 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.590 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.590 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:02.590 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.579328+0000", 2024-09-21T10:11:02.590 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:11:02.590 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.590 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.590 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:02.591 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.579379+0000", 2024-09-21T10:11:02.591 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0772000000000001e-05 2024-09-21T10:11:02.591 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.591 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.591 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:02.591 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.579392+0000", 2024-09-21T10:11:02.591 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3512e-05 2024-09-21T10:11:02.591 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.591 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.591 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:02.591 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.579398+0000", 2024-09-21T10:11:02.591 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1399999999999997e-06 2024-09-21T10:11:02.591 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.592 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.592 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:02.592 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.579461+0000", 2024-09-21T10:11:02.592 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3093000000000001e-05 2024-09-21T10:11:02.592 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.592 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.592 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:02.592 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.579487+0000", 2024-09-21T10:11:02.592 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.5885e-05 2024-09-21T10:11:02.592 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:02.592 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:02.592 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:02.592 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.592 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.592 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36561.0:94832 11.3 11:cc679206:::benchmark_data_smithi107_218812_object94831:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-21T10:11:02.593 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:11:02.579854+0000", 2024-09-21T10:11:02.593 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.00082229000000000004, 2024-09-21T10:11:02.593 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00099023800000000006, 2024-09-21T10:11:02.593 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:02.593 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:11:02.593 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:02.593 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36561", 2024-09-21T10:11:02.593 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:11:02.593 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 94832 2024-09-21T10:11:02.593 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.593 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:02.593 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.593 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:02.593 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.579854+0000", 2024-09-21T10:11:02.594 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:02.594 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.594 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.594 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:02.594 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.579854+0000", 2024-09-21T10:11:02.594 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:02.594 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.594 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.594 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:02.594 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.579852+0000", 2024-09-21T10:11:02.594 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:11:02.594 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.594 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.594 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:02.595 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.579917+0000", 2024-09-21T10:11:02.595 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4950000000000007e-05 2024-09-21T10:11:02.595 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.595 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.595 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:02.595 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.579931+0000", 2024-09-21T10:11:02.595 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3585e-05 2024-09-21T10:11:02.595 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.595 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.595 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:02.595 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.579937+0000", 2024-09-21T10:11:02.595 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4609999999999996e-06 2024-09-21T10:11:02.595 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.595 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.595 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:02.595 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.579988+0000", 2024-09-21T10:11:02.596 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1131000000000002e-05 2024-09-21T10:11:02.596 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.596 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.596 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:02.596 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.580126+0000", 2024-09-21T10:11:02.596 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000137582 2024-09-21T10:11:02.596 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.596 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.596 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,7", 2024-09-21T10:11:02.596 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.580261+0000", 2024-09-21T10:11:02.596 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013489700000000001 2024-09-21T10:11:02.596 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:02.596 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:02.596 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:02.597 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.597 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.597 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36561.0:94817 11.1 11:85db25c6:::benchmark_data_smithi107_218812_object94816:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-21T10:11:02.597 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:11:02.563129+0000", 2024-09-21T10:11:02.597 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.017547289000000001, 2024-09-21T10:11:02.597 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.017782768000000001, 2024-09-21T10:11:02.597 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:02.597 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:11:02.597 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:02.597 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36561", 2024-09-21T10:11:02.597 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:11:02.597 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 94817 2024-09-21T10:11:02.597 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.597 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:02.598 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.598 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:02.598 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.563129+0000", 2024-09-21T10:11:02.598 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:02.598 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.598 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.598 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:02.598 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.563129+0000", 2024-09-21T10:11:02.598 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:02.598 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.598 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.598 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:02.598 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.563129+0000", 2024-09-21T10:11:02.599 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:11:02.599 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.599 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.599 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:02.599 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.563156+0000", 2024-09-21T10:11:02.599 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.6951e-05 2024-09-21T10:11:02.599 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.599 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.599 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:02.599 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.563163+0000", 2024-09-21T10:11:02.599 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8800000000000002e-06 2024-09-21T10:11:02.599 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.599 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.599 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:02.600 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.563166+0000", 2024-09-21T10:11:02.600 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.766e-06 2024-09-21T10:11:02.600 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.600 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.600 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:02.600 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.566643+0000", 2024-09-21T10:11:02.600 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0034771530000000002 2024-09-21T10:11:02.600 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.600 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.600 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:02.600 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.566784+0000", 2024-09-21T10:11:02.600 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000140505 2024-09-21T10:11:02.600 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.600 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.600 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:11:02.601 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.566942+0000", 2024-09-21T10:11:02.601 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015813099999999999 2024-09-21T10:11:02.601 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.601 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.601 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:02.601 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.574086+0000", 2024-09-21T10:11:02.601 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0071438439999999999 2024-09-21T10:11:02.601 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:02.601 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:02.601 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:02.601 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.601 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.601 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36561.0:94825 11.1 11:87b65c96:::benchmark_data_smithi107_218812_object94824:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-09-21T10:11:02.601 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:11:02.567377+0000", 2024-09-21T10:11:02.602 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.013299154000000001, 2024-09-21T10:11:02.602 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.013607862, 2024-09-21T10:11:02.602 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:02.602 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:11:02.602 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:02.602 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36561", 2024-09-21T10:11:02.602 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:11:02.602 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 94825 2024-09-21T10:11:02.602 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.602 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:02.602 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.602 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:02.602 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.567377+0000", 2024-09-21T10:11:02.602 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:02.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.603 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.603 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:02.603 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.567377+0000", 2024-09-21T10:11:02.603 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:02.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.603 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.603 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:02.603 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.567376+0000", 2024-09-21T10:11:02.603 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:11:02.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.603 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.603 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:02.603 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.567471+0000", 2024-09-21T10:11:02.604 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.5292999999999997e-05 2024-09-21T10:11:02.604 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.604 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.604 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:02.604 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.567487+0000", 2024-09-21T10:11:02.604 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5860999999999999e-05 2024-09-21T10:11:02.604 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.604 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.604 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:02.604 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.567493+0000", 2024-09-21T10:11:02.604 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4049999999999997e-06 2024-09-21T10:11:02.604 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.604 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.604 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:02.605 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.569627+0000", 2024-09-21T10:11:02.605 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0021333889999999999 2024-09-21T10:11:02.605 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.605 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.605 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:02.605 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.569744+0000", 2024-09-21T10:11:02.605 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000117009 2024-09-21T10:11:02.605 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.605 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.605 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:11:02.605 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.569890+0000", 2024-09-21T10:11:02.605 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000145716 2024-09-21T10:11:02.605 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:02.605 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:02.606 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:02.606 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:02.574302+0000", 2024-09-21T10:11:02.606 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0044126599999999997 2024-09-21T10:11:02.606 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:02.606 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:02.606 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:02.606 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:02.606 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-21T10:11:02.606 INFO:teuthology.orchestra.run.smithi129.stdout: "num_ops": 5 2024-09-21T10:11:02.606 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:11:02.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:02.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:02.709 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:02.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:02.732 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 70 16 94990 94974 84.777 58.125 0.0311793 0.0117903 2024-09-21T10:11:02.810 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:02.845 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:02.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:02.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:02.948 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:02.979 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:02.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:03.081 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:03.105 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:03.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:03.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:03.207 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:03.251 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:03.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:03.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:03.352 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:03.377 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:03.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:03.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:03.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:03.490 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:11:03.527 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:03.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:03.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:03.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:03.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:03.668 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:03.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:03.720 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:03.733 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 71 16 95910 95894 84.3905 57.5 0.0136292 0.0118436 2024-09-21T10:11:03.770 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:03.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:03.814 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:03.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:03.867 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:11:03.915 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:03.954 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:03.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:03.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:04.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:04.079 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:04.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:04.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:04.180 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:04.220 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:04.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:04.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:04.322 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:04.355 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:04.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:04.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:04.429 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T10:11:04.457 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:04.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:04.494 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:04.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:04.595 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:04.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:04.634 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:04.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:04.714 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:04 smithi107 ceph-mon[102221]: pgmap v934: 109 pgs: 3 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 105 active+clean; 5.7 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 4.9 KiB/s rd, 58 MiB/s wr, 930 op/s; 23015/280929 objects misplaced (8.192%); 938 KiB/s, 15 objects/s recovering 2024-09-21T10:11:04.714 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:04 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2222236193' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:11:04.714 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:04 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:11:04.715 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:04 smithi107 ceph-mon[105022]: pgmap v934: 109 pgs: 3 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 105 active+clean; 5.7 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 4.9 KiB/s rd, 58 MiB/s wr, 930 op/s; 23015/280929 objects misplaced (8.192%); 938 KiB/s, 15 objects/s recovering 2024-09-21T10:11:04.715 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:04 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2222236193' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:11:04.715 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:04 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:11:04.734 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 72 16 96860 96844 84.0431 59.375 0.00705829 0.0118931 2024-09-21T10:11:04.735 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:04.768 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:04.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:04.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:04.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:04.912 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:04.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:04.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:04 smithi129 ceph-mon[88471]: pgmap v934: 109 pgs: 3 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 105 active+clean; 5.7 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 4.9 KiB/s rd, 58 MiB/s wr, 930 op/s; 23015/280929 objects misplaced (8.192%); 938 KiB/s, 15 objects/s recovering 2024-09-21T10:11:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:04 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2222236193' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:11:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:04 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:11:05.013 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:05.049 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:05.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:05.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:05.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:05.189 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:05.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:05.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:05.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:05.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:05.331 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:05.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:05.432 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:05.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:05.466 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:05.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:05.567 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:05.593 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:05.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:05.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:05.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:05 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:11:05.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:05 smithi107 ceph-mon[102221]: osdmap e269: 8 total, 8 up, 7 in 2024-09-21T10:11:05.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:05 smithi107 ceph-mon[102221]: osdmap e270: 8 total, 8 up, 7 in 2024-09-21T10:11:05.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:05 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:11:05.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:05 smithi107 ceph-mon[105022]: osdmap e269: 8 total, 8 up, 7 in 2024-09-21T10:11:05.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:05 smithi107 ceph-mon[105022]: osdmap e270: 8 total, 8 up, 7 in 2024-09-21T10:11:05.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:05 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:11:05.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:05 smithi129 ceph-mon[88471]: osdmap e269: 8 total, 8 up, 7 in 2024-09-21T10:11:05.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:05 smithi129 ceph-mon[88471]: osdmap e270: 8 total, 8 up, 7 in 2024-09-21T10:11:05.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:05.732 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:05.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:05.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:05.734 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 73 16 97813 97797 83.7077 59.5625 0.0136808 0.0119406 2024-09-21T10:11:05.821 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:11:05.822 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:11:05.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:05.867 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:11:05.867 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:11:05.874 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:05.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:05.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:05.911 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:11:05.912 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:11:05.946 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:11:05.946 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:11:05.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:05.977 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:11:05.977 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:11:06.010 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:11:06.010 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:11:06.020 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:06.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:06.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:06.040 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:11:06.041 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:11:06.070 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:11:06.070 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:11:06.122 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:06.143 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:06.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:06.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:06.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:06.267 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:06.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:06.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:06.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:06.399 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:06.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:06.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:06.501 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:06.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:06.535 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:06.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:06.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:06 smithi107 ceph-mon[102221]: pgmap v936: 109 pgs: 3 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 105 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 4.3 KiB/s rd, 33 MiB/s wr, 535 op/s; 23015/283671 objects misplaced (8.113%); 819 KiB/s, 13 objects/s recovering 2024-09-21T10:11:06.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:06 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2001378313' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:06.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:06 smithi107 ceph-mon[105022]: pgmap v936: 109 pgs: 3 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 105 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 4.3 KiB/s rd, 33 MiB/s wr, 535 op/s; 23015/283671 objects misplaced (8.113%); 819 KiB/s, 13 objects/s recovering 2024-09-21T10:11:06.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:06 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2001378313' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:06.637 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:06.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:06 smithi129 ceph-mon[88471]: pgmap v936: 109 pgs: 3 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 105 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 4.3 KiB/s rd, 33 MiB/s wr, 535 op/s; 23015/283671 objects misplaced (8.113%); 819 KiB/s, 13 objects/s recovering 2024-09-21T10:11:06.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:06 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2001378313' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:06.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:06.672 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:06.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:06.734 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 74 16 98470 98454 83.1315 41.0625 0.0195416 0.0120241 2024-09-21T10:11:06.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:06.816 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:06.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:06.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:06.918 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:06.946 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:06.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:06.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:07.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:07.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:07.074 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:07.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:07.175 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:07.204 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:07.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:07.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:07.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:07.341 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:07.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:07.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:07.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T10:11:07.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:07.487 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:07.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:07.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:07.589 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:07.613 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:07.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:07.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:07.714 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:07.736 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 75 16 99310 99294 82.7211 52.5 0.0137866 0.0120829 2024-09-21T10:11:07.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:07.770 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:07.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:07.871 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:07.900 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:07.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:07.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:08.001 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:08.038 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:08.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:08.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:08.140 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:08.167 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:08.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:08.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:08.269 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:08.283 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:08.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:08.315 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:08.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:08.416 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:08.452 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:08.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:08.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:08.497 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:11:08.554 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:08.572 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:11:08.572 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:11:08.588 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:08.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:08.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:08.690 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:08.718 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:08.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:08.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:08.736 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 76 15 100197 100182 82.363 55.5 0.0142114 0.012136 2024-09-21T10:11:08.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:08.855 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:08.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:08.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:08 smithi107 ceph-mon[102221]: pgmap v938: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 50 MiB/s wr, 793 op/s; 23685/292620 objects misplaced (8.094%); 1.9 MiB/s, 31 objects/s recovering 2024-09-21T10:11:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:08 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1639027150' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:11:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:08 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:11:08.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:08 smithi107 ceph-mon[105022]: pgmap v938: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 50 MiB/s wr, 793 op/s; 23685/292620 objects misplaced (8.094%); 1.9 MiB/s, 31 objects/s recovering 2024-09-21T10:11:08.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:08 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1639027150' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:11:08.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:08 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:11:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:08 smithi129 ceph-mon[88471]: pgmap v938: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 50 MiB/s wr, 793 op/s; 23685/292620 objects misplaced (8.094%); 1.9 MiB/s, 31 objects/s recovering 2024-09-21T10:11:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:08 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1639027150' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:11:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:08 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:11:08.957 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:08.992 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:08.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:08.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:09.093 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:09.120 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:09.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:09.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:09.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:09.258 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:09.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:09.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:09.360 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:09.388 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:11:09.394 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:09.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:09.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:09.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:09.532 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:09.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:09.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:09.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:09.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:09.669 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:09.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:09.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:09 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:11:09.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:09 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:11:09.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:09 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:11:09.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:09 smithi129 ceph-mon[88471]: osdmap e271: 8 total, 8 up, 7 in 2024-09-21T10:11:09.736 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 77 16 101139 101123 82.0571 58.8125 0.0243942 0.0121794 2024-09-21T10:11:09.742 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:11:09.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:09.795 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:09.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:09.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:09.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:09 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:11:09.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:09 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:11:09.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:09 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:11:09.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:09 smithi107 ceph-mon[102221]: osdmap e271: 8 total, 8 up, 7 in 2024-09-21T10:11:09.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:09 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:11:09.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:09 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:11:09.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:09 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:11:09.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:09 smithi107 ceph-mon[105022]: osdmap e271: 8 total, 8 up, 7 in 2024-09-21T10:11:09.897 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:09.929 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:09.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:09.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:10.031 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:10.068 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:10.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:10.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:10.164 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:11:10.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:10.264 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:10.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:10.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:10.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:10.428 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:10.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:10.509 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:11:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:11:10.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:11:10.530 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:10.554 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:10.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:10.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:10.656 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:10.694 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:10.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:10.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:10.736 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 78 16 101904 101888 81.6181 47.8125 0.00822054 0.0122453 2024-09-21T10:11:10.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:10.831 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:10.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:10.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:10.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:10 smithi107 ceph-mon[102221]: pgmap v940: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 55 MiB/s wr, 879 op/s; 23685/296757 objects misplaced (7.981%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:11:10.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:10 smithi107 ceph-mon[105022]: pgmap v940: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 55 MiB/s wr, 879 op/s; 23685/296757 objects misplaced (7.981%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:11:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:10 smithi129 ceph-mon[88471]: pgmap v940: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 55 MiB/s wr, 879 op/s; 23685/296757 objects misplaced (7.981%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:11:10.932 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:10.958 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:10.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:10.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:11.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:11.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:11.099 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:11.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:11.200 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:11.233 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:11.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:11.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:11.335 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:11.372 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:11.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:11.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:11.474 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:11.510 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:11.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:11.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:11.612 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:11.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:11.644 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:11.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:11.694 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:11:11.694 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:11:11.736 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 79 16 102842 102826 81.327 58.625 0.0391796 0.0122909 2024-09-21T10:11:11.744 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:11.780 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:11.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:11.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:11.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:11.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:11.940 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:11.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:12.023 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:11:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:11:11] "GET /metrics HTTP/1.1" 200 36804 "" "Prometheus/2.51.0" 2024-09-21T10:11:12.041 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:12.082 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:12.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:12.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:12.185 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:12.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:12.220 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:12.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:12.321 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:12.352 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:12.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:12.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:12.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:11:12.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:12.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:12 smithi107 ceph-mon[105022]: pgmap v941: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 40 MiB/s wr, 637 op/s; 23685/296757 objects misplaced (7.981%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:11:12.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:11:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:11:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:12 smithi107 ceph-mon[102221]: pgmap v941: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 40 MiB/s wr, 637 op/s; 23685/296757 objects misplaced (7.981%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:11:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:11:12.454 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:12.489 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:12.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:12.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:12.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:12.610 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:12.637 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:12.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:12.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:12.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:11:12.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:12.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:12 smithi129 ceph-mon[88471]: pgmap v941: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 40 MiB/s wr, 637 op/s; 23685/296757 objects misplaced (7.981%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:11:12.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:11:12.737 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:11:12.735330+0000 min lat: 0.00109786 max lat: 2.40503 avg lat: 0.0123357 2024-09-21T10:11:12.737 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:11:12.737 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 80 16 103761 103745 81.0284 57.4375 0.00917026 0.0123357 2024-09-21T10:11:12.739 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:12.781 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:12.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:12.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:12.883 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:12.888 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:11:12.913 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:12.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:12.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:13.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:13.058 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:13.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:13.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:13.160 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:13.200 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:13.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:13.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:13.302 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:13.353 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:11:13.355 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:13.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:13.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:13.457 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:13.495 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:13.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:13.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:13.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:13.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:13.639 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:13.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:13.736 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 81 16 104624 104608 80.6939 53.9375 0.0259624 0.0123872 2024-09-21T10:11:13.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:13.779 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:13.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:13.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:13.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:11:13.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:13.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:13.960 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:13.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:14.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:14.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:14.098 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:14.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:14.199 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:14.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:14.242 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:14.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:14.335 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T10:11:14.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:14.384 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:14.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:14.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:14.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:14.522 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:14.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:14.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:14 smithi107 ceph-mon[102221]: pgmap v942: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 62 MiB/s wr, 989 op/s; 24350/307428 objects misplaced (7.921%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T10:11:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:14 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/19809073' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:11:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:14 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:11:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:14 smithi107 ceph-mon[105022]: pgmap v942: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 62 MiB/s wr, 989 op/s; 24350/307428 objects misplaced (7.921%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T10:11:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:14 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/19809073' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:11:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:14 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:11:14.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:14 smithi129 ceph-mon[88471]: pgmap v942: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 62 MiB/s wr, 989 op/s; 24350/307428 objects misplaced (7.921%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T10:11:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:14 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/19809073' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:11:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:14 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:11:14.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:14.670 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:14.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:14.736 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 82 16 105498 105482 80.376 54.625 0.0163691 0.0124363 2024-09-21T10:11:14.772 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:14.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:14.806 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:14.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:14.906 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:14.937 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:14.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:14.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:15.038 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:15.075 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:15.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:15.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:15.177 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:15.206 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:15.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:15.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:15.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:15.348 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:15.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:15.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:15.450 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:15.458 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T10:11:15.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:11:15.483 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:15.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:15.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:15.603 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:15 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:11:15.603 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:15 smithi107 ceph-mon[102221]: osdmap e272: 8 total, 8 up, 7 in 2024-09-21T10:11:15.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:15 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:11:15.603 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:15 smithi107 ceph-mon[105022]: osdmap e272: 8 total, 8 up, 7 in 2024-09-21T10:11:15.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:15.620 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:15.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:15.625 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:15 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:11:15.625 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:15 smithi129 ceph-mon[88471]: osdmap e272: 8 total, 8 up, 7 in 2024-09-21T10:11:15.626 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:11:15.626 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:11:15.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:15.737 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 83 16 106409 106393 80.0926 56.9375 0.016 0.0124801 2024-09-21T10:11:15.779 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:15.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:15.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:15.801 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:11:15.801 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:11:15.831 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:15.850 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:11:15.850 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:11:15.881 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:15.894 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:11:15.894 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:11:15.906 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:15.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:15.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:15.929 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:11:15.930 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:11:15.961 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:11:15.961 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:11:15.990 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:11:15.990 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:11:16.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:16.021 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:11:16.021 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:11:16.035 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:16.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:16.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:16.055 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:11:16.055 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:11:16.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:16.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:16.183 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:16.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:16.284 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:16.302 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:11:16.326 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:16.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:16.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:16.428 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:16.462 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:16.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:16.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:16.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:16 smithi107 ceph-mon[102221]: pgmap v944: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 45 MiB/s wr, 716 op/s; 24350/309819 objects misplaced (7.859%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:11:16.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:16 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1039102369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:16.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:16 smithi107 ceph-mon[105022]: pgmap v944: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 45 MiB/s wr, 716 op/s; 24350/309819 objects misplaced (7.859%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:11:16.509 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:16 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1039102369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:16.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:16.605 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:16.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:16.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:16.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:16 smithi129 ceph-mon[88471]: pgmap v944: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 45 MiB/s wr, 716 op/s; 24350/309819 objects misplaced (7.859%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:11:16.671 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:16 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1039102369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:16.672 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:11:16.706 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:16.738 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 84 16 107280 107264 79.7872 54.4375 0.0135748 0.0125283 2024-09-21T10:11:16.742 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:16.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:16.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:16.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:16.876 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:16.888 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:16.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:16.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:16.990 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:17.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:17.022 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:17.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:17.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:17.160 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:17.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:17.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:17.262 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:17.289 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:17.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:17.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:17.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:17.432 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:17.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:17.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:17.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:17.566 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:17.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:17.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:17.668 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:17.706 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:17.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:17.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:17.738 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 85 16 108182 108166 79.5117 56.375 0.0211276 0.0125717 2024-09-21T10:11:17.808 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:17.836 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:17.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:17.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:17.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:17.983 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:17.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:17.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:18.085 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:18.113 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:18.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:18.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:18.215 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:18.244 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:18.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:18.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:18.346 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:18.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:18.391 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:18.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:18.492 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:18.528 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:18.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:18.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:18.577 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:18 smithi129 ceph-mon[88471]: pgmap v945: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 59 MiB/s wr, 938 op/s; 25005/318963 objects misplaced (7.839%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T10:11:18.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:18 smithi107 ceph-mon[102221]: pgmap v945: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 59 MiB/s wr, 938 op/s; 25005/318963 objects misplaced (7.839%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T10:11:18.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:18 smithi107 ceph-mon[105022]: pgmap v945: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 59 MiB/s wr, 938 op/s; 25005/318963 objects misplaced (7.839%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T10:11:18.631 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:18.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:18.666 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:18.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:18.738 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 86 16 109132 109116 79.2776 59.375 0.0134928 0.012609 2024-09-21T10:11:18.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:18.814 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:18.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:18.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:18.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:18.975 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:18.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:18.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:19.077 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:19.108 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:19.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:19.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:19.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:19.247 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:19.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:19.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:19.357 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:11:19.357 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:11:19.357 INFO:teuthology.orchestra.run.smithi129.stdout: "complaint_time": 30, 2024-09-21T10:11:19.357 INFO:teuthology.orchestra.run.smithi129.stdout: "num_blocked_ops": 0 2024-09-21T10:11:19.357 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:11:19.389 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:19.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:19.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:19.491 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:19.522 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:19.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:19.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:19.623 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:19.646 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:19.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:19.738 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 87 16 110024 110008 79.0071 55.75 0.00392655 0.0126493 2024-09-21T10:11:19.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:19.770 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:19.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:19.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:19.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:19.910 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:19.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:19.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:19.936 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:11:20.012 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:20.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:20.038 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:20.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:20.139 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:20.167 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:20.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:20.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:20.269 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:20.344 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:20.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:20.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:20.379 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:11:20.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:20.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:20.488 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:20.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:20.589 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:20.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:20 smithi107 ceph-mon[102221]: pgmap v946: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 55 MiB/s wr, 885 op/s; 25005/323334 objects misplaced (7.733%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:11:20.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:20 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1134338014' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:11:20.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:20 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:11:20.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:20 smithi107 ceph-mon[105022]: pgmap v946: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 55 MiB/s wr, 885 op/s; 25005/323334 objects misplaced (7.733%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:11:20.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:20 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1134338014' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:11:20.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:20 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:11:20.618 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:20.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:20.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:20.622 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:20.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:20 smithi129 ceph-mon[88471]: pgmap v946: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 55 MiB/s wr, 885 op/s; 25005/323334 objects misplaced (7.733%); 1.8 MiB/s, 29 objects/s recovering 2024-09-21T10:11:20.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:20 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1134338014' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:11:20.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:20 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:11:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:11:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:11:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:11:20.720 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:20.738 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 88 16 110745 110729 78.6212 45.0625 0.0106496 0.0127139 2024-09-21T10:11:20.751 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:20.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:20.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:20.853 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:20.882 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:20.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:20.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:20.984 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:21.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:21.015 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:21.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:21.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:21.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:21.163 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:21.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:21.264 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:21.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:21.299 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:21.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:21.400 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:21.429 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:21.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:21.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:21.531 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:21.565 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:21.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:21.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:21.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:21 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:11:21.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:21 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:11:21.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:21 smithi129 ceph-mon[88471]: osdmap e273: 8 total, 8 up, 7 in 2024-09-21T10:11:21.695 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:21.695 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:21 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:11:21.696 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:21 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:11:21.696 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:21 smithi107 ceph-mon[102221]: osdmap e273: 8 total, 8 up, 7 in 2024-09-21T10:11:21.696 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:21 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:11:21.696 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:21 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:11:21.696 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:21 smithi107 ceph-mon[105022]: osdmap e273: 8 total, 8 up, 7 in 2024-09-21T10:11:21.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:21.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:21.738 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 89 16 111653 111637 78.3754 56.75 0.0147533 0.0127539 2024-09-21T10:11:21.798 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:21.822 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:21.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:21.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:21.924 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:21.947 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:11:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:11:21] "GET /metrics HTTP/1.1" 200 36804 "" "Prometheus/2.51.0" 2024-09-21T10:11:21.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:21.951 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:21.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:22.052 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:22.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:22.082 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:22.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:22.094 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:11:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:11:22.183 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:22.216 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:22.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:22.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:22.318 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:22.339 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:22.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:22.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:22.441 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:22.471 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:22.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:22.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:22.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:22.608 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:22.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:22.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:22.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:22 smithi107 ceph-mon[102221]: pgmap v948: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 41 MiB/s wr, 662 op/s; 25005/323334 objects misplaced (7.733%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:11:22.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:22 smithi107 ceph-mon[105022]: pgmap v948: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 41 MiB/s wr, 662 op/s; 25005/323334 objects misplaced (7.733%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:11:22.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:22 smithi129 ceph-mon[88471]: pgmap v948: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 41 MiB/s wr, 662 op/s; 25005/323334 objects misplaced (7.733%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:11:22.710 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:22.734 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:22.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:22.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:22.738 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 90 4 112484 112480 78.09 52.6875 0.0100904 0.0127982 2024-09-21T10:11:22.833 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:11:22.836 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:22.898 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:22.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:22.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:23.000 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:23.021 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:23.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:23.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:23.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:11:23.109 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:11:23.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:23.147 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:23.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:23.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:23.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:23.272 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:23.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:23.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:23.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:23.395 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:23.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:23.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:23.497 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:23.519 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:23.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:23.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:23.544 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:23 smithi129 ceph-mon[88471]: from='client.36636 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:11:23.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:23.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:23.644 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:23.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:23.739 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total time run: 90.0353 2024-09-21T10:11:23.739 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total writes made: 112484 2024-09-21T10:11:23.739 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Write size: 65536 2024-09-21T10:11:23.739 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object size: 65536 2024-09-21T10:11:23.739 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Bandwidth (MB/sec): 78.0832 2024-09-21T10:11:23.739 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Bandwidth: 35.6774 2024-09-21T10:11:23.739 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max bandwidth (MB/sec): 218.75 2024-09-21T10:11:23.739 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min bandwidth (MB/sec): 0 2024-09-21T10:11:23.739 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average IOPS: 1249 2024-09-21T10:11:23.739 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev IOPS: 570.85 2024-09-21T10:11:23.740 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max IOPS: 3500 2024-09-21T10:11:23.740 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min IOPS: 0 2024-09-21T10:11:23.740 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average Latency(s): 0.0127997 2024-09-21T10:11:23.740 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Latency(s): 0.0296922 2024-09-21T10:11:23.740 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max latency(s): 2.40503 2024-09-21T10:11:23.740 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min latency(s): 0.00109786 2024-09-21T10:11:23.744 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:23.745 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:11:23.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:23.794 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:23.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:23.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:23 smithi107 ceph-mon[102221]: from='client.36636 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:11:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:23 smithi107 ceph-mon[105022]: from='client.36636 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:11:23.894 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:23.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:23.930 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:23.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:24.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:24.072 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:24.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:24.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:24.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:24.210 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:24.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:24.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:24.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:24.344 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:24.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:24.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:24.445 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:24.484 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:24.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:24.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:24.551 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:24 smithi129 ceph-mon[88471]: pgmap v949: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 56 MiB/s wr, 896 op/s; 25682/333882 objects misplaced (7.692%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:11:24.586 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:24.620 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:24.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:24.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:24.721 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:24.759 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:24.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:24.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:24 smithi107 ceph-mon[102221]: pgmap v949: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 56 MiB/s wr, 896 op/s; 25682/333882 objects misplaced (7.692%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:11:24.761 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:24 smithi107 ceph-mon[105022]: pgmap v949: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 56 MiB/s wr, 896 op/s; 25682/333882 objects misplaced (7.692%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:11:24.861 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:24.883 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:24.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:24.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:24.985 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:25.011 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:11:25.011 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 20, 2024-09-21T10:11:25.011 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 600, 2024-09-21T10:11:25.011 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:11:25.011 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.011 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95081 10.c 10:3ce718bd:::benchmark_data_smithi107_184135_object95080:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T10:11:25.011 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.369602+0000", 2024-09-21T10:11:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 318.63706587299998, 2024-09-21T10:11:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0566048509999999, 2024-09-21T10:11:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:11:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:11:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:11:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95081 2024-09-21T10:11:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:25.012 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.369602+0000", 2024-09-21T10:11:25.013 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.013 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.013 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.013 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:25.013 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.369602+0000", 2024-09-21T10:11:25.013 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.013 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.013 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.013 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:25.013 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.369601+0000", 2024-09-21T10:11:25.013 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:11:25.013 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.013 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.013 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:25.014 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.369721+0000", 2024-09-21T10:11:25.014 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012002000000000001 2024-09-21T10:11:25.014 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.014 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.014 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:25.014 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.369736+0000", 2024-09-21T10:11:25.014 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5016e-05 2024-09-21T10:11:25.014 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.014 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.014 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:25.014 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.369742+0000", 2024-09-21T10:11:25.014 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7150000000000003e-06 2024-09-21T10:11:25.014 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.015 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.015 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.015 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.369790+0000", 2024-09-21T10:11:25.015 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8238000000000001e-05 2024-09-21T10:11:25.015 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.015 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.015 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:25.015 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.369927+0000", 2024-09-21T10:11:25.015 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013691699999999999 2024-09-21T10:11:25.015 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.015 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.015 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:11:25.015 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.370082+0000", 2024-09-21T10:11:25.015 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000155616 2024-09-21T10:11:25.016 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.016 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.016 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:25.016 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.373661+0000", 2024-09-21T10:11:25.016 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0035782660000000001 2024-09-21T10:11:25.016 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.016 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.016 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.016 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.377955+0000", 2024-09-21T10:11:25.016 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0042938689999999996 2024-09-21T10:11:25.016 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.016 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.016 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.016 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.398356+0000", 2024-09-21T10:11:25.017 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.020401263999999999 2024-09-21T10:11:25.017 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.017 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.017 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.017 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.398364+0000", 2024-09-21T10:11:25.017 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9160000000000001e-06 2024-09-21T10:11:25.017 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.017 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.017 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.017 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.426137+0000", 2024-09-21T10:11:25.017 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0277736980000001 2024-09-21T10:11:25.017 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.017 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.017 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:11:25.018 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.426207+0000", 2024-09-21T10:11:25.018 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9596000000000002e-05 2024-09-21T10:11:25.018 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.018 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.018 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.018 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.018 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.018 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95085 10.c 10:33c119cf:::benchmark_data_smithi107_184135_object95084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T10:11:25.018 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.374530+0000", 2024-09-21T10:11:25.018 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 318.63213852899997, 2024-09-21T10:11:25.018 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.051753323, 2024-09-21T10:11:25.018 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:25.018 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:11:25.018 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:25.019 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:11:25.019 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:11:25.019 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95085 2024-09-21T10:11:25.019 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.019 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:25.019 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.019 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:25.019 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.374530+0000", 2024-09-21T10:11:25.019 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.019 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.019 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.019 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:25.019 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.374530+0000", 2024-09-21T10:11:25.019 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.020 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.020 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.020 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:25.020 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.374528+0000", 2024-09-21T10:11:25.020 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:11:25.020 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.020 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.020 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:25.020 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.374594+0000", 2024-09-21T10:11:25.020 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6126999999999999e-05 2024-09-21T10:11:25.020 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.020 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.020 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:25.020 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.374607+0000", 2024-09-21T10:11:25.020 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3688e-05 2024-09-21T10:11:25.021 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.021 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.021 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:25.021 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.374613+0000", 2024-09-21T10:11:25.021 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.242e-06 2024-09-21T10:11:25.021 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.021 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.021 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.021 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.374662+0000", 2024-09-21T10:11:25.021 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9156000000000002e-05 2024-09-21T10:11:25.021 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.021 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.021 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:25.021 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.374797+0000", 2024-09-21T10:11:25.022 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000135329 2024-09-21T10:11:25.022 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.022 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.022 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:11:25.022 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.374898+0000", 2024-09-21T10:11:25.022 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000101185 2024-09-21T10:11:25.022 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.022 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.022 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:25.022 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.377741+0000", 2024-09-21T10:11:25.022 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0028426800000000002 2024-09-21T10:11:25.022 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.022 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.022 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.022 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.378262+0000", 2024-09-21T10:11:25.023 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00052137799999999997 2024-09-21T10:11:25.023 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.023 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.023 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.023 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.398390+0000", 2024-09-21T10:11:25.023 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.020127546 2024-09-21T10:11:25.023 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.023 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.023 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.023 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.398395+0000", 2024-09-21T10:11:25.023 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.741e-06 2024-09-21T10:11:25.023 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.023 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.023 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.024 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.426235+0000", 2024-09-21T10:11:25.024 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.027840436 2024-09-21T10:11:25.024 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.024 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.024 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:11:25.024 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.426283+0000", 2024-09-21T10:11:25.024 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7763000000000003e-05 2024-09-21T10:11:25.024 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.024 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.024 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.024 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.024 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.024 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247135 10.1 10:8bb6709e:::benchmark_data_smithi107_184135_object115569:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:11:25.025 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:49.452878+0000", 2024-09-21T10:11:25.025 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 155.55378986900001, 2024-09-21T10:11:25.025 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.958185693, 2024-09-21T10:11:25.025 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:25.025 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:25.025 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:25.025 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:11:25.026 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:11:25.026 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247135 2024-09-21T10:11:25.026 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.026 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:25.026 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.026 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:25.026 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.452878+0000", 2024-09-21T10:11:25.026 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.026 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.026 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.026 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:25.026 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.452878+0000", 2024-09-21T10:11:25.026 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.026 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.027 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.027 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:25.027 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.452876+0000", 2024-09-21T10:11:25.027 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:11:25.027 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.027 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.027 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:25.027 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.452884+0000", 2024-09-21T10:11:25.027 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.5839999999999997e-06 2024-09-21T10:11:25.027 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.027 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.027 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:25.027 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.452897+0000", 2024-09-21T10:11:25.027 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3644e-05 2024-09-21T10:11:25.028 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.028 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.028 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:25.028 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.452904+0000", 2024-09-21T10:11:25.028 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8070000000000002e-06 2024-09-21T10:11:25.028 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.028 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.028 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.028 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.452950+0000", 2024-09-21T10:11:25.028 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5348e-05 2024-09-21T10:11:25.028 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.028 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.028 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.028 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.452959+0000", 2024-09-21T10:11:25.028 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.0620000000000007e-06 2024-09-21T10:11:25.029 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.029 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.029 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.029 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.396167+0000", 2024-09-21T10:11:25.029 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.94320812600000004 2024-09-21T10:11:25.029 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.029 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.029 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:25.029 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.396257+0000", 2024-09-21T10:11:25.029 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.0739e-05 2024-09-21T10:11:25.029 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.029 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.029 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:11:25.030 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.396336+0000", 2024-09-21T10:11:25.030 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.8697999999999998e-05 2024-09-21T10:11:25.030 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.030 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.030 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:25.030 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.405056+0000", 2024-09-21T10:11:25.030 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0087199270000000006 2024-09-21T10:11:25.030 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.030 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.030 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.030 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.410930+0000", 2024-09-21T10:11:25.030 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0058740019999999997 2024-09-21T10:11:25.030 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.030 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.031 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.031 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.410985+0000", 2024-09-21T10:11:25.031 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4898000000000002e-05 2024-09-21T10:11:25.031 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.031 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.031 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:11:25.031 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411015+0000", 2024-09-21T10:11:25.031 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.9828999999999999e-05 2024-09-21T10:11:25.031 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.031 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.031 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:11:25.031 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411064+0000", 2024-09-21T10:11:25.031 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9058999999999999e-05 2024-09-21T10:11:25.031 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.032 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.032 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.032 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.032 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.032 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247136 10.1 10:864af190:::benchmark_data_smithi107_184135_object115570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:11:25.032 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:49.454413+0000", 2024-09-21T10:11:25.032 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 155.55225496200001, 2024-09-21T10:11:25.032 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.95676656000000004, 2024-09-21T10:11:25.032 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:25.032 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:25.032 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:25.032 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:11:25.032 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:11:25.032 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247136 2024-09-21T10:11:25.033 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.033 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:25.033 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.033 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:25.033 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.454413+0000", 2024-09-21T10:11:25.033 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.033 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.033 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.033 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:25.033 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.454413+0000", 2024-09-21T10:11:25.033 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.033 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.033 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.033 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:25.034 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.454412+0000", 2024-09-21T10:11:25.034 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:11:25.034 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.034 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.034 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:25.034 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.454421+0000", 2024-09-21T10:11:25.034 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6030000000000005e-06 2024-09-21T10:11:25.034 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.034 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.034 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:25.034 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.454436+0000", 2024-09-21T10:11:25.034 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5727999999999999e-05 2024-09-21T10:11:25.034 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.034 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.034 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:25.035 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.454442+0000", 2024-09-21T10:11:25.035 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4809999999999997e-06 2024-09-21T10:11:25.035 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.035 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.035 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.035 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.455319+0000", 2024-09-21T10:11:25.035 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00087677499999999999 2024-09-21T10:11:25.035 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.035 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.035 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.035 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.455325+0000", 2024-09-21T10:11:25.035 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6989999999999997e-06 2024-09-21T10:11:25.035 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.035 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.036 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.036 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.398618+0000", 2024-09-21T10:11:25.036 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.94329290099999996 2024-09-21T10:11:25.036 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.036 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.036 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:25.036 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.398701+0000", 2024-09-21T10:11:25.036 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.2421000000000003e-05 2024-09-21T10:11:25.036 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.036 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.036 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:11:25.036 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.398770+0000", 2024-09-21T10:11:25.036 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9162e-05 2024-09-21T10:11:25.036 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.036 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.037 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:25.037 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.408712+0000", 2024-09-21T10:11:25.037 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0099424209999999999 2024-09-21T10:11:25.037 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.037 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.037 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.037 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.410967+0000", 2024-09-21T10:11:25.037 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0022543630000000001 2024-09-21T10:11:25.037 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.037 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.037 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.037 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411099+0000", 2024-09-21T10:11:25.037 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013281999999999999 2024-09-21T10:11:25.037 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.037 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.038 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:11:25.038 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411130+0000", 2024-09-21T10:11:25.038 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0416000000000002e-05 2024-09-21T10:11:25.038 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.038 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.038 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:11:25.038 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411180+0000", 2024-09-21T10:11:25.038 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9802999999999997e-05 2024-09-21T10:11:25.038 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.038 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.038 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.038 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.038 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.038 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247143 10.1 10:86c515c8:::benchmark_data_smithi107_184135_object115577:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:11:25.039 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:49.459691+0000", 2024-09-21T10:11:25.039 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 155.54697705999999, 2024-09-21T10:11:25.039 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.95170592899999995, 2024-09-21T10:11:25.039 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:25.039 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:25.039 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:25.039 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:11:25.039 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:11:25.039 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247143 2024-09-21T10:11:25.039 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.039 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:25.039 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.039 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:25.039 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.459691+0000", 2024-09-21T10:11:25.040 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.040 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.040 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.040 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:25.040 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.459691+0000", 2024-09-21T10:11:25.040 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.040 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.040 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.040 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:25.040 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.459689+0000", 2024-09-21T10:11:25.040 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:11:25.040 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.040 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.041 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:25.041 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.459697+0000", 2024-09-21T10:11:25.041 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2899999999999997e-06 2024-09-21T10:11:25.041 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.041 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.041 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:25.041 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.459709+0000", 2024-09-21T10:11:25.041 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2218000000000001e-05 2024-09-21T10:11:25.041 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.041 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.041 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:25.041 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.459715+0000", 2024-09-21T10:11:25.041 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5140000000000001e-06 2024-09-21T10:11:25.041 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.042 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.042 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.042 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.462126+0000", 2024-09-21T10:11:25.042 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0024117359999999998 2024-09-21T10:11:25.042 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.042 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.042 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.042 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.462135+0000", 2024-09-21T10:11:25.042 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.132e-06 2024-09-21T10:11:25.042 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.042 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.042 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.042 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.400841+0000", 2024-09-21T10:11:25.042 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.93870596299999998 2024-09-21T10:11:25.043 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.043 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.043 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:25.043 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.400922+0000", 2024-09-21T10:11:25.043 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.0877000000000005e-05 2024-09-21T10:11:25.043 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.043 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.043 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:11:25.043 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.400994+0000", 2024-09-21T10:11:25.043 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.1768999999999997e-05 2024-09-21T10:11:25.043 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.043 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.043 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:25.043 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.408719+0000", 2024-09-21T10:11:25.044 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0077246820000000001 2024-09-21T10:11:25.044 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.044 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.044 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.044 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411217+0000", 2024-09-21T10:11:25.044 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0024978880000000002 2024-09-21T10:11:25.044 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.044 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.044 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.044 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411282+0000", 2024-09-21T10:11:25.044 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4969999999999996e-05 2024-09-21T10:11:25.044 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.044 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.044 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:11:25.045 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411310+0000", 2024-09-21T10:11:25.045 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.8738000000000002e-05 2024-09-21T10:11:25.045 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.045 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.045 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:11:25.045 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411397+0000", 2024-09-21T10:11:25.045 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6639999999999997e-05 2024-09-21T10:11:25.045 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.045 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.045 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.045 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.045 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.045 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247190 10.1 10:8be1d16c:::benchmark_data_smithi107_184135_object115624:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:11:25.046 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:49.511817+0000", 2024-09-21T10:11:25.046 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 155.49485091299999, 2024-09-21T10:11:25.046 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.89981461900000004, 2024-09-21T10:11:25.046 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:25.046 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:25.046 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:25.046 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:11:25.046 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:11:25.046 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247190 2024-09-21T10:11:25.046 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.046 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:25.046 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.046 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:25.046 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.511817+0000", 2024-09-21T10:11:25.047 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.047 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.047 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.047 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:25.047 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.511817+0000", 2024-09-21T10:11:25.047 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.047 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.047 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.047 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:25.047 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.511816+0000", 2024-09-21T10:11:25.047 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:11:25.047 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.047 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.047 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:25.047 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.511822+0000", 2024-09-21T10:11:25.048 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.054e-06 2024-09-21T10:11:25.048 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.048 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.048 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:25.048 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.511836+0000", 2024-09-21T10:11:25.048 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4123e-05 2024-09-21T10:11:25.048 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.048 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.048 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:25.048 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.511842+0000", 2024-09-21T10:11:25.048 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4439999999999996e-06 2024-09-21T10:11:25.048 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.048 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.048 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.049 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.511907+0000", 2024-09-21T10:11:25.049 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4968000000000005e-05 2024-09-21T10:11:25.049 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.049 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.049 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.049 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.511915+0000", 2024-09-21T10:11:25.049 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9189999999999999e-06 2024-09-21T10:11:25.049 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.049 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.049 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.049 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.402760+0000", 2024-09-21T10:11:25.049 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.89084475100000005 2024-09-21T10:11:25.049 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.049 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.049 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:25.050 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.402877+0000", 2024-09-21T10:11:25.050 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000116815 2024-09-21T10:11:25.050 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.050 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.050 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:11:25.050 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.403008+0000", 2024-09-21T10:11:25.050 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013146100000000001 2024-09-21T10:11:25.050 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.050 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.050 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:25.050 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.410523+0000", 2024-09-21T10:11:25.050 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0075145780000000001 2024-09-21T10:11:25.050 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.050 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.051 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.051 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411257+0000", 2024-09-21T10:11:25.051 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00073459800000000004 2024-09-21T10:11:25.051 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.051 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.051 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.051 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411548+0000", 2024-09-21T10:11:25.051 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00029048799999999999 2024-09-21T10:11:25.051 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.051 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.051 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:11:25.051 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411576+0000", 2024-09-21T10:11:25.051 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7807999999999999e-05 2024-09-21T10:11:25.051 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.051 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.052 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:11:25.052 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411632+0000", 2024-09-21T10:11:25.052 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6016e-05 2024-09-21T10:11:25.052 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.052 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.052 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.052 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.052 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:25.053 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:25.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:25.054 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247201 10.1 10:8dbea996:::benchmark_data_smithi107_184135_object115635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:11:25.054 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:49.523792+0000", 2024-09-21T10:11:25.054 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 155.482876296, 2024-09-21T10:11:25.054 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.88794138099999997, 2024-09-21T10:11:25.054 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:25.054 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:25.054 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:25.055 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:11:25.055 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:11:25.055 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247201 2024-09-21T10:11:25.055 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.055 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:25.055 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.055 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:25.055 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.523792+0000", 2024-09-21T10:11:25.055 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.055 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.055 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.055 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:25.055 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.523792+0000", 2024-09-21T10:11:25.055 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.056 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.056 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.056 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:25.056 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.523791+0000", 2024-09-21T10:11:25.056 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:11:25.056 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.056 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.056 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:25.056 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.523797+0000", 2024-09-21T10:11:25.056 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0249999999999999e-06 2024-09-21T10:11:25.056 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.056 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.056 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:25.056 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.523824+0000", 2024-09-21T10:11:25.056 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7591999999999999e-05 2024-09-21T10:11:25.057 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.057 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.057 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:25.057 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.523832+0000", 2024-09-21T10:11:25.057 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.259e-06 2024-09-21T10:11:25.057 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.057 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.057 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.057 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.523879+0000", 2024-09-21T10:11:25.057 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6149999999999997e-05 2024-09-21T10:11:25.057 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.057 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.057 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.057 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.523888+0000", 2024-09-21T10:11:25.058 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.696e-06 2024-09-21T10:11:25.058 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.058 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.058 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.058 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.405090+0000", 2024-09-21T10:11:25.058 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.88120210799999998 2024-09-21T10:11:25.058 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.058 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.058 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:25.058 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.405162+0000", 2024-09-21T10:11:25.058 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.1599000000000003e-05 2024-09-21T10:11:25.058 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.058 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.058 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:11:25.059 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.405242+0000", 2024-09-21T10:11:25.059 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.0344000000000002e-05 2024-09-21T10:11:25.059 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.059 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.059 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:25.059 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.410527+0000", 2024-09-21T10:11:25.059 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0052849510000000004 2024-09-21T10:11:25.059 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.059 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.059 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.059 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411519+0000", 2024-09-21T10:11:25.059 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00099159300000000003 2024-09-21T10:11:25.059 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.059 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.059 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.060 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411661+0000", 2024-09-21T10:11:25.060 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014172300000000001 2024-09-21T10:11:25.060 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.060 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.060 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:11:25.060 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411686+0000", 2024-09-21T10:11:25.060 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.5471000000000001e-05 2024-09-21T10:11:25.060 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.060 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.060 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:11:25.060 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411733+0000", 2024-09-21T10:11:25.060 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7036000000000002e-05 2024-09-21T10:11:25.060 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.060 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.061 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.061 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.061 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.061 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247221 10.1 10:8404822e:::benchmark_data_smithi107_184135_object115655:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:11:25.061 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:49.557178+0000", 2024-09-21T10:11:25.061 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 155.44948989299999, 2024-09-21T10:11:25.061 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.85504601800000002, 2024-09-21T10:11:25.061 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:25.061 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:25.061 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:25.061 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:11:25.061 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:11:25.061 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247221 2024-09-21T10:11:25.061 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.062 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:25.062 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.062 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:25.062 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.557178+0000", 2024-09-21T10:11:25.062 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.062 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.062 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.062 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:25.062 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.557178+0000", 2024-09-21T10:11:25.062 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.062 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.062 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.062 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:25.062 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.557177+0000", 2024-09-21T10:11:25.063 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:11:25.063 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.063 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.063 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:25.063 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.557183+0000", 2024-09-21T10:11:25.063 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4799999999999998e-06 2024-09-21T10:11:25.063 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.063 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.063 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:25.063 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.557194+0000", 2024-09-21T10:11:25.063 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1338000000000001e-05 2024-09-21T10:11:25.063 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.063 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.063 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:25.063 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.557200+0000", 2024-09-21T10:11:25.064 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9699999999999996e-06 2024-09-21T10:11:25.064 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.064 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.064 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.064 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.557239+0000", 2024-09-21T10:11:25.064 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8871000000000001e-05 2024-09-21T10:11:25.064 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.064 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.064 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.064 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.557245+0000", 2024-09-21T10:11:25.064 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.908e-06 2024-09-21T10:11:25.064 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.064 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.064 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.065 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.406907+0000", 2024-09-21T10:11:25.065 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.84966192699999998 2024-09-21T10:11:25.065 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.065 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.065 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:25.065 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.406986+0000", 2024-09-21T10:11:25.065 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9424000000000004e-05 2024-09-21T10:11:25.065 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.065 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.065 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:11:25.065 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.407109+0000", 2024-09-21T10:11:25.065 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012250799999999999 2024-09-21T10:11:25.065 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.065 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.065 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:25.066 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.410564+0000", 2024-09-21T10:11:25.066 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0034551999999999999 2024-09-21T10:11:25.066 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.066 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.066 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.066 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411769+0000", 2024-09-21T10:11:25.066 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0012049719999999999 2024-09-21T10:11:25.066 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.066 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.066 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.066 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.412083+0000", 2024-09-21T10:11:25.066 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00031341399999999998 2024-09-21T10:11:25.066 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.066 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.067 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:11:25.067 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.412130+0000", 2024-09-21T10:11:25.067 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7654999999999997e-05 2024-09-21T10:11:25.067 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.067 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.067 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:11:25.067 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.412224+0000", 2024-09-21T10:11:25.067 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.3914999999999998e-05 2024-09-21T10:11:25.067 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.067 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.067 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.067 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.067 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.067 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247224 10.1 10:8b2831c2:::benchmark_data_smithi107_184135_object115658:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:11:25.068 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:49.565141+0000", 2024-09-21T10:11:25.068 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 155.44152653699999, 2024-09-21T10:11:25.068 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.84724627799999996, 2024-09-21T10:11:25.068 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:25.068 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:25.068 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:25.068 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:11:25.068 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:11:25.068 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247224 2024-09-21T10:11:25.068 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.068 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:25.068 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.068 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:25.068 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.565141+0000", 2024-09-21T10:11:25.069 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.069 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.069 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.069 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:25.069 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.565141+0000", 2024-09-21T10:11:25.069 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.069 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.069 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.069 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:25.069 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.565140+0000", 2024-09-21T10:11:25.069 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:11:25.069 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.069 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.069 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:25.070 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.565146+0000", 2024-09-21T10:11:25.070 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3810000000000004e-06 2024-09-21T10:11:25.070 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.070 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.070 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:25.070 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.565157+0000", 2024-09-21T10:11:25.070 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0248000000000001e-05 2024-09-21T10:11:25.070 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.070 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.070 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:25.070 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.565160+0000", 2024-09-21T10:11:25.070 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7120000000000002e-06 2024-09-21T10:11:25.070 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.071 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.071 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.071 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.565194+0000", 2024-09-21T10:11:25.071 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4010999999999997e-05 2024-09-21T10:11:25.071 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.071 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.071 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.071 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.565200+0000", 2024-09-21T10:11:25.071 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6319999999999996e-06 2024-09-21T10:11:25.071 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.071 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.071 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.071 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.408744+0000", 2024-09-21T10:11:25.071 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.84354435000000005 2024-09-21T10:11:25.071 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.072 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.072 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:25.072 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.408810+0000", 2024-09-21T10:11:25.072 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5333000000000006e-05 2024-09-21T10:11:25.072 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.072 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.072 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:11:25.072 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.408877+0000", 2024-09-21T10:11:25.072 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7806999999999996e-05 2024-09-21T10:11:25.072 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.072 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.072 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:25.072 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411187+0000", 2024-09-21T10:11:25.073 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002309256 2024-09-21T10:11:25.073 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.073 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.073 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.073 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411812+0000", 2024-09-21T10:11:25.073 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00062509600000000001 2024-09-21T10:11:25.073 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.073 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.073 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.073 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.412265+0000", 2024-09-21T10:11:25.073 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000453516 2024-09-21T10:11:25.073 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.073 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.074 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:11:25.074 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.412295+0000", 2024-09-21T10:11:25.074 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.9938999999999999e-05 2024-09-21T10:11:25.074 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.074 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.074 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:11:25.074 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.412388+0000", 2024-09-21T10:11:25.074 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.2521000000000004e-05 2024-09-21T10:11:25.074 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.074 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.074 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.074 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.074 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.075 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247247 10.c 10:3f0c3473:::benchmark_data_smithi107_184135_object115681:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:11:25.075 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:50.425960+0000", 2024-09-21T10:11:25.075 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 154.58070836799999, 2024-09-21T10:11:25.075 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0036370640000001, 2024-09-21T10:11:25.075 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:25.075 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:25.075 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:25.075 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:11:25.075 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:11:25.075 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247247 2024-09-21T10:11:25.075 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.075 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:25.075 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.075 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:25.076 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.425960+0000", 2024-09-21T10:11:25.076 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.076 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.076 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.076 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:25.076 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.425960+0000", 2024-09-21T10:11:25.076 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.076 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.076 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.076 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:25.076 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.425958+0000", 2024-09-21T10:11:25.076 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:11:25.076 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.076 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.076 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:25.077 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.425965+0000", 2024-09-21T10:11:25.077 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9349999999999997e-06 2024-09-21T10:11:25.077 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.077 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.077 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:25.077 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.425979+0000", 2024-09-21T10:11:25.077 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4382000000000001e-05 2024-09-21T10:11:25.077 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.077 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.077 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:25.077 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.425986+0000", 2024-09-21T10:11:25.077 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2589999999999997e-06 2024-09-21T10:11:25.077 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.078 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.078 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.078 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.426038+0000", 2024-09-21T10:11:25.078 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2216000000000003e-05 2024-09-21T10:11:25.078 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.078 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.078 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.078 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.426048+0000", 2024-09-21T10:11:25.078 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0292000000000001e-05 2024-09-21T10:11:25.078 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.078 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.078 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.078 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.416064+0000", 2024-09-21T10:11:25.078 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.99001581500000002 2024-09-21T10:11:25.079 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.079 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.079 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:25.079 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.416149+0000", 2024-09-21T10:11:25.079 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.4993000000000004e-05 2024-09-21T10:11:25.079 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.079 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.079 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:11:25.079 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.416565+0000", 2024-09-21T10:11:25.079 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00041579900000000001 2024-09-21T10:11:25.079 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.079 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.079 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:25.079 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.428755+0000", 2024-09-21T10:11:25.080 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.012190256 2024-09-21T10:11:25.080 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.080 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.080 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.080 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.429344+0000", 2024-09-21T10:11:25.080 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000589399 2024-09-21T10:11:25.080 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.080 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.080 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.080 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.429447+0000", 2024-09-21T10:11:25.080 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010212700000000001 2024-09-21T10:11:25.080 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.080 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.081 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:11:25.081 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.429493+0000", 2024-09-21T10:11:25.081 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6418000000000002e-05 2024-09-21T10:11:25.081 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.081 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.081 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:11:25.081 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.429597+0000", 2024-09-21T10:11:25.081 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000103751 2024-09-21T10:11:25.081 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.081 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.081 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.081 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.081 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.081 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247250 10.c 10:36c8fb82:::benchmark_data_smithi107_184135_object115684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:11:25.082 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:50.427099+0000", 2024-09-21T10:11:25.082 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 154.579569499, 2024-09-21T10:11:25.082 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.003121208, 2024-09-21T10:11:25.082 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:25.082 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:25.082 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:25.082 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:11:25.082 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:11:25.082 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247250 2024-09-21T10:11:25.082 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.082 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:25.082 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.082 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:25.082 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.427099+0000", 2024-09-21T10:11:25.083 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.083 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.083 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.083 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:25.083 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.427099+0000", 2024-09-21T10:11:25.083 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.083 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.083 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.083 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:25.083 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.427097+0000", 2024-09-21T10:11:25.083 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:11:25.083 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.083 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.083 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:25.083 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.427103+0000", 2024-09-21T10:11:25.084 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7629999999999997e-06 2024-09-21T10:11:25.084 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.084 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.084 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:25.084 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.427117+0000", 2024-09-21T10:11:25.084 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3387000000000001e-05 2024-09-21T10:11:25.084 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.084 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.084 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:25.084 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.427122+0000", 2024-09-21T10:11:25.084 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7679999999999997e-06 2024-09-21T10:11:25.084 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.084 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.084 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.085 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.427170+0000", 2024-09-21T10:11:25.085 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7683999999999999e-05 2024-09-21T10:11:25.085 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.085 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.085 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.085 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.427178+0000", 2024-09-21T10:11:25.085 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.1089999999999998e-06 2024-09-21T10:11:25.085 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.085 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.085 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.085 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.420448+0000", 2024-09-21T10:11:25.085 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.99326971600000002 2024-09-21T10:11:25.085 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.085 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.086 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:25.086 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.420567+0000", 2024-09-21T10:11:25.086 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011879000000000001 2024-09-21T10:11:25.086 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.086 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.086 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:11:25.086 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.420690+0000", 2024-09-21T10:11:25.086 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012281600000000001 2024-09-21T10:11:25.086 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.086 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.086 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:25.086 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.428762+0000", 2024-09-21T10:11:25.086 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0080721899999999999 2024-09-21T10:11:25.086 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.086 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.087 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.087 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.429402+0000", 2024-09-21T10:11:25.087 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00063987799999999997 2024-09-21T10:11:25.087 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.087 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.087 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.087 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.430063+0000", 2024-09-21T10:11:25.087 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00066114500000000003 2024-09-21T10:11:25.087 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.087 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.087 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:11:25.087 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.430126+0000", 2024-09-21T10:11:25.087 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2983000000000003e-05 2024-09-21T10:11:25.087 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.088 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.088 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:11:25.088 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.430220+0000", 2024-09-21T10:11:25.088 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.4023000000000004e-05 2024-09-21T10:11:25.088 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.088 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.088 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.088 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.088 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.088 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247271 10.c 10:33c1c3c0:::benchmark_data_smithi107_184135_object115705:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:11:25.088 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:50.447124+0000", 2024-09-21T10:11:25.088 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 154.55954389999999, 2024-09-21T10:11:25.088 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.98329654799999999, 2024-09-21T10:11:25.088 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:25.089 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:25.089 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:25.089 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:11:25.089 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:11:25.089 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247271 2024-09-21T10:11:25.089 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.089 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:25.089 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.089 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:25.089 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447124+0000", 2024-09-21T10:11:25.089 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.089 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.089 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.089 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:25.089 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447124+0000", 2024-09-21T10:11:25.090 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.090 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.090 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.090 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:25.090 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447123+0000", 2024-09-21T10:11:25.090 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:11:25.090 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.090 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.090 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:25.090 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447130+0000", 2024-09-21T10:11:25.090 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9640000000000003e-06 2024-09-21T10:11:25.090 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.090 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.090 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:25.091 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447145+0000", 2024-09-21T10:11:25.091 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4280999999999999e-05 2024-09-21T10:11:25.091 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.091 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.091 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:25.091 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447152+0000", 2024-09-21T10:11:25.091 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.1169999999999998e-06 2024-09-21T10:11:25.091 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.091 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.091 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.091 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447213+0000", 2024-09-21T10:11:25.091 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1270000000000001e-05 2024-09-21T10:11:25.091 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.091 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.092 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.092 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447224+0000", 2024-09-21T10:11:25.092 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0813e-05 2024-09-21T10:11:25.092 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.092 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.092 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.092 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.423482+0000", 2024-09-21T10:11:25.092 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.97625846900000002 2024-09-21T10:11:25.092 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.092 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.092 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:25.092 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.423596+0000", 2024-09-21T10:11:25.092 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000114015 2024-09-21T10:11:25.092 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.092 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.093 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:11:25.093 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.423713+0000", 2024-09-21T10:11:25.093 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000116908 2024-09-21T10:11:25.093 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.093 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.093 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:25.093 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.428767+0000", 2024-09-21T10:11:25.093 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0050531769999999998 2024-09-21T10:11:25.093 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.093 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.093 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.093 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.429646+0000", 2024-09-21T10:11:25.093 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00087916000000000003 2024-09-21T10:11:25.093 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.093 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.094 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.094 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.430271+0000", 2024-09-21T10:11:25.094 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00062482399999999995 2024-09-21T10:11:25.094 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.094 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.094 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:11:25.094 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.430315+0000", 2024-09-21T10:11:25.094 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4449000000000001e-05 2024-09-21T10:11:25.094 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.094 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.094 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:11:25.094 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.430421+0000", 2024-09-21T10:11:25.094 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010572499999999999 2024-09-21T10:11:25.094 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.095 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.095 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.095 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.095 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.095 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247278 10.c 10:300615d8:::benchmark_data_smithi107_184135_object115712:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:11:25.095 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:50.460998+0000", 2024-09-21T10:11:25.095 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 154.545670261, 2024-09-21T10:11:25.095 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.96960406300000002, 2024-09-21T10:11:25.095 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:25.095 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:25.095 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:25.095 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:11:25.095 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:11:25.095 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247278 2024-09-21T10:11:25.096 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.096 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:25.096 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.096 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:25.096 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.460998+0000", 2024-09-21T10:11:25.096 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.096 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.096 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.096 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:25.096 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.460998+0000", 2024-09-21T10:11:25.096 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.096 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.096 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.096 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:25.096 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.460997+0000", 2024-09-21T10:11:25.097 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:11:25.097 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.097 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.097 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:25.097 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.461002+0000", 2024-09-21T10:11:25.097 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8620000000000002e-06 2024-09-21T10:11:25.097 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.097 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.097 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:25.097 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.461012+0000", 2024-09-21T10:11:25.097 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0616e-05 2024-09-21T10:11:25.097 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.097 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.097 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:25.098 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.461017+0000", 2024-09-21T10:11:25.098 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4129999999999999e-06 2024-09-21T10:11:25.098 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.098 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.098 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.098 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.461081+0000", 2024-09-21T10:11:25.098 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4398000000000002e-05 2024-09-21T10:11:25.098 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.098 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.098 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.098 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.461090+0000", 2024-09-21T10:11:25.098 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6629999999999999e-06 2024-09-21T10:11:25.098 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.098 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.099 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.099 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.426105+0000", 2024-09-21T10:11:25.099 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.96501570000000003 2024-09-21T10:11:25.099 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.099 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.099 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:25.099 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.426178+0000", 2024-09-21T10:11:25.099 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2175000000000006e-05 2024-09-21T10:11:25.099 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.099 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.099 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:11:25.099 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.426247+0000", 2024-09-21T10:11:25.099 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9805e-05 2024-09-21T10:11:25.099 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.099 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.100 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:25.100 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.429282+0000", 2024-09-21T10:11:25.100 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0030342699999999999 2024-09-21T10:11:25.100 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.100 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.100 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.100 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.429675+0000", 2024-09-21T10:11:25.100 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00039293900000000003 2024-09-21T10:11:25.100 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.100 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.100 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.100 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.430460+0000", 2024-09-21T10:11:25.100 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00078493499999999995 2024-09-21T10:11:25.100 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.101 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.101 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:11:25.101 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.430514+0000", 2024-09-21T10:11:25.101 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4901999999999997e-05 2024-09-21T10:11:25.101 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.101 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.101 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:11:25.101 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.430602+0000", 2024-09-21T10:11:25.101 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.7385000000000004e-05 2024-09-21T10:11:25.101 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.101 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.101 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.101 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.101 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.101 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36561.0:50688 11.e 11:7867821a:::benchmark_data_smithi107_218812_object50687:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-21T10:11:25.102 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:10:19.583524+0000", 2024-09-21T10:11:25.102 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 65.423144468000004, 2024-09-21T10:11:25.102 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.245396748, 2024-09-21T10:11:25.102 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:25.102 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:25.102 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:25.102 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36561", 2024-09-21T10:11:25.102 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:11:25.102 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 50688 2024-09-21T10:11:25.102 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.102 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:25.102 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.102 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:25.102 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.583524+0000", 2024-09-21T10:11:25.103 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.103 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.103 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.103 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:25.103 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.583524+0000", 2024-09-21T10:11:25.103 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.103 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.103 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.103 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:25.103 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.583522+0000", 2024-09-21T10:11:25.103 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:11:25.103 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.103 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.103 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:25.103 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.583579+0000", 2024-09-21T10:11:25.104 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6925999999999997e-05 2024-09-21T10:11:25.104 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.104 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.104 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:25.104 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.583591+0000", 2024-09-21T10:11:25.104 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2527999999999999e-05 2024-09-21T10:11:25.104 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.104 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.104 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:25.104 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.583595+0000", 2024-09-21T10:11:25.104 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.019e-06 2024-09-21T10:11:25.104 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.104 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.104 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.105 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.583637+0000", 2024-09-21T10:11:25.105 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1841000000000001e-05 2024-09-21T10:11:25.105 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.105 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.105 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:25.105 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.583720+0000", 2024-09-21T10:11:25.105 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.2999999999999998e-05 2024-09-21T10:11:25.105 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.105 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.105 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:11:25.105 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.583806+0000", 2024-09-21T10:11:25.105 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6076999999999996e-05 2024-09-21T10:11:25.105 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.105 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.105 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:25.106 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.585516+0000", 2024-09-21T10:11:25.106 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0017102949999999999 2024-09-21T10:11:25.106 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.106 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.106 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.106 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.587893+0000", 2024-09-21T10:11:25.106 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0023768470000000001 2024-09-21T10:11:25.106 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.106 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.106 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.106 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.828840+0000", 2024-09-21T10:11:25.106 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.24094675800000001 2024-09-21T10:11:25.106 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.106 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.107 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:11:25.107 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.828892+0000", 2024-09-21T10:11:25.107 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1731999999999999e-05 2024-09-21T10:11:25.107 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.107 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.107 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:11:25.107 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.828920+0000", 2024-09-21T10:11:25.108 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.8649999999999998e-05 2024-09-21T10:11:25.108 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.108 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.108 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.108 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.108 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.108 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36561.0:50706 11.f 11:f8bf77a7:::benchmark_data_smithi107_218812_object50705:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-21T10:11:25.108 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:10:19.592848+0000", 2024-09-21T10:11:25.109 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 65.413820412000007, 2024-09-21T10:11:25.109 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.24061939600000001, 2024-09-21T10:11:25.109 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:25.109 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:25.109 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:25.109 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36561", 2024-09-21T10:11:25.109 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:11:25.109 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 50706 2024-09-21T10:11:25.109 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.109 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:25.109 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.109 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:25.109 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.592848+0000", 2024-09-21T10:11:25.109 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.110 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.110 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.110 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:25.110 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.592848+0000", 2024-09-21T10:11:25.110 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.110 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.110 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.110 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:25.110 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.592846+0000", 2024-09-21T10:11:25.110 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:11:25.110 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.110 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.110 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:25.110 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.592887+0000", 2024-09-21T10:11:25.111 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0278000000000003e-05 2024-09-21T10:11:25.111 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.111 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.111 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:25.111 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.592901+0000", 2024-09-21T10:11:25.111 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4299e-05 2024-09-21T10:11:25.111 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.111 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.111 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:25.111 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.592907+0000", 2024-09-21T10:11:25.111 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2910000000000004e-06 2024-09-21T10:11:25.111 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.111 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.111 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.112 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.592959+0000", 2024-09-21T10:11:25.112 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1397000000000003e-05 2024-09-21T10:11:25.112 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.112 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.112 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:25.112 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.593065+0000", 2024-09-21T10:11:25.112 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010652100000000001 2024-09-21T10:11:25.112 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.112 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.112 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:11:25.112 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.593190+0000", 2024-09-21T10:11:25.112 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012517 2024-09-21T10:11:25.112 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.112 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.113 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:25.113 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.594917+0000", 2024-09-21T10:11:25.113 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001727042 2024-09-21T10:11:25.113 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.113 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.113 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.113 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.595965+0000", 2024-09-21T10:11:25.113 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0010480909999999999 2024-09-21T10:11:25.113 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.113 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.113 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.113 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.833370+0000", 2024-09-21T10:11:25.113 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23740465799999999 2024-09-21T10:11:25.113 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.113 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.114 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:11:25.114 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.833426+0000", 2024-09-21T10:11:25.114 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6198e-05 2024-09-21T10:11:25.114 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.114 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.114 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:11:25.114 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.833467+0000", 2024-09-21T10:11:25.114 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0781000000000001e-05 2024-09-21T10:11:25.114 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.114 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.114 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.114 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.114 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.115 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36561.0:91073 11.5 11:aba05e3e:::benchmark_data_smithi107_218812_object91072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-21T10:11:25.115 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:10:56.374738+0000", 2024-09-21T10:11:25.115 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 28.631929647, 2024-09-21T10:11:25.115 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.371354148, 2024-09-21T10:11:25.115 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:25.115 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:11:25.115 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:25.115 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36561", 2024-09-21T10:11:25.115 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:11:25.115 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 91073 2024-09-21T10:11:25.115 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.115 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:25.115 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.115 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:25.116 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.374738+0000", 2024-09-21T10:11:25.116 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.116 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.116 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.116 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:25.116 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.374738+0000", 2024-09-21T10:11:25.116 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.116 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.116 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.116 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:25.116 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.374737+0000", 2024-09-21T10:11:25.116 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:11:25.116 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.116 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.116 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:25.117 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.374836+0000", 2024-09-21T10:11:25.117 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.9283000000000005e-05 2024-09-21T10:11:25.117 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.117 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.117 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:25.117 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.374848+0000", 2024-09-21T10:11:25.117 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1473e-05 2024-09-21T10:11:25.117 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.117 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.117 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:25.117 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.374853+0000", 2024-09-21T10:11:25.117 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2719999999999997e-06 2024-09-21T10:11:25.117 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.117 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.118 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.118 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.374934+0000", 2024-09-21T10:11:25.118 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.0448e-05 2024-09-21T10:11:25.118 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.118 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.118 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:25.118 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.375050+0000", 2024-09-21T10:11:25.118 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000116227 2024-09-21T10:11:25.118 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.118 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.118 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:11:25.118 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.375205+0000", 2024-09-21T10:11:25.118 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015526200000000001 2024-09-21T10:11:25.118 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.119 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.119 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:25.119 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.378707+0000", 2024-09-21T10:11:25.119 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003502158 2024-09-21T10:11:25.119 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.119 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.119 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.119 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.378907+0000", 2024-09-21T10:11:25.119 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000199326 2024-09-21T10:11:25.119 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.119 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.119 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.119 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.384513+0000", 2024-09-21T10:11:25.119 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0056069120000000004 2024-09-21T10:11:25.120 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.120 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.120 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.120 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.384522+0000", 2024-09-21T10:11:25.120 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.5169999999999999e-06 2024-09-21T10:11:25.120 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.120 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.120 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.120 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:57.387352+0000", 2024-09-21T10:11:25.120 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0028299380000001 2024-09-21T10:11:25.120 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.120 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.120 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.120 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:57.387363+0000", 2024-09-21T10:11:25.120 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1212999999999999e-05 2024-09-21T10:11:25.121 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.121 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.121 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.121 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.746017+0000", 2024-09-21T10:11:25.121 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3586540110000001 2024-09-21T10:11:25.121 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.121 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.121 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:11:25.121 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.746093+0000", 2024-09-21T10:11:25.121 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.5432000000000006e-05 2024-09-21T10:11:25.121 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.121 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.121 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.121 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.122 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.122 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36561.0:91079 11.5 11:a8488685:::benchmark_data_smithi107_218812_object91078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-21T10:11:25.122 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:10:56.383599+0000", 2024-09-21T10:11:25.122 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 28.623068951, 2024-09-21T10:11:25.122 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.4063422349999999, 2024-09-21T10:11:25.122 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:25.122 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:25.122 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:25.122 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36561", 2024-09-21T10:11:25.122 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:11:25.122 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 91079 2024-09-21T10:11:25.122 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.122 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:25.123 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.123 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:25.123 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.383599+0000", 2024-09-21T10:11:25.123 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.123 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.123 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.123 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:25.123 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.383599+0000", 2024-09-21T10:11:25.123 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.123 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.123 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.123 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:25.123 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.383597+0000", 2024-09-21T10:11:25.123 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:11:25.123 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.124 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.124 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:25.124 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.383690+0000", 2024-09-21T10:11:25.124 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.2906999999999997e-05 2024-09-21T10:11:25.124 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.124 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.124 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:25.124 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.383705+0000", 2024-09-21T10:11:25.124 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4817999999999999e-05 2024-09-21T10:11:25.124 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.124 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.124 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:25.124 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.383713+0000", 2024-09-21T10:11:25.124 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.6599999999999995e-06 2024-09-21T10:11:25.125 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.125 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.125 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.125 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.384546+0000", 2024-09-21T10:11:25.125 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00083367200000000004 2024-09-21T10:11:25.125 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.125 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.125 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.125 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.384553+0000", 2024-09-21T10:11:25.125 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3099999999999997e-06 2024-09-21T10:11:25.125 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.125 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.125 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.125 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:57.387384+0000", 2024-09-21T10:11:25.126 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.00283181 2024-09-21T10:11:25.126 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.126 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.126 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.126 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:57.387391+0000", 2024-09-21T10:11:25.126 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5719999999999999e-06 2024-09-21T10:11:25.126 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.126 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.126 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.126 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.746128+0000", 2024-09-21T10:11:25.126 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3587367399999999 2024-09-21T10:11:25.126 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.126 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.126 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:25.127 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.746197+0000", 2024-09-21T10:11:25.127 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9661999999999998e-05 2024-09-21T10:11:25.127 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.127 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.127 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:11:25.127 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.746285+0000", 2024-09-21T10:11:25.127 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.7943999999999997e-05 2024-09-21T10:11:25.127 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.127 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.127 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:25.127 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.767420+0000", 2024-09-21T10:11:25.127 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.021134673999999999 2024-09-21T10:11:25.127 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.127 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.128 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.128 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.774710+0000", 2024-09-21T10:11:25.128 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0072896499999999999 2024-09-21T10:11:25.128 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.128 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.128 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.128 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.784581+0000", 2024-09-21T10:11:25.128 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0098717130000000007 2024-09-21T10:11:25.128 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.128 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.128 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:11:25.128 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.784639+0000", 2024-09-21T10:11:25.128 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8146999999999997e-05 2024-09-21T10:11:25.129 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.129 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.129 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:11:25.129 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.789941+0000", 2024-09-21T10:11:25.129 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0053018570000000001 2024-09-21T10:11:25.129 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.129 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.129 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.129 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.129 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.129 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36561.0:91098 11.5 11:ab6631c2:::benchmark_data_smithi107_218812_object91097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:11:25.129 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:10:56.419733+0000", 2024-09-21T10:11:25.129 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 28.586934553999999, 2024-09-21T10:11:25.129 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.3754622799999998, 2024-09-21T10:11:25.129 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:25.130 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:25.130 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:25.130 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36561", 2024-09-21T10:11:25.130 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:11:25.130 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 91098 2024-09-21T10:11:25.130 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.130 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:25.130 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.130 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:25.130 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.419733+0000", 2024-09-21T10:11:25.130 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.130 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.130 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.130 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:25.131 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.419733+0000", 2024-09-21T10:11:25.131 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.131 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.131 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.131 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:25.131 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.419733+0000", 2024-09-21T10:11:25.131 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:11:25.131 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.131 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.131 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:25.131 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.419765+0000", 2024-09-21T10:11:25.131 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1600000000000002e-05 2024-09-21T10:11:25.131 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.132 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.132 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:25.132 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.419772+0000", 2024-09-21T10:11:25.132 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3080000000000002e-06 2024-09-21T10:11:25.132 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.132 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.132 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:25.132 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.419776+0000", 2024-09-21T10:11:25.132 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.946e-06 2024-09-21T10:11:25.132 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.132 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.132 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.132 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.419829+0000", 2024-09-21T10:11:25.132 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3248999999999998e-05 2024-09-21T10:11:25.133 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.133 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.133 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.133 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.419835+0000", 2024-09-21T10:11:25.133 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4049999999999997e-06 2024-09-21T10:11:25.133 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.133 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.133 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.133 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:57.387420+0000", 2024-09-21T10:11:25.133 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.96758447299999994 2024-09-21T10:11:25.133 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.133 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.133 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.133 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:57.387426+0000", 2024-09-21T10:11:25.133 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5880000000000003e-06 2024-09-21T10:11:25.134 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.134 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.134 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.134 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.748878+0000", 2024-09-21T10:11:25.134 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.361451953 2024-09-21T10:11:25.134 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.134 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.134 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:25.134 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.749001+0000", 2024-09-21T10:11:25.134 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000122318 2024-09-21T10:11:25.134 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.134 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.134 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:11:25.134 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.749153+0000", 2024-09-21T10:11:25.135 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015237000000000001 2024-09-21T10:11:25.135 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.135 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.135 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:25.135 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.767428+0000", 2024-09-21T10:11:25.135 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.018274420999999999 2024-09-21T10:11:25.135 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.135 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.135 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.135 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.774770+0000", 2024-09-21T10:11:25.135 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0073419870000000003 2024-09-21T10:11:25.135 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.135 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.135 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.136 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.789992+0000", 2024-09-21T10:11:25.136 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.015222037000000001 2024-09-21T10:11:25.136 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.136 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.136 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:11:25.136 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.790046+0000", 2024-09-21T10:11:25.136 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4761e-05 2024-09-21T10:11:25.136 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.136 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.136 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:11:25.136 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.795196+0000", 2024-09-21T10:11:25.136 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0051494139999999997 2024-09-21T10:11:25.136 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.137 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.137 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.137 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.137 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.137 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36561.0:91120 11.5 11:a9d90972:::benchmark_data_smithi107_218812_object91119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:11:25.137 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:10:56.447451+0000", 2024-09-21T10:11:25.137 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 28.559216903999999, 2024-09-21T10:11:25.137 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.3534111200000001, 2024-09-21T10:11:25.137 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:25.137 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:25.137 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:25.137 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36561", 2024-09-21T10:11:25.137 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:11:25.137 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 91120 2024-09-21T10:11:25.138 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.138 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:25.138 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.138 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:25.138 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.447451+0000", 2024-09-21T10:11:25.138 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.138 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.138 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.138 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:25.138 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.447451+0000", 2024-09-21T10:11:25.138 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.138 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.138 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.138 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:25.139 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.447450+0000", 2024-09-21T10:11:25.139 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:11:25.139 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.139 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.139 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:25.139 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.447477+0000", 2024-09-21T10:11:25.139 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.6923e-05 2024-09-21T10:11:25.139 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.139 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.139 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:25.139 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.447489+0000", 2024-09-21T10:11:25.139 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1941e-05 2024-09-21T10:11:25.139 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.139 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.139 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.447493+0000", 2024-09-21T10:11:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4499999999999997e-06 2024-09-21T10:11:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.447538+0000", 2024-09-21T10:11:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5040000000000002e-05 2024-09-21T10:11:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.447545+0000", 2024-09-21T10:11:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.951e-06 2024-09-21T10:11:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.140 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.141 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:57.387446+0000", 2024-09-21T10:11:25.141 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.93990089499999996 2024-09-21T10:11:25.141 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.141 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.141 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.141 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:57.387452+0000", 2024-09-21T10:11:25.141 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3110000000000002e-06 2024-09-21T10:11:25.141 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.141 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.141 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.141 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.752150+0000", 2024-09-21T10:11:25.141 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3646975020000001 2024-09-21T10:11:25.141 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.141 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.142 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:25.142 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.752281+0000", 2024-09-21T10:11:25.142 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013172699999999999 2024-09-21T10:11:25.142 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.142 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.142 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:11:25.142 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.752463+0000", 2024-09-21T10:11:25.142 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00018151500000000001 2024-09-21T10:11:25.142 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.142 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.142 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:25.142 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.767434+0000", 2024-09-21T10:11:25.142 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.014971029 2024-09-21T10:11:25.142 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.143 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.143 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.143 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.774832+0000", 2024-09-21T10:11:25.143 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0073980349999999999 2024-09-21T10:11:25.143 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.143 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.143 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.143 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.795273+0000", 2024-09-21T10:11:25.143 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.020440732999999999 2024-09-21T10:11:25.143 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.143 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.143 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:11:25.143 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.795327+0000", 2024-09-21T10:11:25.143 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4500000000000003e-05 2024-09-21T10:11:25.143 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.144 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.144 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:11:25.144 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.800862+0000", 2024-09-21T10:11:25.144 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0055349400000000003 2024-09-21T10:11:25.144 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.144 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.144 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.144 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.144 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.144 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36561.0:91129 11.5 11:afb7745b:::benchmark_data_smithi107_218812_object91128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:11:25.144 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:10:56.470467+0000", 2024-09-21T10:11:25.144 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 28.536200625999999, 2024-09-21T10:11:25.144 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.3404116099999999, 2024-09-21T10:11:25.144 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:11:25.145 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:11:25.145 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:11:25.145 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36561", 2024-09-21T10:11:25.145 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:11:25.145 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 91129 2024-09-21T10:11:25.145 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.145 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:11:25.145 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.145 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:11:25.145 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.470467+0000", 2024-09-21T10:11:25.145 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.145 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.145 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.145 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:11:25.145 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.470467+0000", 2024-09-21T10:11:25.146 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:11:25.146 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.146 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.146 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:11:25.146 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.470466+0000", 2024-09-21T10:11:25.146 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:11:25.146 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.146 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.146 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:11:25.146 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.470518+0000", 2024-09-21T10:11:25.146 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1638999999999998e-05 2024-09-21T10:11:25.146 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.146 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.146 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:11:25.147 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.470530+0000", 2024-09-21T10:11:25.147 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1764e-05 2024-09-21T10:11:25.147 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.147 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.147 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:11:25.147 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.470534+0000", 2024-09-21T10:11:25.147 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4240000000000003e-06 2024-09-21T10:11:25.147 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.147 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.147 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.147 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.470575+0000", 2024-09-21T10:11:25.147 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1199000000000003e-05 2024-09-21T10:11:25.147 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.147 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.147 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.148 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.470583+0000", 2024-09-21T10:11:25.148 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.0099999999999995e-06 2024-09-21T10:11:25.148 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.148 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.148 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.148 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:57.387473+0000", 2024-09-21T10:11:25.148 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.91688940900000004 2024-09-21T10:11:25.148 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.148 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.148 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:11:25.148 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:57.387479+0000", 2024-09-21T10:11:25.148 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4509999999999997e-06 2024-09-21T10:11:25.148 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.148 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.149 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:11:25.149 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.755649+0000", 2024-09-21T10:11:25.149 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.36817038 2024-09-21T10:11:25.149 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.149 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.149 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:11:25.149 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.755801+0000", 2024-09-21T10:11:25.149 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015123599999999999 2024-09-21T10:11:25.149 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.149 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.149 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:11:25.149 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.756034+0000", 2024-09-21T10:11:25.149 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00023330700000000001 2024-09-21T10:11:25.149 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.149 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.150 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:11:25.150 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.774649+0000", 2024-09-21T10:11:25.150 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.018614616 2024-09-21T10:11:25.150 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.150 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.150 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.150 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.774875+0000", 2024-09-21T10:11:25.150 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000226624 2024-09-21T10:11:25.150 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.150 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.150 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:11:25.150 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.800924+0000", 2024-09-21T10:11:25.150 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.026048737999999998 2024-09-21T10:11:25.150 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.151 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.151 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:11:25.151 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.800975+0000", 2024-09-21T10:11:25.151 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0770999999999999e-05 2024-09-21T10:11:25.151 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:11:25.151 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:11:25.151 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:11:25.151 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.810879+0000", 2024-09-21T10:11:25.151 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0099042560000000002 2024-09-21T10:11:25.151 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.151 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.151 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.151 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:11:25.151 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:11:25.151 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:11:25.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:25.186 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:25.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:25.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:25.288 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:25.320 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:25.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:25.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:25.422 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:25.453 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:25.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:25.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:25.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:25.589 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:25.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:25.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:25.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:25.727 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:25.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:25.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:25.829 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:25.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:25.858 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:25.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:25.959 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:25.984 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:25.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:25.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:26.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:11:26.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:26.141 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:26.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:26.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:26.243 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:26.271 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:26.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:26.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:26.373 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:26.399 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:26.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:26.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:26.490 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":948,"stamp":"2024-09-21T10:11:21.266142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7062263847,"num_objects":107778,"num_object_clones":0,"num_object_copies":323334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":25005,"num_objects_unfound":0,"num_objects_dirty":107778,"num_whiteouts":0,"num_read":49,"num_read_kb":125,"num_write":107840,"num_write_kb":6896367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":602,"num_bytes_recovered":36254387,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101510,"ondisk_log_size":101510,"up":327,"acting":326,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":331,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":25301380,"kb_used_data":20764172,"kb_used_omap":41,"kb_used_meta":4537110,"kb_avail":630914684,"statfs":{"total":671965249536,"available":646056636416,"internally_reserved":0,"allocated":21262512128,"data_stored":21251877959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42519,"internal_metadata":4646001129},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,6,3,7,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":347471872,"num_objects":5302,"num_object_clones":0,"num_object_copies":15906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":655,"num_objects_unfound":0,"num_objects_dirty":5302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5302,"num_write_kb":339328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":8650752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002897"},"pg_stats":[{"pgid":"1.11","version":"77'100","reported_seq":59,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.337926+0000","last_change":"2024-09-21T10:10:57.397786+0000","last_active":"2024-09-21T10:11:14.337926+0000","last_peered":"2024-09-21T10:11:14.337926+0000","last_clean":"2024-09-21T10:11:14.337926+0000","last_became_active":"2024-09-21T10:10:57.396742+0000","last_became_peered":"2024-09-21T10:10:57.396742+0000","last_unstale":"2024-09-21T10:11:14.337926+0000","last_undegraded":"2024-09-21T10:11:14.337926+0000","last_fullsized":"2024-09-21T10:11:14.337926+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:53.631923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:53.631923+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:33:35.364595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":397,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.332269+0000","last_change":"2024-09-21T10:03:38.389577+0000","last_active":"2024-09-21T10:11:14.332269+0000","last_peered":"2024-09-21T10:11:14.332269+0000","last_clean":"2024-09-21T10:11:14.332269+0000","last_became_active":"2024-09-21T10:03:38.389292+0000","last_became_peered":"2024-09-21T10:03:38.389292+0000","last_unstale":"2024-09-21T10:11:14.332269+0000","last_undegraded":"2024-09-21T10:11:14.332269+0000","last_fullsized":"2024-09-21T10:11:14.332269+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:32.113472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":142,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.488558+0000","last_change":"2024-09-21T10:08:38.387871+0000","last_active":"2024-09-21T10:11:08.488558+0000","last_peered":"2024-09-21T10:11:08.488558+0000","last_clean":"2024-09-21T10:11:08.488558+0000","last_became_active":"2024-09-21T10:08:38.376727+0000","last_became_peered":"2024-09-21T10:08:38.376727+0000","last_unstale":"2024-09-21T10:11:08.488558+0000","last_undegraded":"2024-09-21T10:11:08.488558+0000","last_fullsized":"2024-09-21T10:11:08.488558+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:45:54.498950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'100","reported_seq":171,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.488190+0000","last_change":"2024-09-21T10:11:05.438031+0000","last_active":"2024-09-21T10:11:08.488190+0000","last_peered":"2024-09-21T10:11:08.488190+0000","last_clean":"2024-09-21T10:11:08.488190+0000","last_became_active":"2024-09-21T10:11:05.437637+0000","last_became_peered":"2024-09-21T10:11:05.437637+0000","last_unstale":"2024-09-21T10:11:08.488190+0000","last_undegraded":"2024-09-21T10:11:08.488190+0000","last_fullsized":"2024-09-21T10:11:08.488190+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.798577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.798577+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:42.343217+0000","scrub_duration":0.0023622640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":263,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.494107+0000","last_change":"2024-09-21T10:10:57.392782+0000","last_active":"2024-09-21T10:11:08.494107+0000","last_peered":"2024-09-21T10:11:08.494107+0000","last_clean":"2024-09-21T10:11:08.494107+0000","last_became_active":"2024-09-21T10:10:57.392305+0000","last_became_peered":"2024-09-21T10:10:57.392305+0000","last_unstale":"2024-09-21T10:11:08.494107+0000","last_undegraded":"2024-09-21T10:11:08.494107+0000","last_fullsized":"2024-09-21T10:11:08.494107+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:42:01.387622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":318,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.338828+0000","last_change":"2024-09-21T10:09:23.846703+0000","last_active":"2024-09-21T10:11:14.338828+0000","last_peered":"2024-09-21T10:11:14.338828+0000","last_clean":"2024-09-21T10:11:14.338828+0000","last_became_active":"2024-09-21T10:09:23.846415+0000","last_became_peered":"2024-09-21T10:09:23.846415+0000","last_unstale":"2024-09-21T10:11:14.338828+0000","last_undegraded":"2024-09-21T10:11:14.338828+0000","last_fullsized":"2024-09-21T10:11:14.338828+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:23.129285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'100","reported_seq":113,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.341480+0000","last_change":"2024-09-21T10:09:24.841480+0000","last_active":"2024-09-21T10:11:14.341480+0000","last_peered":"2024-09-21T10:11:14.341480+0000","last_clean":"2024-09-21T10:11:14.341480+0000","last_became_active":"2024-09-21T10:09:24.841145+0000","last_became_peered":"2024-09-21T10:09:24.841145+0000","last_unstale":"2024-09-21T10:11:14.341480+0000","last_undegraded":"2024-09-21T10:11:14.341480+0000","last_fullsized":"2024-09-21T10:11:14.341480+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.433227+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:11:30.758205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":23,"num_write":5,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":35780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":44,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.338464+0000","last_change":"2024-09-21T10:09:49.922489+0000","last_active":"2024-09-21T10:11:14.338464+0000","last_peered":"2024-09-21T10:11:14.338464+0000","last_clean":"2024-09-21T10:11:14.338464+0000","last_became_active":"2024-09-21T10:09:49.921629+0000","last_became_peered":"2024-09-21T10:09:49.921629+0000","last_unstale":"2024-09-21T10:11:14.338464+0000","last_undegraded":"2024-09-21T10:11:14.338464+0000","last_fullsized":"2024-09-21T10:11:14.338464+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:50.611979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":144,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.488154+0000","last_change":"2024-09-21T10:09:24.836647+0000","last_active":"2024-09-21T10:11:08.488154+0000","last_peered":"2024-09-21T10:11:08.488154+0000","last_clean":"2024-09-21T10:11:08.488154+0000","last_became_active":"2024-09-21T10:09:24.836278+0000","last_became_peered":"2024-09-21T10:09:24.836278+0000","last_unstale":"2024-09-21T10:11:08.488154+0000","last_undegraded":"2024-09-21T10:11:08.488154+0000","last_fullsized":"2024-09-21T10:11:08.488154+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:08:36.535892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"77'100","reported_seq":15,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.504437+0000","last_change":"2024-09-21T10:10:57.452531+0000","last_active":"2024-09-21T10:11:08.504437+0000","last_peered":"2024-09-21T10:11:08.504437+0000","last_clean":"2024-09-21T10:11:08.504437+0000","last_became_active":"2024-09-21T10:10:57.452150+0000","last_became_peered":"2024-09-21T10:10:57.452150+0000","last_unstale":"2024-09-21T10:11:08.504437+0000","last_undegraded":"2024-09-21T10:11:08.504437+0000","last_fullsized":"2024-09-21T10:11:08.504437+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:00.058077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:00.058077+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":397,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.333469+0000","last_change":"2024-09-21T10:03:38.381552+0000","last_active":"2024-09-21T10:11:14.333469+0000","last_peered":"2024-09-21T10:11:14.333469+0000","last_clean":"2024-09-21T10:11:14.333469+0000","last_became_active":"2024-09-21T10:03:38.380898+0000","last_became_peered":"2024-09-21T10:03:38.380898+0000","last_unstale":"2024-09-21T10:11:14.333469+0000","last_undegraded":"2024-09-21T10:11:14.333469+0000","last_fullsized":"2024-09-21T10:11:14.333469+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:53.409606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":331,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.492992+0000","last_change":"2024-09-21T10:10:57.459057+0000","last_active":"2024-09-21T10:11:08.492992+0000","last_peered":"2024-09-21T10:11:08.492992+0000","last_clean":"2024-09-21T10:11:08.492992+0000","last_became_active":"2024-09-21T10:10:57.451776+0000","last_became_peered":"2024-09-21T10:10:57.451776+0000","last_unstale":"2024-09-21T10:11:08.492992+0000","last_undegraded":"2024-09-21T10:11:08.492992+0000","last_fullsized":"2024-09-21T10:11:08.492992+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:00.138317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"272'6667","reported_seq":6763,"reported_epoch":272,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:11:16.866351+0000","last_change":"2024-09-21T10:10:58.805957+0000","last_active":"2024-09-21T10:11:16.866351+0000","last_peered":"2024-09-21T10:11:16.866351+0000","last_clean":"2024-09-21T10:10:56.364182+0000","last_became_active":"2024-09-21T10:10:58.682930+0000","last_became_peered":"2024-09-21T10:10:58.682930+0000","last_unstale":"2024-09-21T10:11:16.866351+0000","last_undegraded":"2024-09-21T10:11:16.866351+0000","last_fullsized":"2024-09-21T10:11:16.866351+0000","mapping_epoch":267,"log_start":"257'400","ondisk_log_start":"257'400","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6267,"log_dups_size":0,"ondisk_log_size":6267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436928512,"num_objects":6667,"num_object_clones":0,"num_object_copies":20001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6646,"num_objects_unfound":0,"num_objects_dirty":6667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6667,"num_write_kb":426688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[{"shards":"0,1,7","objects":6667}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"77'100","reported_seq":156,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.494040+0000","last_change":"2024-09-21T10:08:38.374573+0000","last_active":"2024-09-21T10:11:08.494040+0000","last_peered":"2024-09-21T10:11:08.494040+0000","last_clean":"2024-09-21T10:11:08.494040+0000","last_became_active":"2024-09-21T10:08:38.374075+0000","last_became_peered":"2024-09-21T10:08:38.374075+0000","last_unstale":"2024-09-21T10:11:08.494040+0000","last_undegraded":"2024-09-21T10:11:08.494040+0000","last_fullsized":"2024-09-21T10:11:08.494040+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:55.259385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:55.259385+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:28:26.784162+0000","scrub_duration":0.0015667260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":187,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.488367+0000","last_change":"2024-09-21T10:09:49.941392+0000","last_active":"2024-09-21T10:11:08.488367+0000","last_peered":"2024-09-21T10:11:08.488367+0000","last_clean":"2024-09-21T10:11:08.488367+0000","last_became_active":"2024-09-21T10:09:49.923535+0000","last_became_peered":"2024-09-21T10:09:49.923535+0000","last_unstale":"2024-09-21T10:11:08.488367+0000","last_undegraded":"2024-09-21T10:11:08.488367+0000","last_fullsized":"2024-09-21T10:11:08.488367+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:16.292163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.503722+0000","last_change":"2024-09-21T10:11:05.435502+0000","last_active":"2024-09-21T10:11:08.503722+0000","last_peered":"2024-09-21T10:11:08.503722+0000","last_clean":"2024-09-21T10:11:08.503722+0000","last_became_active":"2024-09-21T10:11:05.435216+0000","last_became_peered":"2024-09-21T10:11:05.435216+0000","last_unstale":"2024-09-21T10:11:08.503722+0000","last_undegraded":"2024-09-21T10:11:08.503722+0000","last_fullsized":"2024-09-21T10:11:08.503722+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:25.434424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"272'6603","reported_seq":6648,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:16.840285+0000","last_change":"2024-09-21T10:09:49.924187+0000","last_active":"2024-09-21T10:11:16.840285+0000","last_peered":"2024-09-21T10:11:16.840285+0000","last_clean":"2024-09-21T10:11:16.840285+0000","last_became_active":"2024-09-21T10:09:49.923789+0000","last_became_peered":"2024-09-21T10:09:49.923789+0000","last_unstale":"2024-09-21T10:11:16.840285+0000","last_undegraded":"2024-09-21T10:11:16.840285+0000","last_fullsized":"2024-09-21T10:11:16.840285+0000","mapping_epoch":255,"log_start":"257'300","ondisk_log_start":"257'300","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6303,"log_dups_size":0,"ondisk_log_size":6303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:52:49.559425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432734208,"num_objects":6603,"num_object_clones":0,"num_object_copies":19809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6603,"num_write_kb":422592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"77'100","reported_seq":203,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.332135+0000","last_change":"2024-09-21T10:09:23.839211+0000","last_active":"2024-09-21T10:11:14.332135+0000","last_peered":"2024-09-21T10:11:14.332135+0000","last_clean":"2024-09-21T10:11:14.332135+0000","last_became_active":"2024-09-21T10:09:23.838779+0000","last_became_peered":"2024-09-21T10:09:23.838779+0000","last_unstale":"2024-09-21T10:11:14.332135+0000","last_undegraded":"2024-09-21T10:11:14.332135+0000","last_fullsized":"2024-09-21T10:11:14.332135+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.981336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.981336+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:03:15.566873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":166,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.488703+0000","last_change":"2024-09-21T10:09:49.932631+0000","last_active":"2024-09-21T10:11:08.488703+0000","last_peered":"2024-09-21T10:11:08.488703+0000","last_clean":"2024-09-21T10:11:08.488703+0000","last_became_active":"2024-09-21T10:09:49.922897+0000","last_became_peered":"2024-09-21T10:09:49.922897+0000","last_unstale":"2024-09-21T10:11:08.488703+0000","last_undegraded":"2024-09-21T10:11:08.488703+0000","last_fullsized":"2024-09-21T10:11:08.488703+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:40.795476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":192,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.332657+0000","last_change":"2024-09-21T10:08:38.377732+0000","last_active":"2024-09-21T10:11:14.332657+0000","last_peered":"2024-09-21T10:11:14.332657+0000","last_clean":"2024-09-21T10:11:14.332657+0000","last_became_active":"2024-09-21T10:08:38.376601+0000","last_became_peered":"2024-09-21T10:08:38.376601+0000","last_unstale":"2024-09-21T10:11:14.332657+0000","last_undegraded":"2024-09-21T10:11:14.332657+0000","last_fullsized":"2024-09-21T10:11:14.332657+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:02.540158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.5","version":"272'6423","reported_seq":7507,"reported_epoch":272,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T10:11:16.729036+0000","last_change":"2024-09-21T10:10:58.840575+0000","last_active":"2024-09-21T10:11:16.729036+0000","last_peered":"2024-09-21T10:11:16.729036+0000","last_clean":"2024-09-21T10:10:56.374526+0000","last_became_active":"2024-09-21T10:10:58.733275+0000","last_became_peered":"2024-09-21T10:10:58.733275+0000","last_unstale":"2024-09-21T10:11:16.729036+0000","last_undegraded":"2024-09-21T10:11:16.729036+0000","last_fullsized":"2024-09-21T10:10:58.388641+0000","mapping_epoch":267,"log_start":"257'100","ondisk_log_start":"257'100","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6323,"log_dups_size":0,"ondisk_log_size":6323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:21:07.802295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420937728,"num_objects":6423,"num_object_clones":0,"num_object_copies":19269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4954,"num_objects_unfound":0,"num_objects_dirty":6423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6423,"num_write_kb":411072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":524,"num_bytes_recovered":34340864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,6],"avail_no_missing":["7","3","6"],"object_location_counts":[{"shards":"3,6,7","objects":4954},{"shards":"6,7","objects":1469}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"77'100","reported_seq":256,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.493283+0000","last_change":"2024-09-21T10:08:38.382100+0000","last_active":"2024-09-21T10:11:08.493283+0000","last_peered":"2024-09-21T10:11:08.493283+0000","last_clean":"2024-09-21T10:11:08.493283+0000","last_became_active":"2024-09-21T10:08:38.381736+0000","last_became_peered":"2024-09-21T10:08:38.381736+0000","last_unstale":"2024-09-21T10:11:08.493283+0000","last_undegraded":"2024-09-21T10:11:08.493283+0000","last_fullsized":"2024-09-21T10:11:08.493283+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.289859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.289859+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:23.739980+0000","scrub_duration":0.001251809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":152,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.493887+0000","last_change":"2024-09-21T10:10:57.460732+0000","last_active":"2024-09-21T10:11:08.493887+0000","last_peered":"2024-09-21T10:11:08.493887+0000","last_clean":"2024-09-21T10:11:08.493887+0000","last_became_active":"2024-09-21T10:10:57.454689+0000","last_became_peered":"2024-09-21T10:10:57.454689+0000","last_unstale":"2024-09-21T10:11:08.493887+0000","last_undegraded":"2024-09-21T10:11:08.493887+0000","last_fullsized":"2024-09-21T10:11:08.493887+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:12.035022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":272,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.494053+0000","last_change":"2024-09-21T10:03:38.378384+0000","last_active":"2024-09-21T10:11:08.494053+0000","last_peered":"2024-09-21T10:11:08.494053+0000","last_clean":"2024-09-21T10:11:08.494053+0000","last_became_active":"2024-09-21T10:03:38.378072+0000","last_became_peered":"2024-09-21T10:03:38.378072+0000","last_unstale":"2024-09-21T10:11:08.494053+0000","last_undegraded":"2024-09-21T10:11:08.494053+0000","last_fullsized":"2024-09-21T10:11:08.494053+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:54:37.039622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"272'6782","reported_seq":6810,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:16.684590+0000","last_change":"2024-09-21T10:09:42.896949+0000","last_active":"2024-09-21T10:11:16.684590+0000","last_peered":"2024-09-21T10:11:16.684590+0000","last_clean":"2024-09-21T10:11:16.684590+0000","last_became_active":"2024-09-21T10:09:42.896669+0000","last_became_peered":"2024-09-21T10:09:42.896669+0000","last_unstale":"2024-09-21T10:11:16.684590+0000","last_undegraded":"2024-09-21T10:11:16.684590+0000","last_fullsized":"2024-09-21T10:11:16.684590+0000","mapping_epoch":252,"log_start":"257'500","ondisk_log_start":"257'500","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6282,"log_dups_size":0,"ondisk_log_size":6282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:44.279472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444465152,"num_objects":6782,"num_object_clones":0,"num_object_copies":20346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6782,"num_write_kb":434048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"77'100","reported_seq":124,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.488403+0000","last_change":"2024-09-21T10:07:26.188298+0000","last_active":"2024-09-21T10:11:08.488403+0000","last_peered":"2024-09-21T10:11:08.488403+0000","last_clean":"2024-09-21T10:11:08.488403+0000","last_became_active":"2024-09-21T10:07:24.345265+0000","last_became_peered":"2024-09-21T10:07:24.345265+0000","last_unstale":"2024-09-21T10:11:08.488403+0000","last_undegraded":"2024-09-21T10:11:08.488403+0000","last_fullsized":"2024-09-21T10:11:08.488403+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:07:26.188220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:07:26.188220+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:46.381710+0000","scrub_duration":0.021884888000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":201,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.332400+0000","last_change":"2024-09-21T10:07:57.551556+0000","last_active":"2024-09-21T10:11:14.332400+0000","last_peered":"2024-09-21T10:11:14.332400+0000","last_clean":"2024-09-21T10:11:14.332400+0000","last_became_active":"2024-09-21T10:07:57.551249+0000","last_became_peered":"2024-09-21T10:07:57.551249+0000","last_unstale":"2024-09-21T10:11:14.332400+0000","last_undegraded":"2024-09-21T10:11:14.332400+0000","last_fullsized":"2024-09-21T10:11:14.332400+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:57.557152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":298,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.339597+0000","last_change":"2024-09-21T10:09:24.839665+0000","last_active":"2024-09-21T10:11:14.339597+0000","last_peered":"2024-09-21T10:11:14.339597+0000","last_clean":"2024-09-21T10:11:14.339597+0000","last_became_active":"2024-09-21T10:09:24.839353+0000","last_became_peered":"2024-09-21T10:09:24.839353+0000","last_unstale":"2024-09-21T10:11:14.339597+0000","last_undegraded":"2024-09-21T10:11:14.339597+0000","last_fullsized":"2024-09-21T10:11:14.339597+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:51.760073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3","version":"272'6678","reported_seq":6705,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:17.752593+0000","last_change":"2024-09-21T10:09:43.231199+0000","last_active":"2024-09-21T10:11:17.752593+0000","last_peered":"2024-09-21T10:11:17.752593+0000","last_clean":"2024-09-21T10:11:17.752593+0000","last_became_active":"2024-09-21T10:09:43.230922+0000","last_became_peered":"2024-09-21T10:09:43.230922+0000","last_unstale":"2024-09-21T10:11:17.752593+0000","last_undegraded":"2024-09-21T10:11:17.752593+0000","last_fullsized":"2024-09-21T10:11:17.752593+0000","mapping_epoch":252,"log_start":"257'500","ondisk_log_start":"257'500","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6178,"log_dups_size":0,"ondisk_log_size":6178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:39.944122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437649408,"num_objects":6678,"num_object_clones":0,"num_object_copies":20034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6678,"num_write_kb":427392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":146,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.488228+0000","last_change":"2024-09-21T10:04:53.355975+0000","last_active":"2024-09-21T10:11:08.488228+0000","last_peered":"2024-09-21T10:11:08.488228+0000","last_clean":"2024-09-21T10:11:08.488228+0000","last_became_active":"2024-09-21T10:04:42.547505+0000","last_became_peered":"2024-09-21T10:04:42.547505+0000","last_unstale":"2024-09-21T10:11:08.488228+0000","last_undegraded":"2024-09-21T10:11:08.488228+0000","last_fullsized":"2024-09-21T10:11:08.488228+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.355912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.355912+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:43.456830+0000","scrub_duration":0.0017980030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":397,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.341659+0000","last_change":"2024-09-21T10:09:49.939553+0000","last_active":"2024-09-21T10:11:14.341659+0000","last_peered":"2024-09-21T10:11:14.341659+0000","last_clean":"2024-09-21T10:11:14.341659+0000","last_became_active":"2024-09-21T10:09:49.925094+0000","last_became_peered":"2024-09-21T10:09:49.925094+0000","last_unstale":"2024-09-21T10:11:14.341659+0000","last_undegraded":"2024-09-21T10:11:14.341659+0000","last_fullsized":"2024-09-21T10:11:14.341659+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:12:12.933089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":175,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.332480+0000","last_change":"2024-09-21T10:08:38.379747+0000","last_active":"2024-09-21T10:11:14.332480+0000","last_peered":"2024-09-21T10:11:14.332480+0000","last_clean":"2024-09-21T10:11:14.332480+0000","last_became_active":"2024-09-21T10:08:38.378983+0000","last_became_peered":"2024-09-21T10:08:38.378983+0000","last_unstale":"2024-09-21T10:11:14.332480+0000","last_undegraded":"2024-09-21T10:11:14.332480+0000","last_fullsized":"2024-09-21T10:11:14.332480+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:45:08.003016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"272'6793","reported_seq":6844,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.005675+0000","last_change":"2024-09-21T10:09:42.896430+0000","last_active":"2024-09-21T10:11:19.005675+0000","last_peered":"2024-09-21T10:11:19.005675+0000","last_clean":"2024-09-21T10:11:19.005675+0000","last_became_active":"2024-09-21T10:09:42.896019+0000","last_became_peered":"2024-09-21T10:09:42.896019+0000","last_unstale":"2024-09-21T10:11:19.005675+0000","last_undegraded":"2024-09-21T10:11:19.005675+0000","last_fullsized":"2024-09-21T10:11:19.005675+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6793,"log_dups_size":0,"ondisk_log_size":6793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:59:18.633498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445186048,"num_objects":6793,"num_object_clones":0,"num_object_copies":20379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6793,"num_write_kb":434752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":157,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.329231+0000","last_change":"2024-09-21T10:10:57.403722+0000","last_active":"2024-09-21T10:11:14.329231+0000","last_peered":"2024-09-21T10:11:14.329231+0000","last_clean":"2024-09-21T10:11:14.329231+0000","last_became_active":"2024-09-21T10:10:57.394700+0000","last_became_peered":"2024-09-21T10:10:57.394700+0000","last_unstale":"2024-09-21T10:11:14.329231+0000","last_undegraded":"2024-09-21T10:11:14.329231+0000","last_fullsized":"2024-09-21T10:11:14.329231+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:02.152981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:02.152981+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:52:34.887763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":246,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.494194+0000","last_change":"2024-09-21T10:08:38.379849+0000","last_active":"2024-09-21T10:11:08.494194+0000","last_peered":"2024-09-21T10:11:08.494194+0000","last_clean":"2024-09-21T10:11:08.494194+0000","last_became_active":"2024-09-21T10:08:38.379543+0000","last_became_peered":"2024-09-21T10:08:38.379543+0000","last_unstale":"2024-09-21T10:11:08.494194+0000","last_undegraded":"2024-09-21T10:11:08.494194+0000","last_fullsized":"2024-09-21T10:11:08.494194+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:29.041802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":206,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.329990+0000","last_change":"2024-09-21T10:07:24.342950+0000","last_active":"2024-09-21T10:11:14.329990+0000","last_peered":"2024-09-21T10:11:14.329990+0000","last_clean":"2024-09-21T10:11:14.329990+0000","last_became_active":"2024-09-21T10:07:24.342477+0000","last_became_peered":"2024-09-21T10:07:24.342477+0000","last_unstale":"2024-09-21T10:11:14.329990+0000","last_undegraded":"2024-09-21T10:11:14.329990+0000","last_fullsized":"2024-09-21T10:11:14.329990+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:33:13.664050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1","version":"272'6808","reported_seq":6836,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:17.674905+0000","last_change":"2024-09-21T10:09:42.895259+0000","last_active":"2024-09-21T10:11:17.674905+0000","last_peered":"2024-09-21T10:11:17.674905+0000","last_clean":"2024-09-21T10:11:17.674905+0000","last_became_active":"2024-09-21T10:09:42.895087+0000","last_became_peered":"2024-09-21T10:09:42.895087+0000","last_unstale":"2024-09-21T10:11:17.674905+0000","last_undegraded":"2024-09-21T10:11:17.674905+0000","last_fullsized":"2024-09-21T10:11:17.674905+0000","mapping_epoch":252,"log_start":"257'600","ondisk_log_start":"257'600","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6208,"log_dups_size":0,"ondisk_log_size":6208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:40:59.270926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446169088,"num_objects":6808,"num_object_clones":0,"num_object_copies":20424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6808,"num_write_kb":435712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"77'100","reported_seq":300,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.339443+0000","last_change":"2024-09-21T10:10:57.395906+0000","last_active":"2024-09-21T10:11:14.339443+0000","last_peered":"2024-09-21T10:11:14.339443+0000","last_clean":"2024-09-21T10:11:14.339443+0000","last_became_active":"2024-09-21T10:10:57.395366+0000","last_became_peered":"2024-09-21T10:10:57.395366+0000","last_unstale":"2024-09-21T10:11:14.339443+0000","last_undegraded":"2024-09-21T10:11:14.339443+0000","last_fullsized":"2024-09-21T10:11:14.339443+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.540323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.540323+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:34.873336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":298,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.329565+0000","last_change":"2024-09-21T10:07:36.464496+0000","last_active":"2024-09-21T10:11:14.329565+0000","last_peered":"2024-09-21T10:11:14.329565+0000","last_clean":"2024-09-21T10:11:14.329565+0000","last_became_active":"2024-09-21T10:07:36.464172+0000","last_became_peered":"2024-09-21T10:07:36.464172+0000","last_unstale":"2024-09-21T10:11:14.329565+0000","last_undegraded":"2024-09-21T10:11:14.329565+0000","last_fullsized":"2024-09-21T10:11:14.329565+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:09.839709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.492920+0000","last_change":"2024-09-21T10:11:05.442332+0000","last_active":"2024-09-21T10:11:08.492920+0000","last_peered":"2024-09-21T10:11:08.492920+0000","last_clean":"2024-09-21T10:11:08.492920+0000","last_became_active":"2024-09-21T10:11:05.442003+0000","last_became_peered":"2024-09-21T10:11:05.442003+0000","last_unstale":"2024-09-21T10:11:08.492920+0000","last_undegraded":"2024-09-21T10:11:08.492920+0000","last_fullsized":"2024-09-21T10:11:08.492920+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:34.986309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"272'6710","reported_seq":6761,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:16.443576+0000","last_change":"2024-09-21T10:09:43.231401+0000","last_active":"2024-09-21T10:11:16.443576+0000","last_peered":"2024-09-21T10:11:16.443576+0000","last_clean":"2024-09-21T10:11:16.443576+0000","last_became_active":"2024-09-21T10:09:43.230977+0000","last_became_peered":"2024-09-21T10:09:43.230977+0000","last_unstale":"2024-09-21T10:11:16.443576+0000","last_undegraded":"2024-09-21T10:11:16.443576+0000","last_fullsized":"2024-09-21T10:11:16.443576+0000","mapping_epoch":252,"log_start":"257'1000","ondisk_log_start":"257'1000","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5710,"log_dups_size":0,"ondisk_log_size":5710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:15.907809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439746560,"num_objects":6710,"num_object_clones":0,"num_object_copies":20130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6710,"num_write_kb":429440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":120,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.329930+0000","last_change":"2024-09-21T10:08:04.645700+0000","last_active":"2024-09-21T10:11:14.329930+0000","last_peered":"2024-09-21T10:11:14.329930+0000","last_clean":"2024-09-21T10:11:14.329930+0000","last_became_active":"2024-09-21T10:07:57.548189+0000","last_became_peered":"2024-09-21T10:07:57.548189+0000","last_unstale":"2024-09-21T10:11:14.329930+0000","last_undegraded":"2024-09-21T10:11:14.329930+0000","last_fullsized":"2024-09-21T10:11:14.329930+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:48.318010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:48.318010+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T16:31:28.173059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":398,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.332422+0000","last_change":"2024-09-21T10:03:38.386604+0000","last_active":"2024-09-21T10:11:14.332422+0000","last_peered":"2024-09-21T10:11:14.332422+0000","last_clean":"2024-09-21T10:11:14.332422+0000","last_became_active":"2024-09-21T10:03:38.386283+0000","last_became_peered":"2024-09-21T10:03:38.386283+0000","last_unstale":"2024-09-21T10:11:14.332422+0000","last_undegraded":"2024-09-21T10:11:14.332422+0000","last_fullsized":"2024-09-21T10:11:14.332422+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:27:05.944880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":202,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.333538+0000","last_change":"2024-09-21T10:08:38.372486+0000","last_active":"2024-09-21T10:11:14.333538+0000","last_peered":"2024-09-21T10:11:14.333538+0000","last_clean":"2024-09-21T10:11:14.333538+0000","last_became_active":"2024-09-21T10:08:38.370669+0000","last_became_peered":"2024-09-21T10:08:38.370669+0000","last_unstale":"2024-09-21T10:11:14.333538+0000","last_undegraded":"2024-09-21T10:11:14.333538+0000","last_fullsized":"2024-09-21T10:11:14.333538+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:53:23.247341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.a","version":"272'6646","reported_seq":6708,"reported_epoch":272,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:11:16.412404+0000","last_change":"2024-09-21T10:10:58.704699+0000","last_active":"2024-09-21T10:11:16.412404+0000","last_peered":"2024-09-21T10:11:16.412404+0000","last_clean":"2024-09-21T10:10:56.390149+0000","last_became_active":"2024-09-21T10:10:58.682906+0000","last_became_peered":"2024-09-21T10:10:58.682906+0000","last_unstale":"2024-09-21T10:11:16.412404+0000","last_undegraded":"2024-09-21T10:11:16.412404+0000","last_fullsized":"2024-09-21T10:11:16.412404+0000","mapping_epoch":267,"log_start":"257'900","ondisk_log_start":"257'900","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5746,"log_dups_size":0,"ondisk_log_size":5746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435552256,"num_objects":6646,"num_object_clones":0,"num_object_copies":19938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6646,"num_objects_unfound":0,"num_objects_dirty":6646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6646,"num_write_kb":425344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,1],"avail_no_missing":["3","1","6"],"object_location_counts":[{"shards":"1,3,6","objects":6646}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":163,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.488106+0000","last_change":"2024-09-21T10:09:49.924286+0000","last_active":"2024-09-21T10:11:08.488106+0000","last_peered":"2024-09-21T10:11:08.488106+0000","last_clean":"2024-09-21T10:11:08.488106+0000","last_became_active":"2024-09-21T10:09:49.923897+0000","last_became_peered":"2024-09-21T10:09:49.923897+0000","last_unstale":"2024-09-21T10:11:08.488106+0000","last_undegraded":"2024-09-21T10:11:08.488106+0000","last_fullsized":"2024-09-21T10:11:08.488106+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:47.393146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:47.393146+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:44.898083+0000","scrub_duration":0.001601086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":153,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.329523+0000","last_change":"2024-09-21T10:10:57.402946+0000","last_active":"2024-09-21T10:11:14.329523+0000","last_peered":"2024-09-21T10:11:14.329523+0000","last_clean":"2024-09-21T10:11:14.329523+0000","last_became_active":"2024-09-21T10:10:57.395450+0000","last_became_peered":"2024-09-21T10:10:57.395450+0000","last_unstale":"2024-09-21T10:11:14.329523+0000","last_undegraded":"2024-09-21T10:11:14.329523+0000","last_fullsized":"2024-09-21T10:11:14.329523+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:08.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":383,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.333003+0000","last_change":"2024-09-21T10:10:57.390595+0000","last_active":"2024-09-21T10:11:14.333003+0000","last_peered":"2024-09-21T10:11:14.333003+0000","last_clean":"2024-09-21T10:11:14.333003+0000","last_became_active":"2024-09-21T10:10:57.390265+0000","last_became_peered":"2024-09-21T10:10:57.390265+0000","last_unstale":"2024-09-21T10:11:14.333003+0000","last_undegraded":"2024-09-21T10:11:14.333003+0000","last_fullsized":"2024-09-21T10:11:14.333003+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:37.485063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":279,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.493018+0000","last_change":"2024-09-21T10:09:24.837651+0000","last_active":"2024-09-21T10:11:08.493018+0000","last_peered":"2024-09-21T10:11:08.493018+0000","last_clean":"2024-09-21T10:11:08.493018+0000","last_became_active":"2024-09-21T10:09:24.837226+0000","last_became_peered":"2024-09-21T10:09:24.837226+0000","last_unstale":"2024-09-21T10:11:08.493018+0000","last_undegraded":"2024-09-21T10:11:08.493018+0000","last_fullsized":"2024-09-21T10:11:08.493018+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:16.649601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"272'6636","reported_seq":6694,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:16.430903+0000","last_change":"2024-09-21T10:09:49.926338+0000","last_active":"2024-09-21T10:11:16.430903+0000","last_peered":"2024-09-21T10:11:16.430903+0000","last_clean":"2024-09-21T10:11:16.430903+0000","last_became_active":"2024-09-21T10:09:49.926018+0000","last_became_peered":"2024-09-21T10:09:49.926018+0000","last_unstale":"2024-09-21T10:11:16.430903+0000","last_undegraded":"2024-09-21T10:11:16.430903+0000","last_fullsized":"2024-09-21T10:11:16.430903+0000","mapping_epoch":255,"log_start":"257'900","ondisk_log_start":"257'900","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5736,"log_dups_size":0,"ondisk_log_size":5736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:09:49.364660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434896896,"num_objects":6636,"num_object_clones":0,"num_object_copies":19908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6636,"num_write_kb":424704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.493839+0000","last_change":"2024-09-21T10:11:05.443167+0000","last_active":"2024-09-21T10:11:08.493839+0000","last_peered":"2024-09-21T10:11:08.493839+0000","last_clean":"2024-09-21T10:11:08.493839+0000","last_became_active":"2024-09-21T10:11:05.442805+0000","last_became_peered":"2024-09-21T10:11:05.442805+0000","last_unstale":"2024-09-21T10:11:08.493839+0000","last_undegraded":"2024-09-21T10:11:08.493839+0000","last_fullsized":"2024-09-21T10:11:08.493839+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.550551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.550551+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:58:57.998543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":137,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.488129+0000","last_change":"2024-09-21T10:09:23.838944+0000","last_active":"2024-09-21T10:11:08.488129+0000","last_peered":"2024-09-21T10:11:08.488129+0000","last_clean":"2024-09-21T10:11:08.488129+0000","last_became_active":"2024-09-21T10:09:23.837886+0000","last_became_peered":"2024-09-21T10:09:23.837886+0000","last_unstale":"2024-09-21T10:11:08.488129+0000","last_undegraded":"2024-09-21T10:11:08.488129+0000","last_fullsized":"2024-09-21T10:11:08.488129+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:16.928195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":275,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.337421+0000","last_change":"2024-09-21T10:09:24.839770+0000","last_active":"2024-09-21T10:11:14.337421+0000","last_peered":"2024-09-21T10:11:14.337421+0000","last_clean":"2024-09-21T10:11:14.337421+0000","last_became_active":"2024-09-21T10:09:24.839479+0000","last_became_peered":"2024-09-21T10:09:24.839479+0000","last_unstale":"2024-09-21T10:11:14.337421+0000","last_undegraded":"2024-09-21T10:11:14.337421+0000","last_fullsized":"2024-09-21T10:11:14.337421+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:11:19.618193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":44,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.338333+0000","last_change":"2024-09-21T10:09:49.923044+0000","last_active":"2024-09-21T10:11:14.338333+0000","last_peered":"2024-09-21T10:11:14.338333+0000","last_clean":"2024-09-21T10:11:14.338333+0000","last_became_active":"2024-09-21T10:09:49.922711+0000","last_became_peered":"2024-09-21T10:09:49.922711+0000","last_unstale":"2024-09-21T10:11:14.338333+0000","last_undegraded":"2024-09-21T10:11:14.338333+0000","last_fullsized":"2024-09-21T10:11:14.338333+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:14.090084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"272'6759","reported_seq":6825,"reported_epoch":272,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:11:16.392896+0000","last_change":"2024-09-21T10:10:58.695837+0000","last_active":"2024-09-21T10:11:16.392896+0000","last_peered":"2024-09-21T10:11:16.392896+0000","last_clean":"2024-09-21T10:10:56.379248+0000","last_became_active":"2024-09-21T10:10:58.672712+0000","last_became_peered":"2024-09-21T10:10:58.672712+0000","last_unstale":"2024-09-21T10:11:16.392896+0000","last_undegraded":"2024-09-21T10:11:16.392896+0000","last_fullsized":"2024-09-21T10:11:16.392896+0000","mapping_epoch":267,"log_start":"257'1000","ondisk_log_start":"257'1000","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5759,"log_dups_size":0,"ondisk_log_size":5759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442957824,"num_objects":6759,"num_object_clones":0,"num_object_copies":20277,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6759,"num_objects_unfound":0,"num_objects_dirty":6759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6759,"num_write_kb":432576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,0],"avail_no_missing":["3","0","6"],"object_location_counts":[{"shards":"0,3,6","objects":6759}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":177,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.488382+0000","last_change":"2024-09-21T10:09:24.838434+0000","last_active":"2024-09-21T10:11:08.488382+0000","last_peered":"2024-09-21T10:11:08.488382+0000","last_clean":"2024-09-21T10:11:08.488382+0000","last_became_active":"2024-09-21T10:09:24.838066+0000","last_became_peered":"2024-09-21T10:09:24.838066+0000","last_unstale":"2024-09-21T10:11:08.488382+0000","last_undegraded":"2024-09-21T10:11:08.488382+0000","last_fullsized":"2024-09-21T10:11:08.488382+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:52.361137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:52.361137+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:19:57.788195+0000","scrub_duration":0.001255358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":436,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.329695+0000","last_change":"2024-09-21T10:09:23.837859+0000","last_active":"2024-09-21T10:11:14.329695+0000","last_peered":"2024-09-21T10:11:14.329695+0000","last_clean":"2024-09-21T10:11:14.329695+0000","last_became_active":"2024-09-21T10:09:23.837488+0000","last_became_peered":"2024-09-21T10:09:23.837488+0000","last_unstale":"2024-09-21T10:11:14.329695+0000","last_undegraded":"2024-09-21T10:11:14.329695+0000","last_fullsized":"2024-09-21T10:11:14.329695+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:25.215501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.488048+0000","last_change":"2024-09-21T10:10:27.806070+0000","last_active":"2024-09-21T10:11:08.488048+0000","last_peered":"2024-09-21T10:11:08.488048+0000","last_clean":"2024-09-21T10:11:08.488048+0000","last_became_active":"2024-09-21T10:10:27.805718+0000","last_became_peered":"2024-09-21T10:10:27.805718+0000","last_unstale":"2024-09-21T10:11:08.488048+0000","last_undegraded":"2024-09-21T10:11:08.488048+0000","last_fullsized":"2024-09-21T10:11:08.488048+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:31.144118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":430,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.330115+0000","last_change":"2024-09-21T10:10:57.395073+0000","last_active":"2024-09-21T10:11:14.330115+0000","last_peered":"2024-09-21T10:11:14.330115+0000","last_clean":"2024-09-21T10:11:14.330115+0000","last_became_active":"2024-09-21T10:10:57.394674+0000","last_became_peered":"2024-09-21T10:10:57.394674+0000","last_unstale":"2024-09-21T10:11:14.330115+0000","last_undegraded":"2024-09-21T10:11:14.330115+0000","last_fullsized":"2024-09-21T10:11:14.330115+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:48:48.912649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"272'6766","reported_seq":6817,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:16.447101+0000","last_change":"2024-09-21T10:09:43.231554+0000","last_active":"2024-09-21T10:11:16.447101+0000","last_peered":"2024-09-21T10:11:16.447101+0000","last_clean":"2024-09-21T10:11:16.447101+0000","last_became_active":"2024-09-21T10:09:43.231211+0000","last_became_peered":"2024-09-21T10:09:43.231211+0000","last_unstale":"2024-09-21T10:11:16.447101+0000","last_undegraded":"2024-09-21T10:11:16.447101+0000","last_fullsized":"2024-09-21T10:11:16.447101+0000","mapping_epoch":252,"log_start":"257'1100","ondisk_log_start":"257'1100","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5666,"log_dups_size":0,"ondisk_log_size":5666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:16:41.457768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443416576,"num_objects":6766,"num_object_clones":0,"num_object_copies":20298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6766,"num_write_kb":433024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":299,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.337512+0000","last_change":"2024-09-21T10:10:57.398437+0000","last_active":"2024-09-21T10:11:14.337512+0000","last_peered":"2024-09-21T10:11:14.337512+0000","last_clean":"2024-09-21T10:11:14.337512+0000","last_became_active":"2024-09-21T10:10:57.397441+0000","last_became_peered":"2024-09-21T10:10:57.397441+0000","last_unstale":"2024-09-21T10:11:14.337512+0000","last_undegraded":"2024-09-21T10:11:14.337512+0000","last_fullsized":"2024-09-21T10:11:14.337512+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:04.043917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:04.043917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:53:31.813612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":76,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.494467+0000","last_change":"2024-09-21T10:08:26.309323+0000","last_active":"2024-09-21T10:11:08.494467+0000","last_peered":"2024-09-21T10:11:08.494467+0000","last_clean":"2024-09-21T10:11:08.494467+0000","last_became_active":"2024-09-21T10:08:26.309004+0000","last_became_peered":"2024-09-21T10:08:26.309004+0000","last_unstale":"2024-09-21T10:11:08.494467+0000","last_undegraded":"2024-09-21T10:11:08.494467+0000","last_fullsized":"2024-09-21T10:11:08.494467+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:37:20.626702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":109,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.339347+0000","last_change":"2024-09-21T10:10:28.180093+0000","last_active":"2024-09-21T10:11:14.339347+0000","last_peered":"2024-09-21T10:11:14.339347+0000","last_clean":"2024-09-21T10:11:14.339347+0000","last_became_active":"2024-09-21T10:10:28.179559+0000","last_became_peered":"2024-09-21T10:10:28.179559+0000","last_unstale":"2024-09-21T10:11:14.339347+0000","last_undegraded":"2024-09-21T10:11:14.339347+0000","last_fullsized":"2024-09-21T10:11:14.339347+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:52:46.146392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":206,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.330612+0000","last_change":"2024-09-21T10:08:26.307041+0000","last_active":"2024-09-21T10:11:14.330612+0000","last_peered":"2024-09-21T10:11:14.330612+0000","last_clean":"2024-09-21T10:11:14.330612+0000","last_became_active":"2024-09-21T10:08:26.306715+0000","last_became_peered":"2024-09-21T10:08:26.306715+0000","last_unstale":"2024-09-21T10:11:14.330612+0000","last_undegraded":"2024-09-21T10:11:14.330612+0000","last_fullsized":"2024-09-21T10:11:14.330612+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:19:26.192398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"272'6932","reported_seq":6983,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.063371+0000","last_change":"2024-09-21T10:09:43.232893+0000","last_active":"2024-09-21T10:11:19.063371+0000","last_peered":"2024-09-21T10:11:19.063371+0000","last_clean":"2024-09-21T10:11:19.063371+0000","last_became_active":"2024-09-21T10:09:43.232461+0000","last_became_peered":"2024-09-21T10:09:43.232461+0000","last_unstale":"2024-09-21T10:11:19.063371+0000","last_undegraded":"2024-09-21T10:11:19.063371+0000","last_fullsized":"2024-09-21T10:11:19.063371+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6932,"log_dups_size":0,"ondisk_log_size":6932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:39.778833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454295552,"num_objects":6932,"num_object_clones":0,"num_object_copies":20796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6932,"num_write_kb":443648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":246,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.493159+0000","last_change":"2024-09-21T10:04:45.242148+0000","last_active":"2024-09-21T10:11:08.493159+0000","last_peered":"2024-09-21T10:11:08.493159+0000","last_clean":"2024-09-21T10:11:08.493159+0000","last_became_active":"2024-09-21T10:04:42.549844+0000","last_became_peered":"2024-09-21T10:04:42.549844+0000","last_unstale":"2024-09-21T10:11:08.493159+0000","last_undegraded":"2024-09-21T10:11:08.493159+0000","last_fullsized":"2024-09-21T10:11:08.493159+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_deep_scrub":"77'100","last_deep_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.242073+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:03:42.250480+0000","scrub_duration":0.0023466239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.504360+0000","last_change":"2024-09-21T10:10:57.392880+0000","last_active":"2024-09-21T10:11:08.504360+0000","last_peered":"2024-09-21T10:11:08.504360+0000","last_clean":"2024-09-21T10:11:08.504360+0000","last_became_active":"2024-09-21T10:10:57.392539+0000","last_became_peered":"2024-09-21T10:10:57.392539+0000","last_unstale":"2024-09-21T10:11:08.504360+0000","last_undegraded":"2024-09-21T10:11:08.504360+0000","last_fullsized":"2024-09-21T10:11:08.504360+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:49.754955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":319,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.339691+0000","last_change":"2024-09-21T10:09:23.865342+0000","last_active":"2024-09-21T10:11:14.339691+0000","last_peered":"2024-09-21T10:11:14.339691+0000","last_clean":"2024-09-21T10:11:14.339691+0000","last_became_active":"2024-09-21T10:09:23.865008+0000","last_became_peered":"2024-09-21T10:09:23.865008+0000","last_unstale":"2024-09-21T10:11:14.339691+0000","last_undegraded":"2024-09-21T10:11:14.339691+0000","last_fullsized":"2024-09-21T10:11:14.339691+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:03.356101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.504350+0000","last_change":"2024-09-21T10:10:57.394707+0000","last_active":"2024-09-21T10:11:08.504350+0000","last_peered":"2024-09-21T10:11:08.504350+0000","last_clean":"2024-09-21T10:11:08.504350+0000","last_became_active":"2024-09-21T10:10:57.394363+0000","last_became_peered":"2024-09-21T10:10:57.394363+0000","last_unstale":"2024-09-21T10:11:08.504350+0000","last_undegraded":"2024-09-21T10:11:08.504350+0000","last_fullsized":"2024-09-21T10:11:08.504350+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:21.141052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"272'6887","reported_seq":6937,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:19.081855+0000","last_change":"2024-09-21T10:09:43.232987+0000","last_active":"2024-09-21T10:11:19.081855+0000","last_peered":"2024-09-21T10:11:19.081855+0000","last_clean":"2024-09-21T10:11:19.081855+0000","last_became_active":"2024-09-21T10:09:43.232577+0000","last_became_peered":"2024-09-21T10:09:43.232577+0000","last_unstale":"2024-09-21T10:11:19.081855+0000","last_undegraded":"2024-09-21T10:11:19.081855+0000","last_fullsized":"2024-09-21T10:11:19.081855+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6887,"log_dups_size":0,"ondisk_log_size":6887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:14.536774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451346432,"num_objects":6887,"num_object_clones":0,"num_object_copies":20661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6887,"num_write_kb":440768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":263,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.492995+0000","last_change":"2024-09-21T10:09:23.839563+0000","last_active":"2024-09-21T10:11:08.492995+0000","last_peered":"2024-09-21T10:11:08.492995+0000","last_clean":"2024-09-21T10:11:08.492995+0000","last_became_active":"2024-09-21T10:09:23.839207+0000","last_became_peered":"2024-09-21T10:09:23.839207+0000","last_unstale":"2024-09-21T10:11:08.492995+0000","last_undegraded":"2024-09-21T10:11:08.492995+0000","last_fullsized":"2024-09-21T10:11:08.492995+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:49.226496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:49.226496+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:03.960125+0000","scrub_duration":0.001656865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":171,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.330438+0000","last_change":"2024-09-21T10:10:57.395167+0000","last_active":"2024-09-21T10:11:14.330438+0000","last_peered":"2024-09-21T10:11:14.330438+0000","last_clean":"2024-09-21T10:11:14.330438+0000","last_became_active":"2024-09-21T10:10:57.394818+0000","last_became_peered":"2024-09-21T10:10:57.394818+0000","last_unstale":"2024-09-21T10:11:14.330438+0000","last_undegraded":"2024-09-21T10:11:14.330438+0000","last_fullsized":"2024-09-21T10:11:14.330438+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:57.224113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":157,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.493965+0000","last_change":"2024-09-21T10:09:49.922782+0000","last_active":"2024-09-21T10:11:08.493965+0000","last_peered":"2024-09-21T10:11:08.493965+0000","last_clean":"2024-09-21T10:11:08.493965+0000","last_became_active":"2024-09-21T10:09:49.922465+0000","last_became_peered":"2024-09-21T10:09:49.922465+0000","last_unstale":"2024-09-21T10:11:08.493965+0000","last_undegraded":"2024-09-21T10:11:08.493965+0000","last_fullsized":"2024-09-21T10:11:08.493965+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:06.151245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":313,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.339531+0000","last_change":"2024-09-21T10:10:57.397251+0000","last_active":"2024-09-21T10:11:14.339531+0000","last_peered":"2024-09-21T10:11:14.339531+0000","last_clean":"2024-09-21T10:11:14.339531+0000","last_became_active":"2024-09-21T10:10:57.396271+0000","last_became_peered":"2024-09-21T10:10:57.396271+0000","last_unstale":"2024-09-21T10:11:14.339531+0000","last_undegraded":"2024-09-21T10:11:14.339531+0000","last_fullsized":"2024-09-21T10:11:14.339531+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:17.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.f","version":"272'6854","reported_seq":6882,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:16.723618+0000","last_change":"2024-09-21T10:09:43.232625+0000","last_active":"2024-09-21T10:11:16.723618+0000","last_peered":"2024-09-21T10:11:16.723618+0000","last_clean":"2024-09-21T10:11:16.723618+0000","last_became_active":"2024-09-21T10:09:43.232331+0000","last_became_peered":"2024-09-21T10:09:43.232331+0000","last_unstale":"2024-09-21T10:11:16.723618+0000","last_undegraded":"2024-09-21T10:11:16.723618+0000","last_fullsized":"2024-09-21T10:11:16.723618+0000","mapping_epoch":252,"log_start":"257'600","ondisk_log_start":"257'600","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6254,"log_dups_size":0,"ondisk_log_size":6254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:01:16.634826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449183744,"num_objects":6854,"num_object_clones":0,"num_object_copies":20562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6854,"num_write_kb":438656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":117,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.329691+0000","last_change":"2024-09-21T10:11:05.455170+0000","last_active":"2024-09-21T10:11:14.329691+0000","last_peered":"2024-09-21T10:11:14.329691+0000","last_clean":"2024-09-21T10:11:14.329691+0000","last_became_active":"2024-09-21T10:11:05.454607+0000","last_became_peered":"2024-09-21T10:11:05.454607+0000","last_unstale":"2024-09-21T10:11:14.329691+0000","last_undegraded":"2024-09-21T10:11:14.329691+0000","last_fullsized":"2024-09-21T10:11:14.329691+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:59.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:59.099852+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:14.788975+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":404,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.333609+0000","last_change":"2024-09-21T10:09:23.841186+0000","last_active":"2024-09-21T10:11:14.333609+0000","last_peered":"2024-09-21T10:11:14.333609+0000","last_clean":"2024-09-21T10:11:14.333609+0000","last_became_active":"2024-09-21T10:09:23.840854+0000","last_became_peered":"2024-09-21T10:09:23.840854+0000","last_unstale":"2024-09-21T10:11:14.333609+0000","last_undegraded":"2024-09-21T10:11:14.333609+0000","last_fullsized":"2024-09-21T10:11:14.333609+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:29.666358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":94,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.493065+0000","last_change":"2024-09-21T10:08:38.380867+0000","last_active":"2024-09-21T10:11:08.493065+0000","last_peered":"2024-09-21T10:11:08.493065+0000","last_clean":"2024-09-21T10:11:08.493065+0000","last_became_active":"2024-09-21T10:08:38.379599+0000","last_became_peered":"2024-09-21T10:08:38.379599+0000","last_unstale":"2024-09-21T10:11:08.493065+0000","last_undegraded":"2024-09-21T10:11:08.493065+0000","last_fullsized":"2024-09-21T10:11:08.493065+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:50.499470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":401,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.331957+0000","last_change":"2024-09-21T10:09:49.926548+0000","last_active":"2024-09-21T10:11:14.331957+0000","last_peered":"2024-09-21T10:11:14.331957+0000","last_clean":"2024-09-21T10:11:14.331957+0000","last_became_active":"2024-09-21T10:09:49.926255+0000","last_became_peered":"2024-09-21T10:09:49.926255+0000","last_unstale":"2024-09-21T10:11:14.331957+0000","last_undegraded":"2024-09-21T10:11:14.331957+0000","last_fullsized":"2024-09-21T10:11:14.331957+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:49.855878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"272'6807","reported_seq":6835,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:16.736455+0000","last_change":"2024-09-21T10:09:43.231158+0000","last_active":"2024-09-21T10:11:16.736455+0000","last_peered":"2024-09-21T10:11:16.736455+0000","last_clean":"2024-09-21T10:11:16.736455+0000","last_became_active":"2024-09-21T10:09:43.230838+0000","last_became_peered":"2024-09-21T10:09:43.230838+0000","last_unstale":"2024-09-21T10:11:16.736455+0000","last_undegraded":"2024-09-21T10:11:16.736455+0000","last_fullsized":"2024-09-21T10:11:16.736455+0000","mapping_epoch":252,"log_start":"257'500","ondisk_log_start":"257'500","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6307,"log_dups_size":0,"ondisk_log_size":6307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:15.318633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446103552,"num_objects":6807,"num_object_clones":0,"num_object_copies":20421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6807,"num_write_kb":435648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":203,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.332059+0000","last_change":"2024-09-21T10:10:57.390804+0000","last_active":"2024-09-21T10:11:14.332059+0000","last_peered":"2024-09-21T10:11:14.332059+0000","last_clean":"2024-09-21T10:11:14.332059+0000","last_became_active":"2024-09-21T10:10:57.390323+0000","last_became_peered":"2024-09-21T10:10:57.390323+0000","last_unstale":"2024-09-21T10:11:14.332059+0000","last_undegraded":"2024-09-21T10:11:14.332059+0000","last_fullsized":"2024-09-21T10:11:14.332059+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.005477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.005477+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:29:57.702046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":48,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.338597+0000","last_change":"2024-09-21T10:09:49.930280+0000","last_active":"2024-09-21T10:11:14.338597+0000","last_peered":"2024-09-21T10:11:14.338597+0000","last_clean":"2024-09-21T10:11:14.338597+0000","last_became_active":"2024-09-21T10:09:49.919101+0000","last_became_peered":"2024-09-21T10:09:49.919101+0000","last_unstale":"2024-09-21T10:11:14.338597+0000","last_undegraded":"2024-09-21T10:11:14.338597+0000","last_fullsized":"2024-09-21T10:11:14.338597+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:23.998475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":285,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.493425+0000","last_change":"2024-09-21T10:03:38.392767+0000","last_active":"2024-09-21T10:11:08.493425+0000","last_peered":"2024-09-21T10:11:08.493425+0000","last_clean":"2024-09-21T10:11:08.493425+0000","last_became_active":"2024-09-21T10:03:38.392424+0000","last_became_peered":"2024-09-21T10:03:38.392424+0000","last_unstale":"2024-09-21T10:11:08.493425+0000","last_undegraded":"2024-09-21T10:11:08.493425+0000","last_fullsized":"2024-09-21T10:11:08.493425+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:25.358291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":66,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.503779+0000","last_change":"2024-09-21T10:09:24.836360+0000","last_active":"2024-09-21T10:11:08.503779+0000","last_peered":"2024-09-21T10:11:08.503779+0000","last_clean":"2024-09-21T10:11:08.503779+0000","last_became_active":"2024-09-21T10:09:24.836033+0000","last_became_peered":"2024-09-21T10:09:24.836033+0000","last_unstale":"2024-09-21T10:11:08.503779+0000","last_undegraded":"2024-09-21T10:11:08.503779+0000","last_fullsized":"2024-09-21T10:11:08.503779+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:58.368582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":123,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.493304+0000","last_change":"2024-09-21T10:08:38.380189+0000","last_active":"2024-09-21T10:11:08.493304+0000","last_peered":"2024-09-21T10:11:08.493304+0000","last_clean":"2024-09-21T10:11:08.493304+0000","last_became_active":"2024-09-21T10:08:38.379891+0000","last_became_peered":"2024-09-21T10:08:38.379891+0000","last_unstale":"2024-09-21T10:11:08.493304+0000","last_undegraded":"2024-09-21T10:11:08.493304+0000","last_fullsized":"2024-09-21T10:11:08.493304+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:35.820921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":203,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.489478+0000","last_change":"2024-09-21T10:08:38.392563+0000","last_active":"2024-09-21T10:11:08.489478+0000","last_peered":"2024-09-21T10:11:08.489478+0000","last_clean":"2024-09-21T10:11:08.489478+0000","last_became_active":"2024-09-21T10:08:38.378735+0000","last_became_peered":"2024-09-21T10:08:38.378735+0000","last_unstale":"2024-09-21T10:11:08.489478+0000","last_undegraded":"2024-09-21T10:11:08.489478+0000","last_fullsized":"2024-09-21T10:11:08.489478+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:29:41.423208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.494120+0000","last_change":"2024-09-21T10:11:05.453286+0000","last_active":"2024-09-21T10:11:08.494120+0000","last_peered":"2024-09-21T10:11:08.494120+0000","last_clean":"2024-09-21T10:11:08.494120+0000","last_became_active":"2024-09-21T10:11:05.452601+0000","last_became_peered":"2024-09-21T10:11:05.452601+0000","last_unstale":"2024-09-21T10:11:08.494120+0000","last_undegraded":"2024-09-21T10:11:08.494120+0000","last_fullsized":"2024-09-21T10:11:08.494120+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:27.532091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":40,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.330100+0000","last_change":"2024-09-21T10:10:27.833762+0000","last_active":"2024-09-21T10:11:14.330100+0000","last_peered":"2024-09-21T10:11:14.330100+0000","last_clean":"2024-09-21T10:11:14.330100+0000","last_became_active":"2024-09-21T10:10:27.819999+0000","last_became_peered":"2024-09-21T10:10:27.819999+0000","last_unstale":"2024-09-21T10:11:14.330100+0000","last_undegraded":"2024-09-21T10:11:14.330100+0000","last_fullsized":"2024-09-21T10:11:14.330100+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:33.839463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'100","reported_seq":195,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.330322+0000","last_change":"2024-09-21T10:07:24.345428+0000","last_active":"2024-09-21T10:11:14.330322+0000","last_peered":"2024-09-21T10:11:14.330322+0000","last_clean":"2024-09-21T10:11:14.330322+0000","last_became_active":"2024-09-21T10:07:24.343529+0000","last_became_peered":"2024-09-21T10:07:24.343529+0000","last_unstale":"2024-09-21T10:11:14.330322+0000","last_undegraded":"2024-09-21T10:11:14.330322+0000","last_fullsized":"2024-09-21T10:11:14.330322+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.014993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.014993+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:35.677888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.505005+0000","last_change":"2024-09-21T10:10:57.394885+0000","last_active":"2024-09-21T10:11:08.505005+0000","last_peered":"2024-09-21T10:11:08.505005+0000","last_clean":"2024-09-21T10:11:08.505005+0000","last_became_active":"2024-09-21T10:10:57.394584+0000","last_became_peered":"2024-09-21T10:10:57.394584+0000","last_unstale":"2024-09-21T10:11:08.505005+0000","last_undegraded":"2024-09-21T10:11:08.505005+0000","last_fullsized":"2024-09-21T10:11:08.505005+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:52:16.853713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":387,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.332406+0000","last_change":"2024-09-21T10:03:38.425111+0000","last_active":"2024-09-21T10:11:14.332406+0000","last_peered":"2024-09-21T10:11:14.332406+0000","last_clean":"2024-09-21T10:11:14.332406+0000","last_became_active":"2024-09-21T10:03:38.424704+0000","last_became_peered":"2024-09-21T10:03:38.424704+0000","last_unstale":"2024-09-21T10:11:14.332406+0000","last_undegraded":"2024-09-21T10:11:14.332406+0000","last_fullsized":"2024-09-21T10:11:14.332406+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:35.157076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":288,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.339776+0000","last_change":"2024-09-21T10:10:57.397755+0000","last_active":"2024-09-21T10:11:14.339776+0000","last_peered":"2024-09-21T10:11:14.339776+0000","last_clean":"2024-09-21T10:11:14.339776+0000","last_became_active":"2024-09-21T10:10:57.397249+0000","last_became_peered":"2024-09-21T10:10:57.397249+0000","last_unstale":"2024-09-21T10:11:14.339776+0000","last_undegraded":"2024-09-21T10:11:14.339776+0000","last_fullsized":"2024-09-21T10:11:14.339776+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:40.813503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":372,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.331868+0000","last_change":"2024-09-21T09:56:12.576756+0000","last_active":"2024-09-21T10:11:14.331868+0000","last_peered":"2024-09-21T10:11:14.331868+0000","last_clean":"2024-09-21T10:11:14.331868+0000","last_became_active":"2024-09-21T09:56:12.575752+0000","last_became_peered":"2024-09-21T09:56:12.575752+0000","last_unstale":"2024-09-21T10:11:14.331868+0000","last_undegraded":"2024-09-21T10:11:14.331868+0000","last_fullsized":"2024-09-21T10:11:14.331868+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:55.224839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":235,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.493354+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T10:11:08.493354+0000","last_peered":"2024-09-21T10:11:08.493354+0000","last_clean":"2024-09-21T10:11:08.493354+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T10:11:08.493354+0000","last_undegraded":"2024-09-21T10:11:08.493354+0000","last_fullsized":"2024-09-21T10:11:08.493354+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":408,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.332246+0000","last_change":"2024-09-21T10:09:49.928053+0000","last_active":"2024-09-21T10:11:14.332246+0000","last_peered":"2024-09-21T10:11:14.332246+0000","last_clean":"2024-09-21T10:11:14.332246+0000","last_became_active":"2024-09-21T10:09:49.927231+0000","last_became_peered":"2024-09-21T10:09:49.927231+0000","last_unstale":"2024-09-21T10:11:14.332246+0000","last_undegraded":"2024-09-21T10:11:14.332246+0000","last_fullsized":"2024-09-21T10:11:14.332246+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:26.987553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":320,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.339228+0000","last_change":"2024-09-21T10:09:23.840717+0000","last_active":"2024-09-21T10:11:14.339228+0000","last_peered":"2024-09-21T10:11:14.339228+0000","last_clean":"2024-09-21T10:11:14.339228+0000","last_became_active":"2024-09-21T10:09:23.840323+0000","last_became_peered":"2024-09-21T10:09:23.840323+0000","last_unstale":"2024-09-21T10:11:14.339228+0000","last_undegraded":"2024-09-21T10:11:14.339228+0000","last_fullsized":"2024-09-21T10:11:14.339228+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:55.769770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":34,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.329400+0000","last_change":"2024-09-21T10:10:27.816526+0000","last_active":"2024-09-21T10:11:14.329400+0000","last_peered":"2024-09-21T10:11:14.329400+0000","last_clean":"2024-09-21T10:11:14.329400+0000","last_became_active":"2024-09-21T10:10:27.816221+0000","last_became_peered":"2024-09-21T10:10:27.816221+0000","last_unstale":"2024-09-21T10:11:14.329400+0000","last_undegraded":"2024-09-21T10:11:14.329400+0000","last_fullsized":"2024-09-21T10:11:14.329400+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:08.331228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":379,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.332621+0000","last_change":"2024-09-21T10:08:38.379994+0000","last_active":"2024-09-21T10:11:14.332621+0000","last_peered":"2024-09-21T10:11:14.332621+0000","last_clean":"2024-09-21T10:11:14.332621+0000","last_became_active":"2024-09-21T10:08:38.378729+0000","last_became_peered":"2024-09-21T10:08:38.378729+0000","last_unstale":"2024-09-21T10:11:14.332621+0000","last_undegraded":"2024-09-21T10:11:14.332621+0000","last_fullsized":"2024-09-21T10:11:14.332621+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:16.940173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":99,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.330185+0000","last_change":"2024-09-21T10:09:24.844561+0000","last_active":"2024-09-21T10:11:14.330185+0000","last_peered":"2024-09-21T10:11:14.330185+0000","last_clean":"2024-09-21T10:11:14.330185+0000","last_became_active":"2024-09-21T10:09:24.844190+0000","last_became_peered":"2024-09-21T10:09:24.844190+0000","last_unstale":"2024-09-21T10:11:14.330185+0000","last_undegraded":"2024-09-21T10:11:14.330185+0000","last_fullsized":"2024-09-21T10:11:14.330185+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:46:04.775344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":425,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.330537+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:11:14.330537+0000","last_peered":"2024-09-21T10:11:14.330537+0000","last_clean":"2024-09-21T10:11:14.330537+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:11:14.330537+0000","last_undegraded":"2024-09-21T10:11:14.330537+0000","last_fullsized":"2024-09-21T10:11:14.330537+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":44,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.339109+0000","last_change":"2024-09-21T10:09:49.918246+0000","last_active":"2024-09-21T10:11:14.339109+0000","last_peered":"2024-09-21T10:11:14.339109+0000","last_clean":"2024-09-21T10:11:14.339109+0000","last_became_active":"2024-09-21T10:09:49.917911+0000","last_became_peered":"2024-09-21T10:09:49.917911+0000","last_unstale":"2024-09-21T10:11:14.339109+0000","last_undegraded":"2024-09-21T10:11:14.339109+0000","last_fullsized":"2024-09-21T10:11:14.339109+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:30:10.567650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":234,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.341581+0000","last_change":"2024-09-21T10:07:24.347937+0000","last_active":"2024-09-21T10:11:14.341581+0000","last_peered":"2024-09-21T10:11:14.341581+0000","last_clean":"2024-09-21T10:11:14.341581+0000","last_became_active":"2024-09-21T10:07:24.347376+0000","last_became_peered":"2024-09-21T10:07:24.347376+0000","last_unstale":"2024-09-21T10:11:14.341581+0000","last_undegraded":"2024-09-21T10:11:14.341581+0000","last_fullsized":"2024-09-21T10:11:14.341581+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:48:05.644824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":54,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.493635+0000","last_change":"2024-09-21T10:08:38.382518+0000","last_active":"2024-09-21T10:11:08.493635+0000","last_peered":"2024-09-21T10:11:08.493635+0000","last_clean":"2024-09-21T10:11:08.493635+0000","last_became_active":"2024-09-21T10:08:38.372051+0000","last_became_peered":"2024-09-21T10:08:38.372051+0000","last_unstale":"2024-09-21T10:11:08.493635+0000","last_undegraded":"2024-09-21T10:11:08.493635+0000","last_fullsized":"2024-09-21T10:11:08.493635+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:21.360435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":325,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.333135+0000","last_change":"2024-09-21T10:09:24.840173+0000","last_active":"2024-09-21T10:11:14.333135+0000","last_peered":"2024-09-21T10:11:14.333135+0000","last_clean":"2024-09-21T10:11:14.333135+0000","last_became_active":"2024-09-21T10:09:24.839762+0000","last_became_peered":"2024-09-21T10:09:24.839762+0000","last_unstale":"2024-09-21T10:11:14.333135+0000","last_undegraded":"2024-09-21T10:11:14.333135+0000","last_fullsized":"2024-09-21T10:11:14.333135+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:12:53.465203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":68,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.493152+0000","last_change":"2024-09-21T10:08:38.388561+0000","last_active":"2024-09-21T10:11:08.493152+0000","last_peered":"2024-09-21T10:11:08.493152+0000","last_clean":"2024-09-21T10:11:08.493152+0000","last_became_active":"2024-09-21T10:08:38.374558+0000","last_became_peered":"2024-09-21T10:08:38.374558+0000","last_unstale":"2024-09-21T10:11:08.493152+0000","last_undegraded":"2024-09-21T10:11:08.493152+0000","last_fullsized":"2024-09-21T10:11:08.493152+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:24.111438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":409,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.332906+0000","last_change":"2024-09-21T10:09:23.840467+0000","last_active":"2024-09-21T10:11:14.332906+0000","last_peered":"2024-09-21T10:11:14.332906+0000","last_clean":"2024-09-21T10:11:14.332906+0000","last_became_active":"2024-09-21T10:09:23.840177+0000","last_became_peered":"2024-09-21T10:09:23.840177+0000","last_unstale":"2024-09-21T10:11:14.332906+0000","last_undegraded":"2024-09-21T10:11:14.332906+0000","last_fullsized":"2024-09-21T10:11:14.332906+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:17.887307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":398,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.333277+0000","last_change":"2024-09-21T10:11:05.451613+0000","last_active":"2024-09-21T10:11:14.333277+0000","last_peered":"2024-09-21T10:11:14.333277+0000","last_clean":"2024-09-21T10:11:14.333277+0000","last_became_active":"2024-09-21T10:11:05.451075+0000","last_became_peered":"2024-09-21T10:11:05.451075+0000","last_unstale":"2024-09-21T10:11:14.333277+0000","last_undegraded":"2024-09-21T10:11:14.333277+0000","last_fullsized":"2024-09-21T10:11:14.333277+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:18.505045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":104,"reported_epoch":271,"state":"active+clean","last_fresh":"2024-09-21T10:11:08.492771+0000","last_change":"2024-09-21T10:08:38.380508+0000","last_active":"2024-09-21T10:11:08.492771+0000","last_peered":"2024-09-21T10:11:08.492771+0000","last_clean":"2024-09-21T10:11:08.492771+0000","last_became_active":"2024-09-21T10:08:38.380226+0000","last_became_peered":"2024-09-21T10:08:38.380226+0000","last_unstale":"2024-09-21T10:11:08.492771+0000","last_undegraded":"2024-09-21T10:11:08.492771+0000","last_fullsized":"2024-09-21T10:11:08.492771+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:50:38.190118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":63,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.338759+0000","last_change":"2024-09-21T10:10:57.401588+0000","last_active":"2024-09-21T10:11:14.338759+0000","last_peered":"2024-09-21T10:11:14.338759+0000","last_clean":"2024-09-21T10:11:14.338759+0000","last_became_active":"2024-09-21T10:10:57.397724+0000","last_became_peered":"2024-09-21T10:10:57.397724+0000","last_unstale":"2024-09-21T10:11:14.338759+0000","last_undegraded":"2024-09-21T10:11:14.338759+0000","last_fullsized":"2024-09-21T10:11:14.338759+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:47.843630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":7061569536,"num_objects":107751,"num_object_clones":0,"num_object_copies":323253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":25005,"num_objects_unfound":0,"num_objects_dirty":107751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107751,"num_write_kb":6896064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":542,"num_bytes_recovered":35520512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21344092160,"data_stored":21344092160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99351,"ondisk_log_size":99351,"up":48,"acting":47,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":11710,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26580,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":688762,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":49,"num_read_kb":125,"num_write":25,"num_write_kb":271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":722165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626827,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4343296,"kb_used_data":3438776,"kb_used_omap":1,"kb_used_meta":904510,"kb_avail":89401856,"statfs":{"total":95995035648,"available":91547500544,"internally_reserved":0,"allocated":3521306624,"data_stored":3519781058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":926218584},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019285,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":245,"seq":1052266987544,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3566964,"kb_used_data":3422892,"kb_used_omap":8,"kb_used_meta":144055,"kb_avail":90178188,"statfs":{"total":95995035648,"available":92342464512,"internally_reserved":0,"allocated":3505041408,"data_stored":3503523973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8940,"internal_metadata":147512596},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":175,"seq":751619276893,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3748288,"kb_used_data":3417848,"kb_used_omap":12,"kb_used_meta":330419,"kb_avail":89996864,"statfs":{"total":95995035648,"available":92156788736,"internally_reserved":0,"allocated":3499876352,"data_stored":3498350789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12845,"internal_metadata":338349523},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823056,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1013104,"kb_used_data":103848,"kb_used_omap":1,"kb_used_meta":909246,"kb_avail":92732048,"statfs":{"total":95995035648,"available":94957617152,"internally_reserved":0,"allocated":106340352,"data_stored":104826876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931068248},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879479,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4108272,"kb_used_data":3341160,"kb_used_omap":4,"kb_used_meta":767099,"kb_avail":89636880,"statfs":{"total":95995035648,"available":91788165120,"internally_reserved":0,"allocated":3421347840,"data_stored":3419834445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":785509820},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566320,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3281028,"kb_used_data":2623228,"kb_used_omap":6,"kb_used_meta":657785,"kb_avail":90464124,"statfs":{"total":95995035648,"available":92635262976,"internally_reserved":0,"allocated":2686185472,"data_stored":2684656953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":673572263},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860665,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5240428,"kb_used_data":4416420,"kb_used_omap":5,"kb_used_meta":823994,"kb_avail":88504724,"statfs":{"total":95995035648,"available":90628837376,"internally_reserved":0,"allocated":4522414080,"data_stored":4520903865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":843770095},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7236,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9976,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3522953216,"data_stored":3522953216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4530700288,"data_stored":4530700288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2703228928,"data_stored":2703228928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3432579072,"data_stored":3432579072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":105971712,"data_stored":105971712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3535011840,"data_stored":3535011840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3513647104,"data_stored":3513647104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:11:26.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:11:26.501 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:26.525 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:26.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:26.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:26.626 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:26.653 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:26.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:26.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:26.656 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:26 smithi107 ceph-mon[102221]: pgmap v950: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 54 MiB/s wr, 864 op/s; 25682/335454 objects misplaced (7.656%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:11:26.657 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:11:26.657 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:26.658 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:26 smithi107 ceph-mon[105022]: pgmap v950: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 54 MiB/s wr, 864 op/s; 25682/335454 objects misplaced (7.656%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:11:26.658 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:11:26.658 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:26 smithi129 ceph-mon[88471]: pgmap v950: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 54 MiB/s wr, 864 op/s; 25682/335454 objects misplaced (7.656%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:11:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:11:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:26.755 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:26.780 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:26.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:26.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:26.882 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:26.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:26.910 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:26.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:27.011 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:27.041 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:27.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:27.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:27.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:27.156 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:27.161 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:27.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:27.210 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:27.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:27.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:27.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:27.359 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:27.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:27.460 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:27.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:27.491 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:27.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:27.592 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:27.620 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:27.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:27.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:27.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:27.760 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:27.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:27.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:27.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:27.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:27.904 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:27.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:28.005 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:28.032 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:28.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:28.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:28.133 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:28.160 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:28.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:28.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:28.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:28.301 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:28.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:28.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:28.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:28.424 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:28.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:28.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:28.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:28.547 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:28.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:28.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:28.628 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:11:28.628 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:11:28.628 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:11:28.628 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:11:28.628 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:11:28.628 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:11:28.628 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:11:28.628 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:11:28.629 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:11:28.629 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:11:28.629 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:11:28.629 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:11:28.629 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:11:28.629 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:11:28.629 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:11:28.629 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:11:28.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:28.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:28 smithi129 ceph-mon[88471]: pgmap v951: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 39 MiB/s wr, 787 op/s; 25194/332490 objects misplaced (7.577%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T10:11:28.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:28 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2975635423' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:28.685 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:28.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:28.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:28.786 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:28.812 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:28.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:28.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:28.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:28 smithi107 ceph-mon[102221]: pgmap v951: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 39 MiB/s wr, 787 op/s; 25194/332490 objects misplaced (7.577%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T10:11:28.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:28 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2975635423' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:28.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:28 smithi107 ceph-mon[105022]: pgmap v951: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 39 MiB/s wr, 787 op/s; 25194/332490 objects misplaced (7.577%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T10:11:28.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:28 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2975635423' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:28.914 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:28.937 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:28.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:28.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:29.038 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:29.073 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:29.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:29.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:29.175 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:29.210 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:29.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:29.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:29.312 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:29.339 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:29.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:29.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:29.441 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:29.470 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:29.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:29.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:29.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:29.611 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:29.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:29.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:29.713 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:29.741 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:29.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:29.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:29.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:29.876 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:29.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:29.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:29.978 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:30.004 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:30.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:30.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:30.144 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:30.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:30.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:30.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:30.274 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:30.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:30.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:30.376 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:30.401 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:30.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:30.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:30.502 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:30.532 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:30.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:30.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:30.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:30 smithi107 ceph-mon[102221]: pgmap v952: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 30 MiB/s wr, 760 op/s; 25194/328929 objects misplaced (7.659%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T10:11:30.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:30 smithi107 ceph-mon[105022]: pgmap v952: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 30 MiB/s wr, 760 op/s; 25194/328929 objects misplaced (7.659%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T10:11:30.633 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:30.666 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:30.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:30.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:30 smithi129 ceph-mon[88471]: pgmap v952: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 30 MiB/s wr, 760 op/s; 25194/328929 objects misplaced (7.659%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T10:11:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:11:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:11:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:11:30.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:30.804 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:30.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:30.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:30.907 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:30.947 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:30.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:30.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:31.049 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:31.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:31.082 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:31.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:31.183 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:31.222 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:31.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:31.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:31.325 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:31.363 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:31.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:31.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:31.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:31.506 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:31.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:31.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:31.608 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:31.639 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:31.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:31.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:31.673 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:31.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:31.799 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:31.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:31.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:31.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:31.935 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:31.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:31.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:32.037 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:32.063 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:32.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:32.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:11:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:11:31] "GET /metrics HTTP/1.1" 200 36813 "" "Prometheus/2.51.0" 2024-09-21T10:11:32.165 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:32.199 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:32.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:32.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:32.301 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:32.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:32.334 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:32.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:32.427 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:11:32.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:32 smithi107 ceph-mon[102221]: pgmap v953: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.7 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 93 B/s rd, 27 MiB/s wr, 697 op/s; 25194/328929 objects misplaced (7.659%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T10:11:32.429 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:32 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2021247150' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:11:32.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:32 smithi107 ceph-mon[105022]: pgmap v953: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.7 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 93 B/s rd, 27 MiB/s wr, 697 op/s; 25194/328929 objects misplaced (7.659%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T10:11:32.430 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:32 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2021247150' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:11:32.436 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:32.477 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:32.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:32.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:32.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:32.613 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:32.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:32.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:32.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:32 smithi129 ceph-mon[88471]: pgmap v953: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.7 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 93 B/s rd, 27 MiB/s wr, 697 op/s; 25194/328929 objects misplaced (7.659%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T10:11:32.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:32 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2021247150' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:11:32.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:32.757 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:32.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:32.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:32.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:32.905 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:32.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:32.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:32.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:11:33.007 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:33.035 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:33.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:33.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:33.137 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:33.166 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:33.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:33.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:33.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:33.296 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:33.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:33.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:33.398 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:33.428 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:33.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:33.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:33.529 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:33.559 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:33.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:33.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:33.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:33 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:11:33.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:33 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2021247150' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:11:33.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:33 smithi107 ceph-mon[102221]: osdmap e274: 8 total, 8 up, 7 in 2024-09-21T10:11:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:33 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:11:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:33 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2021247150' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:11:33.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:33 smithi107 ceph-mon[105022]: osdmap e274: 8 total, 8 up, 7 in 2024-09-21T10:11:33.661 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:33.703 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:33.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:33.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:33.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:33.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:33.851 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:33.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:33 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:11:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:33 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2021247150' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:11:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:33 smithi129 ceph-mon[88471]: osdmap e274: 8 total, 8 up, 7 in 2024-09-21T10:11:33.953 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:33.978 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:33.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:33.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:34.080 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:34.106 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:34.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:34.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:34.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:34.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:34.264 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:34.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:34.267 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:11:34.267 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:11:34.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:34.413 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:34.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:34.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:34.515 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:34.544 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:34.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:34.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:34.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:34 smithi107 ceph-mon[102221]: pgmap v955: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 7.6 MiB/s wr, 759 op/s; 24203/318408 objects misplaced (7.601%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T10:11:34.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:34 smithi107 ceph-mon[105022]: pgmap v955: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 7.6 MiB/s wr, 759 op/s; 24203/318408 objects misplaced (7.601%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T10:11:34.646 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:34.675 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:34.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:34.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:34.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:34.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:34.816 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:34.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:34.917 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:34.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:34 smithi129 ceph-mon[88471]: pgmap v955: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 7.6 MiB/s wr, 759 op/s; 24203/318408 objects misplaced (7.601%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T10:11:34.945 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:34.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:34.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:35.047 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:35.070 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:35.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:35.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:35.172 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:35.196 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:35.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:35.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:35.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:35.350 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:35.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:35.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:35.451 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:35.477 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:35.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:35.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:35.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:35.616 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:35.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:35.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:35.717 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:35.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:11:35.743 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:35.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:35.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:35.845 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:35.869 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:35.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:35.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:35.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:11:35.971 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:35.997 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:35.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:35.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:36.098 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:36.133 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:36.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:36.143 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:36.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:36.244 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:36.269 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:36.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:36.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:36.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:36.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:36.409 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:36.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:36.510 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:36.544 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:11:36.549 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:11:36.550 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:36 smithi107 ceph-mon[105022]: pgmap v956: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 4.0 MiB/s wr, 786 op/s; 24203/315708 objects misplaced (7.666%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T10:11:36.552 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:36 smithi107 ceph-mon[102221]: pgmap v956: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 4.0 MiB/s wr, 786 op/s; 24203/315708 objects misplaced (7.666%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T10:11:36.559 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:36.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:36.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:36.661 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:36.690 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:36.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:36.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:36.791 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:36.824 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:36.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:36.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:36 smithi129 ceph-mon[88471]: pgmap v956: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 4.0 MiB/s wr, 786 op/s; 24203/315708 objects misplaced (7.666%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T10:11:36.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:36.959 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:36.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:36.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:37.061 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:37.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:37.093 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:37.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:37.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:37.225 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:37.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:37.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:37.296 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:37.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:37.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:37.397 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:37.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:37.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:37.541 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:37.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:37.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:37.643 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:37.674 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:37.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:37.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:37.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:37.816 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:37.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:37.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:37 smithi107 ceph-mon[102221]: from='client.46448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:11:37.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:37 smithi107 ceph-mon[105022]: from='client.46448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:11:37.919 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:37 smithi129 ceph-mon[88471]: from='client.46448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:11:37.946 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:37.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:37.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:38.047 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:38.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:38.085 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:38.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:38.186 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:38.212 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:38.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:38.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:38.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:38.347 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:38.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:38.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:38.449 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:38.477 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:38.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:38.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:38.572 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:38.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:38.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:38 smithi107 ceph-mon[102221]: pgmap v957: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 23143/306627 objects misplaced (7.548%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:11:38.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:38 smithi107 ceph-mon[105022]: pgmap v957: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 23143/306627 objects misplaced (7.548%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:11:38.645 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:38.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:38.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:38.747 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:38.783 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:38.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:38.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:38.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:38 smithi129 ceph-mon[88471]: pgmap v957: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 23143/306627 objects misplaced (7.548%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:11:38.926 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:38.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:38.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:39.028 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:39.071 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:39.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:39.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:39.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:39.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:39.209 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:39.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:39.309 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:39.350 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:39.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:39.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:39.452 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:39.481 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:39.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:39.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:39.565 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":956,"stamp":"2024-09-21T10:11:35.269827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6895605823,"num_objects":105236,"num_object_clones":0,"num_object_copies":315708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":24203,"num_objects_unfound":0,"num_objects_dirty":105236,"num_whiteouts":0,"num_read":50,"num_read_kb":126,"num_write":119850,"num_write_kb":7199280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1182,"num_bytes_recovered":74265267,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103720,"ondisk_log_size":103720,"up":327,"acting":326,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":331,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":24833460,"kb_used_data":20266872,"kb_used_omap":41,"kb_used_meta":4566294,"kb_avail":631382604,"statfs":{"total":671965249536,"available":646535786496,"internally_reserved":0,"allocated":20753276928,"data_stored":20742552203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42519,"internal_metadata":4675885545},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-431357952,"num_objects":-6582,"num_object_clones":0,"num_object_copies":-19746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1479,"num_objects_unfound":0,"num_objects_dirty":-6582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7872,"num_write_kb":41280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":27918336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002616"},"pg_stats":[{"pgid":"1.11","version":"77'100","reported_seq":61,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.385671+0000","last_change":"2024-09-21T10:10:57.397786+0000","last_active":"2024-09-21T10:11:20.385671+0000","last_peered":"2024-09-21T10:11:20.385671+0000","last_clean":"2024-09-21T10:11:20.385671+0000","last_became_active":"2024-09-21T10:10:57.396742+0000","last_became_peered":"2024-09-21T10:10:57.396742+0000","last_unstale":"2024-09-21T10:11:20.385671+0000","last_undegraded":"2024-09-21T10:11:20.385671+0000","last_fullsized":"2024-09-21T10:11:20.385671+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:53.631923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:53.631923+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:33:35.364595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":399,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.390897+0000","last_change":"2024-09-21T10:03:38.389577+0000","last_active":"2024-09-21T10:11:20.390897+0000","last_peered":"2024-09-21T10:11:20.390897+0000","last_clean":"2024-09-21T10:11:20.390897+0000","last_became_active":"2024-09-21T10:03:38.389292+0000","last_became_peered":"2024-09-21T10:03:38.389292+0000","last_unstale":"2024-09-21T10:11:20.390897+0000","last_undegraded":"2024-09-21T10:11:20.390897+0000","last_fullsized":"2024-09-21T10:11:20.390897+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:32.113472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":143,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.331286+0000","last_change":"2024-09-21T10:08:38.387871+0000","last_active":"2024-09-21T10:11:14.331286+0000","last_peered":"2024-09-21T10:11:14.331286+0000","last_clean":"2024-09-21T10:11:14.331286+0000","last_became_active":"2024-09-21T10:08:38.376727+0000","last_became_peered":"2024-09-21T10:08:38.376727+0000","last_unstale":"2024-09-21T10:11:14.331286+0000","last_undegraded":"2024-09-21T10:11:14.331286+0000","last_fullsized":"2024-09-21T10:11:14.331286+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:45:54.498950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'100","reported_seq":172,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.331064+0000","last_change":"2024-09-21T10:11:05.438031+0000","last_active":"2024-09-21T10:11:14.331064+0000","last_peered":"2024-09-21T10:11:14.331064+0000","last_clean":"2024-09-21T10:11:14.331064+0000","last_became_active":"2024-09-21T10:11:05.437637+0000","last_became_peered":"2024-09-21T10:11:05.437637+0000","last_unstale":"2024-09-21T10:11:14.331064+0000","last_undegraded":"2024-09-21T10:11:14.331064+0000","last_fullsized":"2024-09-21T10:11:14.331064+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.798577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.798577+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:42.343217+0000","scrub_duration":0.0023622640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":264,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.333720+0000","last_change":"2024-09-21T10:10:57.392782+0000","last_active":"2024-09-21T10:11:14.333720+0000","last_peered":"2024-09-21T10:11:14.333720+0000","last_clean":"2024-09-21T10:11:14.333720+0000","last_became_active":"2024-09-21T10:10:57.392305+0000","last_became_peered":"2024-09-21T10:10:57.392305+0000","last_unstale":"2024-09-21T10:11:14.333720+0000","last_undegraded":"2024-09-21T10:11:14.333720+0000","last_fullsized":"2024-09-21T10:11:14.333720+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:42:01.387622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":320,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.387028+0000","last_change":"2024-09-21T10:09:23.846703+0000","last_active":"2024-09-21T10:11:20.387028+0000","last_peered":"2024-09-21T10:11:20.387028+0000","last_clean":"2024-09-21T10:11:20.387028+0000","last_became_active":"2024-09-21T10:09:23.846415+0000","last_became_peered":"2024-09-21T10:09:23.846415+0000","last_unstale":"2024-09-21T10:11:20.387028+0000","last_undegraded":"2024-09-21T10:11:20.387028+0000","last_fullsized":"2024-09-21T10:11:20.387028+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:23.129285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'100","reported_seq":115,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.390694+0000","last_change":"2024-09-21T10:09:24.841480+0000","last_active":"2024-09-21T10:11:20.390694+0000","last_peered":"2024-09-21T10:11:20.390694+0000","last_clean":"2024-09-21T10:11:20.390694+0000","last_became_active":"2024-09-21T10:09:24.841145+0000","last_became_peered":"2024-09-21T10:09:24.841145+0000","last_unstale":"2024-09-21T10:11:20.390694+0000","last_undegraded":"2024-09-21T10:11:20.390694+0000","last_fullsized":"2024-09-21T10:11:20.390694+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.433227+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:11:30.758205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":23,"num_write":5,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":35780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":46,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.386029+0000","last_change":"2024-09-21T10:09:49.922489+0000","last_active":"2024-09-21T10:11:20.386029+0000","last_peered":"2024-09-21T10:11:20.386029+0000","last_clean":"2024-09-21T10:11:20.386029+0000","last_became_active":"2024-09-21T10:09:49.921629+0000","last_became_peered":"2024-09-21T10:09:49.921629+0000","last_unstale":"2024-09-21T10:11:20.386029+0000","last_undegraded":"2024-09-21T10:11:20.386029+0000","last_fullsized":"2024-09-21T10:11:20.386029+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:50.611979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":145,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.330980+0000","last_change":"2024-09-21T10:09:24.836647+0000","last_active":"2024-09-21T10:11:14.330980+0000","last_peered":"2024-09-21T10:11:14.330980+0000","last_clean":"2024-09-21T10:11:14.330980+0000","last_became_active":"2024-09-21T10:09:24.836278+0000","last_became_peered":"2024-09-21T10:09:24.836278+0000","last_unstale":"2024-09-21T10:11:14.330980+0000","last_undegraded":"2024-09-21T10:11:14.330980+0000","last_fullsized":"2024-09-21T10:11:14.330980+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:08:36.535892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"77'100","reported_seq":17,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.370967+0000","last_change":"2024-09-21T10:10:57.452531+0000","last_active":"2024-09-21T10:11:20.370967+0000","last_peered":"2024-09-21T10:11:20.370967+0000","last_clean":"2024-09-21T10:11:20.370967+0000","last_became_active":"2024-09-21T10:10:57.452150+0000","last_became_peered":"2024-09-21T10:10:57.452150+0000","last_unstale":"2024-09-21T10:11:20.370967+0000","last_undegraded":"2024-09-21T10:11:20.370967+0000","last_fullsized":"2024-09-21T10:11:20.370967+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:00.058077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:00.058077+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":399,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.392417+0000","last_change":"2024-09-21T10:03:38.381552+0000","last_active":"2024-09-21T10:11:20.392417+0000","last_peered":"2024-09-21T10:11:20.392417+0000","last_clean":"2024-09-21T10:11:20.392417+0000","last_became_active":"2024-09-21T10:03:38.380898+0000","last_became_peered":"2024-09-21T10:03:38.380898+0000","last_unstale":"2024-09-21T10:11:20.392417+0000","last_undegraded":"2024-09-21T10:11:20.392417+0000","last_fullsized":"2024-09-21T10:11:20.392417+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:53.409606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":333,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.390182+0000","last_change":"2024-09-21T10:10:57.459057+0000","last_active":"2024-09-21T10:11:20.390182+0000","last_peered":"2024-09-21T10:11:20.390182+0000","last_clean":"2024-09-21T10:11:20.390182+0000","last_became_active":"2024-09-21T10:10:57.451776+0000","last_became_peered":"2024-09-21T10:10:57.451776+0000","last_unstale":"2024-09-21T10:11:20.390182+0000","last_undegraded":"2024-09-21T10:11:20.390182+0000","last_fullsized":"2024-09-21T10:11:20.390182+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:00.138317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"273'7377","reported_seq":7475,"reported_epoch":273,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:11:31.834451+0000","last_change":"2024-09-21T10:10:58.805957+0000","last_active":"2024-09-21T10:11:31.834451+0000","last_peered":"2024-09-21T10:11:31.834451+0000","last_clean":"2024-09-21T10:10:56.364182+0000","last_became_active":"2024-09-21T10:10:58.682930+0000","last_became_peered":"2024-09-21T10:10:58.682930+0000","last_unstale":"2024-09-21T10:11:31.834451+0000","last_undegraded":"2024-09-21T10:11:31.834451+0000","last_fullsized":"2024-09-21T10:11:31.834451+0000","mapping_epoch":267,"log_start":"257'1100","ondisk_log_start":"257'1100","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6277,"log_dups_size":0,"ondisk_log_size":6277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432996352,"num_objects":6607,"num_object_clones":0,"num_object_copies":19821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6586,"num_objects_unfound":0,"num_objects_dirty":6607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7377,"num_write_kb":447488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[{"shards":"0,1,7","objects":6607}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"77'100","reported_seq":157,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.333658+0000","last_change":"2024-09-21T10:08:38.374573+0000","last_active":"2024-09-21T10:11:14.333658+0000","last_peered":"2024-09-21T10:11:14.333658+0000","last_clean":"2024-09-21T10:11:14.333658+0000","last_became_active":"2024-09-21T10:08:38.374075+0000","last_became_peered":"2024-09-21T10:08:38.374075+0000","last_unstale":"2024-09-21T10:11:14.333658+0000","last_undegraded":"2024-09-21T10:11:14.333658+0000","last_fullsized":"2024-09-21T10:11:14.333658+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:55.259385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:55.259385+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:28:26.784162+0000","scrub_duration":0.0015667260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":188,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.331521+0000","last_change":"2024-09-21T10:09:49.941392+0000","last_active":"2024-09-21T10:11:14.331521+0000","last_peered":"2024-09-21T10:11:14.331521+0000","last_clean":"2024-09-21T10:11:14.331521+0000","last_became_active":"2024-09-21T10:09:49.923535+0000","last_became_peered":"2024-09-21T10:09:49.923535+0000","last_unstale":"2024-09-21T10:11:14.331521+0000","last_undegraded":"2024-09-21T10:11:14.331521+0000","last_fullsized":"2024-09-21T10:11:14.331521+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:16.292163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.370889+0000","last_change":"2024-09-21T10:11:05.435502+0000","last_active":"2024-09-21T10:11:20.370889+0000","last_peered":"2024-09-21T10:11:20.370889+0000","last_clean":"2024-09-21T10:11:20.370889+0000","last_became_active":"2024-09-21T10:11:05.435216+0000","last_became_peered":"2024-09-21T10:11:05.435216+0000","last_unstale":"2024-09-21T10:11:20.370889+0000","last_undegraded":"2024-09-21T10:11:20.370889+0000","last_fullsized":"2024-09-21T10:11:20.370889+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:25.434424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"273'7346","reported_seq":7393,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:31.877699+0000","last_change":"2024-09-21T10:09:49.924187+0000","last_active":"2024-09-21T10:11:31.877699+0000","last_peered":"2024-09-21T10:11:31.877699+0000","last_clean":"2024-09-21T10:11:31.877699+0000","last_became_active":"2024-09-21T10:09:49.923789+0000","last_became_peered":"2024-09-21T10:09:49.923789+0000","last_unstale":"2024-09-21T10:11:31.877699+0000","last_undegraded":"2024-09-21T10:11:31.877699+0000","last_fullsized":"2024-09-21T10:11:31.877699+0000","mapping_epoch":255,"log_start":"257'1000","ondisk_log_start":"257'1000","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6346,"log_dups_size":0,"ondisk_log_size":6346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:52:49.559425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427294720,"num_objects":6520,"num_object_clones":0,"num_object_copies":19560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7346,"num_write_kb":443712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"77'100","reported_seq":206,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.390795+0000","last_change":"2024-09-21T10:09:23.839211+0000","last_active":"2024-09-21T10:11:20.390795+0000","last_peered":"2024-09-21T10:11:20.390795+0000","last_clean":"2024-09-21T10:11:20.390795+0000","last_became_active":"2024-09-21T10:09:23.838779+0000","last_became_peered":"2024-09-21T10:09:23.838779+0000","last_unstale":"2024-09-21T10:11:20.390795+0000","last_undegraded":"2024-09-21T10:11:20.390795+0000","last_fullsized":"2024-09-21T10:11:20.390795+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.981336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.981336+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:03:15.566873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":167,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.331371+0000","last_change":"2024-09-21T10:09:49.932631+0000","last_active":"2024-09-21T10:11:14.331371+0000","last_peered":"2024-09-21T10:11:14.331371+0000","last_clean":"2024-09-21T10:11:14.331371+0000","last_became_active":"2024-09-21T10:09:49.922897+0000","last_became_peered":"2024-09-21T10:09:49.922897+0000","last_unstale":"2024-09-21T10:11:14.331371+0000","last_undegraded":"2024-09-21T10:11:14.331371+0000","last_fullsized":"2024-09-21T10:11:14.331371+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:40.795476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":194,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.405981+0000","last_change":"2024-09-21T10:08:38.377732+0000","last_active":"2024-09-21T10:11:20.405981+0000","last_peered":"2024-09-21T10:11:20.405981+0000","last_clean":"2024-09-21T10:11:20.405981+0000","last_became_active":"2024-09-21T10:08:38.376601+0000","last_became_peered":"2024-09-21T10:08:38.376601+0000","last_unstale":"2024-09-21T10:11:20.405981+0000","last_undegraded":"2024-09-21T10:11:20.405981+0000","last_fullsized":"2024-09-21T10:11:20.405981+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:02.540158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.5","version":"273'7150","reported_seq":9398,"reported_epoch":273,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T10:11:31.748339+0000","last_change":"2024-09-21T10:10:58.840575+0000","last_active":"2024-09-21T10:11:31.748339+0000","last_peered":"2024-09-21T10:11:31.748339+0000","last_clean":"2024-09-21T10:10:56.374526+0000","last_became_active":"2024-09-21T10:10:58.733275+0000","last_became_peered":"2024-09-21T10:10:58.733275+0000","last_unstale":"2024-09-21T10:11:31.748339+0000","last_undegraded":"2024-09-21T10:11:31.748339+0000","last_fullsized":"2024-09-21T10:10:58.388641+0000","mapping_epoch":267,"log_start":"257'800","ondisk_log_start":"257'800","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6350,"log_dups_size":0,"ondisk_log_size":6350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:21:07.802295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417202176,"num_objects":6366,"num_object_clones":0,"num_object_copies":19098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4374,"num_objects_unfound":0,"num_objects_dirty":6366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7150,"num_write_kb":432512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1104,"num_bytes_recovered":72351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,6],"avail_no_missing":["7","3","6"],"object_location_counts":[{"shards":"3,6,7","objects":4374},{"shards":"6,7","objects":1992}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"77'100","reported_seq":258,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.388233+0000","last_change":"2024-09-21T10:08:38.382100+0000","last_active":"2024-09-21T10:11:20.388233+0000","last_peered":"2024-09-21T10:11:20.388233+0000","last_clean":"2024-09-21T10:11:20.388233+0000","last_became_active":"2024-09-21T10:08:38.381736+0000","last_became_peered":"2024-09-21T10:08:38.381736+0000","last_unstale":"2024-09-21T10:11:20.388233+0000","last_undegraded":"2024-09-21T10:11:20.388233+0000","last_fullsized":"2024-09-21T10:11:20.388233+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.289859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.289859+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:23.739980+0000","scrub_duration":0.001251809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":153,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.333522+0000","last_change":"2024-09-21T10:10:57.460732+0000","last_active":"2024-09-21T10:11:14.333522+0000","last_peered":"2024-09-21T10:11:14.333522+0000","last_clean":"2024-09-21T10:11:14.333522+0000","last_became_active":"2024-09-21T10:10:57.454689+0000","last_became_peered":"2024-09-21T10:10:57.454689+0000","last_unstale":"2024-09-21T10:11:14.333522+0000","last_undegraded":"2024-09-21T10:11:14.333522+0000","last_fullsized":"2024-09-21T10:11:14.333522+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:12.035022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":274,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.388631+0000","last_change":"2024-09-21T10:03:38.378384+0000","last_active":"2024-09-21T10:11:20.388631+0000","last_peered":"2024-09-21T10:11:20.388631+0000","last_clean":"2024-09-21T10:11:20.388631+0000","last_became_active":"2024-09-21T10:03:38.378072+0000","last_became_peered":"2024-09-21T10:03:38.378072+0000","last_unstale":"2024-09-21T10:11:20.388631+0000","last_undegraded":"2024-09-21T10:11:20.388631+0000","last_fullsized":"2024-09-21T10:11:20.388631+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:54:37.039622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"273'7526","reported_seq":7555,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:31.714352+0000","last_change":"2024-09-21T10:09:42.896949+0000","last_active":"2024-09-21T10:11:31.714352+0000","last_peered":"2024-09-21T10:11:31.714352+0000","last_clean":"2024-09-21T10:11:31.714352+0000","last_became_active":"2024-09-21T10:09:42.896669+0000","last_became_peered":"2024-09-21T10:09:42.896669+0000","last_unstale":"2024-09-21T10:11:31.714352+0000","last_undegraded":"2024-09-21T10:11:31.714352+0000","last_fullsized":"2024-09-21T10:11:31.714352+0000","mapping_epoch":252,"log_start":"257'1200","ondisk_log_start":"257'1200","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6326,"log_dups_size":0,"ondisk_log_size":6326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:44.279472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438960128,"num_objects":6698,"num_object_clones":0,"num_object_copies":20094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7526,"num_write_kb":455168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"77'100","reported_seq":125,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.331250+0000","last_change":"2024-09-21T10:07:26.188298+0000","last_active":"2024-09-21T10:11:14.331250+0000","last_peered":"2024-09-21T10:11:14.331250+0000","last_clean":"2024-09-21T10:11:14.331250+0000","last_became_active":"2024-09-21T10:07:24.345265+0000","last_became_peered":"2024-09-21T10:07:24.345265+0000","last_unstale":"2024-09-21T10:11:14.331250+0000","last_undegraded":"2024-09-21T10:11:14.331250+0000","last_fullsized":"2024-09-21T10:11:14.331250+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:07:26.188220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:07:26.188220+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:46.381710+0000","scrub_duration":0.021884888000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":203,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.391015+0000","last_change":"2024-09-21T10:07:57.551556+0000","last_active":"2024-09-21T10:11:20.391015+0000","last_peered":"2024-09-21T10:11:20.391015+0000","last_clean":"2024-09-21T10:11:20.391015+0000","last_became_active":"2024-09-21T10:07:57.551249+0000","last_became_peered":"2024-09-21T10:07:57.551249+0000","last_unstale":"2024-09-21T10:11:20.391015+0000","last_undegraded":"2024-09-21T10:11:20.391015+0000","last_fullsized":"2024-09-21T10:11:20.391015+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:57.557152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":300,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.382773+0000","last_change":"2024-09-21T10:09:24.839665+0000","last_active":"2024-09-21T10:11:20.382773+0000","last_peered":"2024-09-21T10:11:20.382773+0000","last_clean":"2024-09-21T10:11:20.382773+0000","last_became_active":"2024-09-21T10:09:24.839353+0000","last_became_peered":"2024-09-21T10:09:24.839353+0000","last_unstale":"2024-09-21T10:11:20.382773+0000","last_undegraded":"2024-09-21T10:11:20.382773+0000","last_fullsized":"2024-09-21T10:11:20.382773+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:51.760073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3","version":"274'7403","reported_seq":7432,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-21T10:11:32.761298+0000","last_change":"2024-09-21T10:09:43.231199+0000","last_active":"2024-09-21T10:11:32.761298+0000","last_peered":"2024-09-21T10:11:32.761298+0000","last_clean":"2024-09-21T10:11:32.761298+0000","last_became_active":"2024-09-21T10:09:43.230922+0000","last_became_peered":"2024-09-21T10:09:43.230922+0000","last_unstale":"2024-09-21T10:11:32.761298+0000","last_undegraded":"2024-09-21T10:11:32.761298+0000","last_fullsized":"2024-09-21T10:11:32.761298+0000","mapping_epoch":252,"log_start":"257'1200","ondisk_log_start":"257'1200","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6203,"log_dups_size":0,"ondisk_log_size":6203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:39.944122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424345600,"num_objects":6475,"num_object_clones":0,"num_object_copies":19425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7403,"num_write_kb":444096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":147,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.331026+0000","last_change":"2024-09-21T10:04:53.355975+0000","last_active":"2024-09-21T10:11:14.331026+0000","last_peered":"2024-09-21T10:11:14.331026+0000","last_clean":"2024-09-21T10:11:14.331026+0000","last_became_active":"2024-09-21T10:04:42.547505+0000","last_became_peered":"2024-09-21T10:04:42.547505+0000","last_unstale":"2024-09-21T10:11:14.331026+0000","last_undegraded":"2024-09-21T10:11:14.331026+0000","last_fullsized":"2024-09-21T10:11:14.331026+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.355912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.355912+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:43.456830+0000","scrub_duration":0.0017980030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":399,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.390874+0000","last_change":"2024-09-21T10:09:49.939553+0000","last_active":"2024-09-21T10:11:20.390874+0000","last_peered":"2024-09-21T10:11:20.390874+0000","last_clean":"2024-09-21T10:11:20.390874+0000","last_became_active":"2024-09-21T10:09:49.925094+0000","last_became_peered":"2024-09-21T10:09:49.925094+0000","last_unstale":"2024-09-21T10:11:20.390874+0000","last_undegraded":"2024-09-21T10:11:20.390874+0000","last_fullsized":"2024-09-21T10:11:20.390874+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:12:12.933089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":177,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.393587+0000","last_change":"2024-09-21T10:08:38.379747+0000","last_active":"2024-09-21T10:11:20.393587+0000","last_peered":"2024-09-21T10:11:20.393587+0000","last_clean":"2024-09-21T10:11:20.393587+0000","last_became_active":"2024-09-21T10:08:38.378983+0000","last_became_peered":"2024-09-21T10:08:38.378983+0000","last_unstale":"2024-09-21T10:11:20.393587+0000","last_undegraded":"2024-09-21T10:11:20.393587+0000","last_fullsized":"2024-09-21T10:11:20.393587+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:45:08.003016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"274'7562","reported_seq":7617,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-21T10:11:34.034685+0000","last_change":"2024-09-21T10:09:42.896430+0000","last_active":"2024-09-21T10:11:34.034685+0000","last_peered":"2024-09-21T10:11:34.034685+0000","last_clean":"2024-09-21T10:11:34.034685+0000","last_became_active":"2024-09-21T10:09:42.896019+0000","last_became_peered":"2024-09-21T10:09:42.896019+0000","last_unstale":"2024-09-21T10:11:34.034685+0000","last_undegraded":"2024-09-21T10:11:34.034685+0000","last_fullsized":"2024-09-21T10:11:34.034685+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":7562,"log_dups_size":0,"ondisk_log_size":7562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:59:18.633498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421396480,"num_objects":6430,"num_object_clones":0,"num_object_copies":19290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7562,"num_write_kb":447744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":161,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-21T10:11:32.434681+0000","last_change":"2024-09-21T10:10:57.403722+0000","last_active":"2024-09-21T10:11:32.434681+0000","last_peered":"2024-09-21T10:11:32.434681+0000","last_clean":"2024-09-21T10:11:32.434681+0000","last_became_active":"2024-09-21T10:10:57.394700+0000","last_became_peered":"2024-09-21T10:10:57.394700+0000","last_unstale":"2024-09-21T10:11:32.434681+0000","last_undegraded":"2024-09-21T10:11:32.434681+0000","last_fullsized":"2024-09-21T10:11:32.434681+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:02.152981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:02.152981+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:52:34.887763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":248,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.388763+0000","last_change":"2024-09-21T10:08:38.379849+0000","last_active":"2024-09-21T10:11:20.388763+0000","last_peered":"2024-09-21T10:11:20.388763+0000","last_clean":"2024-09-21T10:11:20.388763+0000","last_became_active":"2024-09-21T10:08:38.379543+0000","last_became_peered":"2024-09-21T10:08:38.379543+0000","last_unstale":"2024-09-21T10:11:20.388763+0000","last_undegraded":"2024-09-21T10:11:20.388763+0000","last_fullsized":"2024-09-21T10:11:20.388763+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:29.041802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":210,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-21T10:11:32.436285+0000","last_change":"2024-09-21T10:07:24.342950+0000","last_active":"2024-09-21T10:11:32.436285+0000","last_peered":"2024-09-21T10:11:32.436285+0000","last_clean":"2024-09-21T10:11:32.436285+0000","last_became_active":"2024-09-21T10:07:24.342477+0000","last_became_peered":"2024-09-21T10:07:24.342477+0000","last_unstale":"2024-09-21T10:11:32.436285+0000","last_undegraded":"2024-09-21T10:11:32.436285+0000","last_fullsized":"2024-09-21T10:11:32.436285+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:33:13.664050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1","version":"274'7554","reported_seq":7582,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-21T10:11:32.791189+0000","last_change":"2024-09-21T10:09:42.895259+0000","last_active":"2024-09-21T10:11:32.791189+0000","last_peered":"2024-09-21T10:11:32.791189+0000","last_clean":"2024-09-21T10:11:32.791189+0000","last_became_active":"2024-09-21T10:09:42.895087+0000","last_became_peered":"2024-09-21T10:09:42.895087+0000","last_unstale":"2024-09-21T10:11:32.791189+0000","last_undegraded":"2024-09-21T10:11:32.791189+0000","last_fullsized":"2024-09-21T10:11:32.791189+0000","mapping_epoch":252,"log_start":"257'1400","ondisk_log_start":"257'1400","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6154,"log_dups_size":0,"ondisk_log_size":6154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:40:59.270926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430047232,"num_objects":6562,"num_object_clones":0,"num_object_copies":19686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7554,"num_write_kb":451712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"77'100","reported_seq":302,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.383710+0000","last_change":"2024-09-21T10:10:57.395906+0000","last_active":"2024-09-21T10:11:20.383710+0000","last_peered":"2024-09-21T10:11:20.383710+0000","last_clean":"2024-09-21T10:11:20.383710+0000","last_became_active":"2024-09-21T10:10:57.395366+0000","last_became_peered":"2024-09-21T10:10:57.395366+0000","last_unstale":"2024-09-21T10:11:20.383710+0000","last_undegraded":"2024-09-21T10:11:20.383710+0000","last_fullsized":"2024-09-21T10:11:20.383710+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.540323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.540323+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:34.873336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":302,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-21T10:11:32.435053+0000","last_change":"2024-09-21T10:07:36.464496+0000","last_active":"2024-09-21T10:11:32.435053+0000","last_peered":"2024-09-21T10:11:32.435053+0000","last_clean":"2024-09-21T10:11:32.435053+0000","last_became_active":"2024-09-21T10:07:36.464172+0000","last_became_peered":"2024-09-21T10:07:36.464172+0000","last_unstale":"2024-09-21T10:11:32.435053+0000","last_undegraded":"2024-09-21T10:11:32.435053+0000","last_fullsized":"2024-09-21T10:11:32.435053+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:09.839709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.388457+0000","last_change":"2024-09-21T10:11:05.442332+0000","last_active":"2024-09-21T10:11:20.388457+0000","last_peered":"2024-09-21T10:11:20.388457+0000","last_clean":"2024-09-21T10:11:20.388457+0000","last_became_active":"2024-09-21T10:11:05.442003+0000","last_became_peered":"2024-09-21T10:11:05.442003+0000","last_unstale":"2024-09-21T10:11:20.388457+0000","last_undegraded":"2024-09-21T10:11:20.388457+0000","last_fullsized":"2024-09-21T10:11:20.388457+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:34.986309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"273'7457","reported_seq":7510,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:31.433988+0000","last_change":"2024-09-21T10:09:43.231401+0000","last_active":"2024-09-21T10:11:31.433988+0000","last_peered":"2024-09-21T10:11:31.433988+0000","last_clean":"2024-09-21T10:11:31.433988+0000","last_became_active":"2024-09-21T10:09:43.230977+0000","last_became_peered":"2024-09-21T10:09:43.230977+0000","last_unstale":"2024-09-21T10:11:31.433988+0000","last_undegraded":"2024-09-21T10:11:31.433988+0000","last_fullsized":"2024-09-21T10:11:31.433988+0000","mapping_epoch":252,"log_start":"259'1700","ondisk_log_start":"259'1700","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5757,"log_dups_size":0,"ondisk_log_size":5757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:15.907809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433520640,"num_objects":6615,"num_object_clones":0,"num_object_copies":19845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7457,"num_write_kb":450304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":124,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-21T10:11:32.436478+0000","last_change":"2024-09-21T10:08:04.645700+0000","last_active":"2024-09-21T10:11:32.436478+0000","last_peered":"2024-09-21T10:11:32.436478+0000","last_clean":"2024-09-21T10:11:32.436478+0000","last_became_active":"2024-09-21T10:07:57.548189+0000","last_became_peered":"2024-09-21T10:07:57.548189+0000","last_unstale":"2024-09-21T10:11:32.436478+0000","last_undegraded":"2024-09-21T10:11:32.436478+0000","last_fullsized":"2024-09-21T10:11:32.436478+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:48.318010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:48.318010+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T16:31:28.173059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":401,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.393201+0000","last_change":"2024-09-21T10:03:38.386604+0000","last_active":"2024-09-21T10:11:20.393201+0000","last_peered":"2024-09-21T10:11:20.393201+0000","last_clean":"2024-09-21T10:11:20.393201+0000","last_became_active":"2024-09-21T10:03:38.386283+0000","last_became_peered":"2024-09-21T10:03:38.386283+0000","last_unstale":"2024-09-21T10:11:20.393201+0000","last_undegraded":"2024-09-21T10:11:20.393201+0000","last_fullsized":"2024-09-21T10:11:20.393201+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:27:05.944880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":204,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.392490+0000","last_change":"2024-09-21T10:08:38.372486+0000","last_active":"2024-09-21T10:11:20.392490+0000","last_peered":"2024-09-21T10:11:20.392490+0000","last_clean":"2024-09-21T10:11:20.392490+0000","last_became_active":"2024-09-21T10:08:38.370669+0000","last_became_peered":"2024-09-21T10:08:38.370669+0000","last_unstale":"2024-09-21T10:11:20.392490+0000","last_undegraded":"2024-09-21T10:11:20.392490+0000","last_fullsized":"2024-09-21T10:11:20.392490+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:53:23.247341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.a","version":"273'7357","reported_seq":7419,"reported_epoch":273,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:11:31.466171+0000","last_change":"2024-09-21T10:10:58.704699+0000","last_active":"2024-09-21T10:11:31.466171+0000","last_peered":"2024-09-21T10:11:31.466171+0000","last_clean":"2024-09-21T10:10:56.390149+0000","last_became_active":"2024-09-21T10:10:58.682906+0000","last_became_peered":"2024-09-21T10:10:58.682906+0000","last_unstale":"2024-09-21T10:11:31.466171+0000","last_undegraded":"2024-09-21T10:11:31.466171+0000","last_fullsized":"2024-09-21T10:11:31.466171+0000","mapping_epoch":267,"log_start":"258'1600","ondisk_log_start":"258'1600","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5757,"log_dups_size":0,"ondisk_log_size":5757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429981696,"num_objects":6561,"num_object_clones":0,"num_object_copies":19683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6561,"num_objects_unfound":0,"num_objects_dirty":6561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7357,"num_write_kb":445376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,1],"avail_no_missing":["3","1","6"],"object_location_counts":[{"shards":"1,3,6","objects":6561}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":164,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.330978+0000","last_change":"2024-09-21T10:09:49.924286+0000","last_active":"2024-09-21T10:11:14.330978+0000","last_peered":"2024-09-21T10:11:14.330978+0000","last_clean":"2024-09-21T10:11:14.330978+0000","last_became_active":"2024-09-21T10:09:49.923897+0000","last_became_peered":"2024-09-21T10:09:49.923897+0000","last_unstale":"2024-09-21T10:11:14.330978+0000","last_undegraded":"2024-09-21T10:11:14.330978+0000","last_fullsized":"2024-09-21T10:11:14.330978+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:47.393146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:47.393146+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:44.898083+0000","scrub_duration":0.001601086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":157,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-21T10:11:32.435381+0000","last_change":"2024-09-21T10:10:57.402946+0000","last_active":"2024-09-21T10:11:32.435381+0000","last_peered":"2024-09-21T10:11:32.435381+0000","last_clean":"2024-09-21T10:11:32.435381+0000","last_became_active":"2024-09-21T10:10:57.395450+0000","last_became_peered":"2024-09-21T10:10:57.395450+0000","last_unstale":"2024-09-21T10:11:32.435381+0000","last_undegraded":"2024-09-21T10:11:32.435381+0000","last_fullsized":"2024-09-21T10:11:32.435381+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:08.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":385,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.395394+0000","last_change":"2024-09-21T10:10:57.390595+0000","last_active":"2024-09-21T10:11:20.395394+0000","last_peered":"2024-09-21T10:11:20.395394+0000","last_clean":"2024-09-21T10:11:20.395394+0000","last_became_active":"2024-09-21T10:10:57.390265+0000","last_became_peered":"2024-09-21T10:10:57.390265+0000","last_unstale":"2024-09-21T10:11:20.395394+0000","last_undegraded":"2024-09-21T10:11:20.395394+0000","last_fullsized":"2024-09-21T10:11:20.395394+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:37.485063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":281,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.388506+0000","last_change":"2024-09-21T10:09:24.837651+0000","last_active":"2024-09-21T10:11:20.388506+0000","last_peered":"2024-09-21T10:11:20.388506+0000","last_clean":"2024-09-21T10:11:20.388506+0000","last_became_active":"2024-09-21T10:09:24.837226+0000","last_became_peered":"2024-09-21T10:09:24.837226+0000","last_unstale":"2024-09-21T10:11:20.388506+0000","last_undegraded":"2024-09-21T10:11:20.388506+0000","last_fullsized":"2024-09-21T10:11:20.388506+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:16.649601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"273'7396","reported_seq":7456,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:31.461134+0000","last_change":"2024-09-21T10:09:49.926338+0000","last_active":"2024-09-21T10:11:31.461134+0000","last_peered":"2024-09-21T10:11:31.461134+0000","last_clean":"2024-09-21T10:11:31.461134+0000","last_became_active":"2024-09-21T10:09:49.926018+0000","last_became_peered":"2024-09-21T10:09:49.926018+0000","last_unstale":"2024-09-21T10:11:31.461134+0000","last_undegraded":"2024-09-21T10:11:31.461134+0000","last_fullsized":"2024-09-21T10:11:31.461134+0000","mapping_epoch":255,"log_start":"258'1700","ondisk_log_start":"258'1700","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5696,"log_dups_size":0,"ondisk_log_size":5696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:09:49.364660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430309376,"num_objects":6566,"num_object_clones":0,"num_object_copies":19698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7396,"num_write_kb":446784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":14,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.387720+0000","last_change":"2024-09-21T10:11:05.443167+0000","last_active":"2024-09-21T10:11:20.387720+0000","last_peered":"2024-09-21T10:11:20.387720+0000","last_clean":"2024-09-21T10:11:20.387720+0000","last_became_active":"2024-09-21T10:11:05.442805+0000","last_became_peered":"2024-09-21T10:11:05.442805+0000","last_unstale":"2024-09-21T10:11:20.387720+0000","last_undegraded":"2024-09-21T10:11:20.387720+0000","last_fullsized":"2024-09-21T10:11:20.387720+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.550551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.550551+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:58:57.998543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":138,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.331226+0000","last_change":"2024-09-21T10:09:23.838944+0000","last_active":"2024-09-21T10:11:14.331226+0000","last_peered":"2024-09-21T10:11:14.331226+0000","last_clean":"2024-09-21T10:11:14.331226+0000","last_became_active":"2024-09-21T10:09:23.837886+0000","last_became_peered":"2024-09-21T10:09:23.837886+0000","last_unstale":"2024-09-21T10:11:14.331226+0000","last_undegraded":"2024-09-21T10:11:14.331226+0000","last_fullsized":"2024-09-21T10:11:14.331226+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:16.928195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":278,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.385474+0000","last_change":"2024-09-21T10:09:24.839770+0000","last_active":"2024-09-21T10:11:20.385474+0000","last_peered":"2024-09-21T10:11:20.385474+0000","last_clean":"2024-09-21T10:11:20.385474+0000","last_became_active":"2024-09-21T10:09:24.839479+0000","last_became_peered":"2024-09-21T10:09:24.839479+0000","last_unstale":"2024-09-21T10:11:20.385474+0000","last_undegraded":"2024-09-21T10:11:20.385474+0000","last_fullsized":"2024-09-21T10:11:20.385474+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:11:19.618193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":46,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.385741+0000","last_change":"2024-09-21T10:09:49.923044+0000","last_active":"2024-09-21T10:11:20.385741+0000","last_peered":"2024-09-21T10:11:20.385741+0000","last_clean":"2024-09-21T10:11:20.385741+0000","last_became_active":"2024-09-21T10:09:49.922711+0000","last_became_peered":"2024-09-21T10:09:49.922711+0000","last_unstale":"2024-09-21T10:11:20.385741+0000","last_undegraded":"2024-09-21T10:11:20.385741+0000","last_fullsized":"2024-09-21T10:11:20.385741+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:14.090084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"273'7578","reported_seq":7645,"reported_epoch":273,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:11:31.460248+0000","last_change":"2024-09-21T10:10:58.695837+0000","last_active":"2024-09-21T10:11:31.460248+0000","last_peered":"2024-09-21T10:11:31.460248+0000","last_clean":"2024-09-21T10:10:56.379248+0000","last_became_active":"2024-09-21T10:10:58.672712+0000","last_became_peered":"2024-09-21T10:10:58.672712+0000","last_unstale":"2024-09-21T10:11:31.460248+0000","last_undegraded":"2024-09-21T10:11:31.460248+0000","last_fullsized":"2024-09-21T10:11:31.460248+0000","mapping_epoch":267,"log_start":"259'1900","ondisk_log_start":"259'1900","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5678,"log_dups_size":0,"ondisk_log_size":5678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437911552,"num_objects":6682,"num_object_clones":0,"num_object_copies":20046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6682,"num_objects_unfound":0,"num_objects_dirty":6682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7578,"num_write_kb":456320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,0],"avail_no_missing":["3","0","6"],"object_location_counts":[{"shards":"0,3,6","objects":6682}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":178,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.331071+0000","last_change":"2024-09-21T10:09:24.838434+0000","last_active":"2024-09-21T10:11:14.331071+0000","last_peered":"2024-09-21T10:11:14.331071+0000","last_clean":"2024-09-21T10:11:14.331071+0000","last_became_active":"2024-09-21T10:09:24.838066+0000","last_became_peered":"2024-09-21T10:09:24.838066+0000","last_unstale":"2024-09-21T10:11:14.331071+0000","last_undegraded":"2024-09-21T10:11:14.331071+0000","last_fullsized":"2024-09-21T10:11:14.331071+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:52.361137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:52.361137+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:19:57.788195+0000","scrub_duration":0.001255358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":440,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-21T10:11:32.435176+0000","last_change":"2024-09-21T10:09:23.837859+0000","last_active":"2024-09-21T10:11:32.435176+0000","last_peered":"2024-09-21T10:11:32.435176+0000","last_clean":"2024-09-21T10:11:32.435176+0000","last_became_active":"2024-09-21T10:09:23.837488+0000","last_became_peered":"2024-09-21T10:09:23.837488+0000","last_unstale":"2024-09-21T10:11:32.435176+0000","last_undegraded":"2024-09-21T10:11:32.435176+0000","last_fullsized":"2024-09-21T10:11:32.435176+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:25.215501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.330920+0000","last_change":"2024-09-21T10:10:27.806070+0000","last_active":"2024-09-21T10:11:14.330920+0000","last_peered":"2024-09-21T10:11:14.330920+0000","last_clean":"2024-09-21T10:11:14.330920+0000","last_became_active":"2024-09-21T10:10:27.805718+0000","last_became_peered":"2024-09-21T10:10:27.805718+0000","last_unstale":"2024-09-21T10:11:14.330920+0000","last_undegraded":"2024-09-21T10:11:14.330920+0000","last_fullsized":"2024-09-21T10:11:14.330920+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:31.144118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":434,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-21T10:11:32.440348+0000","last_change":"2024-09-21T10:10:57.395073+0000","last_active":"2024-09-21T10:11:32.440348+0000","last_peered":"2024-09-21T10:11:32.440348+0000","last_clean":"2024-09-21T10:11:32.440348+0000","last_became_active":"2024-09-21T10:10:57.394674+0000","last_became_peered":"2024-09-21T10:10:57.394674+0000","last_unstale":"2024-09-21T10:11:32.440348+0000","last_undegraded":"2024-09-21T10:11:32.440348+0000","last_fullsized":"2024-09-21T10:11:32.440348+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:48:48.912649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"273'7592","reported_seq":7643,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:31.450370+0000","last_change":"2024-09-21T10:09:43.231554+0000","last_active":"2024-09-21T10:11:31.450370+0000","last_peered":"2024-09-21T10:11:31.450370+0000","last_clean":"2024-09-21T10:11:31.450370+0000","last_became_active":"2024-09-21T10:09:43.231211+0000","last_became_peered":"2024-09-21T10:09:43.231211+0000","last_unstale":"2024-09-21T10:11:31.450370+0000","last_undegraded":"2024-09-21T10:11:31.450370+0000","last_fullsized":"2024-09-21T10:11:31.450370+0000","mapping_epoch":252,"log_start":"259'1900","ondisk_log_start":"259'1900","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5692,"log_dups_size":0,"ondisk_log_size":5692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:16:41.457768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439353344,"num_objects":6704,"num_object_clones":0,"num_object_copies":20112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7592,"num_write_kb":457472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":302,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.385622+0000","last_change":"2024-09-21T10:10:57.398437+0000","last_active":"2024-09-21T10:11:20.385622+0000","last_peered":"2024-09-21T10:11:20.385622+0000","last_clean":"2024-09-21T10:11:20.385622+0000","last_became_active":"2024-09-21T10:10:57.397441+0000","last_became_peered":"2024-09-21T10:10:57.397441+0000","last_unstale":"2024-09-21T10:11:20.385622+0000","last_undegraded":"2024-09-21T10:11:20.385622+0000","last_fullsized":"2024-09-21T10:11:20.385622+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:04.043917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:04.043917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:53:31.813612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":78,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.388752+0000","last_change":"2024-09-21T10:08:26.309323+0000","last_active":"2024-09-21T10:11:20.388752+0000","last_peered":"2024-09-21T10:11:20.388752+0000","last_clean":"2024-09-21T10:11:20.388752+0000","last_became_active":"2024-09-21T10:08:26.309004+0000","last_became_peered":"2024-09-21T10:08:26.309004+0000","last_unstale":"2024-09-21T10:11:20.388752+0000","last_undegraded":"2024-09-21T10:11:20.388752+0000","last_fullsized":"2024-09-21T10:11:20.388752+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:37:20.626702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":112,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.382948+0000","last_change":"2024-09-21T10:10:28.180093+0000","last_active":"2024-09-21T10:11:20.382948+0000","last_peered":"2024-09-21T10:11:20.382948+0000","last_clean":"2024-09-21T10:11:20.382948+0000","last_became_active":"2024-09-21T10:10:28.179559+0000","last_became_peered":"2024-09-21T10:10:28.179559+0000","last_unstale":"2024-09-21T10:11:20.382948+0000","last_undegraded":"2024-09-21T10:11:20.382948+0000","last_fullsized":"2024-09-21T10:11:20.382948+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:52:46.146392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":210,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-21T10:11:32.435737+0000","last_change":"2024-09-21T10:08:26.307041+0000","last_active":"2024-09-21T10:11:32.435737+0000","last_peered":"2024-09-21T10:11:32.435737+0000","last_clean":"2024-09-21T10:11:32.435737+0000","last_became_active":"2024-09-21T10:08:26.306715+0000","last_became_peered":"2024-09-21T10:08:26.306715+0000","last_unstale":"2024-09-21T10:11:32.435737+0000","last_undegraded":"2024-09-21T10:11:32.435737+0000","last_fullsized":"2024-09-21T10:11:32.435737+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:19:26.192398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"274'7696","reported_seq":7752,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-21T10:11:34.043166+0000","last_change":"2024-09-21T10:09:43.232893+0000","last_active":"2024-09-21T10:11:34.043166+0000","last_peered":"2024-09-21T10:11:34.043166+0000","last_clean":"2024-09-21T10:11:34.043166+0000","last_became_active":"2024-09-21T10:09:43.232461+0000","last_became_peered":"2024-09-21T10:09:43.232461+0000","last_unstale":"2024-09-21T10:11:34.043166+0000","last_undegraded":"2024-09-21T10:11:34.043166+0000","last_fullsized":"2024-09-21T10:11:34.043166+0000","mapping_epoch":252,"log_start":"257'100","ondisk_log_start":"257'100","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":7596,"log_dups_size":0,"ondisk_log_size":7596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:39.778833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428539928,"num_objects":6540,"num_object_clones":0,"num_object_copies":19620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6540,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7696,"num_write_kb":455489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":248,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.388158+0000","last_change":"2024-09-21T10:04:45.242148+0000","last_active":"2024-09-21T10:11:20.388158+0000","last_peered":"2024-09-21T10:11:20.388158+0000","last_clean":"2024-09-21T10:11:20.388158+0000","last_became_active":"2024-09-21T10:04:42.549844+0000","last_became_peered":"2024-09-21T10:04:42.549844+0000","last_unstale":"2024-09-21T10:11:20.388158+0000","last_undegraded":"2024-09-21T10:11:20.388158+0000","last_fullsized":"2024-09-21T10:11:20.388158+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_deep_scrub":"77'100","last_deep_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.242073+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:03:42.250480+0000","scrub_duration":0.0023466239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":17,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.371188+0000","last_change":"2024-09-21T10:10:57.392880+0000","last_active":"2024-09-21T10:11:20.371188+0000","last_peered":"2024-09-21T10:11:20.371188+0000","last_clean":"2024-09-21T10:11:20.371188+0000","last_became_active":"2024-09-21T10:10:57.392539+0000","last_became_peered":"2024-09-21T10:10:57.392539+0000","last_unstale":"2024-09-21T10:11:20.371188+0000","last_undegraded":"2024-09-21T10:11:20.371188+0000","last_fullsized":"2024-09-21T10:11:20.371188+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:49.754955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":322,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.382620+0000","last_change":"2024-09-21T10:09:23.865342+0000","last_active":"2024-09-21T10:11:20.382620+0000","last_peered":"2024-09-21T10:11:20.382620+0000","last_clean":"2024-09-21T10:11:20.382620+0000","last_became_active":"2024-09-21T10:09:23.865008+0000","last_became_peered":"2024-09-21T10:09:23.865008+0000","last_unstale":"2024-09-21T10:11:20.382620+0000","last_undegraded":"2024-09-21T10:11:20.382620+0000","last_fullsized":"2024-09-21T10:11:20.382620+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:03.356101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.370845+0000","last_change":"2024-09-21T10:10:57.394707+0000","last_active":"2024-09-21T10:11:20.370845+0000","last_peered":"2024-09-21T10:11:20.370845+0000","last_clean":"2024-09-21T10:11:20.370845+0000","last_became_active":"2024-09-21T10:10:57.394363+0000","last_became_peered":"2024-09-21T10:10:57.394363+0000","last_unstale":"2024-09-21T10:11:20.370845+0000","last_undegraded":"2024-09-21T10:11:20.370845+0000","last_fullsized":"2024-09-21T10:11:20.370845+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:21.141052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"274'7650","reported_seq":7705,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-21T10:11:34.087070+0000","last_change":"2024-09-21T10:09:43.232987+0000","last_active":"2024-09-21T10:11:34.087070+0000","last_peered":"2024-09-21T10:11:34.087070+0000","last_clean":"2024-09-21T10:11:34.087070+0000","last_became_active":"2024-09-21T10:09:43.232577+0000","last_became_peered":"2024-09-21T10:09:43.232577+0000","last_unstale":"2024-09-21T10:11:34.087070+0000","last_undegraded":"2024-09-21T10:11:34.087070+0000","last_fullsized":"2024-09-21T10:11:34.087070+0000","mapping_epoch":252,"log_start":"257'100","ondisk_log_start":"257'100","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":7550,"log_dups_size":0,"ondisk_log_size":7550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:14.536774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425459712,"num_objects":6492,"num_object_clones":0,"num_object_copies":19476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7650,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":265,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.387969+0000","last_change":"2024-09-21T10:09:23.839563+0000","last_active":"2024-09-21T10:11:20.387969+0000","last_peered":"2024-09-21T10:11:20.387969+0000","last_clean":"2024-09-21T10:11:20.387969+0000","last_became_active":"2024-09-21T10:09:23.839207+0000","last_became_peered":"2024-09-21T10:09:23.839207+0000","last_unstale":"2024-09-21T10:11:20.387969+0000","last_undegraded":"2024-09-21T10:11:20.387969+0000","last_fullsized":"2024-09-21T10:11:20.387969+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:49.226496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:49.226496+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:03.960125+0000","scrub_duration":0.001656865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":175,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-21T10:11:32.435589+0000","last_change":"2024-09-21T10:10:57.395167+0000","last_active":"2024-09-21T10:11:32.435589+0000","last_peered":"2024-09-21T10:11:32.435589+0000","last_clean":"2024-09-21T10:11:32.435589+0000","last_became_active":"2024-09-21T10:10:57.394818+0000","last_became_peered":"2024-09-21T10:10:57.394818+0000","last_unstale":"2024-09-21T10:11:32.435589+0000","last_undegraded":"2024-09-21T10:11:32.435589+0000","last_fullsized":"2024-09-21T10:11:32.435589+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:57.224113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":158,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.333589+0000","last_change":"2024-09-21T10:09:49.922782+0000","last_active":"2024-09-21T10:11:14.333589+0000","last_peered":"2024-09-21T10:11:14.333589+0000","last_clean":"2024-09-21T10:11:14.333589+0000","last_became_active":"2024-09-21T10:09:49.922465+0000","last_became_peered":"2024-09-21T10:09:49.922465+0000","last_unstale":"2024-09-21T10:11:14.333589+0000","last_undegraded":"2024-09-21T10:11:14.333589+0000","last_fullsized":"2024-09-21T10:11:14.333589+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:06.151245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":316,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.383862+0000","last_change":"2024-09-21T10:10:57.397251+0000","last_active":"2024-09-21T10:11:20.383862+0000","last_peered":"2024-09-21T10:11:20.383862+0000","last_clean":"2024-09-21T10:11:20.383862+0000","last_became_active":"2024-09-21T10:10:57.396271+0000","last_became_peered":"2024-09-21T10:10:57.396271+0000","last_unstale":"2024-09-21T10:11:20.383862+0000","last_undegraded":"2024-09-21T10:11:20.383862+0000","last_fullsized":"2024-09-21T10:11:20.383862+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:17.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.f","version":"273'7574","reported_seq":7603,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:31.736646+0000","last_change":"2024-09-21T10:09:43.232625+0000","last_active":"2024-09-21T10:11:31.736646+0000","last_peered":"2024-09-21T10:11:31.736646+0000","last_clean":"2024-09-21T10:11:31.736646+0000","last_became_active":"2024-09-21T10:09:43.232331+0000","last_became_peered":"2024-09-21T10:09:43.232331+0000","last_unstale":"2024-09-21T10:11:31.736646+0000","last_undegraded":"2024-09-21T10:11:31.736646+0000","last_fullsized":"2024-09-21T10:11:31.736646+0000","mapping_epoch":252,"log_start":"257'1300","ondisk_log_start":"257'1300","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6274,"log_dups_size":0,"ondisk_log_size":6274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:01:16.634826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439353344,"num_objects":6704,"num_object_clones":0,"num_object_copies":20112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7574,"num_write_kb":456896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":121,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-21T10:11:32.435987+0000","last_change":"2024-09-21T10:11:05.455170+0000","last_active":"2024-09-21T10:11:32.435987+0000","last_peered":"2024-09-21T10:11:32.435987+0000","last_clean":"2024-09-21T10:11:32.435987+0000","last_became_active":"2024-09-21T10:11:05.454607+0000","last_became_peered":"2024-09-21T10:11:05.454607+0000","last_unstale":"2024-09-21T10:11:32.435987+0000","last_undegraded":"2024-09-21T10:11:32.435987+0000","last_fullsized":"2024-09-21T10:11:32.435987+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:59.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:59.099852+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:14.788975+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":407,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.392627+0000","last_change":"2024-09-21T10:09:23.841186+0000","last_active":"2024-09-21T10:11:20.392627+0000","last_peered":"2024-09-21T10:11:20.392627+0000","last_clean":"2024-09-21T10:11:20.392627+0000","last_became_active":"2024-09-21T10:09:23.840854+0000","last_became_peered":"2024-09-21T10:09:23.840854+0000","last_unstale":"2024-09-21T10:11:20.392627+0000","last_undegraded":"2024-09-21T10:11:20.392627+0000","last_fullsized":"2024-09-21T10:11:20.392627+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:29.666358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":96,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.388040+0000","last_change":"2024-09-21T10:08:38.380867+0000","last_active":"2024-09-21T10:11:20.388040+0000","last_peered":"2024-09-21T10:11:20.388040+0000","last_clean":"2024-09-21T10:11:20.388040+0000","last_became_active":"2024-09-21T10:08:38.379599+0000","last_became_peered":"2024-09-21T10:08:38.379599+0000","last_unstale":"2024-09-21T10:11:20.388040+0000","last_undegraded":"2024-09-21T10:11:20.388040+0000","last_fullsized":"2024-09-21T10:11:20.388040+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:50.499470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":403,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.391084+0000","last_change":"2024-09-21T10:09:49.926548+0000","last_active":"2024-09-21T10:11:20.391084+0000","last_peered":"2024-09-21T10:11:20.391084+0000","last_clean":"2024-09-21T10:11:20.391084+0000","last_became_active":"2024-09-21T10:09:49.926255+0000","last_became_peered":"2024-09-21T10:09:49.926255+0000","last_unstale":"2024-09-21T10:11:20.391084+0000","last_undegraded":"2024-09-21T10:11:20.391084+0000","last_fullsized":"2024-09-21T10:11:20.391084+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:49.855878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"273'7543","reported_seq":7572,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:31.714846+0000","last_change":"2024-09-21T10:09:43.231158+0000","last_active":"2024-09-21T10:11:31.714846+0000","last_peered":"2024-09-21T10:11:31.714846+0000","last_clean":"2024-09-21T10:11:31.714846+0000","last_became_active":"2024-09-21T10:09:43.230838+0000","last_became_peered":"2024-09-21T10:09:43.230838+0000","last_unstale":"2024-09-21T10:11:31.714846+0000","last_undegraded":"2024-09-21T10:11:31.714846+0000","last_fullsized":"2024-09-21T10:11:31.714846+0000","mapping_epoch":252,"log_start":"257'1200","ondisk_log_start":"257'1200","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6343,"log_dups_size":0,"ondisk_log_size":6343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:15.318633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438239232,"num_objects":6687,"num_object_clones":0,"num_object_copies":20061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7543,"num_write_kb":455360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":205,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.390664+0000","last_change":"2024-09-21T10:10:57.390804+0000","last_active":"2024-09-21T10:11:20.390664+0000","last_peered":"2024-09-21T10:11:20.390664+0000","last_clean":"2024-09-21T10:11:20.390664+0000","last_became_active":"2024-09-21T10:10:57.390323+0000","last_became_peered":"2024-09-21T10:10:57.390323+0000","last_unstale":"2024-09-21T10:11:20.390664+0000","last_undegraded":"2024-09-21T10:11:20.390664+0000","last_fullsized":"2024-09-21T10:11:20.390664+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.005477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.005477+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:29:57.702046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":50,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.386211+0000","last_change":"2024-09-21T10:09:49.930280+0000","last_active":"2024-09-21T10:11:20.386211+0000","last_peered":"2024-09-21T10:11:20.386211+0000","last_clean":"2024-09-21T10:11:20.386211+0000","last_became_active":"2024-09-21T10:09:49.919101+0000","last_became_peered":"2024-09-21T10:09:49.919101+0000","last_unstale":"2024-09-21T10:11:20.386211+0000","last_undegraded":"2024-09-21T10:11:20.386211+0000","last_fullsized":"2024-09-21T10:11:20.386211+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:23.998475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":287,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.388395+0000","last_change":"2024-09-21T10:03:38.392767+0000","last_active":"2024-09-21T10:11:20.388395+0000","last_peered":"2024-09-21T10:11:20.388395+0000","last_clean":"2024-09-21T10:11:20.388395+0000","last_became_active":"2024-09-21T10:03:38.392424+0000","last_became_peered":"2024-09-21T10:03:38.392424+0000","last_unstale":"2024-09-21T10:11:20.388395+0000","last_undegraded":"2024-09-21T10:11:20.388395+0000","last_fullsized":"2024-09-21T10:11:20.388395+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:25.358291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":68,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.370889+0000","last_change":"2024-09-21T10:09:24.836360+0000","last_active":"2024-09-21T10:11:20.370889+0000","last_peered":"2024-09-21T10:11:20.370889+0000","last_clean":"2024-09-21T10:11:20.370889+0000","last_became_active":"2024-09-21T10:09:24.836033+0000","last_became_peered":"2024-09-21T10:09:24.836033+0000","last_unstale":"2024-09-21T10:11:20.370889+0000","last_undegraded":"2024-09-21T10:11:20.370889+0000","last_fullsized":"2024-09-21T10:11:20.370889+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:58.368582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":125,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.388105+0000","last_change":"2024-09-21T10:08:38.380189+0000","last_active":"2024-09-21T10:11:20.388105+0000","last_peered":"2024-09-21T10:11:20.388105+0000","last_clean":"2024-09-21T10:11:20.388105+0000","last_became_active":"2024-09-21T10:08:38.379891+0000","last_became_peered":"2024-09-21T10:08:38.379891+0000","last_unstale":"2024-09-21T10:11:20.388105+0000","last_undegraded":"2024-09-21T10:11:20.388105+0000","last_fullsized":"2024-09-21T10:11:20.388105+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:35.820921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":204,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-21T10:11:14.331507+0000","last_change":"2024-09-21T10:08:38.392563+0000","last_active":"2024-09-21T10:11:14.331507+0000","last_peered":"2024-09-21T10:11:14.331507+0000","last_clean":"2024-09-21T10:11:14.331507+0000","last_became_active":"2024-09-21T10:08:38.378735+0000","last_became_peered":"2024-09-21T10:08:38.378735+0000","last_unstale":"2024-09-21T10:11:14.331507+0000","last_undegraded":"2024-09-21T10:11:14.331507+0000","last_fullsized":"2024-09-21T10:11:14.331507+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:29:41.423208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.388691+0000","last_change":"2024-09-21T10:11:05.453286+0000","last_active":"2024-09-21T10:11:20.388691+0000","last_peered":"2024-09-21T10:11:20.388691+0000","last_clean":"2024-09-21T10:11:20.388691+0000","last_became_active":"2024-09-21T10:11:05.452601+0000","last_became_peered":"2024-09-21T10:11:05.452601+0000","last_unstale":"2024-09-21T10:11:20.388691+0000","last_undegraded":"2024-09-21T10:11:20.388691+0000","last_fullsized":"2024-09-21T10:11:20.388691+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:27.532091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":44,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-21T10:11:32.436203+0000","last_change":"2024-09-21T10:10:27.833762+0000","last_active":"2024-09-21T10:11:32.436203+0000","last_peered":"2024-09-21T10:11:32.436203+0000","last_clean":"2024-09-21T10:11:32.436203+0000","last_became_active":"2024-09-21T10:10:27.819999+0000","last_became_peered":"2024-09-21T10:10:27.819999+0000","last_unstale":"2024-09-21T10:11:32.436203+0000","last_undegraded":"2024-09-21T10:11:32.436203+0000","last_fullsized":"2024-09-21T10:11:32.436203+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:33.839463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'100","reported_seq":199,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-21T10:11:32.435398+0000","last_change":"2024-09-21T10:07:24.345428+0000","last_active":"2024-09-21T10:11:32.435398+0000","last_peered":"2024-09-21T10:11:32.435398+0000","last_clean":"2024-09-21T10:11:32.435398+0000","last_became_active":"2024-09-21T10:07:24.343529+0000","last_became_peered":"2024-09-21T10:07:24.343529+0000","last_unstale":"2024-09-21T10:11:32.435398+0000","last_undegraded":"2024-09-21T10:11:32.435398+0000","last_fullsized":"2024-09-21T10:11:32.435398+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.014993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.014993+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:35.677888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.371157+0000","last_change":"2024-09-21T10:10:57.394885+0000","last_active":"2024-09-21T10:11:20.371157+0000","last_peered":"2024-09-21T10:11:20.371157+0000","last_clean":"2024-09-21T10:11:20.371157+0000","last_became_active":"2024-09-21T10:10:57.394584+0000","last_became_peered":"2024-09-21T10:10:57.394584+0000","last_unstale":"2024-09-21T10:11:20.371157+0000","last_undegraded":"2024-09-21T10:11:20.371157+0000","last_fullsized":"2024-09-21T10:11:20.371157+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:52:16.853713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":389,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.405774+0000","last_change":"2024-09-21T10:03:38.425111+0000","last_active":"2024-09-21T10:11:20.405774+0000","last_peered":"2024-09-21T10:11:20.405774+0000","last_clean":"2024-09-21T10:11:20.405774+0000","last_became_active":"2024-09-21T10:03:38.424704+0000","last_became_peered":"2024-09-21T10:03:38.424704+0000","last_unstale":"2024-09-21T10:11:20.405774+0000","last_undegraded":"2024-09-21T10:11:20.405774+0000","last_fullsized":"2024-09-21T10:11:20.405774+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:35.157076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":290,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.383177+0000","last_change":"2024-09-21T10:10:57.397755+0000","last_active":"2024-09-21T10:11:20.383177+0000","last_peered":"2024-09-21T10:11:20.383177+0000","last_clean":"2024-09-21T10:11:20.383177+0000","last_became_active":"2024-09-21T10:10:57.397249+0000","last_became_peered":"2024-09-21T10:10:57.397249+0000","last_unstale":"2024-09-21T10:11:20.383177+0000","last_undegraded":"2024-09-21T10:11:20.383177+0000","last_fullsized":"2024-09-21T10:11:20.383177+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:40.813503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":374,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.390979+0000","last_change":"2024-09-21T09:56:12.576756+0000","last_active":"2024-09-21T10:11:20.390979+0000","last_peered":"2024-09-21T10:11:20.390979+0000","last_clean":"2024-09-21T10:11:20.390979+0000","last_became_active":"2024-09-21T09:56:12.575752+0000","last_became_peered":"2024-09-21T09:56:12.575752+0000","last_unstale":"2024-09-21T10:11:20.390979+0000","last_undegraded":"2024-09-21T10:11:20.390979+0000","last_fullsized":"2024-09-21T10:11:20.390979+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:55.224839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":237,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.388330+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T10:11:20.388330+0000","last_peered":"2024-09-21T10:11:20.388330+0000","last_clean":"2024-09-21T10:11:20.388330+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T10:11:20.388330+0000","last_undegraded":"2024-09-21T10:11:20.388330+0000","last_fullsized":"2024-09-21T10:11:20.388330+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":410,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.391178+0000","last_change":"2024-09-21T10:09:49.928053+0000","last_active":"2024-09-21T10:11:20.391178+0000","last_peered":"2024-09-21T10:11:20.391178+0000","last_clean":"2024-09-21T10:11:20.391178+0000","last_became_active":"2024-09-21T10:09:49.927231+0000","last_became_peered":"2024-09-21T10:09:49.927231+0000","last_unstale":"2024-09-21T10:11:20.391178+0000","last_undegraded":"2024-09-21T10:11:20.391178+0000","last_fullsized":"2024-09-21T10:11:20.391178+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:26.987553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":323,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.386527+0000","last_change":"2024-09-21T10:09:23.840717+0000","last_active":"2024-09-21T10:11:20.386527+0000","last_peered":"2024-09-21T10:11:20.386527+0000","last_clean":"2024-09-21T10:11:20.386527+0000","last_became_active":"2024-09-21T10:09:23.840323+0000","last_became_peered":"2024-09-21T10:09:23.840323+0000","last_unstale":"2024-09-21T10:11:20.386527+0000","last_undegraded":"2024-09-21T10:11:20.386527+0000","last_fullsized":"2024-09-21T10:11:20.386527+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:55.769770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":38,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-21T10:11:32.434924+0000","last_change":"2024-09-21T10:10:27.816526+0000","last_active":"2024-09-21T10:11:32.434924+0000","last_peered":"2024-09-21T10:11:32.434924+0000","last_clean":"2024-09-21T10:11:32.434924+0000","last_became_active":"2024-09-21T10:10:27.816221+0000","last_became_peered":"2024-09-21T10:10:27.816221+0000","last_unstale":"2024-09-21T10:11:32.434924+0000","last_undegraded":"2024-09-21T10:11:32.434924+0000","last_fullsized":"2024-09-21T10:11:32.434924+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:08.331228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":381,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.391155+0000","last_change":"2024-09-21T10:08:38.379994+0000","last_active":"2024-09-21T10:11:20.391155+0000","last_peered":"2024-09-21T10:11:20.391155+0000","last_clean":"2024-09-21T10:11:20.391155+0000","last_became_active":"2024-09-21T10:08:38.378729+0000","last_became_peered":"2024-09-21T10:08:38.378729+0000","last_unstale":"2024-09-21T10:11:20.391155+0000","last_undegraded":"2024-09-21T10:11:20.391155+0000","last_fullsized":"2024-09-21T10:11:20.391155+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:16.940173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":103,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-21T10:11:32.436693+0000","last_change":"2024-09-21T10:09:24.844561+0000","last_active":"2024-09-21T10:11:32.436693+0000","last_peered":"2024-09-21T10:11:32.436693+0000","last_clean":"2024-09-21T10:11:32.436693+0000","last_became_active":"2024-09-21T10:09:24.844190+0000","last_became_peered":"2024-09-21T10:09:24.844190+0000","last_unstale":"2024-09-21T10:11:32.436693+0000","last_undegraded":"2024-09-21T10:11:32.436693+0000","last_fullsized":"2024-09-21T10:11:32.436693+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:46:04.775344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":430,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-21T10:11:32.435669+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:11:32.435669+0000","last_peered":"2024-09-21T10:11:32.435669+0000","last_clean":"2024-09-21T10:11:32.435669+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:11:32.435669+0000","last_undegraded":"2024-09-21T10:11:32.435669+0000","last_fullsized":"2024-09-21T10:11:32.435669+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":46,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.382618+0000","last_change":"2024-09-21T10:09:49.918246+0000","last_active":"2024-09-21T10:11:20.382618+0000","last_peered":"2024-09-21T10:11:20.382618+0000","last_clean":"2024-09-21T10:11:20.382618+0000","last_became_active":"2024-09-21T10:09:49.917911+0000","last_became_peered":"2024-09-21T10:09:49.917911+0000","last_unstale":"2024-09-21T10:11:20.382618+0000","last_undegraded":"2024-09-21T10:11:20.382618+0000","last_fullsized":"2024-09-21T10:11:20.382618+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:30:10.567650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":236,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.390788+0000","last_change":"2024-09-21T10:07:24.347937+0000","last_active":"2024-09-21T10:11:20.390788+0000","last_peered":"2024-09-21T10:11:20.390788+0000","last_clean":"2024-09-21T10:11:20.390788+0000","last_became_active":"2024-09-21T10:07:24.347376+0000","last_became_peered":"2024-09-21T10:07:24.347376+0000","last_unstale":"2024-09-21T10:11:20.390788+0000","last_undegraded":"2024-09-21T10:11:20.390788+0000","last_fullsized":"2024-09-21T10:11:20.390788+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:48:05.644824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":56,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.387967+0000","last_change":"2024-09-21T10:08:38.382518+0000","last_active":"2024-09-21T10:11:20.387967+0000","last_peered":"2024-09-21T10:11:20.387967+0000","last_clean":"2024-09-21T10:11:20.387967+0000","last_became_active":"2024-09-21T10:08:38.372051+0000","last_became_peered":"2024-09-21T10:08:38.372051+0000","last_unstale":"2024-09-21T10:11:20.387967+0000","last_undegraded":"2024-09-21T10:11:20.387967+0000","last_fullsized":"2024-09-21T10:11:20.387967+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:21.360435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":327,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.395535+0000","last_change":"2024-09-21T10:09:24.840173+0000","last_active":"2024-09-21T10:11:20.395535+0000","last_peered":"2024-09-21T10:11:20.395535+0000","last_clean":"2024-09-21T10:11:20.395535+0000","last_became_active":"2024-09-21T10:09:24.839762+0000","last_became_peered":"2024-09-21T10:09:24.839762+0000","last_unstale":"2024-09-21T10:11:20.395535+0000","last_undegraded":"2024-09-21T10:11:20.395535+0000","last_fullsized":"2024-09-21T10:11:20.395535+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:12:53.465203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":70,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.388635+0000","last_change":"2024-09-21T10:08:38.388561+0000","last_active":"2024-09-21T10:11:20.388635+0000","last_peered":"2024-09-21T10:11:20.388635+0000","last_clean":"2024-09-21T10:11:20.388635+0000","last_became_active":"2024-09-21T10:08:38.374558+0000","last_became_peered":"2024-09-21T10:08:38.374558+0000","last_unstale":"2024-09-21T10:11:20.388635+0000","last_undegraded":"2024-09-21T10:11:20.388635+0000","last_fullsized":"2024-09-21T10:11:20.388635+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:24.111438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":411,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.406530+0000","last_change":"2024-09-21T10:09:23.840467+0000","last_active":"2024-09-21T10:11:20.406530+0000","last_peered":"2024-09-21T10:11:20.406530+0000","last_clean":"2024-09-21T10:11:20.406530+0000","last_became_active":"2024-09-21T10:09:23.840177+0000","last_became_peered":"2024-09-21T10:09:23.840177+0000","last_unstale":"2024-09-21T10:11:20.406530+0000","last_undegraded":"2024-09-21T10:11:20.406530+0000","last_fullsized":"2024-09-21T10:11:20.406530+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:17.887307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":400,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.391676+0000","last_change":"2024-09-21T10:11:05.451613+0000","last_active":"2024-09-21T10:11:20.391676+0000","last_peered":"2024-09-21T10:11:20.391676+0000","last_clean":"2024-09-21T10:11:20.391676+0000","last_became_active":"2024-09-21T10:11:05.451075+0000","last_became_peered":"2024-09-21T10:11:05.451075+0000","last_unstale":"2024-09-21T10:11:20.391676+0000","last_undegraded":"2024-09-21T10:11:20.391676+0000","last_fullsized":"2024-09-21T10:11:20.391676+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:18.505045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":106,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.387717+0000","last_change":"2024-09-21T10:08:38.380508+0000","last_active":"2024-09-21T10:11:20.387717+0000","last_peered":"2024-09-21T10:11:20.387717+0000","last_clean":"2024-09-21T10:11:20.387717+0000","last_became_active":"2024-09-21T10:08:38.380226+0000","last_became_peered":"2024-09-21T10:08:38.380226+0000","last_unstale":"2024-09-21T10:11:20.387717+0000","last_undegraded":"2024-09-21T10:11:20.387717+0000","last_fullsized":"2024-09-21T10:11:20.387717+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:50:38.190118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":65,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-21T10:11:20.382993+0000","last_change":"2024-09-21T10:10:57.401588+0000","last_active":"2024-09-21T10:11:20.382993+0000","last_peered":"2024-09-21T10:11:20.382993+0000","last_clean":"2024-09-21T10:11:20.382993+0000","last_became_active":"2024-09-21T10:10:57.397724+0000","last_became_peered":"2024-09-21T10:10:57.397724+0000","last_unstale":"2024-09-21T10:11:20.382993+0000","last_undegraded":"2024-09-21T10:11:20.382993+0000","last_fullsized":"2024-09-21T10:11:20.382993+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:47.843630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":6894911512,"num_objects":105209,"num_object_clones":0,"num_object_copies":315627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":24203,"num_objects_unfound":0,"num_objects_dirty":105209,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119761,"num_write_kb":7198977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1122,"num_bytes_recovered":73531392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20638150656,"data_stored":20638138440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101561,"ondisk_log_size":101561,"up":48,"acting":47,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":11710,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26580,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":688762,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":49,"num_read_kb":125,"num_write":25,"num_write_kb":271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":722165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626830,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4250324,"kb_used_data":3358284,"kb_used_omap":1,"kb_used_meta":892030,"kb_avail":89494828,"statfs":{"total":95995035648,"available":91642703872,"internally_reserved":0,"allocated":3438882816,"data_stored":3437345344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":913439064},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019288,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":245,"seq":1052266987547,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3553860,"kb_used_data":3404220,"kb_used_omap":8,"kb_used_meta":149623,"kb_avail":90191292,"statfs":{"total":95995035648,"available":92355883008,"internally_reserved":0,"allocated":3485921280,"data_stored":3484396011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8940,"internal_metadata":153214228},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":175,"seq":751619276896,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3740432,"kb_used_data":3401672,"kb_used_omap":12,"kb_used_meta":338739,"kb_avail":90004720,"statfs":{"total":95995035648,"available":92164833280,"internally_reserved":0,"allocated":3483312128,"data_stored":3481778731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12845,"internal_metadata":346869203},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823059,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1041300,"kb_used_data":132040,"kb_used_omap":1,"kb_used_meta":909246,"kb_avail":92703852,"statfs":{"total":95995035648,"available":94928744448,"internally_reserved":0,"allocated":135208960,"data_stored":133679816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":931068248},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879482,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4124480,"kb_used_data":3339320,"kb_used_omap":4,"kb_used_meta":785147,"kb_avail":89620672,"statfs":{"total":95995035648,"available":91771568128,"internally_reserved":0,"allocated":3419463680,"data_stored":3417942451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":803990972},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566323,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3175848,"kb_used_data":2499808,"kb_used_omap":6,"kb_used_meta":675833,"kb_avail":90569304,"statfs":{"total":95995035648,"available":92742967296,"internally_reserved":0,"allocated":2559803392,"data_stored":2558255133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":692053415},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860668,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4947216,"kb_used_data":4131528,"kb_used_omap":5,"kb_used_meta":815674,"kb_avail":88797936,"statfs":{"total":95995035648,"available":90929086464,"internally_reserved":0,"allocated":4230684672,"data_stored":4229154717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":835250415},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7236,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9976,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3463970816,"data_stored":3463970816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4207349760,"data_stored":4207345688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2555645952,"data_stored":2555641880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3396665344,"data_stored":3396665344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":131268608,"data_stored":131268608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3420524544,"data_stored":3420520472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3462725632,"data_stored":3462725632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:11:39.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T10:11:39.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:39.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:39.657 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:39.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:39.758 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:39.784 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:39.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:39.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:39.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:39.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:39.936 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:39.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:40.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:40.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:40.099 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:40.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:40.200 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:40.227 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:40.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:40.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:40.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:40.373 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:40.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:40.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:40.475 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:40.514 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:40.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:40.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:40.616 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:40.644 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:40.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:40.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:40.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:40 smithi129 ceph-mon[88471]: pgmap v958: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 888 op/s; 23143/302274 objects misplaced (7.656%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:11:40.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:11:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:11:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:11:40.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:40.784 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:40.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:40 smithi107 ceph-mon[102221]: pgmap v958: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 888 op/s; 23143/302274 objects misplaced (7.656%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:11:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:40 smithi107 ceph-mon[105022]: pgmap v958: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 888 op/s; 23143/302274 objects misplaced (7.656%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:11:40.887 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:40.925 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:40.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:40.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:41.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:41.063 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:41.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:41.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:41.165 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:41.192 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:41.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:41.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:41.294 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:41.328 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:41.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:41.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:41.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:41.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:41.471 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:41.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:41.476 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:11:41.504 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:41 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4149846234' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:11:41.505 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:41.505 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:41 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4149846234' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:11:41.505 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:41.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:41.614 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:41.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:41.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:41.647 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:11:41.648 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:11:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:41 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4149846234' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:11:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:41.688 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:11:41.688 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:11:41.715 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:41.724 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:11:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:11:41.742 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:41.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:41.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:41.758 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:11:41.758 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:11:41.795 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:11:41.795 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:11:41.826 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:11:41.826 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:11:41.844 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:41.857 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:11:41.857 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:11:41.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:41.876 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:41.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:41.888 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:11:41.888 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:11:41.977 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:42.013 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:42.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:42.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:42.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:11:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:11:41] "GET /metrics HTTP/1.1" 200 36812 "" "Prometheus/2.51.0" 2024-09-21T10:11:42.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:42.158 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:42.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:42.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:42.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:42.300 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:42.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:42.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:42.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:42.449 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:42.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:42.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:42.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:42.583 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:42.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:42.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:42.685 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:42.720 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:42.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:42.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:42.822 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:42.849 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:42.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:42.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:42.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:42 smithi107 ceph-mon[102221]: pgmap v959: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 888 op/s; 23143/302274 objects misplaced (7.656%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:11:42.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:42 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:11:42.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:42 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4149846234' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:11:42.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:42 smithi107 ceph-mon[102221]: osdmap e275: 8 total, 8 up, 7 in 2024-09-21T10:11:42.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:42 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2697679411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:42.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:42 smithi107 ceph-mon[105022]: pgmap v959: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 888 op/s; 23143/302274 objects misplaced (7.656%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:11:42.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:42 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:11:42.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:42 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4149846234' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:11:42.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:42 smithi107 ceph-mon[105022]: osdmap e275: 8 total, 8 up, 7 in 2024-09-21T10:11:42.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:42 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2697679411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:42 smithi129 ceph-mon[88471]: pgmap v959: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 888 op/s; 23143/302274 objects misplaced (7.656%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:11:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:42 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:11:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:42 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4149846234' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:11:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:42 smithi129 ceph-mon[88471]: osdmap e275: 8 total, 8 up, 7 in 2024-09-21T10:11:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:42 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2697679411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:42.954 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:42.981 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:42.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:42.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:43.082 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:43.116 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:43.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:43.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:43.218 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:43.249 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:43.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:43.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:43.351 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:43.381 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:43.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:43.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:43.483 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:43.512 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:43.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:43.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:43.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:43.649 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:43.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:43.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:43.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:43.789 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:43.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:43.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:43.890 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:43.917 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:43.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:43.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:44.018 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:44.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:44.064 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:44.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:44.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:44.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:44.202 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:44.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:44.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:44.304 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:44.334 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:44.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:44.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:44.436 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:44.474 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:44.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:44.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:44.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:44.597 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:44 smithi107 ceph-mon[102221]: pgmap v961: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 906 op/s; 22088/291201 objects misplaced (7.585%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T10:11:44.597 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:44 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:11:44.598 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:44 smithi107 ceph-mon[105022]: pgmap v961: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 906 op/s; 22088/291201 objects misplaced (7.585%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T10:11:44.598 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:44 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:11:44.609 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:44.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:44.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:44.710 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:44.760 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:44.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:44.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:44.863 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:44.877 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:11:44.877 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:11:44.878 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-09-21T10:11:44.878 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-09-21T10:11:44.878 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:11:44.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:44.906 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:44.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:44 smithi129 ceph-mon[88471]: pgmap v961: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 906 op/s; 22088/291201 objects misplaced (7.585%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T10:11:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:44 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:11:44.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:11:45.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:45.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:45.060 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:45.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:45.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:45.190 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:45.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:45.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:45.292 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:45.321 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:45.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:45.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:45.422 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:45.460 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:45.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:45.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:45.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:45.597 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:45.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:45.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:45.699 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:45.731 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:45.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:45.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:45.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:45.844 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:11:45.875 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:45.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:45.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:45.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:46.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:11:46.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:46.056 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:46.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:46.157 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:46.187 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:46.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:46.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:46.289 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:46.336 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:46.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:46.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:46.438 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:46.472 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:46.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:46.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:46.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:46.606 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:46.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:46.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:46.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:46 smithi129 ceph-mon[88471]: pgmap v962: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 905 op/s; 22088/288534 objects misplaced (7.655%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T10:11:46.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:46.746 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:46.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:46.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:46.848 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:46.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:46 smithi107 ceph-mon[102221]: pgmap v962: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 905 op/s; 22088/288534 objects misplaced (7.655%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T10:11:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:46 smithi107 ceph-mon[105022]: pgmap v962: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 905 op/s; 22088/288534 objects misplaced (7.655%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T10:11:46.878 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:46.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:46.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:46.980 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:47.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:47.026 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:47.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:47.127 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:47.158 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:47.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:47.259 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:47.289 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:47.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:47.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:47.428 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:47.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:47.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:47.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:47.565 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:47.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:47.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:47.668 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:47.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:47.705 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:47.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:47.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:47.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:47.849 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:47.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:47.950 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:47.979 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:47.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:47.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:48.081 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:48.117 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:48.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:48.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:48.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:48.260 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:48.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:48.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:48.362 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:48.401 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:48.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:48.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:48.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:48.533 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:48.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:48.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:48.635 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:48.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:48 smithi129 ceph-mon[88471]: pgmap v963: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 917 op/s; 21022/279105 objects misplaced (7.532%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:11:48.669 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:48.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:48.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:48.772 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:48.799 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:48.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:48.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:48.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:48 smithi107 ceph-mon[102221]: pgmap v963: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 917 op/s; 21022/279105 objects misplaced (7.532%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:11:48.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:48 smithi107 ceph-mon[105022]: pgmap v963: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 917 op/s; 21022/279105 objects misplaced (7.532%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:11:48.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:48.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:48.943 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:48.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:49.044 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:49.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:49.079 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:49.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:49.180 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:49.202 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:49.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:49.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:49.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:49.341 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:49.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:49.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:49.443 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:49.476 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:49.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:49.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:49.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:49.608 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:49.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:49.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:49.710 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:49.737 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:49.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:49.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:49.838 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:49.868 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:49.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:49.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:49.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:50.017 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:50.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:50.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:50.119 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:50.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:50.158 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:50.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:50.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:50.281 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:50.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:50.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:50.383 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:50.406 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:50.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:50.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:50.496 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:11:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:11:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:11:50.509 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:50.541 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:50.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:50.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:50.642 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:50.668 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:50.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:50.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:50.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:50.804 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:50.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:50.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:50.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:50 smithi107 ceph-mon[102221]: pgmap v964: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 911 op/s; 21022/274917 objects misplaced (7.647%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:11:50.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:50 smithi107 ceph-mon[105022]: pgmap v964: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 911 op/s; 21022/274917 objects misplaced (7.647%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:11:50.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:50.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:50 smithi129 ceph-mon[88471]: pgmap v964: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 911 op/s; 21022/274917 objects misplaced (7.647%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:11:50.942 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:50.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:50.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:51.044 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:51.080 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:51.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:51.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:51.181 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:51.207 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:51.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:51.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:51.309 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:51.338 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:51.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:51.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:51.440 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:51.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:51.470 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:51.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:51.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:51.618 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:51.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:51.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:51.649 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:51.719 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:51.743 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:51.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:51.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:51.845 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:51.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:51 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:11:51.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:51 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:11:51.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:11:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:11:51] "GET /metrics HTTP/1.1" 200 36812 "" "Prometheus/2.51.0" 2024-09-21T10:11:51.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:51 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:11:51.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:51 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:11:51.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:51.884 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:51.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:51.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:51.994 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:52.044 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:52.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:52.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:52.146 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:52.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:51 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:11:52.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:51 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:11:52.173 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:52.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:52.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:52.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:52.324 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:52.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:52.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:52.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:52.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:52.462 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:52.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:52.563 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:52.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:52.594 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:52.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:52.695 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:52.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:52.762 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:52.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:52.863 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:52.921 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:52.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:52.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:53.023 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:53.050 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:53.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:53.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:53.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:52 smithi107 ceph-mon[102221]: pgmap v965: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 911 op/s; 21022/274917 objects misplaced (7.647%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:11:53.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:52 smithi107 ceph-mon[105022]: pgmap v965: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 911 op/s; 21022/274917 objects misplaced (7.647%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:11:53.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:53.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:52 smithi129 ceph-mon[88471]: pgmap v965: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 911 op/s; 21022/274917 objects misplaced (7.647%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:11:53.195 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:53.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:53.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:53.297 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:53.322 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:53.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:53.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:53.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:53.469 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:53.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:53.536 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:11:53.536 INFO:teuthology.orchestra.run.smithi107.stdout:{"fsid":"86c49514-77fc-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1471,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":275,"num_osds":8,"num_up_osds":8,"osd_up_since":1726913362,"num_in_osds":7,"osd_in_since":1726913317,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":105},{"state_name":"active+remapped+backfill_wait","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":109,"num_pools":5,"num_objects":91639,"data_bytes":6004512831,"bytes_used":23165894656,"bytes_avail":648799354880,"bytes_total":671965249536,"misplaced_objects":21022,"misplaced_total":274917,"misplaced_ratio":0.076466715408650604,"recovering_objects_per_sec":41,"recovering_bytes_per_sec":2725408,"recovering_keys_per_sec":0,"num_objects_recovered":416,"num_bytes_recovered":27262976,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":911},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi107.sedied","status":"up:active","gid":24557}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.107:8443/","prometheus":"http://172.21.15.107:9283/"}},"servicemap":{"epoch":30,"modified":"2024-09-21T10:10:23.246058+0000","services":{}},"progress_events":{"24983b40-6a19-4e8e-8a8a-55bfd2201591":{"message":"Global Recovery Event (0s)\n [............................] ","progress":0,"add_to_ceph_s":true},"2fcfd131-0268-49fe-9638-9f737ef274d3":{"message":"Upgrade to 19.3.0-5092-g62da54b4 (9m)\n [============................] (remaining: 11m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-09-21T10:11:53.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:53.612 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:53.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:53.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:53.713 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:53.716 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:11:53.736 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:53.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:53.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:53.838 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:53.870 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:53.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:53.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:53.971 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:54.002 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:54.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:54.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:54.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:54.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:53 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/678573738' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:11:54.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:53 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/678573738' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:11:54.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:54.139 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:54.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:54.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:53 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/678573738' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:11:54.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:54.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:54.295 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:54.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:54.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:54.442 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:54.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:54.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:54.544 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:54.570 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:54.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:54.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:54.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:54.676 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-21T10:11:54.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:11:54.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:54.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:54.826 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:54.856 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:54.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:54.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:54.943 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:54 smithi107 ceph-mon[102221]: pgmap v966: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s; 19928/264009 objects misplaced (7.548%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T10:11:54.943 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:54 smithi107 ceph-mon[105022]: pgmap v966: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s; 19928/264009 objects misplaced (7.548%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T10:11:54.957 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:54.987 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:54.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:54.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:55.089 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:55.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:55.136 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:55.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:54 smithi129 ceph-mon[88471]: pgmap v966: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s; 19928/264009 objects misplaced (7.548%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T10:11:55.237 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:55.265 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:55.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:55.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:55.367 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:55.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:55.418 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:55.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:55.519 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:55.561 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:55.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:55.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:55.570 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:11:55.570 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:11:55.570 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:11:55.570 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:11:55.595 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:11:55.595 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:11:55.632 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:11:55.633 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:11:55.661 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:11:55.662 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:11:55.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:55.693 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:11:55.693 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:11:55.709 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:55.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:55.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:55.724 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:11:55.724 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:11:55.757 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:11:55.757 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:11:55.812 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:55.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:55.848 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:55.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:55 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1908857384' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:55.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:55 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1908857384' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:55.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:55.988 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:55.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:55.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:56.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:56.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:56.138 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:56.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:56.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:55 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1908857384' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:11:56.242 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:56.271 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:56.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:56.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:56.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:56.409 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:56.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:56.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:56.510 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:56.546 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:56.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:56.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:56.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:56.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:56.691 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:56.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:56.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:56.836 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:56.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:56.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:56.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:56.972 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:56.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:56.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:57.074 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:57.089 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:56 smithi107 ceph-mon[102221]: pgmap v967: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 824 op/s; 19928/261516 objects misplaced (7.620%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:11:57.090 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:57.090 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:56 smithi107 ceph-mon[105022]: pgmap v967: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 824 op/s; 19928/261516 objects misplaced (7.620%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:11:57.090 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:57.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:57.111 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:57.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:56 smithi129 ceph-mon[88471]: pgmap v967: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 824 op/s; 19928/261516 objects misplaced (7.620%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:11:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:57.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:57.253 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:57.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:57.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:57.355 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:57.389 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:57.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:57.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:57.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:57.525 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:57.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:57.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:57.627 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:57.657 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:57.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:57.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:57.758 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:11:57.787 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:57.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:57.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:11:57.889 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:57.917 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:57.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:57.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:58.019 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:58.049 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:58.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:58.151 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:58.179 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:58.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:58.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:58.281 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:58.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:58.329 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:58.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:58.374 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:11:58.374 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:11:58.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:58.463 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:58.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:58.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:58.565 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:58.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:58.598 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:58.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:58.699 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:58.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:58.740 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:58.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:58.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:11:58.878 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:58.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:58.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:11:58.980 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:59.007 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:59.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:59.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:11:59.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:11:59.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:11:58 smithi107 ceph-mon[102221]: pgmap v968: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1k op/s; 18850/252522 objects misplaced (7.465%); 3.6 MiB/s, 56 objects/s recovering 2024-09-21T10:11:59.111 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:11:58 smithi107 ceph-mon[105022]: pgmap v968: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1k op/s; 18850/252522 objects misplaced (7.465%); 3.6 MiB/s, 56 objects/s recovering 2024-09-21T10:11:59.147 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:59.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:59.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:11:59.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:11:58 smithi129 ceph-mon[88471]: pgmap v968: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1k op/s; 18850/252522 objects misplaced (7.465%); 3.6 MiB/s, 56 objects/s recovering 2024-09-21T10:11:59.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:11:59.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:59.290 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:59.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:11:59.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:11:59.427 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:11:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:59.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:11:59.528 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:59.556 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:59.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:59.658 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:11:59.683 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:59.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:59.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:11:59.785 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:11:59.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:59.815 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:59.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:11:59.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:11:59.916 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:11:59.943 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:11:59.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:11:59.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:00.044 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:00.067 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:00.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:00.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:00.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:00.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:00.222 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:00.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:00.323 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:00.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:00.365 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:00.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:00.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:00.516 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:00.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:00.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:00.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:00.669 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:12:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:12:00.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:12:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:00.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:00.771 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:00.805 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:00.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:00.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:00.844 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:12:00.907 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:00.935 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:00.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:00.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:01.036 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:01.065 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:01.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:01.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:01.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:00 smithi107 ceph-mon[102221]: pgmap v969: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 848 op/s; 18850/248535 objects misplaced (7.584%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:12:01.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:00 smithi107 ceph-mon[102221]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-21T10:12:01.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:00 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3503420925' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:01.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:00 smithi107 ceph-mon[105022]: pgmap v969: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 848 op/s; 18850/248535 objects misplaced (7.584%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:12:01.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:00 smithi107 ceph-mon[105022]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-21T10:12:01.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:00 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3503420925' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:01.166 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:00 smithi129 ceph-mon[88471]: pgmap v969: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 848 op/s; 18850/248535 objects misplaced (7.584%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:12:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:00 smithi129 ceph-mon[88471]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-21T10:12:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:00 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3503420925' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:01.205 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:01.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:01.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:01.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:01.344 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:01.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:01.445 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:01.472 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:01.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:01.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:01.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:01.620 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:01.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:01.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:01.636 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:01.723 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:01.755 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:01.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:01.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:01.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:01.896 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:01.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:01.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:01.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:02.035 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:02.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:02.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:02.065 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:01 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:12:02.065 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:01 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3503420925' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:12:02.066 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:01 smithi107 ceph-mon[102221]: osdmap e276: 8 total, 8 up, 7 in 2024-09-21T10:12:02.066 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:12:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:12:01] "GET /metrics HTTP/1.1" 200 36809 "" "Prometheus/2.51.0" 2024-09-21T10:12:02.066 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:01 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:12:02.066 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:01 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3503420925' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:12:02.066 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:01 smithi107 ceph-mon[105022]: osdmap e276: 8 total, 8 up, 7 in 2024-09-21T10:12:02.084 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:02.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:01 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:12:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:01 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3503420925' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:12:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:01 smithi129 ceph-mon[88471]: osdmap e276: 8 total, 8 up, 7 in 2024-09-21T10:12:02.189 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:02.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:02.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:02.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:02.329 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:02.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:02.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:02.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:02.471 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:02.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:02.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:02.573 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:02.604 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:02.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:02.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:02.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:02.748 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:02.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:02.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:02.850 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:02.874 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:02.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:02.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:02.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:03.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:03.011 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:03.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:03.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:02 smithi107 ceph-mon[102221]: pgmap v971: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 878 op/s; 18850/248535 objects misplaced (7.584%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T10:12:03.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:02 smithi107 ceph-mon[105022]: pgmap v971: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 878 op/s; 18850/248535 objects misplaced (7.584%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T10:12:03.111 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:03.148 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:03.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:03.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:03.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:02 smithi129 ceph-mon[88471]: pgmap v971: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 878 op/s; 18850/248535 objects misplaced (7.584%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T10:12:03.249 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:03.279 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:03.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:03.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:03.380 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:03.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:12:03.407 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:03.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:03.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:03.510 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:03.557 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:03.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:03.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:03.659 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:03.686 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:03.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:03.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:03.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:03.821 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:03.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:03.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:03.923 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:03.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:03.962 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:03.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:04.062 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:04.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:04.093 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:04.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:04.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:12:04.193 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:04.223 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:04.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:04.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:04.325 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:04.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:04.352 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:04.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:04.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:04.495 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:04.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:04.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:04.597 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:04.627 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:04.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:04.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:04.730 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:04.765 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:04.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:04.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:04.867 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:04.903 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:04.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:04.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:05.005 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:05.034 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:05.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:05.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:05.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:04 smithi107 ceph-mon[102221]: pgmap v972: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 889 op/s; 17628/237303 objects misplaced (7.428%); 3.3 MiB/s, 53 objects/s recovering 2024-09-21T10:12:05.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:04 smithi107 ceph-mon[105022]: pgmap v972: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 889 op/s; 17628/237303 objects misplaced (7.428%); 3.3 MiB/s, 53 objects/s recovering 2024-09-21T10:12:05.137 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:05.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:04 smithi129 ceph-mon[88471]: pgmap v972: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 889 op/s; 17628/237303 objects misplaced (7.428%); 3.3 MiB/s, 53 objects/s recovering 2024-09-21T10:12:05.170 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:05.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:05.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:05.305 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:05.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:05.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:05.407 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:05.433 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:05.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:05.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:05.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:05.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:05.574 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:05.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:05.674 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:05.709 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:05.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:05.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:05.811 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:05.842 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:05.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:05.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:05.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:05.982 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:05.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:05.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:06.084 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:06.111 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:06.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:06.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:06.212 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:06.239 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:06.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:06.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:06.341 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:06.376 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:06.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:06.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:06.478 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:06.508 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:06.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:06.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:06.610 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:06.641 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:06.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:06.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:06.742 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:06.769 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:06.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:06.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:06.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:06.906 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:06.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:06.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:07.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:07.029 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:07.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:07.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:07.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:06 smithi107 ceph-mon[102221]: pgmap v973: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 892 op/s; 17628/234723 objects misplaced (7.510%); 3.3 MiB/s, 53 objects/s recovering 2024-09-21T10:12:07.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:06 smithi107 ceph-mon[105022]: pgmap v973: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 892 op/s; 17628/234723 objects misplaced (7.510%); 3.3 MiB/s, 53 objects/s recovering 2024-09-21T10:12:07.130 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:07.158 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:07.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:07.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:07.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:06 smithi129 ceph-mon[88471]: pgmap v973: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 892 op/s; 17628/234723 objects misplaced (7.510%); 3.3 MiB/s, 53 objects/s recovering 2024-09-21T10:12:07.260 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:07.289 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:07.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:07.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:07.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:07.428 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:07.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:07.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:07.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:07.565 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:07.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:07.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:07.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:07.715 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:07.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:07.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:07.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:07.859 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:07.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:07.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:07.961 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:07.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:07.997 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:07.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:08.098 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:08.129 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:08.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:08.231 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:08.255 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:08.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:08.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:08.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:08.392 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:08.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:08.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:08.494 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:08.519 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:08.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:08.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:08.620 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:08.650 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:08.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:08.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:08.751 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:08.776 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:08.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:08.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:08.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:08.916 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:08.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:08.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:09.018 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:09.020 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:08 smithi107 ceph-mon[102221]: pgmap v974: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 925 op/s; 16456/224754 objects misplaced (7.322%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T10:12:09.020 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:08 smithi107 ceph-mon[105022]: pgmap v974: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 925 op/s; 16456/224754 objects misplaced (7.322%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T10:12:09.044 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:09.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:09.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:09.145 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:09.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:08 smithi129 ceph-mon[88471]: pgmap v974: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 925 op/s; 16456/224754 objects misplaced (7.322%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T10:12:09.170 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:09.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:09.204 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:12:09.204 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:12:09.272 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:09.300 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:09.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:09.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:09.402 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:09.429 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:09.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:09.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:09.491 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:12:09.492 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:12:09.531 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:09.551 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:09.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:09.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:09.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:09.673 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:09.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:09.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:09.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:09.796 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:09.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:09.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:09.898 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:09.921 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:09.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:10.022 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:10.055 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:10.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:10.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:10.156 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:10.183 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:10.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:10.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:10.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:10.321 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:10.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:10.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:10.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:10.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:10.454 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:10.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:10.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:10.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:10.595 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:10.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:10.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:12:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:12:10.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:12:10.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:10.735 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:10.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:10.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:10.837 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:10.868 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:10.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:10.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:10.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:11.005 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:11.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:11.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:11.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:12:11.107 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:11.154 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:11.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:11.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:11.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:10 smithi129 ceph-mon[88471]: pgmap v975: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 929 op/s; 16456/220635 objects misplaced (7.458%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T10:12:11.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:10 smithi129 ceph-mon[88471]: from='client.46472 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:12:11.255 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:11.292 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:11.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:11.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:11.314 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:11.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:10 smithi107 ceph-mon[105022]: pgmap v975: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 929 op/s; 16456/220635 objects misplaced (7.458%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T10:12:11.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:10 smithi107 ceph-mon[105022]: from='client.46472 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:12:11.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:10 smithi107 ceph-mon[102221]: pgmap v975: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 929 op/s; 16456/220635 objects misplaced (7.458%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T10:12:11.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:10 smithi107 ceph-mon[102221]: from='client.46472 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:12:11.394 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:11.419 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:11.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:11.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:11.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:11.560 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:11.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:11.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:11.662 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:11.697 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:11.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:11.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:11.798 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:11.831 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:11.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:11.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:11.935 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:11.950 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":975,"stamp":"2024-09-21T10:12:09.286225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4818704447,"num_objects":73545,"num_object_clones":0,"num_object_copies":220635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16456,"num_objects_unfound":0,"num_objects_dirty":73545,"num_whiteouts":0,"num_read":50,"num_read_kb":126,"num_write":151541,"num_write_kb":7199280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2851,"num_bytes_recovered":183644851,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103411,"ondisk_log_size":103411,"up":327,"acting":326,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":331,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19411280,"kb_used_data":14482328,"kb_used_omap":41,"kb_used_meta":4928406,"kb_avail":636804784,"statfs":{"total":671965249536,"available":652088098816,"internally_reserved":0,"allocated":14829903872,"data_stored":14819038135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42519,"internal_metadata":5046688233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-609484800,"num_objects":-9300,"num_object_clones":0,"num_object_copies":-27900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2394,"num_objects_unfound":0,"num_objects_dirty":-9300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9300,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":557,"num_bytes_recovered":36503552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.009347"},"pg_stats":[{"pgid":"1.11","version":"77'100","reported_seq":67,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.866402+0000","last_change":"2024-09-21T10:10:57.397786+0000","last_active":"2024-09-21T10:12:00.866402+0000","last_peered":"2024-09-21T10:12:00.866402+0000","last_clean":"2024-09-21T10:12:00.866402+0000","last_became_active":"2024-09-21T10:10:57.396742+0000","last_became_peered":"2024-09-21T10:10:57.396742+0000","last_unstale":"2024-09-21T10:12:00.866402+0000","last_undegraded":"2024-09-21T10:12:00.866402+0000","last_fullsized":"2024-09-21T10:12:00.866402+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:53.631923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:53.631923+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:33:35.364595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":405,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.861501+0000","last_change":"2024-09-21T10:03:38.389577+0000","last_active":"2024-09-21T10:12:00.861501+0000","last_peered":"2024-09-21T10:12:00.861501+0000","last_clean":"2024-09-21T10:12:00.861501+0000","last_became_active":"2024-09-21T10:03:38.389292+0000","last_became_peered":"2024-09-21T10:03:38.389292+0000","last_unstale":"2024-09-21T10:12:00.861501+0000","last_undegraded":"2024-09-21T10:12:00.861501+0000","last_fullsized":"2024-09-21T10:12:00.861501+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:32.113472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":146,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.486932+0000","last_change":"2024-09-21T10:08:38.387871+0000","last_active":"2024-09-21T10:11:41.486932+0000","last_peered":"2024-09-21T10:11:41.486932+0000","last_clean":"2024-09-21T10:11:41.486932+0000","last_became_active":"2024-09-21T10:08:38.376727+0000","last_became_peered":"2024-09-21T10:08:38.376727+0000","last_unstale":"2024-09-21T10:11:41.486932+0000","last_undegraded":"2024-09-21T10:11:41.486932+0000","last_fullsized":"2024-09-21T10:11:41.486932+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:45:54.498950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'100","reported_seq":175,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.486585+0000","last_change":"2024-09-21T10:11:05.438031+0000","last_active":"2024-09-21T10:11:41.486585+0000","last_peered":"2024-09-21T10:11:41.486585+0000","last_clean":"2024-09-21T10:11:41.486585+0000","last_became_active":"2024-09-21T10:11:05.437637+0000","last_became_peered":"2024-09-21T10:11:05.437637+0000","last_unstale":"2024-09-21T10:11:41.486585+0000","last_undegraded":"2024-09-21T10:11:41.486585+0000","last_fullsized":"2024-09-21T10:11:41.486585+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.798577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.798577+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:42.343217+0000","scrub_duration":0.0023622640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":267,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.491799+0000","last_change":"2024-09-21T10:10:57.392782+0000","last_active":"2024-09-21T10:11:41.491799+0000","last_peered":"2024-09-21T10:11:41.491799+0000","last_clean":"2024-09-21T10:11:41.491799+0000","last_became_active":"2024-09-21T10:10:57.392305+0000","last_became_peered":"2024-09-21T10:10:57.392305+0000","last_unstale":"2024-09-21T10:11:41.491799+0000","last_undegraded":"2024-09-21T10:11:41.491799+0000","last_fullsized":"2024-09-21T10:11:41.491799+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:42:01.387622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":327,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.873546+0000","last_change":"2024-09-21T10:09:23.846703+0000","last_active":"2024-09-21T10:12:00.873546+0000","last_peered":"2024-09-21T10:12:00.873546+0000","last_clean":"2024-09-21T10:12:00.873546+0000","last_became_active":"2024-09-21T10:09:23.846415+0000","last_became_peered":"2024-09-21T10:09:23.846415+0000","last_unstale":"2024-09-21T10:12:00.873546+0000","last_undegraded":"2024-09-21T10:12:00.873546+0000","last_fullsized":"2024-09-21T10:12:00.873546+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:23.129285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'100","reported_seq":121,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.861957+0000","last_change":"2024-09-21T10:09:24.841480+0000","last_active":"2024-09-21T10:12:00.861957+0000","last_peered":"2024-09-21T10:12:00.861957+0000","last_clean":"2024-09-21T10:12:00.861957+0000","last_became_active":"2024-09-21T10:09:24.841145+0000","last_became_peered":"2024-09-21T10:09:24.841145+0000","last_unstale":"2024-09-21T10:12:00.861957+0000","last_undegraded":"2024-09-21T10:12:00.861957+0000","last_fullsized":"2024-09-21T10:12:00.861957+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.433227+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":23,"num_write":5,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":35780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":52,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.865412+0000","last_change":"2024-09-21T10:09:49.922489+0000","last_active":"2024-09-21T10:12:00.865412+0000","last_peered":"2024-09-21T10:12:00.865412+0000","last_clean":"2024-09-21T10:12:00.865412+0000","last_became_active":"2024-09-21T10:09:49.921629+0000","last_became_peered":"2024-09-21T10:09:49.921629+0000","last_unstale":"2024-09-21T10:12:00.865412+0000","last_undegraded":"2024-09-21T10:12:00.865412+0000","last_fullsized":"2024-09-21T10:12:00.865412+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:50.611979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":148,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.486446+0000","last_change":"2024-09-21T10:09:24.836647+0000","last_active":"2024-09-21T10:11:41.486446+0000","last_peered":"2024-09-21T10:11:41.486446+0000","last_clean":"2024-09-21T10:11:41.486446+0000","last_became_active":"2024-09-21T10:09:24.836278+0000","last_became_peered":"2024-09-21T10:09:24.836278+0000","last_unstale":"2024-09-21T10:11:41.486446+0000","last_undegraded":"2024-09-21T10:11:41.486446+0000","last_fullsized":"2024-09-21T10:11:41.486446+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:08:36.535892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"77'100","reported_seq":19,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.498660+0000","last_change":"2024-09-21T10:10:57.452531+0000","last_active":"2024-09-21T10:11:41.498660+0000","last_peered":"2024-09-21T10:11:41.498660+0000","last_clean":"2024-09-21T10:11:41.498660+0000","last_became_active":"2024-09-21T10:10:57.452150+0000","last_became_peered":"2024-09-21T10:10:57.452150+0000","last_unstale":"2024-09-21T10:11:41.498660+0000","last_undegraded":"2024-09-21T10:11:41.498660+0000","last_fullsized":"2024-09-21T10:11:41.498660+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:00.058077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:00.058077+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":405,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.861968+0000","last_change":"2024-09-21T10:03:38.381552+0000","last_active":"2024-09-21T10:12:00.861968+0000","last_peered":"2024-09-21T10:12:00.861968+0000","last_clean":"2024-09-21T10:12:00.861968+0000","last_became_active":"2024-09-21T10:03:38.380898+0000","last_became_peered":"2024-09-21T10:03:38.380898+0000","last_unstale":"2024-09-21T10:12:00.861968+0000","last_undegraded":"2024-09-21T10:12:00.861968+0000","last_fullsized":"2024-09-21T10:12:00.861968+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:53.409606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":335,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.491644+0000","last_change":"2024-09-21T10:10:57.459057+0000","last_active":"2024-09-21T10:11:41.491644+0000","last_peered":"2024-09-21T10:11:41.491644+0000","last_clean":"2024-09-21T10:11:41.491644+0000","last_became_active":"2024-09-21T10:10:57.451776+0000","last_became_peered":"2024-09-21T10:10:57.451776+0000","last_unstale":"2024-09-21T10:11:41.491644+0000","last_undegraded":"2024-09-21T10:11:41.491644+0000","last_fullsized":"2024-09-21T10:11:41.491644+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:00.138317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"276'9431","reported_seq":9536,"reported_epoch":276,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:12:06.902535+0000","last_change":"2024-09-21T10:10:58.805957+0000","last_active":"2024-09-21T10:12:06.902535+0000","last_peered":"2024-09-21T10:12:06.902535+0000","last_clean":"2024-09-21T10:10:56.364182+0000","last_became_active":"2024-09-21T10:10:58.682930+0000","last_became_peered":"2024-09-21T10:10:58.682930+0000","last_unstale":"2024-09-21T10:12:06.902535+0000","last_undegraded":"2024-09-21T10:12:06.902535+0000","last_fullsized":"2024-09-21T10:12:06.902535+0000","mapping_epoch":267,"log_start":"259'3100","ondisk_log_start":"259'3100","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6331,"log_dups_size":0,"ondisk_log_size":6331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298385408,"num_objects":4553,"num_object_clones":0,"num_object_copies":13659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4536,"num_objects_unfound":0,"num_objects_dirty":4553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9431,"num_write_kb":447488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[{"shards":"0,1,7","objects":4553}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"77'100","reported_seq":160,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.491736+0000","last_change":"2024-09-21T10:08:38.374573+0000","last_active":"2024-09-21T10:11:41.491736+0000","last_peered":"2024-09-21T10:11:41.491736+0000","last_clean":"2024-09-21T10:11:41.491736+0000","last_became_active":"2024-09-21T10:08:38.374075+0000","last_became_peered":"2024-09-21T10:08:38.374075+0000","last_unstale":"2024-09-21T10:11:41.491736+0000","last_undegraded":"2024-09-21T10:11:41.491736+0000","last_fullsized":"2024-09-21T10:11:41.491736+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:55.259385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:55.259385+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:28:26.784162+0000","scrub_duration":0.0015667260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":191,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.486993+0000","last_change":"2024-09-21T10:09:49.941392+0000","last_active":"2024-09-21T10:11:41.486993+0000","last_peered":"2024-09-21T10:11:41.486993+0000","last_clean":"2024-09-21T10:11:41.486993+0000","last_became_active":"2024-09-21T10:09:49.923535+0000","last_became_peered":"2024-09-21T10:09:49.923535+0000","last_unstale":"2024-09-21T10:11:41.486993+0000","last_undegraded":"2024-09-21T10:11:41.486993+0000","last_fullsized":"2024-09-21T10:11:41.486993+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:16.292163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.498693+0000","last_change":"2024-09-21T10:11:05.435502+0000","last_active":"2024-09-21T10:11:41.498693+0000","last_peered":"2024-09-21T10:11:41.498693+0000","last_clean":"2024-09-21T10:11:41.498693+0000","last_became_active":"2024-09-21T10:11:05.435216+0000","last_became_peered":"2024-09-21T10:11:05.435216+0000","last_unstale":"2024-09-21T10:11:41.498693+0000","last_undegraded":"2024-09-21T10:11:41.498693+0000","last_fullsized":"2024-09-21T10:11:41.498693+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:25.434424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"276'9254","reported_seq":9308,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:06.962617+0000","last_change":"2024-09-21T10:09:49.924187+0000","last_active":"2024-09-21T10:12:06.962617+0000","last_peered":"2024-09-21T10:12:06.962617+0000","last_clean":"2024-09-21T10:12:06.962617+0000","last_became_active":"2024-09-21T10:09:49.923789+0000","last_became_peered":"2024-09-21T10:09:49.923789+0000","last_unstale":"2024-09-21T10:12:06.962617+0000","last_undegraded":"2024-09-21T10:12:06.962617+0000","last_fullsized":"2024-09-21T10:12:06.962617+0000","mapping_epoch":255,"log_start":"259'3000","ondisk_log_start":"259'3000","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6254,"log_dups_size":0,"ondisk_log_size":6254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:52:49.559425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302252032,"num_objects":4612,"num_object_clones":0,"num_object_copies":13836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9254,"num_write_kb":443712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"77'100","reported_seq":212,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.861364+0000","last_change":"2024-09-21T10:09:23.839211+0000","last_active":"2024-09-21T10:12:00.861364+0000","last_peered":"2024-09-21T10:12:00.861364+0000","last_clean":"2024-09-21T10:12:00.861364+0000","last_became_active":"2024-09-21T10:09:23.838779+0000","last_became_peered":"2024-09-21T10:09:23.838779+0000","last_unstale":"2024-09-21T10:12:00.861364+0000","last_undegraded":"2024-09-21T10:12:00.861364+0000","last_fullsized":"2024-09-21T10:12:00.861364+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.981336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.981336+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:03:15.566873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":170,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.487049+0000","last_change":"2024-09-21T10:09:49.932631+0000","last_active":"2024-09-21T10:11:41.487049+0000","last_peered":"2024-09-21T10:11:41.487049+0000","last_clean":"2024-09-21T10:11:41.487049+0000","last_became_active":"2024-09-21T10:09:49.922897+0000","last_became_peered":"2024-09-21T10:09:49.922897+0000","last_unstale":"2024-09-21T10:11:41.487049+0000","last_undegraded":"2024-09-21T10:11:41.487049+0000","last_fullsized":"2024-09-21T10:11:41.487049+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:40.795476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":200,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.862938+0000","last_change":"2024-09-21T10:08:38.377732+0000","last_active":"2024-09-21T10:12:00.862938+0000","last_peered":"2024-09-21T10:12:00.862938+0000","last_clean":"2024-09-21T10:12:00.862938+0000","last_became_active":"2024-09-21T10:08:38.376601+0000","last_became_peered":"2024-09-21T10:08:38.376601+0000","last_unstale":"2024-09-21T10:12:00.862938+0000","last_undegraded":"2024-09-21T10:12:00.862938+0000","last_fullsized":"2024-09-21T10:12:00.862938+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:02.540158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.5","version":"276'9059","reported_seq":14647,"reported_epoch":276,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T10:12:06.815160+0000","last_change":"2024-09-21T10:10:58.840575+0000","last_active":"2024-09-21T10:12:06.815160+0000","last_peered":"2024-09-21T10:12:06.815160+0000","last_clean":"2024-09-21T10:10:56.374526+0000","last_became_active":"2024-09-21T10:10:58.733275+0000","last_became_peered":"2024-09-21T10:10:58.733275+0000","last_unstale":"2024-09-21T10:12:06.815160+0000","last_undegraded":"2024-09-21T10:12:06.815160+0000","last_fullsized":"2024-09-21T10:10:58.388641+0000","mapping_epoch":267,"log_start":"259'2800","ondisk_log_start":"259'2800","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6259,"log_dups_size":0,"ondisk_log_size":6259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:21:07.802295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292093952,"num_objects":4457,"num_object_clones":0,"num_object_copies":13371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2705,"num_objects_unfound":0,"num_objects_dirty":4457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9059,"num_write_kb":432512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2773,"num_bytes_recovered":181731328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,6],"avail_no_missing":["7","3","6"],"object_location_counts":[{"shards":"3,6,7","objects":2705},{"shards":"6,7","objects":1752}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"77'100","reported_seq":260,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.492247+0000","last_change":"2024-09-21T10:08:38.382100+0000","last_active":"2024-09-21T10:11:41.492247+0000","last_peered":"2024-09-21T10:11:41.492247+0000","last_clean":"2024-09-21T10:11:41.492247+0000","last_became_active":"2024-09-21T10:08:38.381736+0000","last_became_peered":"2024-09-21T10:08:38.381736+0000","last_unstale":"2024-09-21T10:11:41.492247+0000","last_undegraded":"2024-09-21T10:11:41.492247+0000","last_fullsized":"2024-09-21T10:11:41.492247+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.289859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.289859+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:23.739980+0000","scrub_duration":0.001251809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":156,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.491600+0000","last_change":"2024-09-21T10:10:57.460732+0000","last_active":"2024-09-21T10:11:41.491600+0000","last_peered":"2024-09-21T10:11:41.491600+0000","last_clean":"2024-09-21T10:11:41.491600+0000","last_became_active":"2024-09-21T10:10:57.454689+0000","last_became_peered":"2024-09-21T10:10:57.454689+0000","last_unstale":"2024-09-21T10:11:41.491600+0000","last_undegraded":"2024-09-21T10:11:41.491600+0000","last_fullsized":"2024-09-21T10:11:41.491600+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:12.035022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":276,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.491662+0000","last_change":"2024-09-21T10:03:38.378384+0000","last_active":"2024-09-21T10:11:41.491662+0000","last_peered":"2024-09-21T10:11:41.491662+0000","last_clean":"2024-09-21T10:11:41.491662+0000","last_became_active":"2024-09-21T10:03:38.378072+0000","last_became_peered":"2024-09-21T10:03:38.378072+0000","last_unstale":"2024-09-21T10:11:41.491662+0000","last_undegraded":"2024-09-21T10:11:41.491662+0000","last_fullsized":"2024-09-21T10:11:41.491662+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:54:37.039622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"276'9521","reported_seq":9553,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:06.782704+0000","last_change":"2024-09-21T10:09:42.896949+0000","last_active":"2024-09-21T10:12:06.782704+0000","last_peered":"2024-09-21T10:12:06.782704+0000","last_clean":"2024-09-21T10:12:06.782704+0000","last_became_active":"2024-09-21T10:09:42.896669+0000","last_became_peered":"2024-09-21T10:09:42.896669+0000","last_unstale":"2024-09-21T10:12:06.782704+0000","last_undegraded":"2024-09-21T10:12:06.782704+0000","last_fullsized":"2024-09-21T10:12:06.782704+0000","mapping_epoch":252,"log_start":"259'3200","ondisk_log_start":"259'3200","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6321,"log_dups_size":0,"ondisk_log_size":6321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:44.279472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308215808,"num_objects":4703,"num_object_clones":0,"num_object_copies":14109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9521,"num_write_kb":455168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"77'100","reported_seq":128,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.486752+0000","last_change":"2024-09-21T10:07:26.188298+0000","last_active":"2024-09-21T10:11:41.486752+0000","last_peered":"2024-09-21T10:11:41.486752+0000","last_clean":"2024-09-21T10:11:41.486752+0000","last_became_active":"2024-09-21T10:07:24.345265+0000","last_became_peered":"2024-09-21T10:07:24.345265+0000","last_unstale":"2024-09-21T10:11:41.486752+0000","last_undegraded":"2024-09-21T10:11:41.486752+0000","last_fullsized":"2024-09-21T10:11:41.486752+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:07:26.188220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:07:26.188220+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:46.381710+0000","scrub_duration":0.021884888000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":209,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.861631+0000","last_change":"2024-09-21T10:07:57.551556+0000","last_active":"2024-09-21T10:12:00.861631+0000","last_peered":"2024-09-21T10:12:00.861631+0000","last_clean":"2024-09-21T10:12:00.861631+0000","last_became_active":"2024-09-21T10:07:57.551249+0000","last_became_peered":"2024-09-21T10:07:57.551249+0000","last_unstale":"2024-09-21T10:12:00.861631+0000","last_undegraded":"2024-09-21T10:12:00.861631+0000","last_fullsized":"2024-09-21T10:12:00.861631+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:57.557152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":307,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.864959+0000","last_change":"2024-09-21T10:09:24.839665+0000","last_active":"2024-09-21T10:12:00.864959+0000","last_peered":"2024-09-21T10:12:00.864959+0000","last_clean":"2024-09-21T10:12:00.864959+0000","last_became_active":"2024-09-21T10:09:24.839353+0000","last_became_peered":"2024-09-21T10:09:24.839353+0000","last_unstale":"2024-09-21T10:12:00.864959+0000","last_undegraded":"2024-09-21T10:12:00.864959+0000","last_fullsized":"2024-09-21T10:12:00.864959+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:51.760073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3","version":"276'9265","reported_seq":9297,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:07.741969+0000","last_change":"2024-09-21T10:09:43.231199+0000","last_active":"2024-09-21T10:12:07.741969+0000","last_peered":"2024-09-21T10:12:07.741969+0000","last_clean":"2024-09-21T10:12:07.741969+0000","last_became_active":"2024-09-21T10:09:43.230922+0000","last_became_peered":"2024-09-21T10:09:43.230922+0000","last_unstale":"2024-09-21T10:12:07.741969+0000","last_undegraded":"2024-09-21T10:12:07.741969+0000","last_fullsized":"2024-09-21T10:12:07.741969+0000","mapping_epoch":252,"log_start":"259'3100","ondisk_log_start":"259'3100","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6165,"log_dups_size":0,"ondisk_log_size":6165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:39.944122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302317568,"num_objects":4613,"num_object_clones":0,"num_object_copies":13839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9265,"num_write_kb":444096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":150,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.486558+0000","last_change":"2024-09-21T10:04:53.355975+0000","last_active":"2024-09-21T10:11:41.486558+0000","last_peered":"2024-09-21T10:11:41.486558+0000","last_clean":"2024-09-21T10:11:41.486558+0000","last_became_active":"2024-09-21T10:04:42.547505+0000","last_became_peered":"2024-09-21T10:04:42.547505+0000","last_unstale":"2024-09-21T10:11:41.486558+0000","last_undegraded":"2024-09-21T10:11:41.486558+0000","last_fullsized":"2024-09-21T10:11:41.486558+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.355912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.355912+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:43.456830+0000","scrub_duration":0.0017980030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":406,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.862134+0000","last_change":"2024-09-21T10:09:49.939553+0000","last_active":"2024-09-21T10:12:00.862134+0000","last_peered":"2024-09-21T10:12:00.862134+0000","last_clean":"2024-09-21T10:12:00.862134+0000","last_became_active":"2024-09-21T10:09:49.925094+0000","last_became_peered":"2024-09-21T10:09:49.925094+0000","last_unstale":"2024-09-21T10:12:00.862134+0000","last_undegraded":"2024-09-21T10:12:00.862134+0000","last_fullsized":"2024-09-21T10:12:00.862134+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:12:12.933089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":183,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.861716+0000","last_change":"2024-09-21T10:08:38.379747+0000","last_active":"2024-09-21T10:12:00.861716+0000","last_peered":"2024-09-21T10:12:00.861716+0000","last_clean":"2024-09-21T10:12:00.861716+0000","last_became_active":"2024-09-21T10:08:38.378983+0000","last_became_peered":"2024-09-21T10:08:38.378983+0000","last_unstale":"2024-09-21T10:12:00.861716+0000","last_undegraded":"2024-09-21T10:12:00.861716+0000","last_fullsized":"2024-09-21T10:12:00.861716+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:45:08.003016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"276'9526","reported_seq":9586,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:09.120329+0000","last_change":"2024-09-21T10:09:42.896430+0000","last_active":"2024-09-21T10:12:09.120329+0000","last_peered":"2024-09-21T10:12:09.120329+0000","last_clean":"2024-09-21T10:12:09.120329+0000","last_became_active":"2024-09-21T10:09:42.896019+0000","last_became_peered":"2024-09-21T10:09:42.896019+0000","last_unstale":"2024-09-21T10:12:09.120329+0000","last_undegraded":"2024-09-21T10:12:09.120329+0000","last_fullsized":"2024-09-21T10:12:09.120329+0000","mapping_epoch":252,"log_start":"259'2000","ondisk_log_start":"259'2000","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":7526,"log_dups_size":0,"ondisk_log_size":7526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:59:18.633498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292683776,"num_objects":4466,"num_object_clones":0,"num_object_copies":13398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9526,"num_write_kb":447744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":165,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.860004+0000","last_change":"2024-09-21T10:10:57.403722+0000","last_active":"2024-09-21T10:12:00.860004+0000","last_peered":"2024-09-21T10:12:00.860004+0000","last_clean":"2024-09-21T10:12:00.860004+0000","last_became_active":"2024-09-21T10:10:57.394700+0000","last_became_peered":"2024-09-21T10:10:57.394700+0000","last_unstale":"2024-09-21T10:12:00.860004+0000","last_undegraded":"2024-09-21T10:12:00.860004+0000","last_fullsized":"2024-09-21T10:12:00.860004+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:02.152981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:02.152981+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:52:34.887763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":250,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.491800+0000","last_change":"2024-09-21T10:08:38.379849+0000","last_active":"2024-09-21T10:11:41.491800+0000","last_peered":"2024-09-21T10:11:41.491800+0000","last_clean":"2024-09-21T10:11:41.491800+0000","last_became_active":"2024-09-21T10:08:38.379543+0000","last_became_peered":"2024-09-21T10:08:38.379543+0000","last_unstale":"2024-09-21T10:11:41.491800+0000","last_undegraded":"2024-09-21T10:11:41.491800+0000","last_fullsized":"2024-09-21T10:11:41.491800+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:29.041802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":215,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.862557+0000","last_change":"2024-09-21T10:07:24.342950+0000","last_active":"2024-09-21T10:12:00.862557+0000","last_peered":"2024-09-21T10:12:00.862557+0000","last_clean":"2024-09-21T10:12:00.862557+0000","last_became_active":"2024-09-21T10:07:24.342477+0000","last_became_peered":"2024-09-21T10:07:24.342477+0000","last_unstale":"2024-09-21T10:12:00.862557+0000","last_undegraded":"2024-09-21T10:12:00.862557+0000","last_fullsized":"2024-09-21T10:12:00.862557+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:33:13.664050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1","version":"276'9556","reported_seq":9587,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:07.799616+0000","last_change":"2024-09-21T10:09:42.895259+0000","last_active":"2024-09-21T10:12:07.799616+0000","last_peered":"2024-09-21T10:12:07.799616+0000","last_clean":"2024-09-21T10:12:07.799616+0000","last_became_active":"2024-09-21T10:09:42.895087+0000","last_became_peered":"2024-09-21T10:09:42.895087+0000","last_unstale":"2024-09-21T10:12:07.799616+0000","last_undegraded":"2024-09-21T10:12:07.799616+0000","last_fullsized":"2024-09-21T10:12:07.799616+0000","mapping_epoch":252,"log_start":"259'3400","ondisk_log_start":"259'3400","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6156,"log_dups_size":0,"ondisk_log_size":6156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:40:59.270926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298844160,"num_objects":4560,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9556,"num_write_kb":451712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"77'100","reported_seq":309,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.865164+0000","last_change":"2024-09-21T10:10:57.395906+0000","last_active":"2024-09-21T10:12:00.865164+0000","last_peered":"2024-09-21T10:12:00.865164+0000","last_clean":"2024-09-21T10:12:00.865164+0000","last_became_active":"2024-09-21T10:10:57.395366+0000","last_became_peered":"2024-09-21T10:10:57.395366+0000","last_unstale":"2024-09-21T10:12:00.865164+0000","last_undegraded":"2024-09-21T10:12:00.865164+0000","last_fullsized":"2024-09-21T10:12:00.865164+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.540323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.540323+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:34.873336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":307,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.860241+0000","last_change":"2024-09-21T10:07:36.464496+0000","last_active":"2024-09-21T10:12:00.860241+0000","last_peered":"2024-09-21T10:12:00.860241+0000","last_clean":"2024-09-21T10:12:00.860241+0000","last_became_active":"2024-09-21T10:07:36.464172+0000","last_became_peered":"2024-09-21T10:07:36.464172+0000","last_unstale":"2024-09-21T10:12:00.860241+0000","last_undegraded":"2024-09-21T10:12:00.860241+0000","last_fullsized":"2024-09-21T10:12:00.860241+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:09.839709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":16,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.492056+0000","last_change":"2024-09-21T10:11:05.442332+0000","last_active":"2024-09-21T10:11:41.492056+0000","last_peered":"2024-09-21T10:11:41.492056+0000","last_clean":"2024-09-21T10:11:41.492056+0000","last_became_active":"2024-09-21T10:11:05.442003+0000","last_became_peered":"2024-09-21T10:11:05.442003+0000","last_unstale":"2024-09-21T10:11:41.492056+0000","last_undegraded":"2024-09-21T10:11:41.492056+0000","last_fullsized":"2024-09-21T10:11:41.492056+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:34.986309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"276'9389","reported_seq":9449,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:06.516365+0000","last_change":"2024-09-21T10:09:43.231401+0000","last_active":"2024-09-21T10:12:06.516365+0000","last_peered":"2024-09-21T10:12:06.516365+0000","last_clean":"2024-09-21T10:12:06.516365+0000","last_became_active":"2024-09-21T10:09:43.230977+0000","last_became_peered":"2024-09-21T10:09:43.230977+0000","last_unstale":"2024-09-21T10:12:06.516365+0000","last_undegraded":"2024-09-21T10:12:06.516365+0000","last_fullsized":"2024-09-21T10:12:06.516365+0000","mapping_epoch":252,"log_start":"259'3700","ondisk_log_start":"259'3700","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5689,"log_dups_size":0,"ondisk_log_size":5689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:15.907809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306905088,"num_objects":4683,"num_object_clones":0,"num_object_copies":14049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9389,"num_write_kb":450304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":128,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.860727+0000","last_change":"2024-09-21T10:08:04.645700+0000","last_active":"2024-09-21T10:12:00.860727+0000","last_peered":"2024-09-21T10:12:00.860727+0000","last_clean":"2024-09-21T10:12:00.860727+0000","last_became_active":"2024-09-21T10:07:57.548189+0000","last_became_peered":"2024-09-21T10:07:57.548189+0000","last_unstale":"2024-09-21T10:12:00.860727+0000","last_undegraded":"2024-09-21T10:12:00.860727+0000","last_fullsized":"2024-09-21T10:12:00.860727+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:48.318010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:48.318010+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T16:31:28.173059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":407,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.862809+0000","last_change":"2024-09-21T10:03:38.386604+0000","last_active":"2024-09-21T10:12:00.862809+0000","last_peered":"2024-09-21T10:12:00.862809+0000","last_clean":"2024-09-21T10:12:00.862809+0000","last_became_active":"2024-09-21T10:03:38.386283+0000","last_became_peered":"2024-09-21T10:03:38.386283+0000","last_unstale":"2024-09-21T10:12:00.862809+0000","last_undegraded":"2024-09-21T10:12:00.862809+0000","last_fullsized":"2024-09-21T10:12:00.862809+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:27:05.944880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":210,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.862049+0000","last_change":"2024-09-21T10:08:38.372486+0000","last_active":"2024-09-21T10:12:00.862049+0000","last_peered":"2024-09-21T10:12:00.862049+0000","last_clean":"2024-09-21T10:12:00.862049+0000","last_became_active":"2024-09-21T10:08:38.370669+0000","last_became_peered":"2024-09-21T10:08:38.370669+0000","last_unstale":"2024-09-21T10:12:00.862049+0000","last_undegraded":"2024-09-21T10:12:00.862049+0000","last_fullsized":"2024-09-21T10:12:00.862049+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:53:23.247341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.a","version":"276'9347","reported_seq":9418,"reported_epoch":276,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:12:06.487214+0000","last_change":"2024-09-21T10:10:58.704699+0000","last_active":"2024-09-21T10:12:06.487214+0000","last_peered":"2024-09-21T10:12:06.487214+0000","last_clean":"2024-09-21T10:10:56.390149+0000","last_became_active":"2024-09-21T10:10:58.682906+0000","last_became_peered":"2024-09-21T10:10:58.682906+0000","last_unstale":"2024-09-21T10:12:06.487214+0000","last_undegraded":"2024-09-21T10:12:06.487214+0000","last_fullsized":"2024-09-21T10:12:06.487214+0000","mapping_epoch":267,"log_start":"259'3600","ondisk_log_start":"259'3600","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5747,"log_dups_size":0,"ondisk_log_size":5747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299565056,"num_objects":4571,"num_object_clones":0,"num_object_copies":13713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4571,"num_objects_unfound":0,"num_objects_dirty":4571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9347,"num_write_kb":445376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,1],"avail_no_missing":["3","1","6"],"object_location_counts":[{"shards":"1,3,6","objects":4571}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":167,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.486499+0000","last_change":"2024-09-21T10:09:49.924286+0000","last_active":"2024-09-21T10:11:41.486499+0000","last_peered":"2024-09-21T10:11:41.486499+0000","last_clean":"2024-09-21T10:11:41.486499+0000","last_became_active":"2024-09-21T10:09:49.923897+0000","last_became_peered":"2024-09-21T10:09:49.923897+0000","last_unstale":"2024-09-21T10:11:41.486499+0000","last_undegraded":"2024-09-21T10:11:41.486499+0000","last_fullsized":"2024-09-21T10:11:41.486499+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:47.393146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:47.393146+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:44.898083+0000","scrub_duration":0.001601086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":162,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.860529+0000","last_change":"2024-09-21T10:10:57.402946+0000","last_active":"2024-09-21T10:12:00.860529+0000","last_peered":"2024-09-21T10:12:00.860529+0000","last_clean":"2024-09-21T10:12:00.860529+0000","last_became_active":"2024-09-21T10:10:57.395450+0000","last_became_peered":"2024-09-21T10:10:57.395450+0000","last_unstale":"2024-09-21T10:12:00.860529+0000","last_undegraded":"2024-09-21T10:12:00.860529+0000","last_fullsized":"2024-09-21T10:12:00.860529+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:08.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":391,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.863691+0000","last_change":"2024-09-21T10:10:57.390595+0000","last_active":"2024-09-21T10:12:00.863691+0000","last_peered":"2024-09-21T10:12:00.863691+0000","last_clean":"2024-09-21T10:12:00.863691+0000","last_became_active":"2024-09-21T10:10:57.390265+0000","last_became_peered":"2024-09-21T10:10:57.390265+0000","last_unstale":"2024-09-21T10:12:00.863691+0000","last_undegraded":"2024-09-21T10:12:00.863691+0000","last_fullsized":"2024-09-21T10:12:00.863691+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:37.485063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":283,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.498455+0000","last_change":"2024-09-21T10:09:24.837651+0000","last_active":"2024-09-21T10:11:41.498455+0000","last_peered":"2024-09-21T10:11:41.498455+0000","last_clean":"2024-09-21T10:11:41.498455+0000","last_became_active":"2024-09-21T10:09:24.837226+0000","last_became_peered":"2024-09-21T10:09:24.837226+0000","last_unstale":"2024-09-21T10:11:41.498455+0000","last_undegraded":"2024-09-21T10:11:41.498455+0000","last_fullsized":"2024-09-21T10:11:41.498455+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:16.649601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"276'9374","reported_seq":9441,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:06.531263+0000","last_change":"2024-09-21T10:09:49.926338+0000","last_active":"2024-09-21T10:12:06.531263+0000","last_peered":"2024-09-21T10:12:06.531263+0000","last_clean":"2024-09-21T10:12:06.531263+0000","last_became_active":"2024-09-21T10:09:49.926018+0000","last_became_peered":"2024-09-21T10:09:49.926018+0000","last_unstale":"2024-09-21T10:12:06.531263+0000","last_undegraded":"2024-09-21T10:12:06.531263+0000","last_fullsized":"2024-09-21T10:12:06.531263+0000","mapping_epoch":255,"log_start":"259'3700","ondisk_log_start":"259'3700","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5674,"log_dups_size":0,"ondisk_log_size":5674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:09:49.364660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300679168,"num_objects":4588,"num_object_clones":0,"num_object_copies":13764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9374,"num_write_kb":446784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":16,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.491494+0000","last_change":"2024-09-21T10:11:05.443167+0000","last_active":"2024-09-21T10:11:41.491494+0000","last_peered":"2024-09-21T10:11:41.491494+0000","last_clean":"2024-09-21T10:11:41.491494+0000","last_became_active":"2024-09-21T10:11:05.442805+0000","last_became_peered":"2024-09-21T10:11:05.442805+0000","last_unstale":"2024-09-21T10:11:41.491494+0000","last_undegraded":"2024-09-21T10:11:41.491494+0000","last_fullsized":"2024-09-21T10:11:41.491494+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.550551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.550551+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:58:57.998543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":141,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.486495+0000","last_change":"2024-09-21T10:09:23.838944+0000","last_active":"2024-09-21T10:11:41.486495+0000","last_peered":"2024-09-21T10:11:41.486495+0000","last_clean":"2024-09-21T10:11:41.486495+0000","last_became_active":"2024-09-21T10:09:23.837886+0000","last_became_peered":"2024-09-21T10:09:23.837886+0000","last_unstale":"2024-09-21T10:11:41.486495+0000","last_undegraded":"2024-09-21T10:11:41.486495+0000","last_fullsized":"2024-09-21T10:11:41.486495+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:16.928195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":284,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.866212+0000","last_change":"2024-09-21T10:09:24.839770+0000","last_active":"2024-09-21T10:12:00.866212+0000","last_peered":"2024-09-21T10:12:00.866212+0000","last_clean":"2024-09-21T10:12:00.866212+0000","last_became_active":"2024-09-21T10:09:24.839479+0000","last_became_peered":"2024-09-21T10:09:24.839479+0000","last_unstale":"2024-09-21T10:12:00.866212+0000","last_undegraded":"2024-09-21T10:12:00.866212+0000","last_fullsized":"2024-09-21T10:12:00.866212+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:11:19.618193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":53,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.865374+0000","last_change":"2024-09-21T10:09:49.923044+0000","last_active":"2024-09-21T10:12:00.865374+0000","last_peered":"2024-09-21T10:12:00.865374+0000","last_clean":"2024-09-21T10:12:00.865374+0000","last_became_active":"2024-09-21T10:09:49.922711+0000","last_became_peered":"2024-09-21T10:09:49.922711+0000","last_unstale":"2024-09-21T10:12:00.865374+0000","last_undegraded":"2024-09-21T10:12:00.865374+0000","last_fullsized":"2024-09-21T10:12:00.865374+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:14.090084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"276'9616","reported_seq":9691,"reported_epoch":276,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:12:06.497122+0000","last_change":"2024-09-21T10:10:58.695837+0000","last_active":"2024-09-21T10:12:06.497122+0000","last_peered":"2024-09-21T10:12:06.497122+0000","last_clean":"2024-09-21T10:10:56.379248+0000","last_became_active":"2024-09-21T10:10:58.672712+0000","last_became_peered":"2024-09-21T10:10:58.672712+0000","last_unstale":"2024-09-21T10:12:06.497122+0000","last_undegraded":"2024-09-21T10:12:06.497122+0000","last_fullsized":"2024-09-21T10:12:06.497122+0000","mapping_epoch":267,"log_start":"261'3900","ondisk_log_start":"261'3900","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5716,"log_dups_size":0,"ondisk_log_size":5716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304349184,"num_objects":4644,"num_object_clones":0,"num_object_copies":13932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4644,"num_objects_unfound":0,"num_objects_dirty":4644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9616,"num_write_kb":456320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,0],"avail_no_missing":["3","0","6"],"object_location_counts":[{"shards":"0,3,6","objects":4644}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":181,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.486791+0000","last_change":"2024-09-21T10:09:24.838434+0000","last_active":"2024-09-21T10:11:41.486791+0000","last_peered":"2024-09-21T10:11:41.486791+0000","last_clean":"2024-09-21T10:11:41.486791+0000","last_became_active":"2024-09-21T10:09:24.838066+0000","last_became_peered":"2024-09-21T10:09:24.838066+0000","last_unstale":"2024-09-21T10:11:41.486791+0000","last_undegraded":"2024-09-21T10:11:41.486791+0000","last_fullsized":"2024-09-21T10:11:41.486791+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:52.361137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:52.361137+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:19:57.788195+0000","scrub_duration":0.001255358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":444,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.860377+0000","last_change":"2024-09-21T10:09:23.837859+0000","last_active":"2024-09-21T10:12:00.860377+0000","last_peered":"2024-09-21T10:12:00.860377+0000","last_clean":"2024-09-21T10:12:00.860377+0000","last_became_active":"2024-09-21T10:09:23.837488+0000","last_became_peered":"2024-09-21T10:09:23.837488+0000","last_unstale":"2024-09-21T10:12:00.860377+0000","last_undegraded":"2024-09-21T10:12:00.860377+0000","last_fullsized":"2024-09-21T10:12:00.860377+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:25.215501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.486372+0000","last_change":"2024-09-21T10:10:27.806070+0000","last_active":"2024-09-21T10:11:41.486372+0000","last_peered":"2024-09-21T10:11:41.486372+0000","last_clean":"2024-09-21T10:11:41.486372+0000","last_became_active":"2024-09-21T10:10:27.805718+0000","last_became_peered":"2024-09-21T10:10:27.805718+0000","last_unstale":"2024-09-21T10:11:41.486372+0000","last_undegraded":"2024-09-21T10:11:41.486372+0000","last_fullsized":"2024-09-21T10:11:41.486372+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:31.144118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":438,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.860161+0000","last_change":"2024-09-21T10:10:57.395073+0000","last_active":"2024-09-21T10:12:00.860161+0000","last_peered":"2024-09-21T10:12:00.860161+0000","last_clean":"2024-09-21T10:12:00.860161+0000","last_became_active":"2024-09-21T10:10:57.394674+0000","last_became_peered":"2024-09-21T10:10:57.394674+0000","last_unstale":"2024-09-21T10:12:00.860161+0000","last_undegraded":"2024-09-21T10:12:00.860161+0000","last_fullsized":"2024-09-21T10:12:00.860161+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:48:48.912649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"276'9604","reported_seq":9664,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:06.501318+0000","last_change":"2024-09-21T10:09:43.231554+0000","last_active":"2024-09-21T10:12:06.501318+0000","last_peered":"2024-09-21T10:12:06.501318+0000","last_clean":"2024-09-21T10:12:06.501318+0000","last_became_active":"2024-09-21T10:09:43.231211+0000","last_became_peered":"2024-09-21T10:09:43.231211+0000","last_unstale":"2024-09-21T10:12:06.501318+0000","last_undegraded":"2024-09-21T10:12:06.501318+0000","last_fullsized":"2024-09-21T10:12:06.501318+0000","mapping_epoch":252,"log_start":"262'3900","ondisk_log_start":"262'3900","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5704,"log_dups_size":0,"ondisk_log_size":5704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:16:41.457768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307494912,"num_objects":4692,"num_object_clones":0,"num_object_copies":14076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9604,"num_write_kb":457472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":308,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.866309+0000","last_change":"2024-09-21T10:10:57.398437+0000","last_active":"2024-09-21T10:12:00.866309+0000","last_peered":"2024-09-21T10:12:00.866309+0000","last_clean":"2024-09-21T10:12:00.866309+0000","last_became_active":"2024-09-21T10:10:57.397441+0000","last_became_peered":"2024-09-21T10:10:57.397441+0000","last_unstale":"2024-09-21T10:12:00.866309+0000","last_undegraded":"2024-09-21T10:12:00.866309+0000","last_fullsized":"2024-09-21T10:12:00.866309+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:04.043917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:04.043917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:53:31.813612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":80,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.491870+0000","last_change":"2024-09-21T10:08:26.309323+0000","last_active":"2024-09-21T10:11:41.491870+0000","last_peered":"2024-09-21T10:11:41.491870+0000","last_clean":"2024-09-21T10:11:41.491870+0000","last_became_active":"2024-09-21T10:08:26.309004+0000","last_became_peered":"2024-09-21T10:08:26.309004+0000","last_unstale":"2024-09-21T10:11:41.491870+0000","last_undegraded":"2024-09-21T10:11:41.491870+0000","last_fullsized":"2024-09-21T10:11:41.491870+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:37:20.626702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":118,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.865088+0000","last_change":"2024-09-21T10:10:28.180093+0000","last_active":"2024-09-21T10:12:00.865088+0000","last_peered":"2024-09-21T10:12:00.865088+0000","last_clean":"2024-09-21T10:12:00.865088+0000","last_became_active":"2024-09-21T10:10:28.179559+0000","last_became_peered":"2024-09-21T10:10:28.179559+0000","last_unstale":"2024-09-21T10:12:00.865088+0000","last_undegraded":"2024-09-21T10:12:00.865088+0000","last_fullsized":"2024-09-21T10:12:00.865088+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:52:46.146392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":214,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.866063+0000","last_change":"2024-09-21T10:08:26.307041+0000","last_active":"2024-09-21T10:12:00.866063+0000","last_peered":"2024-09-21T10:12:00.866063+0000","last_clean":"2024-09-21T10:12:00.866063+0000","last_became_active":"2024-09-21T10:08:26.306715+0000","last_became_peered":"2024-09-21T10:08:26.306715+0000","last_unstale":"2024-09-21T10:12:00.866063+0000","last_undegraded":"2024-09-21T10:12:00.866063+0000","last_fullsized":"2024-09-21T10:12:00.866063+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:19:26.192398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"276'9676","reported_seq":9737,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:09.154026+0000","last_change":"2024-09-21T10:09:43.232893+0000","last_active":"2024-09-21T10:12:09.154026+0000","last_peered":"2024-09-21T10:12:09.154026+0000","last_clean":"2024-09-21T10:12:09.154026+0000","last_became_active":"2024-09-21T10:09:43.232461+0000","last_became_peered":"2024-09-21T10:09:43.232461+0000","last_unstale":"2024-09-21T10:12:09.154026+0000","last_undegraded":"2024-09-21T10:12:09.154026+0000","last_fullsized":"2024-09-21T10:12:09.154026+0000","mapping_epoch":252,"log_start":"259'2100","ondisk_log_start":"259'2100","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":7576,"log_dups_size":0,"ondisk_log_size":7576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:39.778833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298778648,"num_objects":4560,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4560,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9676,"num_write_kb":455489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":250,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.492177+0000","last_change":"2024-09-21T10:04:45.242148+0000","last_active":"2024-09-21T10:11:41.492177+0000","last_peered":"2024-09-21T10:11:41.492177+0000","last_clean":"2024-09-21T10:11:41.492177+0000","last_became_active":"2024-09-21T10:04:42.549844+0000","last_became_peered":"2024-09-21T10:04:42.549844+0000","last_unstale":"2024-09-21T10:11:41.492177+0000","last_undegraded":"2024-09-21T10:11:41.492177+0000","last_fullsized":"2024-09-21T10:11:41.492177+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_deep_scrub":"77'100","last_deep_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.242073+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:03:42.250480+0000","scrub_duration":0.0023466239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.503108+0000","last_change":"2024-09-21T10:10:57.392880+0000","last_active":"2024-09-21T10:11:41.503108+0000","last_peered":"2024-09-21T10:11:41.503108+0000","last_clean":"2024-09-21T10:11:41.503108+0000","last_became_active":"2024-09-21T10:10:57.392539+0000","last_became_peered":"2024-09-21T10:10:57.392539+0000","last_unstale":"2024-09-21T10:11:41.503108+0000","last_undegraded":"2024-09-21T10:11:41.503108+0000","last_fullsized":"2024-09-21T10:11:41.503108+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:49.754955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":328,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.864707+0000","last_change":"2024-09-21T10:09:23.865342+0000","last_active":"2024-09-21T10:12:00.864707+0000","last_peered":"2024-09-21T10:12:00.864707+0000","last_clean":"2024-09-21T10:12:00.864707+0000","last_became_active":"2024-09-21T10:09:23.865008+0000","last_became_peered":"2024-09-21T10:09:23.865008+0000","last_unstale":"2024-09-21T10:12:00.864707+0000","last_undegraded":"2024-09-21T10:12:00.864707+0000","last_fullsized":"2024-09-21T10:12:00.864707+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:03.356101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.498575+0000","last_change":"2024-09-21T10:10:57.394707+0000","last_active":"2024-09-21T10:11:41.498575+0000","last_peered":"2024-09-21T10:11:41.498575+0000","last_clean":"2024-09-21T10:11:41.498575+0000","last_became_active":"2024-09-21T10:10:57.394363+0000","last_became_peered":"2024-09-21T10:10:57.394363+0000","last_unstale":"2024-09-21T10:11:41.498575+0000","last_undegraded":"2024-09-21T10:11:41.498575+0000","last_fullsized":"2024-09-21T10:11:41.498575+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:21.141052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"276'9644","reported_seq":9704,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:09.163194+0000","last_change":"2024-09-21T10:09:43.232987+0000","last_active":"2024-09-21T10:12:09.163194+0000","last_peered":"2024-09-21T10:12:09.163194+0000","last_clean":"2024-09-21T10:12:09.163194+0000","last_became_active":"2024-09-21T10:09:43.232577+0000","last_became_peered":"2024-09-21T10:09:43.232577+0000","last_unstale":"2024-09-21T10:12:09.163194+0000","last_undegraded":"2024-09-21T10:12:09.163194+0000","last_fullsized":"2024-09-21T10:12:09.163194+0000","mapping_epoch":252,"log_start":"259'2100","ondisk_log_start":"259'2100","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":7544,"log_dups_size":0,"ondisk_log_size":7544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:14.536774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294780928,"num_objects":4498,"num_object_clones":0,"num_object_copies":13494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9644,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":267,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.492149+0000","last_change":"2024-09-21T10:09:23.839563+0000","last_active":"2024-09-21T10:11:41.492149+0000","last_peered":"2024-09-21T10:11:41.492149+0000","last_clean":"2024-09-21T10:11:41.492149+0000","last_became_active":"2024-09-21T10:09:23.839207+0000","last_became_peered":"2024-09-21T10:09:23.839207+0000","last_unstale":"2024-09-21T10:11:41.492149+0000","last_undegraded":"2024-09-21T10:11:41.492149+0000","last_fullsized":"2024-09-21T10:11:41.492149+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:49.226496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:49.226496+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:03.960125+0000","scrub_duration":0.001656865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":179,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.865895+0000","last_change":"2024-09-21T10:10:57.395167+0000","last_active":"2024-09-21T10:12:00.865895+0000","last_peered":"2024-09-21T10:12:00.865895+0000","last_clean":"2024-09-21T10:12:00.865895+0000","last_became_active":"2024-09-21T10:10:57.394818+0000","last_became_peered":"2024-09-21T10:10:57.394818+0000","last_unstale":"2024-09-21T10:12:00.865895+0000","last_undegraded":"2024-09-21T10:12:00.865895+0000","last_fullsized":"2024-09-21T10:12:00.865895+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:57.224113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":161,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.491666+0000","last_change":"2024-09-21T10:09:49.922782+0000","last_active":"2024-09-21T10:11:41.491666+0000","last_peered":"2024-09-21T10:11:41.491666+0000","last_clean":"2024-09-21T10:11:41.491666+0000","last_became_active":"2024-09-21T10:09:49.922465+0000","last_became_peered":"2024-09-21T10:09:49.922465+0000","last_unstale":"2024-09-21T10:11:41.491666+0000","last_undegraded":"2024-09-21T10:11:41.491666+0000","last_fullsized":"2024-09-21T10:11:41.491666+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:06.151245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":322,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.865240+0000","last_change":"2024-09-21T10:10:57.397251+0000","last_active":"2024-09-21T10:12:00.865240+0000","last_peered":"2024-09-21T10:12:00.865240+0000","last_clean":"2024-09-21T10:12:00.865240+0000","last_became_active":"2024-09-21T10:10:57.396271+0000","last_became_peered":"2024-09-21T10:10:57.396271+0000","last_unstale":"2024-09-21T10:12:00.865240+0000","last_undegraded":"2024-09-21T10:12:00.865240+0000","last_fullsized":"2024-09-21T10:12:00.865240+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:17.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.f","version":"276'9584","reported_seq":9616,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:06.710048+0000","last_change":"2024-09-21T10:09:43.232625+0000","last_active":"2024-09-21T10:12:06.710048+0000","last_peered":"2024-09-21T10:12:06.710048+0000","last_clean":"2024-09-21T10:12:06.710048+0000","last_became_active":"2024-09-21T10:09:43.232331+0000","last_became_peered":"2024-09-21T10:09:43.232331+0000","last_unstale":"2024-09-21T10:12:06.710048+0000","last_undegraded":"2024-09-21T10:12:06.710048+0000","last_fullsized":"2024-09-21T10:12:06.710048+0000","mapping_epoch":252,"log_start":"259'3300","ondisk_log_start":"259'3300","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6284,"log_dups_size":0,"ondisk_log_size":6284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:01:16.634826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307625984,"num_objects":4694,"num_object_clones":0,"num_object_copies":14082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9584,"num_write_kb":456896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":125,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.862256+0000","last_change":"2024-09-21T10:11:05.455170+0000","last_active":"2024-09-21T10:12:00.862256+0000","last_peered":"2024-09-21T10:12:00.862256+0000","last_clean":"2024-09-21T10:12:00.862256+0000","last_became_active":"2024-09-21T10:11:05.454607+0000","last_became_peered":"2024-09-21T10:11:05.454607+0000","last_unstale":"2024-09-21T10:12:00.862256+0000","last_undegraded":"2024-09-21T10:12:00.862256+0000","last_fullsized":"2024-09-21T10:12:00.862256+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:59.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:59.099852+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:14.788975+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":413,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.862127+0000","last_change":"2024-09-21T10:09:23.841186+0000","last_active":"2024-09-21T10:12:00.862127+0000","last_peered":"2024-09-21T10:12:00.862127+0000","last_clean":"2024-09-21T10:12:00.862127+0000","last_became_active":"2024-09-21T10:09:23.840854+0000","last_became_peered":"2024-09-21T10:09:23.840854+0000","last_unstale":"2024-09-21T10:12:00.862127+0000","last_undegraded":"2024-09-21T10:12:00.862127+0000","last_fullsized":"2024-09-21T10:12:00.862127+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:29.666358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":98,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.492221+0000","last_change":"2024-09-21T10:08:38.380867+0000","last_active":"2024-09-21T10:11:41.492221+0000","last_peered":"2024-09-21T10:11:41.492221+0000","last_clean":"2024-09-21T10:11:41.492221+0000","last_became_active":"2024-09-21T10:08:38.379599+0000","last_became_peered":"2024-09-21T10:08:38.379599+0000","last_unstale":"2024-09-21T10:11:41.492221+0000","last_undegraded":"2024-09-21T10:11:41.492221+0000","last_fullsized":"2024-09-21T10:11:41.492221+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:50.499470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":409,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.861306+0000","last_change":"2024-09-21T10:09:49.926548+0000","last_active":"2024-09-21T10:12:00.861306+0000","last_peered":"2024-09-21T10:12:00.861306+0000","last_clean":"2024-09-21T10:12:00.861306+0000","last_became_active":"2024-09-21T10:09:49.926255+0000","last_became_peered":"2024-09-21T10:09:49.926255+0000","last_unstale":"2024-09-21T10:12:00.861306+0000","last_undegraded":"2024-09-21T10:12:00.861306+0000","last_fullsized":"2024-09-21T10:12:00.861306+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:49.855878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"276'9606","reported_seq":9638,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:06.776083+0000","last_change":"2024-09-21T10:09:43.231158+0000","last_active":"2024-09-21T10:12:06.776083+0000","last_peered":"2024-09-21T10:12:06.776083+0000","last_clean":"2024-09-21T10:12:06.776083+0000","last_became_active":"2024-09-21T10:09:43.230838+0000","last_became_peered":"2024-09-21T10:09:43.230838+0000","last_unstale":"2024-09-21T10:12:06.776083+0000","last_undegraded":"2024-09-21T10:12:06.776083+0000","last_fullsized":"2024-09-21T10:12:06.776083+0000","mapping_epoch":252,"log_start":"259'3300","ondisk_log_start":"259'3300","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6306,"log_dups_size":0,"ondisk_log_size":6306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:15.318633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303038464,"num_objects":4624,"num_object_clones":0,"num_object_copies":13872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9606,"num_write_kb":455360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":211,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.861262+0000","last_change":"2024-09-21T10:10:57.390804+0000","last_active":"2024-09-21T10:12:00.861262+0000","last_peered":"2024-09-21T10:12:00.861262+0000","last_clean":"2024-09-21T10:12:00.861262+0000","last_became_active":"2024-09-21T10:10:57.390323+0000","last_became_peered":"2024-09-21T10:10:57.390323+0000","last_unstale":"2024-09-21T10:12:00.861262+0000","last_undegraded":"2024-09-21T10:12:00.861262+0000","last_fullsized":"2024-09-21T10:12:00.861262+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.005477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.005477+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:29:57.702046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":57,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.865519+0000","last_change":"2024-09-21T10:09:49.930280+0000","last_active":"2024-09-21T10:12:00.865519+0000","last_peered":"2024-09-21T10:12:00.865519+0000","last_clean":"2024-09-21T10:12:00.865519+0000","last_became_active":"2024-09-21T10:09:49.919101+0000","last_became_peered":"2024-09-21T10:09:49.919101+0000","last_unstale":"2024-09-21T10:12:00.865519+0000","last_undegraded":"2024-09-21T10:12:00.865519+0000","last_fullsized":"2024-09-21T10:12:00.865519+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:23.998475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":289,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.492377+0000","last_change":"2024-09-21T10:03:38.392767+0000","last_active":"2024-09-21T10:11:41.492377+0000","last_peered":"2024-09-21T10:11:41.492377+0000","last_clean":"2024-09-21T10:11:41.492377+0000","last_became_active":"2024-09-21T10:03:38.392424+0000","last_became_peered":"2024-09-21T10:03:38.392424+0000","last_unstale":"2024-09-21T10:11:41.492377+0000","last_undegraded":"2024-09-21T10:11:41.492377+0000","last_fullsized":"2024-09-21T10:11:41.492377+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:25.358291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":70,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.498679+0000","last_change":"2024-09-21T10:09:24.836360+0000","last_active":"2024-09-21T10:11:41.498679+0000","last_peered":"2024-09-21T10:11:41.498679+0000","last_clean":"2024-09-21T10:11:41.498679+0000","last_became_active":"2024-09-21T10:09:24.836033+0000","last_became_peered":"2024-09-21T10:09:24.836033+0000","last_unstale":"2024-09-21T10:11:41.498679+0000","last_undegraded":"2024-09-21T10:11:41.498679+0000","last_fullsized":"2024-09-21T10:11:41.498679+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:58.368582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":127,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.491723+0000","last_change":"2024-09-21T10:08:38.380189+0000","last_active":"2024-09-21T10:11:41.491723+0000","last_peered":"2024-09-21T10:11:41.491723+0000","last_clean":"2024-09-21T10:11:41.491723+0000","last_became_active":"2024-09-21T10:08:38.379891+0000","last_became_peered":"2024-09-21T10:08:38.379891+0000","last_unstale":"2024-09-21T10:11:41.491723+0000","last_undegraded":"2024-09-21T10:11:41.491723+0000","last_fullsized":"2024-09-21T10:11:41.491723+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:35.820921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":207,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.489815+0000","last_change":"2024-09-21T10:08:38.392563+0000","last_active":"2024-09-21T10:11:41.489815+0000","last_peered":"2024-09-21T10:11:41.489815+0000","last_clean":"2024-09-21T10:11:41.489815+0000","last_became_active":"2024-09-21T10:08:38.378735+0000","last_became_peered":"2024-09-21T10:08:38.378735+0000","last_unstale":"2024-09-21T10:11:41.489815+0000","last_undegraded":"2024-09-21T10:11:41.489815+0000","last_fullsized":"2024-09-21T10:11:41.489815+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:29:41.423208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.491728+0000","last_change":"2024-09-21T10:11:05.453286+0000","last_active":"2024-09-21T10:11:41.491728+0000","last_peered":"2024-09-21T10:11:41.491728+0000","last_clean":"2024-09-21T10:11:41.491728+0000","last_became_active":"2024-09-21T10:11:05.452601+0000","last_became_peered":"2024-09-21T10:11:05.452601+0000","last_unstale":"2024-09-21T10:11:41.491728+0000","last_undegraded":"2024-09-21T10:11:41.491728+0000","last_fullsized":"2024-09-21T10:11:41.491728+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:27.532091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":48,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.862475+0000","last_change":"2024-09-21T10:10:27.833762+0000","last_active":"2024-09-21T10:12:00.862475+0000","last_peered":"2024-09-21T10:12:00.862475+0000","last_clean":"2024-09-21T10:12:00.862475+0000","last_became_active":"2024-09-21T10:10:27.819999+0000","last_became_peered":"2024-09-21T10:10:27.819999+0000","last_unstale":"2024-09-21T10:12:00.862475+0000","last_undegraded":"2024-09-21T10:12:00.862475+0000","last_fullsized":"2024-09-21T10:12:00.862475+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:33.839463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'100","reported_seq":204,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.865603+0000","last_change":"2024-09-21T10:07:24.345428+0000","last_active":"2024-09-21T10:12:00.865603+0000","last_peered":"2024-09-21T10:12:00.865603+0000","last_clean":"2024-09-21T10:12:00.865603+0000","last_became_active":"2024-09-21T10:07:24.343529+0000","last_became_peered":"2024-09-21T10:07:24.343529+0000","last_unstale":"2024-09-21T10:12:00.865603+0000","last_undegraded":"2024-09-21T10:12:00.865603+0000","last_fullsized":"2024-09-21T10:12:00.865603+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.014993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.014993+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:35.677888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.498940+0000","last_change":"2024-09-21T10:10:57.394885+0000","last_active":"2024-09-21T10:11:41.498940+0000","last_peered":"2024-09-21T10:11:41.498940+0000","last_clean":"2024-09-21T10:11:41.498940+0000","last_became_active":"2024-09-21T10:10:57.394584+0000","last_became_peered":"2024-09-21T10:10:57.394584+0000","last_unstale":"2024-09-21T10:11:41.498940+0000","last_undegraded":"2024-09-21T10:11:41.498940+0000","last_fullsized":"2024-09-21T10:11:41.498940+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:52:16.853713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":395,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.861180+0000","last_change":"2024-09-21T10:03:38.425111+0000","last_active":"2024-09-21T10:12:00.861180+0000","last_peered":"2024-09-21T10:12:00.861180+0000","last_clean":"2024-09-21T10:12:00.861180+0000","last_became_active":"2024-09-21T10:03:38.424704+0000","last_became_peered":"2024-09-21T10:03:38.424704+0000","last_unstale":"2024-09-21T10:12:00.861180+0000","last_undegraded":"2024-09-21T10:12:00.861180+0000","last_fullsized":"2024-09-21T10:12:00.861180+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:35.157076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":297,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.866007+0000","last_change":"2024-09-21T10:10:57.397755+0000","last_active":"2024-09-21T10:12:00.866007+0000","last_peered":"2024-09-21T10:12:00.866007+0000","last_clean":"2024-09-21T10:12:00.866007+0000","last_became_active":"2024-09-21T10:10:57.397249+0000","last_became_peered":"2024-09-21T10:10:57.397249+0000","last_unstale":"2024-09-21T10:12:00.866007+0000","last_undegraded":"2024-09-21T10:12:00.866007+0000","last_fullsized":"2024-09-21T10:12:00.866007+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:40.813503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":380,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.861180+0000","last_change":"2024-09-21T09:56:12.576756+0000","last_active":"2024-09-21T10:12:00.861180+0000","last_peered":"2024-09-21T10:12:00.861180+0000","last_clean":"2024-09-21T10:12:00.861180+0000","last_became_active":"2024-09-21T09:56:12.575752+0000","last_became_peered":"2024-09-21T09:56:12.575752+0000","last_unstale":"2024-09-21T10:12:00.861180+0000","last_undegraded":"2024-09-21T10:12:00.861180+0000","last_fullsized":"2024-09-21T10:12:00.861180+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:55.224839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":239,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.492310+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T10:11:41.492310+0000","last_peered":"2024-09-21T10:11:41.492310+0000","last_clean":"2024-09-21T10:11:41.492310+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T10:11:41.492310+0000","last_undegraded":"2024-09-21T10:11:41.492310+0000","last_fullsized":"2024-09-21T10:11:41.492310+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":416,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.863011+0000","last_change":"2024-09-21T10:09:49.928053+0000","last_active":"2024-09-21T10:12:00.863011+0000","last_peered":"2024-09-21T10:12:00.863011+0000","last_clean":"2024-09-21T10:12:00.863011+0000","last_became_active":"2024-09-21T10:09:49.927231+0000","last_became_peered":"2024-09-21T10:09:49.927231+0000","last_unstale":"2024-09-21T10:12:00.863011+0000","last_undegraded":"2024-09-21T10:12:00.863011+0000","last_fullsized":"2024-09-21T10:12:00.863011+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:26.987553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":329,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.867308+0000","last_change":"2024-09-21T10:09:23.840717+0000","last_active":"2024-09-21T10:12:00.867308+0000","last_peered":"2024-09-21T10:12:00.867308+0000","last_clean":"2024-09-21T10:12:00.867308+0000","last_became_active":"2024-09-21T10:09:23.840323+0000","last_became_peered":"2024-09-21T10:09:23.840323+0000","last_unstale":"2024-09-21T10:12:00.867308+0000","last_undegraded":"2024-09-21T10:12:00.867308+0000","last_fullsized":"2024-09-21T10:12:00.867308+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:55.769770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":42,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.860085+0000","last_change":"2024-09-21T10:10:27.816526+0000","last_active":"2024-09-21T10:12:00.860085+0000","last_peered":"2024-09-21T10:12:00.860085+0000","last_clean":"2024-09-21T10:12:00.860085+0000","last_became_active":"2024-09-21T10:10:27.816221+0000","last_became_peered":"2024-09-21T10:10:27.816221+0000","last_unstale":"2024-09-21T10:12:00.860085+0000","last_undegraded":"2024-09-21T10:12:00.860085+0000","last_fullsized":"2024-09-21T10:12:00.860085+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:08.331228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":387,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.862298+0000","last_change":"2024-09-21T10:08:38.379994+0000","last_active":"2024-09-21T10:12:00.862298+0000","last_peered":"2024-09-21T10:12:00.862298+0000","last_clean":"2024-09-21T10:12:00.862298+0000","last_became_active":"2024-09-21T10:08:38.378729+0000","last_became_peered":"2024-09-21T10:08:38.378729+0000","last_unstale":"2024-09-21T10:12:00.862298+0000","last_undegraded":"2024-09-21T10:12:00.862298+0000","last_fullsized":"2024-09-21T10:12:00.862298+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:16.940173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":107,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.860939+0000","last_change":"2024-09-21T10:09:24.844561+0000","last_active":"2024-09-21T10:12:00.860939+0000","last_peered":"2024-09-21T10:12:00.860939+0000","last_clean":"2024-09-21T10:12:00.860939+0000","last_became_active":"2024-09-21T10:09:24.844190+0000","last_became_peered":"2024-09-21T10:09:24.844190+0000","last_unstale":"2024-09-21T10:12:00.860939+0000","last_undegraded":"2024-09-21T10:12:00.860939+0000","last_fullsized":"2024-09-21T10:12:00.860939+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:46:04.775344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":434,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.865986+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:12:00.865986+0000","last_peered":"2024-09-21T10:12:00.865986+0000","last_clean":"2024-09-21T10:12:00.865986+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:12:00.865986+0000","last_undegraded":"2024-09-21T10:12:00.865986+0000","last_fullsized":"2024-09-21T10:12:00.865986+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":52,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.864796+0000","last_change":"2024-09-21T10:09:49.918246+0000","last_active":"2024-09-21T10:12:00.864796+0000","last_peered":"2024-09-21T10:12:00.864796+0000","last_clean":"2024-09-21T10:12:00.864796+0000","last_became_active":"2024-09-21T10:09:49.917911+0000","last_became_peered":"2024-09-21T10:09:49.917911+0000","last_unstale":"2024-09-21T10:12:00.864796+0000","last_undegraded":"2024-09-21T10:12:00.864796+0000","last_fullsized":"2024-09-21T10:12:00.864796+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:30:10.567650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":242,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.862055+0000","last_change":"2024-09-21T10:07:24.347937+0000","last_active":"2024-09-21T10:12:00.862055+0000","last_peered":"2024-09-21T10:12:00.862055+0000","last_clean":"2024-09-21T10:12:00.862055+0000","last_became_active":"2024-09-21T10:07:24.347376+0000","last_became_peered":"2024-09-21T10:07:24.347376+0000","last_unstale":"2024-09-21T10:12:00.862055+0000","last_undegraded":"2024-09-21T10:12:00.862055+0000","last_fullsized":"2024-09-21T10:12:00.862055+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:48:05.644824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":58,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.491517+0000","last_change":"2024-09-21T10:08:38.382518+0000","last_active":"2024-09-21T10:11:41.491517+0000","last_peered":"2024-09-21T10:11:41.491517+0000","last_clean":"2024-09-21T10:11:41.491517+0000","last_became_active":"2024-09-21T10:08:38.372051+0000","last_became_peered":"2024-09-21T10:08:38.372051+0000","last_unstale":"2024-09-21T10:11:41.491517+0000","last_undegraded":"2024-09-21T10:11:41.491517+0000","last_fullsized":"2024-09-21T10:11:41.491517+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:21.360435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":333,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.863925+0000","last_change":"2024-09-21T10:09:24.840173+0000","last_active":"2024-09-21T10:12:00.863925+0000","last_peered":"2024-09-21T10:12:00.863925+0000","last_clean":"2024-09-21T10:12:00.863925+0000","last_became_active":"2024-09-21T10:09:24.839762+0000","last_became_peered":"2024-09-21T10:09:24.839762+0000","last_unstale":"2024-09-21T10:12:00.863925+0000","last_undegraded":"2024-09-21T10:12:00.863925+0000","last_fullsized":"2024-09-21T10:12:00.863925+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:12:53.465203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":72,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.499050+0000","last_change":"2024-09-21T10:08:38.388561+0000","last_active":"2024-09-21T10:11:41.499050+0000","last_peered":"2024-09-21T10:11:41.499050+0000","last_clean":"2024-09-21T10:11:41.499050+0000","last_became_active":"2024-09-21T10:08:38.374558+0000","last_became_peered":"2024-09-21T10:08:38.374558+0000","last_unstale":"2024-09-21T10:11:41.499050+0000","last_undegraded":"2024-09-21T10:11:41.499050+0000","last_fullsized":"2024-09-21T10:11:41.499050+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:24.111438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":417,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.863067+0000","last_change":"2024-09-21T10:09:23.840467+0000","last_active":"2024-09-21T10:12:00.863067+0000","last_peered":"2024-09-21T10:12:00.863067+0000","last_clean":"2024-09-21T10:12:00.863067+0000","last_became_active":"2024-09-21T10:09:23.840177+0000","last_became_peered":"2024-09-21T10:09:23.840177+0000","last_unstale":"2024-09-21T10:12:00.863067+0000","last_undegraded":"2024-09-21T10:12:00.863067+0000","last_fullsized":"2024-09-21T10:12:00.863067+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:17.887307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":406,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.861737+0000","last_change":"2024-09-21T10:11:05.451613+0000","last_active":"2024-09-21T10:12:00.861737+0000","last_peered":"2024-09-21T10:12:00.861737+0000","last_clean":"2024-09-21T10:12:00.861737+0000","last_became_active":"2024-09-21T10:11:05.451075+0000","last_became_peered":"2024-09-21T10:11:05.451075+0000","last_unstale":"2024-09-21T10:12:00.861737+0000","last_undegraded":"2024-09-21T10:12:00.861737+0000","last_fullsized":"2024-09-21T10:12:00.861737+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:18.505045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":108,"reported_epoch":275,"state":"active+clean","last_fresh":"2024-09-21T10:11:41.491439+0000","last_change":"2024-09-21T10:08:38.380508+0000","last_active":"2024-09-21T10:11:41.491439+0000","last_peered":"2024-09-21T10:11:41.491439+0000","last_clean":"2024-09-21T10:11:41.491439+0000","last_became_active":"2024-09-21T10:08:38.380226+0000","last_became_peered":"2024-09-21T10:08:38.380226+0000","last_unstale":"2024-09-21T10:11:41.491439+0000","last_undegraded":"2024-09-21T10:11:41.491439+0000","last_fullsized":"2024-09-21T10:11:41.491439+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:50:38.190118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":72,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.865625+0000","last_change":"2024-09-21T10:10:57.401588+0000","last_active":"2024-09-21T10:12:00.865625+0000","last_peered":"2024-09-21T10:12:00.865625+0000","last_clean":"2024-09-21T10:12:00.865625+0000","last_became_active":"2024-09-21T10:10:57.397724+0000","last_became_peered":"2024-09-21T10:10:57.397724+0000","last_unstale":"2024-09-21T10:12:00.865625+0000","last_undegraded":"2024-09-21T10:12:00.865625+0000","last_fullsized":"2024-09-21T10:12:00.865625+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:47.843630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":4818010136,"num_objects":73518,"num_object_clones":0,"num_object_copies":220554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16456,"num_objects_unfound":0,"num_objects_dirty":73518,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151452,"num_write_kb":7198977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2791,"num_bytes_recovered":182910976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14695149568,"data_stored":14695137352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101252,"ondisk_log_size":101252,"up":48,"acting":47,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":11710,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26580,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":688762,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":49,"num_read_kb":125,"num_write":25,"num_write_kb":271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":722165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626837,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3254276,"kb_used_data":2349756,"kb_used_omap":1,"kb_used_meta":904510,"kb_avail":90490876,"statfs":{"total":95995035648,"available":92662657024,"internally_reserved":0,"allocated":2406150144,"data_stored":2404589170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":926218584},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019295,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":245,"seq":1052266987554,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2536564,"kb_used_data":2382636,"kb_used_omap":8,"kb_used_meta":153783,"kb_avail":91208588,"statfs":{"total":95995035648,"available":93397594112,"internally_reserved":0,"allocated":2439819264,"data_stored":2438270493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8940,"internal_metadata":157474068},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":175,"seq":751619276903,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2727104,"kb_used_data":2392504,"kb_used_omap":12,"kb_used_meta":334579,"kb_avail":91018048,"statfs":{"total":95995035648,"available":93202481152,"internally_reserved":0,"allocated":2449924096,"data_stored":2448367197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12845,"internal_metadata":342609363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823066,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1302320,"kb_used_data":117160,"kb_used_omap":1,"kb_used_meta":1185150,"kb_avail":92442832,"statfs":{"total":95995035648,"available":94661459968,"internally_reserved":0,"allocated":119971840,"data_stored":118427028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1213593944},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879489,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3248496,"kb_used_data":2445224,"kb_used_omap":4,"kb_used_meta":803195,"kb_avail":90496656,"statfs":{"total":95995035648,"available":92668575744,"internally_reserved":0,"allocated":2503909376,"data_stored":2502364645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":822472124},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566330,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2432200,"kb_used_data":1743808,"kb_used_omap":6,"kb_used_meta":688313,"kb_avail":91312952,"statfs":{"total":95995035648,"available":93504462848,"internally_reserved":0,"allocated":1785659392,"data_stored":1784095465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":704832935},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860674,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3910320,"kb_used_data":3051240,"kb_used_omap":5,"kb_used_meta":858874,"kb_avail":89834832,"statfs":{"total":95995035648,"available":91990867968,"internally_reserved":0,"allocated":3124469760,"data_stored":3122924137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":879487215},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7236,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9976,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2410807296,"data_stored":2410807296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3098611712,"data_stored":3098607640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1776881664,"data_stored":1776877592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2481782784,"data_stored":2481782784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":116260864,"data_stored":116260864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2386497536,"data_stored":2386493464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2424307712,"data_stored":2424307712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:12:11.955 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:11.961 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:11.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:11.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:12.055 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:12:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:12:11] "GET /metrics HTTP/1.1" 200 36813 "" "Prometheus/2.51.0" 2024-09-21T10:12:12.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:12.101 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:12.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:12.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:12.203 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:12.264 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:12.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:12.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:12.315 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:12:12.315 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:12.315 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:12 smithi107 ceph-mon[102221]: pgmap v976: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 890 op/s; 16456/220635 objects misplaced (7.458%); 3.3 MiB/s, 53 objects/s recovering 2024-09-21T10:12:12.315 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:12:12.315 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:12.315 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:12 smithi107 ceph-mon[105022]: pgmap v976: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 890 op/s; 16456/220635 objects misplaced (7.458%); 3.3 MiB/s, 53 objects/s recovering 2024-09-21T10:12:12.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:12.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:12.411 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:12.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:12.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:12.561 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:12.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:12.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:12.663 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:12:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:12 smithi129 ceph-mon[88471]: pgmap v976: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 890 op/s; 16456/220635 objects misplaced (7.458%); 3.3 MiB/s, 53 objects/s recovering 2024-09-21T10:12:12.698 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:12.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:12.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:12.800 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:12.831 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:12.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:12.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:12.933 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:12.964 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:12.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:12.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:13.066 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:13.104 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:13.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:13.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:13.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:13.244 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:13.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:13.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:13.346 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:13.388 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:13.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:13.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:13.490 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:13.517 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:13.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:13.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:13.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:13.657 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:13.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:13.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:13.759 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:13.789 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:13.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:13.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:13.891 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:13.916 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:13.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:13.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:14.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:14.054 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:14.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:14.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:14.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:14.201 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:14.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:14.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:14.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:14.340 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:14.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:14.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:14.441 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:14.469 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:14.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:14.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:14.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:14 smithi107 ceph-mon[102221]: pgmap v977: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 15377/210255 objects misplaced (7.314%); 4.7 MiB/s, 75 objects/s recovering 2024-09-21T10:12:14.611 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:14 smithi107 ceph-mon[105022]: pgmap v977: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 15377/210255 objects misplaced (7.314%); 4.7 MiB/s, 75 objects/s recovering 2024-09-21T10:12:14.613 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:14.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:14.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:14 smithi129 ceph-mon[88471]: pgmap v977: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 15377/210255 objects misplaced (7.314%); 4.7 MiB/s, 75 objects/s recovering 2024-09-21T10:12:14.715 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:14.750 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:14.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:14.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:14.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:14.892 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:14.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:14.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:14.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:12:14.993 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:15.021 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:15.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:15.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:15.047 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:15.122 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:15.145 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:15.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:15.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:15.247 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:15.273 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:15.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:15.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:15.375 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:15.403 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:15.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:15.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:15.505 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:15.532 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:15.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:15.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:15.634 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:15.664 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:15.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:15.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:15.766 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:15.791 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:15.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:15.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:15.893 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:15.923 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:15.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:15.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:16.024 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:16.054 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:16.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:16.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:16.156 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:16.183 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:16.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:16.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:16.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:16.316 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:16.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:16.417 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:16.439 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:16.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:16.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:16.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:16.573 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:16.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:16.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:16.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:16 smithi107 ceph-mon[102221]: pgmap v978: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 818 op/s; 15377/207795 objects misplaced (7.400%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:12:16.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:16 smithi107 ceph-mon[105022]: pgmap v978: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 818 op/s; 15377/207795 objects misplaced (7.400%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:12:16.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:16 smithi129 ceph-mon[88471]: pgmap v978: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 818 op/s; 15377/207795 objects misplaced (7.400%); 3.3 MiB/s, 52 objects/s recovering 2024-09-21T10:12:16.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:16.696 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:16.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:16.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:16.798 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:16.824 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:16.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:16.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:16.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:16.964 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:16.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:16.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:17.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:17.103 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:17.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:17.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:17.205 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:17.237 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:17.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:17.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:17.339 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:17.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:17.369 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:17.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:17.470 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:17.512 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:17.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:17.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:17.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:17.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:17.707 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:17.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:17.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:17.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:17.852 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:17.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:17.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:17.993 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:17.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:17.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:18.095 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:18.122 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:18.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:18.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:18.224 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:18.266 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:18.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:18.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:18.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:18.374 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:12:18.416 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:18.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:18.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:18.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:18 smithi107 ceph-mon[102221]: pgmap v979: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 995 op/s; 14212/198870 objects misplaced (7.146%); 5.3 MiB/s, 84 objects/s recovering 2024-09-21T10:12:18.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:18 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/69291068' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:12:18.446 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:18 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1535050456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:18.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:18 smithi107 ceph-mon[105022]: pgmap v979: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 995 op/s; 14212/198870 objects misplaced (7.146%); 5.3 MiB/s, 84 objects/s recovering 2024-09-21T10:12:18.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:18 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/69291068' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:12:18.447 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:18 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1535050456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:18.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:18.557 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:18.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:18.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:18.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:18 smithi129 ceph-mon[88471]: pgmap v979: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 995 op/s; 14212/198870 objects misplaced (7.146%); 5.3 MiB/s, 84 objects/s recovering 2024-09-21T10:12:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:18 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/69291068' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:12:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:18 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1535050456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:18.700 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:18.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:18.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:18.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:18.839 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:18.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:18.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:18.941 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:18.982 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:18.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:18.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:19.084 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:19.114 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:19.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:19.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:19.216 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:19.247 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:19.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:19.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:19.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:19.396 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:19.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:19.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:19.497 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:19.524 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:19.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:19.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:19.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:19 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:12:19.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:19 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/69291068' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:12:19.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:19 smithi107 ceph-mon[102221]: osdmap e277: 8 total, 8 up, 7 in 2024-09-21T10:12:19.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:19 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:12:19.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:19 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/69291068' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:12:19.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:19 smithi107 ceph-mon[105022]: osdmap e277: 8 total, 8 up, 7 in 2024-09-21T10:12:19.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:19.667 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:19.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:19.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:19 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:12:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:19 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/69291068' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:12:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:19 smithi129 ceph-mon[88471]: osdmap e277: 8 total, 8 up, 7 in 2024-09-21T10:12:19.722 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:12:19.722 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:12:19.722 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:12:19.722 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:12:19.722 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:12:19.722 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:12:19.722 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:12:19.722 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:12:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:12:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:12:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:12:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:12:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:12:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:12:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:12:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:12:19.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:19.813 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:19.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:19.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:19.915 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:19.951 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:19.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:19.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:20.053 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:20.087 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:20.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:20.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:20.189 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:20.215 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:20.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:20.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:20.316 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:20.344 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:20.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:20.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:20.445 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:20.473 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:20.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:20.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:20.574 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:20.580 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:20 smithi107 ceph-mon[102221]: pgmap v981: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 14212/194784 objects misplaced (7.296%); 4.6 MiB/s, 72 objects/s recovering 2024-09-21T10:12:20.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:20 smithi107 ceph-mon[105022]: pgmap v981: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 14212/194784 objects misplaced (7.296%); 4.6 MiB/s, 72 objects/s recovering 2024-09-21T10:12:20.602 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:20.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:20.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:20.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:20 smithi129 ceph-mon[88471]: pgmap v981: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 14212/194784 objects misplaced (7.296%); 4.6 MiB/s, 72 objects/s recovering 2024-09-21T10:12:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:12:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:12:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:12:20.705 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:20.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:20.741 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:20.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:20.842 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:20.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:20.869 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:20.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:20.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:21.009 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:21.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:21.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:21.111 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:21.153 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:21.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:21.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:21.254 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:21.290 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:12:21.290 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:12:21.291 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-09-21T10:12:21.291 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-09-21T10:12:21.291 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:12:21.296 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:21.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:21.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:21.398 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:21.427 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:21.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:21.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:21.530 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:21.555 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:21.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:21.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:21.657 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:21.690 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:21.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:21.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:21.792 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:21.824 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:21.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:21.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:21.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:21.963 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:21.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:22.008 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:12:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:12:21] "GET /metrics HTTP/1.1" 200 36813 "" "Prometheus/2.51.0" 2024-09-21T10:12:22.064 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:22.094 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:22.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:22.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:22.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:22.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:22.233 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:22.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:22.334 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:22.365 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:22.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:22.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:22.467 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:22.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:22.495 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:22.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:22.563 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:12:22.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:22.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:22.643 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:22.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:22.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:22 smithi129 ceph-mon[88471]: pgmap v982: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 14212/194784 objects misplaced (7.296%); 4.6 MiB/s, 72 objects/s recovering 2024-09-21T10:12:22.743 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:22.822 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:22.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:22.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:22.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:22 smithi107 ceph-mon[102221]: pgmap v982: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 14212/194784 objects misplaced (7.296%); 4.6 MiB/s, 72 objects/s recovering 2024-09-21T10:12:22.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:22 smithi107 ceph-mon[105022]: pgmap v982: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 14212/194784 objects misplaced (7.296%); 4.6 MiB/s, 72 objects/s recovering 2024-09-21T10:12:22.861 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:22.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:22.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:22.977 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:22.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:23.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:12:23.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:23.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:23.140 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:23.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:23.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:23.306 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:23.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:23.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:23.408 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:23.440 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:23.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:23.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:23.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:23.638 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:23.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:23.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:23.740 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:23.765 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:23.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:23.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:23.867 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:23.897 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:23.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:23.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:23.998 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:24.032 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:24.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:24.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:24.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:12:24.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:24.181 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:24.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:24.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:24.286 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:24.307 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:24.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:24.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:24.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:24.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:24.442 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:24.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:24.543 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:24.570 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:24.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:24.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:24.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:24 smithi129 ceph-mon[88471]: pgmap v983: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 884 op/s; 12950/183711 objects misplaced (7.049%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:12:24.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:24.710 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:24.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:24.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:24.811 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:24.848 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:24.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:24.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:24.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:24 smithi107 ceph-mon[105022]: pgmap v983: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 884 op/s; 12950/183711 objects misplaced (7.049%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:12:24.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:24 smithi107 ceph-mon[102221]: pgmap v983: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 884 op/s; 12950/183711 objects misplaced (7.049%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:12:24.949 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:24.988 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:24.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:24.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:25.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:25.126 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:25.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:25.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:25.228 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:25.256 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:25.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:25.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:25.358 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:25.386 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:25.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:25.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:25.488 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:25.515 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:25.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:25.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:25.617 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:25.646 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:25.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:25.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:25.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:25.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:25.781 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:25.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:25.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:25.918 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:25.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:25.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:26.019 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:26.047 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:26.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:26.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:26.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:26.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:26.210 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:26.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:26.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:26.335 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:26.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:26.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:26.436 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:26.465 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:26.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:26.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:26.567 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:26.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:26.595 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:26.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:26.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:26 smithi107 ceph-mon[102221]: pgmap v984: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 891 op/s; 12950/181050 objects misplaced (7.153%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:12:26.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:12:26.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:26.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:26 smithi107 ceph-mon[105022]: pgmap v984: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 891 op/s; 12950/181050 objects misplaced (7.153%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:12:26.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:12:26.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:26 smithi129 ceph-mon[88471]: pgmap v984: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 891 op/s; 12950/181050 objects misplaced (7.153%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:12:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:12:26.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:26.697 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:26.723 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:26.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:26.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:26.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:26.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:26.875 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:26.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:26.976 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:27.004 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:27.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:27.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:27.096 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:27.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:27.171 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:27.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:27.180 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:27.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:27.182 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:27.281 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:27.307 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:27.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:27.409 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:27.436 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:27.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:27.537 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:27.566 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:27.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:27.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:27.581 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:12:27.581 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:12:27.581 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:12:27.581 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:12:27.581 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:12:27.581 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:12:27.581 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:12:27.581 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:12:27.581 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:12:27.582 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:12:27.668 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:27.696 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:27.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:27.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:27.798 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:27.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:27.837 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:27.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:12:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:12:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:12:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:12:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:12:27.938 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:27.964 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:27.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:27.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:28.066 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:28.091 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:28.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:28.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:28.192 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:28.215 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:28.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:28.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:28.317 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:28.345 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:28.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:28.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:28.447 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:28.489 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:28.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:28.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:28.591 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:28.619 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:28.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:28.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:28.721 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:28.748 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:28.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:28.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:28.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:28.856 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:28 smithi107 ceph-mon[102221]: pgmap v985: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 899 op/s; 11574/171882 objects misplaced (6.734%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:12:28.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:28 smithi107 ceph-mon[105022]: pgmap v985: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 899 op/s; 11574/171882 objects misplaced (6.734%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:12:28.888 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:28.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:28.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:28.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:28 smithi129 ceph-mon[88471]: pgmap v985: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 899 op/s; 11574/171882 objects misplaced (6.734%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:12:28.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:29.029 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:29.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:29.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:29.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:29.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:29.172 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:29.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:29.273 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:29.295 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:12:29.299 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:12:29.314 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:29.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:29.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:29.416 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:29.458 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:29.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:29.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:29.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:29.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:29.613 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:29.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:29.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:29.748 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:29.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:29.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:29.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:29.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:29 smithi107 ceph-mon[102221]: from='client.46484 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:12:29.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:29 smithi107 ceph-mon[105022]: from='client.46484 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:12:29.884 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:29.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:29.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:29 smithi129 ceph-mon[88471]: from='client.46484 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:12:29.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:30.030 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:30.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:30.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:30.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:30.173 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:30.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:30.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:30.274 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:30.306 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:30.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:30.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:30.410 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:30.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:30.439 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:30.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:30.540 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:30.570 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:12:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:12:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:12:30.576 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:30.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:30.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:30.678 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:30.706 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:30.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:30.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:30.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:30.847 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:30.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:30.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:30.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:30 smithi107 ceph-mon[102221]: pgmap v986: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.4 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 895 op/s; 11574/167925 objects misplaced (6.892%); 5.4 MiB/s, 86 objects/s recovering 2024-09-21T10:12:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:30 smithi107 ceph-mon[105022]: pgmap v986: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.4 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 895 op/s; 11574/167925 objects misplaced (6.892%); 5.4 MiB/s, 86 objects/s recovering 2024-09-21T10:12:30.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:30 smithi129 ceph-mon[88471]: pgmap v986: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.4 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 895 op/s; 11574/167925 objects misplaced (6.892%); 5.4 MiB/s, 86 objects/s recovering 2024-09-21T10:12:30.949 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:30.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:30.988 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:30.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:31.089 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:31.135 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:31.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:31.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:31.237 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:31.266 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:31.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:31.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:31.368 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:31.396 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:31.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:31.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:31.498 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:31.528 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:31.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:31.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:31.630 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:31.657 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:31.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:31.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:31.687 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":985,"stamp":"2024-09-21T10:12:27.294096+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3753678911,"num_objects":57294,"num_object_clones":0,"num_object_copies":171882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11574,"num_objects_unfound":0,"num_objects_dirty":57294,"num_whiteouts":0,"num_read":50,"num_read_kb":126,"num_write":167792,"num_write_kb":7199280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4527,"num_bytes_recovered":293483187,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103562,"ondisk_log_size":103562,"up":327,"acting":326,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":331,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":16402624,"kb_used_data":11450248,"kb_used_omap":41,"kb_used_meta":4952022,"kb_avail":639813440,"statfs":{"total":671965249536,"available":655168962560,"internally_reserved":0,"allocated":11725053952,"data_stored":11714133377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42519,"internal_metadata":5070871017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,8,11,5,5,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-589561856,"num_objects":-8996,"num_object_clones":0,"num_object_copies":-26988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2638,"num_objects_unfound":0,"num_objects_dirty":-8996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8996,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":947,"num_bytes_recovered":62062592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002775"},"pg_stats":[{"pgid":"1.11","version":"77'100","reported_seq":70,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.400415+0000","last_change":"2024-09-21T10:10:57.397786+0000","last_active":"2024-09-21T10:12:18.400415+0000","last_peered":"2024-09-21T10:12:18.400415+0000","last_clean":"2024-09-21T10:12:18.400415+0000","last_became_active":"2024-09-21T10:10:57.396742+0000","last_became_peered":"2024-09-21T10:10:57.396742+0000","last_unstale":"2024-09-21T10:12:18.400415+0000","last_undegraded":"2024-09-21T10:12:18.400415+0000","last_fullsized":"2024-09-21T10:12:18.400415+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:53.631923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:53.631923+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:33:35.364595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":407,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.394692+0000","last_change":"2024-09-21T10:03:38.389577+0000","last_active":"2024-09-21T10:12:18.394692+0000","last_peered":"2024-09-21T10:12:18.394692+0000","last_clean":"2024-09-21T10:12:18.394692+0000","last_became_active":"2024-09-21T10:03:38.389292+0000","last_became_peered":"2024-09-21T10:03:38.389292+0000","last_unstale":"2024-09-21T10:12:18.394692+0000","last_undegraded":"2024-09-21T10:12:18.394692+0000","last_fullsized":"2024-09-21T10:12:18.394692+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:32.113472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":147,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.843508+0000","last_change":"2024-09-21T10:08:38.387871+0000","last_active":"2024-09-21T10:12:00.843508+0000","last_peered":"2024-09-21T10:12:00.843508+0000","last_clean":"2024-09-21T10:12:00.843508+0000","last_became_active":"2024-09-21T10:08:38.376727+0000","last_became_peered":"2024-09-21T10:08:38.376727+0000","last_unstale":"2024-09-21T10:12:00.843508+0000","last_undegraded":"2024-09-21T10:12:00.843508+0000","last_fullsized":"2024-09-21T10:12:00.843508+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:45:54.498950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'100","reported_seq":176,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.843120+0000","last_change":"2024-09-21T10:11:05.438031+0000","last_active":"2024-09-21T10:12:00.843120+0000","last_peered":"2024-09-21T10:12:00.843120+0000","last_clean":"2024-09-21T10:12:00.843120+0000","last_became_active":"2024-09-21T10:11:05.437637+0000","last_became_peered":"2024-09-21T10:11:05.437637+0000","last_unstale":"2024-09-21T10:12:00.843120+0000","last_undegraded":"2024-09-21T10:12:00.843120+0000","last_fullsized":"2024-09-21T10:12:00.843120+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.798577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.798577+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:42.343217+0000","scrub_duration":0.0023622640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":268,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.846033+0000","last_change":"2024-09-21T10:10:57.392782+0000","last_active":"2024-09-21T10:12:00.846033+0000","last_peered":"2024-09-21T10:12:00.846033+0000","last_clean":"2024-09-21T10:12:00.846033+0000","last_became_active":"2024-09-21T10:10:57.392305+0000","last_became_peered":"2024-09-21T10:10:57.392305+0000","last_unstale":"2024-09-21T10:12:00.846033+0000","last_undegraded":"2024-09-21T10:12:00.846033+0000","last_fullsized":"2024-09-21T10:12:00.846033+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:42:01.387622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":329,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.413750+0000","last_change":"2024-09-21T10:09:23.846703+0000","last_active":"2024-09-21T10:12:18.413750+0000","last_peered":"2024-09-21T10:12:18.413750+0000","last_clean":"2024-09-21T10:12:18.413750+0000","last_became_active":"2024-09-21T10:09:23.846415+0000","last_became_peered":"2024-09-21T10:09:23.846415+0000","last_unstale":"2024-09-21T10:12:18.413750+0000","last_undegraded":"2024-09-21T10:12:18.413750+0000","last_fullsized":"2024-09-21T10:12:18.413750+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:23.129285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'100","reported_seq":123,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.394215+0000","last_change":"2024-09-21T10:09:24.841480+0000","last_active":"2024-09-21T10:12:18.394215+0000","last_peered":"2024-09-21T10:12:18.394215+0000","last_clean":"2024-09-21T10:12:18.394215+0000","last_became_active":"2024-09-21T10:09:24.841145+0000","last_became_peered":"2024-09-21T10:09:24.841145+0000","last_unstale":"2024-09-21T10:12:18.394215+0000","last_undegraded":"2024-09-21T10:12:18.394215+0000","last_fullsized":"2024-09-21T10:12:18.394215+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.433227+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":23,"num_write":5,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":35780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":55,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.401178+0000","last_change":"2024-09-21T10:09:49.922489+0000","last_active":"2024-09-21T10:12:18.401178+0000","last_peered":"2024-09-21T10:12:18.401178+0000","last_clean":"2024-09-21T10:12:18.401178+0000","last_became_active":"2024-09-21T10:09:49.921629+0000","last_became_peered":"2024-09-21T10:09:49.921629+0000","last_unstale":"2024-09-21T10:12:18.401178+0000","last_undegraded":"2024-09-21T10:12:18.401178+0000","last_fullsized":"2024-09-21T10:12:18.401178+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:50.611979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":149,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.843300+0000","last_change":"2024-09-21T10:09:24.836647+0000","last_active":"2024-09-21T10:12:00.843300+0000","last_peered":"2024-09-21T10:12:00.843300+0000","last_clean":"2024-09-21T10:12:00.843300+0000","last_became_active":"2024-09-21T10:09:24.836278+0000","last_became_peered":"2024-09-21T10:09:24.836278+0000","last_unstale":"2024-09-21T10:12:00.843300+0000","last_undegraded":"2024-09-21T10:12:00.843300+0000","last_fullsized":"2024-09-21T10:12:00.843300+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:08:36.535892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"77'100","reported_seq":20,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.872443+0000","last_change":"2024-09-21T10:10:57.452531+0000","last_active":"2024-09-21T10:12:00.872443+0000","last_peered":"2024-09-21T10:12:00.872443+0000","last_clean":"2024-09-21T10:12:00.872443+0000","last_became_active":"2024-09-21T10:10:57.452150+0000","last_became_peered":"2024-09-21T10:10:57.452150+0000","last_unstale":"2024-09-21T10:12:00.872443+0000","last_undegraded":"2024-09-21T10:12:00.872443+0000","last_fullsized":"2024-09-21T10:12:00.872443+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:00.058077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:00.058077+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":407,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.393226+0000","last_change":"2024-09-21T10:03:38.381552+0000","last_active":"2024-09-21T10:12:18.393226+0000","last_peered":"2024-09-21T10:12:18.393226+0000","last_clean":"2024-09-21T10:12:18.393226+0000","last_became_active":"2024-09-21T10:03:38.380898+0000","last_became_peered":"2024-09-21T10:03:38.380898+0000","last_unstale":"2024-09-21T10:12:18.393226+0000","last_undegraded":"2024-09-21T10:12:18.393226+0000","last_fullsized":"2024-09-21T10:12:18.393226+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:53.409606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":336,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.852345+0000","last_change":"2024-09-21T10:10:57.459057+0000","last_active":"2024-09-21T10:12:00.852345+0000","last_peered":"2024-09-21T10:12:00.852345+0000","last_clean":"2024-09-21T10:12:00.852345+0000","last_became_active":"2024-09-21T10:10:57.451776+0000","last_became_peered":"2024-09-21T10:10:57.451776+0000","last_unstale":"2024-09-21T10:12:00.852345+0000","last_undegraded":"2024-09-21T10:12:00.852345+0000","last_fullsized":"2024-09-21T10:12:00.852345+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:00.138317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"277'10473","reported_seq":10579,"reported_epoch":277,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:12:26.996609+0000","last_change":"2024-09-21T10:10:58.805957+0000","last_active":"2024-09-21T10:12:26.996609+0000","last_peered":"2024-09-21T10:12:26.996609+0000","last_clean":"2024-09-21T10:10:56.364182+0000","last_became_active":"2024-09-21T10:10:58.682930+0000","last_became_peered":"2024-09-21T10:10:58.682930+0000","last_unstale":"2024-09-21T10:12:26.996609+0000","last_undegraded":"2024-09-21T10:12:26.996609+0000","last_fullsized":"2024-09-21T10:12:26.996609+0000","mapping_epoch":267,"log_start":"263'4200","ondisk_log_start":"263'4200","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6273,"log_dups_size":0,"ondisk_log_size":6273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230096896,"num_objects":3511,"num_object_clones":0,"num_object_copies":10533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3498,"num_objects_unfound":0,"num_objects_dirty":3511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10473,"num_write_kb":447488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[{"shards":"0,1,7","objects":3511}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"77'100","reported_seq":161,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.845993+0000","last_change":"2024-09-21T10:08:38.374573+0000","last_active":"2024-09-21T10:12:00.845993+0000","last_peered":"2024-09-21T10:12:00.845993+0000","last_clean":"2024-09-21T10:12:00.845993+0000","last_became_active":"2024-09-21T10:08:38.374075+0000","last_became_peered":"2024-09-21T10:08:38.374075+0000","last_unstale":"2024-09-21T10:12:00.845993+0000","last_undegraded":"2024-09-21T10:12:00.845993+0000","last_fullsized":"2024-09-21T10:12:00.845993+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:55.259385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:55.259385+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:28:26.784162+0000","scrub_duration":0.0015667260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":192,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.843668+0000","last_change":"2024-09-21T10:09:49.941392+0000","last_active":"2024-09-21T10:12:00.843668+0000","last_peered":"2024-09-21T10:12:00.843668+0000","last_clean":"2024-09-21T10:12:00.843668+0000","last_became_active":"2024-09-21T10:09:49.923535+0000","last_became_peered":"2024-09-21T10:09:49.923535+0000","last_unstale":"2024-09-21T10:12:00.843668+0000","last_undegraded":"2024-09-21T10:12:00.843668+0000","last_fullsized":"2024-09-21T10:12:00.843668+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:16.292163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":30,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.872055+0000","last_change":"2024-09-21T10:11:05.435502+0000","last_active":"2024-09-21T10:12:00.872055+0000","last_peered":"2024-09-21T10:12:00.872055+0000","last_clean":"2024-09-21T10:12:00.872055+0000","last_became_active":"2024-09-21T10:11:05.435216+0000","last_became_peered":"2024-09-21T10:11:05.435216+0000","last_unstale":"2024-09-21T10:12:00.872055+0000","last_undegraded":"2024-09-21T10:12:00.872055+0000","last_fullsized":"2024-09-21T10:12:00.872055+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:25.434424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"277'10362","reported_seq":10418,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:26.968858+0000","last_change":"2024-09-21T10:09:49.924187+0000","last_active":"2024-09-21T10:12:26.968858+0000","last_peered":"2024-09-21T10:12:26.968858+0000","last_clean":"2024-09-21T10:12:26.968858+0000","last_became_active":"2024-09-21T10:09:49.923789+0000","last_became_peered":"2024-09-21T10:09:49.923789+0000","last_unstale":"2024-09-21T10:12:26.968858+0000","last_undegraded":"2024-09-21T10:12:26.968858+0000","last_fullsized":"2024-09-21T10:12:26.968858+0000","mapping_epoch":255,"log_start":"263'4100","ondisk_log_start":"263'4100","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6262,"log_dups_size":0,"ondisk_log_size":6262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:52:49.559425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229638144,"num_objects":3504,"num_object_clones":0,"num_object_copies":10512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10362,"num_write_kb":443712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"77'100","reported_seq":214,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.394549+0000","last_change":"2024-09-21T10:09:23.839211+0000","last_active":"2024-09-21T10:12:18.394549+0000","last_peered":"2024-09-21T10:12:18.394549+0000","last_clean":"2024-09-21T10:12:18.394549+0000","last_became_active":"2024-09-21T10:09:23.838779+0000","last_became_peered":"2024-09-21T10:09:23.838779+0000","last_unstale":"2024-09-21T10:12:18.394549+0000","last_undegraded":"2024-09-21T10:12:18.394549+0000","last_fullsized":"2024-09-21T10:12:18.394549+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.981336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.981336+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:03:15.566873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":171,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.843764+0000","last_change":"2024-09-21T10:09:49.932631+0000","last_active":"2024-09-21T10:12:00.843764+0000","last_peered":"2024-09-21T10:12:00.843764+0000","last_clean":"2024-09-21T10:12:00.843764+0000","last_became_active":"2024-09-21T10:09:49.922897+0000","last_became_peered":"2024-09-21T10:09:49.922897+0000","last_unstale":"2024-09-21T10:12:00.843764+0000","last_undegraded":"2024-09-21T10:12:00.843764+0000","last_fullsized":"2024-09-21T10:12:00.843764+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:40.795476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":202,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.397238+0000","last_change":"2024-09-21T10:08:38.377732+0000","last_active":"2024-09-21T10:12:18.397238+0000","last_peered":"2024-09-21T10:12:18.397238+0000","last_clean":"2024-09-21T10:12:18.397238+0000","last_became_active":"2024-09-21T10:08:38.376601+0000","last_became_peered":"2024-09-21T10:08:38.376601+0000","last_unstale":"2024-09-21T10:12:18.397238+0000","last_undegraded":"2024-09-21T10:12:18.397238+0000","last_fullsized":"2024-09-21T10:12:18.397238+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:02.540158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.5","version":"277'10156","reported_seq":19095,"reported_epoch":277,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T10:12:26.833598+0000","last_change":"2024-09-21T10:10:58.840575+0000","last_active":"2024-09-21T10:12:26.833598+0000","last_peered":"2024-09-21T10:12:26.833598+0000","last_clean":"2024-09-21T10:10:56.374526+0000","last_became_active":"2024-09-21T10:10:58.733275+0000","last_became_peered":"2024-09-21T10:10:58.733275+0000","last_unstale":"2024-09-21T10:12:26.833598+0000","last_undegraded":"2024-09-21T10:12:26.833598+0000","last_fullsized":"2024-09-21T10:10:58.388641+0000","mapping_epoch":267,"log_start":"263'3900","ondisk_log_start":"263'3900","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6256,"log_dups_size":0,"ondisk_log_size":6256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:21:07.802295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220200960,"num_objects":3360,"num_object_clones":0,"num_object_copies":10080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1029,"num_objects_unfound":0,"num_objects_dirty":3360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10156,"num_write_kb":432512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4449,"num_bytes_recovered":291569664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,6],"avail_no_missing":["7","3","6"],"object_location_counts":[{"shards":"3,6,7","objects":1029},{"shards":"6,7","objects":2331}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"77'100","reported_seq":261,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.851854+0000","last_change":"2024-09-21T10:08:38.382100+0000","last_active":"2024-09-21T10:12:00.851854+0000","last_peered":"2024-09-21T10:12:00.851854+0000","last_clean":"2024-09-21T10:12:00.851854+0000","last_became_active":"2024-09-21T10:08:38.381736+0000","last_became_peered":"2024-09-21T10:08:38.381736+0000","last_unstale":"2024-09-21T10:12:00.851854+0000","last_undegraded":"2024-09-21T10:12:00.851854+0000","last_fullsized":"2024-09-21T10:12:00.851854+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.289859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.289859+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:23.739980+0000","scrub_duration":0.001251809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":157,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.845904+0000","last_change":"2024-09-21T10:10:57.460732+0000","last_active":"2024-09-21T10:12:00.845904+0000","last_peered":"2024-09-21T10:12:00.845904+0000","last_clean":"2024-09-21T10:12:00.845904+0000","last_became_active":"2024-09-21T10:10:57.454689+0000","last_became_peered":"2024-09-21T10:10:57.454689+0000","last_unstale":"2024-09-21T10:12:00.845904+0000","last_undegraded":"2024-09-21T10:12:00.845904+0000","last_fullsized":"2024-09-21T10:12:00.845904+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:12.035022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":277,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.852611+0000","last_change":"2024-09-21T10:03:38.378384+0000","last_active":"2024-09-21T10:12:00.852611+0000","last_peered":"2024-09-21T10:12:00.852611+0000","last_clean":"2024-09-21T10:12:00.852611+0000","last_became_active":"2024-09-21T10:03:38.378072+0000","last_became_peered":"2024-09-21T10:03:38.378072+0000","last_unstale":"2024-09-21T10:12:00.852611+0000","last_undegraded":"2024-09-21T10:12:00.852611+0000","last_fullsized":"2024-09-21T10:12:00.852611+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:54:37.039622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"277'10616","reported_seq":10649,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:26.827150+0000","last_change":"2024-09-21T10:09:42.896949+0000","last_active":"2024-09-21T10:12:26.827150+0000","last_peered":"2024-09-21T10:12:26.827150+0000","last_clean":"2024-09-21T10:12:26.827150+0000","last_became_active":"2024-09-21T10:09:42.896669+0000","last_became_peered":"2024-09-21T10:09:42.896669+0000","last_unstale":"2024-09-21T10:12:26.827150+0000","last_undegraded":"2024-09-21T10:12:26.827150+0000","last_fullsized":"2024-09-21T10:12:26.827150+0000","mapping_epoch":252,"log_start":"263'4300","ondisk_log_start":"263'4300","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6316,"log_dups_size":0,"ondisk_log_size":6316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:44.279472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236453888,"num_objects":3608,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10616,"num_write_kb":455168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"77'100","reported_seq":129,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.843598+0000","last_change":"2024-09-21T10:07:26.188298+0000","last_active":"2024-09-21T10:12:00.843598+0000","last_peered":"2024-09-21T10:12:00.843598+0000","last_clean":"2024-09-21T10:12:00.843598+0000","last_became_active":"2024-09-21T10:07:24.345265+0000","last_became_peered":"2024-09-21T10:07:24.345265+0000","last_unstale":"2024-09-21T10:12:00.843598+0000","last_undegraded":"2024-09-21T10:12:00.843598+0000","last_fullsized":"2024-09-21T10:12:00.843598+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:07:26.188220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:07:26.188220+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:46.381710+0000","scrub_duration":0.021884888000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":211,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.394830+0000","last_change":"2024-09-21T10:07:57.551556+0000","last_active":"2024-09-21T10:12:18.394830+0000","last_peered":"2024-09-21T10:12:18.394830+0000","last_clean":"2024-09-21T10:12:18.394830+0000","last_became_active":"2024-09-21T10:07:57.551249+0000","last_became_peered":"2024-09-21T10:07:57.551249+0000","last_unstale":"2024-09-21T10:12:18.394830+0000","last_undegraded":"2024-09-21T10:12:18.394830+0000","last_fullsized":"2024-09-21T10:12:18.394830+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:57.557152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":309,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.400648+0000","last_change":"2024-09-21T10:09:24.839665+0000","last_active":"2024-09-21T10:12:18.400648+0000","last_peered":"2024-09-21T10:12:18.400648+0000","last_clean":"2024-09-21T10:12:18.400648+0000","last_became_active":"2024-09-21T10:09:24.839353+0000","last_became_peered":"2024-09-21T10:09:24.839353+0000","last_unstale":"2024-09-21T10:12:18.400648+0000","last_undegraded":"2024-09-21T10:12:18.400648+0000","last_fullsized":"2024-09-21T10:12:18.400648+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:51.760073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3","version":"277'10067","reported_seq":10100,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:22.790540+0000","last_change":"2024-09-21T10:09:43.231199+0000","last_active":"2024-09-21T10:12:22.790540+0000","last_peered":"2024-09-21T10:12:22.790540+0000","last_clean":"2024-09-21T10:12:22.790540+0000","last_became_active":"2024-09-21T10:09:43.230922+0000","last_became_peered":"2024-09-21T10:09:43.230922+0000","last_unstale":"2024-09-21T10:12:22.790540+0000","last_undegraded":"2024-09-21T10:12:22.790540+0000","last_fullsized":"2024-09-21T10:12:22.790540+0000","mapping_epoch":252,"log_start":"263'3900","ondisk_log_start":"263'3900","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6167,"log_dups_size":0,"ondisk_log_size":6167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:39.944122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249757696,"num_objects":3811,"num_object_clones":0,"num_object_copies":11433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10067,"num_write_kb":444096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":151,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.843448+0000","last_change":"2024-09-21T10:04:53.355975+0000","last_active":"2024-09-21T10:12:00.843448+0000","last_peered":"2024-09-21T10:12:00.843448+0000","last_clean":"2024-09-21T10:12:00.843448+0000","last_became_active":"2024-09-21T10:04:42.547505+0000","last_became_peered":"2024-09-21T10:04:42.547505+0000","last_unstale":"2024-09-21T10:12:00.843448+0000","last_undegraded":"2024-09-21T10:12:00.843448+0000","last_fullsized":"2024-09-21T10:12:00.843448+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.355912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.355912+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:43.456830+0000","scrub_duration":0.0017980030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":408,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.394373+0000","last_change":"2024-09-21T10:09:49.939553+0000","last_active":"2024-09-21T10:12:18.394373+0000","last_peered":"2024-09-21T10:12:18.394373+0000","last_clean":"2024-09-21T10:12:18.394373+0000","last_became_active":"2024-09-21T10:09:49.925094+0000","last_became_peered":"2024-09-21T10:09:49.925094+0000","last_unstale":"2024-09-21T10:12:18.394373+0000","last_undegraded":"2024-09-21T10:12:18.394373+0000","last_fullsized":"2024-09-21T10:12:18.394373+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:12:12.933089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":185,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.394931+0000","last_change":"2024-09-21T10:08:38.379747+0000","last_active":"2024-09-21T10:12:18.394931+0000","last_peered":"2024-09-21T10:12:18.394931+0000","last_clean":"2024-09-21T10:12:18.394931+0000","last_became_active":"2024-09-21T10:08:38.378983+0000","last_became_peered":"2024-09-21T10:08:38.378983+0000","last_unstale":"2024-09-21T10:12:18.394931+0000","last_undegraded":"2024-09-21T10:12:18.394931+0000","last_fullsized":"2024-09-21T10:12:18.394931+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:45:08.003016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"277'10347","reported_seq":10409,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:24.172317+0000","last_change":"2024-09-21T10:09:42.896430+0000","last_active":"2024-09-21T10:12:24.172317+0000","last_peered":"2024-09-21T10:12:24.172317+0000","last_clean":"2024-09-21T10:12:24.172317+0000","last_became_active":"2024-09-21T10:09:42.896019+0000","last_became_peered":"2024-09-21T10:09:42.896019+0000","last_unstale":"2024-09-21T10:12:24.172317+0000","last_undegraded":"2024-09-21T10:12:24.172317+0000","last_fullsized":"2024-09-21T10:12:24.172317+0000","mapping_epoch":252,"log_start":"259'2800","ondisk_log_start":"259'2800","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":7547,"log_dups_size":0,"ondisk_log_size":7547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:59:18.633498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238878720,"num_objects":3645,"num_object_clones":0,"num_object_copies":10935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10347,"num_write_kb":447744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":168,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.373366+0000","last_change":"2024-09-21T10:10:57.403722+0000","last_active":"2024-09-21T10:12:18.373366+0000","last_peered":"2024-09-21T10:12:18.373366+0000","last_clean":"2024-09-21T10:12:18.373366+0000","last_became_active":"2024-09-21T10:10:57.394700+0000","last_became_peered":"2024-09-21T10:10:57.394700+0000","last_unstale":"2024-09-21T10:12:18.373366+0000","last_undegraded":"2024-09-21T10:12:18.373366+0000","last_fullsized":"2024-09-21T10:12:18.373366+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:02.152981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:02.152981+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:52:34.887763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":251,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.852659+0000","last_change":"2024-09-21T10:08:38.379849+0000","last_active":"2024-09-21T10:12:00.852659+0000","last_peered":"2024-09-21T10:12:00.852659+0000","last_clean":"2024-09-21T10:12:00.852659+0000","last_became_active":"2024-09-21T10:08:38.379543+0000","last_became_peered":"2024-09-21T10:08:38.379543+0000","last_unstale":"2024-09-21T10:12:00.852659+0000","last_undegraded":"2024-09-21T10:12:00.852659+0000","last_fullsized":"2024-09-21T10:12:00.852659+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:29.041802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":217,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.381573+0000","last_change":"2024-09-21T10:07:24.342950+0000","last_active":"2024-09-21T10:12:18.381573+0000","last_peered":"2024-09-21T10:12:18.381573+0000","last_clean":"2024-09-21T10:12:18.381573+0000","last_became_active":"2024-09-21T10:07:24.342477+0000","last_became_peered":"2024-09-21T10:07:24.342477+0000","last_unstale":"2024-09-21T10:12:18.381573+0000","last_undegraded":"2024-09-21T10:12:18.381573+0000","last_fullsized":"2024-09-21T10:12:18.381573+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:33:13.664050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1","version":"277'10407","reported_seq":10440,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:22.775604+0000","last_change":"2024-09-21T10:09:42.895259+0000","last_active":"2024-09-21T10:12:22.775604+0000","last_peered":"2024-09-21T10:12:22.775604+0000","last_clean":"2024-09-21T10:12:22.775604+0000","last_became_active":"2024-09-21T10:09:42.895087+0000","last_became_peered":"2024-09-21T10:09:42.895087+0000","last_unstale":"2024-09-21T10:12:22.775604+0000","last_undegraded":"2024-09-21T10:12:22.775604+0000","last_fullsized":"2024-09-21T10:12:22.775604+0000","mapping_epoch":252,"log_start":"263'4200","ondisk_log_start":"263'4200","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6207,"log_dups_size":0,"ondisk_log_size":6207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:40:59.270926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243073024,"num_objects":3709,"num_object_clones":0,"num_object_copies":11127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10407,"num_write_kb":451712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"77'100","reported_seq":311,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.401035+0000","last_change":"2024-09-21T10:10:57.395906+0000","last_active":"2024-09-21T10:12:18.401035+0000","last_peered":"2024-09-21T10:12:18.401035+0000","last_clean":"2024-09-21T10:12:18.401035+0000","last_became_active":"2024-09-21T10:10:57.395366+0000","last_became_peered":"2024-09-21T10:10:57.395366+0000","last_unstale":"2024-09-21T10:12:18.401035+0000","last_undegraded":"2024-09-21T10:12:18.401035+0000","last_fullsized":"2024-09-21T10:12:18.401035+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.540323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.540323+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:34.873336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":309,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.373669+0000","last_change":"2024-09-21T10:07:36.464496+0000","last_active":"2024-09-21T10:12:18.373669+0000","last_peered":"2024-09-21T10:12:18.373669+0000","last_clean":"2024-09-21T10:12:18.373669+0000","last_became_active":"2024-09-21T10:07:36.464172+0000","last_became_peered":"2024-09-21T10:07:36.464172+0000","last_unstale":"2024-09-21T10:12:18.373669+0000","last_undegraded":"2024-09-21T10:12:18.373669+0000","last_fullsized":"2024-09-21T10:12:18.373669+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:09.839709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.851809+0000","last_change":"2024-09-21T10:11:05.442332+0000","last_active":"2024-09-21T10:12:00.851809+0000","last_peered":"2024-09-21T10:12:00.851809+0000","last_clean":"2024-09-21T10:12:00.851809+0000","last_became_active":"2024-09-21T10:11:05.442003+0000","last_became_peered":"2024-09-21T10:11:05.442003+0000","last_unstale":"2024-09-21T10:12:00.851809+0000","last_undegraded":"2024-09-21T10:12:00.851809+0000","last_fullsized":"2024-09-21T10:12:00.851809+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:34.986309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"277'10523","reported_seq":10585,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:26.435551+0000","last_change":"2024-09-21T10:09:43.231401+0000","last_active":"2024-09-21T10:12:26.435551+0000","last_peered":"2024-09-21T10:12:26.435551+0000","last_clean":"2024-09-21T10:12:26.435551+0000","last_became_active":"2024-09-21T10:09:43.230977+0000","last_became_peered":"2024-09-21T10:09:43.230977+0000","last_unstale":"2024-09-21T10:12:26.435551+0000","last_undegraded":"2024-09-21T10:12:26.435551+0000","last_fullsized":"2024-09-21T10:12:26.435551+0000","mapping_epoch":252,"log_start":"263'4800","ondisk_log_start":"263'4800","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5723,"log_dups_size":0,"ondisk_log_size":5723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:15.907809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232587264,"num_objects":3549,"num_object_clones":0,"num_object_copies":10647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":450304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":131,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.375373+0000","last_change":"2024-09-21T10:08:04.645700+0000","last_active":"2024-09-21T10:12:18.375373+0000","last_peered":"2024-09-21T10:12:18.375373+0000","last_clean":"2024-09-21T10:12:18.375373+0000","last_became_active":"2024-09-21T10:07:57.548189+0000","last_became_peered":"2024-09-21T10:07:57.548189+0000","last_unstale":"2024-09-21T10:12:18.375373+0000","last_undegraded":"2024-09-21T10:12:18.375373+0000","last_fullsized":"2024-09-21T10:12:18.375373+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:48.318010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:48.318010+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T16:31:28.173059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":409,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.395864+0000","last_change":"2024-09-21T10:03:38.386604+0000","last_active":"2024-09-21T10:12:18.395864+0000","last_peered":"2024-09-21T10:12:18.395864+0000","last_clean":"2024-09-21T10:12:18.395864+0000","last_became_active":"2024-09-21T10:03:38.386283+0000","last_became_peered":"2024-09-21T10:03:38.386283+0000","last_unstale":"2024-09-21T10:12:18.395864+0000","last_undegraded":"2024-09-21T10:12:18.395864+0000","last_fullsized":"2024-09-21T10:12:18.395864+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:27:05.944880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":212,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.393307+0000","last_change":"2024-09-21T10:08:38.372486+0000","last_active":"2024-09-21T10:12:18.393307+0000","last_peered":"2024-09-21T10:12:18.393307+0000","last_clean":"2024-09-21T10:12:18.393307+0000","last_became_active":"2024-09-21T10:08:38.370669+0000","last_became_peered":"2024-09-21T10:08:38.370669+0000","last_unstale":"2024-09-21T10:12:18.393307+0000","last_undegraded":"2024-09-21T10:12:18.393307+0000","last_fullsized":"2024-09-21T10:12:18.393307+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:53:23.247341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.a","version":"277'10430","reported_seq":10503,"reported_epoch":277,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:12:26.537307+0000","last_change":"2024-09-21T10:10:58.704699+0000","last_active":"2024-09-21T10:12:26.537307+0000","last_peered":"2024-09-21T10:12:26.537307+0000","last_clean":"2024-09-21T10:10:56.390149+0000","last_became_active":"2024-09-21T10:10:58.682906+0000","last_became_peered":"2024-09-21T10:10:58.682906+0000","last_unstale":"2024-09-21T10:12:26.537307+0000","last_undegraded":"2024-09-21T10:12:26.537307+0000","last_fullsized":"2024-09-21T10:12:26.537307+0000","mapping_epoch":267,"log_start":"263'4700","ondisk_log_start":"263'4700","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5730,"log_dups_size":0,"ondisk_log_size":5730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228589568,"num_objects":3488,"num_object_clones":0,"num_object_copies":10464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3488,"num_objects_unfound":0,"num_objects_dirty":3488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10430,"num_write_kb":445376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,1],"avail_no_missing":["3","1","6"],"object_location_counts":[{"shards":"1,3,6","objects":3488}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":168,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.843033+0000","last_change":"2024-09-21T10:09:49.924286+0000","last_active":"2024-09-21T10:12:00.843033+0000","last_peered":"2024-09-21T10:12:00.843033+0000","last_clean":"2024-09-21T10:12:00.843033+0000","last_became_active":"2024-09-21T10:09:49.923897+0000","last_became_peered":"2024-09-21T10:09:49.923897+0000","last_unstale":"2024-09-21T10:12:00.843033+0000","last_undegraded":"2024-09-21T10:12:00.843033+0000","last_fullsized":"2024-09-21T10:12:00.843033+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:47.393146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:47.393146+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:44.898083+0000","scrub_duration":0.001601086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":164,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.373844+0000","last_change":"2024-09-21T10:10:57.402946+0000","last_active":"2024-09-21T10:12:18.373844+0000","last_peered":"2024-09-21T10:12:18.373844+0000","last_clean":"2024-09-21T10:12:18.373844+0000","last_became_active":"2024-09-21T10:10:57.395450+0000","last_became_peered":"2024-09-21T10:10:57.395450+0000","last_unstale":"2024-09-21T10:12:18.373844+0000","last_undegraded":"2024-09-21T10:12:18.373844+0000","last_fullsized":"2024-09-21T10:12:18.373844+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:08.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":393,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.397113+0000","last_change":"2024-09-21T10:10:57.390595+0000","last_active":"2024-09-21T10:12:18.397113+0000","last_peered":"2024-09-21T10:12:18.397113+0000","last_clean":"2024-09-21T10:12:18.397113+0000","last_became_active":"2024-09-21T10:10:57.390265+0000","last_became_peered":"2024-09-21T10:10:57.390265+0000","last_unstale":"2024-09-21T10:12:18.397113+0000","last_undegraded":"2024-09-21T10:12:18.397113+0000","last_fullsized":"2024-09-21T10:12:18.397113+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:37.485063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":284,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.852095+0000","last_change":"2024-09-21T10:09:24.837651+0000","last_active":"2024-09-21T10:12:00.852095+0000","last_peered":"2024-09-21T10:12:00.852095+0000","last_clean":"2024-09-21T10:12:00.852095+0000","last_became_active":"2024-09-21T10:09:24.837226+0000","last_became_peered":"2024-09-21T10:09:24.837226+0000","last_unstale":"2024-09-21T10:12:00.852095+0000","last_undegraded":"2024-09-21T10:12:00.852095+0000","last_fullsized":"2024-09-21T10:12:00.852095+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:16.649601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"277'10458","reported_seq":10527,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:26.566666+0000","last_change":"2024-09-21T10:09:49.926338+0000","last_active":"2024-09-21T10:12:26.566666+0000","last_peered":"2024-09-21T10:12:26.566666+0000","last_clean":"2024-09-21T10:12:26.566666+0000","last_became_active":"2024-09-21T10:09:49.926018+0000","last_became_peered":"2024-09-21T10:09:49.926018+0000","last_unstale":"2024-09-21T10:12:26.566666+0000","last_undegraded":"2024-09-21T10:12:26.566666+0000","last_fullsized":"2024-09-21T10:12:26.566666+0000","mapping_epoch":255,"log_start":"263'4700","ondisk_log_start":"263'4700","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5758,"log_dups_size":0,"ondisk_log_size":5758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:09:49.364660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229638144,"num_objects":3504,"num_object_clones":0,"num_object_copies":10512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10458,"num_write_kb":446784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":17,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.851611+0000","last_change":"2024-09-21T10:11:05.443167+0000","last_active":"2024-09-21T10:12:00.851611+0000","last_peered":"2024-09-21T10:12:00.851611+0000","last_clean":"2024-09-21T10:12:00.851611+0000","last_became_active":"2024-09-21T10:11:05.442805+0000","last_became_peered":"2024-09-21T10:11:05.442805+0000","last_unstale":"2024-09-21T10:12:00.851611+0000","last_undegraded":"2024-09-21T10:12:00.851611+0000","last_fullsized":"2024-09-21T10:12:00.851611+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.550551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.550551+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:58:57.998543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":142,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.843055+0000","last_change":"2024-09-21T10:09:23.838944+0000","last_active":"2024-09-21T10:12:00.843055+0000","last_peered":"2024-09-21T10:12:00.843055+0000","last_clean":"2024-09-21T10:12:00.843055+0000","last_became_active":"2024-09-21T10:09:23.837886+0000","last_became_peered":"2024-09-21T10:09:23.837886+0000","last_unstale":"2024-09-21T10:12:00.843055+0000","last_undegraded":"2024-09-21T10:12:00.843055+0000","last_fullsized":"2024-09-21T10:12:00.843055+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:16.928195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":286,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.400241+0000","last_change":"2024-09-21T10:09:24.839770+0000","last_active":"2024-09-21T10:12:18.400241+0000","last_peered":"2024-09-21T10:12:18.400241+0000","last_clean":"2024-09-21T10:12:18.400241+0000","last_became_active":"2024-09-21T10:09:24.839479+0000","last_became_peered":"2024-09-21T10:09:24.839479+0000","last_unstale":"2024-09-21T10:12:18.400241+0000","last_undegraded":"2024-09-21T10:12:18.400241+0000","last_fullsized":"2024-09-21T10:12:18.400241+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:11:19.618193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":55,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.401079+0000","last_change":"2024-09-21T10:09:49.923044+0000","last_active":"2024-09-21T10:12:18.401079+0000","last_peered":"2024-09-21T10:12:18.401079+0000","last_clean":"2024-09-21T10:12:18.401079+0000","last_became_active":"2024-09-21T10:09:49.922711+0000","last_became_peered":"2024-09-21T10:09:49.922711+0000","last_unstale":"2024-09-21T10:12:18.401079+0000","last_undegraded":"2024-09-21T10:12:18.401079+0000","last_fullsized":"2024-09-21T10:12:18.401079+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:14.090084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"277'10701","reported_seq":10778,"reported_epoch":277,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:12:26.515966+0000","last_change":"2024-09-21T10:10:58.695837+0000","last_active":"2024-09-21T10:12:26.515966+0000","last_peered":"2024-09-21T10:12:26.515966+0000","last_clean":"2024-09-21T10:10:56.379248+0000","last_became_active":"2024-09-21T10:10:58.672712+0000","last_became_peered":"2024-09-21T10:10:58.672712+0000","last_unstale":"2024-09-21T10:12:26.515966+0000","last_undegraded":"2024-09-21T10:12:26.515966+0000","last_fullsized":"2024-09-21T10:12:26.515966+0000","mapping_epoch":267,"log_start":"263'5000","ondisk_log_start":"263'5000","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5701,"log_dups_size":0,"ondisk_log_size":5701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233242624,"num_objects":3559,"num_object_clones":0,"num_object_copies":10677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3559,"num_objects_unfound":0,"num_objects_dirty":3559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10701,"num_write_kb":456320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,0],"avail_no_missing":["3","0","6"],"object_location_counts":[{"shards":"0,3,6","objects":3559}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":182,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.843228+0000","last_change":"2024-09-21T10:09:24.838434+0000","last_active":"2024-09-21T10:12:00.843228+0000","last_peered":"2024-09-21T10:12:00.843228+0000","last_clean":"2024-09-21T10:12:00.843228+0000","last_became_active":"2024-09-21T10:09:24.838066+0000","last_became_peered":"2024-09-21T10:09:24.838066+0000","last_unstale":"2024-09-21T10:12:00.843228+0000","last_undegraded":"2024-09-21T10:12:00.843228+0000","last_fullsized":"2024-09-21T10:12:00.843228+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:52.361137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:52.361137+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:19:57.788195+0000","scrub_duration":0.001255358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":447,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.373809+0000","last_change":"2024-09-21T10:09:23.837859+0000","last_active":"2024-09-21T10:12:18.373809+0000","last_peered":"2024-09-21T10:12:18.373809+0000","last_clean":"2024-09-21T10:12:18.373809+0000","last_became_active":"2024-09-21T10:09:23.837488+0000","last_became_peered":"2024-09-21T10:09:23.837488+0000","last_unstale":"2024-09-21T10:12:18.373809+0000","last_undegraded":"2024-09-21T10:12:18.373809+0000","last_fullsized":"2024-09-21T10:12:18.373809+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:25.215501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":26,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.842975+0000","last_change":"2024-09-21T10:10:27.806070+0000","last_active":"2024-09-21T10:12:00.842975+0000","last_peered":"2024-09-21T10:12:00.842975+0000","last_clean":"2024-09-21T10:12:00.842975+0000","last_became_active":"2024-09-21T10:10:27.805718+0000","last_became_peered":"2024-09-21T10:10:27.805718+0000","last_unstale":"2024-09-21T10:12:00.842975+0000","last_undegraded":"2024-09-21T10:12:00.842975+0000","last_fullsized":"2024-09-21T10:12:00.842975+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:31.144118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":440,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.374163+0000","last_change":"2024-09-21T10:10:57.395073+0000","last_active":"2024-09-21T10:12:18.374163+0000","last_peered":"2024-09-21T10:12:18.374163+0000","last_clean":"2024-09-21T10:12:18.374163+0000","last_became_active":"2024-09-21T10:10:57.394674+0000","last_became_peered":"2024-09-21T10:10:57.394674+0000","last_unstale":"2024-09-21T10:12:18.374163+0000","last_undegraded":"2024-09-21T10:12:18.374163+0000","last_fullsized":"2024-09-21T10:12:18.374163+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:48:48.912649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"277'10650","reported_seq":10712,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:26.539198+0000","last_change":"2024-09-21T10:09:43.231554+0000","last_active":"2024-09-21T10:12:26.539198+0000","last_peered":"2024-09-21T10:12:26.539198+0000","last_clean":"2024-09-21T10:12:26.539198+0000","last_became_active":"2024-09-21T10:09:43.231211+0000","last_became_peered":"2024-09-21T10:09:43.231211+0000","last_unstale":"2024-09-21T10:12:26.539198+0000","last_undegraded":"2024-09-21T10:12:26.539198+0000","last_fullsized":"2024-09-21T10:12:26.539198+0000","mapping_epoch":252,"log_start":"263'4900","ondisk_log_start":"263'4900","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5750,"log_dups_size":0,"ondisk_log_size":5750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:16:41.457768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238944256,"num_objects":3646,"num_object_clones":0,"num_object_copies":10938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10650,"num_write_kb":457472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":310,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.400328+0000","last_change":"2024-09-21T10:10:57.398437+0000","last_active":"2024-09-21T10:12:18.400328+0000","last_peered":"2024-09-21T10:12:18.400328+0000","last_clean":"2024-09-21T10:12:18.400328+0000","last_became_active":"2024-09-21T10:10:57.397441+0000","last_became_peered":"2024-09-21T10:10:57.397441+0000","last_unstale":"2024-09-21T10:12:18.400328+0000","last_undegraded":"2024-09-21T10:12:18.400328+0000","last_fullsized":"2024-09-21T10:12:18.400328+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:04.043917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:04.043917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:53:31.813612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":81,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.852721+0000","last_change":"2024-09-21T10:08:26.309323+0000","last_active":"2024-09-21T10:12:00.852721+0000","last_peered":"2024-09-21T10:12:00.852721+0000","last_clean":"2024-09-21T10:12:00.852721+0000","last_became_active":"2024-09-21T10:08:26.309004+0000","last_became_peered":"2024-09-21T10:08:26.309004+0000","last_unstale":"2024-09-21T10:12:00.852721+0000","last_undegraded":"2024-09-21T10:12:00.852721+0000","last_fullsized":"2024-09-21T10:12:00.852721+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:37:20.626702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":120,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.400911+0000","last_change":"2024-09-21T10:10:28.180093+0000","last_active":"2024-09-21T10:12:18.400911+0000","last_peered":"2024-09-21T10:12:18.400911+0000","last_clean":"2024-09-21T10:12:18.400911+0000","last_became_active":"2024-09-21T10:10:28.179559+0000","last_became_peered":"2024-09-21T10:10:28.179559+0000","last_unstale":"2024-09-21T10:12:18.400911+0000","last_undegraded":"2024-09-21T10:12:18.400911+0000","last_fullsized":"2024-09-21T10:12:18.400911+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:52:46.146392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":216,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.376221+0000","last_change":"2024-09-21T10:08:26.307041+0000","last_active":"2024-09-21T10:12:18.376221+0000","last_peered":"2024-09-21T10:12:18.376221+0000","last_clean":"2024-09-21T10:12:18.376221+0000","last_became_active":"2024-09-21T10:08:26.306715+0000","last_became_peered":"2024-09-21T10:08:26.306715+0000","last_unstale":"2024-09-21T10:12:18.376221+0000","last_undegraded":"2024-09-21T10:12:18.376221+0000","last_fullsized":"2024-09-21T10:12:18.376221+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:19:26.192398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"277'10569","reported_seq":10632,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:24.161887+0000","last_change":"2024-09-21T10:09:43.232893+0000","last_active":"2024-09-21T10:12:24.161887+0000","last_peered":"2024-09-21T10:12:24.161887+0000","last_clean":"2024-09-21T10:12:24.161887+0000","last_became_active":"2024-09-21T10:09:43.232461+0000","last_became_peered":"2024-09-21T10:09:43.232461+0000","last_unstale":"2024-09-21T10:12:24.161887+0000","last_undegraded":"2024-09-21T10:12:24.161887+0000","last_fullsized":"2024-09-21T10:12:24.161887+0000","mapping_epoch":252,"log_start":"259'3000","ondisk_log_start":"259'3000","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":7569,"log_dups_size":0,"ondisk_log_size":7569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:39.778833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240255000,"num_objects":3667,"num_object_clones":0,"num_object_copies":11001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3667,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10569,"num_write_kb":455489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":251,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.851768+0000","last_change":"2024-09-21T10:04:45.242148+0000","last_active":"2024-09-21T10:12:00.851768+0000","last_peered":"2024-09-21T10:12:00.851768+0000","last_clean":"2024-09-21T10:12:00.851768+0000","last_became_active":"2024-09-21T10:04:42.549844+0000","last_became_peered":"2024-09-21T10:04:42.549844+0000","last_unstale":"2024-09-21T10:12:00.851768+0000","last_undegraded":"2024-09-21T10:12:00.851768+0000","last_fullsized":"2024-09-21T10:12:00.851768+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_deep_scrub":"77'100","last_deep_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.242073+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:03:42.250480+0000","scrub_duration":0.0023466239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.872629+0000","last_change":"2024-09-21T10:10:57.392880+0000","last_active":"2024-09-21T10:12:00.872629+0000","last_peered":"2024-09-21T10:12:00.872629+0000","last_clean":"2024-09-21T10:12:00.872629+0000","last_became_active":"2024-09-21T10:10:57.392539+0000","last_became_peered":"2024-09-21T10:10:57.392539+0000","last_unstale":"2024-09-21T10:12:00.872629+0000","last_undegraded":"2024-09-21T10:12:00.872629+0000","last_fullsized":"2024-09-21T10:12:00.872629+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:49.754955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":330,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.397935+0000","last_change":"2024-09-21T10:09:23.865342+0000","last_active":"2024-09-21T10:12:18.397935+0000","last_peered":"2024-09-21T10:12:18.397935+0000","last_clean":"2024-09-21T10:12:18.397935+0000","last_became_active":"2024-09-21T10:09:23.865008+0000","last_became_peered":"2024-09-21T10:09:23.865008+0000","last_unstale":"2024-09-21T10:12:18.397935+0000","last_undegraded":"2024-09-21T10:12:18.397935+0000","last_fullsized":"2024-09-21T10:12:18.397935+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:03.356101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":20,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.872356+0000","last_change":"2024-09-21T10:10:57.394707+0000","last_active":"2024-09-21T10:12:00.872356+0000","last_peered":"2024-09-21T10:12:00.872356+0000","last_clean":"2024-09-21T10:12:00.872356+0000","last_became_active":"2024-09-21T10:10:57.394363+0000","last_became_peered":"2024-09-21T10:10:57.394363+0000","last_unstale":"2024-09-21T10:12:00.872356+0000","last_undegraded":"2024-09-21T10:12:00.872356+0000","last_fullsized":"2024-09-21T10:12:00.872356+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:21.141052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"277'10460","reported_seq":10522,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:24.192696+0000","last_change":"2024-09-21T10:09:43.232987+0000","last_active":"2024-09-21T10:12:24.192696+0000","last_peered":"2024-09-21T10:12:24.192696+0000","last_clean":"2024-09-21T10:12:24.192696+0000","last_became_active":"2024-09-21T10:09:43.232577+0000","last_became_peered":"2024-09-21T10:09:43.232577+0000","last_unstale":"2024-09-21T10:12:24.192696+0000","last_undegraded":"2024-09-21T10:12:24.192696+0000","last_fullsized":"2024-09-21T10:12:24.192696+0000","mapping_epoch":252,"log_start":"259'2900","ondisk_log_start":"259'2900","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":7560,"log_dups_size":0,"ondisk_log_size":7560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:14.536774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241303552,"num_objects":3682,"num_object_clones":0,"num_object_copies":11046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10460,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":268,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.851918+0000","last_change":"2024-09-21T10:09:23.839563+0000","last_active":"2024-09-21T10:12:00.851918+0000","last_peered":"2024-09-21T10:12:00.851918+0000","last_clean":"2024-09-21T10:12:00.851918+0000","last_became_active":"2024-09-21T10:09:23.839207+0000","last_became_peered":"2024-09-21T10:09:23.839207+0000","last_unstale":"2024-09-21T10:12:00.851918+0000","last_undegraded":"2024-09-21T10:12:00.851918+0000","last_fullsized":"2024-09-21T10:12:00.851918+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:49.226496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:49.226496+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:03.960125+0000","scrub_duration":0.001656865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":181,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.376055+0000","last_change":"2024-09-21T10:10:57.395167+0000","last_active":"2024-09-21T10:12:18.376055+0000","last_peered":"2024-09-21T10:12:18.376055+0000","last_clean":"2024-09-21T10:12:18.376055+0000","last_became_active":"2024-09-21T10:10:57.394818+0000","last_became_peered":"2024-09-21T10:10:57.394818+0000","last_unstale":"2024-09-21T10:12:18.376055+0000","last_undegraded":"2024-09-21T10:12:18.376055+0000","last_fullsized":"2024-09-21T10:12:18.376055+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:57.224113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":162,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.845948+0000","last_change":"2024-09-21T10:09:49.922782+0000","last_active":"2024-09-21T10:12:00.845948+0000","last_peered":"2024-09-21T10:12:00.845948+0000","last_clean":"2024-09-21T10:12:00.845948+0000","last_became_active":"2024-09-21T10:09:49.922465+0000","last_became_peered":"2024-09-21T10:09:49.922465+0000","last_unstale":"2024-09-21T10:12:00.845948+0000","last_undegraded":"2024-09-21T10:12:00.845948+0000","last_fullsized":"2024-09-21T10:12:00.845948+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:06.151245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":324,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.401122+0000","last_change":"2024-09-21T10:10:57.397251+0000","last_active":"2024-09-21T10:12:18.401122+0000","last_peered":"2024-09-21T10:12:18.401122+0000","last_clean":"2024-09-21T10:12:18.401122+0000","last_became_active":"2024-09-21T10:10:57.396271+0000","last_became_peered":"2024-09-21T10:10:57.396271+0000","last_unstale":"2024-09-21T10:12:18.401122+0000","last_undegraded":"2024-09-21T10:12:18.401122+0000","last_fullsized":"2024-09-21T10:12:18.401122+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:17.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.f","version":"277'10766","reported_seq":10799,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:26.813877+0000","last_change":"2024-09-21T10:09:43.232625+0000","last_active":"2024-09-21T10:12:26.813877+0000","last_peered":"2024-09-21T10:12:26.813877+0000","last_clean":"2024-09-21T10:12:26.813877+0000","last_became_active":"2024-09-21T10:09:43.232331+0000","last_became_peered":"2024-09-21T10:09:43.232331+0000","last_unstale":"2024-09-21T10:12:26.813877+0000","last_undegraded":"2024-09-21T10:12:26.813877+0000","last_fullsized":"2024-09-21T10:12:26.813877+0000","mapping_epoch":252,"log_start":"263'4500","ondisk_log_start":"263'4500","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6266,"log_dups_size":0,"ondisk_log_size":6266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:01:16.634826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230162432,"num_objects":3512,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10766,"num_write_kb":456896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":127,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.381253+0000","last_change":"2024-09-21T10:11:05.455170+0000","last_active":"2024-09-21T10:12:18.381253+0000","last_peered":"2024-09-21T10:12:18.381253+0000","last_clean":"2024-09-21T10:12:18.381253+0000","last_became_active":"2024-09-21T10:11:05.454607+0000","last_became_peered":"2024-09-21T10:11:05.454607+0000","last_unstale":"2024-09-21T10:12:18.381253+0000","last_undegraded":"2024-09-21T10:12:18.381253+0000","last_fullsized":"2024-09-21T10:12:18.381253+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:59.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:59.099852+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:14.788975+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":415,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.393377+0000","last_change":"2024-09-21T10:09:23.841186+0000","last_active":"2024-09-21T10:12:18.393377+0000","last_peered":"2024-09-21T10:12:18.393377+0000","last_clean":"2024-09-21T10:12:18.393377+0000","last_became_active":"2024-09-21T10:09:23.840854+0000","last_became_peered":"2024-09-21T10:09:23.840854+0000","last_unstale":"2024-09-21T10:12:18.393377+0000","last_undegraded":"2024-09-21T10:12:18.393377+0000","last_fullsized":"2024-09-21T10:12:18.393377+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:29.666358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":99,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.852007+0000","last_change":"2024-09-21T10:08:38.380867+0000","last_active":"2024-09-21T10:12:00.852007+0000","last_peered":"2024-09-21T10:12:00.852007+0000","last_clean":"2024-09-21T10:12:00.852007+0000","last_became_active":"2024-09-21T10:08:38.379599+0000","last_became_peered":"2024-09-21T10:08:38.379599+0000","last_unstale":"2024-09-21T10:12:00.852007+0000","last_undegraded":"2024-09-21T10:12:00.852007+0000","last_fullsized":"2024-09-21T10:12:00.852007+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:50.499470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":412,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.395583+0000","last_change":"2024-09-21T10:09:49.926548+0000","last_active":"2024-09-21T10:12:18.395583+0000","last_peered":"2024-09-21T10:12:18.395583+0000","last_clean":"2024-09-21T10:12:18.395583+0000","last_became_active":"2024-09-21T10:09:49.926255+0000","last_became_peered":"2024-09-21T10:09:49.926255+0000","last_unstale":"2024-09-21T10:12:18.395583+0000","last_undegraded":"2024-09-21T10:12:18.395583+0000","last_fullsized":"2024-09-21T10:12:18.395583+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:49.855878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"277'10718","reported_seq":10751,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:26.812141+0000","last_change":"2024-09-21T10:09:43.231158+0000","last_active":"2024-09-21T10:12:26.812141+0000","last_peered":"2024-09-21T10:12:26.812141+0000","last_clean":"2024-09-21T10:12:26.812141+0000","last_became_active":"2024-09-21T10:09:43.230838+0000","last_became_peered":"2024-09-21T10:09:43.230838+0000","last_unstale":"2024-09-21T10:12:26.812141+0000","last_undegraded":"2024-09-21T10:12:26.812141+0000","last_fullsized":"2024-09-21T10:12:26.812141+0000","mapping_epoch":252,"log_start":"263'4400","ondisk_log_start":"263'4400","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6318,"log_dups_size":0,"ondisk_log_size":6318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:15.318633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230162432,"num_objects":3512,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10718,"num_write_kb":455360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":213,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.394442+0000","last_change":"2024-09-21T10:10:57.390804+0000","last_active":"2024-09-21T10:12:18.394442+0000","last_peered":"2024-09-21T10:12:18.394442+0000","last_clean":"2024-09-21T10:12:18.394442+0000","last_became_active":"2024-09-21T10:10:57.390323+0000","last_became_peered":"2024-09-21T10:10:57.390323+0000","last_unstale":"2024-09-21T10:12:18.394442+0000","last_undegraded":"2024-09-21T10:12:18.394442+0000","last_fullsized":"2024-09-21T10:12:18.394442+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.005477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.005477+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:29:57.702046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":59,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.401281+0000","last_change":"2024-09-21T10:09:49.930280+0000","last_active":"2024-09-21T10:12:18.401281+0000","last_peered":"2024-09-21T10:12:18.401281+0000","last_clean":"2024-09-21T10:12:18.401281+0000","last_became_active":"2024-09-21T10:09:49.919101+0000","last_became_peered":"2024-09-21T10:09:49.919101+0000","last_unstale":"2024-09-21T10:12:18.401281+0000","last_undegraded":"2024-09-21T10:12:18.401281+0000","last_fullsized":"2024-09-21T10:12:18.401281+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:23.998475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":290,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.852019+0000","last_change":"2024-09-21T10:03:38.392767+0000","last_active":"2024-09-21T10:12:00.852019+0000","last_peered":"2024-09-21T10:12:00.852019+0000","last_clean":"2024-09-21T10:12:00.852019+0000","last_became_active":"2024-09-21T10:03:38.392424+0000","last_became_peered":"2024-09-21T10:03:38.392424+0000","last_unstale":"2024-09-21T10:12:00.852019+0000","last_undegraded":"2024-09-21T10:12:00.852019+0000","last_fullsized":"2024-09-21T10:12:00.852019+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:25.358291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":71,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.872079+0000","last_change":"2024-09-21T10:09:24.836360+0000","last_active":"2024-09-21T10:12:00.872079+0000","last_peered":"2024-09-21T10:12:00.872079+0000","last_clean":"2024-09-21T10:12:00.872079+0000","last_became_active":"2024-09-21T10:09:24.836033+0000","last_became_peered":"2024-09-21T10:09:24.836033+0000","last_unstale":"2024-09-21T10:12:00.872079+0000","last_undegraded":"2024-09-21T10:12:00.872079+0000","last_fullsized":"2024-09-21T10:12:00.872079+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:58.368582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":128,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.851714+0000","last_change":"2024-09-21T10:08:38.380189+0000","last_active":"2024-09-21T10:12:00.851714+0000","last_peered":"2024-09-21T10:12:00.851714+0000","last_clean":"2024-09-21T10:12:00.851714+0000","last_became_active":"2024-09-21T10:08:38.379891+0000","last_became_peered":"2024-09-21T10:08:38.379891+0000","last_unstale":"2024-09-21T10:12:00.851714+0000","last_undegraded":"2024-09-21T10:12:00.851714+0000","last_fullsized":"2024-09-21T10:12:00.851714+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:35.820921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":208,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.843470+0000","last_change":"2024-09-21T10:08:38.392563+0000","last_active":"2024-09-21T10:12:00.843470+0000","last_peered":"2024-09-21T10:12:00.843470+0000","last_clean":"2024-09-21T10:12:00.843470+0000","last_became_active":"2024-09-21T10:08:38.378735+0000","last_became_peered":"2024-09-21T10:08:38.378735+0000","last_unstale":"2024-09-21T10:12:00.843470+0000","last_undegraded":"2024-09-21T10:12:00.843470+0000","last_fullsized":"2024-09-21T10:12:00.843470+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:29:41.423208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.852645+0000","last_change":"2024-09-21T10:11:05.453286+0000","last_active":"2024-09-21T10:12:00.852645+0000","last_peered":"2024-09-21T10:12:00.852645+0000","last_clean":"2024-09-21T10:12:00.852645+0000","last_became_active":"2024-09-21T10:11:05.452601+0000","last_became_peered":"2024-09-21T10:11:05.452601+0000","last_unstale":"2024-09-21T10:12:00.852645+0000","last_undegraded":"2024-09-21T10:12:00.852645+0000","last_fullsized":"2024-09-21T10:12:00.852645+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:27.532091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":50,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.381490+0000","last_change":"2024-09-21T10:10:27.833762+0000","last_active":"2024-09-21T10:12:18.381490+0000","last_peered":"2024-09-21T10:12:18.381490+0000","last_clean":"2024-09-21T10:12:18.381490+0000","last_became_active":"2024-09-21T10:10:27.819999+0000","last_became_peered":"2024-09-21T10:10:27.819999+0000","last_unstale":"2024-09-21T10:12:18.381490+0000","last_undegraded":"2024-09-21T10:12:18.381490+0000","last_fullsized":"2024-09-21T10:12:18.381490+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:33.839463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'100","reported_seq":206,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.375843+0000","last_change":"2024-09-21T10:07:24.345428+0000","last_active":"2024-09-21T10:12:18.375843+0000","last_peered":"2024-09-21T10:12:18.375843+0000","last_clean":"2024-09-21T10:12:18.375843+0000","last_became_active":"2024-09-21T10:07:24.343529+0000","last_became_peered":"2024-09-21T10:07:24.343529+0000","last_unstale":"2024-09-21T10:12:18.375843+0000","last_undegraded":"2024-09-21T10:12:18.375843+0000","last_fullsized":"2024-09-21T10:12:18.375843+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.014993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.014993+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:35.677888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.872552+0000","last_change":"2024-09-21T10:10:57.394885+0000","last_active":"2024-09-21T10:12:00.872552+0000","last_peered":"2024-09-21T10:12:00.872552+0000","last_clean":"2024-09-21T10:12:00.872552+0000","last_became_active":"2024-09-21T10:10:57.394584+0000","last_became_peered":"2024-09-21T10:10:57.394584+0000","last_unstale":"2024-09-21T10:12:00.872552+0000","last_undegraded":"2024-09-21T10:12:00.872552+0000","last_fullsized":"2024-09-21T10:12:00.872552+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:52:16.853713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":397,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.396599+0000","last_change":"2024-09-21T10:03:38.425111+0000","last_active":"2024-09-21T10:12:18.396599+0000","last_peered":"2024-09-21T10:12:18.396599+0000","last_clean":"2024-09-21T10:12:18.396599+0000","last_became_active":"2024-09-21T10:03:38.424704+0000","last_became_peered":"2024-09-21T10:03:38.424704+0000","last_unstale":"2024-09-21T10:12:18.396599+0000","last_undegraded":"2024-09-21T10:12:18.396599+0000","last_fullsized":"2024-09-21T10:12:18.396599+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:35.157076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":299,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.399880+0000","last_change":"2024-09-21T10:10:57.397755+0000","last_active":"2024-09-21T10:12:18.399880+0000","last_peered":"2024-09-21T10:12:18.399880+0000","last_clean":"2024-09-21T10:12:18.399880+0000","last_became_active":"2024-09-21T10:10:57.397249+0000","last_became_peered":"2024-09-21T10:10:57.397249+0000","last_unstale":"2024-09-21T10:12:18.399880+0000","last_undegraded":"2024-09-21T10:12:18.399880+0000","last_fullsized":"2024-09-21T10:12:18.399880+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:40.813503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":382,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.395478+0000","last_change":"2024-09-21T09:56:12.576756+0000","last_active":"2024-09-21T10:12:18.395478+0000","last_peered":"2024-09-21T10:12:18.395478+0000","last_clean":"2024-09-21T10:12:18.395478+0000","last_became_active":"2024-09-21T09:56:12.575752+0000","last_became_peered":"2024-09-21T09:56:12.575752+0000","last_unstale":"2024-09-21T10:12:18.395478+0000","last_undegraded":"2024-09-21T10:12:18.395478+0000","last_fullsized":"2024-09-21T10:12:18.395478+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:55.224839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":240,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.851939+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T10:12:00.851939+0000","last_peered":"2024-09-21T10:12:00.851939+0000","last_clean":"2024-09-21T10:12:00.851939+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T10:12:00.851939+0000","last_undegraded":"2024-09-21T10:12:00.851939+0000","last_fullsized":"2024-09-21T10:12:00.851939+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":418,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.395667+0000","last_change":"2024-09-21T10:09:49.928053+0000","last_active":"2024-09-21T10:12:18.395667+0000","last_peered":"2024-09-21T10:12:18.395667+0000","last_clean":"2024-09-21T10:12:18.395667+0000","last_became_active":"2024-09-21T10:09:49.927231+0000","last_became_peered":"2024-09-21T10:09:49.927231+0000","last_unstale":"2024-09-21T10:12:18.395667+0000","last_undegraded":"2024-09-21T10:12:18.395667+0000","last_fullsized":"2024-09-21T10:12:18.395667+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:26.987553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":331,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.401634+0000","last_change":"2024-09-21T10:09:23.840717+0000","last_active":"2024-09-21T10:12:18.401634+0000","last_peered":"2024-09-21T10:12:18.401634+0000","last_clean":"2024-09-21T10:12:18.401634+0000","last_became_active":"2024-09-21T10:09:23.840323+0000","last_became_peered":"2024-09-21T10:09:23.840323+0000","last_unstale":"2024-09-21T10:12:18.401634+0000","last_undegraded":"2024-09-21T10:12:18.401634+0000","last_fullsized":"2024-09-21T10:12:18.401634+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:55.769770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":44,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.373520+0000","last_change":"2024-09-21T10:10:27.816526+0000","last_active":"2024-09-21T10:12:18.373520+0000","last_peered":"2024-09-21T10:12:18.373520+0000","last_clean":"2024-09-21T10:12:18.373520+0000","last_became_active":"2024-09-21T10:10:27.816221+0000","last_became_peered":"2024-09-21T10:10:27.816221+0000","last_unstale":"2024-09-21T10:12:18.373520+0000","last_undegraded":"2024-09-21T10:12:18.373520+0000","last_fullsized":"2024-09-21T10:12:18.373520+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:08.331228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":389,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.395179+0000","last_change":"2024-09-21T10:08:38.379994+0000","last_active":"2024-09-21T10:12:18.395179+0000","last_peered":"2024-09-21T10:12:18.395179+0000","last_clean":"2024-09-21T10:12:18.395179+0000","last_became_active":"2024-09-21T10:08:38.378729+0000","last_became_peered":"2024-09-21T10:08:38.378729+0000","last_unstale":"2024-09-21T10:12:18.395179+0000","last_undegraded":"2024-09-21T10:12:18.395179+0000","last_fullsized":"2024-09-21T10:12:18.395179+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:16.940173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":109,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.375756+0000","last_change":"2024-09-21T10:09:24.844561+0000","last_active":"2024-09-21T10:12:18.375756+0000","last_peered":"2024-09-21T10:12:18.375756+0000","last_clean":"2024-09-21T10:12:18.375756+0000","last_became_active":"2024-09-21T10:09:24.844190+0000","last_became_peered":"2024-09-21T10:09:24.844190+0000","last_unstale":"2024-09-21T10:12:18.375756+0000","last_undegraded":"2024-09-21T10:12:18.375756+0000","last_fullsized":"2024-09-21T10:12:18.375756+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:46:04.775344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":436,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.376141+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:12:18.376141+0000","last_peered":"2024-09-21T10:12:18.376141+0000","last_clean":"2024-09-21T10:12:18.376141+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:12:18.376141+0000","last_undegraded":"2024-09-21T10:12:18.376141+0000","last_fullsized":"2024-09-21T10:12:18.376141+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":55,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.400486+0000","last_change":"2024-09-21T10:09:49.918246+0000","last_active":"2024-09-21T10:12:18.400486+0000","last_peered":"2024-09-21T10:12:18.400486+0000","last_clean":"2024-09-21T10:12:18.400486+0000","last_became_active":"2024-09-21T10:09:49.917911+0000","last_became_peered":"2024-09-21T10:09:49.917911+0000","last_unstale":"2024-09-21T10:12:18.400486+0000","last_undegraded":"2024-09-21T10:12:18.400486+0000","last_fullsized":"2024-09-21T10:12:18.400486+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:30:10.567650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":245,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.394330+0000","last_change":"2024-09-21T10:07:24.347937+0000","last_active":"2024-09-21T10:12:18.394330+0000","last_peered":"2024-09-21T10:12:18.394330+0000","last_clean":"2024-09-21T10:12:18.394330+0000","last_became_active":"2024-09-21T10:07:24.347376+0000","last_became_peered":"2024-09-21T10:07:24.347376+0000","last_unstale":"2024-09-21T10:12:18.394330+0000","last_undegraded":"2024-09-21T10:12:18.394330+0000","last_fullsized":"2024-09-21T10:12:18.394330+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:48:05.644824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":59,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.851852+0000","last_change":"2024-09-21T10:08:38.382518+0000","last_active":"2024-09-21T10:12:00.851852+0000","last_peered":"2024-09-21T10:12:00.851852+0000","last_clean":"2024-09-21T10:12:00.851852+0000","last_became_active":"2024-09-21T10:08:38.372051+0000","last_became_peered":"2024-09-21T10:08:38.372051+0000","last_unstale":"2024-09-21T10:12:00.851852+0000","last_undegraded":"2024-09-21T10:12:00.851852+0000","last_fullsized":"2024-09-21T10:12:00.851852+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:21.360435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":336,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.397279+0000","last_change":"2024-09-21T10:09:24.840173+0000","last_active":"2024-09-21T10:12:18.397279+0000","last_peered":"2024-09-21T10:12:18.397279+0000","last_clean":"2024-09-21T10:12:18.397279+0000","last_became_active":"2024-09-21T10:09:24.839762+0000","last_became_peered":"2024-09-21T10:09:24.839762+0000","last_unstale":"2024-09-21T10:12:18.397279+0000","last_undegraded":"2024-09-21T10:12:18.397279+0000","last_fullsized":"2024-09-21T10:12:18.397279+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:12:53.465203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":73,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.852242+0000","last_change":"2024-09-21T10:08:38.388561+0000","last_active":"2024-09-21T10:12:00.852242+0000","last_peered":"2024-09-21T10:12:00.852242+0000","last_clean":"2024-09-21T10:12:00.852242+0000","last_became_active":"2024-09-21T10:08:38.374558+0000","last_became_peered":"2024-09-21T10:08:38.374558+0000","last_unstale":"2024-09-21T10:12:00.852242+0000","last_undegraded":"2024-09-21T10:12:00.852242+0000","last_fullsized":"2024-09-21T10:12:00.852242+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:24.111438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":419,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.397456+0000","last_change":"2024-09-21T10:09:23.840467+0000","last_active":"2024-09-21T10:12:18.397456+0000","last_peered":"2024-09-21T10:12:18.397456+0000","last_clean":"2024-09-21T10:12:18.397456+0000","last_became_active":"2024-09-21T10:09:23.840177+0000","last_became_peered":"2024-09-21T10:09:23.840177+0000","last_unstale":"2024-09-21T10:12:18.397456+0000","last_undegraded":"2024-09-21T10:12:18.397456+0000","last_fullsized":"2024-09-21T10:12:18.397456+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:17.887307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":409,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.395775+0000","last_change":"2024-09-21T10:11:05.451613+0000","last_active":"2024-09-21T10:12:18.395775+0000","last_peered":"2024-09-21T10:12:18.395775+0000","last_clean":"2024-09-21T10:12:18.395775+0000","last_became_active":"2024-09-21T10:11:05.451075+0000","last_became_peered":"2024-09-21T10:11:05.451075+0000","last_unstale":"2024-09-21T10:12:18.395775+0000","last_undegraded":"2024-09-21T10:12:18.395775+0000","last_fullsized":"2024-09-21T10:12:18.395775+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:18.505045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":109,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-09-21T10:12:00.851725+0000","last_change":"2024-09-21T10:08:38.380508+0000","last_active":"2024-09-21T10:12:00.851725+0000","last_peered":"2024-09-21T10:12:00.851725+0000","last_clean":"2024-09-21T10:12:00.851725+0000","last_became_active":"2024-09-21T10:08:38.380226+0000","last_became_peered":"2024-09-21T10:08:38.380226+0000","last_unstale":"2024-09-21T10:12:00.851725+0000","last_undegraded":"2024-09-21T10:12:00.851725+0000","last_fullsized":"2024-09-21T10:12:00.851725+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:50:38.190118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":74,"reported_epoch":277,"state":"active+clean","last_fresh":"2024-09-21T10:12:18.399477+0000","last_change":"2024-09-21T10:10:57.401588+0000","last_active":"2024-09-21T10:12:18.399477+0000","last_peered":"2024-09-21T10:12:18.399477+0000","last_clean":"2024-09-21T10:12:18.399477+0000","last_became_active":"2024-09-21T10:10:57.397724+0000","last_became_peered":"2024-09-21T10:10:57.397724+0000","last_unstale":"2024-09-21T10:12:18.399477+0000","last_undegraded":"2024-09-21T10:12:18.399477+0000","last_fullsized":"2024-09-21T10:12:18.399477+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:47.843630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3752984600,"num_objects":57267,"num_object_clones":0,"num_object_copies":171801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":11574,"num_objects_unfound":0,"num_objects_dirty":57267,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167703,"num_write_kb":7198977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4467,"num_bytes_recovered":292749312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11590119424,"data_stored":11590107208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101403,"ondisk_log_size":101403,"up":48,"acting":47,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":11710,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26580,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":688762,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":49,"num_read_kb":125,"num_write":25,"num_write_kb":271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":722165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626840,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2832212,"kb_used_data":1936012,"kb_used_omap":1,"kb_used_meta":896190,"kb_avail":90912940,"statfs":{"total":95995035648,"available":93094850560,"internally_reserved":0,"allocated":1982476288,"data_stored":1980907480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":917698904},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019298,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":245,"seq":1052266987558,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2019972,"kb_used_data":1829948,"kb_used_omap":8,"kb_used_meta":189879,"kb_avail":91725180,"statfs":{"total":95995035648,"available":93926584320,"internally_reserved":0,"allocated":1873866752,"data_stored":1872310147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8940,"internal_metadata":194436372},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":175,"seq":751619276907,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2160720,"kb_used_data":1826120,"kb_used_omap":12,"kb_used_meta":334579,"kb_avail":91584432,"statfs":{"total":95995035648,"available":93782458368,"internally_reserved":0,"allocated":1869946880,"data_stored":1868382147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12845,"internal_metadata":342609363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823069,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1324544,"kb_used_data":139384,"kb_used_omap":1,"kb_used_meta":1185150,"kb_avail":92420608,"statfs":{"total":95995035648,"available":94638702592,"internally_reserved":0,"allocated":142729216,"data_stored":141176570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1213593944},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879493,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2753728,"kb_used_data":1958776,"kb_used_omap":4,"kb_used_meta":794875,"kb_avail":90991424,"statfs":{"total":95995035648,"available":93175218176,"internally_reserved":0,"allocated":2005786624,"data_stored":2004234059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":813952444},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,7,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566333,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2114328,"kb_used_data":1425936,"kb_used_omap":6,"kb_used_meta":688313,"kb_avail":91630824,"statfs":{"total":95995035648,"available":93829963776,"internally_reserved":0,"allocated":1460158464,"data_stored":1458586703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":704832935},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860678,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3197120,"kb_used_data":2334072,"kb_used_omap":5,"kb_used_meta":863034,"kb_avail":90548032,"statfs":{"total":95995035648,"available":92721184768,"internally_reserved":0,"allocated":2390089728,"data_stored":2388536271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":883747055},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7236,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9976,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1844903936,"data_stored":1844903936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2368606208,"data_stored":2368602136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1446187008,"data_stored":1446182936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1982726144,"data_stored":1982726144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":140050432,"data_stored":140050432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1958612992,"data_stored":1958608920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1849032704,"data_stored":1849032704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:12:31.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T10:12:31.759 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:31.784 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:31.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:31.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:31.886 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:31.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:31.913 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:31.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:31.934 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:12:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:12:31] "GET /metrics HTTP/1.1" 200 36812 "" "Prometheus/2.51.0" 2024-09-21T10:12:32.014 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:32.042 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:32.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:32.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:32.143 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:32.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:32.173 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:32.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:32.274 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:32.299 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:32.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:32.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:32.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:32.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:32.437 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:32.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:32.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:32.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:32.571 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:32.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:32.671 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:32.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:32.701 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:32.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:32.802 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:32.832 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:32.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:32.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:32.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:32 smithi107 ceph-mon[102221]: pgmap v987: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 745 op/s; 11574/167925 objects misplaced (6.892%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T10:12:32.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:32 smithi107 ceph-mon[105022]: pgmap v987: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 745 op/s; 11574/167925 objects misplaced (6.892%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T10:12:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:32 smithi129 ceph-mon[88471]: pgmap v987: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 745 op/s; 11574/167925 objects misplaced (6.892%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T10:12:32.934 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:32.968 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:32.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:32.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:33.070 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:33.097 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:33.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:33.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:33.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:33.236 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:33.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:33.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:33.338 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:33.365 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:33.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:33.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:33.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:33.505 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:33.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:33.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:33.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:33.628 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:33.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:33.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:33.730 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:33.758 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:33.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:33.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:33.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:33.893 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:33.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:33.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:33.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:34.031 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:34.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:34.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:34.132 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:34.166 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:34.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:34.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:34.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:34.306 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:34.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:34.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:34.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:34.465 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:34.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:34.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:34.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:34.583 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:34 smithi107 ceph-mon[102221]: pgmap v988: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s; 10094/156831 objects misplaced (6.436%); 8.3 MiB/s, 133 objects/s recovering 2024-09-21T10:12:34.583 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:34 smithi107 ceph-mon[102221]: osdmap e278: 8 total, 8 up, 7 in 2024-09-21T10:12:34.603 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:34.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:34.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:34.705 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:34.728 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:34.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:34.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:34.830 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:34.833 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:34 smithi107 ceph-mon[105022]: pgmap v988: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s; 10094/156831 objects misplaced (6.436%); 8.3 MiB/s, 133 objects/s recovering 2024-09-21T10:12:34.834 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:34 smithi107 ceph-mon[105022]: osdmap e278: 8 total, 8 up, 7 in 2024-09-21T10:12:34.853 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:34.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:34.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:34 smithi129 ceph-mon[88471]: pgmap v988: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s; 10094/156831 objects misplaced (6.436%); 8.3 MiB/s, 133 objects/s recovering 2024-09-21T10:12:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:34 smithi129 ceph-mon[88471]: osdmap e278: 8 total, 8 up, 7 in 2024-09-21T10:12:34.955 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:34.977 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:34.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:34.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:35.079 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:35.100 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:35.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:35.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:35.133 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:12:35.133 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:12:35.133 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:12:35.134 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:12:35.134 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:12:35.134 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:12:35.134 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:12:35.134 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:12:35.134 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:12:35.134 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:12:35.135 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:12:35.135 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:12:35.135 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:12:35.135 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:12:35.135 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:12:35.135 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:12:35.202 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:35.225 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:35.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:35.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:35.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:35.350 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:35.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:35.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:35.452 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:35.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:35.475 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:35.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:35.575 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:35.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:35.606 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:35.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:35.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:35.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:35.753 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:35.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:35.841 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:35 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2896958330' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:35.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:35 smithi107 ceph-mon[102221]: osdmap e279: 8 total, 8 up, 7 in 2024-09-21T10:12:35.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:35 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2896958330' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:35.842 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:35 smithi107 ceph-mon[105022]: osdmap e279: 8 total, 8 up, 7 in 2024-09-21T10:12:35.855 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:35.878 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:35.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:35.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:35.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:35 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2896958330' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:35.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:35 smithi129 ceph-mon[88471]: osdmap e279: 8 total, 8 up, 7 in 2024-09-21T10:12:35.981 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:36.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:36.008 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:36.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:36.020 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:36.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:36.148 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:36.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:36.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:36.250 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:36.275 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:36.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:36.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:36.377 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:36.402 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:36.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:36.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:36.504 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:36.530 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:36.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:36.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:36.632 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:36.658 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:36.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:36.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:36.683 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:12:36.683 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:12:36.697 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:36 smithi107 ceph-mon[102221]: pgmap v990: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 895 op/s; 10094/154173 objects misplaced (6.547%); 7.4 MiB/s, 117 objects/s recovering 2024-09-21T10:12:36.698 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:36 smithi107 ceph-mon[105022]: pgmap v990: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 895 op/s; 10094/154173 objects misplaced (6.547%); 7.4 MiB/s, 117 objects/s recovering 2024-09-21T10:12:36.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:36.794 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:36.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:36.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:36.896 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:36 smithi129 ceph-mon[88471]: pgmap v990: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+remapped+backfill_wait, 105 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 895 op/s; 10094/154173 objects misplaced (6.547%); 7.4 MiB/s, 117 objects/s recovering 2024-09-21T10:12:36.922 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:36.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:36.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:37.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:37.059 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:37.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:37.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:37.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:37.198 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:37.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:37.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:37.299 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:37.325 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:37.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:37.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:37.427 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:37.470 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:37.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:37.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:37.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:37.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:37.615 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:37.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:37.716 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:37.754 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:37.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:37.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:37.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:37.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:37 smithi107 ceph-mon[102221]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-21T10:12:37.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:37 smithi107 ceph-mon[105022]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-21T10:12:37.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:12:37.898 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:37.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:37.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:37 smithi129 ceph-mon[88471]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-21T10:12:38.000 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:38.025 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:38.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:38.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:38.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:38.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:12:38.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:38.171 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:38.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:38.272 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:38.321 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:38.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:38.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:38.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:38.448 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:38.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:38.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:38.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:38.583 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:38.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:38.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:38.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:38.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:38.718 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:38.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:38.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:38.853 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:38.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:38.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:38 smithi129 ceph-mon[88471]: pgmap v992: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 3.0 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 882 op/s; 8252/146748 objects misplaced (5.623%); 14 MiB/s, 225 objects/s recovering 2024-09-21T10:12:38.955 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:38.981 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:38.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:38.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:39.083 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:39.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:38 smithi107 ceph-mon[102221]: pgmap v992: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 3.0 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 882 op/s; 8252/146748 objects misplaced (5.623%); 14 MiB/s, 225 objects/s recovering 2024-09-21T10:12:39.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:38 smithi107 ceph-mon[105022]: pgmap v992: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 3.0 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 882 op/s; 8252/146748 objects misplaced (5.623%); 14 MiB/s, 225 objects/s recovering 2024-09-21T10:12:39.114 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:39.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:39.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:39.216 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:39.237 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:39.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:39.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:39.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:39.360 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:39.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:39.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:39.461 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:39.506 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:39.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:39.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:39.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:39.647 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:39.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:39.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:39.748 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:39.770 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:39.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:39.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:39.872 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:39.896 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:39.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:39.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:39.997 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:40.025 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:40.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:40.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:40.127 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:40.155 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:40.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:40.257 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:40.281 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:40.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:40.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:40.382 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:40.410 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:40.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:40.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:40.511 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:40.539 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:40.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:40.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:40.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:40.655 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:12:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:12:40.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:12:40.671 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:12:40.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:40.685 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:40.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:40.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:40.823 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:40.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:40.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:40 smithi107 ceph-mon[102221]: pgmap v993: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 8252/143433 objects misplaced (5.753%); 14 MiB/s, 225 objects/s recovering 2024-09-21T10:12:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:40 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3328376965' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:40 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:40 smithi107 ceph-mon[105022]: pgmap v993: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 8252/143433 objects misplaced (5.753%); 14 MiB/s, 225 objects/s recovering 2024-09-21T10:12:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:40 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3328376965' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:40 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:40 smithi129 ceph-mon[88471]: pgmap v993: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 8252/143433 objects misplaced (5.753%); 14 MiB/s, 225 objects/s recovering 2024-09-21T10:12:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:40 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3328376965' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:40 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:12:40.924 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:40.952 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:40.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:40.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:41.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:41.088 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:41.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:41.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:41.190 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:41.215 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:41.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:41.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:41.317 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:41.342 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:41.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:41.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:41.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:41.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:41.489 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:41.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:41.589 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:41.615 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:41.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:41.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:41.717 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:41.750 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:41.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:41.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:41.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:41 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:12:41.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:41 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:12:41.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:41 smithi107 ceph-mon[102221]: osdmap e280: 8 total, 8 up, 7 in 2024-09-21T10:12:41.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:12:41.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:41.772 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:41 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:12:41.772 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:41 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:12:41.772 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:41 smithi107 ceph-mon[105022]: osdmap e280: 8 total, 8 up, 7 in 2024-09-21T10:12:41.772 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:12:41.772 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:41.852 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:41.881 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:41.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:41.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:41 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:12:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:41 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:12:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:41 smithi129 ceph-mon[88471]: osdmap e280: 8 total, 8 up, 7 in 2024-09-21T10:12:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:12:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:41.983 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:42.010 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:42.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:42.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:42.086 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:12:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:12:41] "GET /metrics HTTP/1.1" 200 36812 "" "Prometheus/2.51.0" 2024-09-21T10:12:42.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:42.158 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:42.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:42.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:42.260 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:42.286 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:42.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:42.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:42.388 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:42.421 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:42.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:42.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:42.523 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:42.550 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:42.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:42.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:42.562 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:42.570 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:42.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:42.693 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:42.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:42.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:42.705 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:42 smithi107 ceph-mon[102221]: pgmap v995: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 525 op/s; 8252/143433 objects misplaced (5.753%); 11 MiB/s, 169 objects/s recovering 2024-09-21T10:12:42.705 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:42 smithi107 ceph-mon[105022]: pgmap v995: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 525 op/s; 8252/143433 objects misplaced (5.753%); 11 MiB/s, 169 objects/s recovering 2024-09-21T10:12:42.795 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:42.822 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:42.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:42.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:42 smithi129 ceph-mon[88471]: pgmap v995: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 525 op/s; 8252/143433 objects misplaced (5.753%); 11 MiB/s, 169 objects/s recovering 2024-09-21T10:12:42.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:42.967 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:42.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:42.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:43.068 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:43.085 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:12:43.085 INFO:teuthology.orchestra.run.smithi107.stdout:{"fsid":"86c49514-77fc-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1521,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":280,"num_osds":8,"num_up_osds":8,"osd_up_since":1726913362,"num_in_osds":7,"osd_in_since":1726913317,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":106},{"state_name":"active+remapped+backfill_wait","count":2},{"state_name":"active+remapped+backfilling","count":1}],"num_pgs":109,"num_pools":5,"num_objects":47811,"data_bytes":3132201023,"bytes_used":15147454464,"bytes_avail":656817795072,"bytes_total":671965249536,"misplaced_objects":8252,"misplaced_total":143433,"misplaced_ratio":0.057532088152656639,"recovering_objects_per_sec":169,"recovering_bytes_per_sec":11077635,"recovering_keys_per_sec":0,"num_objects_recovered":1152,"num_bytes_recovered":75497472,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":525},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi107.sedied","status":"up:active","gid":24557}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.107:8443/","prometheus":"http://172.21.15.107:9283/"}},"servicemap":{"epoch":30,"modified":"2024-09-21T10:10:23.246058+0000","services":{}},"progress_events":{"24983b40-6a19-4e8e-8a8a-55bfd2201591":{"message":"Global Recovery Event (0s)\n [............................] ","progress":0,"add_to_ceph_s":true},"2fcfd131-0268-49fe-9638-9f737ef274d3":{"message":"Upgrade to 19.3.0-5092-g62da54b4 (9m)\n [============................] (remaining: 11m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-09-21T10:12:43.092 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:43.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:43.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:43.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:43.228 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:43.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:43.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:43.330 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:43.370 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:43.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:43.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:43.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:43.513 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:43.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:43.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:43.615 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:43.644 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:43.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:43.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:43.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:43.784 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:43.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:43.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:43 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3462262301' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:12:43.926 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:43.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:43.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:43.973 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:43 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3462262301' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:12:43.973 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:43 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3462262301' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:12:43.987 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:12:44.028 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:44.054 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:44.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:44.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:44.155 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:44.182 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:44.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:44.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:44.284 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:44.311 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:44.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:44.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:44.413 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:44.440 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:44.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:44.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:44.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:44.586 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:44.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:44.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:44.688 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:44.713 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:44.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:44.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:44.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:44.851 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:44.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:44.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:44.953 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:44.984 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:44.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:44.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:45.086 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:44 smithi107 ceph-mon[102221]: pgmap v996: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 877 op/s; 6915/133116 objects misplaced (5.195%); 14 MiB/s, 218 objects/s recovering 2024-09-21T10:12:45.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:44 smithi107 ceph-mon[105022]: pgmap v996: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 877 op/s; 6915/133116 objects misplaced (5.195%); 14 MiB/s, 218 objects/s recovering 2024-09-21T10:12:45.112 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:45.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:45.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:45.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:44 smithi129 ceph-mon[88471]: pgmap v996: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 877 op/s; 6915/133116 objects misplaced (5.195%); 14 MiB/s, 218 objects/s recovering 2024-09-21T10:12:45.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:45.221 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-21T10:12:45.221 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:12:45.257 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:45.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:45.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:45.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:45.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:45.409 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:45.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:45.510 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:45.534 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:45.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:45.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:45.636 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:45.664 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:45.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:45.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:45.765 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:45.797 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:45.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:45.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:45.898 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:45.922 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:45.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:45.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:46.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:46.065 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:46.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:46.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:46.167 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:46.194 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:46.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:46.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:46.296 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:46.324 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:46.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:46.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:46.425 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:46.465 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:46.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:46.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:46.561 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:46.566 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:46.590 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:46.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:46.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:46.628 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:46.692 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:46.715 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:46.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:46.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:46.790 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:46 smithi107 ceph-mon[102221]: pgmap v997: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 794 op/s; 6915/130761 objects misplaced (5.288%); 11 MiB/s, 178 objects/s recovering 2024-09-21T10:12:46.794 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:46 smithi107 ceph-mon[105022]: pgmap v997: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 794 op/s; 6915/130761 objects misplaced (5.288%); 11 MiB/s, 178 objects/s recovering 2024-09-21T10:12:46.817 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:46.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:46.860 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:46.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:46.961 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:47.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:47.002 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:47.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:47.103 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:47.136 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:47.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:47.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:47.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:46 smithi129 ceph-mon[88471]: pgmap v997: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 794 op/s; 6915/130761 objects misplaced (5.288%); 11 MiB/s, 178 objects/s recovering 2024-09-21T10:12:47.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:47.270 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:47.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:47.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:47.371 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:47.394 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:47.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:47.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:47.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:47.542 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:47.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:47.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:47.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:47.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:47.678 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:47.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:47.779 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:47.802 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:47.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:47.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:47.904 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:47.931 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:47.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:47.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:48.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:48.065 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:48.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:48.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:48.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:48.210 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:48.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:48.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:48.312 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:48.336 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:48.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:48.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:48.438 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:48.464 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:48.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:48.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:48.565 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:48.589 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:48.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:48.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:48.691 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:48.717 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:48.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:48.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:48.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:48.854 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:48.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:48.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:48.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:48 smithi107 ceph-mon[102221]: pgmap v998: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 836 op/s; 5608/121635 objects misplaced (4.611%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T10:12:48.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:48 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/945569515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:48.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:48 smithi107 ceph-mon[105022]: pgmap v998: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 836 op/s; 5608/121635 objects misplaced (4.611%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T10:12:48.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:48 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/945569515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:48.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:48.976 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:48.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:48.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:49.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:49.116 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:49.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:49.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:48 smithi129 ceph-mon[88471]: pgmap v998: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 836 op/s; 5608/121635 objects misplaced (4.611%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T10:12:49.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:48 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/945569515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:12:49.217 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:49.241 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:49.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:49.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:49.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:49.384 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:49.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:49.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:49.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:49.521 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:49.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:49.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:49.623 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:49.658 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:49.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:49.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:49.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:49.796 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:49.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:49.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:49.898 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:49.923 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:49.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:49.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:50.025 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:50.046 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:50.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:50.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:50.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:50.184 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:50.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:50.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:50.286 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:50.319 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:50.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:50.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:50.421 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:50.444 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:50.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:50.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:50.546 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:50.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:50.577 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:50.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:12:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:12:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:12:50.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:50.682 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:12:50.682 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:12:50.682 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:12:50.682 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:12:50.682 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:12:50.682 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:12:50.682 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:12:50.682 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:12:50.682 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:12:50.683 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:12:50.683 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:12:50.683 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:12:50.683 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:12:50.683 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:12:50.683 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:12:50.683 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:12:50.715 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:50.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:50.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:50.817 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:50.860 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:50.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:50.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:50.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:51.000 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:51.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:51.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:51.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:51.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:50 smithi107 ceph-mon[102221]: pgmap v999: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 860 op/s; 5608/117612 objects misplaced (4.768%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T10:12:51.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:50 smithi107 ceph-mon[105022]: pgmap v999: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 860 op/s; 5608/117612 objects misplaced (4.768%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T10:12:51.138 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:51.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:51.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:51.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:50 smithi129 ceph-mon[88471]: pgmap v999: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 860 op/s; 5608/117612 objects misplaced (4.768%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T10:12:51.239 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:51.264 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:51.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:51.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:51.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:51.401 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:51.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:51.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:51.503 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:51.530 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:51.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:51.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:51.632 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:51.659 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:51.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:51.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:51.761 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:51.789 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:51.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:51.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:51.891 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:51.916 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:51.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:51.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:52.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:52.049 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:12:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:12:51] "GET /metrics HTTP/1.1" 200 36812 "" "Prometheus/2.51.0" 2024-09-21T10:12:52.068 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:52.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:52.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:52.135 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:12:52.135 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:12:52.170 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:52.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:52.198 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:52.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:52.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:52.354 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:52.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:52.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:52.455 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:52.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:52.493 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:52.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:52.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:52.633 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:52.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:52.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:52.734 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:52.761 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:52.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:52.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:52.852 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:52 smithi107 ceph-mon[102221]: pgmap v1000: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 5608/117612 objects misplaced (4.768%); 7.2 MiB/s, 114 objects/s recovering 2024-09-21T10:12:52.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:52 smithi107 ceph-mon[105022]: pgmap v1000: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 5608/117612 objects misplaced (4.768%); 7.2 MiB/s, 114 objects/s recovering 2024-09-21T10:12:52.863 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:52.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:12:52.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:52.909 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:52.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:53.010 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:53.035 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:53.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:53.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:53.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:52 smithi129 ceph-mon[88471]: pgmap v1000: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 5608/117612 objects misplaced (4.768%); 7.2 MiB/s, 114 objects/s recovering 2024-09-21T10:12:53.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:53.175 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:53.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:53.276 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:53.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:53.329 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:53.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:53.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:53.478 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:53.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:53.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:53.580 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:53.615 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:53.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:53.715 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:53.741 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:53.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:53.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:53.843 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:53.870 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:53.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:53.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:53.972 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:53.996 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:53.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:53.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:54.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:12:54.098 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:54.122 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:54.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:54.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:54.225 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:54.246 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:54.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:54.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:54.348 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:54.369 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:54.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:54.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:54.471 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:54.492 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:54.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:54.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:54.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:54.616 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:54.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:54.705 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:12:54.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:54.761 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:54.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:54.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:54.863 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:54.891 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:54.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:54.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:54.993 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:55.016 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:54 smithi107 ceph-mon[102221]: pgmap v1001: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 4472/106467 objects misplaced (4.200%); 9.1 MiB/s, 145 objects/s recovering 2024-09-21T10:12:55.017 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:54 smithi107 ceph-mon[102221]: osdmap e281: 8 total, 8 up, 7 in 2024-09-21T10:12:55.017 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:54 smithi107 ceph-mon[105022]: pgmap v1001: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 4472/106467 objects misplaced (4.200%); 9.1 MiB/s, 145 objects/s recovering 2024-09-21T10:12:55.017 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:54 smithi107 ceph-mon[105022]: osdmap e281: 8 total, 8 up, 7 in 2024-09-21T10:12:55.019 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:55.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:55.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:55.121 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:55.147 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:55.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:55.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:54 smithi129 ceph-mon[88471]: pgmap v1001: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 4472/106467 objects misplaced (4.200%); 9.1 MiB/s, 145 objects/s recovering 2024-09-21T10:12:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:54 smithi129 ceph-mon[88471]: osdmap e281: 8 total, 8 up, 7 in 2024-09-21T10:12:55.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:55.297 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:55.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:55.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:55.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:55.432 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:55.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:55.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:55.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:55.571 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:55.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:55.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:55.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:55.714 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:55.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:55.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:55.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:55.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:55.860 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:55.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:55.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:56.002 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:56.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:56.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:56.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:55 smithi107 ceph-mon[102221]: osdmap e282: 8 total, 8 up, 7 in 2024-09-21T10:12:56.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:55 smithi107 ceph-mon[105022]: osdmap e282: 8 total, 8 up, 7 in 2024-09-21T10:12:56.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:56.147 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:56.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:56.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:55 smithi129 ceph-mon[88471]: osdmap e282: 8 total, 8 up, 7 in 2024-09-21T10:12:56.248 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:56.272 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:56.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:56.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:56.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:56.414 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:56.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:56.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:56.515 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:56.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:56.538 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:56.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:56.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:56.680 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:56.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:56.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:56.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:56.802 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:56.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:56.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:56.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:56.948 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:56.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:56.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:57.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:57.085 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:57.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:57.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:57.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:56 smithi107 ceph-mon[102221]: pgmap v1004: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 731 op/s; 4472/104082 objects misplaced (4.297%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T10:12:57.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:57.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:56 smithi107 ceph-mon[105022]: pgmap v1004: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 731 op/s; 4472/104082 objects misplaced (4.297%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T10:12:57.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:56 smithi129 ceph-mon[88471]: pgmap v1004: 109 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_wait, 106 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 731 op/s; 4472/104082 objects misplaced (4.297%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T10:12:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:57.187 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:57.208 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:57.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:57.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:57.310 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:57.331 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:57.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:57.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:57.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:57.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:57.458 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:57.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:57.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:12:57.590 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:57.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:12:57.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:57.729 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:57.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:57.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:57.831 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:12:57.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:57.859 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:57.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:12:57.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:57.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:58.000 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:58.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:58.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:12:58.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:57 smithi107 ceph-mon[102221]: from='client.56116 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:12:58.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:57 smithi107 ceph-mon[105022]: from='client.56116 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:12:58.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:58.132 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:58.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:12:58.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:57 smithi129 ceph-mon[88471]: from='client.56116 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:12:58.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:58.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:58.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:58.364 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:58.394 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:58.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:58.496 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:12:58.526 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:58.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:58.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:12:58.628 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:58.657 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:58.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:58.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:58.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:58.767 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:12:58.806 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:58.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:58.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:58.908 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:58.932 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:58.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:58.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:59.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:59.073 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:59.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:59.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:59.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:58 smithi107 ceph-mon[102221]: pgmap v1005: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 874 op/s; 2921/96630 objects misplaced (3.023%); 13 MiB/s, 208 objects/s recovering 2024-09-21T10:12:59.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:58 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3471105768' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:12:59.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:58 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:12:59.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:58 smithi107 ceph-mon[105022]: pgmap v1005: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 874 op/s; 2921/96630 objects misplaced (3.023%); 13 MiB/s, 208 objects/s recovering 2024-09-21T10:12:59.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:58 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3471105768' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:12:59.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:58 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:12:59.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:58 smithi129 ceph-mon[88471]: pgmap v1005: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 874 op/s; 2921/96630 objects misplaced (3.023%); 13 MiB/s, 208 objects/s recovering 2024-09-21T10:12:59.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:58 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3471105768' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:12:59.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:58 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:12:59.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:59.207 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:59.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:59.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:59.309 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:59.347 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:59.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:59.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:59.448 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:12:59.473 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:59.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:59.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:12:59.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:12:59.606 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:59.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:59.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:12:59.710 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:12:59.733 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:12:59.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:59.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:12:59.737 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:12:59.739 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:12:59.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:12:59.871 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:12:59.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:12:59.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:12:59.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:00.013 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:00.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:00.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:00.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:59 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:13:00.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:59 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:13:00.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:12:59 smithi107 ceph-mon[102221]: osdmap e283: 8 total, 8 up, 7 in 2024-09-21T10:13:00.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:59 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:13:00.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:59 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:13:00.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:12:59 smithi107 ceph-mon[105022]: osdmap e283: 8 total, 8 up, 7 in 2024-09-21T10:13:00.116 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:00.141 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:00.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:00.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:00.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:59 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:13:00.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:59 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:13:00.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:12:59 smithi129 ceph-mon[88471]: osdmap e283: 8 total, 8 up, 7 in 2024-09-21T10:13:00.244 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:00.270 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:00.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:00.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:00.371 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:00.398 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:00.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:00.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:00.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:00.543 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:00.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:00.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:00.645 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:13:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:13:00.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:13:00.673 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:00.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:00.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:00.775 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:00.805 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:00.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:00.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:00.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:00 smithi107 ceph-mon[102221]: pgmap v1007: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 713 op/s; 2921/93618 objects misplaced (3.120%); 12 MiB/s, 190 objects/s recovering 2024-09-21T10:13:00.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:00 smithi107 ceph-mon[105022]: pgmap v1007: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 713 op/s; 2921/93618 objects misplaced (3.120%); 12 MiB/s, 190 objects/s recovering 2024-09-21T10:13:00.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:00.945 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:00.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:00.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:01.012 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1004,"stamp":"2024-09-21T10:12:55.302216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2272565311,"num_objects":34694,"num_object_clones":0,"num_object_copies":104082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4472,"num_objects_unfound":0,"num_objects_dirty":34694,"num_whiteouts":0,"num_read":50,"num_read_kb":126,"num_write":190392,"num_write_kb":7199280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8079,"num_bytes_recovered":526267059,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104062,"ondisk_log_size":104062,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":330,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":12308260,"kb_used_data":6976680,"kb_used_omap":41,"kb_used_meta":5331286,"kb_avail":643907804,"statfs":{"total":671965249536,"available":659361591296,"internally_reserved":0,"allocated":7144120320,"data_stored":7133027014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42519,"internal_metadata":5459237353},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,3,7,6,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-383451136,"num_objects":-5851,"num_object_clones":0,"num_object_copies":-17553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1136,"num_objects_unfound":0,"num_objects_dirty":-5851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5851,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":526,"num_bytes_recovered":34471936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002307"},"pg_stats":[{"pgid":"1.11","version":"77'100","reported_seq":76,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.694801+0000","last_change":"2024-09-21T10:10:57.397786+0000","last_active":"2024-09-21T10:12:40.694801+0000","last_peered":"2024-09-21T10:12:40.694801+0000","last_clean":"2024-09-21T10:12:40.694801+0000","last_became_active":"2024-09-21T10:10:57.396742+0000","last_became_peered":"2024-09-21T10:10:57.396742+0000","last_unstale":"2024-09-21T10:12:40.694801+0000","last_undegraded":"2024-09-21T10:12:40.694801+0000","last_fullsized":"2024-09-21T10:12:40.694801+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:53.631923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:53.631923+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:33:35.364595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":414,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.685559+0000","last_change":"2024-09-21T10:03:38.389577+0000","last_active":"2024-09-21T10:12:40.685559+0000","last_peered":"2024-09-21T10:12:40.685559+0000","last_clean":"2024-09-21T10:12:40.685559+0000","last_became_active":"2024-09-21T10:03:38.389292+0000","last_became_peered":"2024-09-21T10:03:38.389292+0000","last_unstale":"2024-09-21T10:12:40.685559+0000","last_undegraded":"2024-09-21T10:12:40.685559+0000","last_fullsized":"2024-09-21T10:12:40.685559+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:32.113472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":150,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.480264+0000","last_change":"2024-09-21T10:08:38.387871+0000","last_active":"2024-09-21T10:12:35.480264+0000","last_peered":"2024-09-21T10:12:35.480264+0000","last_clean":"2024-09-21T10:12:35.480264+0000","last_became_active":"2024-09-21T10:08:38.376727+0000","last_became_peered":"2024-09-21T10:08:38.376727+0000","last_unstale":"2024-09-21T10:12:35.480264+0000","last_undegraded":"2024-09-21T10:12:35.480264+0000","last_fullsized":"2024-09-21T10:12:35.480264+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:45:54.498950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'100","reported_seq":179,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.480024+0000","last_change":"2024-09-21T10:11:05.438031+0000","last_active":"2024-09-21T10:12:35.480024+0000","last_peered":"2024-09-21T10:12:35.480024+0000","last_clean":"2024-09-21T10:12:35.480024+0000","last_became_active":"2024-09-21T10:11:05.437637+0000","last_became_peered":"2024-09-21T10:11:05.437637+0000","last_unstale":"2024-09-21T10:12:35.480024+0000","last_undegraded":"2024-09-21T10:12:35.480024+0000","last_fullsized":"2024-09-21T10:12:35.480024+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.798577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.798577+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:42.343217+0000","scrub_duration":0.0023622640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":271,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.480663+0000","last_change":"2024-09-21T10:10:57.392782+0000","last_active":"2024-09-21T10:12:35.480663+0000","last_peered":"2024-09-21T10:12:35.480663+0000","last_clean":"2024-09-21T10:12:35.480663+0000","last_became_active":"2024-09-21T10:10:57.392305+0000","last_became_peered":"2024-09-21T10:10:57.392305+0000","last_unstale":"2024-09-21T10:12:35.480663+0000","last_undegraded":"2024-09-21T10:12:35.480663+0000","last_fullsized":"2024-09-21T10:12:35.480663+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:42:01.387622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":335,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.685908+0000","last_change":"2024-09-21T10:09:23.846703+0000","last_active":"2024-09-21T10:12:40.685908+0000","last_peered":"2024-09-21T10:12:40.685908+0000","last_clean":"2024-09-21T10:12:40.685908+0000","last_became_active":"2024-09-21T10:09:23.846415+0000","last_became_peered":"2024-09-21T10:09:23.846415+0000","last_unstale":"2024-09-21T10:12:40.685908+0000","last_undegraded":"2024-09-21T10:12:40.685908+0000","last_fullsized":"2024-09-21T10:12:40.685908+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:23.129285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'100","reported_seq":129,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.684269+0000","last_change":"2024-09-21T10:09:24.841480+0000","last_active":"2024-09-21T10:12:40.684269+0000","last_peered":"2024-09-21T10:12:40.684269+0000","last_clean":"2024-09-21T10:12:40.684269+0000","last_became_active":"2024-09-21T10:09:24.841145+0000","last_became_peered":"2024-09-21T10:09:24.841145+0000","last_unstale":"2024-09-21T10:12:40.684269+0000","last_undegraded":"2024-09-21T10:12:40.684269+0000","last_fullsized":"2024-09-21T10:12:40.684269+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.433227+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":23,"num_write":5,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":35780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":61,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.683597+0000","last_change":"2024-09-21T10:09:49.922489+0000","last_active":"2024-09-21T10:12:40.683597+0000","last_peered":"2024-09-21T10:12:40.683597+0000","last_clean":"2024-09-21T10:12:40.683597+0000","last_became_active":"2024-09-21T10:09:49.921629+0000","last_became_peered":"2024-09-21T10:09:49.921629+0000","last_unstale":"2024-09-21T10:12:40.683597+0000","last_undegraded":"2024-09-21T10:12:40.683597+0000","last_fullsized":"2024-09-21T10:12:40.683597+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:50.611979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":152,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.479993+0000","last_change":"2024-09-21T10:09:24.836647+0000","last_active":"2024-09-21T10:12:35.479993+0000","last_peered":"2024-09-21T10:12:35.479993+0000","last_clean":"2024-09-21T10:12:35.479993+0000","last_became_active":"2024-09-21T10:09:24.836278+0000","last_became_peered":"2024-09-21T10:09:24.836278+0000","last_unstale":"2024-09-21T10:12:35.479993+0000","last_undegraded":"2024-09-21T10:12:35.479993+0000","last_fullsized":"2024-09-21T10:12:35.479993+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:08:36.535892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"77'100","reported_seq":23,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.482046+0000","last_change":"2024-09-21T10:10:57.452531+0000","last_active":"2024-09-21T10:12:35.482046+0000","last_peered":"2024-09-21T10:12:35.482046+0000","last_clean":"2024-09-21T10:12:35.482046+0000","last_became_active":"2024-09-21T10:10:57.452150+0000","last_became_peered":"2024-09-21T10:10:57.452150+0000","last_unstale":"2024-09-21T10:12:35.482046+0000","last_undegraded":"2024-09-21T10:12:35.482046+0000","last_fullsized":"2024-09-21T10:12:35.482046+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:00.058077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:00.058077+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":413,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.686784+0000","last_change":"2024-09-21T10:03:38.381552+0000","last_active":"2024-09-21T10:12:40.686784+0000","last_peered":"2024-09-21T10:12:40.686784+0000","last_clean":"2024-09-21T10:12:40.686784+0000","last_became_active":"2024-09-21T10:03:38.380898+0000","last_became_peered":"2024-09-21T10:03:38.380898+0000","last_unstale":"2024-09-21T10:12:40.686784+0000","last_undegraded":"2024-09-21T10:12:40.686784+0000","last_fullsized":"2024-09-21T10:12:40.686784+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:53.409606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":339,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.482715+0000","last_change":"2024-09-21T10:10:57.459057+0000","last_active":"2024-09-21T10:12:35.482715+0000","last_peered":"2024-09-21T10:12:35.482715+0000","last_clean":"2024-09-21T10:12:35.482715+0000","last_became_active":"2024-09-21T10:10:57.451776+0000","last_became_peered":"2024-09-21T10:10:57.451776+0000","last_unstale":"2024-09-21T10:12:35.482715+0000","last_undegraded":"2024-09-21T10:12:35.482715+0000","last_fullsized":"2024-09-21T10:12:35.482715+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:00.138317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"280'11834","reported_seq":11947,"reported_epoch":280,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:12:52.032250+0000","last_change":"2024-09-21T10:10:58.805957+0000","last_active":"2024-09-21T10:12:52.032250+0000","last_peered":"2024-09-21T10:12:52.032250+0000","last_clean":"2024-09-21T10:10:56.364182+0000","last_became_active":"2024-09-21T10:10:58.682930+0000","last_became_peered":"2024-09-21T10:10:58.682930+0000","last_unstale":"2024-09-21T10:12:52.032250+0000","last_undegraded":"2024-09-21T10:12:52.032250+0000","last_fullsized":"2024-09-21T10:12:52.032250+0000","mapping_epoch":267,"log_start":"265'5500","ondisk_log_start":"265'5500","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6334,"log_dups_size":0,"ondisk_log_size":6334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140902400,"num_objects":2150,"num_object_clones":0,"num_object_copies":6450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2143,"num_objects_unfound":0,"num_objects_dirty":2150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11834,"num_write_kb":447488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[{"shards":"0,1,7","objects":2150}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"77'100","reported_seq":164,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.480587+0000","last_change":"2024-09-21T10:08:38.374573+0000","last_active":"2024-09-21T10:12:35.480587+0000","last_peered":"2024-09-21T10:12:35.480587+0000","last_clean":"2024-09-21T10:12:35.480587+0000","last_became_active":"2024-09-21T10:08:38.374075+0000","last_became_peered":"2024-09-21T10:08:38.374075+0000","last_unstale":"2024-09-21T10:12:35.480587+0000","last_undegraded":"2024-09-21T10:12:35.480587+0000","last_fullsized":"2024-09-21T10:12:35.480587+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:55.259385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:55.259385+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:28:26.784162+0000","scrub_duration":0.0015667260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":195,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.480481+0000","last_change":"2024-09-21T10:09:49.941392+0000","last_active":"2024-09-21T10:12:35.480481+0000","last_peered":"2024-09-21T10:12:35.480481+0000","last_clean":"2024-09-21T10:12:35.480481+0000","last_became_active":"2024-09-21T10:09:49.923535+0000","last_became_peered":"2024-09-21T10:09:49.923535+0000","last_unstale":"2024-09-21T10:12:35.480481+0000","last_undegraded":"2024-09-21T10:12:35.480481+0000","last_fullsized":"2024-09-21T10:12:35.480481+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:16.292163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":33,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.481972+0000","last_change":"2024-09-21T10:11:05.435502+0000","last_active":"2024-09-21T10:12:35.481972+0000","last_peered":"2024-09-21T10:12:35.481972+0000","last_clean":"2024-09-21T10:12:35.481972+0000","last_became_active":"2024-09-21T10:11:05.435216+0000","last_became_peered":"2024-09-21T10:11:05.435216+0000","last_unstale":"2024-09-21T10:12:35.481972+0000","last_undegraded":"2024-09-21T10:12:35.481972+0000","last_fullsized":"2024-09-21T10:12:35.481972+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:25.434424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"280'11664","reported_seq":11726,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:52.034964+0000","last_change":"2024-09-21T10:09:49.924187+0000","last_active":"2024-09-21T10:12:52.034964+0000","last_peered":"2024-09-21T10:12:52.034964+0000","last_clean":"2024-09-21T10:12:52.034964+0000","last_became_active":"2024-09-21T10:09:49.923789+0000","last_became_peered":"2024-09-21T10:09:49.923789+0000","last_unstale":"2024-09-21T10:12:52.034964+0000","last_undegraded":"2024-09-21T10:12:52.034964+0000","last_fullsized":"2024-09-21T10:12:52.034964+0000","mapping_epoch":255,"log_start":"265'5400","ondisk_log_start":"265'5400","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6264,"log_dups_size":0,"ondisk_log_size":6264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:52:49.559425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144310272,"num_objects":2202,"num_object_clones":0,"num_object_copies":6606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11664,"num_write_kb":443712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"77'100","reported_seq":220,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.685414+0000","last_change":"2024-09-21T10:09:23.839211+0000","last_active":"2024-09-21T10:12:40.685414+0000","last_peered":"2024-09-21T10:12:40.685414+0000","last_clean":"2024-09-21T10:12:40.685414+0000","last_became_active":"2024-09-21T10:09:23.838779+0000","last_became_peered":"2024-09-21T10:09:23.838779+0000","last_unstale":"2024-09-21T10:12:40.685414+0000","last_undegraded":"2024-09-21T10:12:40.685414+0000","last_fullsized":"2024-09-21T10:12:40.685414+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.981336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.981336+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:03:15.566873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":174,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.480604+0000","last_change":"2024-09-21T10:09:49.932631+0000","last_active":"2024-09-21T10:12:35.480604+0000","last_peered":"2024-09-21T10:12:35.480604+0000","last_clean":"2024-09-21T10:12:35.480604+0000","last_became_active":"2024-09-21T10:09:49.922897+0000","last_became_peered":"2024-09-21T10:09:49.922897+0000","last_unstale":"2024-09-21T10:12:35.480604+0000","last_undegraded":"2024-09-21T10:12:35.480604+0000","last_fullsized":"2024-09-21T10:12:35.480604+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:40.795476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":208,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.687564+0000","last_change":"2024-09-21T10:08:38.377732+0000","last_active":"2024-09-21T10:12:40.687564+0000","last_peered":"2024-09-21T10:12:40.687564+0000","last_clean":"2024-09-21T10:12:40.687564+0000","last_became_active":"2024-09-21T10:08:38.376601+0000","last_became_peered":"2024-09-21T10:08:38.376601+0000","last_unstale":"2024-09-21T10:12:40.687564+0000","last_undegraded":"2024-09-21T10:12:40.687564+0000","last_fullsized":"2024-09-21T10:12:40.687564+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:02.540158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.5","version":"280'11417","reported_seq":22430,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:51.848179+0000","last_change":"2024-09-21T10:12:35.556125+0000","last_active":"2024-09-21T10:12:51.848179+0000","last_peered":"2024-09-21T10:12:51.848179+0000","last_clean":"2024-09-21T10:12:51.848179+0000","last_became_active":"2024-09-21T10:12:35.483696+0000","last_became_peered":"2024-09-21T10:12:35.483696+0000","last_unstale":"2024-09-21T10:12:51.848179+0000","last_undegraded":"2024-09-21T10:12:51.848179+0000","last_fullsized":"2024-09-21T10:12:51.848179+0000","mapping_epoch":278,"log_start":"265'5100","ondisk_log_start":"265'5100","created":252,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6317,"log_dups_size":0,"ondisk_log_size":6317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:54:35.212835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137560064,"num_objects":2099,"num_object_clones":0,"num_object_copies":6297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11417,"num_write_kb":432512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5478,"num_bytes_recovered":359006208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"77'100","reported_seq":264,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.483089+0000","last_change":"2024-09-21T10:08:38.382100+0000","last_active":"2024-09-21T10:12:35.483089+0000","last_peered":"2024-09-21T10:12:35.483089+0000","last_clean":"2024-09-21T10:12:35.483089+0000","last_became_active":"2024-09-21T10:08:38.381736+0000","last_became_peered":"2024-09-21T10:08:38.381736+0000","last_unstale":"2024-09-21T10:12:35.483089+0000","last_undegraded":"2024-09-21T10:12:35.483089+0000","last_fullsized":"2024-09-21T10:12:35.483089+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.289859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.289859+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:23.739980+0000","scrub_duration":0.001251809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":160,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.480421+0000","last_change":"2024-09-21T10:10:57.460732+0000","last_active":"2024-09-21T10:12:35.480421+0000","last_peered":"2024-09-21T10:12:35.480421+0000","last_clean":"2024-09-21T10:12:35.480421+0000","last_became_active":"2024-09-21T10:10:57.454689+0000","last_became_peered":"2024-09-21T10:10:57.454689+0000","last_unstale":"2024-09-21T10:12:35.480421+0000","last_undegraded":"2024-09-21T10:12:35.480421+0000","last_fullsized":"2024-09-21T10:12:35.480421+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:12.035022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":280,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.482798+0000","last_change":"2024-09-21T10:03:38.378384+0000","last_active":"2024-09-21T10:12:35.482798+0000","last_peered":"2024-09-21T10:12:35.482798+0000","last_clean":"2024-09-21T10:12:35.482798+0000","last_became_active":"2024-09-21T10:03:38.378072+0000","last_became_peered":"2024-09-21T10:03:38.378072+0000","last_unstale":"2024-09-21T10:12:35.482798+0000","last_undegraded":"2024-09-21T10:12:35.482798+0000","last_fullsized":"2024-09-21T10:12:35.482798+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:54:37.039622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"280'11973","reported_seq":12009,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:51.763805+0000","last_change":"2024-09-21T10:09:42.896949+0000","last_active":"2024-09-21T10:12:51.763805+0000","last_peered":"2024-09-21T10:12:51.763805+0000","last_clean":"2024-09-21T10:12:51.763805+0000","last_became_active":"2024-09-21T10:09:42.896669+0000","last_became_peered":"2024-09-21T10:09:42.896669+0000","last_unstale":"2024-09-21T10:12:51.763805+0000","last_undegraded":"2024-09-21T10:12:51.763805+0000","last_fullsized":"2024-09-21T10:12:51.763805+0000","mapping_epoch":252,"log_start":"265'5700","ondisk_log_start":"265'5700","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6273,"log_dups_size":0,"ondisk_log_size":6273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:44.279472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147521536,"num_objects":2251,"num_object_clones":0,"num_object_copies":6753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11973,"num_write_kb":455168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"77'100","reported_seq":132,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.480364+0000","last_change":"2024-09-21T10:07:26.188298+0000","last_active":"2024-09-21T10:12:35.480364+0000","last_peered":"2024-09-21T10:12:35.480364+0000","last_clean":"2024-09-21T10:12:35.480364+0000","last_became_active":"2024-09-21T10:07:24.345265+0000","last_became_peered":"2024-09-21T10:07:24.345265+0000","last_unstale":"2024-09-21T10:12:35.480364+0000","last_undegraded":"2024-09-21T10:12:35.480364+0000","last_fullsized":"2024-09-21T10:12:35.480364+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:07:26.188220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:07:26.188220+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:46.381710+0000","scrub_duration":0.021884888000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":218,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.685685+0000","last_change":"2024-09-21T10:07:57.551556+0000","last_active":"2024-09-21T10:12:40.685685+0000","last_peered":"2024-09-21T10:12:40.685685+0000","last_clean":"2024-09-21T10:12:40.685685+0000","last_became_active":"2024-09-21T10:07:57.551249+0000","last_became_peered":"2024-09-21T10:07:57.551249+0000","last_unstale":"2024-09-21T10:12:40.685685+0000","last_undegraded":"2024-09-21T10:12:40.685685+0000","last_fullsized":"2024-09-21T10:12:40.685685+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:57.557152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":315,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.683750+0000","last_change":"2024-09-21T10:09:24.839665+0000","last_active":"2024-09-21T10:12:40.683750+0000","last_peered":"2024-09-21T10:12:40.683750+0000","last_clean":"2024-09-21T10:12:40.683750+0000","last_became_active":"2024-09-21T10:09:24.839353+0000","last_became_peered":"2024-09-21T10:09:24.839353+0000","last_unstale":"2024-09-21T10:12:40.683750+0000","last_undegraded":"2024-09-21T10:12:40.683750+0000","last_fullsized":"2024-09-21T10:12:40.683750+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:51.760073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3","version":"280'11678","reported_seq":11714,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:52.810185+0000","last_change":"2024-09-21T10:09:43.231199+0000","last_active":"2024-09-21T10:12:52.810185+0000","last_peered":"2024-09-21T10:12:52.810185+0000","last_clean":"2024-09-21T10:12:52.810185+0000","last_became_active":"2024-09-21T10:09:43.230922+0000","last_became_peered":"2024-09-21T10:09:43.230922+0000","last_unstale":"2024-09-21T10:12:52.810185+0000","last_undegraded":"2024-09-21T10:12:52.810185+0000","last_fullsized":"2024-09-21T10:12:52.810185+0000","mapping_epoch":252,"log_start":"265'5500","ondisk_log_start":"265'5500","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6178,"log_dups_size":0,"ondisk_log_size":6178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:39.944122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144179200,"num_objects":2200,"num_object_clones":0,"num_object_copies":6600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11678,"num_write_kb":444096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":154,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.480081+0000","last_change":"2024-09-21T10:04:53.355975+0000","last_active":"2024-09-21T10:12:35.480081+0000","last_peered":"2024-09-21T10:12:35.480081+0000","last_clean":"2024-09-21T10:12:35.480081+0000","last_became_active":"2024-09-21T10:04:42.547505+0000","last_became_peered":"2024-09-21T10:04:42.547505+0000","last_unstale":"2024-09-21T10:12:35.480081+0000","last_undegraded":"2024-09-21T10:12:35.480081+0000","last_fullsized":"2024-09-21T10:12:35.480081+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.355912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.355912+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:43.456830+0000","scrub_duration":0.0017980030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":414,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.684433+0000","last_change":"2024-09-21T10:09:49.939553+0000","last_active":"2024-09-21T10:12:40.684433+0000","last_peered":"2024-09-21T10:12:40.684433+0000","last_clean":"2024-09-21T10:12:40.684433+0000","last_became_active":"2024-09-21T10:09:49.925094+0000","last_became_peered":"2024-09-21T10:09:49.925094+0000","last_unstale":"2024-09-21T10:12:40.684433+0000","last_undegraded":"2024-09-21T10:12:40.684433+0000","last_fullsized":"2024-09-21T10:12:40.684433+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:12:12.933089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":191,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.686554+0000","last_change":"2024-09-21T10:08:38.379747+0000","last_active":"2024-09-21T10:12:40.686554+0000","last_peered":"2024-09-21T10:12:40.686554+0000","last_clean":"2024-09-21T10:12:40.686554+0000","last_became_active":"2024-09-21T10:08:38.378983+0000","last_became_peered":"2024-09-21T10:08:38.378983+0000","last_unstale":"2024-09-21T10:12:40.686554+0000","last_undegraded":"2024-09-21T10:12:40.686554+0000","last_fullsized":"2024-09-21T10:12:40.686554+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:45:08.003016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"281'11924","reported_seq":11994,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-21T10:12:53.965881+0000","last_change":"2024-09-21T10:09:42.896430+0000","last_active":"2024-09-21T10:12:53.965881+0000","last_peered":"2024-09-21T10:12:53.965881+0000","last_clean":"2024-09-21T10:12:53.965881+0000","last_became_active":"2024-09-21T10:09:42.896019+0000","last_became_peered":"2024-09-21T10:09:42.896019+0000","last_unstale":"2024-09-21T10:12:53.965881+0000","last_undegraded":"2024-09-21T10:12:53.965881+0000","last_fullsized":"2024-09-21T10:12:53.965881+0000","mapping_epoch":252,"log_start":"263'4400","ondisk_log_start":"263'4400","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":7524,"log_dups_size":0,"ondisk_log_size":7524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:59:18.633498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135528448,"num_objects":2068,"num_object_clones":0,"num_object_copies":6204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11924,"num_write_kb":447744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":176,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-21T10:12:53.712405+0000","last_change":"2024-09-21T10:10:57.403722+0000","last_active":"2024-09-21T10:12:53.712405+0000","last_peered":"2024-09-21T10:12:53.712405+0000","last_clean":"2024-09-21T10:12:53.712405+0000","last_became_active":"2024-09-21T10:10:57.394700+0000","last_became_peered":"2024-09-21T10:10:57.394700+0000","last_unstale":"2024-09-21T10:12:53.712405+0000","last_undegraded":"2024-09-21T10:12:53.712405+0000","last_fullsized":"2024-09-21T10:12:53.712405+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:02.152981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:02.152981+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:52:34.887763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":254,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.482900+0000","last_change":"2024-09-21T10:08:38.379849+0000","last_active":"2024-09-21T10:12:35.482900+0000","last_peered":"2024-09-21T10:12:35.482900+0000","last_clean":"2024-09-21T10:12:35.482900+0000","last_became_active":"2024-09-21T10:08:38.379543+0000","last_became_peered":"2024-09-21T10:08:38.379543+0000","last_unstale":"2024-09-21T10:12:35.482900+0000","last_undegraded":"2024-09-21T10:12:35.482900+0000","last_fullsized":"2024-09-21T10:12:35.482900+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:29.041802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":225,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-21T10:12:53.715346+0000","last_change":"2024-09-21T10:07:24.342950+0000","last_active":"2024-09-21T10:12:53.715346+0000","last_peered":"2024-09-21T10:12:53.715346+0000","last_clean":"2024-09-21T10:12:53.715346+0000","last_became_active":"2024-09-21T10:07:24.342477+0000","last_became_peered":"2024-09-21T10:07:24.342477+0000","last_unstale":"2024-09-21T10:12:53.715346+0000","last_undegraded":"2024-09-21T10:12:53.715346+0000","last_fullsized":"2024-09-21T10:12:53.715346+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:33:13.664050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1","version":"280'12050","reported_seq":12086,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:52.824233+0000","last_change":"2024-09-21T10:09:42.895259+0000","last_active":"2024-09-21T10:12:52.824233+0000","last_peered":"2024-09-21T10:12:52.824233+0000","last_clean":"2024-09-21T10:12:52.824233+0000","last_became_active":"2024-09-21T10:09:42.895087+0000","last_became_peered":"2024-09-21T10:09:42.895087+0000","last_unstale":"2024-09-21T10:12:52.824233+0000","last_undegraded":"2024-09-21T10:12:52.824233+0000","last_fullsized":"2024-09-21T10:12:52.824233+0000","mapping_epoch":252,"log_start":"268'5900","ondisk_log_start":"268'5900","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6150,"log_dups_size":0,"ondisk_log_size":6150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:40:59.270926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135397376,"num_objects":2066,"num_object_clones":0,"num_object_copies":6198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12050,"num_write_kb":451712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"77'100","reported_seq":317,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.683923+0000","last_change":"2024-09-21T10:10:57.395906+0000","last_active":"2024-09-21T10:12:40.683923+0000","last_peered":"2024-09-21T10:12:40.683923+0000","last_clean":"2024-09-21T10:12:40.683923+0000","last_became_active":"2024-09-21T10:10:57.395366+0000","last_became_peered":"2024-09-21T10:10:57.395366+0000","last_unstale":"2024-09-21T10:12:40.683923+0000","last_undegraded":"2024-09-21T10:12:40.683923+0000","last_fullsized":"2024-09-21T10:12:40.683923+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.540323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.540323+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:34.873336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":317,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-21T10:12:53.712438+0000","last_change":"2024-09-21T10:07:36.464496+0000","last_active":"2024-09-21T10:12:53.712438+0000","last_peered":"2024-09-21T10:12:53.712438+0000","last_clean":"2024-09-21T10:12:53.712438+0000","last_became_active":"2024-09-21T10:07:36.464172+0000","last_became_peered":"2024-09-21T10:07:36.464172+0000","last_unstale":"2024-09-21T10:12:53.712438+0000","last_undegraded":"2024-09-21T10:12:53.712438+0000","last_fullsized":"2024-09-21T10:12:53.712438+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:09.839709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":20,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.483031+0000","last_change":"2024-09-21T10:11:05.442332+0000","last_active":"2024-09-21T10:12:35.483031+0000","last_peered":"2024-09-21T10:12:35.483031+0000","last_clean":"2024-09-21T10:12:35.483031+0000","last_became_active":"2024-09-21T10:11:05.442003+0000","last_became_peered":"2024-09-21T10:11:05.442003+0000","last_unstale":"2024-09-21T10:12:35.483031+0000","last_undegraded":"2024-09-21T10:12:35.483031+0000","last_fullsized":"2024-09-21T10:12:35.483031+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:34.986309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"280'11852","reported_seq":11920,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:51.476425+0000","last_change":"2024-09-21T10:09:43.231401+0000","last_active":"2024-09-21T10:12:51.476425+0000","last_peered":"2024-09-21T10:12:51.476425+0000","last_clean":"2024-09-21T10:12:51.476425+0000","last_became_active":"2024-09-21T10:09:43.230977+0000","last_became_peered":"2024-09-21T10:09:43.230977+0000","last_unstale":"2024-09-21T10:12:51.476425+0000","last_undegraded":"2024-09-21T10:12:51.476425+0000","last_fullsized":"2024-09-21T10:12:51.476425+0000","mapping_epoch":252,"log_start":"268'6000","ondisk_log_start":"268'6000","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5852,"log_dups_size":0,"ondisk_log_size":5852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:15.907809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145489920,"num_objects":2220,"num_object_clones":0,"num_object_copies":6660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11852,"num_write_kb":450304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":139,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-21T10:12:53.712779+0000","last_change":"2024-09-21T10:08:04.645700+0000","last_active":"2024-09-21T10:12:53.712779+0000","last_peered":"2024-09-21T10:12:53.712779+0000","last_clean":"2024-09-21T10:12:53.712779+0000","last_became_active":"2024-09-21T10:07:57.548189+0000","last_became_peered":"2024-09-21T10:07:57.548189+0000","last_unstale":"2024-09-21T10:12:53.712779+0000","last_undegraded":"2024-09-21T10:12:53.712779+0000","last_fullsized":"2024-09-21T10:12:53.712779+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:48.318010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:48.318010+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T16:31:28.173059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":415,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.687240+0000","last_change":"2024-09-21T10:03:38.386604+0000","last_active":"2024-09-21T10:12:40.687240+0000","last_peered":"2024-09-21T10:12:40.687240+0000","last_clean":"2024-09-21T10:12:40.687240+0000","last_became_active":"2024-09-21T10:03:38.386283+0000","last_became_peered":"2024-09-21T10:03:38.386283+0000","last_unstale":"2024-09-21T10:12:40.687240+0000","last_undegraded":"2024-09-21T10:12:40.687240+0000","last_fullsized":"2024-09-21T10:12:40.687240+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:27:05.944880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":218,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.686876+0000","last_change":"2024-09-21T10:08:38.372486+0000","last_active":"2024-09-21T10:12:40.686876+0000","last_peered":"2024-09-21T10:12:40.686876+0000","last_clean":"2024-09-21T10:12:40.686876+0000","last_became_active":"2024-09-21T10:08:38.370669+0000","last_became_peered":"2024-09-21T10:08:38.370669+0000","last_unstale":"2024-09-21T10:12:40.686876+0000","last_undegraded":"2024-09-21T10:12:40.686876+0000","last_fullsized":"2024-09-21T10:12:40.686876+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:53:23.247341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.a","version":"280'11766","reported_seq":11845,"reported_epoch":280,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:12:51.615985+0000","last_change":"2024-09-21T10:10:58.704699+0000","last_active":"2024-09-21T10:12:51.615985+0000","last_peered":"2024-09-21T10:12:51.615985+0000","last_clean":"2024-09-21T10:10:56.390149+0000","last_became_active":"2024-09-21T10:10:58.682906+0000","last_became_peered":"2024-09-21T10:10:58.682906+0000","last_unstale":"2024-09-21T10:12:51.615985+0000","last_undegraded":"2024-09-21T10:12:51.615985+0000","last_fullsized":"2024-09-21T10:12:51.615985+0000","mapping_epoch":267,"log_start":"268'5900","ondisk_log_start":"268'5900","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5866,"log_dups_size":0,"ondisk_log_size":5866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141033472,"num_objects":2152,"num_object_clones":0,"num_object_copies":6456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2152,"num_objects_unfound":0,"num_objects_dirty":2152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11766,"num_write_kb":445376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,1],"avail_no_missing":["3","1","6"],"object_location_counts":[{"shards":"1,3,6","objects":2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":171,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.479940+0000","last_change":"2024-09-21T10:09:49.924286+0000","last_active":"2024-09-21T10:12:35.479940+0000","last_peered":"2024-09-21T10:12:35.479940+0000","last_clean":"2024-09-21T10:12:35.479940+0000","last_became_active":"2024-09-21T10:09:49.923897+0000","last_became_peered":"2024-09-21T10:09:49.923897+0000","last_unstale":"2024-09-21T10:12:35.479940+0000","last_undegraded":"2024-09-21T10:12:35.479940+0000","last_fullsized":"2024-09-21T10:12:35.479940+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:47.393146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:47.393146+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:44.898083+0000","scrub_duration":0.001601086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":172,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-21T10:12:53.713493+0000","last_change":"2024-09-21T10:10:57.402946+0000","last_active":"2024-09-21T10:12:53.713493+0000","last_peered":"2024-09-21T10:12:53.713493+0000","last_clean":"2024-09-21T10:12:53.713493+0000","last_became_active":"2024-09-21T10:10:57.395450+0000","last_became_peered":"2024-09-21T10:10:57.395450+0000","last_unstale":"2024-09-21T10:12:53.713493+0000","last_undegraded":"2024-09-21T10:12:53.713493+0000","last_fullsized":"2024-09-21T10:12:53.713493+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:08.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":400,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.685050+0000","last_change":"2024-09-21T10:10:57.390595+0000","last_active":"2024-09-21T10:12:40.685050+0000","last_peered":"2024-09-21T10:12:40.685050+0000","last_clean":"2024-09-21T10:12:40.685050+0000","last_became_active":"2024-09-21T10:10:57.390265+0000","last_became_peered":"2024-09-21T10:10:57.390265+0000","last_unstale":"2024-09-21T10:12:40.685050+0000","last_undegraded":"2024-09-21T10:12:40.685050+0000","last_fullsized":"2024-09-21T10:12:40.685050+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:37.485063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":287,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.483212+0000","last_change":"2024-09-21T10:09:24.837651+0000","last_active":"2024-09-21T10:12:35.483212+0000","last_peered":"2024-09-21T10:12:35.483212+0000","last_clean":"2024-09-21T10:12:35.483212+0000","last_became_active":"2024-09-21T10:09:24.837226+0000","last_became_peered":"2024-09-21T10:09:24.837226+0000","last_unstale":"2024-09-21T10:12:35.483212+0000","last_undegraded":"2024-09-21T10:12:35.483212+0000","last_fullsized":"2024-09-21T10:12:35.483212+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:16.649601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"280'11781","reported_seq":11855,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:51.646175+0000","last_change":"2024-09-21T10:09:49.926338+0000","last_active":"2024-09-21T10:12:51.646175+0000","last_peered":"2024-09-21T10:12:51.646175+0000","last_clean":"2024-09-21T10:12:51.646175+0000","last_became_active":"2024-09-21T10:09:49.926018+0000","last_became_peered":"2024-09-21T10:09:49.926018+0000","last_unstale":"2024-09-21T10:12:51.646175+0000","last_undegraded":"2024-09-21T10:12:51.646175+0000","last_fullsized":"2024-09-21T10:12:51.646175+0000","mapping_epoch":255,"log_start":"268'6000","ondisk_log_start":"268'6000","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5781,"log_dups_size":0,"ondisk_log_size":5781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:09:49.364660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142934016,"num_objects":2181,"num_object_clones":0,"num_object_copies":6543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11781,"num_write_kb":446784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":20,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.482796+0000","last_change":"2024-09-21T10:11:05.443167+0000","last_active":"2024-09-21T10:12:35.482796+0000","last_peered":"2024-09-21T10:12:35.482796+0000","last_clean":"2024-09-21T10:12:35.482796+0000","last_became_active":"2024-09-21T10:11:05.442805+0000","last_became_peered":"2024-09-21T10:11:05.442805+0000","last_unstale":"2024-09-21T10:12:35.482796+0000","last_undegraded":"2024-09-21T10:12:35.482796+0000","last_fullsized":"2024-09-21T10:12:35.482796+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.550551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.550551+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:58:57.998543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":145,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.479991+0000","last_change":"2024-09-21T10:09:23.838944+0000","last_active":"2024-09-21T10:12:35.479991+0000","last_peered":"2024-09-21T10:12:35.479991+0000","last_clean":"2024-09-21T10:12:35.479991+0000","last_became_active":"2024-09-21T10:09:23.837886+0000","last_became_peered":"2024-09-21T10:09:23.837886+0000","last_unstale":"2024-09-21T10:12:35.479991+0000","last_undegraded":"2024-09-21T10:12:35.479991+0000","last_fullsized":"2024-09-21T10:12:35.479991+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:16.928195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":292,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.694623+0000","last_change":"2024-09-21T10:09:24.839770+0000","last_active":"2024-09-21T10:12:40.694623+0000","last_peered":"2024-09-21T10:12:40.694623+0000","last_clean":"2024-09-21T10:12:40.694623+0000","last_became_active":"2024-09-21T10:09:24.839479+0000","last_became_peered":"2024-09-21T10:09:24.839479+0000","last_unstale":"2024-09-21T10:12:40.694623+0000","last_undegraded":"2024-09-21T10:12:40.694623+0000","last_fullsized":"2024-09-21T10:12:40.694623+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:11:19.618193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":61,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.683501+0000","last_change":"2024-09-21T10:09:49.923044+0000","last_active":"2024-09-21T10:12:40.683501+0000","last_peered":"2024-09-21T10:12:40.683501+0000","last_clean":"2024-09-21T10:12:40.683501+0000","last_became_active":"2024-09-21T10:09:49.922711+0000","last_became_peered":"2024-09-21T10:09:49.922711+0000","last_unstale":"2024-09-21T10:12:40.683501+0000","last_undegraded":"2024-09-21T10:12:40.683501+0000","last_fullsized":"2024-09-21T10:12:40.683501+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:14.090084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"280'12053","reported_seq":17183,"reported_epoch":280,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T10:12:51.643382+0000","last_change":"2024-09-21T10:12:33.823280+0000","last_active":"2024-09-21T10:12:51.643382+0000","last_peered":"2024-09-21T10:12:51.643382+0000","last_clean":"2024-09-21T10:10:56.379248+0000","last_became_active":"2024-09-21T10:10:58.672712+0000","last_became_peered":"2024-09-21T10:10:58.672712+0000","last_unstale":"2024-09-21T10:12:51.643382+0000","last_undegraded":"2024-09-21T10:12:51.643382+0000","last_fullsized":"2024-09-21T10:12:51.643382+0000","mapping_epoch":267,"log_start":"270'6200","ondisk_log_start":"270'6200","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5853,"log_dups_size":0,"ondisk_log_size":5853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144637952,"num_objects":2207,"num_object_clones":0,"num_object_copies":6621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":177,"num_objects_unfound":0,"num_objects_dirty":2207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12053,"num_write_kb":456320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2523,"num_bytes_recovered":165347328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,0],"avail_no_missing":["3","0","6"],"object_location_counts":[{"shards":"0,3,6","objects":2207}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":185,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.480134+0000","last_change":"2024-09-21T10:09:24.838434+0000","last_active":"2024-09-21T10:12:35.480134+0000","last_peered":"2024-09-21T10:12:35.480134+0000","last_clean":"2024-09-21T10:12:35.480134+0000","last_became_active":"2024-09-21T10:09:24.838066+0000","last_became_peered":"2024-09-21T10:09:24.838066+0000","last_unstale":"2024-09-21T10:12:35.480134+0000","last_undegraded":"2024-09-21T10:12:35.480134+0000","last_fullsized":"2024-09-21T10:12:35.480134+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:52.361137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:52.361137+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:19:57.788195+0000","scrub_duration":0.001255358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":455,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-21T10:12:53.712565+0000","last_change":"2024-09-21T10:09:23.837859+0000","last_active":"2024-09-21T10:12:53.712565+0000","last_peered":"2024-09-21T10:12:53.712565+0000","last_clean":"2024-09-21T10:12:53.712565+0000","last_became_active":"2024-09-21T10:09:23.837488+0000","last_became_peered":"2024-09-21T10:09:23.837488+0000","last_unstale":"2024-09-21T10:12:53.712565+0000","last_undegraded":"2024-09-21T10:12:53.712565+0000","last_fullsized":"2024-09-21T10:12:53.712565+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:25.215501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.479913+0000","last_change":"2024-09-21T10:10:27.806070+0000","last_active":"2024-09-21T10:12:35.479913+0000","last_peered":"2024-09-21T10:12:35.479913+0000","last_clean":"2024-09-21T10:12:35.479913+0000","last_became_active":"2024-09-21T10:10:27.805718+0000","last_became_peered":"2024-09-21T10:10:27.805718+0000","last_unstale":"2024-09-21T10:12:35.479913+0000","last_undegraded":"2024-09-21T10:12:35.479913+0000","last_fullsized":"2024-09-21T10:12:35.479913+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:31.144118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":448,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-21T10:12:53.713277+0000","last_change":"2024-09-21T10:10:57.395073+0000","last_active":"2024-09-21T10:12:53.713277+0000","last_peered":"2024-09-21T10:12:53.713277+0000","last_clean":"2024-09-21T10:12:53.713277+0000","last_became_active":"2024-09-21T10:10:57.394674+0000","last_became_peered":"2024-09-21T10:10:57.394674+0000","last_unstale":"2024-09-21T10:12:53.713277+0000","last_undegraded":"2024-09-21T10:12:53.713277+0000","last_fullsized":"2024-09-21T10:12:53.713277+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:48:48.912649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"280'12005","reported_seq":12072,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:51.634302+0000","last_change":"2024-09-21T10:09:43.231554+0000","last_active":"2024-09-21T10:12:51.634302+0000","last_peered":"2024-09-21T10:12:51.634302+0000","last_clean":"2024-09-21T10:12:51.634302+0000","last_became_active":"2024-09-21T10:09:43.231211+0000","last_became_peered":"2024-09-21T10:09:43.231211+0000","last_unstale":"2024-09-21T10:12:51.634302+0000","last_undegraded":"2024-09-21T10:12:51.634302+0000","last_fullsized":"2024-09-21T10:12:51.634302+0000","mapping_epoch":252,"log_start":"270'6200","ondisk_log_start":"270'6200","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5805,"log_dups_size":0,"ondisk_log_size":5805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:16:41.457768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150142976,"num_objects":2291,"num_object_clones":0,"num_object_copies":6873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12005,"num_write_kb":457472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":316,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.694719+0000","last_change":"2024-09-21T10:10:57.398437+0000","last_active":"2024-09-21T10:12:40.694719+0000","last_peered":"2024-09-21T10:12:40.694719+0000","last_clean":"2024-09-21T10:12:40.694719+0000","last_became_active":"2024-09-21T10:10:57.397441+0000","last_became_peered":"2024-09-21T10:10:57.397441+0000","last_unstale":"2024-09-21T10:12:40.694719+0000","last_undegraded":"2024-09-21T10:12:40.694719+0000","last_fullsized":"2024-09-21T10:12:40.694719+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:04.043917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:04.043917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:53:31.813612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":84,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.483161+0000","last_change":"2024-09-21T10:08:26.309323+0000","last_active":"2024-09-21T10:12:35.483161+0000","last_peered":"2024-09-21T10:12:35.483161+0000","last_clean":"2024-09-21T10:12:35.483161+0000","last_became_active":"2024-09-21T10:08:26.309004+0000","last_became_peered":"2024-09-21T10:08:26.309004+0000","last_unstale":"2024-09-21T10:12:35.483161+0000","last_undegraded":"2024-09-21T10:12:35.483161+0000","last_fullsized":"2024-09-21T10:12:35.483161+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:37:20.626702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":126,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.683893+0000","last_change":"2024-09-21T10:10:28.180093+0000","last_active":"2024-09-21T10:12:40.683893+0000","last_peered":"2024-09-21T10:12:40.683893+0000","last_clean":"2024-09-21T10:12:40.683893+0000","last_became_active":"2024-09-21T10:10:28.179559+0000","last_became_peered":"2024-09-21T10:10:28.179559+0000","last_unstale":"2024-09-21T10:12:40.683893+0000","last_undegraded":"2024-09-21T10:12:40.683893+0000","last_fullsized":"2024-09-21T10:12:40.683893+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:52:46.146392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":225,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-21T10:12:53.715768+0000","last_change":"2024-09-21T10:08:26.307041+0000","last_active":"2024-09-21T10:12:53.715768+0000","last_peered":"2024-09-21T10:12:53.715768+0000","last_clean":"2024-09-21T10:12:53.715768+0000","last_became_active":"2024-09-21T10:08:26.306715+0000","last_became_peered":"2024-09-21T10:08:26.306715+0000","last_unstale":"2024-09-21T10:12:53.715768+0000","last_undegraded":"2024-09-21T10:12:53.715768+0000","last_fullsized":"2024-09-21T10:12:53.715768+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:19:26.192398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"281'12146","reported_seq":12217,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-21T10:12:53.948326+0000","last_change":"2024-09-21T10:09:43.232893+0000","last_active":"2024-09-21T10:12:53.948326+0000","last_peered":"2024-09-21T10:12:53.948326+0000","last_clean":"2024-09-21T10:12:53.948326+0000","last_became_active":"2024-09-21T10:09:43.232461+0000","last_became_peered":"2024-09-21T10:09:43.232461+0000","last_unstale":"2024-09-21T10:12:53.948326+0000","last_undegraded":"2024-09-21T10:12:53.948326+0000","last_fullsized":"2024-09-21T10:12:53.948326+0000","mapping_epoch":252,"log_start":"263'4600","ondisk_log_start":"263'4600","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":7546,"log_dups_size":0,"ondisk_log_size":7546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:39.778833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136904728,"num_objects":2090,"num_object_clones":0,"num_object_copies":6270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2090,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12146,"num_write_kb":455489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":254,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.482964+0000","last_change":"2024-09-21T10:04:45.242148+0000","last_active":"2024-09-21T10:12:35.482964+0000","last_peered":"2024-09-21T10:12:35.482964+0000","last_clean":"2024-09-21T10:12:35.482964+0000","last_became_active":"2024-09-21T10:04:42.549844+0000","last_became_peered":"2024-09-21T10:04:42.549844+0000","last_unstale":"2024-09-21T10:12:35.482964+0000","last_undegraded":"2024-09-21T10:12:35.482964+0000","last_fullsized":"2024-09-21T10:12:35.482964+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_deep_scrub":"77'100","last_deep_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.242073+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:03:42.250480+0000","scrub_duration":0.0023466239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":23,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.482373+0000","last_change":"2024-09-21T10:10:57.392880+0000","last_active":"2024-09-21T10:12:35.482373+0000","last_peered":"2024-09-21T10:12:35.482373+0000","last_clean":"2024-09-21T10:12:35.482373+0000","last_became_active":"2024-09-21T10:10:57.392539+0000","last_became_peered":"2024-09-21T10:10:57.392539+0000","last_unstale":"2024-09-21T10:12:35.482373+0000","last_undegraded":"2024-09-21T10:12:35.482373+0000","last_fullsized":"2024-09-21T10:12:35.482373+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:49.754955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":336,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.683508+0000","last_change":"2024-09-21T10:09:23.865342+0000","last_active":"2024-09-21T10:12:40.683508+0000","last_peered":"2024-09-21T10:12:40.683508+0000","last_clean":"2024-09-21T10:12:40.683508+0000","last_became_active":"2024-09-21T10:09:23.865008+0000","last_became_peered":"2024-09-21T10:09:23.865008+0000","last_unstale":"2024-09-21T10:12:40.683508+0000","last_undegraded":"2024-09-21T10:12:40.683508+0000","last_fullsized":"2024-09-21T10:12:40.683508+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:03.356101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.481892+0000","last_change":"2024-09-21T10:10:57.394707+0000","last_active":"2024-09-21T10:12:35.481892+0000","last_peered":"2024-09-21T10:12:35.481892+0000","last_clean":"2024-09-21T10:12:35.481892+0000","last_became_active":"2024-09-21T10:10:57.394363+0000","last_became_peered":"2024-09-21T10:10:57.394363+0000","last_unstale":"2024-09-21T10:12:35.481892+0000","last_undegraded":"2024-09-21T10:12:35.481892+0000","last_fullsized":"2024-09-21T10:12:35.481892+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:21.141052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"281'12036","reported_seq":12106,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-21T10:12:53.996302+0000","last_change":"2024-09-21T10:09:43.232987+0000","last_active":"2024-09-21T10:12:53.996302+0000","last_peered":"2024-09-21T10:12:53.996302+0000","last_clean":"2024-09-21T10:12:53.996302+0000","last_became_active":"2024-09-21T10:09:43.232577+0000","last_became_peered":"2024-09-21T10:09:43.232577+0000","last_unstale":"2024-09-21T10:12:53.996302+0000","last_undegraded":"2024-09-21T10:12:53.996302+0000","last_fullsized":"2024-09-21T10:12:53.996302+0000","mapping_epoch":252,"log_start":"263'4500","ondisk_log_start":"263'4500","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":7536,"log_dups_size":0,"ondisk_log_size":7536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:14.536774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138018816,"num_objects":2106,"num_object_clones":0,"num_object_copies":6318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12036,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":271,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.483074+0000","last_change":"2024-09-21T10:09:23.839563+0000","last_active":"2024-09-21T10:12:35.483074+0000","last_peered":"2024-09-21T10:12:35.483074+0000","last_clean":"2024-09-21T10:12:35.483074+0000","last_became_active":"2024-09-21T10:09:23.839207+0000","last_became_peered":"2024-09-21T10:09:23.839207+0000","last_unstale":"2024-09-21T10:12:35.483074+0000","last_undegraded":"2024-09-21T10:12:35.483074+0000","last_fullsized":"2024-09-21T10:12:35.483074+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:49.226496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:49.226496+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:03.960125+0000","scrub_duration":0.001656865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":190,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-21T10:12:53.715688+0000","last_change":"2024-09-21T10:10:57.395167+0000","last_active":"2024-09-21T10:12:53.715688+0000","last_peered":"2024-09-21T10:12:53.715688+0000","last_clean":"2024-09-21T10:12:53.715688+0000","last_became_active":"2024-09-21T10:10:57.394818+0000","last_became_peered":"2024-09-21T10:10:57.394818+0000","last_unstale":"2024-09-21T10:12:53.715688+0000","last_undegraded":"2024-09-21T10:12:53.715688+0000","last_fullsized":"2024-09-21T10:12:53.715688+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:57.224113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":165,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.480507+0000","last_change":"2024-09-21T10:09:49.922782+0000","last_active":"2024-09-21T10:12:35.480507+0000","last_peered":"2024-09-21T10:12:35.480507+0000","last_clean":"2024-09-21T10:12:35.480507+0000","last_became_active":"2024-09-21T10:09:49.922465+0000","last_became_peered":"2024-09-21T10:09:49.922465+0000","last_unstale":"2024-09-21T10:12:35.480507+0000","last_undegraded":"2024-09-21T10:12:35.480507+0000","last_fullsized":"2024-09-21T10:12:35.480507+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:06.151245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":330,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.683976+0000","last_change":"2024-09-21T10:10:57.397251+0000","last_active":"2024-09-21T10:12:40.683976+0000","last_peered":"2024-09-21T10:12:40.683976+0000","last_clean":"2024-09-21T10:12:40.683976+0000","last_became_active":"2024-09-21T10:10:57.396271+0000","last_became_peered":"2024-09-21T10:10:57.396271+0000","last_unstale":"2024-09-21T10:12:40.683976+0000","last_undegraded":"2024-09-21T10:12:40.683976+0000","last_fullsized":"2024-09-21T10:12:40.683976+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:17.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.f","version":"280'12087","reported_seq":12123,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:51.787702+0000","last_change":"2024-09-21T10:09:43.232625+0000","last_active":"2024-09-21T10:12:51.787702+0000","last_peered":"2024-09-21T10:12:51.787702+0000","last_clean":"2024-09-21T10:12:51.787702+0000","last_became_active":"2024-09-21T10:09:43.232331+0000","last_became_peered":"2024-09-21T10:09:43.232331+0000","last_unstale":"2024-09-21T10:12:51.787702+0000","last_undegraded":"2024-09-21T10:12:51.787702+0000","last_fullsized":"2024-09-21T10:12:51.787702+0000","mapping_epoch":252,"log_start":"265'5800","ondisk_log_start":"265'5800","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6287,"log_dups_size":0,"ondisk_log_size":6287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:01:16.634826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143589376,"num_objects":2191,"num_object_clones":0,"num_object_copies":6573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12087,"num_write_kb":456896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":136,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-21T10:12:53.714995+0000","last_change":"2024-09-21T10:11:05.455170+0000","last_active":"2024-09-21T10:12:53.714995+0000","last_peered":"2024-09-21T10:12:53.714995+0000","last_clean":"2024-09-21T10:12:53.714995+0000","last_became_active":"2024-09-21T10:11:05.454607+0000","last_became_peered":"2024-09-21T10:11:05.454607+0000","last_unstale":"2024-09-21T10:12:53.714995+0000","last_undegraded":"2024-09-21T10:12:53.714995+0000","last_fullsized":"2024-09-21T10:12:53.714995+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:59.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:59.099852+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:14.788975+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":421,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.686960+0000","last_change":"2024-09-21T10:09:23.841186+0000","last_active":"2024-09-21T10:12:40.686960+0000","last_peered":"2024-09-21T10:12:40.686960+0000","last_clean":"2024-09-21T10:12:40.686960+0000","last_became_active":"2024-09-21T10:09:23.840854+0000","last_became_peered":"2024-09-21T10:09:23.840854+0000","last_unstale":"2024-09-21T10:12:40.686960+0000","last_undegraded":"2024-09-21T10:12:40.686960+0000","last_fullsized":"2024-09-21T10:12:40.686960+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:29.666358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":102,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.483101+0000","last_change":"2024-09-21T10:08:38.380867+0000","last_active":"2024-09-21T10:12:35.483101+0000","last_peered":"2024-09-21T10:12:35.483101+0000","last_clean":"2024-09-21T10:12:35.483101+0000","last_became_active":"2024-09-21T10:08:38.379599+0000","last_became_peered":"2024-09-21T10:08:38.379599+0000","last_unstale":"2024-09-21T10:12:35.483101+0000","last_undegraded":"2024-09-21T10:12:35.483101+0000","last_fullsized":"2024-09-21T10:12:35.483101+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:50.499470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":418,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.687070+0000","last_change":"2024-09-21T10:09:49.926548+0000","last_active":"2024-09-21T10:12:40.687070+0000","last_peered":"2024-09-21T10:12:40.687070+0000","last_clean":"2024-09-21T10:12:40.687070+0000","last_became_active":"2024-09-21T10:09:49.926255+0000","last_became_peered":"2024-09-21T10:09:49.926255+0000","last_unstale":"2024-09-21T10:12:40.687070+0000","last_undegraded":"2024-09-21T10:12:40.687070+0000","last_fullsized":"2024-09-21T10:12:40.687070+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:49.855878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"280'12037","reported_seq":12073,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:51.842552+0000","last_change":"2024-09-21T10:09:43.231158+0000","last_active":"2024-09-21T10:12:51.842552+0000","last_peered":"2024-09-21T10:12:51.842552+0000","last_clean":"2024-09-21T10:12:51.842552+0000","last_became_active":"2024-09-21T10:09:43.230838+0000","last_became_peered":"2024-09-21T10:09:43.230838+0000","last_unstale":"2024-09-21T10:12:51.842552+0000","last_undegraded":"2024-09-21T10:12:51.842552+0000","last_fullsized":"2024-09-21T10:12:51.842552+0000","mapping_epoch":252,"log_start":"265'5700","ondisk_log_start":"265'5700","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6337,"log_dups_size":0,"ondisk_log_size":6337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:15.318633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143720448,"num_objects":2193,"num_object_clones":0,"num_object_copies":6579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12037,"num_write_kb":455360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":219,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.685307+0000","last_change":"2024-09-21T10:10:57.390804+0000","last_active":"2024-09-21T10:12:40.685307+0000","last_peered":"2024-09-21T10:12:40.685307+0000","last_clean":"2024-09-21T10:12:40.685307+0000","last_became_active":"2024-09-21T10:10:57.390323+0000","last_became_peered":"2024-09-21T10:10:57.390323+0000","last_unstale":"2024-09-21T10:12:40.685307+0000","last_undegraded":"2024-09-21T10:12:40.685307+0000","last_fullsized":"2024-09-21T10:12:40.685307+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.005477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.005477+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:29:57.702046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":65,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.683683+0000","last_change":"2024-09-21T10:09:49.930280+0000","last_active":"2024-09-21T10:12:40.683683+0000","last_peered":"2024-09-21T10:12:40.683683+0000","last_clean":"2024-09-21T10:12:40.683683+0000","last_became_active":"2024-09-21T10:09:49.919101+0000","last_became_peered":"2024-09-21T10:09:49.919101+0000","last_unstale":"2024-09-21T10:12:40.683683+0000","last_undegraded":"2024-09-21T10:12:40.683683+0000","last_fullsized":"2024-09-21T10:12:40.683683+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:23.998475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":293,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.483239+0000","last_change":"2024-09-21T10:03:38.392767+0000","last_active":"2024-09-21T10:12:35.483239+0000","last_peered":"2024-09-21T10:12:35.483239+0000","last_clean":"2024-09-21T10:12:35.483239+0000","last_became_active":"2024-09-21T10:03:38.392424+0000","last_became_peered":"2024-09-21T10:03:38.392424+0000","last_unstale":"2024-09-21T10:12:35.483239+0000","last_undegraded":"2024-09-21T10:12:35.483239+0000","last_fullsized":"2024-09-21T10:12:35.483239+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:25.358291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":74,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.481957+0000","last_change":"2024-09-21T10:09:24.836360+0000","last_active":"2024-09-21T10:12:35.481957+0000","last_peered":"2024-09-21T10:12:35.481957+0000","last_clean":"2024-09-21T10:12:35.481957+0000","last_became_active":"2024-09-21T10:09:24.836033+0000","last_became_peered":"2024-09-21T10:09:24.836033+0000","last_unstale":"2024-09-21T10:12:35.481957+0000","last_undegraded":"2024-09-21T10:12:35.481957+0000","last_fullsized":"2024-09-21T10:12:35.481957+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:58.368582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":131,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.483040+0000","last_change":"2024-09-21T10:08:38.380189+0000","last_active":"2024-09-21T10:12:35.483040+0000","last_peered":"2024-09-21T10:12:35.483040+0000","last_clean":"2024-09-21T10:12:35.483040+0000","last_became_active":"2024-09-21T10:08:38.379891+0000","last_became_peered":"2024-09-21T10:08:38.379891+0000","last_unstale":"2024-09-21T10:12:35.483040+0000","last_undegraded":"2024-09-21T10:12:35.483040+0000","last_fullsized":"2024-09-21T10:12:35.483040+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:35.820921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":211,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.480834+0000","last_change":"2024-09-21T10:08:38.392563+0000","last_active":"2024-09-21T10:12:35.480834+0000","last_peered":"2024-09-21T10:12:35.480834+0000","last_clean":"2024-09-21T10:12:35.480834+0000","last_became_active":"2024-09-21T10:08:38.378735+0000","last_became_peered":"2024-09-21T10:08:38.378735+0000","last_unstale":"2024-09-21T10:12:35.480834+0000","last_undegraded":"2024-09-21T10:12:35.480834+0000","last_fullsized":"2024-09-21T10:12:35.480834+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:29:41.423208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":20,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.482902+0000","last_change":"2024-09-21T10:11:05.453286+0000","last_active":"2024-09-21T10:12:35.482902+0000","last_peered":"2024-09-21T10:12:35.482902+0000","last_clean":"2024-09-21T10:12:35.482902+0000","last_became_active":"2024-09-21T10:11:05.452601+0000","last_became_peered":"2024-09-21T10:11:05.452601+0000","last_unstale":"2024-09-21T10:12:35.482902+0000","last_undegraded":"2024-09-21T10:12:35.482902+0000","last_fullsized":"2024-09-21T10:12:35.482902+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:27.532091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":59,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-21T10:12:53.715294+0000","last_change":"2024-09-21T10:10:27.833762+0000","last_active":"2024-09-21T10:12:53.715294+0000","last_peered":"2024-09-21T10:12:53.715294+0000","last_clean":"2024-09-21T10:12:53.715294+0000","last_became_active":"2024-09-21T10:10:27.819999+0000","last_became_peered":"2024-09-21T10:10:27.819999+0000","last_unstale":"2024-09-21T10:12:53.715294+0000","last_undegraded":"2024-09-21T10:12:53.715294+0000","last_fullsized":"2024-09-21T10:12:53.715294+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:33.839463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'100","reported_seq":214,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-21T10:12:53.715492+0000","last_change":"2024-09-21T10:07:24.345428+0000","last_active":"2024-09-21T10:12:53.715492+0000","last_peered":"2024-09-21T10:12:53.715492+0000","last_clean":"2024-09-21T10:12:53.715492+0000","last_became_active":"2024-09-21T10:07:24.343529+0000","last_became_peered":"2024-09-21T10:07:24.343529+0000","last_unstale":"2024-09-21T10:12:53.715492+0000","last_undegraded":"2024-09-21T10:12:53.715492+0000","last_fullsized":"2024-09-21T10:12:53.715492+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.014993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.014993+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:35.677888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":23,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.482482+0000","last_change":"2024-09-21T10:10:57.394885+0000","last_active":"2024-09-21T10:12:35.482482+0000","last_peered":"2024-09-21T10:12:35.482482+0000","last_clean":"2024-09-21T10:12:35.482482+0000","last_became_active":"2024-09-21T10:10:57.394584+0000","last_became_peered":"2024-09-21T10:10:57.394584+0000","last_unstale":"2024-09-21T10:12:35.482482+0000","last_undegraded":"2024-09-21T10:12:35.482482+0000","last_fullsized":"2024-09-21T10:12:35.482482+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:52:16.853713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":404,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.687306+0000","last_change":"2024-09-21T10:03:38.425111+0000","last_active":"2024-09-21T10:12:40.687306+0000","last_peered":"2024-09-21T10:12:40.687306+0000","last_clean":"2024-09-21T10:12:40.687306+0000","last_became_active":"2024-09-21T10:03:38.424704+0000","last_became_peered":"2024-09-21T10:03:38.424704+0000","last_unstale":"2024-09-21T10:12:40.687306+0000","last_undegraded":"2024-09-21T10:12:40.687306+0000","last_fullsized":"2024-09-21T10:12:40.687306+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:35.157076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":305,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.685234+0000","last_change":"2024-09-21T10:10:57.397755+0000","last_active":"2024-09-21T10:12:40.685234+0000","last_peered":"2024-09-21T10:12:40.685234+0000","last_clean":"2024-09-21T10:12:40.685234+0000","last_became_active":"2024-09-21T10:10:57.397249+0000","last_became_peered":"2024-09-21T10:10:57.397249+0000","last_unstale":"2024-09-21T10:12:40.685234+0000","last_undegraded":"2024-09-21T10:12:40.685234+0000","last_fullsized":"2024-09-21T10:12:40.685234+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:40.813503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":389,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.687046+0000","last_change":"2024-09-21T09:56:12.576756+0000","last_active":"2024-09-21T10:12:40.687046+0000","last_peered":"2024-09-21T10:12:40.687046+0000","last_clean":"2024-09-21T10:12:40.687046+0000","last_became_active":"2024-09-21T09:56:12.575752+0000","last_became_peered":"2024-09-21T09:56:12.575752+0000","last_unstale":"2024-09-21T10:12:40.687046+0000","last_undegraded":"2024-09-21T10:12:40.687046+0000","last_fullsized":"2024-09-21T10:12:40.687046+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:55.224839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":243,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.483171+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T10:12:35.483171+0000","last_peered":"2024-09-21T10:12:35.483171+0000","last_clean":"2024-09-21T10:12:35.483171+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T10:12:35.483171+0000","last_undegraded":"2024-09-21T10:12:35.483171+0000","last_fullsized":"2024-09-21T10:12:35.483171+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":424,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.687137+0000","last_change":"2024-09-21T10:09:49.928053+0000","last_active":"2024-09-21T10:12:40.687137+0000","last_peered":"2024-09-21T10:12:40.687137+0000","last_clean":"2024-09-21T10:12:40.687137+0000","last_became_active":"2024-09-21T10:09:49.927231+0000","last_became_peered":"2024-09-21T10:09:49.927231+0000","last_unstale":"2024-09-21T10:12:40.687137+0000","last_undegraded":"2024-09-21T10:12:40.687137+0000","last_fullsized":"2024-09-21T10:12:40.687137+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:26.987553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":337,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.684053+0000","last_change":"2024-09-21T10:09:23.840717+0000","last_active":"2024-09-21T10:12:40.684053+0000","last_peered":"2024-09-21T10:12:40.684053+0000","last_clean":"2024-09-21T10:12:40.684053+0000","last_became_active":"2024-09-21T10:09:23.840323+0000","last_became_peered":"2024-09-21T10:09:23.840323+0000","last_unstale":"2024-09-21T10:12:40.684053+0000","last_undegraded":"2024-09-21T10:12:40.684053+0000","last_fullsized":"2024-09-21T10:12:40.684053+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:55.769770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":53,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-21T10:12:53.712337+0000","last_change":"2024-09-21T10:10:27.816526+0000","last_active":"2024-09-21T10:12:53.712337+0000","last_peered":"2024-09-21T10:12:53.712337+0000","last_clean":"2024-09-21T10:12:53.712337+0000","last_became_active":"2024-09-21T10:10:27.816221+0000","last_became_peered":"2024-09-21T10:10:27.816221+0000","last_unstale":"2024-09-21T10:12:53.712337+0000","last_undegraded":"2024-09-21T10:12:53.712337+0000","last_fullsized":"2024-09-21T10:12:53.712337+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:08.331228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":395,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.687135+0000","last_change":"2024-09-21T10:08:38.379994+0000","last_active":"2024-09-21T10:12:40.687135+0000","last_peered":"2024-09-21T10:12:40.687135+0000","last_clean":"2024-09-21T10:12:40.687135+0000","last_became_active":"2024-09-21T10:08:38.378729+0000","last_became_peered":"2024-09-21T10:08:38.378729+0000","last_unstale":"2024-09-21T10:12:40.687135+0000","last_undegraded":"2024-09-21T10:12:40.687135+0000","last_fullsized":"2024-09-21T10:12:40.687135+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:16.940173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":117,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-21T10:12:53.713037+0000","last_change":"2024-09-21T10:09:24.844561+0000","last_active":"2024-09-21T10:12:53.713037+0000","last_peered":"2024-09-21T10:12:53.713037+0000","last_clean":"2024-09-21T10:12:53.713037+0000","last_became_active":"2024-09-21T10:09:24.844190+0000","last_became_peered":"2024-09-21T10:09:24.844190+0000","last_unstale":"2024-09-21T10:12:53.713037+0000","last_undegraded":"2024-09-21T10:12:53.713037+0000","last_fullsized":"2024-09-21T10:12:53.713037+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:46:04.775344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":444,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-21T10:12:53.715719+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:12:53.715719+0000","last_peered":"2024-09-21T10:12:53.715719+0000","last_clean":"2024-09-21T10:12:53.715719+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:12:53.715719+0000","last_undegraded":"2024-09-21T10:12:53.715719+0000","last_fullsized":"2024-09-21T10:12:53.715719+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":61,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.683596+0000","last_change":"2024-09-21T10:09:49.918246+0000","last_active":"2024-09-21T10:12:40.683596+0000","last_peered":"2024-09-21T10:12:40.683596+0000","last_clean":"2024-09-21T10:12:40.683596+0000","last_became_active":"2024-09-21T10:09:49.917911+0000","last_became_peered":"2024-09-21T10:09:49.917911+0000","last_unstale":"2024-09-21T10:12:40.683596+0000","last_undegraded":"2024-09-21T10:12:40.683596+0000","last_fullsized":"2024-09-21T10:12:40.683596+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:30:10.567650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":251,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.684358+0000","last_change":"2024-09-21T10:07:24.347937+0000","last_active":"2024-09-21T10:12:40.684358+0000","last_peered":"2024-09-21T10:12:40.684358+0000","last_clean":"2024-09-21T10:12:40.684358+0000","last_became_active":"2024-09-21T10:07:24.347376+0000","last_became_peered":"2024-09-21T10:07:24.347376+0000","last_unstale":"2024-09-21T10:12:40.684358+0000","last_undegraded":"2024-09-21T10:12:40.684358+0000","last_fullsized":"2024-09-21T10:12:40.684358+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:48:05.644824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":62,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.483346+0000","last_change":"2024-09-21T10:08:38.382518+0000","last_active":"2024-09-21T10:12:35.483346+0000","last_peered":"2024-09-21T10:12:35.483346+0000","last_clean":"2024-09-21T10:12:35.483346+0000","last_became_active":"2024-09-21T10:08:38.372051+0000","last_became_peered":"2024-09-21T10:08:38.372051+0000","last_unstale":"2024-09-21T10:12:35.483346+0000","last_undegraded":"2024-09-21T10:12:35.483346+0000","last_fullsized":"2024-09-21T10:12:35.483346+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:21.360435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":342,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.685138+0000","last_change":"2024-09-21T10:09:24.840173+0000","last_active":"2024-09-21T10:12:40.685138+0000","last_peered":"2024-09-21T10:12:40.685138+0000","last_clean":"2024-09-21T10:12:40.685138+0000","last_became_active":"2024-09-21T10:09:24.839762+0000","last_became_peered":"2024-09-21T10:09:24.839762+0000","last_unstale":"2024-09-21T10:12:40.685138+0000","last_undegraded":"2024-09-21T10:12:40.685138+0000","last_fullsized":"2024-09-21T10:12:40.685138+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:12:53.465203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":76,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.483265+0000","last_change":"2024-09-21T10:08:38.388561+0000","last_active":"2024-09-21T10:12:35.483265+0000","last_peered":"2024-09-21T10:12:35.483265+0000","last_clean":"2024-09-21T10:12:35.483265+0000","last_became_active":"2024-09-21T10:08:38.374558+0000","last_became_peered":"2024-09-21T10:08:38.374558+0000","last_unstale":"2024-09-21T10:12:35.483265+0000","last_undegraded":"2024-09-21T10:12:35.483265+0000","last_fullsized":"2024-09-21T10:12:35.483265+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:24.111438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":425,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.687811+0000","last_change":"2024-09-21T10:09:23.840467+0000","last_active":"2024-09-21T10:12:40.687811+0000","last_peered":"2024-09-21T10:12:40.687811+0000","last_clean":"2024-09-21T10:12:40.687811+0000","last_became_active":"2024-09-21T10:09:23.840177+0000","last_became_peered":"2024-09-21T10:09:23.840177+0000","last_unstale":"2024-09-21T10:12:40.687811+0000","last_undegraded":"2024-09-21T10:12:40.687811+0000","last_fullsized":"2024-09-21T10:12:40.687811+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:17.887307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":415,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.684556+0000","last_change":"2024-09-21T10:11:05.451613+0000","last_active":"2024-09-21T10:12:40.684556+0000","last_peered":"2024-09-21T10:12:40.684556+0000","last_clean":"2024-09-21T10:12:40.684556+0000","last_became_active":"2024-09-21T10:11:05.451075+0000","last_became_peered":"2024-09-21T10:11:05.451075+0000","last_unstale":"2024-09-21T10:12:40.684556+0000","last_undegraded":"2024-09-21T10:12:40.684556+0000","last_fullsized":"2024-09-21T10:12:40.684556+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:18.505045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":112,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-21T10:12:35.482624+0000","last_change":"2024-09-21T10:08:38.380508+0000","last_active":"2024-09-21T10:12:35.482624+0000","last_peered":"2024-09-21T10:12:35.482624+0000","last_clean":"2024-09-21T10:12:35.482624+0000","last_became_active":"2024-09-21T10:08:38.380226+0000","last_became_peered":"2024-09-21T10:08:38.380226+0000","last_unstale":"2024-09-21T10:12:35.482624+0000","last_undegraded":"2024-09-21T10:12:35.482624+0000","last_fullsized":"2024-09-21T10:12:35.482624+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:50:38.190118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":80,"reported_epoch":280,"state":"active+clean","last_fresh":"2024-09-21T10:12:40.684855+0000","last_change":"2024-09-21T10:10:57.401588+0000","last_active":"2024-09-21T10:12:40.684855+0000","last_peered":"2024-09-21T10:12:40.684855+0000","last_clean":"2024-09-21T10:12:40.684855+0000","last_became_active":"2024-09-21T10:10:57.397724+0000","last_became_peered":"2024-09-21T10:10:57.397724+0000","last_unstale":"2024-09-21T10:12:40.684855+0000","last_undegraded":"2024-09-21T10:12:40.684855+0000","last_fullsized":"2024-09-21T10:12:40.684855+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:47.843630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":2271871000,"num_objects":34667,"num_object_clones":0,"num_object_copies":104001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4472,"num_objects_unfound":0,"num_objects_dirty":34667,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190303,"num_write_kb":7198977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8019,"num_bytes_recovered":525533184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7015116800,"data_stored":7015104584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101903,"ondisk_log_size":101903,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":11710,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26580,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":688762,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":49,"num_read_kb":125,"num_write":25,"num_write_kb":271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":722165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626846,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2015108,"kb_used_data":1114748,"kb_used_omap":1,"kb_used_meta":900350,"kb_avail":91730044,"statfs":{"total":95995035648,"available":93931565056,"internally_reserved":0,"allocated":1141501952,"data_stored":1139909592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":921958744},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019304,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":245,"seq":1052266987563,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1317620,"kb_used_data":1123564,"kb_used_omap":8,"kb_used_meta":194039,"kb_avail":92427532,"statfs":{"total":95995035648,"available":94645792768,"internally_reserved":0,"allocated":1150529536,"data_stored":1148949379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8940,"internal_metadata":198696212},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":175,"seq":751619276912,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1491072,"kb_used_data":1148152,"kb_used_omap":12,"kb_used_meta":342899,"kb_avail":92254080,"statfs":{"total":95995035648,"available":94468177920,"internally_reserved":0,"allocated":1175707648,"data_stored":1174119363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12845,"internal_metadata":351129043},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823075,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1764720,"kb_used_data":271080,"kb_used_omap":1,"kb_used_meta":1493630,"kb_avail":91980432,"statfs":{"total":95995035648,"available":94187962368,"internally_reserved":0,"allocated":277585920,"data_stored":276009722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1529477464},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879498,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1824624,"kb_used_data":1011560,"kb_used_omap":4,"kb_used_meta":812923,"kb_avail":91920528,"statfs":{"total":95995035648,"available":94126620672,"internally_reserved":0,"allocated":1035837440,"data_stored":1034261323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":832433596},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566339,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1517724,"kb_used_data":807184,"kb_used_omap":6,"kb_used_meta":710521,"kb_avail":92227428,"statfs":{"total":95995035648,"available":94440886272,"internally_reserved":0,"allocated":826556416,"data_stored":824953236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":727573927},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860683,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2377392,"kb_used_data":1500392,"kb_used_omap":5,"kb_used_meta":876922,"kb_avail":91367760,"statfs":{"total":95995035648,"available":93560586240,"internally_reserved":0,"allocated":1536401408,"data_stored":1534824399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":897968367},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7236,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9976,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1141899264,"data_stored":1141899264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1503268864,"data_stored":1503264792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":822349824,"data_stored":822345752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1008599040,"data_stored":1008599040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":272695296,"data_stored":272695296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1113657344,"data_stored":1113653272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1152647168,"data_stored":1152647168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:13:01.069 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:01.077 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:01.092 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:01.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:01.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:00 smithi129 ceph-mon[88471]: pgmap v1007: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 713 op/s; 2921/93618 objects misplaced (3.120%); 12 MiB/s, 190 objects/s recovering 2024-09-21T10:13:01.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:01.252 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:01.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:01.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:01.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:01.401 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:01.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:01.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:01.502 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:01.532 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:01.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:01.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:01.634 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:01.661 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:01.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:01.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:01.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:13:01.762 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:01.788 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:01.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:01.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:01.803 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:13:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:13:01] "GET /metrics HTTP/1.1" 200 36807 "" "Prometheus/2.51.0" 2024-09-21T10:13:01.890 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:01.913 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:01.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:01.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:02.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:02.055 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:02.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:02.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:02.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:02.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:02.220 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:02.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:02.321 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:02.347 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:02.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:02.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:02.448 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:02.474 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:02.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:02.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:02.576 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:02.601 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:02.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:02.703 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:02.730 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:02.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:02.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:02.831 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:02.859 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:02.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:02.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:02.891 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:13:02.892 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:13:02.892 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-09-21T10:13:02.892 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-09-21T10:13:02.892 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:13:02.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:02.988 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:02.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:02.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:03.089 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:03.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:02 smithi107 ceph-mon[102221]: pgmap v1008: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 563 op/s; 2921/93618 objects misplaced (3.120%); 9.4 MiB/s, 150 objects/s recovering 2024-09-21T10:13:03.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:02 smithi107 ceph-mon[105022]: pgmap v1008: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 563 op/s; 2921/93618 objects misplaced (3.120%); 9.4 MiB/s, 150 objects/s recovering 2024-09-21T10:13:03.110 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:03.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:03.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:03.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:02 smithi129 ceph-mon[88471]: pgmap v1008: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 563 op/s; 2921/93618 objects misplaced (3.120%); 9.4 MiB/s, 150 objects/s recovering 2024-09-21T10:13:03.212 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:03.233 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:03.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:03.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:03.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:03.356 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:03.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:03.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:03.457 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:03.479 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:03.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:03.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:03.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:03.602 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:03.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:03.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:03.704 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:03.725 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:03.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:03.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:03.827 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:03.848 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:03.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:03.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:03.950 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:03.977 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:03.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:03.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:04.070 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:13:04.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:04.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:03 smithi107 ceph-mon[102221]: osdmap e284: 8 total, 8 up, 7 in 2024-09-21T10:13:04.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:03 smithi107 ceph-mon[105022]: osdmap e284: 8 total, 8 up, 7 in 2024-09-21T10:13:04.129 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:04.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:04.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:03 smithi129 ceph-mon[88471]: osdmap e284: 8 total, 8 up, 7 in 2024-09-21T10:13:04.231 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:04.258 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:04.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:04.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:04.360 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:04.386 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:04.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:04.487 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:04.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:04.513 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:04.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:04.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:04.650 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:04.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:04.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:04.752 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:04.777 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:04.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:04.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:04.878 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:04.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:04.911 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:04.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:05.012 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:05.034 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:05.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:05.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:05.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:04 smithi107 ceph-mon[105022]: pgmap v1010: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 1.7 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 870 op/s; 1714/83163 objects misplaced (2.061%); 16 MiB/s, 255 objects/s recovering 2024-09-21T10:13:05.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:04 smithi107 ceph-mon[105022]: osdmap e285: 8 total, 8 up, 7 in 2024-09-21T10:13:05.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:04 smithi107 ceph-mon[102221]: pgmap v1010: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 1.7 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 870 op/s; 1714/83163 objects misplaced (2.061%); 16 MiB/s, 255 objects/s recovering 2024-09-21T10:13:05.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:04 smithi107 ceph-mon[102221]: osdmap e285: 8 total, 8 up, 7 in 2024-09-21T10:13:05.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:05.158 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:05.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:05.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:05.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:04 smithi129 ceph-mon[88471]: pgmap v1010: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 1.7 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 870 op/s; 1714/83163 objects misplaced (2.061%); 16 MiB/s, 255 objects/s recovering 2024-09-21T10:13:05.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:04 smithi129 ceph-mon[88471]: osdmap e285: 8 total, 8 up, 7 in 2024-09-21T10:13:05.259 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:05.287 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:05.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:05.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:05.388 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:05.424 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:05.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:05.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:05.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:05.551 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:05.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:05.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:05.653 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:05.679 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:05.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:05.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:05.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:05.815 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:05.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:05.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:05.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:13:05.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:05.960 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:05.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:05.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:06.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:06.098 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:06.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:06.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:06.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:06.256 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:06.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:06.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:06.358 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:06.381 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:06.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:06.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:06.449 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:06.468 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:06.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:06.559 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:06.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:06.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:06.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:06.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:06.708 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:06.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:06.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:06.853 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:06.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:06.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:06.897 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:06 smithi107 ceph-mon[102221]: pgmap v1012: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 736 op/s; 1714/80949 objects misplaced (2.117%); 8.6 MiB/s, 137 objects/s recovering 2024-09-21T10:13:06.897 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:06 smithi107 ceph-mon[105022]: pgmap v1012: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 736 op/s; 1714/80949 objects misplaced (2.117%); 8.6 MiB/s, 137 objects/s recovering 2024-09-21T10:13:06.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:06.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:06.996 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:06.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:07.097 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:07.121 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:07.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:07.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:07.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:06 smithi129 ceph-mon[88471]: pgmap v1012: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 736 op/s; 1714/80949 objects misplaced (2.117%); 8.6 MiB/s, 137 objects/s recovering 2024-09-21T10:13:07.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:07.268 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:07.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:07.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:07.370 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:07.401 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:07.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:07.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:07.502 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:07.527 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:07.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:07.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:07.628 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:07.653 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:07.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:07.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:07.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:07.785 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:07.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:07.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:07.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:07.923 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:07.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:07.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:08.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:08.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:08.058 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:08.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:08.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:07 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2533877034' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:08.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:07 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2533877034' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:08.159 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:08.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:07 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2533877034' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:08.185 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:08.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:08.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:08.252 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:13:08.252 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:13:08.252 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:13:08.252 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:13:08.252 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:13:08.252 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:13:08.252 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:13:08.252 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:13:08.252 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:13:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:13:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:13:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:13:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:13:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:13:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:13:08.253 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:13:08.286 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:08.330 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:08.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:08.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:08.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:08.474 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:08.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:08.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:08.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:08.614 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:08.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:08.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:08.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:08.751 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:08.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:08.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:08.853 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:08 smithi107 ceph-mon[102221]: pgmap v1013: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.5 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 843 op/s; 528/73380 objects misplaced (0.720%); 15 MiB/s, 241 objects/s recovering 2024-09-21T10:13:08.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:08 smithi107 ceph-mon[105022]: pgmap v1013: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.5 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 843 op/s; 528/73380 objects misplaced (0.720%); 15 MiB/s, 241 objects/s recovering 2024-09-21T10:13:08.879 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:08.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:08.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:08.903 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:08.981 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:09.008 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:09.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:09.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:09.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:09.149 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:09.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:09.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:09.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:08 smithi129 ceph-mon[88471]: pgmap v1013: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.5 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 843 op/s; 528/73380 objects misplaced (0.720%); 15 MiB/s, 241 objects/s recovering 2024-09-21T10:13:09.250 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:09.279 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:09.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:09.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:09.380 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:09.410 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:09.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:09.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:09.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:09.551 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:09.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:09.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:09.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:09.691 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:09.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:09.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:09.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:09.828 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:09.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:09.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:09.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:09.972 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:09.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:09.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:10.074 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:10.101 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:10.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:10.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:10.203 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:10.226 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:10.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:10.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:10.328 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:10.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:10.354 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:10.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:10.455 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:10.479 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:10.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:10.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:10.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:10.624 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:10.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:10.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:10.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:13:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:13:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:13:10.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:10.751 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:10.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:10.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:10.853 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:10.887 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:10.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:10.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:10.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:13:10.985 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:10 smithi129 ceph-mon[88471]: pgmap v1014: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.4 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 528/69516 objects misplaced (0.760%); 15 MiB/s, 241 objects/s recovering 2024-09-21T10:13:10.988 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:11.012 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:11.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:11.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:11.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:10 smithi107 ceph-mon[102221]: pgmap v1014: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.4 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 528/69516 objects misplaced (0.760%); 15 MiB/s, 241 objects/s recovering 2024-09-21T10:13:11.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:10 smithi107 ceph-mon[105022]: pgmap v1014: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.4 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 528/69516 objects misplaced (0.760%); 15 MiB/s, 241 objects/s recovering 2024-09-21T10:13:11.114 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:11.135 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:11.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:11.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:11.237 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:11.260 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:11.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:11.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:11.362 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:11.384 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:11.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:11.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:11.486 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:11.507 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:11.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:11.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:11.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:11.632 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:11.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:11.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:11.665 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:13:11.667 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:13:11.733 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:11.755 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:11.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:11.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:11.857 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:11.878 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:11.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:11.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:11.980 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:12.006 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:12.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:12.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:12.050 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:11 smithi107 ceph-mon[102221]: osdmap e286: 8 total, 8 up, 7 in 2024-09-21T10:13:12.050 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:11 smithi107 ceph-mon[102221]: from='client.36717 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:13:12.050 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:12.050 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:13:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:13:11] "GET /metrics HTTP/1.1" 200 36810 "" "Prometheus/2.51.0" 2024-09-21T10:13:12.051 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:11 smithi107 ceph-mon[105022]: osdmap e286: 8 total, 8 up, 7 in 2024-09-21T10:13:12.051 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:11 smithi107 ceph-mon[105022]: from='client.36717 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:13:12.051 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:12.107 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:12.140 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:12.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:12.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:12.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:11 smithi129 ceph-mon[88471]: osdmap e286: 8 total, 8 up, 7 in 2024-09-21T10:13:12.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:11 smithi129 ceph-mon[88471]: from='client.36717 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:13:12.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:12.243 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:12.276 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:12.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:12.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:12.378 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:12.402 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:12.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:12.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:12.504 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:12.516 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1015,"stamp":"2024-09-21T10:13:10.834686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1517459519,"num_objects":23172,"num_object_clones":0,"num_object_copies":69516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":528,"num_objects_unfound":0,"num_objects_dirty":23172,"num_whiteouts":0,"num_read":50,"num_read_kb":126,"num_write":201914,"num_write_kb":7199280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11151,"num_bytes_recovered":727593651,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104184,"ondisk_log_size":104184,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":329,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":9947400,"kb_used_data":4602120,"kb_used_omap":41,"kb_used_meta":5345174,"kb_avail":646268664,"statfs":{"total":671965249536,"available":661779111936,"internally_reserved":0,"allocated":4712570880,"data_stored":4701209137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42519,"internal_metadata":5473458665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,6,5,6,7,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-335544320,"num_objects":-5120,"num_object_clones":0,"num_object_copies":-15360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1186,"num_objects_unfound":0,"num_objects_dirty":-5120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5120,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1029,"num_bytes_recovered":67436544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.064752"},"pg_stats":[{"pgid":"1.11","version":"77'100","reported_seq":86,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.790893+0000","last_change":"2024-09-21T10:10:57.397786+0000","last_active":"2024-09-21T10:13:03.790893+0000","last_peered":"2024-09-21T10:13:03.790893+0000","last_clean":"2024-09-21T10:13:03.790893+0000","last_became_active":"2024-09-21T10:10:57.396742+0000","last_became_peered":"2024-09-21T10:10:57.396742+0000","last_unstale":"2024-09-21T10:13:03.790893+0000","last_undegraded":"2024-09-21T10:13:03.790893+0000","last_fullsized":"2024-09-21T10:13:03.790893+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:53.631923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:53.631923+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:33:35.364595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":424,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.794514+0000","last_change":"2024-09-21T10:03:38.389577+0000","last_active":"2024-09-21T10:13:03.794514+0000","last_peered":"2024-09-21T10:13:03.794514+0000","last_clean":"2024-09-21T10:13:03.794514+0000","last_became_active":"2024-09-21T10:03:38.389292+0000","last_became_peered":"2024-09-21T10:03:38.389292+0000","last_unstale":"2024-09-21T10:13:03.794514+0000","last_undegraded":"2024-09-21T10:13:03.794514+0000","last_fullsized":"2024-09-21T10:13:03.794514+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:32.113472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":155,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.779980+0000","last_change":"2024-09-21T10:08:38.387871+0000","last_active":"2024-09-21T10:13:02.779980+0000","last_peered":"2024-09-21T10:13:02.779980+0000","last_clean":"2024-09-21T10:13:02.779980+0000","last_became_active":"2024-09-21T10:08:38.376727+0000","last_became_peered":"2024-09-21T10:08:38.376727+0000","last_unstale":"2024-09-21T10:13:02.779980+0000","last_undegraded":"2024-09-21T10:13:02.779980+0000","last_fullsized":"2024-09-21T10:13:02.779980+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:45:54.498950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'100","reported_seq":184,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.779672+0000","last_change":"2024-09-21T10:11:05.438031+0000","last_active":"2024-09-21T10:13:02.779672+0000","last_peered":"2024-09-21T10:13:02.779672+0000","last_clean":"2024-09-21T10:13:02.779672+0000","last_became_active":"2024-09-21T10:11:05.437637+0000","last_became_peered":"2024-09-21T10:11:05.437637+0000","last_unstale":"2024-09-21T10:13:02.779672+0000","last_undegraded":"2024-09-21T10:13:02.779672+0000","last_fullsized":"2024-09-21T10:13:02.779672+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.798577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.798577+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:42.343217+0000","scrub_duration":0.0023622640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":276,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.780274+0000","last_change":"2024-09-21T10:10:57.392782+0000","last_active":"2024-09-21T10:13:02.780274+0000","last_peered":"2024-09-21T10:13:02.780274+0000","last_clean":"2024-09-21T10:13:02.780274+0000","last_became_active":"2024-09-21T10:10:57.392305+0000","last_became_peered":"2024-09-21T10:10:57.392305+0000","last_unstale":"2024-09-21T10:13:02.780274+0000","last_undegraded":"2024-09-21T10:13:02.780274+0000","last_fullsized":"2024-09-21T10:13:02.780274+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:42:01.387622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":345,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.791358+0000","last_change":"2024-09-21T10:09:23.846703+0000","last_active":"2024-09-21T10:13:03.791358+0000","last_peered":"2024-09-21T10:13:03.791358+0000","last_clean":"2024-09-21T10:13:03.791358+0000","last_became_active":"2024-09-21T10:09:23.846415+0000","last_became_peered":"2024-09-21T10:09:23.846415+0000","last_unstale":"2024-09-21T10:13:03.791358+0000","last_undegraded":"2024-09-21T10:13:03.791358+0000","last_fullsized":"2024-09-21T10:13:03.791358+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:23.129285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'100","reported_seq":139,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.794525+0000","last_change":"2024-09-21T10:09:24.841480+0000","last_active":"2024-09-21T10:13:03.794525+0000","last_peered":"2024-09-21T10:13:03.794525+0000","last_clean":"2024-09-21T10:13:03.794525+0000","last_became_active":"2024-09-21T10:09:24.841145+0000","last_became_peered":"2024-09-21T10:09:24.841145+0000","last_unstale":"2024-09-21T10:13:03.794525+0000","last_undegraded":"2024-09-21T10:13:03.794525+0000","last_fullsized":"2024-09-21T10:13:03.794525+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.433227+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:13:41.719395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":23,"num_write":5,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":35780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":71,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.790881+0000","last_change":"2024-09-21T10:09:49.922489+0000","last_active":"2024-09-21T10:13:03.790881+0000","last_peered":"2024-09-21T10:13:03.790881+0000","last_clean":"2024-09-21T10:13:03.790881+0000","last_became_active":"2024-09-21T10:09:49.921629+0000","last_became_peered":"2024-09-21T10:09:49.921629+0000","last_unstale":"2024-09-21T10:13:03.790881+0000","last_undegraded":"2024-09-21T10:13:03.790881+0000","last_fullsized":"2024-09-21T10:13:03.790881+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:50.611979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":157,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.779654+0000","last_change":"2024-09-21T10:09:24.836647+0000","last_active":"2024-09-21T10:13:02.779654+0000","last_peered":"2024-09-21T10:13:02.779654+0000","last_clean":"2024-09-21T10:13:02.779654+0000","last_became_active":"2024-09-21T10:09:24.836278+0000","last_became_peered":"2024-09-21T10:09:24.836278+0000","last_unstale":"2024-09-21T10:13:02.779654+0000","last_undegraded":"2024-09-21T10:13:02.779654+0000","last_fullsized":"2024-09-21T10:13:02.779654+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:08:36.535892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"77'100","reported_seq":28,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.790244+0000","last_change":"2024-09-21T10:10:57.452531+0000","last_active":"2024-09-21T10:13:02.790244+0000","last_peered":"2024-09-21T10:13:02.790244+0000","last_clean":"2024-09-21T10:13:02.790244+0000","last_became_active":"2024-09-21T10:10:57.452150+0000","last_became_peered":"2024-09-21T10:10:57.452150+0000","last_unstale":"2024-09-21T10:13:02.790244+0000","last_undegraded":"2024-09-21T10:13:02.790244+0000","last_fullsized":"2024-09-21T10:13:02.790244+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:00.058077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:00.058077+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":424,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.794896+0000","last_change":"2024-09-21T10:03:38.381552+0000","last_active":"2024-09-21T10:13:03.794896+0000","last_peered":"2024-09-21T10:13:03.794896+0000","last_clean":"2024-09-21T10:13:03.794896+0000","last_became_active":"2024-09-21T10:03:38.380898+0000","last_became_peered":"2024-09-21T10:03:38.380898+0000","last_unstale":"2024-09-21T10:13:03.794896+0000","last_undegraded":"2024-09-21T10:13:03.794896+0000","last_fullsized":"2024-09-21T10:13:03.794896+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:53.409606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":344,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.778977+0000","last_change":"2024-09-21T10:10:57.459057+0000","last_active":"2024-09-21T10:13:02.778977+0000","last_peered":"2024-09-21T10:13:02.778977+0000","last_clean":"2024-09-21T10:13:02.778977+0000","last_became_active":"2024-09-21T10:10:57.451776+0000","last_became_peered":"2024-09-21T10:10:57.451776+0000","last_unstale":"2024-09-21T10:13:02.778977+0000","last_undegraded":"2024-09-21T10:13:02.778977+0000","last_fullsized":"2024-09-21T10:13:02.778977+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:00.138317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"285'12528","reported_seq":16503,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.081926+0000","last_change":"2024-09-21T10:13:03.848496+0000","last_active":"2024-09-21T10:13:07.081926+0000","last_peered":"2024-09-21T10:13:07.081926+0000","last_clean":"2024-09-21T10:13:07.081926+0000","last_became_active":"2024-09-21T10:13:03.793148+0000","last_became_peered":"2024-09-21T10:13:03.793148+0000","last_unstale":"2024-09-21T10:13:07.081926+0000","last_undegraded":"2024-09-21T10:13:07.081926+0000","last_fullsized":"2024-09-21T10:13:07.081926+0000","mapping_epoch":284,"log_start":"270'6100","ondisk_log_start":"270'6100","created":252,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6428,"log_dups_size":0,"ondisk_log_size":6428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:00:29.314783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95420416,"num_objects":1456,"num_object_clones":0,"num_object_copies":4368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12528,"num_write_kb":447488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1937,"num_bytes_recovered":126943232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"77'100","reported_seq":169,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.780281+0000","last_change":"2024-09-21T10:08:38.374573+0000","last_active":"2024-09-21T10:13:02.780281+0000","last_peered":"2024-09-21T10:13:02.780281+0000","last_clean":"2024-09-21T10:13:02.780281+0000","last_became_active":"2024-09-21T10:08:38.374075+0000","last_became_peered":"2024-09-21T10:08:38.374075+0000","last_unstale":"2024-09-21T10:13:02.780281+0000","last_undegraded":"2024-09-21T10:13:02.780281+0000","last_fullsized":"2024-09-21T10:13:02.780281+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:55.259385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:55.259385+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:28:26.784162+0000","scrub_duration":0.0015667260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":200,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.780387+0000","last_change":"2024-09-21T10:09:49.941392+0000","last_active":"2024-09-21T10:13:02.780387+0000","last_peered":"2024-09-21T10:13:02.780387+0000","last_clean":"2024-09-21T10:13:02.780387+0000","last_became_active":"2024-09-21T10:09:49.923535+0000","last_became_peered":"2024-09-21T10:09:49.923535+0000","last_unstale":"2024-09-21T10:13:02.780387+0000","last_undegraded":"2024-09-21T10:13:02.780387+0000","last_fullsized":"2024-09-21T10:13:02.780387+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:16.292163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":38,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.791670+0000","last_change":"2024-09-21T10:11:05.435502+0000","last_active":"2024-09-21T10:13:02.791670+0000","last_peered":"2024-09-21T10:13:02.791670+0000","last_clean":"2024-09-21T10:13:02.791670+0000","last_became_active":"2024-09-21T10:11:05.435216+0000","last_became_peered":"2024-09-21T10:11:05.435216+0000","last_unstale":"2024-09-21T10:13:02.791670+0000","last_undegraded":"2024-09-21T10:13:02.791670+0000","last_fullsized":"2024-09-21T10:13:02.791670+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:25.434424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"285'12368","reported_seq":12440,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.046201+0000","last_change":"2024-09-21T10:09:49.924187+0000","last_active":"2024-09-21T10:13:07.046201+0000","last_peered":"2024-09-21T10:13:07.046201+0000","last_clean":"2024-09-21T10:13:07.046201+0000","last_became_active":"2024-09-21T10:09:49.923789+0000","last_became_peered":"2024-09-21T10:09:49.923789+0000","last_unstale":"2024-09-21T10:13:07.046201+0000","last_undegraded":"2024-09-21T10:13:07.046201+0000","last_fullsized":"2024-09-21T10:13:07.046201+0000","mapping_epoch":255,"log_start":"268'5900","ondisk_log_start":"268'5900","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6468,"log_dups_size":0,"ondisk_log_size":6468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:52:49.559425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98172928,"num_objects":1498,"num_object_clones":0,"num_object_copies":4494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12368,"num_write_kb":443712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"77'100","reported_seq":230,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.794364+0000","last_change":"2024-09-21T10:09:23.839211+0000","last_active":"2024-09-21T10:13:03.794364+0000","last_peered":"2024-09-21T10:13:03.794364+0000","last_clean":"2024-09-21T10:13:03.794364+0000","last_became_active":"2024-09-21T10:09:23.838779+0000","last_became_peered":"2024-09-21T10:09:23.838779+0000","last_unstale":"2024-09-21T10:13:03.794364+0000","last_undegraded":"2024-09-21T10:13:03.794364+0000","last_fullsized":"2024-09-21T10:13:03.794364+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.981336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.981336+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:03:15.566873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":179,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.780434+0000","last_change":"2024-09-21T10:09:49.932631+0000","last_active":"2024-09-21T10:13:02.780434+0000","last_peered":"2024-09-21T10:13:02.780434+0000","last_clean":"2024-09-21T10:13:02.780434+0000","last_became_active":"2024-09-21T10:09:49.922897+0000","last_became_peered":"2024-09-21T10:09:49.922897+0000","last_unstale":"2024-09-21T10:13:02.780434+0000","last_undegraded":"2024-09-21T10:13:02.780434+0000","last_fullsized":"2024-09-21T10:13:02.780434+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:40.795476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":219,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.794456+0000","last_change":"2024-09-21T10:08:38.377732+0000","last_active":"2024-09-21T10:13:03.794456+0000","last_peered":"2024-09-21T10:13:03.794456+0000","last_clean":"2024-09-21T10:13:03.794456+0000","last_became_active":"2024-09-21T10:08:38.376601+0000","last_became_peered":"2024-09-21T10:08:38.376601+0000","last_unstale":"2024-09-21T10:13:03.794456+0000","last_undegraded":"2024-09-21T10:13:03.794456+0000","last_fullsized":"2024-09-21T10:13:03.794456+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:02.540158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.5","version":"285'12097","reported_seq":23115,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:06.837510+0000","last_change":"2024-09-21T10:12:35.556125+0000","last_active":"2024-09-21T10:13:06.837510+0000","last_peered":"2024-09-21T10:13:06.837510+0000","last_clean":"2024-09-21T10:13:06.837510+0000","last_became_active":"2024-09-21T10:12:35.483696+0000","last_became_peered":"2024-09-21T10:12:35.483696+0000","last_unstale":"2024-09-21T10:13:06.837510+0000","last_undegraded":"2024-09-21T10:13:06.837510+0000","last_fullsized":"2024-09-21T10:13:06.837510+0000","mapping_epoch":278,"log_start":"268'5800","ondisk_log_start":"268'5800","created":252,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6297,"log_dups_size":0,"ondisk_log_size":6297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:54:35.212835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92995584,"num_objects":1419,"num_object_clones":0,"num_object_copies":4257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12097,"num_write_kb":432512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5478,"num_bytes_recovered":359006208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"77'100","reported_seq":269,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.779178+0000","last_change":"2024-09-21T10:08:38.382100+0000","last_active":"2024-09-21T10:13:02.779178+0000","last_peered":"2024-09-21T10:13:02.779178+0000","last_clean":"2024-09-21T10:13:02.779178+0000","last_became_active":"2024-09-21T10:08:38.381736+0000","last_became_peered":"2024-09-21T10:08:38.381736+0000","last_unstale":"2024-09-21T10:13:02.779178+0000","last_undegraded":"2024-09-21T10:13:02.779178+0000","last_fullsized":"2024-09-21T10:13:02.779178+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.289859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.289859+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:23.739980+0000","scrub_duration":0.001251809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":165,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.780198+0000","last_change":"2024-09-21T10:10:57.460732+0000","last_active":"2024-09-21T10:13:02.780198+0000","last_peered":"2024-09-21T10:13:02.780198+0000","last_clean":"2024-09-21T10:13:02.780198+0000","last_became_active":"2024-09-21T10:10:57.454689+0000","last_became_peered":"2024-09-21T10:10:57.454689+0000","last_unstale":"2024-09-21T10:13:02.780198+0000","last_undegraded":"2024-09-21T10:13:02.780198+0000","last_fullsized":"2024-09-21T10:13:02.780198+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:12.035022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":285,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.778772+0000","last_change":"2024-09-21T10:03:38.378384+0000","last_active":"2024-09-21T10:13:02.778772+0000","last_peered":"2024-09-21T10:13:02.778772+0000","last_clean":"2024-09-21T10:13:02.778772+0000","last_became_active":"2024-09-21T10:03:38.378072+0000","last_became_peered":"2024-09-21T10:03:38.378072+0000","last_unstale":"2024-09-21T10:13:02.778772+0000","last_undegraded":"2024-09-21T10:13:02.778772+0000","last_fullsized":"2024-09-21T10:13:02.778772+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:54:37.039622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"285'12723","reported_seq":12764,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:06.827776+0000","last_change":"2024-09-21T10:09:42.896949+0000","last_active":"2024-09-21T10:13:06.827776+0000","last_peered":"2024-09-21T10:13:06.827776+0000","last_clean":"2024-09-21T10:13:06.827776+0000","last_became_active":"2024-09-21T10:09:42.896669+0000","last_became_peered":"2024-09-21T10:09:42.896669+0000","last_unstale":"2024-09-21T10:13:06.827776+0000","last_undegraded":"2024-09-21T10:13:06.827776+0000","last_fullsized":"2024-09-21T10:13:06.827776+0000","mapping_epoch":252,"log_start":"271'6400","ondisk_log_start":"271'6400","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6323,"log_dups_size":0,"ondisk_log_size":6323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:44.279472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98369536,"num_objects":1501,"num_object_clones":0,"num_object_copies":4503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12723,"num_write_kb":455168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"77'100","reported_seq":137,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.780227+0000","last_change":"2024-09-21T10:07:26.188298+0000","last_active":"2024-09-21T10:13:02.780227+0000","last_peered":"2024-09-21T10:13:02.780227+0000","last_clean":"2024-09-21T10:13:02.780227+0000","last_became_active":"2024-09-21T10:07:24.345265+0000","last_became_peered":"2024-09-21T10:07:24.345265+0000","last_unstale":"2024-09-21T10:13:02.780227+0000","last_undegraded":"2024-09-21T10:13:02.780227+0000","last_fullsized":"2024-09-21T10:13:02.780227+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:07:26.188220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:07:26.188220+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:46.381710+0000","scrub_duration":0.021884888000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":228,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.794771+0000","last_change":"2024-09-21T10:07:57.551556+0000","last_active":"2024-09-21T10:13:03.794771+0000","last_peered":"2024-09-21T10:13:03.794771+0000","last_clean":"2024-09-21T10:13:03.794771+0000","last_became_active":"2024-09-21T10:07:57.551249+0000","last_became_peered":"2024-09-21T10:07:57.551249+0000","last_unstale":"2024-09-21T10:13:03.794771+0000","last_undegraded":"2024-09-21T10:13:03.794771+0000","last_fullsized":"2024-09-21T10:13:03.794771+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:57.557152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":325,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.791451+0000","last_change":"2024-09-21T10:09:24.839665+0000","last_active":"2024-09-21T10:13:03.791451+0000","last_peered":"2024-09-21T10:13:03.791451+0000","last_clean":"2024-09-21T10:13:03.791451+0000","last_became_active":"2024-09-21T10:09:24.839353+0000","last_became_peered":"2024-09-21T10:09:24.839353+0000","last_unstale":"2024-09-21T10:13:03.791451+0000","last_undegraded":"2024-09-21T10:13:03.791451+0000","last_fullsized":"2024-09-21T10:13:03.791451+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:51.760073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3","version":"285'12409","reported_seq":12450,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.822349+0000","last_change":"2024-09-21T10:09:43.231199+0000","last_active":"2024-09-21T10:13:07.822349+0000","last_peered":"2024-09-21T10:13:07.822349+0000","last_clean":"2024-09-21T10:13:07.822349+0000","last_became_active":"2024-09-21T10:09:43.230922+0000","last_became_peered":"2024-09-21T10:09:43.230922+0000","last_unstale":"2024-09-21T10:13:07.822349+0000","last_undegraded":"2024-09-21T10:13:07.822349+0000","last_fullsized":"2024-09-21T10:13:07.822349+0000","mapping_epoch":252,"log_start":"271'6200","ondisk_log_start":"271'6200","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6209,"log_dups_size":0,"ondisk_log_size":6209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:39.944122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96272384,"num_objects":1469,"num_object_clones":0,"num_object_copies":4407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12409,"num_write_kb":444096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":159,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.779737+0000","last_change":"2024-09-21T10:04:53.355975+0000","last_active":"2024-09-21T10:13:02.779737+0000","last_peered":"2024-09-21T10:13:02.779737+0000","last_clean":"2024-09-21T10:13:02.779737+0000","last_became_active":"2024-09-21T10:04:42.547505+0000","last_became_peered":"2024-09-21T10:04:42.547505+0000","last_unstale":"2024-09-21T10:13:02.779737+0000","last_undegraded":"2024-09-21T10:13:02.779737+0000","last_fullsized":"2024-09-21T10:13:02.779737+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.355912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.355912+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:43.456830+0000","scrub_duration":0.0017980030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":424,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.794773+0000","last_change":"2024-09-21T10:09:49.939553+0000","last_active":"2024-09-21T10:13:03.794773+0000","last_peered":"2024-09-21T10:13:03.794773+0000","last_clean":"2024-09-21T10:13:03.794773+0000","last_became_active":"2024-09-21T10:09:49.925094+0000","last_became_peered":"2024-09-21T10:09:49.925094+0000","last_unstale":"2024-09-21T10:13:03.794773+0000","last_undegraded":"2024-09-21T10:13:03.794773+0000","last_fullsized":"2024-09-21T10:13:03.794773+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:12:12.933089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":201,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.794921+0000","last_change":"2024-09-21T10:08:38.379747+0000","last_active":"2024-09-21T10:13:03.794921+0000","last_peered":"2024-09-21T10:13:03.794921+0000","last_clean":"2024-09-21T10:13:03.794921+0000","last_became_active":"2024-09-21T10:08:38.378983+0000","last_became_peered":"2024-09-21T10:08:38.378983+0000","last_unstale":"2024-09-21T10:13:03.794921+0000","last_undegraded":"2024-09-21T10:13:03.794921+0000","last_fullsized":"2024-09-21T10:13:03.794921+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:45:08.003016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"285'12632","reported_seq":12710,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:09.282934+0000","last_change":"2024-09-21T10:09:42.896430+0000","last_active":"2024-09-21T10:13:09.282934+0000","last_peered":"2024-09-21T10:13:09.282934+0000","last_clean":"2024-09-21T10:13:09.282934+0000","last_became_active":"2024-09-21T10:09:42.896019+0000","last_became_peered":"2024-09-21T10:09:42.896019+0000","last_unstale":"2024-09-21T10:13:09.282934+0000","last_undegraded":"2024-09-21T10:13:09.282934+0000","last_fullsized":"2024-09-21T10:13:09.282934+0000","mapping_epoch":252,"log_start":"264'5100","ondisk_log_start":"264'5100","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":7532,"log_dups_size":0,"ondisk_log_size":7532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:59:18.633498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89128960,"num_objects":1360,"num_object_clones":0,"num_object_copies":4080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12632,"num_write_kb":447744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":184,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.855635+0000","last_change":"2024-09-21T10:10:57.403722+0000","last_active":"2024-09-21T10:13:03.855635+0000","last_peered":"2024-09-21T10:13:03.855635+0000","last_clean":"2024-09-21T10:13:03.855635+0000","last_became_active":"2024-09-21T10:10:57.394700+0000","last_became_peered":"2024-09-21T10:10:57.394700+0000","last_unstale":"2024-09-21T10:13:03.855635+0000","last_undegraded":"2024-09-21T10:13:03.855635+0000","last_fullsized":"2024-09-21T10:13:03.855635+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:02.152981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:02.152981+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:52:34.887763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":259,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.778864+0000","last_change":"2024-09-21T10:08:38.379849+0000","last_active":"2024-09-21T10:13:02.778864+0000","last_peered":"2024-09-21T10:13:02.778864+0000","last_clean":"2024-09-21T10:13:02.778864+0000","last_became_active":"2024-09-21T10:08:38.379543+0000","last_became_peered":"2024-09-21T10:08:38.379543+0000","last_unstale":"2024-09-21T10:13:02.778864+0000","last_undegraded":"2024-09-21T10:13:02.778864+0000","last_fullsized":"2024-09-21T10:13:02.778864+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:29.041802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":233,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.856496+0000","last_change":"2024-09-21T10:07:24.342950+0000","last_active":"2024-09-21T10:13:03.856496+0000","last_peered":"2024-09-21T10:13:03.856496+0000","last_clean":"2024-09-21T10:13:03.856496+0000","last_became_active":"2024-09-21T10:07:24.342477+0000","last_became_peered":"2024-09-21T10:07:24.342477+0000","last_unstale":"2024-09-21T10:13:03.856496+0000","last_undegraded":"2024-09-21T10:13:03.856496+0000","last_fullsized":"2024-09-21T10:13:03.856496+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:33:13.664050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1","version":"285'12736","reported_seq":12776,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:07.838965+0000","last_change":"2024-09-21T10:09:42.895259+0000","last_active":"2024-09-21T10:13:07.838965+0000","last_peered":"2024-09-21T10:13:07.838965+0000","last_clean":"2024-09-21T10:13:07.838965+0000","last_became_active":"2024-09-21T10:09:42.895087+0000","last_became_peered":"2024-09-21T10:09:42.895087+0000","last_unstale":"2024-09-21T10:13:07.838965+0000","last_undegraded":"2024-09-21T10:13:07.838965+0000","last_fullsized":"2024-09-21T10:13:07.838965+0000","mapping_epoch":252,"log_start":"271'6600","ondisk_log_start":"271'6600","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6136,"log_dups_size":0,"ondisk_log_size":6136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:40:59.270926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90439680,"num_objects":1380,"num_object_clones":0,"num_object_copies":4140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12736,"num_write_kb":451712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"77'100","reported_seq":327,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.791557+0000","last_change":"2024-09-21T10:10:57.395906+0000","last_active":"2024-09-21T10:13:03.791557+0000","last_peered":"2024-09-21T10:13:03.791557+0000","last_clean":"2024-09-21T10:13:03.791557+0000","last_became_active":"2024-09-21T10:10:57.395366+0000","last_became_peered":"2024-09-21T10:10:57.395366+0000","last_unstale":"2024-09-21T10:13:03.791557+0000","last_undegraded":"2024-09-21T10:13:03.791557+0000","last_fullsized":"2024-09-21T10:13:03.791557+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.540323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.540323+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:34.873336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":325,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.855865+0000","last_change":"2024-09-21T10:07:36.464496+0000","last_active":"2024-09-21T10:13:03.855865+0000","last_peered":"2024-09-21T10:13:03.855865+0000","last_clean":"2024-09-21T10:13:03.855865+0000","last_became_active":"2024-09-21T10:07:36.464172+0000","last_became_peered":"2024-09-21T10:07:36.464172+0000","last_unstale":"2024-09-21T10:13:03.855865+0000","last_undegraded":"2024-09-21T10:13:03.855865+0000","last_fullsized":"2024-09-21T10:13:03.855865+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:09.839709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.778870+0000","last_change":"2024-09-21T10:11:05.442332+0000","last_active":"2024-09-21T10:13:02.778870+0000","last_peered":"2024-09-21T10:13:02.778870+0000","last_clean":"2024-09-21T10:13:02.778870+0000","last_became_active":"2024-09-21T10:11:05.442003+0000","last_became_peered":"2024-09-21T10:11:05.442003+0000","last_unstale":"2024-09-21T10:13:02.778870+0000","last_undegraded":"2024-09-21T10:13:02.778870+0000","last_fullsized":"2024-09-21T10:13:02.778870+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:34.986309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"285'12576","reported_seq":12654,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:06.692653+0000","last_change":"2024-09-21T10:09:43.231401+0000","last_active":"2024-09-21T10:13:06.692653+0000","last_peered":"2024-09-21T10:13:06.692653+0000","last_clean":"2024-09-21T10:13:06.692653+0000","last_became_active":"2024-09-21T10:09:43.230977+0000","last_became_peered":"2024-09-21T10:09:43.230977+0000","last_unstale":"2024-09-21T10:13:06.692653+0000","last_undegraded":"2024-09-21T10:13:06.692653+0000","last_fullsized":"2024-09-21T10:13:06.692653+0000","mapping_epoch":252,"log_start":"272'6800","ondisk_log_start":"272'6800","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5776,"log_dups_size":0,"ondisk_log_size":5776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:15.907809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98041856,"num_objects":1496,"num_object_clones":0,"num_object_copies":4488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12576,"num_write_kb":450304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":147,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.855937+0000","last_change":"2024-09-21T10:08:04.645700+0000","last_active":"2024-09-21T10:13:03.855937+0000","last_peered":"2024-09-21T10:13:03.855937+0000","last_clean":"2024-09-21T10:13:03.855937+0000","last_became_active":"2024-09-21T10:07:57.548189+0000","last_became_peered":"2024-09-21T10:07:57.548189+0000","last_unstale":"2024-09-21T10:13:03.855937+0000","last_undegraded":"2024-09-21T10:13:03.855937+0000","last_fullsized":"2024-09-21T10:13:03.855937+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:48.318010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:48.318010+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T16:31:28.173059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":425,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.794253+0000","last_change":"2024-09-21T10:03:38.386604+0000","last_active":"2024-09-21T10:13:03.794253+0000","last_peered":"2024-09-21T10:13:03.794253+0000","last_clean":"2024-09-21T10:13:03.794253+0000","last_became_active":"2024-09-21T10:03:38.386283+0000","last_became_peered":"2024-09-21T10:03:38.386283+0000","last_unstale":"2024-09-21T10:13:03.794253+0000","last_undegraded":"2024-09-21T10:13:03.794253+0000","last_fullsized":"2024-09-21T10:13:03.794253+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:27:05.944880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":229,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.794854+0000","last_change":"2024-09-21T10:08:38.372486+0000","last_active":"2024-09-21T10:13:03.794854+0000","last_peered":"2024-09-21T10:13:03.794854+0000","last_clean":"2024-09-21T10:13:03.794854+0000","last_became_active":"2024-09-21T10:08:38.370669+0000","last_became_peered":"2024-09-21T10:08:38.370669+0000","last_unstale":"2024-09-21T10:13:03.794854+0000","last_undegraded":"2024-09-21T10:13:03.794854+0000","last_fullsized":"2024-09-21T10:13:03.794854+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:53:23.247341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.a","version":"285'12488","reported_seq":14534,"reported_epoch":285,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T10:13:06.691842+0000","last_change":"2024-09-21T10:13:02.326578+0000","last_active":"2024-09-21T10:13:06.691842+0000","last_peered":"2024-09-21T10:13:06.691842+0000","last_clean":"2024-09-21T10:10:56.390149+0000","last_became_active":"2024-09-21T10:10:58.682906+0000","last_became_peered":"2024-09-21T10:10:58.682906+0000","last_unstale":"2024-09-21T10:13:06.691842+0000","last_undegraded":"2024-09-21T10:13:06.691842+0000","last_fullsized":"2024-09-21T10:13:06.691842+0000","mapping_epoch":267,"log_start":"272'6700","ondisk_log_start":"272'6700","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5788,"log_dups_size":0,"ondisk_log_size":5788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93716480,"num_objects":1430,"num_object_clones":0,"num_object_copies":4290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":528,"num_objects_unfound":0,"num_objects_dirty":1430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12488,"num_write_kb":445376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":979,"num_bytes_recovered":64159744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,1],"avail_no_missing":["3","1","6"],"object_location_counts":[{"shards":"1,3,6","objects":1430}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":176,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.779590+0000","last_change":"2024-09-21T10:09:49.924286+0000","last_active":"2024-09-21T10:13:02.779590+0000","last_peered":"2024-09-21T10:13:02.779590+0000","last_clean":"2024-09-21T10:13:02.779590+0000","last_became_active":"2024-09-21T10:09:49.923897+0000","last_became_peered":"2024-09-21T10:09:49.923897+0000","last_unstale":"2024-09-21T10:13:02.779590+0000","last_undegraded":"2024-09-21T10:13:02.779590+0000","last_fullsized":"2024-09-21T10:13:02.779590+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:47.393146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:47.393146+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:44.898083+0000","scrub_duration":0.001601086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":180,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.856380+0000","last_change":"2024-09-21T10:10:57.402946+0000","last_active":"2024-09-21T10:13:03.856380+0000","last_peered":"2024-09-21T10:13:03.856380+0000","last_clean":"2024-09-21T10:13:03.856380+0000","last_became_active":"2024-09-21T10:10:57.395450+0000","last_became_peered":"2024-09-21T10:10:57.395450+0000","last_unstale":"2024-09-21T10:13:03.856380+0000","last_undegraded":"2024-09-21T10:13:03.856380+0000","last_fullsized":"2024-09-21T10:13:03.856380+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:08.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":410,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.795241+0000","last_change":"2024-09-21T10:10:57.390595+0000","last_active":"2024-09-21T10:13:03.795241+0000","last_peered":"2024-09-21T10:13:03.795241+0000","last_clean":"2024-09-21T10:13:03.795241+0000","last_became_active":"2024-09-21T10:10:57.390265+0000","last_became_peered":"2024-09-21T10:10:57.390265+0000","last_unstale":"2024-09-21T10:13:03.795241+0000","last_undegraded":"2024-09-21T10:13:03.795241+0000","last_fullsized":"2024-09-21T10:13:03.795241+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:37.485063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":292,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.779333+0000","last_change":"2024-09-21T10:09:24.837651+0000","last_active":"2024-09-21T10:13:02.779333+0000","last_peered":"2024-09-21T10:13:02.779333+0000","last_clean":"2024-09-21T10:13:02.779333+0000","last_became_active":"2024-09-21T10:09:24.837226+0000","last_became_peered":"2024-09-21T10:09:24.837226+0000","last_unstale":"2024-09-21T10:13:02.779333+0000","last_undegraded":"2024-09-21T10:13:02.779333+0000","last_fullsized":"2024-09-21T10:13:02.779333+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:16.649601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"285'12522","reported_seq":12607,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:06.657176+0000","last_change":"2024-09-21T10:09:49.926338+0000","last_active":"2024-09-21T10:13:06.657176+0000","last_peered":"2024-09-21T10:13:06.657176+0000","last_clean":"2024-09-21T10:13:06.657176+0000","last_became_active":"2024-09-21T10:09:49.926018+0000","last_became_peered":"2024-09-21T10:09:49.926018+0000","last_unstale":"2024-09-21T10:13:06.657176+0000","last_undegraded":"2024-09-21T10:13:06.657176+0000","last_fullsized":"2024-09-21T10:13:06.657176+0000","mapping_epoch":255,"log_start":"272'6700","ondisk_log_start":"272'6700","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5822,"log_dups_size":0,"ondisk_log_size":5822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:09:49.364660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94371840,"num_objects":1440,"num_object_clones":0,"num_object_copies":4320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12522,"num_write_kb":446784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":25,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.779098+0000","last_change":"2024-09-21T10:11:05.443167+0000","last_active":"2024-09-21T10:13:02.779098+0000","last_peered":"2024-09-21T10:13:02.779098+0000","last_clean":"2024-09-21T10:13:02.779098+0000","last_became_active":"2024-09-21T10:11:05.442805+0000","last_became_peered":"2024-09-21T10:11:05.442805+0000","last_unstale":"2024-09-21T10:13:02.779098+0000","last_undegraded":"2024-09-21T10:13:02.779098+0000","last_fullsized":"2024-09-21T10:13:02.779098+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.550551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.550551+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:58:57.998543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":150,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.779588+0000","last_change":"2024-09-21T10:09:23.838944+0000","last_active":"2024-09-21T10:13:02.779588+0000","last_peered":"2024-09-21T10:13:02.779588+0000","last_clean":"2024-09-21T10:13:02.779588+0000","last_became_active":"2024-09-21T10:09:23.837886+0000","last_became_peered":"2024-09-21T10:09:23.837886+0000","last_unstale":"2024-09-21T10:13:02.779588+0000","last_undegraded":"2024-09-21T10:13:02.779588+0000","last_fullsized":"2024-09-21T10:13:02.779588+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:16.928195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":302,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.790667+0000","last_change":"2024-09-21T10:09:24.839770+0000","last_active":"2024-09-21T10:13:03.790667+0000","last_peered":"2024-09-21T10:13:03.790667+0000","last_clean":"2024-09-21T10:13:03.790667+0000","last_became_active":"2024-09-21T10:09:24.839479+0000","last_became_peered":"2024-09-21T10:09:24.839479+0000","last_unstale":"2024-09-21T10:13:03.790667+0000","last_undegraded":"2024-09-21T10:13:03.790667+0000","last_fullsized":"2024-09-21T10:13:03.790667+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:11:19.618193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":71,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.790781+0000","last_change":"2024-09-21T10:09:49.923044+0000","last_active":"2024-09-21T10:13:03.790781+0000","last_peered":"2024-09-21T10:13:03.790781+0000","last_clean":"2024-09-21T10:13:03.790781+0000","last_became_active":"2024-09-21T10:09:49.922711+0000","last_became_peered":"2024-09-21T10:09:49.922711+0000","last_unstale":"2024-09-21T10:13:03.790781+0000","last_undegraded":"2024-09-21T10:13:03.790781+0000","last_fullsized":"2024-09-21T10:13:03.790781+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:14.090084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"285'12779","reported_seq":18278,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:06.700505+0000","last_change":"2024-09-21T10:12:54.765806+0000","last_active":"2024-09-21T10:13:06.700505+0000","last_peered":"2024-09-21T10:13:06.700505+0000","last_clean":"2024-09-21T10:13:06.700505+0000","last_became_active":"2024-09-21T10:12:54.721010+0000","last_became_peered":"2024-09-21T10:12:54.721010+0000","last_unstale":"2024-09-21T10:13:06.700505+0000","last_undegraded":"2024-09-21T10:13:06.700505+0000","last_fullsized":"2024-09-21T10:13:06.700505+0000","mapping_epoch":281,"log_start":"273'7000","ondisk_log_start":"273'7000","created":252,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5779,"log_dups_size":0,"ondisk_log_size":5779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:35.210888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97058816,"num_objects":1481,"num_object_clones":0,"num_object_copies":4443,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12779,"num_write_kb":456320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2697,"num_bytes_recovered":176750592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":190,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.779811+0000","last_change":"2024-09-21T10:09:24.838434+0000","last_active":"2024-09-21T10:13:02.779811+0000","last_peered":"2024-09-21T10:13:02.779811+0000","last_clean":"2024-09-21T10:13:02.779811+0000","last_became_active":"2024-09-21T10:09:24.838066+0000","last_became_peered":"2024-09-21T10:09:24.838066+0000","last_unstale":"2024-09-21T10:13:02.779811+0000","last_undegraded":"2024-09-21T10:13:02.779811+0000","last_fullsized":"2024-09-21T10:13:02.779811+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:52.361137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:52.361137+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:19:57.788195+0000","scrub_duration":0.001255358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.855967+0000","last_change":"2024-09-21T10:09:23.837859+0000","last_active":"2024-09-21T10:13:03.855967+0000","last_peered":"2024-09-21T10:13:03.855967+0000","last_clean":"2024-09-21T10:13:03.855967+0000","last_became_active":"2024-09-21T10:09:23.837488+0000","last_became_peered":"2024-09-21T10:09:23.837488+0000","last_unstale":"2024-09-21T10:13:03.855967+0000","last_undegraded":"2024-09-21T10:13:03.855967+0000","last_fullsized":"2024-09-21T10:13:03.855967+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:25.215501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":34,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.778537+0000","last_change":"2024-09-21T10:10:27.806070+0000","last_active":"2024-09-21T10:13:02.778537+0000","last_peered":"2024-09-21T10:13:02.778537+0000","last_clean":"2024-09-21T10:13:02.778537+0000","last_became_active":"2024-09-21T10:10:27.805718+0000","last_became_peered":"2024-09-21T10:10:27.805718+0000","last_unstale":"2024-09-21T10:13:02.778537+0000","last_undegraded":"2024-09-21T10:13:02.778537+0000","last_fullsized":"2024-09-21T10:13:02.778537+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:31.144118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":457,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.857970+0000","last_change":"2024-09-21T10:10:57.395073+0000","last_active":"2024-09-21T10:13:03.857970+0000","last_peered":"2024-09-21T10:13:03.857970+0000","last_clean":"2024-09-21T10:13:03.857970+0000","last_became_active":"2024-09-21T10:10:57.394674+0000","last_became_peered":"2024-09-21T10:10:57.394674+0000","last_unstale":"2024-09-21T10:13:03.857970+0000","last_undegraded":"2024-09-21T10:13:03.857970+0000","last_fullsized":"2024-09-21T10:13:03.857970+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:48:48.912649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"285'12757","reported_seq":12835,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:06.676468+0000","last_change":"2024-09-21T10:09:43.231554+0000","last_active":"2024-09-21T10:13:06.676468+0000","last_peered":"2024-09-21T10:13:06.676468+0000","last_clean":"2024-09-21T10:13:06.676468+0000","last_became_active":"2024-09-21T10:09:43.231211+0000","last_became_peered":"2024-09-21T10:09:43.231211+0000","last_unstale":"2024-09-21T10:13:06.676468+0000","last_undegraded":"2024-09-21T10:13:06.676468+0000","last_fullsized":"2024-09-21T10:13:06.676468+0000","mapping_epoch":252,"log_start":"272'6900","ondisk_log_start":"272'6900","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5857,"log_dups_size":0,"ondisk_log_size":5857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:16:41.457768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100859904,"num_objects":1539,"num_object_clones":0,"num_object_copies":4617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":457472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":326,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.790785+0000","last_change":"2024-09-21T10:10:57.398437+0000","last_active":"2024-09-21T10:13:03.790785+0000","last_peered":"2024-09-21T10:13:03.790785+0000","last_clean":"2024-09-21T10:13:03.790785+0000","last_became_active":"2024-09-21T10:10:57.397441+0000","last_became_peered":"2024-09-21T10:10:57.397441+0000","last_unstale":"2024-09-21T10:13:03.790785+0000","last_undegraded":"2024-09-21T10:13:03.790785+0000","last_fullsized":"2024-09-21T10:13:03.790785+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:04.043917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:04.043917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:53:31.813612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":89,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.778865+0000","last_change":"2024-09-21T10:08:26.309323+0000","last_active":"2024-09-21T10:13:02.778865+0000","last_peered":"2024-09-21T10:13:02.778865+0000","last_clean":"2024-09-21T10:13:02.778865+0000","last_became_active":"2024-09-21T10:08:26.309004+0000","last_became_peered":"2024-09-21T10:08:26.309004+0000","last_unstale":"2024-09-21T10:13:02.778865+0000","last_undegraded":"2024-09-21T10:13:02.778865+0000","last_fullsized":"2024-09-21T10:13:02.778865+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:37:20.626702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":136,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.791529+0000","last_change":"2024-09-21T10:10:28.180093+0000","last_active":"2024-09-21T10:13:03.791529+0000","last_peered":"2024-09-21T10:13:03.791529+0000","last_clean":"2024-09-21T10:13:03.791529+0000","last_became_active":"2024-09-21T10:10:28.179559+0000","last_became_peered":"2024-09-21T10:10:28.179559+0000","last_unstale":"2024-09-21T10:13:03.791529+0000","last_undegraded":"2024-09-21T10:13:03.791529+0000","last_fullsized":"2024-09-21T10:13:03.791529+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:52:46.146392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":233,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.856442+0000","last_change":"2024-09-21T10:08:26.307041+0000","last_active":"2024-09-21T10:13:03.856442+0000","last_peered":"2024-09-21T10:13:03.856442+0000","last_clean":"2024-09-21T10:13:03.856442+0000","last_became_active":"2024-09-21T10:08:26.306715+0000","last_became_peered":"2024-09-21T10:08:26.306715+0000","last_unstale":"2024-09-21T10:13:03.856442+0000","last_undegraded":"2024-09-21T10:13:03.856442+0000","last_fullsized":"2024-09-21T10:13:03.856442+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:19:26.192398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"285'12911","reported_seq":12990,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:09.248938+0000","last_change":"2024-09-21T10:09:43.232893+0000","last_active":"2024-09-21T10:13:09.248938+0000","last_peered":"2024-09-21T10:13:09.248938+0000","last_clean":"2024-09-21T10:13:09.248938+0000","last_became_active":"2024-09-21T10:09:43.232461+0000","last_became_peered":"2024-09-21T10:09:43.232461+0000","last_unstale":"2024-09-21T10:13:09.248938+0000","last_undegraded":"2024-09-21T10:13:09.248938+0000","last_fullsized":"2024-09-21T10:13:09.248938+0000","mapping_epoch":252,"log_start":"265'5400","ondisk_log_start":"265'5400","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":7511,"log_dups_size":0,"ondisk_log_size":7511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:39.778833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86769688,"num_objects":1325,"num_object_clones":0,"num_object_copies":3975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1325,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12911,"num_write_kb":455489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":259,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.779098+0000","last_change":"2024-09-21T10:04:45.242148+0000","last_active":"2024-09-21T10:13:02.779098+0000","last_peered":"2024-09-21T10:13:02.779098+0000","last_clean":"2024-09-21T10:13:02.779098+0000","last_became_active":"2024-09-21T10:04:42.549844+0000","last_became_peered":"2024-09-21T10:04:42.549844+0000","last_unstale":"2024-09-21T10:13:02.779098+0000","last_undegraded":"2024-09-21T10:13:02.779098+0000","last_fullsized":"2024-09-21T10:13:02.779098+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_deep_scrub":"77'100","last_deep_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.242073+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:03:42.250480+0000","scrub_duration":0.0023466239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":28,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.790388+0000","last_change":"2024-09-21T10:10:57.392880+0000","last_active":"2024-09-21T10:13:02.790388+0000","last_peered":"2024-09-21T10:13:02.790388+0000","last_clean":"2024-09-21T10:13:02.790388+0000","last_became_active":"2024-09-21T10:10:57.392539+0000","last_became_peered":"2024-09-21T10:10:57.392539+0000","last_unstale":"2024-09-21T10:13:02.790388+0000","last_undegraded":"2024-09-21T10:13:02.790388+0000","last_fullsized":"2024-09-21T10:13:02.790388+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:49.754955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":346,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.790754+0000","last_change":"2024-09-21T10:09:23.865342+0000","last_active":"2024-09-21T10:13:03.790754+0000","last_peered":"2024-09-21T10:13:03.790754+0000","last_clean":"2024-09-21T10:13:03.790754+0000","last_became_active":"2024-09-21T10:09:23.865008+0000","last_became_peered":"2024-09-21T10:09:23.865008+0000","last_unstale":"2024-09-21T10:13:03.790754+0000","last_undegraded":"2024-09-21T10:13:03.790754+0000","last_fullsized":"2024-09-21T10:13:03.790754+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:03.356101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":28,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.790148+0000","last_change":"2024-09-21T10:10:57.394707+0000","last_active":"2024-09-21T10:13:02.790148+0000","last_peered":"2024-09-21T10:13:02.790148+0000","last_clean":"2024-09-21T10:13:02.790148+0000","last_became_active":"2024-09-21T10:10:57.394363+0000","last_became_peered":"2024-09-21T10:10:57.394363+0000","last_unstale":"2024-09-21T10:13:02.790148+0000","last_undegraded":"2024-09-21T10:13:02.790148+0000","last_fullsized":"2024-09-21T10:13:02.790148+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:21.141052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"285'12747","reported_seq":12824,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:09.294719+0000","last_change":"2024-09-21T10:09:43.232987+0000","last_active":"2024-09-21T10:13:09.294719+0000","last_peered":"2024-09-21T10:13:09.294719+0000","last_clean":"2024-09-21T10:13:09.294719+0000","last_became_active":"2024-09-21T10:09:43.232577+0000","last_became_peered":"2024-09-21T10:09:43.232577+0000","last_unstale":"2024-09-21T10:13:09.294719+0000","last_undegraded":"2024-09-21T10:13:09.294719+0000","last_fullsized":"2024-09-21T10:13:09.294719+0000","mapping_epoch":252,"log_start":"264'5200","ondisk_log_start":"264'5200","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":7547,"log_dups_size":0,"ondisk_log_size":7547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:14.536774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91422720,"num_objects":1395,"num_object_clones":0,"num_object_copies":4185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12747,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":276,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.778945+0000","last_change":"2024-09-21T10:09:23.839563+0000","last_active":"2024-09-21T10:13:02.778945+0000","last_peered":"2024-09-21T10:13:02.778945+0000","last_clean":"2024-09-21T10:13:02.778945+0000","last_became_active":"2024-09-21T10:09:23.839207+0000","last_became_peered":"2024-09-21T10:09:23.839207+0000","last_unstale":"2024-09-21T10:13:02.778945+0000","last_undegraded":"2024-09-21T10:13:02.778945+0000","last_fullsized":"2024-09-21T10:13:02.778945+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:49.226496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:49.226496+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:03.960125+0000","scrub_duration":0.001656865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":198,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.856291+0000","last_change":"2024-09-21T10:10:57.395167+0000","last_active":"2024-09-21T10:13:03.856291+0000","last_peered":"2024-09-21T10:13:03.856291+0000","last_clean":"2024-09-21T10:13:03.856291+0000","last_became_active":"2024-09-21T10:10:57.394818+0000","last_became_peered":"2024-09-21T10:10:57.394818+0000","last_unstale":"2024-09-21T10:13:03.856291+0000","last_undegraded":"2024-09-21T10:13:03.856291+0000","last_fullsized":"2024-09-21T10:13:03.856291+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:57.224113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":170,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.780188+0000","last_change":"2024-09-21T10:09:49.922782+0000","last_active":"2024-09-21T10:13:02.780188+0000","last_peered":"2024-09-21T10:13:02.780188+0000","last_clean":"2024-09-21T10:13:02.780188+0000","last_became_active":"2024-09-21T10:09:49.922465+0000","last_became_peered":"2024-09-21T10:09:49.922465+0000","last_unstale":"2024-09-21T10:13:02.780188+0000","last_undegraded":"2024-09-21T10:13:02.780188+0000","last_fullsized":"2024-09-21T10:13:02.780188+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:06.151245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":340,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.791606+0000","last_change":"2024-09-21T10:10:57.397251+0000","last_active":"2024-09-21T10:13:03.791606+0000","last_peered":"2024-09-21T10:13:03.791606+0000","last_clean":"2024-09-21T10:13:03.791606+0000","last_became_active":"2024-09-21T10:10:57.396271+0000","last_became_peered":"2024-09-21T10:10:57.396271+0000","last_unstale":"2024-09-21T10:13:03.791606+0000","last_undegraded":"2024-09-21T10:13:03.791606+0000","last_fullsized":"2024-09-21T10:13:03.791606+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:17.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.f","version":"285'12784","reported_seq":12825,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:06.796950+0000","last_change":"2024-09-21T10:09:43.232625+0000","last_active":"2024-09-21T10:13:06.796950+0000","last_peered":"2024-09-21T10:13:06.796950+0000","last_clean":"2024-09-21T10:13:06.796950+0000","last_became_active":"2024-09-21T10:09:43.232331+0000","last_became_peered":"2024-09-21T10:09:43.232331+0000","last_unstale":"2024-09-21T10:13:06.796950+0000","last_undegraded":"2024-09-21T10:13:06.796950+0000","last_fullsized":"2024-09-21T10:13:06.796950+0000","mapping_epoch":252,"log_start":"271'6500","ondisk_log_start":"271'6500","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6284,"log_dups_size":0,"ondisk_log_size":6284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:01:16.634826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97910784,"num_objects":1494,"num_object_clones":0,"num_object_copies":4482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12784,"num_write_kb":456896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":144,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.856174+0000","last_change":"2024-09-21T10:11:05.455170+0000","last_active":"2024-09-21T10:13:03.856174+0000","last_peered":"2024-09-21T10:13:03.856174+0000","last_clean":"2024-09-21T10:13:03.856174+0000","last_became_active":"2024-09-21T10:11:05.454607+0000","last_became_peered":"2024-09-21T10:11:05.454607+0000","last_unstale":"2024-09-21T10:13:03.856174+0000","last_undegraded":"2024-09-21T10:13:03.856174+0000","last_fullsized":"2024-09-21T10:13:03.856174+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:59.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:59.099852+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:14.788975+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":431,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.794963+0000","last_change":"2024-09-21T10:09:23.841186+0000","last_active":"2024-09-21T10:13:03.794963+0000","last_peered":"2024-09-21T10:13:03.794963+0000","last_clean":"2024-09-21T10:13:03.794963+0000","last_became_active":"2024-09-21T10:09:23.840854+0000","last_became_peered":"2024-09-21T10:09:23.840854+0000","last_unstale":"2024-09-21T10:13:03.794963+0000","last_undegraded":"2024-09-21T10:13:03.794963+0000","last_fullsized":"2024-09-21T10:13:03.794963+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:29.666358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":107,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.779014+0000","last_change":"2024-09-21T10:08:38.380867+0000","last_active":"2024-09-21T10:13:02.779014+0000","last_peered":"2024-09-21T10:13:02.779014+0000","last_clean":"2024-09-21T10:13:02.779014+0000","last_became_active":"2024-09-21T10:08:38.379599+0000","last_became_peered":"2024-09-21T10:08:38.379599+0000","last_unstale":"2024-09-21T10:13:02.779014+0000","last_undegraded":"2024-09-21T10:13:02.779014+0000","last_fullsized":"2024-09-21T10:13:02.779014+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:50.499470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":428,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.794121+0000","last_change":"2024-09-21T10:09:49.926548+0000","last_active":"2024-09-21T10:13:03.794121+0000","last_peered":"2024-09-21T10:13:03.794121+0000","last_clean":"2024-09-21T10:13:03.794121+0000","last_became_active":"2024-09-21T10:09:49.926255+0000","last_became_peered":"2024-09-21T10:09:49.926255+0000","last_unstale":"2024-09-21T10:13:03.794121+0000","last_undegraded":"2024-09-21T10:13:03.794121+0000","last_fullsized":"2024-09-21T10:13:03.794121+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:49.855878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"285'12768","reported_seq":12809,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:06.738693+0000","last_change":"2024-09-21T10:09:43.231158+0000","last_active":"2024-09-21T10:13:06.738693+0000","last_peered":"2024-09-21T10:13:06.738693+0000","last_clean":"2024-09-21T10:13:06.738693+0000","last_became_active":"2024-09-21T10:09:43.230838+0000","last_became_peered":"2024-09-21T10:09:43.230838+0000","last_unstale":"2024-09-21T10:13:06.738693+0000","last_undegraded":"2024-09-21T10:13:06.738693+0000","last_fullsized":"2024-09-21T10:13:06.738693+0000","mapping_epoch":252,"log_start":"271'6500","ondisk_log_start":"271'6500","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6268,"log_dups_size":0,"ondisk_log_size":6268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:15.318633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95813632,"num_objects":1462,"num_object_clones":0,"num_object_copies":4386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":455360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":230,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.794670+0000","last_change":"2024-09-21T10:10:57.390804+0000","last_active":"2024-09-21T10:13:03.794670+0000","last_peered":"2024-09-21T10:13:03.794670+0000","last_clean":"2024-09-21T10:13:03.794670+0000","last_became_active":"2024-09-21T10:10:57.390323+0000","last_became_peered":"2024-09-21T10:10:57.390323+0000","last_unstale":"2024-09-21T10:13:03.794670+0000","last_undegraded":"2024-09-21T10:13:03.794670+0000","last_fullsized":"2024-09-21T10:13:03.794670+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.005477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.005477+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:29:57.702046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":75,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.791688+0000","last_change":"2024-09-21T10:09:49.930280+0000","last_active":"2024-09-21T10:13:03.791688+0000","last_peered":"2024-09-21T10:13:03.791688+0000","last_clean":"2024-09-21T10:13:03.791688+0000","last_became_active":"2024-09-21T10:09:49.919101+0000","last_became_peered":"2024-09-21T10:09:49.919101+0000","last_unstale":"2024-09-21T10:13:03.791688+0000","last_undegraded":"2024-09-21T10:13:03.791688+0000","last_fullsized":"2024-09-21T10:13:03.791688+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:23.998475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":298,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.779249+0000","last_change":"2024-09-21T10:03:38.392767+0000","last_active":"2024-09-21T10:13:02.779249+0000","last_peered":"2024-09-21T10:13:02.779249+0000","last_clean":"2024-09-21T10:13:02.779249+0000","last_became_active":"2024-09-21T10:03:38.392424+0000","last_became_peered":"2024-09-21T10:03:38.392424+0000","last_unstale":"2024-09-21T10:13:02.779249+0000","last_undegraded":"2024-09-21T10:13:02.779249+0000","last_fullsized":"2024-09-21T10:13:02.779249+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:25.358291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":79,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.790159+0000","last_change":"2024-09-21T10:09:24.836360+0000","last_active":"2024-09-21T10:13:02.790159+0000","last_peered":"2024-09-21T10:13:02.790159+0000","last_clean":"2024-09-21T10:13:02.790159+0000","last_became_active":"2024-09-21T10:09:24.836033+0000","last_became_peered":"2024-09-21T10:09:24.836033+0000","last_unstale":"2024-09-21T10:13:02.790159+0000","last_undegraded":"2024-09-21T10:13:02.790159+0000","last_fullsized":"2024-09-21T10:13:02.790159+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:58.368582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":136,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.778932+0000","last_change":"2024-09-21T10:08:38.380189+0000","last_active":"2024-09-21T10:13:02.778932+0000","last_peered":"2024-09-21T10:13:02.778932+0000","last_clean":"2024-09-21T10:13:02.778932+0000","last_became_active":"2024-09-21T10:08:38.379891+0000","last_became_peered":"2024-09-21T10:08:38.379891+0000","last_unstale":"2024-09-21T10:13:02.778932+0000","last_undegraded":"2024-09-21T10:13:02.778932+0000","last_fullsized":"2024-09-21T10:13:02.778932+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:35.820921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":216,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.781915+0000","last_change":"2024-09-21T10:08:38.392563+0000","last_active":"2024-09-21T10:13:02.781915+0000","last_peered":"2024-09-21T10:13:02.781915+0000","last_clean":"2024-09-21T10:13:02.781915+0000","last_became_active":"2024-09-21T10:08:38.378735+0000","last_became_peered":"2024-09-21T10:08:38.378735+0000","last_unstale":"2024-09-21T10:13:02.781915+0000","last_undegraded":"2024-09-21T10:13:02.781915+0000","last_fullsized":"2024-09-21T10:13:02.781915+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:29:41.423208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.778779+0000","last_change":"2024-09-21T10:11:05.453286+0000","last_active":"2024-09-21T10:13:02.778779+0000","last_peered":"2024-09-21T10:13:02.778779+0000","last_clean":"2024-09-21T10:13:02.778779+0000","last_became_active":"2024-09-21T10:11:05.452601+0000","last_became_peered":"2024-09-21T10:11:05.452601+0000","last_unstale":"2024-09-21T10:13:02.778779+0000","last_undegraded":"2024-09-21T10:13:02.778779+0000","last_fullsized":"2024-09-21T10:13:02.778779+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:27.532091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":67,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.856406+0000","last_change":"2024-09-21T10:10:27.833762+0000","last_active":"2024-09-21T10:13:03.856406+0000","last_peered":"2024-09-21T10:13:03.856406+0000","last_clean":"2024-09-21T10:13:03.856406+0000","last_became_active":"2024-09-21T10:10:27.819999+0000","last_became_peered":"2024-09-21T10:10:27.819999+0000","last_unstale":"2024-09-21T10:13:03.856406+0000","last_undegraded":"2024-09-21T10:13:03.856406+0000","last_fullsized":"2024-09-21T10:13:03.856406+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:33.839463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'100","reported_seq":222,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.856122+0000","last_change":"2024-09-21T10:07:24.345428+0000","last_active":"2024-09-21T10:13:03.856122+0000","last_peered":"2024-09-21T10:13:03.856122+0000","last_clean":"2024-09-21T10:13:03.856122+0000","last_became_active":"2024-09-21T10:07:24.343529+0000","last_became_peered":"2024-09-21T10:07:24.343529+0000","last_unstale":"2024-09-21T10:13:03.856122+0000","last_undegraded":"2024-09-21T10:13:03.856122+0000","last_fullsized":"2024-09-21T10:13:03.856122+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.014993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.014993+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:35.677888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":28,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.790518+0000","last_change":"2024-09-21T10:10:57.394885+0000","last_active":"2024-09-21T10:13:02.790518+0000","last_peered":"2024-09-21T10:13:02.790518+0000","last_clean":"2024-09-21T10:13:02.790518+0000","last_became_active":"2024-09-21T10:10:57.394584+0000","last_became_peered":"2024-09-21T10:10:57.394584+0000","last_unstale":"2024-09-21T10:13:02.790518+0000","last_undegraded":"2024-09-21T10:13:02.790518+0000","last_fullsized":"2024-09-21T10:13:02.790518+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:52:16.853713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":414,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.794303+0000","last_change":"2024-09-21T10:03:38.425111+0000","last_active":"2024-09-21T10:13:03.794303+0000","last_peered":"2024-09-21T10:13:03.794303+0000","last_clean":"2024-09-21T10:13:03.794303+0000","last_became_active":"2024-09-21T10:03:38.424704+0000","last_became_peered":"2024-09-21T10:03:38.424704+0000","last_unstale":"2024-09-21T10:13:03.794303+0000","last_undegraded":"2024-09-21T10:13:03.794303+0000","last_fullsized":"2024-09-21T10:13:03.794303+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:35.157076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":315,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.790996+0000","last_change":"2024-09-21T10:10:57.397755+0000","last_active":"2024-09-21T10:13:03.790996+0000","last_peered":"2024-09-21T10:13:03.790996+0000","last_clean":"2024-09-21T10:13:03.790996+0000","last_became_active":"2024-09-21T10:10:57.397249+0000","last_became_peered":"2024-09-21T10:10:57.397249+0000","last_unstale":"2024-09-21T10:13:03.790996+0000","last_undegraded":"2024-09-21T10:13:03.790996+0000","last_fullsized":"2024-09-21T10:13:03.790996+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:40.813503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":399,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.794066+0000","last_change":"2024-09-21T09:56:12.576756+0000","last_active":"2024-09-21T10:13:03.794066+0000","last_peered":"2024-09-21T10:13:03.794066+0000","last_clean":"2024-09-21T10:13:03.794066+0000","last_became_active":"2024-09-21T09:56:12.575752+0000","last_became_peered":"2024-09-21T09:56:12.575752+0000","last_unstale":"2024-09-21T10:13:03.794066+0000","last_undegraded":"2024-09-21T10:13:03.794066+0000","last_fullsized":"2024-09-21T10:13:03.794066+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:55.224839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":248,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.779232+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T10:13:02.779232+0000","last_peered":"2024-09-21T10:13:02.779232+0000","last_clean":"2024-09-21T10:13:02.779232+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T10:13:02.779232+0000","last_undegraded":"2024-09-21T10:13:02.779232+0000","last_fullsized":"2024-09-21T10:13:02.779232+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":435,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.794191+0000","last_change":"2024-09-21T10:09:49.928053+0000","last_active":"2024-09-21T10:13:03.794191+0000","last_peered":"2024-09-21T10:13:03.794191+0000","last_clean":"2024-09-21T10:13:03.794191+0000","last_became_active":"2024-09-21T10:09:49.927231+0000","last_became_peered":"2024-09-21T10:09:49.927231+0000","last_unstale":"2024-09-21T10:13:03.794191+0000","last_undegraded":"2024-09-21T10:13:03.794191+0000","last_fullsized":"2024-09-21T10:13:03.794191+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:26.987553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":347,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.791951+0000","last_change":"2024-09-21T10:09:23.840717+0000","last_active":"2024-09-21T10:13:03.791951+0000","last_peered":"2024-09-21T10:13:03.791951+0000","last_clean":"2024-09-21T10:13:03.791951+0000","last_became_active":"2024-09-21T10:09:23.840323+0000","last_became_peered":"2024-09-21T10:09:23.840323+0000","last_unstale":"2024-09-21T10:13:03.791951+0000","last_undegraded":"2024-09-21T10:13:03.791951+0000","last_fullsized":"2024-09-21T10:13:03.791951+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:55.769770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":61,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.855704+0000","last_change":"2024-09-21T10:10:27.816526+0000","last_active":"2024-09-21T10:13:03.855704+0000","last_peered":"2024-09-21T10:13:03.855704+0000","last_clean":"2024-09-21T10:13:03.855704+0000","last_became_active":"2024-09-21T10:10:27.816221+0000","last_became_peered":"2024-09-21T10:10:27.816221+0000","last_unstale":"2024-09-21T10:13:03.855704+0000","last_undegraded":"2024-09-21T10:13:03.855704+0000","last_fullsized":"2024-09-21T10:13:03.855704+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:08.331228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":405,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.794509+0000","last_change":"2024-09-21T10:08:38.379994+0000","last_active":"2024-09-21T10:13:03.794509+0000","last_peered":"2024-09-21T10:13:03.794509+0000","last_clean":"2024-09-21T10:13:03.794509+0000","last_became_active":"2024-09-21T10:08:38.378729+0000","last_became_peered":"2024-09-21T10:08:38.378729+0000","last_unstale":"2024-09-21T10:13:03.794509+0000","last_undegraded":"2024-09-21T10:13:03.794509+0000","last_fullsized":"2024-09-21T10:13:03.794509+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:16.940173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":125,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.856157+0000","last_change":"2024-09-21T10:09:24.844561+0000","last_active":"2024-09-21T10:13:03.856157+0000","last_peered":"2024-09-21T10:13:03.856157+0000","last_clean":"2024-09-21T10:13:03.856157+0000","last_became_active":"2024-09-21T10:09:24.844190+0000","last_became_peered":"2024-09-21T10:09:24.844190+0000","last_unstale":"2024-09-21T10:13:03.856157+0000","last_undegraded":"2024-09-21T10:13:03.856157+0000","last_fullsized":"2024-09-21T10:13:03.856157+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:46:04.775344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":452,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.856369+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:13:03.856369+0000","last_peered":"2024-09-21T10:13:03.856369+0000","last_clean":"2024-09-21T10:13:03.856369+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:13:03.856369+0000","last_undegraded":"2024-09-21T10:13:03.856369+0000","last_fullsized":"2024-09-21T10:13:03.856369+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":71,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.791300+0000","last_change":"2024-09-21T10:09:49.918246+0000","last_active":"2024-09-21T10:13:03.791300+0000","last_peered":"2024-09-21T10:13:03.791300+0000","last_clean":"2024-09-21T10:13:03.791300+0000","last_became_active":"2024-09-21T10:09:49.917911+0000","last_became_peered":"2024-09-21T10:09:49.917911+0000","last_unstale":"2024-09-21T10:13:03.791300+0000","last_undegraded":"2024-09-21T10:13:03.791300+0000","last_fullsized":"2024-09-21T10:13:03.791300+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:30:10.567650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":261,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.794688+0000","last_change":"2024-09-21T10:07:24.347937+0000","last_active":"2024-09-21T10:13:03.794688+0000","last_peered":"2024-09-21T10:13:03.794688+0000","last_clean":"2024-09-21T10:13:03.794688+0000","last_became_active":"2024-09-21T10:07:24.347376+0000","last_became_peered":"2024-09-21T10:07:24.347376+0000","last_unstale":"2024-09-21T10:13:03.794688+0000","last_undegraded":"2024-09-21T10:13:03.794688+0000","last_fullsized":"2024-09-21T10:13:03.794688+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:48:05.644824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":67,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.779223+0000","last_change":"2024-09-21T10:08:38.382518+0000","last_active":"2024-09-21T10:13:02.779223+0000","last_peered":"2024-09-21T10:13:02.779223+0000","last_clean":"2024-09-21T10:13:02.779223+0000","last_became_active":"2024-09-21T10:08:38.372051+0000","last_became_peered":"2024-09-21T10:08:38.372051+0000","last_unstale":"2024-09-21T10:13:02.779223+0000","last_undegraded":"2024-09-21T10:13:02.779223+0000","last_fullsized":"2024-09-21T10:13:02.779223+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:21.360435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":352,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.795296+0000","last_change":"2024-09-21T10:09:24.840173+0000","last_active":"2024-09-21T10:13:03.795296+0000","last_peered":"2024-09-21T10:13:03.795296+0000","last_clean":"2024-09-21T10:13:03.795296+0000","last_became_active":"2024-09-21T10:09:24.839762+0000","last_became_peered":"2024-09-21T10:09:24.839762+0000","last_unstale":"2024-09-21T10:13:03.795296+0000","last_undegraded":"2024-09-21T10:13:03.795296+0000","last_fullsized":"2024-09-21T10:13:03.795296+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:12:53.465203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":81,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.779450+0000","last_change":"2024-09-21T10:08:38.388561+0000","last_active":"2024-09-21T10:13:02.779450+0000","last_peered":"2024-09-21T10:13:02.779450+0000","last_clean":"2024-09-21T10:13:02.779450+0000","last_became_active":"2024-09-21T10:08:38.374558+0000","last_became_peered":"2024-09-21T10:08:38.374558+0000","last_unstale":"2024-09-21T10:13:02.779450+0000","last_undegraded":"2024-09-21T10:13:02.779450+0000","last_fullsized":"2024-09-21T10:13:02.779450+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:24.111438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":436,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.794870+0000","last_change":"2024-09-21T10:09:23.840467+0000","last_active":"2024-09-21T10:13:03.794870+0000","last_peered":"2024-09-21T10:13:03.794870+0000","last_clean":"2024-09-21T10:13:03.794870+0000","last_became_active":"2024-09-21T10:09:23.840177+0000","last_became_peered":"2024-09-21T10:09:23.840177+0000","last_unstale":"2024-09-21T10:13:03.794870+0000","last_undegraded":"2024-09-21T10:13:03.794870+0000","last_fullsized":"2024-09-21T10:13:03.794870+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:17.887307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":425,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.794793+0000","last_change":"2024-09-21T10:11:05.451613+0000","last_active":"2024-09-21T10:13:03.794793+0000","last_peered":"2024-09-21T10:13:03.794793+0000","last_clean":"2024-09-21T10:13:03.794793+0000","last_became_active":"2024-09-21T10:11:05.451075+0000","last_became_peered":"2024-09-21T10:11:05.451075+0000","last_unstale":"2024-09-21T10:13:03.794793+0000","last_undegraded":"2024-09-21T10:13:03.794793+0000","last_fullsized":"2024-09-21T10:13:03.794793+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:18.505045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":117,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-09-21T10:13:02.778779+0000","last_change":"2024-09-21T10:08:38.380508+0000","last_active":"2024-09-21T10:13:02.778779+0000","last_peered":"2024-09-21T10:13:02.778779+0000","last_clean":"2024-09-21T10:13:02.778779+0000","last_became_active":"2024-09-21T10:08:38.380226+0000","last_became_peered":"2024-09-21T10:08:38.380226+0000","last_unstale":"2024-09-21T10:13:02.778779+0000","last_undegraded":"2024-09-21T10:13:02.778779+0000","last_fullsized":"2024-09-21T10:13:02.778779+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:50:38.190118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":90,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-09-21T10:13:03.790807+0000","last_change":"2024-09-21T10:10:57.401588+0000","last_active":"2024-09-21T10:13:03.790807+0000","last_peered":"2024-09-21T10:13:03.790807+0000","last_clean":"2024-09-21T10:13:03.790807+0000","last_became_active":"2024-09-21T10:10:57.397724+0000","last_became_peered":"2024-09-21T10:10:57.397724+0000","last_unstale":"2024-09-21T10:13:03.790807+0000","last_undegraded":"2024-09-21T10:13:03.790807+0000","last_fullsized":"2024-09-21T10:13:03.790807+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:47.843630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":1516765208,"num_objects":23145,"num_object_clones":0,"num_object_copies":69435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":528,"num_objects_unfound":0,"num_objects_dirty":23145,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201825,"num_write_kb":7198977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11091,"num_bytes_recovered":726859776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4573442048,"data_stored":4573429832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102025,"ondisk_log_size":102025,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":11710,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26580,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":688762,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":49,"num_read_kb":125,"num_write":25,"num_write_kb":271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":722165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626849,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1630356,"kb_used_data":725836,"kb_used_omap":1,"kb_used_meta":904510,"kb_avail":92114796,"statfs":{"total":95995035648,"available":94325551104,"internally_reserved":0,"allocated":743256064,"data_stored":741624232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":926218584},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019307,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":245,"seq":1052266987566,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":879492,"kb_used_data":667388,"kb_used_omap":8,"kb_used_meta":212087,"kb_avail":92865660,"statfs":{"total":95995035648,"available":95094435840,"internally_reserved":0,"allocated":683405312,"data_stored":681785683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8940,"internal_metadata":217177364},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":175,"seq":751619276915,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1116308,"kb_used_data":781704,"kb_used_omap":12,"kb_used_meta":334579,"kb_avail":92628844,"statfs":{"total":95995035648,"available":94851936256,"internally_reserved":0,"allocated":800464896,"data_stored":798837139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12845,"internal_metadata":342609363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823078,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1829632,"kb_used_data":335992,"kb_used_omap":1,"kb_used_meta":1493630,"kb_avail":91915520,"statfs":{"total":95995035648,"available":94121492480,"internally_reserved":0,"allocated":344055808,"data_stored":342440138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1529477464},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879501,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1503812,"kb_used_data":686712,"kb_used_omap":4,"kb_used_meta":817083,"kb_avail":92241340,"statfs":{"total":95995035648,"available":94455132160,"internally_reserved":0,"allocated":703193088,"data_stored":701577499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":836693436},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566342,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1253464,"kb_used_data":547088,"kb_used_omap":6,"kb_used_meta":706361,"kb_avail":92491688,"statfs":{"total":95995035648,"available":94711488512,"internally_reserved":0,"allocated":560218112,"data_stored":558583327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":723314087},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860687,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1734336,"kb_used_data":857400,"kb_used_omap":5,"kb_used_meta":876922,"kb_avail":92010816,"statfs":{"total":95995035648,"available":94219075584,"internally_reserved":0,"allocated":877977600,"data_stored":876361119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":897968367},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,1,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7236,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9976,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":670892032,"data_stored":670892032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":837226496,"data_stored":837222424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":531763200,"data_stored":531759128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":677380096,"data_stored":677380096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":338362368,"data_stored":338362368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":736497664,"data_stored":736493592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":781320192,"data_stored":781320192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:13:12.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T10:13:12.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:12.543 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:12.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:12.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:12.694 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:12.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:12.796 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:12.823 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:12.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:12.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:12.924 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:12.952 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:12.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:12.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:13.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:13.089 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:13.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:13.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:13.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:12 smithi107 ceph-mon[102221]: pgmap v1016: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 568 op/s; 528/69516 objects misplaced (0.760%); 8.0 MiB/s, 128 objects/s recovering 2024-09-21T10:13:13.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:12 smithi107 ceph-mon[102221]: osdmap e287: 8 total, 8 up, 7 in 2024-09-21T10:13:13.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:12 smithi107 ceph-mon[105022]: pgmap v1016: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 568 op/s; 528/69516 objects misplaced (0.760%); 8.0 MiB/s, 128 objects/s recovering 2024-09-21T10:13:13.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:12 smithi107 ceph-mon[105022]: osdmap e287: 8 total, 8 up, 7 in 2024-09-21T10:13:13.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:12 smithi129 ceph-mon[88471]: pgmap v1016: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 568 op/s; 528/69516 objects misplaced (0.760%); 8.0 MiB/s, 128 objects/s recovering 2024-09-21T10:13:13.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:12 smithi129 ceph-mon[88471]: osdmap e287: 8 total, 8 up, 7 in 2024-09-21T10:13:13.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:13.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:13.231 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:13.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:13.332 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:13.355 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:13.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:13.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:13.457 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:13.482 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:13.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:13.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:13.583 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:13.596 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:13.613 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:13.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:13.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:13.649 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:13.715 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:13.741 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:13.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:13.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:13.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:13.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:13.896 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:13.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:13.997 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:14.020 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:14.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:14.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:14.122 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:14.158 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:14.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:14.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:14.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:14.301 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:14.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:14.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:14.402 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:14.430 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:14.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:14.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:14.532 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:14.557 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:14.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:14.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:14.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:14.667 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:13:14.668 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:13:14.698 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:14.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:14.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:14.799 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:14.841 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:14.841 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:14 smithi107 ceph-mon[102221]: pgmap v1018: 109 pgs: 1 peering, 108 active+clean; 1.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 880 op/s; 12 MiB/s, 189 objects/s recovering 2024-09-21T10:13:14.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:14.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:14.943 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:14.969 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:14.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:14.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:15.071 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:15.099 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:15.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:15.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:15.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:14 smithi107 ceph-mon[105022]: pgmap v1018: 109 pgs: 1 peering, 108 active+clean; 1.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 880 op/s; 12 MiB/s, 189 objects/s recovering 2024-09-21T10:13:15.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:14 smithi129 ceph-mon[88471]: pgmap v1018: 109 pgs: 1 peering, 108 active+clean; 1.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 880 op/s; 12 MiB/s, 189 objects/s recovering 2024-09-21T10:13:15.202 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:15.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:15.227 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:15.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:15.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:15.366 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:15.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:15.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:15.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:15.503 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:15.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:15.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:15.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:15.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:15.639 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:15.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:15.740 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:15.764 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:15.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:15.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:15.866 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:15.892 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:15.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:15.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:15.994 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:16.020 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:16.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:16.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:16.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:15 smithi107 ceph-mon[102221]: 1.12 scrub starts 2024-09-21T10:13:16.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:15 smithi107 ceph-mon[102221]: 1.12 scrub ok 2024-09-21T10:13:16.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:15 smithi107 ceph-mon[105022]: 1.12 scrub starts 2024-09-21T10:13:16.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:15 smithi107 ceph-mon[105022]: 1.12 scrub ok 2024-09-21T10:13:16.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:16.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:16.162 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:16.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:16.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:15 smithi129 ceph-mon[88471]: 1.12 scrub starts 2024-09-21T10:13:16.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:15 smithi129 ceph-mon[88471]: 1.12 scrub ok 2024-09-21T10:13:16.263 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:16.292 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:16.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:16.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:16.394 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:16.419 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:16.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:16.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:16.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:16.561 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:16.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:16.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:16.663 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:16.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:16.690 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:16.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:16.791 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:16.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:16.824 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:16.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:16.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:16.962 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:16.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:16.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:17.064 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:17.088 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:17.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:17.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:17.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:16 smithi107 ceph-mon[102221]: pgmap v1019: 109 pgs: 1 peering, 108 active+clean; 1.2 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 658 op/s; 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:13:17.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:16 smithi107 ceph-mon[105022]: pgmap v1019: 109 pgs: 1 peering, 108 active+clean; 1.2 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 658 op/s; 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:13:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:16 smithi129 ceph-mon[88471]: pgmap v1019: 109 pgs: 1 peering, 108 active+clean; 1.2 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 658 op/s; 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:13:17.190 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:17.225 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:17.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:17.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:17.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:17.366 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:17.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:17.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:17.468 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:17.494 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:17.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:17.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:17.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:17.636 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:17.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:17.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:17.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:17.776 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:17.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:17.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:17.877 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:17.901 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:17.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:17.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:18.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:18.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:18.036 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:18.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:18.137 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:18.163 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:18.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:18.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:18.266 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:18.291 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:18.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:18.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:18.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:18.435 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:18.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:18.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:18.537 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:18.564 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:18.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:18.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:18.665 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:18.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:18.691 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:18.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:18.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:18.819 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:18.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:18.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:18.861 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:13:18.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:18.958 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:18.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:18.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:19.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:19.099 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:19.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:19.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:19.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:18 smithi129 ceph-mon[88471]: pgmap v1020: 109 pgs: 109 active+clean; 997 MiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 900 op/s; 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:13:19.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:18 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4004015820' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:19.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:18 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:19.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:19.237 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:19.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:19.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:19.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:18 smithi107 ceph-mon[102221]: pgmap v1020: 109 pgs: 109 active+clean; 997 MiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 900 op/s; 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:13:19.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:18 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4004015820' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:19.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:18 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:19.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:18 smithi107 ceph-mon[105022]: pgmap v1020: 109 pgs: 109 active+clean; 997 MiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 900 op/s; 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:13:19.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:18 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4004015820' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:19.324 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:18 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:19.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:19.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:19.376 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:19.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:19.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:19.516 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:19.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:19.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:19.618 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:19.642 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:19.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:19.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:19.744 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:19.771 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:19.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:19.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:19.873 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:19.914 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:19.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:19.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:20.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:20.053 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:20.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:20.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:20.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:19 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:13:20.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:19 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:13:20.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:19 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:13:20.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:19 smithi107 ceph-mon[102221]: osdmap e288: 8 total, 8 up, 7 in 2024-09-21T10:13:20.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:19 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:13:20.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:19 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:13:20.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:19 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:13:20.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:19 smithi107 ceph-mon[105022]: osdmap e288: 8 total, 8 up, 7 in 2024-09-21T10:13:20.154 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:20.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:19 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:13:20.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:19 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:13:20.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:19 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:13:20.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:19 smithi129 ceph-mon[88471]: osdmap e288: 8 total, 8 up, 7 in 2024-09-21T10:13:20.178 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:20.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:20.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:20.280 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:20.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:20.312 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:20.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:20.413 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:20.451 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:20.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:20.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:20.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:20.592 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:20.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:20.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:13:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:13:20.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:13:20.694 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:20.721 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:20.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:20.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:20.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:20.864 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:20.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:20.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:20.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:21.004 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:21.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:21.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:21.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:21.141 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:21.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:21.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:21.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:20 smithi129 ceph-mon[88471]: pgmap v1022: 109 pgs: 109 active+clean; 957 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 977 op/s; 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:13:21.192 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:20 smithi107 ceph-mon[102221]: pgmap v1022: 109 pgs: 109 active+clean; 957 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 977 op/s; 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:13:21.193 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:20 smithi107 ceph-mon[105022]: pgmap v1022: 109 pgs: 109 active+clean; 957 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 977 op/s; 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:13:21.243 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:21.279 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:21.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:21.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:21.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:21.417 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:21.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:21.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:21.519 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:21.545 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:21.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:21.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:21.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:21.691 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:21.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:21.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:21.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:21.827 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:21.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:21.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:21.929 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:21.930 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:13:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:13:21] "GET /metrics HTTP/1.1" 200 36810 "" "Prometheus/2.51.0" 2024-09-21T10:13:21.969 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:21.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:21.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:22.027 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:13:22.027 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:13:22.069 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:13:22.069 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:13:22.071 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:22.097 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:22.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:22.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:22.114 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:13:22.114 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:13:22.156 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:13:22.156 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:13:22.187 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:13:22.187 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:13:22.196 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:13:22.197 INFO:teuthology.orchestra.run.smithi107.stdout:{"fsid":"86c49514-77fc-11ef-baf6-efdc52797490","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1560,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":288,"num_osds":8,"num_up_osds":8,"osd_up_since":1726913362,"num_in_osds":7,"osd_in_since":1726913317,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":109}],"num_pgs":109,"num_pools":5,"num_objects":14417,"data_bytes":943730540,"bytes_used":8421187584,"bytes_avail":663544061952,"bytes_total":671965249536,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":791},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi107.sedied","status":"up:active","gid":24557}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.107:8443/","prometheus":"http://172.21.15.107:9283/"}},"servicemap":{"epoch":30,"modified":"2024-09-21T10:10:23.246058+0000","services":{}},"progress_events":{"24983b40-6a19-4e8e-8a8a-55bfd2201591":{"message":"Global Recovery Event (2m)\n [===========================.] (remaining: 1s)","progress":0.99082571268081665,"add_to_ceph_s":true},"2fcfd131-0268-49fe-9638-9f737ef274d3":{"message":"Upgrade to 19.3.0-5092-g62da54b4 (9m)\n [============................] (remaining: 11m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-09-21T10:13:22.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:22.227 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:13:22.228 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:13:22.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:22.247 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:22.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:22.260 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:13:22.260 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:13:22.290 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:13:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:13:22.349 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:22.373 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:22.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:22.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:22.475 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:22.506 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:22.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:22.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:22.608 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:22.640 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:22.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:22.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:22.742 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:22.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:13:22.771 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:22.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:22.873 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:22.929 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:22.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:22.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:23.032 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:23.056 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:23.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:23.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:13:23.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:22 smithi107 ceph-mon[102221]: pgmap v1023: 109 pgs: 109 active+clean; 900 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 791 op/s 2024-09-21T10:13:23.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:22 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/575525446' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:23.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:22 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/251849275' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:13:23.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:22 smithi107 ceph-mon[105022]: pgmap v1023: 109 pgs: 109 active+clean; 900 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 791 op/s 2024-09-21T10:13:23.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:22 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/575525446' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:23.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:22 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/251849275' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:13:23.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:23.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:22 smithi129 ceph-mon[88471]: pgmap v1023: 109 pgs: 109 active+clean; 900 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 791 op/s 2024-09-21T10:13:23.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:22 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/575525446' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:23.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:22 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/251849275' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:13:23.220 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:23.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:23.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:23.322 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:23.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:23.367 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:23.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:23.468 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:23.496 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:23.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:23.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:23.598 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:23.625 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:23.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:23.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:23.726 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:23.752 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:23.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:23.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:23.854 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:23.879 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:23.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:23.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:23.981 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:24.012 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:24.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:24.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:24.113 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:24.146 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:24.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:24.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:24.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:24.292 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:24.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:24.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:24.394 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:24.434 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:24.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:24.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:24.536 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:24.560 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:24.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:24.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:24.662 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:24.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:24.689 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:24.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:24.768 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-21T10:13:24.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:13:24.791 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:24.845 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:24.874 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:24.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:24.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:24.884 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:24.976 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:24.999 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:25.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:25.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:25.000 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:24 smithi107 ceph-mon[102221]: pgmap v1024: 109 pgs: 109 active+clean; 663 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 929 op/s 2024-09-21T10:13:25.001 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:24 smithi107 ceph-mon[105022]: pgmap v1024: 109 pgs: 109 active+clean; 663 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 929 op/s 2024-09-21T10:13:25.101 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:25.125 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:25.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:25.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:24 smithi129 ceph-mon[88471]: pgmap v1024: 109 pgs: 109 active+clean; 663 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 929 op/s 2024-09-21T10:13:25.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:13:25.227 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:25.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:25.319 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:25.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:25.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:25.470 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:25.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:25.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:25.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:25.619 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:25.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:25.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:25.722 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:25.745 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:25.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:25.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:25.848 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:25.874 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:25.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:25.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:25.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:26.007 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:26.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:26.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:26.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:26.157 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:26.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:26.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:26.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:26.293 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:26.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:26.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:26.395 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:26.417 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:26.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:26.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:26.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:26.567 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:26.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:26.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:26.669 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:26.696 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:26.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:26.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:26.798 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:26.847 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:26.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:26.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:26.948 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:26.974 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:26.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:26.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:27.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:27.112 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:26 smithi107 ceph-mon[102221]: pgmap v1025: 109 pgs: 109 active+clean; 608 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 942 op/s 2024-09-21T10:13:27.112 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:27.112 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:27.112 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:27.112 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]} : dispatch 2024-09-21T10:13:27.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:26 smithi107 ceph-mon[105022]: pgmap v1025: 109 pgs: 109 active+clean; 608 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 942 op/s 2024-09-21T10:13:27.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:27.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:27.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:27.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]} : dispatch 2024-09-21T10:13:27.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:27.120 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:27.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:26 smithi129 ceph-mon[88471]: pgmap v1025: 109 pgs: 109 active+clean; 608 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 942 op/s 2024-09-21T10:13:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]} : dispatch 2024-09-21T10:13:27.220 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:27.247 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:27.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:27.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:27.349 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:27.375 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:27.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:27.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:27.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:27.491 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:27.538 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:27.569 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:27.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:27.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:27.671 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:27.696 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:27.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:27.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:27.798 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:27.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:27.850 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:27.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:27.951 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:27.985 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:27.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:27.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:28.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:28.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:28.137 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:28.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:28.238 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:28.261 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:28.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:28.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:28.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:28.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:28.408 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:28.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:28.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:28.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:28.570 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:28.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:28.607 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:28 smithi107 ceph-mon[102221]: pgmap v1026: 109 pgs: 109 active+clean; 416 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 927 op/s 2024-09-21T10:13:28.607 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:28 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]': finished 2024-09-21T10:13:28.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:28 smithi107 ceph-mon[102221]: osdmap e289: 8 total, 8 up, 7 in 2024-09-21T10:13:28.608 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:28 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:13:28.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:28 smithi107 ceph-mon[105022]: pgmap v1026: 109 pgs: 109 active+clean; 416 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 927 op/s 2024-09-21T10:13:28.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:28 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]': finished 2024-09-21T10:13:28.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:28 smithi107 ceph-mon[105022]: osdmap e289: 8 total, 8 up, 7 in 2024-09-21T10:13:28.608 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:28 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:13:28.609 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:13:28.609 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:13:28.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.609 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:220475 11.8 11:10910921:::benchmark_data_smithi107_218812_object107988:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-21T10:13:28.609 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:28.312702+0000", 2024-09-21T10:13:28.609 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.019751722999999999, 2024-09-21T10:13:28.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019781951999999998, 2024-09-21T10:13:28.609 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:28.609 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:28.610 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:13:28.610 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:28.610 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:28.610 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:28.610 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 220475 2024-09-21T10:13:28.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.610 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:28.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:28.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.312702+0000", 2024-09-21T10:13:28.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:28.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.312702+0000", 2024-09-21T10:13:28.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:28.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.312703+0000", 2024-09-21T10:13:28.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8700000000000005e-07 2024-09-21T10:13:28.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:28.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.312707+0000", 2024-09-21T10:13:28.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0799999999999999e-06 2024-09-21T10:13:28.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:28.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.312717+0000", 2024-09-21T10:13:28.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0275000000000001e-05 2024-09-21T10:13:28.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:28.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.312724+0000", 2024-09-21T10:13:28.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.985e-06 2024-09-21T10:13:28.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:28.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.316989+0000", 2024-09-21T10:13:28.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004265044 2024-09-21T10:13:28.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:28.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.317088+0000", 2024-09-21T10:13:28.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8988999999999997e-05 2024-09-21T10:13:28.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:13:28.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.317250+0000", 2024-09-21T10:13:28.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016181100000000001 2024-09-21T10:13:28.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:28.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.327074+0000", 2024-09-21T10:13:28.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098246940000000001 2024-09-21T10:13:28.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:28.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.327205+0000", 2024-09-21T10:13:28.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013086200000000001 2024-09-21T10:13:28.614 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.614 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:28.614 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.614 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:220476 11.8 11:172bd52d:::benchmark_data_smithi107_218812_object107989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-21T10:13:28.614 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:28.312736+0000", 2024-09-21T10:13:28.614 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.019716984999999999, 2024-09-21T10:13:28.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019821333, 2024-09-21T10:13:28.615 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:28.615 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:28.615 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:13:28.615 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:28.615 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:28.615 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:28.615 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 220476 2024-09-21T10:13:28.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.615 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:28.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:28.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.312736+0000", 2024-09-21T10:13:28.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:28.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.312736+0000", 2024-09-21T10:13:28.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:28.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.312737+0000", 2024-09-21T10:13:28.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4799999999999998e-07 2024-09-21T10:13:28.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:28.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.312740+0000", 2024-09-21T10:13:28.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.554e-06 2024-09-21T10:13:28.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:28.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.312747+0000", 2024-09-21T10:13:28.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0520000000000004e-06 2024-09-21T10:13:28.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:28.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.312761+0000", 2024-09-21T10:13:28.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3944999999999999e-05 2024-09-21T10:13:28.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:28.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.320966+0000", 2024-09-21T10:13:28.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0082052219999999999 2024-09-21T10:13:28.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:28.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.321064+0000", 2024-09-21T10:13:28.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8270999999999995e-05 2024-09-21T10:13:28.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:13:28.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.321162+0000", 2024-09-21T10:13:28.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7474999999999998e-05 2024-09-21T10:13:28.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:28.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.327269+0000", 2024-09-21T10:13:28.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0061072069999999999 2024-09-21T10:13:28.619 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.619 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:28.619 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.619 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:220478 11.0 11:0b61182e:::benchmark_data_smithi107_218812_object107991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-21T10:13:28.619 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:28.314710+0000", 2024-09-21T10:13:28.619 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.017743524, 2024-09-21T10:13:28.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017911624000000001, 2024-09-21T10:13:28.619 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:28.619 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:28.619 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:13:28.619 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:28.620 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:28.620 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:28.620 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 220478 2024-09-21T10:13:28.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.620 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:28.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:28.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314710+0000", 2024-09-21T10:13:28.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:28.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314710+0000", 2024-09-21T10:13:28.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:28.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314711+0000", 2024-09-21T10:13:28.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.02e-06 2024-09-21T10:13:28.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:28.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314716+0000", 2024-09-21T10:13:28.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7299999999999996e-06 2024-09-21T10:13:28.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:28.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314728+0000", 2024-09-21T10:13:28.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.187e-05 2024-09-21T10:13:28.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:28.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314735+0000", 2024-09-21T10:13:28.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9489999999999999e-06 2024-09-21T10:13:28.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:28.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.320564+0000", 2024-09-21T10:13:28.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0058291740000000003 2024-09-21T10:13:28.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:28.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.320658+0000", 2024-09-21T10:13:28.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3845999999999996e-05 2024-09-21T10:13:28.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,7", 2024-09-21T10:13:28.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.320752+0000", 2024-09-21T10:13:28.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4475999999999995e-05 2024-09-21T10:13:28.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:28.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.327848+0000", 2024-09-21T10:13:28.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0070954939999999999 2024-09-21T10:13:28.623 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.623 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:28.623 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.624 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:220479 11.b 11:dbb29c19:::benchmark_data_smithi107_218812_object107992:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-21T10:13:28.624 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:28.314747+0000", 2024-09-21T10:13:28.624 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.017706815000000001, 2024-09-21T10:13:28.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017936802000000002, 2024-09-21T10:13:28.624 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:28.624 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:28.624 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:13:28.624 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:28.624 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:28.624 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:28.624 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 220479 2024-09-21T10:13:28.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.624 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:28.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:28.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314747+0000", 2024-09-21T10:13:28.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:28.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314747+0000", 2024-09-21T10:13:28.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:28.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314747+0000", 2024-09-21T10:13:28.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8800000000000003e-07 2024-09-21T10:13:28.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.626 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:28.626 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314749+0000", 2024-09-21T10:13:28.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2110000000000001e-06 2024-09-21T10:13:28.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.626 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:28.626 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314755+0000", 2024-09-21T10:13:28.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6910000000000002e-06 2024-09-21T10:13:28.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.626 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:28.626 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314760+0000", 2024-09-21T10:13:28.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9010000000000002e-06 2024-09-21T10:13:28.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:28.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.315563+0000", 2024-09-21T10:13:28.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00080285899999999995 2024-09-21T10:13:28.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:28.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.315683+0000", 2024-09-21T10:13:28.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000120587 2024-09-21T10:13:28.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:13:28.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.315802+0000", 2024-09-21T10:13:28.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000118662 2024-09-21T10:13:28.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:28.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.320243+0000", 2024-09-21T10:13:28.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044412130000000003 2024-09-21T10:13:28.628 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.628 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:28.628 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.628 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:220481 11.0 11:0a57615a:::benchmark_data_smithi107_218812_object107994:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-21T10:13:28.628 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:28.314772+0000", 2024-09-21T10:13:28.628 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.017681375999999999, 2024-09-21T10:13:28.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017971593000000001, 2024-09-21T10:13:28.629 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:28.629 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:28.629 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:13:28.629 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:28.629 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:28.629 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:28.629 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 220481 2024-09-21T10:13:28.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.629 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:28.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:28.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314772+0000", 2024-09-21T10:13:28.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:28.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314772+0000", 2024-09-21T10:13:28.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:28.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314772+0000", 2024-09-21T10:13:28.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.96e-07 2024-09-21T10:13:28.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:28.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314774+0000", 2024-09-21T10:13:28.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.787e-06 2024-09-21T10:13:28.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:28.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314780+0000", 2024-09-21T10:13:28.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3929999999999999e-06 2024-09-21T10:13:28.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:28.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314784+0000", 2024-09-21T10:13:28.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3869999999999997e-06 2024-09-21T10:13:28.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:28.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.327878+0000", 2024-09-21T10:13:28.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013093642000000001 2024-09-21T10:13:28.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:28.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.327976+0000", 2024-09-21T10:13:28.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8018999999999998e-05 2024-09-21T10:13:28.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,7", 2024-09-21T10:13:28.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328079+0000", 2024-09-21T10:13:28.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103458 2024-09-21T10:13:28.632 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.632 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:28.632 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.632 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:220482 11.0 11.78e24ec0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-21T10:13:28.632 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:28.314789+0000", 2024-09-21T10:13:28.632 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.017664097, 2024-09-21T10:13:28.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018008356, 2024-09-21T10:13:28.633 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:28.633 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:28.633 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2024-09-21T10:13:28.633 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:28.633 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:28.633 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:28.633 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 220482 2024-09-21T10:13:28.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.633 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:28.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.633 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:28.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314789+0000", 2024-09-21T10:13:28.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.633 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:28.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314789+0000", 2024-09-21T10:13:28.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:28.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314790+0000", 2024-09-21T10:13:28.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.22e-07 2024-09-21T10:13:28.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:28.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314791+0000", 2024-09-21T10:13:28.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.53e-06 2024-09-21T10:13:28.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:28.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314796+0000", 2024-09-21T10:13:28.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9350000000000002e-06 2024-09-21T10:13:28.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:28.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.314800+0000", 2024-09-21T10:13:28.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0090000000000001e-06 2024-09-21T10:13:28.635 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.635 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:28.635 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.635 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:220489 11.9 11:9658b0e1:::benchmark_data_smithi107_218812_object108002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-21T10:13:28.635 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:28.319887+0000", 2024-09-21T10:13:28.636 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.012566417, 2024-09-21T10:13:28.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012962519, 2024-09-21T10:13:28.636 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:28.636 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:28.636 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:13:28.636 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:28.636 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:28.636 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:28.636 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 220489 2024-09-21T10:13:28.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.636 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:28.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:28.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.319887+0000", 2024-09-21T10:13:28.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:28.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.319887+0000", 2024-09-21T10:13:28.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:28.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.319888+0000", 2024-09-21T10:13:28.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.026e-06 2024-09-21T10:13:28.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:28.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.319893+0000", 2024-09-21T10:13:28.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4959999999999998e-06 2024-09-21T10:13:28.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:28.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.319907+0000", 2024-09-21T10:13:28.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4737e-05 2024-09-21T10:13:28.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:28.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.319915+0000", 2024-09-21T10:13:28.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9310000000000008e-06 2024-09-21T10:13:28.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:28.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.325676+0000", 2024-09-21T10:13:28.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0057604170000000003 2024-09-21T10:13:28.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:28.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.325778+0000", 2024-09-21T10:13:28.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101885 2024-09-21T10:13:28.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:13:28.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.325889+0000", 2024-09-21T10:13:28.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011184 2024-09-21T10:13:28.639 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.639 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:28.639 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.639 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:220490 11.a 11:5b5ddca6:::benchmark_data_smithi107_218812_object108003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-21T10:13:28.640 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:28.323795+0000", 2024-09-21T10:13:28.640 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0086584269999999998, 2024-09-21T10:13:28.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0091105739999999998, 2024-09-21T10:13:28.640 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:28.640 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:28.640 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:13:28.640 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:28.640 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:28.640 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:28.640 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 220490 2024-09-21T10:13:28.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.640 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:28.640 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.640 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:28.640 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.323795+0000", 2024-09-21T10:13:28.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:28.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.323795+0000", 2024-09-21T10:13:28.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:28.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.323797+0000", 2024-09-21T10:13:28.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.796e-06 2024-09-21T10:13:28.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:28.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.323803+0000", 2024-09-21T10:13:28.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1500000000000004e-06 2024-09-21T10:13:28.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:28.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.323823+0000", 2024-09-21T10:13:28.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9675000000000001e-05 2024-09-21T10:13:28.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:28.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.323830+0000", 2024-09-21T10:13:28.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7610000000000007e-06 2024-09-21T10:13:28.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:28.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.323893+0000", 2024-09-21T10:13:28.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2959999999999994e-05 2024-09-21T10:13:28.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:28.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.324709+0000", 2024-09-21T10:13:28.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00081580599999999997 2024-09-21T10:13:28.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:13:28.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.324808+0000", 2024-09-21T10:13:28.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9073999999999994e-05 2024-09-21T10:13:28.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:28.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328864+0000", 2024-09-21T10:13:28.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0040561599999999996 2024-09-21T10:13:28.643 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.643 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:28.643 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.644 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.36561.0:220475 11.8 e289/281)", 2024-09-21T10:13:28.644 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:28.328630+0000", 2024-09-21T10:13:28.644 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0038232040000000002, 2024-09-21T10:13:28.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043308870000000003, 2024-09-21T10:13:28.644 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:28.644 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:28.644 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2024-09-21T10:13:28.644 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:28.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.644 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:28.644 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328630+0000", 2024-09-21T10:13:28.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:28.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328630+0000", 2024-09-21T10:13:28.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:28.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328631+0000", 2024-09-21T10:13:28.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2900000000000003e-07 2024-09-21T10:13:28.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:28.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328634+0000", 2024-09-21T10:13:28.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4740000000000001e-06 2024-09-21T10:13:28.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:28.646 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328644+0000", 2024-09-21T10:13:28.646 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8879999999999992e-06 2024-09-21T10:13:28.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:28.646 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328650+0000", 2024-09-21T10:13:28.646 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8810000000000001e-06 2024-09-21T10:13:28.646 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.646 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:28.646 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.647 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.36561.0:220476 11.8 e289/281)", 2024-09-21T10:13:28.647 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:28.328662+0000", 2024-09-21T10:13:28.647 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0037919529999999998, 2024-09-21T10:13:28.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043329919999999999, 2024-09-21T10:13:28.647 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:28.647 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:28.647 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2024-09-21T10:13:28.647 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:28.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:28.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328662+0000", 2024-09-21T10:13:28.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:28.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328662+0000", 2024-09-21T10:13:28.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:28.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328662+0000", 2024-09-21T10:13:28.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8799999999999998e-07 2024-09-21T10:13:28.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:28.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328664+0000", 2024-09-21T10:13:28.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7889999999999999e-06 2024-09-21T10:13:28.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:28.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328669+0000", 2024-09-21T10:13:28.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1900000000000003e-06 2024-09-21T10:13:28.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:28.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328673+0000", 2024-09-21T10:13:28.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9589999999999997e-06 2024-09-21T10:13:28.649 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.649 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:28.649 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.649 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.36561.0:220476 11.8 e289/281)", 2024-09-21T10:13:28.649 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:28.328661+0000", 2024-09-21T10:13:28.649 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0037921809999999999, 2024-09-21T10:13:28.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043666240000000004, 2024-09-21T10:13:28.649 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:28.649 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:28.650 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2024-09-21T10:13:28.650 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:28.650 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.650 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:28.650 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328661+0000", 2024-09-21T10:13:28.650 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.650 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.650 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:28.650 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328661+0000", 2024-09-21T10:13:28.650 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.650 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.650 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:28.650 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328662+0000", 2024-09-21T10:13:28.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3700000000000001e-07 2024-09-21T10:13:28.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:28.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328664+0000", 2024-09-21T10:13:28.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.232e-06 2024-09-21T10:13:28.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:28.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328670+0000", 2024-09-21T10:13:28.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5339999999999999e-06 2024-09-21T10:13:28.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.651 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:28.651 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328674+0000", 2024-09-21T10:13:28.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5340000000000001e-06 2024-09-21T10:13:28.652 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.652 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:28.652 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.652 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:220493 11.8 11.f2887c58 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-21T10:13:28.652 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:28.328707+0000", 2024-09-21T10:13:28.652 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0037464780000000001, 2024-09-21T10:13:28.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043565399999999999, 2024-09-21T10:13:28.652 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:28.652 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:28.652 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2024-09-21T10:13:28.652 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:28.653 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:28.653 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:28.653 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 220493 2024-09-21T10:13:28.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.653 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:28.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:28.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328707+0000", 2024-09-21T10:13:28.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:28.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328707+0000", 2024-09-21T10:13:28.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:28.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328708+0000", 2024-09-21T10:13:28.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9099999999999991e-07 2024-09-21T10:13:28.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:28.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328711+0000", 2024-09-21T10:13:28.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2619999999999999e-06 2024-09-21T10:13:28.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:28.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328719+0000", 2024-09-21T10:13:28.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8690000000000004e-06 2024-09-21T10:13:28.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:28.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328724+0000", 2024-09-21T10:13:28.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3120000000000001e-06 2024-09-21T10:13:28.655 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.655 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:28.655 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.655 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:220494 11.a 11:532b483c:::benchmark_data_smithi107_218812_object108007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-09-21T10:13:28.655 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:28.328735+0000", 2024-09-21T10:13:28.655 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.003718247, 2024-09-21T10:13:28.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043769890000000004, 2024-09-21T10:13:28.655 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:28.656 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:28.656 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:13:28.656 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:28.656 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:28.656 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:28.656 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 220494 2024-09-21T10:13:28.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.656 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:28.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.656 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:28.656 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328735+0000", 2024-09-21T10:13:28.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.656 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:28.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328735+0000", 2024-09-21T10:13:28.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:28.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328736+0000", 2024-09-21T10:13:28.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7099999999999997e-07 2024-09-21T10:13:28.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:28.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328737+0000", 2024-09-21T10:13:28.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.798e-06 2024-09-21T10:13:28.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.658 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:28.658 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328743+0000", 2024-09-21T10:13:28.658 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5820000000000001e-06 2024-09-21T10:13:28.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.658 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:28.658 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328748+0000", 2024-09-21T10:13:28.658 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9030000000000003e-06 2024-09-21T10:13:28.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.658 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:28.658 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328921+0000", 2024-09-21T10:13:28.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000173427 2024-09-21T10:13:28.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.659 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:28.659 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.329058+0000", 2024-09-21T10:13:28.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000136639 2024-09-21T10:13:28.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.659 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:13:28.659 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.329201+0000", 2024-09-21T10:13:28.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014273199999999999 2024-09-21T10:13:28.659 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.659 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:28.659 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.660 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.36561.0:220479 11.b e289/252)", 2024-09-21T10:13:28.660 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:28.328750+0000", 2024-09-21T10:13:28.660 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.003703212, 2024-09-21T10:13:28.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044175799999999999, 2024-09-21T10:13:28.660 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:28.660 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:28.660 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2024-09-21T10:13:28.660 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:28.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.660 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:28.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328750+0000", 2024-09-21T10:13:28.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:28.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328750+0000", 2024-09-21T10:13:28.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:28.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328750+0000", 2024-09-21T10:13:28.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.37e-07 2024-09-21T10:13:28.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:28.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328752+0000", 2024-09-21T10:13:28.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8619999999999999e-06 2024-09-21T10:13:28.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.662 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:28.662 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328757+0000", 2024-09-21T10:13:28.662 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5639999999999998e-06 2024-09-21T10:13:28.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.662 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:28.662 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328760+0000", 2024-09-21T10:13:28.662 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4309999999999998e-06 2024-09-21T10:13:28.662 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.662 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:28.662 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.662 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.662 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.36561.0:220479 11.b e289/252)", 2024-09-21T10:13:28.663 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:28.328782+0000", 2024-09-21T10:13:28.663 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.003671835, 2024-09-21T10:13:28.663 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004449504, 2024-09-21T10:13:28.663 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:28.663 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:28.663 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2024-09-21T10:13:28.663 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:28.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.663 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:28.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328782+0000", 2024-09-21T10:13:28.663 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.663 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:28.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328782+0000", 2024-09-21T10:13:28.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.664 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:28.664 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328782+0000", 2024-09-21T10:13:28.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3699999999999999e-07 2024-09-21T10:13:28.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.664 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:28.664 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328784+0000", 2024-09-21T10:13:28.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1849999999999998e-06 2024-09-21T10:13:28.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.664 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:28.664 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328790+0000", 2024-09-21T10:13:28.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1380000000000004e-06 2024-09-21T10:13:28.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:28.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328794+0000", 2024-09-21T10:13:28.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0579999999999999e-06 2024-09-21T10:13:28.665 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.665 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:28.665 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.665 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.36561.0:220478 11.0 e289/252)", 2024-09-21T10:13:28.665 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:28.328800+0000", 2024-09-21T10:13:28.665 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0036531480000000002, 2024-09-21T10:13:28.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044655399999999996, 2024-09-21T10:13:28.666 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:28.666 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:28.666 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2024-09-21T10:13:28.666 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:28.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:28.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328800+0000", 2024-09-21T10:13:28.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:28.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328800+0000", 2024-09-21T10:13:28.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:28.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.667 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:28.667 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328801+0000", 2024-09-21T10:13:28.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.92e-07 2024-09-21T10:13:28.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.667 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:28.667 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328802+0000", 2024-09-21T10:13:28.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.55e-06 2024-09-21T10:13:28.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.667 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:28.667 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328807+0000", 2024-09-21T10:13:28.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7530000000000001e-06 2024-09-21T10:13:28.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:28.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:28.668 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:28.668 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:28.328810+0000", 2024-09-21T10:13:28.668 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3749999999999999e-06 2024-09-21T10:13:28.668 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.668 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:28.668 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.668 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:28.668 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-09-21T10:13:28.668 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 16 2024-09-21T10:13:28.668 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:13:28.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:28 smithi129 ceph-mon[88471]: pgmap v1026: 109 pgs: 109 active+clean; 416 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 927 op/s 2024-09-21T10:13:28.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:28 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]': finished 2024-09-21T10:13:28.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:28 smithi129 ceph-mon[88471]: osdmap e289: 8 total, 8 up, 7 in 2024-09-21T10:13:28.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:28 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:13:28.671 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:28.698 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:28.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:28.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:28.799 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:28.832 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:28.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:28.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:28.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:28.968 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:28.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:28.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:29.070 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:29.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:29.102 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:29.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:29.202 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:29.239 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:29.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:29.341 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:29.379 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:29.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:29.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:29.480 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:29.506 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:29.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:29.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:29.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:29.650 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:29.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:29.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:29.751 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:29.781 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:29.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:29.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:29.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:29 smithi107 ceph-mon[102221]: osdmap e290: 8 total, 8 up, 7 in 2024-09-21T10:13:29.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:29 smithi107 ceph-mon[105022]: osdmap e290: 8 total, 8 up, 7 in 2024-09-21T10:13:29.882 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:29.893 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:13:29.914 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:29.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:29.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:29 smithi129 ceph-mon[88471]: osdmap e290: 8 total, 8 up, 7 in 2024-09-21T10:13:30.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:30.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:30.091 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:30.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:30.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:30.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:30.239 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:30.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:30.340 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:30.370 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:30.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:30.472 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:30.496 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:13:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:13:30.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:13:30.509 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:30.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:30.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:30.573 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:13:30.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:30.649 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:30.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:30.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:30.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:30.792 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:30.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:30.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:30.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:30 smithi107 ceph-mon[102221]: pgmap v1029: 109 pgs: 1 peering, 108 active+clean; 381 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-21T10:13:30.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:30 smithi107 ceph-mon[102221]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:13:30.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:30 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1989203483' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:13:30.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:30 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:13:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:30 smithi107 ceph-mon[105022]: pgmap v1029: 109 pgs: 1 peering, 108 active+clean; 381 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-21T10:13:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:30 smithi107 ceph-mon[105022]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:13:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:30 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1989203483' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:13:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:30 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:13:30.894 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:30.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:30 smithi129 ceph-mon[88471]: pgmap v1029: 109 pgs: 1 peering, 108 active+clean; 381 MiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-21T10:13:30.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:30 smithi129 ceph-mon[88471]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:13:30.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:30 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1989203483' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:13:30.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:30 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:13:30.936 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:30.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:30.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:31.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:31.072 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:31.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:31.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:31.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:31.216 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:31.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:31.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:31.319 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:31.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:31.359 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:31.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:31.460 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:31.498 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:31.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:31.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:31.600 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:31.624 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:31.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:31.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:31 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:13:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:31 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:13:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:31 smithi129 ceph-mon[88471]: osdmap e291: 8 total, 8 up, 7 in 2024-09-21T10:13:31.716 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:31 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:13:31.716 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:31 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:13:31.717 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:31 smithi107 ceph-mon[102221]: osdmap e291: 8 total, 8 up, 7 in 2024-09-21T10:13:31.718 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:31 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:13:31.718 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:31 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:13:31.718 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:31 smithi107 ceph-mon[105022]: osdmap e291: 8 total, 8 up, 7 in 2024-09-21T10:13:31.726 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:31.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:31.756 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:31.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:31.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:31.892 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:31.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:31.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:31.994 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:32.019 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:32.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:32.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:13:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:13:31] "GET /metrics HTTP/1.1" 200 36783 "" "Prometheus/2.51.0" 2024-09-21T10:13:32.120 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:32.144 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:32.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:32.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:32.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:32.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:32.306 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:32.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:32.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:32.444 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:32.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:32.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:32.545 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:32.583 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:32.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:32.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:32.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:32.725 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:32.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:32.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:32.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:32.863 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:32 smithi107 ceph-mon[102221]: pgmap v1031: 109 pgs: 1 peering, 108 active+clean; 326 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 750 op/s 2024-09-21T10:13:32.863 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:32 smithi107 ceph-mon[105022]: pgmap v1031: 109 pgs: 1 peering, 108 active+clean; 326 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 750 op/s 2024-09-21T10:13:32.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:32.894 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:32.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:32 smithi129 ceph-mon[88471]: pgmap v1031: 109 pgs: 1 peering, 108 active+clean; 326 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 750 op/s 2024-09-21T10:13:32.995 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:33.027 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:33.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:33.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:33.086 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Removed 112484 objects 2024-09-21T10:13:33.086 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Clean up completed and total clean up time :129.341 2024-09-21T10:13:33.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:33.150 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:33.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:33.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:33.252 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:33.274 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:33.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:33.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:33.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:33.399 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:33.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:33.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:33.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:33.521 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:33.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:33.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:33.531 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:13:33.531 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:13:33.531 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:13:33.532 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:13:33.532 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:13:33.532 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:13:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:13:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:13:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:13:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:13:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:13:33.532 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:13:33.533 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:13:33.533 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:13:33.533 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:13:33.533 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:13:33.623 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:33.645 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:33.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:33.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:33.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:33.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:13:33.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:33.769 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:33.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:33.789 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:33 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:33.789 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:33 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:33.789 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:33 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/75258253' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:33.789 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:33 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:33.789 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:33 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:33.789 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:33 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/75258253' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:33.870 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:33.891 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:33.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:33 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:33 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:33 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/75258253' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:33.993 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:34.015 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:34.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:34.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:34.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:34.137 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:34.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:34.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:34.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:34.259 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:34.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:34.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:34.361 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:34.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:34.386 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:34.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:34.487 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:34.510 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:34.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:34.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:34.612 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:34.635 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:34.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:34.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:34.737 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:34.761 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:34.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:34.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:34.863 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:34.886 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:34.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:34.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:34 smithi129 ceph-mon[88471]: pgmap v1032: 109 pgs: 109 active+clean; 91 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 866 op/s 2024-09-21T10:13:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:34 smithi129 ceph-mon[88471]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:13:34.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:35.014 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:35.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:35.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:34 smithi107 ceph-mon[102221]: pgmap v1032: 109 pgs: 109 active+clean; 91 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 866 op/s 2024-09-21T10:13:35.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:34 smithi107 ceph-mon[102221]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:13:35.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:34 smithi107 ceph-mon[105022]: pgmap v1032: 109 pgs: 109 active+clean; 91 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 866 op/s 2024-09-21T10:13:35.112 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:34 smithi107 ceph-mon[105022]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:13:35.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:35.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:35.148 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:35.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:35.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:35.276 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:35.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:35.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:35.378 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:35.404 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:35.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:35.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:35.506 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:35.531 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:35.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:35.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:35.633 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:35.659 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:35.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:35.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:35.760 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:35.786 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:35.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:35.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:35.888 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:35.913 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:35.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:35.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:36.015 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:36.039 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:36.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:36.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:36.141 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:36.163 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:36.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:36.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:36.255 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:36.265 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:36.291 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:36.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:36.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:36.352 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:13:36.353 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:13:36.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:36.413 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:36.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:36.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:36.515 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:36.536 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:36.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:36.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:36.637 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:36.663 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:36.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:36.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:36.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:36.785 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:36.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:36.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:36.887 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:36.903 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:36 smithi107 ceph-mon[102221]: pgmap v1033: 109 pgs: 109 active+clean; 50 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 751 op/s 2024-09-21T10:13:36.904 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:36 smithi107 ceph-mon[105022]: pgmap v1033: 109 pgs: 109 active+clean; 50 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 751 op/s 2024-09-21T10:13:36.912 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:36.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:36.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:36 smithi129 ceph-mon[88471]: pgmap v1033: 109 pgs: 109 active+clean; 50 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 751 op/s 2024-09-21T10:13:37.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:37.036 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:37.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:37.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:37.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:37.158 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:37.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:37.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:37.238 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1033,"stamp":"2024-09-21T10:13:35.322549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":52506452,"num_objects":817,"num_object_clones":0,"num_object_copies":2451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":817,"num_whiteouts":0,"num_read":36,"num_read_kb":92,"num_write":224261,"num_write_kb":7199206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11635,"num_bytes_recovered":759324941,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105038,"ondisk_log_size":105038,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":329,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":5747912,"kb_used_data":363712,"kb_used_omap":41,"kb_used_meta":5384022,"kb_avail":650468152,"statfs":{"total":671965249536,"available":666079387648,"internally_reserved":0,"allocated":372441088,"data_stored":360844089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42519,"internal_metadata":5513239017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,7,3,6,8,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-383844376,"num_objects":-5858,"num_object_clones":0,"num_object_copies":-17574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5858,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.793162"},"pg_stats":[{"pgid":"1.11","version":"77'100","reported_seq":98,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.584917+0000","last_change":"2024-09-21T10:10:57.397786+0000","last_active":"2024-09-21T10:13:30.584917+0000","last_peered":"2024-09-21T10:13:30.584917+0000","last_clean":"2024-09-21T10:13:30.584917+0000","last_became_active":"2024-09-21T10:10:57.396742+0000","last_became_peered":"2024-09-21T10:10:57.396742+0000","last_unstale":"2024-09-21T10:13:30.584917+0000","last_undegraded":"2024-09-21T10:13:30.584917+0000","last_fullsized":"2024-09-21T10:13:30.584917+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:53.631923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:53.631923+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:33:35.364595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":436,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.573267+0000","last_change":"2024-09-21T10:03:38.389577+0000","last_active":"2024-09-21T10:13:30.573267+0000","last_peered":"2024-09-21T10:13:30.573267+0000","last_clean":"2024-09-21T10:13:30.573267+0000","last_became_active":"2024-09-21T10:03:38.389292+0000","last_became_peered":"2024-09-21T10:03:38.389292+0000","last_unstale":"2024-09-21T10:13:30.573267+0000","last_undegraded":"2024-09-21T10:13:30.573267+0000","last_fullsized":"2024-09-21T10:13:30.573267+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:32.113472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":161,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.536724+0000","last_change":"2024-09-21T10:08:38.387871+0000","last_active":"2024-09-21T10:13:28.536724+0000","last_peered":"2024-09-21T10:13:28.536724+0000","last_clean":"2024-09-21T10:13:28.536724+0000","last_became_active":"2024-09-21T10:08:38.376727+0000","last_became_peered":"2024-09-21T10:08:38.376727+0000","last_unstale":"2024-09-21T10:13:28.536724+0000","last_undegraded":"2024-09-21T10:13:28.536724+0000","last_fullsized":"2024-09-21T10:13:28.536724+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:45:54.498950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'100","reported_seq":190,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.536525+0000","last_change":"2024-09-21T10:11:05.438031+0000","last_active":"2024-09-21T10:13:28.536525+0000","last_peered":"2024-09-21T10:13:28.536525+0000","last_clean":"2024-09-21T10:13:28.536525+0000","last_became_active":"2024-09-21T10:11:05.437637+0000","last_became_peered":"2024-09-21T10:11:05.437637+0000","last_unstale":"2024-09-21T10:13:28.536525+0000","last_undegraded":"2024-09-21T10:13:28.536525+0000","last_fullsized":"2024-09-21T10:13:28.536525+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.798577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.798577+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:42.343217+0000","scrub_duration":0.0023622640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":282,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.537095+0000","last_change":"2024-09-21T10:10:57.392782+0000","last_active":"2024-09-21T10:13:28.537095+0000","last_peered":"2024-09-21T10:13:28.537095+0000","last_clean":"2024-09-21T10:13:28.537095+0000","last_became_active":"2024-09-21T10:10:57.392305+0000","last_became_peered":"2024-09-21T10:10:57.392305+0000","last_unstale":"2024-09-21T10:13:28.537095+0000","last_undegraded":"2024-09-21T10:13:28.537095+0000","last_fullsized":"2024-09-21T10:13:28.537095+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:42:01.387622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":357,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.581031+0000","last_change":"2024-09-21T10:09:23.846703+0000","last_active":"2024-09-21T10:13:30.581031+0000","last_peered":"2024-09-21T10:13:30.581031+0000","last_clean":"2024-09-21T10:13:30.581031+0000","last_became_active":"2024-09-21T10:09:23.846415+0000","last_became_peered":"2024-09-21T10:09:23.846415+0000","last_unstale":"2024-09-21T10:13:30.581031+0000","last_undegraded":"2024-09-21T10:13:30.581031+0000","last_fullsized":"2024-09-21T10:13:30.581031+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:23.129285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'100","reported_seq":152,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.573754+0000","last_change":"2024-09-21T10:09:24.841480+0000","last_active":"2024-09-21T10:13:30.573754+0000","last_peered":"2024-09-21T10:13:30.573754+0000","last_clean":"2024-09-21T10:13:30.573754+0000","last_became_active":"2024-09-21T10:09:24.841145+0000","last_became_peered":"2024-09-21T10:09:24.841145+0000","last_unstale":"2024-09-21T10:13:30.573754+0000","last_undegraded":"2024-09-21T10:13:30.573754+0000","last_fullsized":"2024-09-21T10:13:30.573754+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.433227+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:13:41.719395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":23,"num_write":5,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":35780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":83,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.582748+0000","last_change":"2024-09-21T10:09:49.922489+0000","last_active":"2024-09-21T10:13:30.582748+0000","last_peered":"2024-09-21T10:13:30.582748+0000","last_clean":"2024-09-21T10:13:30.582748+0000","last_became_active":"2024-09-21T10:09:49.921629+0000","last_became_peered":"2024-09-21T10:09:49.921629+0000","last_unstale":"2024-09-21T10:13:30.582748+0000","last_undegraded":"2024-09-21T10:13:30.582748+0000","last_fullsized":"2024-09-21T10:13:30.582748+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:50.611979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":163,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.536423+0000","last_change":"2024-09-21T10:09:24.836647+0000","last_active":"2024-09-21T10:13:28.536423+0000","last_peered":"2024-09-21T10:13:28.536423+0000","last_clean":"2024-09-21T10:13:28.536423+0000","last_became_active":"2024-09-21T10:09:24.836278+0000","last_became_peered":"2024-09-21T10:09:24.836278+0000","last_unstale":"2024-09-21T10:13:28.536423+0000","last_undegraded":"2024-09-21T10:13:28.536423+0000","last_fullsized":"2024-09-21T10:13:28.536423+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:08:36.535892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"77'100","reported_seq":44,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.540885+0000","last_change":"2024-09-21T10:13:14.804339+0000","last_active":"2024-09-21T10:13:28.540885+0000","last_peered":"2024-09-21T10:13:28.540885+0000","last_clean":"2024-09-21T10:13:28.540885+0000","last_became_active":"2024-09-21T10:10:57.452150+0000","last_became_peered":"2024-09-21T10:10:57.452150+0000","last_unstale":"2024-09-21T10:13:28.540885+0000","last_undegraded":"2024-09-21T10:13:28.540885+0000","last_fullsized":"2024-09-21T10:13:28.540885+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:13:14.804256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:13:14.804256+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:33.744954+0000","scrub_duration":0.002846973,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":436,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.572984+0000","last_change":"2024-09-21T10:03:38.381552+0000","last_active":"2024-09-21T10:13:30.572984+0000","last_peered":"2024-09-21T10:13:30.572984+0000","last_clean":"2024-09-21T10:13:30.572984+0000","last_became_active":"2024-09-21T10:03:38.380898+0000","last_became_peered":"2024-09-21T10:03:38.380898+0000","last_unstale":"2024-09-21T10:13:30.572984+0000","last_undegraded":"2024-09-21T10:13:30.572984+0000","last_fullsized":"2024-09-21T10:13:30.572984+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:53.409606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":350,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.537355+0000","last_change":"2024-09-21T10:10:57.459057+0000","last_active":"2024-09-21T10:13:28.537355+0000","last_peered":"2024-09-21T10:13:28.537355+0000","last_clean":"2024-09-21T10:13:28.537355+0000","last_became_active":"2024-09-21T10:10:57.451776+0000","last_became_peered":"2024-09-21T10:10:57.451776+0000","last_unstale":"2024-09-21T10:13:28.537355+0000","last_undegraded":"2024-09-21T10:13:28.537355+0000","last_fullsized":"2024-09-21T10:13:28.537355+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:00.138317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"291'13921","reported_seq":17908,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:32.076977+0000","last_change":"2024-09-21T10:13:03.848496+0000","last_active":"2024-09-21T10:13:32.076977+0000","last_peered":"2024-09-21T10:13:32.076977+0000","last_clean":"2024-09-21T10:13:32.076977+0000","last_became_active":"2024-09-21T10:13:03.793148+0000","last_became_peered":"2024-09-21T10:13:03.793148+0000","last_unstale":"2024-09-21T10:13:32.076977+0000","last_undegraded":"2024-09-21T10:13:32.076977+0000","last_fullsized":"2024-09-21T10:13:32.076977+0000","mapping_epoch":284,"log_start":"274'7500","ondisk_log_start":"274'7500","created":252,"last_epoch_clean":285,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6421,"log_dups_size":0,"ondisk_log_size":6421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:00:29.314783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4128768,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13921,"num_write_kb":447488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1937,"num_bytes_recovered":126943232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"77'100","reported_seq":175,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.537020+0000","last_change":"2024-09-21T10:08:38.374573+0000","last_active":"2024-09-21T10:13:28.537020+0000","last_peered":"2024-09-21T10:13:28.537020+0000","last_clean":"2024-09-21T10:13:28.537020+0000","last_became_active":"2024-09-21T10:08:38.374075+0000","last_became_peered":"2024-09-21T10:08:38.374075+0000","last_unstale":"2024-09-21T10:13:28.537020+0000","last_undegraded":"2024-09-21T10:13:28.537020+0000","last_fullsized":"2024-09-21T10:13:28.537020+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:55.259385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:55.259385+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:28:26.784162+0000","scrub_duration":0.0015667260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":206,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.536961+0000","last_change":"2024-09-21T10:09:49.941392+0000","last_active":"2024-09-21T10:13:28.536961+0000","last_peered":"2024-09-21T10:13:28.536961+0000","last_clean":"2024-09-21T10:13:28.536961+0000","last_became_active":"2024-09-21T10:09:49.923535+0000","last_became_peered":"2024-09-21T10:09:49.923535+0000","last_unstale":"2024-09-21T10:13:28.536961+0000","last_undegraded":"2024-09-21T10:13:28.536961+0000","last_fullsized":"2024-09-21T10:13:28.536961+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:16.292163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":44,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.545484+0000","last_change":"2024-09-21T10:11:05.435502+0000","last_active":"2024-09-21T10:13:28.545484+0000","last_peered":"2024-09-21T10:13:28.545484+0000","last_clean":"2024-09-21T10:13:28.545484+0000","last_became_active":"2024-09-21T10:11:05.435216+0000","last_became_peered":"2024-09-21T10:11:05.435216+0000","last_unstale":"2024-09-21T10:13:28.545484+0000","last_undegraded":"2024-09-21T10:13:28.545484+0000","last_fullsized":"2024-09-21T10:13:28.545484+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:25.434424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"291'13813","reported_seq":13896,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:32.098746+0000","last_change":"2024-09-21T10:09:49.924187+0000","last_active":"2024-09-21T10:13:32.098746+0000","last_peered":"2024-09-21T10:13:32.098746+0000","last_clean":"2024-09-21T10:13:32.098746+0000","last_became_active":"2024-09-21T10:09:49.923789+0000","last_became_peered":"2024-09-21T10:09:49.923789+0000","last_unstale":"2024-09-21T10:13:32.098746+0000","last_undegraded":"2024-09-21T10:13:32.098746+0000","last_fullsized":"2024-09-21T10:13:32.098746+0000","mapping_epoch":255,"log_start":"274'7400","ondisk_log_start":"274'7400","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6413,"log_dups_size":0,"ondisk_log_size":6413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:52:49.559425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3473408,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13813,"num_write_kb":443712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"77'100","reported_seq":242,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.573095+0000","last_change":"2024-09-21T10:09:23.839211+0000","last_active":"2024-09-21T10:13:30.573095+0000","last_peered":"2024-09-21T10:13:30.573095+0000","last_clean":"2024-09-21T10:13:30.573095+0000","last_became_active":"2024-09-21T10:09:23.838779+0000","last_became_peered":"2024-09-21T10:09:23.838779+0000","last_unstale":"2024-09-21T10:13:30.573095+0000","last_undegraded":"2024-09-21T10:13:30.573095+0000","last_fullsized":"2024-09-21T10:13:30.573095+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.981336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.981336+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:03:15.566873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":185,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.537039+0000","last_change":"2024-09-21T10:09:49.932631+0000","last_active":"2024-09-21T10:13:28.537039+0000","last_peered":"2024-09-21T10:13:28.537039+0000","last_clean":"2024-09-21T10:13:28.537039+0000","last_became_active":"2024-09-21T10:09:49.922897+0000","last_became_peered":"2024-09-21T10:09:49.922897+0000","last_unstale":"2024-09-21T10:13:28.537039+0000","last_undegraded":"2024-09-21T10:13:28.537039+0000","last_fullsized":"2024-09-21T10:13:28.537039+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:40.795476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":231,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.585309+0000","last_change":"2024-09-21T10:08:38.377732+0000","last_active":"2024-09-21T10:13:30.585309+0000","last_peered":"2024-09-21T10:13:30.585309+0000","last_clean":"2024-09-21T10:13:30.585309+0000","last_became_active":"2024-09-21T10:08:38.376601+0000","last_became_peered":"2024-09-21T10:08:38.376601+0000","last_unstale":"2024-09-21T10:13:30.585309+0000","last_undegraded":"2024-09-21T10:13:30.585309+0000","last_fullsized":"2024-09-21T10:13:30.585309+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:02.540158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.5","version":"291'13440","reported_seq":24464,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:31.845812+0000","last_change":"2024-09-21T10:12:35.556125+0000","last_active":"2024-09-21T10:13:31.845812+0000","last_peered":"2024-09-21T10:13:31.845812+0000","last_clean":"2024-09-21T10:13:31.845812+0000","last_became_active":"2024-09-21T10:12:35.483696+0000","last_became_peered":"2024-09-21T10:12:35.483696+0000","last_unstale":"2024-09-21T10:13:31.845812+0000","last_undegraded":"2024-09-21T10:13:31.845812+0000","last_fullsized":"2024-09-21T10:13:31.845812+0000","mapping_epoch":278,"log_start":"273'7100","ondisk_log_start":"273'7100","created":252,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6340,"log_dups_size":0,"ondisk_log_size":6340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:54:35.212835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4980736,"num_objects":76,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13440,"num_write_kb":432512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5478,"num_bytes_recovered":359006208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"77'100","reported_seq":275,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.536028+0000","last_change":"2024-09-21T10:08:38.382100+0000","last_active":"2024-09-21T10:13:28.536028+0000","last_peered":"2024-09-21T10:13:28.536028+0000","last_clean":"2024-09-21T10:13:28.536028+0000","last_became_active":"2024-09-21T10:08:38.381736+0000","last_became_peered":"2024-09-21T10:08:38.381736+0000","last_unstale":"2024-09-21T10:13:28.536028+0000","last_undegraded":"2024-09-21T10:13:28.536028+0000","last_fullsized":"2024-09-21T10:13:28.536028+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.289859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.289859+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:23.739980+0000","scrub_duration":0.001251809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":171,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.536934+0000","last_change":"2024-09-21T10:10:57.460732+0000","last_active":"2024-09-21T10:13:28.536934+0000","last_peered":"2024-09-21T10:13:28.536934+0000","last_clean":"2024-09-21T10:13:28.536934+0000","last_became_active":"2024-09-21T10:10:57.454689+0000","last_became_peered":"2024-09-21T10:10:57.454689+0000","last_unstale":"2024-09-21T10:13:28.536934+0000","last_undegraded":"2024-09-21T10:13:28.536934+0000","last_fullsized":"2024-09-21T10:13:28.536934+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:12.035022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":291,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.535599+0000","last_change":"2024-09-21T10:03:38.378384+0000","last_active":"2024-09-21T10:13:28.535599+0000","last_peered":"2024-09-21T10:13:28.535599+0000","last_clean":"2024-09-21T10:13:28.535599+0000","last_became_active":"2024-09-21T10:03:38.378072+0000","last_became_peered":"2024-09-21T10:03:38.378072+0000","last_unstale":"2024-09-21T10:13:28.535599+0000","last_undegraded":"2024-09-21T10:13:28.535599+0000","last_fullsized":"2024-09-21T10:13:28.535599+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:54:37.039622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"291'14148","reported_seq":14195,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:31.815376+0000","last_change":"2024-09-21T10:09:42.896949+0000","last_active":"2024-09-21T10:13:31.815376+0000","last_peered":"2024-09-21T10:13:31.815376+0000","last_clean":"2024-09-21T10:13:31.815376+0000","last_became_active":"2024-09-21T10:09:42.896669+0000","last_became_peered":"2024-09-21T10:09:42.896669+0000","last_unstale":"2024-09-21T10:13:31.815376+0000","last_undegraded":"2024-09-21T10:13:31.815376+0000","last_fullsized":"2024-09-21T10:13:31.815376+0000","mapping_epoch":252,"log_start":"274'7800","ondisk_log_start":"274'7800","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6348,"log_dups_size":0,"ondisk_log_size":6348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:58:44.279472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4980736,"num_objects":76,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14148,"num_write_kb":455168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.e","version":"77'100","reported_seq":143,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.536829+0000","last_change":"2024-09-21T10:07:26.188298+0000","last_active":"2024-09-21T10:13:28.536829+0000","last_peered":"2024-09-21T10:13:28.536829+0000","last_clean":"2024-09-21T10:13:28.536829+0000","last_became_active":"2024-09-21T10:07:24.345265+0000","last_became_peered":"2024-09-21T10:07:24.345265+0000","last_unstale":"2024-09-21T10:13:28.536829+0000","last_undegraded":"2024-09-21T10:13:28.536829+0000","last_fullsized":"2024-09-21T10:13:28.536829+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:07:26.188220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:07:26.188220+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:46.381710+0000","scrub_duration":0.021884888000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":240,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.573449+0000","last_change":"2024-09-21T10:07:57.551556+0000","last_active":"2024-09-21T10:13:30.573449+0000","last_peered":"2024-09-21T10:13:30.573449+0000","last_clean":"2024-09-21T10:13:30.573449+0000","last_became_active":"2024-09-21T10:07:57.551249+0000","last_became_peered":"2024-09-21T10:07:57.551249+0000","last_unstale":"2024-09-21T10:13:30.573449+0000","last_undegraded":"2024-09-21T10:13:30.573449+0000","last_fullsized":"2024-09-21T10:13:30.573449+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:57.557152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":337,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.583139+0000","last_change":"2024-09-21T10:09:24.839665+0000","last_active":"2024-09-21T10:13:30.583139+0000","last_peered":"2024-09-21T10:13:30.583139+0000","last_clean":"2024-09-21T10:13:30.583139+0000","last_became_active":"2024-09-21T10:09:24.839353+0000","last_became_peered":"2024-09-21T10:09:24.839353+0000","last_unstale":"2024-09-21T10:13:30.583139+0000","last_undegraded":"2024-09-21T10:13:30.583139+0000","last_fullsized":"2024-09-21T10:13:30.583139+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:51.760073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.3","version":"291'13870","reported_seq":13917,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:32.785498+0000","last_change":"2024-09-21T10:09:43.231199+0000","last_active":"2024-09-21T10:13:32.785498+0000","last_peered":"2024-09-21T10:13:32.785498+0000","last_clean":"2024-09-21T10:13:32.785498+0000","last_became_active":"2024-09-21T10:09:43.230922+0000","last_became_peered":"2024-09-21T10:09:43.230922+0000","last_unstale":"2024-09-21T10:13:32.785498+0000","last_undegraded":"2024-09-21T10:13:32.785498+0000","last_fullsized":"2024-09-21T10:13:32.785498+0000","mapping_epoch":252,"log_start":"274'7700","ondisk_log_start":"274'7700","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6170,"log_dups_size":0,"ondisk_log_size":6170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:39.944122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":524288,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13870,"num_write_kb":444096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":165,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.536524+0000","last_change":"2024-09-21T10:04:53.355975+0000","last_active":"2024-09-21T10:13:28.536524+0000","last_peered":"2024-09-21T10:13:28.536524+0000","last_clean":"2024-09-21T10:13:28.536524+0000","last_became_active":"2024-09-21T10:04:42.547505+0000","last_became_peered":"2024-09-21T10:04:42.547505+0000","last_unstale":"2024-09-21T10:13:28.536524+0000","last_undegraded":"2024-09-21T10:13:28.536524+0000","last_fullsized":"2024-09-21T10:13:28.536524+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.355912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.355912+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:43.456830+0000","scrub_duration":0.0017980030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":436,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.573937+0000","last_change":"2024-09-21T10:09:49.939553+0000","last_active":"2024-09-21T10:13:30.573937+0000","last_peered":"2024-09-21T10:13:30.573937+0000","last_clean":"2024-09-21T10:13:30.573937+0000","last_became_active":"2024-09-21T10:09:49.925094+0000","last_became_peered":"2024-09-21T10:09:49.925094+0000","last_unstale":"2024-09-21T10:13:30.573937+0000","last_undegraded":"2024-09-21T10:13:30.573937+0000","last_fullsized":"2024-09-21T10:13:30.573937+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:12:12.933089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":214,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.573540+0000","last_change":"2024-09-21T10:08:38.379747+0000","last_active":"2024-09-21T10:13:30.573540+0000","last_peered":"2024-09-21T10:13:30.573540+0000","last_clean":"2024-09-21T10:13:30.573540+0000","last_became_active":"2024-09-21T10:08:38.378983+0000","last_became_peered":"2024-09-21T10:08:38.378983+0000","last_unstale":"2024-09-21T10:13:30.573540+0000","last_undegraded":"2024-09-21T10:13:30.573540+0000","last_fullsized":"2024-09-21T10:13:30.573540+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:45:08.003016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"291'13992","reported_seq":14082,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:33.081628+0000","last_change":"2024-09-21T10:09:42.896430+0000","last_active":"2024-09-21T10:13:33.081628+0000","last_peered":"2024-09-21T10:13:33.081628+0000","last_clean":"2024-09-21T10:13:33.081628+0000","last_became_active":"2024-09-21T10:09:42.896019+0000","last_became_peered":"2024-09-21T10:09:42.896019+0000","last_unstale":"2024-09-21T10:13:33.081628+0000","last_undegraded":"2024-09-21T10:13:33.081628+0000","last_fullsized":"2024-09-21T10:13:33.081628+0000","mapping_epoch":252,"log_start":"271'6400","ondisk_log_start":"271'6400","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":7592,"log_dups_size":0,"ondisk_log_size":7592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:59:18.633498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13992,"num_write_kb":447744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":196,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.563999+0000","last_change":"2024-09-21T10:10:57.403722+0000","last_active":"2024-09-21T10:13:30.563999+0000","last_peered":"2024-09-21T10:13:30.563999+0000","last_clean":"2024-09-21T10:13:30.563999+0000","last_became_active":"2024-09-21T10:10:57.394700+0000","last_became_peered":"2024-09-21T10:10:57.394700+0000","last_unstale":"2024-09-21T10:13:30.563999+0000","last_undegraded":"2024-09-21T10:13:30.563999+0000","last_fullsized":"2024-09-21T10:13:30.563999+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:02.152981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:02.152981+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:52:34.887763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":265,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.535783+0000","last_change":"2024-09-21T10:08:38.379849+0000","last_active":"2024-09-21T10:13:28.535783+0000","last_peered":"2024-09-21T10:13:28.535783+0000","last_clean":"2024-09-21T10:13:28.535783+0000","last_became_active":"2024-09-21T10:08:38.379543+0000","last_became_peered":"2024-09-21T10:08:38.379543+0000","last_unstale":"2024-09-21T10:13:28.535783+0000","last_undegraded":"2024-09-21T10:13:28.535783+0000","last_fullsized":"2024-09-21T10:13:28.535783+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:29.041802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":245,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.565615+0000","last_change":"2024-09-21T10:07:24.342950+0000","last_active":"2024-09-21T10:13:30.565615+0000","last_peered":"2024-09-21T10:13:30.565615+0000","last_clean":"2024-09-21T10:13:30.565615+0000","last_became_active":"2024-09-21T10:07:24.342477+0000","last_became_peered":"2024-09-21T10:07:24.342477+0000","last_unstale":"2024-09-21T10:13:30.565615+0000","last_undegraded":"2024-09-21T10:13:30.565615+0000","last_fullsized":"2024-09-21T10:13:30.565615+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:33:13.664050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1","version":"291'14107","reported_seq":14153,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:32.804692+0000","last_change":"2024-09-21T10:09:42.895259+0000","last_active":"2024-09-21T10:13:32.804692+0000","last_peered":"2024-09-21T10:13:32.804692+0000","last_clean":"2024-09-21T10:13:32.804692+0000","last_became_active":"2024-09-21T10:09:42.895087+0000","last_became_peered":"2024-09-21T10:09:42.895087+0000","last_unstale":"2024-09-21T10:13:32.804692+0000","last_undegraded":"2024-09-21T10:13:32.804692+0000","last_fullsized":"2024-09-21T10:13:32.804692+0000","mapping_epoch":252,"log_start":"274'7900","ondisk_log_start":"274'7900","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6207,"log_dups_size":0,"ondisk_log_size":6207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:40:59.270926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":589824,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14107,"num_write_kb":451712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"77'100","reported_seq":339,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.583384+0000","last_change":"2024-09-21T10:10:57.395906+0000","last_active":"2024-09-21T10:13:30.583384+0000","last_peered":"2024-09-21T10:13:30.583384+0000","last_clean":"2024-09-21T10:13:30.583384+0000","last_became_active":"2024-09-21T10:10:57.395366+0000","last_became_peered":"2024-09-21T10:10:57.395366+0000","last_unstale":"2024-09-21T10:13:30.583384+0000","last_undegraded":"2024-09-21T10:13:30.583384+0000","last_fullsized":"2024-09-21T10:13:30.583384+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.540323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.540323+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:34.873336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":337,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.564344+0000","last_change":"2024-09-21T10:07:36.464496+0000","last_active":"2024-09-21T10:13:30.564344+0000","last_peered":"2024-09-21T10:13:30.564344+0000","last_clean":"2024-09-21T10:13:30.564344+0000","last_became_active":"2024-09-21T10:07:36.464172+0000","last_became_peered":"2024-09-21T10:07:36.464172+0000","last_unstale":"2024-09-21T10:13:30.564344+0000","last_undegraded":"2024-09-21T10:13:30.564344+0000","last_fullsized":"2024-09-21T10:13:30.564344+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:09.839709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.535333+0000","last_change":"2024-09-21T10:11:05.442332+0000","last_active":"2024-09-21T10:13:28.535333+0000","last_peered":"2024-09-21T10:13:28.535333+0000","last_clean":"2024-09-21T10:13:28.535333+0000","last_became_active":"2024-09-21T10:11:05.442003+0000","last_became_peered":"2024-09-21T10:11:05.442003+0000","last_unstale":"2024-09-21T10:13:28.535333+0000","last_undegraded":"2024-09-21T10:13:28.535333+0000","last_fullsized":"2024-09-21T10:13:28.535333+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:34.986309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"291'14011","reported_seq":14099,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:31.857254+0000","last_change":"2024-09-21T10:09:43.231401+0000","last_active":"2024-09-21T10:13:31.857254+0000","last_peered":"2024-09-21T10:13:31.857254+0000","last_clean":"2024-09-21T10:13:31.857254+0000","last_became_active":"2024-09-21T10:09:43.230977+0000","last_became_peered":"2024-09-21T10:09:43.230977+0000","last_unstale":"2024-09-21T10:13:31.857254+0000","last_undegraded":"2024-09-21T10:13:31.857254+0000","last_fullsized":"2024-09-21T10:13:31.857254+0000","mapping_epoch":252,"log_start":"275'8100","ondisk_log_start":"275'8100","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5911,"log_dups_size":0,"ondisk_log_size":5911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:15.907809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3997696,"num_objects":61,"num_object_clones":0,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14011,"num_write_kb":450304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":159,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.564152+0000","last_change":"2024-09-21T10:08:04.645700+0000","last_active":"2024-09-21T10:13:30.564152+0000","last_peered":"2024-09-21T10:13:30.564152+0000","last_clean":"2024-09-21T10:13:30.564152+0000","last_became_active":"2024-09-21T10:07:57.548189+0000","last_became_peered":"2024-09-21T10:07:57.548189+0000","last_unstale":"2024-09-21T10:13:30.564152+0000","last_undegraded":"2024-09-21T10:13:30.564152+0000","last_fullsized":"2024-09-21T10:13:30.564152+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:48.318010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:48.318010+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T16:31:28.173059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":438,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.574704+0000","last_change":"2024-09-21T10:03:38.386604+0000","last_active":"2024-09-21T10:13:30.574704+0000","last_peered":"2024-09-21T10:13:30.574704+0000","last_clean":"2024-09-21T10:13:30.574704+0000","last_became_active":"2024-09-21T10:03:38.386283+0000","last_became_peered":"2024-09-21T10:03:38.386283+0000","last_unstale":"2024-09-21T10:13:30.574704+0000","last_undegraded":"2024-09-21T10:13:30.574704+0000","last_fullsized":"2024-09-21T10:13:30.574704+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:27:05.944880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":241,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.573029+0000","last_change":"2024-09-21T10:08:38.372486+0000","last_active":"2024-09-21T10:13:30.573029+0000","last_peered":"2024-09-21T10:13:30.573029+0000","last_clean":"2024-09-21T10:13:30.573029+0000","last_became_active":"2024-09-21T10:08:38.370669+0000","last_became_peered":"2024-09-21T10:08:38.370669+0000","last_unstale":"2024-09-21T10:13:30.573029+0000","last_undegraded":"2024-09-21T10:13:30.573029+0000","last_fullsized":"2024-09-21T10:13:30.573029+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:53:23.247341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.a","version":"291'13855","reported_seq":16894,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:31.785970+0000","last_change":"2024-09-21T10:13:11.885940+0000","last_active":"2024-09-21T10:13:31.785970+0000","last_peered":"2024-09-21T10:13:31.785970+0000","last_clean":"2024-09-21T10:13:31.785970+0000","last_became_active":"2024-09-21T10:13:11.856247+0000","last_became_peered":"2024-09-21T10:13:11.856247+0000","last_unstale":"2024-09-21T10:13:31.785970+0000","last_undegraded":"2024-09-21T10:13:31.785970+0000","last_fullsized":"2024-09-21T10:13:31.785970+0000","mapping_epoch":286,"log_start":"274'7900","ondisk_log_start":"274'7900","created":252,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5955,"log_dups_size":0,"ondisk_log_size":5955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:57:27.193951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4128768,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13855,"num_write_kb":445376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1464,"num_bytes_recovered":95944704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":182,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.536430+0000","last_change":"2024-09-21T10:09:49.924286+0000","last_active":"2024-09-21T10:13:28.536430+0000","last_peered":"2024-09-21T10:13:28.536430+0000","last_clean":"2024-09-21T10:13:28.536430+0000","last_became_active":"2024-09-21T10:09:49.923897+0000","last_became_peered":"2024-09-21T10:09:49.923897+0000","last_unstale":"2024-09-21T10:13:28.536430+0000","last_undegraded":"2024-09-21T10:13:28.536430+0000","last_fullsized":"2024-09-21T10:13:28.536430+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:47.393146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:47.393146+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:44.898083+0000","scrub_duration":0.001601086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":192,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.564678+0000","last_change":"2024-09-21T10:10:57.402946+0000","last_active":"2024-09-21T10:13:30.564678+0000","last_peered":"2024-09-21T10:13:30.564678+0000","last_clean":"2024-09-21T10:13:30.564678+0000","last_became_active":"2024-09-21T10:10:57.395450+0000","last_became_peered":"2024-09-21T10:10:57.395450+0000","last_unstale":"2024-09-21T10:13:30.564678+0000","last_undegraded":"2024-09-21T10:13:30.564678+0000","last_fullsized":"2024-09-21T10:13:30.564678+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:08.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":422,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.577620+0000","last_change":"2024-09-21T10:10:57.390595+0000","last_active":"2024-09-21T10:13:30.577620+0000","last_peered":"2024-09-21T10:13:30.577620+0000","last_clean":"2024-09-21T10:13:30.577620+0000","last_became_active":"2024-09-21T10:10:57.390265+0000","last_became_peered":"2024-09-21T10:10:57.390265+0000","last_unstale":"2024-09-21T10:13:30.577620+0000","last_undegraded":"2024-09-21T10:13:30.577620+0000","last_fullsized":"2024-09-21T10:13:30.577620+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:37.485063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":298,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.535958+0000","last_change":"2024-09-21T10:09:24.837651+0000","last_active":"2024-09-21T10:13:28.535958+0000","last_peered":"2024-09-21T10:13:28.535958+0000","last_clean":"2024-09-21T10:13:28.535958+0000","last_became_active":"2024-09-21T10:09:24.837226+0000","last_became_peered":"2024-09-21T10:09:24.837226+0000","last_unstale":"2024-09-21T10:13:28.535958+0000","last_undegraded":"2024-09-21T10:13:28.535958+0000","last_fullsized":"2024-09-21T10:13:28.535958+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:16.649601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"291'13870","reported_seq":13967,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:31.810428+0000","last_change":"2024-09-21T10:09:49.926338+0000","last_active":"2024-09-21T10:13:31.810428+0000","last_peered":"2024-09-21T10:13:31.810428+0000","last_clean":"2024-09-21T10:13:31.810428+0000","last_became_active":"2024-09-21T10:09:49.926018+0000","last_became_peered":"2024-09-21T10:09:49.926018+0000","last_unstale":"2024-09-21T10:13:31.810428+0000","last_undegraded":"2024-09-21T10:13:31.810428+0000","last_fullsized":"2024-09-21T10:13:31.810428+0000","mapping_epoch":255,"log_start":"274'7900","ondisk_log_start":"274'7900","created":252,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5970,"log_dups_size":0,"ondisk_log_size":5970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:09:49.364660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6029312,"num_objects":92,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13870,"num_write_kb":446784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":31,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.535026+0000","last_change":"2024-09-21T10:11:05.443167+0000","last_active":"2024-09-21T10:13:28.535026+0000","last_peered":"2024-09-21T10:13:28.535026+0000","last_clean":"2024-09-21T10:13:28.535026+0000","last_became_active":"2024-09-21T10:11:05.442805+0000","last_became_peered":"2024-09-21T10:11:05.442805+0000","last_unstale":"2024-09-21T10:13:28.535026+0000","last_undegraded":"2024-09-21T10:13:28.535026+0000","last_fullsized":"2024-09-21T10:13:28.535026+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.550551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.550551+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:58:57.998543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":156,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.536448+0000","last_change":"2024-09-21T10:09:23.838944+0000","last_active":"2024-09-21T10:13:28.536448+0000","last_peered":"2024-09-21T10:13:28.536448+0000","last_clean":"2024-09-21T10:13:28.536448+0000","last_became_active":"2024-09-21T10:09:23.837886+0000","last_became_peered":"2024-09-21T10:09:23.837886+0000","last_unstale":"2024-09-21T10:13:28.536448+0000","last_undegraded":"2024-09-21T10:13:28.536448+0000","last_fullsized":"2024-09-21T10:13:28.536448+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:16.928195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":315,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.584766+0000","last_change":"2024-09-21T10:09:24.839770+0000","last_active":"2024-09-21T10:13:30.584766+0000","last_peered":"2024-09-21T10:13:30.584766+0000","last_clean":"2024-09-21T10:13:30.584766+0000","last_became_active":"2024-09-21T10:09:24.839479+0000","last_became_peered":"2024-09-21T10:09:24.839479+0000","last_unstale":"2024-09-21T10:13:30.584766+0000","last_undegraded":"2024-09-21T10:13:30.584766+0000","last_fullsized":"2024-09-21T10:13:30.584766+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:11:19.618193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":83,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.582664+0000","last_change":"2024-09-21T10:09:49.923044+0000","last_active":"2024-09-21T10:13:30.582664+0000","last_peered":"2024-09-21T10:13:30.582664+0000","last_clean":"2024-09-21T10:13:30.582664+0000","last_became_active":"2024-09-21T10:09:49.922711+0000","last_became_peered":"2024-09-21T10:09:49.922711+0000","last_unstale":"2024-09-21T10:13:30.582664+0000","last_undegraded":"2024-09-21T10:13:30.582664+0000","last_fullsized":"2024-09-21T10:13:30.582664+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:14.090084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"291'14177","reported_seq":19688,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:31.857751+0000","last_change":"2024-09-21T10:12:54.765806+0000","last_active":"2024-09-21T10:13:31.857751+0000","last_peered":"2024-09-21T10:13:31.857751+0000","last_clean":"2024-09-21T10:13:31.857751+0000","last_became_active":"2024-09-21T10:12:54.721010+0000","last_became_peered":"2024-09-21T10:12:54.721010+0000","last_unstale":"2024-09-21T10:13:31.857751+0000","last_undegraded":"2024-09-21T10:13:31.857751+0000","last_fullsized":"2024-09-21T10:13:31.857751+0000","mapping_epoch":281,"log_start":"275'8200","ondisk_log_start":"275'8200","created":252,"last_epoch_clean":282,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5977,"log_dups_size":0,"ondisk_log_size":5977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:35.210888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5439488,"num_objects":83,"num_object_clones":0,"num_object_copies":249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14177,"num_write_kb":456320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2697,"num_bytes_recovered":176750592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":196,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.536524+0000","last_change":"2024-09-21T10:09:24.838434+0000","last_active":"2024-09-21T10:13:28.536524+0000","last_peered":"2024-09-21T10:13:28.536524+0000","last_clean":"2024-09-21T10:13:28.536524+0000","last_became_active":"2024-09-21T10:09:24.838066+0000","last_became_peered":"2024-09-21T10:09:24.838066+0000","last_unstale":"2024-09-21T10:13:28.536524+0000","last_undegraded":"2024-09-21T10:13:28.536524+0000","last_fullsized":"2024-09-21T10:13:28.536524+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:52.361137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:52.361137+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:19:57.788195+0000","scrub_duration":0.001255358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":475,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.564481+0000","last_change":"2024-09-21T10:09:23.837859+0000","last_active":"2024-09-21T10:13:30.564481+0000","last_peered":"2024-09-21T10:13:30.564481+0000","last_clean":"2024-09-21T10:13:30.564481+0000","last_became_active":"2024-09-21T10:09:23.837488+0000","last_became_peered":"2024-09-21T10:09:23.837488+0000","last_unstale":"2024-09-21T10:13:30.564481+0000","last_undegraded":"2024-09-21T10:13:30.564481+0000","last_fullsized":"2024-09-21T10:13:30.564481+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:25.215501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":40,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.536353+0000","last_change":"2024-09-21T10:10:27.806070+0000","last_active":"2024-09-21T10:13:28.536353+0000","last_peered":"2024-09-21T10:13:28.536353+0000","last_clean":"2024-09-21T10:13:28.536353+0000","last_became_active":"2024-09-21T10:10:27.805718+0000","last_became_peered":"2024-09-21T10:10:27.805718+0000","last_unstale":"2024-09-21T10:13:28.536353+0000","last_undegraded":"2024-09-21T10:13:28.536353+0000","last_fullsized":"2024-09-21T10:13:28.536353+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:31.144118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":469,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.566408+0000","last_change":"2024-09-21T10:10:57.395073+0000","last_active":"2024-09-21T10:13:30.566408+0000","last_peered":"2024-09-21T10:13:30.566408+0000","last_clean":"2024-09-21T10:13:30.566408+0000","last_became_active":"2024-09-21T10:10:57.394674+0000","last_became_peered":"2024-09-21T10:10:57.394674+0000","last_unstale":"2024-09-21T10:13:30.566408+0000","last_undegraded":"2024-09-21T10:13:30.566408+0000","last_fullsized":"2024-09-21T10:13:30.566408+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:48:48.912649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"291'14222","reported_seq":14312,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:31.835893+0000","last_change":"2024-09-21T10:09:43.231554+0000","last_active":"2024-09-21T10:13:31.835893+0000","last_peered":"2024-09-21T10:13:31.835893+0000","last_clean":"2024-09-21T10:13:31.835893+0000","last_became_active":"2024-09-21T10:09:43.231211+0000","last_became_peered":"2024-09-21T10:09:43.231211+0000","last_unstale":"2024-09-21T10:13:31.835893+0000","last_undegraded":"2024-09-21T10:13:31.835893+0000","last_fullsized":"2024-09-21T10:13:31.835893+0000","mapping_epoch":252,"log_start":"275'8300","ondisk_log_start":"275'8300","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":5922,"log_dups_size":0,"ondisk_log_size":5922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:16:41.457768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4849664,"num_objects":74,"num_object_clones":0,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14222,"num_write_kb":457472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":338,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.584815+0000","last_change":"2024-09-21T10:10:57.398437+0000","last_active":"2024-09-21T10:13:30.584815+0000","last_peered":"2024-09-21T10:13:30.584815+0000","last_clean":"2024-09-21T10:13:30.584815+0000","last_became_active":"2024-09-21T10:10:57.397441+0000","last_became_peered":"2024-09-21T10:10:57.397441+0000","last_unstale":"2024-09-21T10:13:30.584815+0000","last_undegraded":"2024-09-21T10:13:30.584815+0000","last_fullsized":"2024-09-21T10:13:30.584815+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:04.043917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:04.043917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:53:31.813612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":95,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.535867+0000","last_change":"2024-09-21T10:08:26.309323+0000","last_active":"2024-09-21T10:13:28.535867+0000","last_peered":"2024-09-21T10:13:28.535867+0000","last_clean":"2024-09-21T10:13:28.535867+0000","last_became_active":"2024-09-21T10:08:26.309004+0000","last_became_peered":"2024-09-21T10:08:26.309004+0000","last_unstale":"2024-09-21T10:13:28.535867+0000","last_undegraded":"2024-09-21T10:13:28.535867+0000","last_fullsized":"2024-09-21T10:13:28.535867+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:37:20.626702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":148,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.583294+0000","last_change":"2024-09-21T10:10:28.180093+0000","last_active":"2024-09-21T10:13:30.583294+0000","last_peered":"2024-09-21T10:13:30.583294+0000","last_clean":"2024-09-21T10:13:30.583294+0000","last_became_active":"2024-09-21T10:10:28.179559+0000","last_became_peered":"2024-09-21T10:10:28.179559+0000","last_unstale":"2024-09-21T10:13:30.583294+0000","last_undegraded":"2024-09-21T10:13:30.583294+0000","last_fullsized":"2024-09-21T10:13:30.583294+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:52:46.146392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":245,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.565294+0000","last_change":"2024-09-21T10:08:26.307041+0000","last_active":"2024-09-21T10:13:30.565294+0000","last_peered":"2024-09-21T10:13:30.565294+0000","last_clean":"2024-09-21T10:13:30.565294+0000","last_became_active":"2024-09-21T10:08:26.306715+0000","last_became_peered":"2024-09-21T10:08:26.306715+0000","last_unstale":"2024-09-21T10:13:30.565294+0000","last_undegraded":"2024-09-21T10:13:30.565294+0000","last_fullsized":"2024-09-21T10:13:30.565294+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:19:26.192398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"291'14236","reported_seq":14327,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:33.071002+0000","last_change":"2024-09-21T10:09:43.232893+0000","last_active":"2024-09-21T10:13:33.071002+0000","last_peered":"2024-09-21T10:13:33.071002+0000","last_clean":"2024-09-21T10:13:33.071002+0000","last_became_active":"2024-09-21T10:09:43.232461+0000","last_became_peered":"2024-09-21T10:09:43.232461+0000","last_unstale":"2024-09-21T10:13:33.071002+0000","last_undegraded":"2024-09-21T10:13:33.071002+0000","last_fullsized":"2024-09-21T10:13:33.071002+0000","mapping_epoch":252,"log_start":"272'6700","ondisk_log_start":"272'6700","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":7536,"log_dups_size":0,"ondisk_log_size":7536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:39.778833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14236,"num_write_kb":455489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":265,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.535564+0000","last_change":"2024-09-21T10:04:45.242148+0000","last_active":"2024-09-21T10:13:28.535564+0000","last_peered":"2024-09-21T10:13:28.535564+0000","last_clean":"2024-09-21T10:13:28.535564+0000","last_became_active":"2024-09-21T10:04:42.549844+0000","last_became_peered":"2024-09-21T10:04:42.549844+0000","last_unstale":"2024-09-21T10:13:28.535564+0000","last_undegraded":"2024-09-21T10:13:28.535564+0000","last_fullsized":"2024-09-21T10:13:28.535564+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_deep_scrub":"77'100","last_deep_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.242073+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:03:42.250480+0000","scrub_duration":0.0023466239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":34,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.541321+0000","last_change":"2024-09-21T10:10:57.392880+0000","last_active":"2024-09-21T10:13:28.541321+0000","last_peered":"2024-09-21T10:13:28.541321+0000","last_clean":"2024-09-21T10:13:28.541321+0000","last_became_active":"2024-09-21T10:10:57.392539+0000","last_became_peered":"2024-09-21T10:10:57.392539+0000","last_unstale":"2024-09-21T10:13:28.541321+0000","last_undegraded":"2024-09-21T10:13:28.541321+0000","last_fullsized":"2024-09-21T10:13:28.541321+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:49.754955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":358,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.579332+0000","last_change":"2024-09-21T10:09:23.865342+0000","last_active":"2024-09-21T10:13:30.579332+0000","last_peered":"2024-09-21T10:13:30.579332+0000","last_clean":"2024-09-21T10:13:30.579332+0000","last_became_active":"2024-09-21T10:09:23.865008+0000","last_became_peered":"2024-09-21T10:09:23.865008+0000","last_unstale":"2024-09-21T10:13:30.579332+0000","last_undegraded":"2024-09-21T10:13:30.579332+0000","last_fullsized":"2024-09-21T10:13:30.579332+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:03.356101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":34,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.540800+0000","last_change":"2024-09-21T10:10:57.394707+0000","last_active":"2024-09-21T10:13:28.540800+0000","last_peered":"2024-09-21T10:13:28.540800+0000","last_clean":"2024-09-21T10:13:28.540800+0000","last_became_active":"2024-09-21T10:10:57.394363+0000","last_became_peered":"2024-09-21T10:10:57.394363+0000","last_unstale":"2024-09-21T10:13:28.540800+0000","last_undegraded":"2024-09-21T10:13:28.540800+0000","last_fullsized":"2024-09-21T10:13:28.540800+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:21.141052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"291'14142","reported_seq":14232,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:33.038849+0000","last_change":"2024-09-21T10:09:43.232987+0000","last_active":"2024-09-21T10:13:33.038849+0000","last_peered":"2024-09-21T10:13:33.038849+0000","last_clean":"2024-09-21T10:13:33.038849+0000","last_became_active":"2024-09-21T10:09:43.232577+0000","last_became_peered":"2024-09-21T10:09:43.232577+0000","last_unstale":"2024-09-21T10:13:33.038849+0000","last_undegraded":"2024-09-21T10:13:33.038849+0000","last_fullsized":"2024-09-21T10:13:33.038849+0000","mapping_epoch":252,"log_start":"272'6600","ondisk_log_start":"272'6600","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":7542,"log_dups_size":0,"ondisk_log_size":7542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:16:14.536774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14142,"num_write_kb":452544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":282,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.535443+0000","last_change":"2024-09-21T10:09:23.839563+0000","last_active":"2024-09-21T10:13:28.535443+0000","last_peered":"2024-09-21T10:13:28.535443+0000","last_clean":"2024-09-21T10:13:28.535443+0000","last_became_active":"2024-09-21T10:09:23.839207+0000","last_became_peered":"2024-09-21T10:09:23.839207+0000","last_unstale":"2024-09-21T10:13:28.535443+0000","last_undegraded":"2024-09-21T10:13:28.535443+0000","last_fullsized":"2024-09-21T10:13:28.535443+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:49.226496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:49.226496+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:03.960125+0000","scrub_duration":0.001656865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":210,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.565157+0000","last_change":"2024-09-21T10:10:57.395167+0000","last_active":"2024-09-21T10:13:30.565157+0000","last_peered":"2024-09-21T10:13:30.565157+0000","last_clean":"2024-09-21T10:13:30.565157+0000","last_became_active":"2024-09-21T10:10:57.394818+0000","last_became_peered":"2024-09-21T10:10:57.394818+0000","last_unstale":"2024-09-21T10:13:30.565157+0000","last_undegraded":"2024-09-21T10:13:30.565157+0000","last_fullsized":"2024-09-21T10:13:30.565157+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:57.224113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":176,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.537143+0000","last_change":"2024-09-21T10:09:49.922782+0000","last_active":"2024-09-21T10:13:28.537143+0000","last_peered":"2024-09-21T10:13:28.537143+0000","last_clean":"2024-09-21T10:13:28.537143+0000","last_became_active":"2024-09-21T10:09:49.922465+0000","last_became_peered":"2024-09-21T10:09:49.922465+0000","last_unstale":"2024-09-21T10:13:28.537143+0000","last_undegraded":"2024-09-21T10:13:28.537143+0000","last_fullsized":"2024-09-21T10:13:28.537143+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:06.151245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":353,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.583508+0000","last_change":"2024-09-21T10:10:57.397251+0000","last_active":"2024-09-21T10:13:30.583508+0000","last_peered":"2024-09-21T10:13:30.583508+0000","last_clean":"2024-09-21T10:13:30.583508+0000","last_became_active":"2024-09-21T10:10:57.396271+0000","last_became_peered":"2024-09-21T10:10:57.396271+0000","last_unstale":"2024-09-21T10:13:30.583508+0000","last_undegraded":"2024-09-21T10:13:30.583508+0000","last_fullsized":"2024-09-21T10:13:30.583508+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:17.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.f","version":"291'14218","reported_seq":14265,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:31.816367+0000","last_change":"2024-09-21T10:09:43.232625+0000","last_active":"2024-09-21T10:13:31.816367+0000","last_peered":"2024-09-21T10:13:31.816367+0000","last_clean":"2024-09-21T10:13:31.816367+0000","last_became_active":"2024-09-21T10:09:43.232331+0000","last_became_peered":"2024-09-21T10:09:43.232331+0000","last_unstale":"2024-09-21T10:13:31.816367+0000","last_undegraded":"2024-09-21T10:13:31.816367+0000","last_fullsized":"2024-09-21T10:13:31.816367+0000","mapping_epoch":252,"log_start":"274'7900","ondisk_log_start":"274'7900","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6318,"log_dups_size":0,"ondisk_log_size":6318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:01:16.634826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932160,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14218,"num_write_kb":456896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":156,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.565468+0000","last_change":"2024-09-21T10:11:05.455170+0000","last_active":"2024-09-21T10:13:30.565468+0000","last_peered":"2024-09-21T10:13:30.565468+0000","last_clean":"2024-09-21T10:13:30.565468+0000","last_became_active":"2024-09-21T10:11:05.454607+0000","last_became_peered":"2024-09-21T10:11:05.454607+0000","last_unstale":"2024-09-21T10:13:30.565468+0000","last_undegraded":"2024-09-21T10:13:30.565468+0000","last_fullsized":"2024-09-21T10:13:30.565468+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:59.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:59.099852+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:14.788975+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":444,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.573307+0000","last_change":"2024-09-21T10:09:23.841186+0000","last_active":"2024-09-21T10:13:30.573307+0000","last_peered":"2024-09-21T10:13:30.573307+0000","last_clean":"2024-09-21T10:13:30.573307+0000","last_became_active":"2024-09-21T10:09:23.840854+0000","last_became_peered":"2024-09-21T10:09:23.840854+0000","last_unstale":"2024-09-21T10:13:30.573307+0000","last_undegraded":"2024-09-21T10:13:30.573307+0000","last_fullsized":"2024-09-21T10:13:30.573307+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:29.666358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":113,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.535977+0000","last_change":"2024-09-21T10:08:38.380867+0000","last_active":"2024-09-21T10:13:28.535977+0000","last_peered":"2024-09-21T10:13:28.535977+0000","last_clean":"2024-09-21T10:13:28.535977+0000","last_became_active":"2024-09-21T10:08:38.379599+0000","last_became_peered":"2024-09-21T10:08:38.379599+0000","last_unstale":"2024-09-21T10:13:28.535977+0000","last_undegraded":"2024-09-21T10:13:28.535977+0000","last_fullsized":"2024-09-21T10:13:28.535977+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:50.499470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":440,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.574436+0000","last_change":"2024-09-21T10:09:49.926548+0000","last_active":"2024-09-21T10:13:30.574436+0000","last_peered":"2024-09-21T10:13:30.574436+0000","last_clean":"2024-09-21T10:13:30.574436+0000","last_became_active":"2024-09-21T10:09:49.926255+0000","last_became_peered":"2024-09-21T10:09:49.926255+0000","last_unstale":"2024-09-21T10:13:30.574436+0000","last_undegraded":"2024-09-21T10:13:30.574436+0000","last_fullsized":"2024-09-21T10:13:30.574436+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:49.855878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"291'14157","reported_seq":14204,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:31.827506+0000","last_change":"2024-09-21T10:09:43.231158+0000","last_active":"2024-09-21T10:13:31.827506+0000","last_peered":"2024-09-21T10:13:31.827506+0000","last_clean":"2024-09-21T10:13:31.827506+0000","last_became_active":"2024-09-21T10:09:43.230838+0000","last_became_peered":"2024-09-21T10:09:43.230838+0000","last_unstale":"2024-09-21T10:13:31.827506+0000","last_undegraded":"2024-09-21T10:13:31.827506+0000","last_fullsized":"2024-09-21T10:13:31.827506+0000","mapping_epoch":252,"log_start":"274'7900","ondisk_log_start":"274'7900","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:09:41.875156+0000","last_clean_scrub_stamp":"2024-09-21T10:09:41.875156+0000","objects_scrubbed":0,"log_size":6257,"log_dups_size":0,"ondisk_log_size":6257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:15.318633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4784128,"num_objects":73,"num_object_clones":0,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14157,"num_write_kb":455360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":242,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.572985+0000","last_change":"2024-09-21T10:10:57.390804+0000","last_active":"2024-09-21T10:13:30.572985+0000","last_peered":"2024-09-21T10:13:30.572985+0000","last_clean":"2024-09-21T10:13:30.572985+0000","last_became_active":"2024-09-21T10:10:57.390323+0000","last_became_peered":"2024-09-21T10:10:57.390323+0000","last_unstale":"2024-09-21T10:13:30.572985+0000","last_undegraded":"2024-09-21T10:13:30.572985+0000","last_fullsized":"2024-09-21T10:13:30.572985+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.005477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.005477+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:29:57.702046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":87,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.582843+0000","last_change":"2024-09-21T10:09:49.930280+0000","last_active":"2024-09-21T10:13:30.582843+0000","last_peered":"2024-09-21T10:13:30.582843+0000","last_clean":"2024-09-21T10:13:30.582843+0000","last_became_active":"2024-09-21T10:09:49.919101+0000","last_became_peered":"2024-09-21T10:09:49.919101+0000","last_unstale":"2024-09-21T10:13:30.582843+0000","last_undegraded":"2024-09-21T10:13:30.582843+0000","last_fullsized":"2024-09-21T10:13:30.582843+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:23.998475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":304,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.535912+0000","last_change":"2024-09-21T10:03:38.392767+0000","last_active":"2024-09-21T10:13:28.535912+0000","last_peered":"2024-09-21T10:13:28.535912+0000","last_clean":"2024-09-21T10:13:28.535912+0000","last_became_active":"2024-09-21T10:03:38.392424+0000","last_became_peered":"2024-09-21T10:03:38.392424+0000","last_unstale":"2024-09-21T10:13:28.535912+0000","last_undegraded":"2024-09-21T10:13:28.535912+0000","last_fullsized":"2024-09-21T10:13:28.535912+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:25.358291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":85,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.540890+0000","last_change":"2024-09-21T10:09:24.836360+0000","last_active":"2024-09-21T10:13:28.540890+0000","last_peered":"2024-09-21T10:13:28.540890+0000","last_clean":"2024-09-21T10:13:28.540890+0000","last_became_active":"2024-09-21T10:09:24.836033+0000","last_became_peered":"2024-09-21T10:09:24.836033+0000","last_unstale":"2024-09-21T10:13:28.540890+0000","last_undegraded":"2024-09-21T10:13:28.540890+0000","last_fullsized":"2024-09-21T10:13:28.540890+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:58.368582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":142,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.535194+0000","last_change":"2024-09-21T10:08:38.380189+0000","last_active":"2024-09-21T10:13:28.535194+0000","last_peered":"2024-09-21T10:13:28.535194+0000","last_clean":"2024-09-21T10:13:28.535194+0000","last_became_active":"2024-09-21T10:08:38.379891+0000","last_became_peered":"2024-09-21T10:08:38.379891+0000","last_unstale":"2024-09-21T10:13:28.535194+0000","last_undegraded":"2024-09-21T10:13:28.535194+0000","last_fullsized":"2024-09-21T10:13:28.535194+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:35.820921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":222,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.536907+0000","last_change":"2024-09-21T10:08:38.392563+0000","last_active":"2024-09-21T10:13:28.536907+0000","last_peered":"2024-09-21T10:13:28.536907+0000","last_clean":"2024-09-21T10:13:28.536907+0000","last_became_active":"2024-09-21T10:08:38.378735+0000","last_became_peered":"2024-09-21T10:08:38.378735+0000","last_unstale":"2024-09-21T10:13:28.536907+0000","last_undegraded":"2024-09-21T10:13:28.536907+0000","last_fullsized":"2024-09-21T10:13:28.536907+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:29:41.423208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":31,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.535700+0000","last_change":"2024-09-21T10:11:05.453286+0000","last_active":"2024-09-21T10:13:28.535700+0000","last_peered":"2024-09-21T10:13:28.535700+0000","last_clean":"2024-09-21T10:13:28.535700+0000","last_became_active":"2024-09-21T10:11:05.452601+0000","last_became_peered":"2024-09-21T10:11:05.452601+0000","last_unstale":"2024-09-21T10:13:28.535700+0000","last_undegraded":"2024-09-21T10:13:28.535700+0000","last_fullsized":"2024-09-21T10:13:28.535700+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:27.532091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":79,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.565621+0000","last_change":"2024-09-21T10:10:27.833762+0000","last_active":"2024-09-21T10:13:30.565621+0000","last_peered":"2024-09-21T10:13:30.565621+0000","last_clean":"2024-09-21T10:13:30.565621+0000","last_became_active":"2024-09-21T10:10:27.819999+0000","last_became_peered":"2024-09-21T10:10:27.819999+0000","last_unstale":"2024-09-21T10:13:30.565621+0000","last_undegraded":"2024-09-21T10:13:30.565621+0000","last_fullsized":"2024-09-21T10:13:30.565621+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:33.839463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'100","reported_seq":234,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.564960+0000","last_change":"2024-09-21T10:07:24.345428+0000","last_active":"2024-09-21T10:13:30.564960+0000","last_peered":"2024-09-21T10:13:30.564960+0000","last_clean":"2024-09-21T10:13:30.564960+0000","last_became_active":"2024-09-21T10:07:24.343529+0000","last_became_peered":"2024-09-21T10:07:24.343529+0000","last_unstale":"2024-09-21T10:13:30.564960+0000","last_undegraded":"2024-09-21T10:13:30.564960+0000","last_fullsized":"2024-09-21T10:13:30.564960+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.014993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.014993+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:35.677888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":34,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.541163+0000","last_change":"2024-09-21T10:10:57.394885+0000","last_active":"2024-09-21T10:13:28.541163+0000","last_peered":"2024-09-21T10:13:28.541163+0000","last_clean":"2024-09-21T10:13:28.541163+0000","last_became_active":"2024-09-21T10:10:57.394584+0000","last_became_peered":"2024-09-21T10:10:57.394584+0000","last_unstale":"2024-09-21T10:13:28.541163+0000","last_undegraded":"2024-09-21T10:13:28.541163+0000","last_fullsized":"2024-09-21T10:13:28.541163+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:52:16.853713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":426,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.585101+0000","last_change":"2024-09-21T10:03:38.425111+0000","last_active":"2024-09-21T10:13:30.585101+0000","last_peered":"2024-09-21T10:13:30.585101+0000","last_clean":"2024-09-21T10:13:30.585101+0000","last_became_active":"2024-09-21T10:03:38.424704+0000","last_became_peered":"2024-09-21T10:03:38.424704+0000","last_unstale":"2024-09-21T10:13:30.585101+0000","last_undegraded":"2024-09-21T10:13:30.585101+0000","last_fullsized":"2024-09-21T10:13:30.585101+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:35.157076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":327,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.579997+0000","last_change":"2024-09-21T10:10:57.397755+0000","last_active":"2024-09-21T10:13:30.579997+0000","last_peered":"2024-09-21T10:13:30.579997+0000","last_clean":"2024-09-21T10:13:30.579997+0000","last_became_active":"2024-09-21T10:10:57.397249+0000","last_became_peered":"2024-09-21T10:10:57.397249+0000","last_unstale":"2024-09-21T10:13:30.579997+0000","last_undegraded":"2024-09-21T10:13:30.579997+0000","last_fullsized":"2024-09-21T10:13:30.579997+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:40.813503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":80,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.548892+0000","last_change":"2024-09-21T10:13:28.546311+0000","last_active":"2024-09-21T10:13:28.548892+0000","last_peered":"2024-09-21T10:13:28.548892+0000","last_clean":"2024-09-21T10:13:28.548892+0000","last_became_active":"2024-09-21T10:13:28.545887+0000","last_became_peered":"2024-09-21T10:13:28.545887+0000","last_unstale":"2024-09-21T10:13:28.548892+0000","last_undegraded":"2024-09-21T10:13:28.548892+0000","last_fullsized":"2024-09-21T10:13:28.548892+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:32:06.883624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":254,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.535833+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T10:13:28.535833+0000","last_peered":"2024-09-21T10:13:28.535833+0000","last_clean":"2024-09-21T10:13:28.535833+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T10:13:28.535833+0000","last_undegraded":"2024-09-21T10:13:28.535833+0000","last_fullsized":"2024-09-21T10:13:28.535833+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":447,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.574513+0000","last_change":"2024-09-21T10:09:49.928053+0000","last_active":"2024-09-21T10:13:30.574513+0000","last_peered":"2024-09-21T10:13:30.574513+0000","last_clean":"2024-09-21T10:13:30.574513+0000","last_became_active":"2024-09-21T10:09:49.927231+0000","last_became_peered":"2024-09-21T10:09:49.927231+0000","last_unstale":"2024-09-21T10:13:30.574513+0000","last_undegraded":"2024-09-21T10:13:30.574513+0000","last_fullsized":"2024-09-21T10:13:30.574513+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:26.987553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":360,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.588777+0000","last_change":"2024-09-21T10:09:23.840717+0000","last_active":"2024-09-21T10:13:30.588777+0000","last_peered":"2024-09-21T10:13:30.588777+0000","last_clean":"2024-09-21T10:13:30.588777+0000","last_became_active":"2024-09-21T10:09:23.840323+0000","last_became_peered":"2024-09-21T10:09:23.840323+0000","last_unstale":"2024-09-21T10:13:30.588777+0000","last_undegraded":"2024-09-21T10:13:30.588777+0000","last_fullsized":"2024-09-21T10:13:30.588777+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:55.769770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":73,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.564187+0000","last_change":"2024-09-21T10:10:27.816526+0000","last_active":"2024-09-21T10:13:30.564187+0000","last_peered":"2024-09-21T10:13:30.564187+0000","last_clean":"2024-09-21T10:13:30.564187+0000","last_became_active":"2024-09-21T10:10:27.816221+0000","last_became_peered":"2024-09-21T10:10:27.816221+0000","last_unstale":"2024-09-21T10:13:30.564187+0000","last_undegraded":"2024-09-21T10:13:30.564187+0000","last_fullsized":"2024-09-21T10:13:30.564187+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:08.331228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":417,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.572287+0000","last_change":"2024-09-21T10:08:38.379994+0000","last_active":"2024-09-21T10:13:30.572287+0000","last_peered":"2024-09-21T10:13:30.572287+0000","last_clean":"2024-09-21T10:13:30.572287+0000","last_became_active":"2024-09-21T10:08:38.378729+0000","last_became_peered":"2024-09-21T10:08:38.378729+0000","last_unstale":"2024-09-21T10:13:30.572287+0000","last_undegraded":"2024-09-21T10:13:30.572287+0000","last_fullsized":"2024-09-21T10:13:30.572287+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:16.940173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":137,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.564368+0000","last_change":"2024-09-21T10:09:24.844561+0000","last_active":"2024-09-21T10:13:30.564368+0000","last_peered":"2024-09-21T10:13:30.564368+0000","last_clean":"2024-09-21T10:13:30.564368+0000","last_became_active":"2024-09-21T10:09:24.844190+0000","last_became_peered":"2024-09-21T10:09:24.844190+0000","last_unstale":"2024-09-21T10:13:30.564368+0000","last_undegraded":"2024-09-21T10:13:30.564368+0000","last_fullsized":"2024-09-21T10:13:30.564368+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:46:04.775344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":464,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.565239+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:13:30.565239+0000","last_peered":"2024-09-21T10:13:30.565239+0000","last_clean":"2024-09-21T10:13:30.565239+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:13:30.565239+0000","last_undegraded":"2024-09-21T10:13:30.565239+0000","last_fullsized":"2024-09-21T10:13:30.565239+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":83,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.582962+0000","last_change":"2024-09-21T10:09:49.918246+0000","last_active":"2024-09-21T10:13:30.582962+0000","last_peered":"2024-09-21T10:13:30.582962+0000","last_clean":"2024-09-21T10:13:30.582962+0000","last_became_active":"2024-09-21T10:09:49.917911+0000","last_became_peered":"2024-09-21T10:09:49.917911+0000","last_unstale":"2024-09-21T10:13:30.582962+0000","last_undegraded":"2024-09-21T10:13:30.582962+0000","last_fullsized":"2024-09-21T10:13:30.582962+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:30:10.567650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":273,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.573855+0000","last_change":"2024-09-21T10:07:24.347937+0000","last_active":"2024-09-21T10:13:30.573855+0000","last_peered":"2024-09-21T10:13:30.573855+0000","last_clean":"2024-09-21T10:13:30.573855+0000","last_became_active":"2024-09-21T10:07:24.347376+0000","last_became_peered":"2024-09-21T10:07:24.347376+0000","last_unstale":"2024-09-21T10:13:30.573855+0000","last_undegraded":"2024-09-21T10:13:30.573855+0000","last_fullsized":"2024-09-21T10:13:30.573855+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:48:05.644824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":73,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.535281+0000","last_change":"2024-09-21T10:08:38.382518+0000","last_active":"2024-09-21T10:13:28.535281+0000","last_peered":"2024-09-21T10:13:28.535281+0000","last_clean":"2024-09-21T10:13:28.535281+0000","last_became_active":"2024-09-21T10:08:38.372051+0000","last_became_peered":"2024-09-21T10:08:38.372051+0000","last_unstale":"2024-09-21T10:13:28.535281+0000","last_undegraded":"2024-09-21T10:13:28.535281+0000","last_fullsized":"2024-09-21T10:13:28.535281+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:21.360435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":364,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.577762+0000","last_change":"2024-09-21T10:09:24.840173+0000","last_active":"2024-09-21T10:13:30.577762+0000","last_peered":"2024-09-21T10:13:30.577762+0000","last_clean":"2024-09-21T10:13:30.577762+0000","last_became_active":"2024-09-21T10:09:24.839762+0000","last_became_peered":"2024-09-21T10:09:24.839762+0000","last_unstale":"2024-09-21T10:13:30.577762+0000","last_undegraded":"2024-09-21T10:13:30.577762+0000","last_fullsized":"2024-09-21T10:13:30.577762+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:12:53.465203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":87,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.535922+0000","last_change":"2024-09-21T10:08:38.388561+0000","last_active":"2024-09-21T10:13:28.535922+0000","last_peered":"2024-09-21T10:13:28.535922+0000","last_clean":"2024-09-21T10:13:28.535922+0000","last_became_active":"2024-09-21T10:08:38.374558+0000","last_became_peered":"2024-09-21T10:08:38.374558+0000","last_unstale":"2024-09-21T10:13:28.535922+0000","last_undegraded":"2024-09-21T10:13:28.535922+0000","last_fullsized":"2024-09-21T10:13:28.535922+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:24.111438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":448,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.585515+0000","last_change":"2024-09-21T10:09:23.840467+0000","last_active":"2024-09-21T10:13:30.585515+0000","last_peered":"2024-09-21T10:13:30.585515+0000","last_clean":"2024-09-21T10:13:30.585515+0000","last_became_active":"2024-09-21T10:09:23.840177+0000","last_became_peered":"2024-09-21T10:09:23.840177+0000","last_unstale":"2024-09-21T10:13:30.585515+0000","last_undegraded":"2024-09-21T10:13:30.585515+0000","last_fullsized":"2024-09-21T10:13:30.585515+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:17.887307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":437,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.584287+0000","last_change":"2024-09-21T10:11:05.451613+0000","last_active":"2024-09-21T10:13:30.584287+0000","last_peered":"2024-09-21T10:13:30.584287+0000","last_clean":"2024-09-21T10:13:30.584287+0000","last_became_active":"2024-09-21T10:11:05.451075+0000","last_became_peered":"2024-09-21T10:11:05.451075+0000","last_unstale":"2024-09-21T10:13:30.584287+0000","last_undegraded":"2024-09-21T10:13:30.584287+0000","last_fullsized":"2024-09-21T10:13:30.584287+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:18.505045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":123,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-21T10:13:28.536012+0000","last_change":"2024-09-21T10:08:38.380508+0000","last_active":"2024-09-21T10:13:28.536012+0000","last_peered":"2024-09-21T10:13:28.536012+0000","last_clean":"2024-09-21T10:13:28.536012+0000","last_became_active":"2024-09-21T10:08:38.380226+0000","last_became_peered":"2024-09-21T10:08:38.380226+0000","last_unstale":"2024-09-21T10:13:28.536012+0000","last_undegraded":"2024-09-21T10:13:28.536012+0000","last_fullsized":"2024-09-21T10:13:28.536012+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:50:38.190118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":102,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.579697+0000","last_change":"2024-09-21T10:10:57.401588+0000","last_active":"2024-09-21T10:13:30.579697+0000","last_peered":"2024-09-21T10:13:30.579697+0000","last_clean":"2024-09-21T10:13:30.579697+0000","last_became_active":"2024-09-21T10:10:57.397724+0000","last_became_peered":"2024-09-21T10:10:57.397724+0000","last_unstale":"2024-09-21T10:13:30.579697+0000","last_undegraded":"2024-09-21T10:13:30.579697+0000","last_fullsized":"2024-09-21T10:13:30.579697+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:47.843630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":51838976,"num_objects":791,"num_object_clones":0,"num_object_copies":2373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":791,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224179,"num_write_kb":7198977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11576,"num_bytes_recovered":758644736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":283648000,"data_stored":283639856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102879,"ondisk_log_size":102879,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":11710,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26580,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":661927,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":91,"num_write":18,"num_write_kb":197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":668495,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626854,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":917736,"kb_used_data":17376,"kb_used_omap":1,"kb_used_meta":900350,"kb_avail":92827416,"statfs":{"total":95995035648,"available":95055273984,"internally_reserved":0,"allocated":17793024,"data_stored":16125869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":921958744},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019312,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":245,"seq":1052266987571,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":253148,"kb_used_data":36816,"kb_used_omap":8,"kb_used_meta":216247,"kb_avail":93492004,"statfs":{"total":95995035648,"available":95735812096,"internally_reserved":0,"allocated":37699584,"data_stored":36044632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8940,"internal_metadata":221437204},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":175,"seq":751619276920,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":437864,"kb_used_data":99100,"kb_used_omap":12,"kb_used_meta":338739,"kb_avail":93307288,"statfs":{"total":95995035648,"available":95546662912,"internally_reserved":0,"allocated":101478400,"data_stored":99815320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12845,"internal_metadata":346869203},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823083,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1513492,"kb_used_data":7372,"kb_used_omap":1,"kb_used_meta":1506110,"kb_avail":92231660,"statfs":{"total":95995035648,"available":94445219840,"internally_reserved":0,"allocated":7548928,"data_stored":5897935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1542256984},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879506,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":881560,"kb_used_data":64460,"kb_used_omap":4,"kb_used_meta":817083,"kb_avail":92863592,"statfs":{"total":95995035648,"available":95092318208,"internally_reserved":0,"allocated":66007040,"data_stored":64356128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":836693436},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566347,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":710440,"kb_used_data":4064,"kb_used_omap":6,"kb_used_meta":706361,"kb_avail":93034712,"statfs":{"total":95995035648,"available":95267545088,"internally_reserved":0,"allocated":4161536,"data_stored":2495500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":723314087},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860691,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1033672,"kb_used_data":134524,"kb_used_omap":5,"kb_used_meta":899130,"kb_avail":92711480,"statfs":{"total":95995035648,"available":94936555520,"internally_reserved":0,"allocated":137752576,"data_stored":136108705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":920709359},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7236,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9976,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26214400,"data_stored":26214400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":133763072,"data_stored":133759000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32505856,"data_stored":32505856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":7147520,"data_stored":7143448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":84017152,"data_stored":84017152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:13:37.260 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:37.286 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:37.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:37.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:37.355 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:37.388 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:37.412 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:37.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:37.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:37.514 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:37.538 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:37.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:37.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:37.640 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:37.663 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:37.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:37.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:37.765 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:37.788 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:37.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:37.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:37.890 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:37.912 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:37.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:37.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:37 smithi129 ceph-mon[88471]: from='client.46541 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:13:37.970 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:37 smithi107 ceph-mon[102221]: from='client.46541 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:13:37.970 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:37 smithi107 ceph-mon[105022]: from='client.46541 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:13:38.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:13:38.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:38.068 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:38.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:38.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:38.169 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:38.195 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:38.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:38.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:38.296 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:38.320 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:38.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:38.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:38.395 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_6 2024-09-21T10:13:38.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-09-21T10:13:38.422 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:38.460 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:38.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:38.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:38.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:38.583 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:38.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:38.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:38.684 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:38.709 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:38.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:38.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:38.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:38.833 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:38.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:38.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:38.847 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:38 smithi107 ceph-mon[102221]: pgmap v1034: 109 pgs: 109 active+clean; 1.7 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 689 op/s 2024-09-21T10:13:38.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:38 smithi107 ceph-mon[105022]: pgmap v1034: 109 pgs: 109 active+clean; 1.7 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 689 op/s 2024-09-21T10:13:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:38 smithi129 ceph-mon[88471]: pgmap v1034: 109 pgs: 109 active+clean; 1.7 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 689 op/s 2024-09-21T10:13:38.934 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:38.960 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:38.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:38.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:39.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:39.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:39.084 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:39.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:39.185 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:39.207 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:39.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:39.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:39.308 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:39.334 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:39.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:39.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:39.436 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:39.461 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:39.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:39.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:39.563 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:39.588 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:39.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:39.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:39.689 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:39.714 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:39.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:39.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:39.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:39.839 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:39.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:39.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:39.941 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:39.967 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:39.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:39.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:40.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:40.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:40.094 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:40.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:40.195 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:40.221 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:40.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:40.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:40.243 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:13:40.323 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:40.345 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:40.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:40.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:40.447 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:40.473 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:40.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:40.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:40.574 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:40.600 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:40.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:40.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:40.628 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:13:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:13:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:13:40.702 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:40.724 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:40.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:40.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:40.796 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:40.796 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:40.825 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:40.847 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:40.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:40.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:40.888 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:40 smithi129 ceph-mon[88471]: pgmap v1035: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 608 op/s 2024-09-21T10:13:40.891 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:40 smithi107 ceph-mon[102221]: pgmap v1035: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 608 op/s 2024-09-21T10:13:40.891 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:40 smithi107 ceph-mon[105022]: pgmap v1035: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 608 op/s 2024-09-21T10:13:40.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:40.970 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:40.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:40.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:41.072 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:41.094 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:41.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:41.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:41.196 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:41.217 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:41.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:41.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:41.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:41.342 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:41.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:41.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:41.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:41.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:41.465 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:41.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:41.566 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:41.587 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:41.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:41.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:41.688 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:41.709 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:41.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:41.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:41.811 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:41.832 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:41.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:41.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:41.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:13:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:13:41] "GET /metrics HTTP/1.1" 200 36777 "" "Prometheus/2.51.0" 2024-09-21T10:13:41.934 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:41.955 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:41.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:41.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:42.057 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:42.078 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:42.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:42.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:42.110 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:13:42.110 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:13:42.179 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:42.200 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:42.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:42.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:42.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:42.324 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:42.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:42.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:42.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:42.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:42.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:42 smithi107 ceph-mon[102221]: pgmap v1036: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 484 op/s 2024-09-21T10:13:42.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:42.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:42.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:42.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:42 smithi107 ceph-mon[105022]: pgmap v1036: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 484 op/s 2024-09-21T10:13:42.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:42 smithi129 ceph-mon[88471]: pgmap v1036: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 484 op/s 2024-09-21T10:13:42.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:42.426 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:42.448 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:42.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:42.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:42.549 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:42.571 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:42.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:42.672 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:42.694 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:42.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:42.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:42.796 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:42.817 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:42.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:42.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:42.873 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:13:42.918 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:42.940 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:42.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:42.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:43.041 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:43.063 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:43.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:43.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:43.165 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:43.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:43.187 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:43.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:43.288 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:43.309 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:43.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:43.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:43.411 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:43.433 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:43.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:43.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:43.534 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:43.556 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:43.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:43.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:43.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:43.679 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:43.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:43.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:43.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:43.803 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:43.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:43.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:43.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:43.927 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:43.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:43.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:44.029 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:44.052 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:44.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:44.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:44.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:44.175 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:44.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:44.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:44.276 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:44.297 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:44.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:44.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:44.384 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:13:44.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:44.420 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:44.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:44.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:44.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:44.543 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:44.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:44.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:44.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:44.666 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:44.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:44.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:44 smithi129 ceph-mon[88471]: pgmap v1037: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 434 op/s 2024-09-21T10:13:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:44 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1687334681' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:44.757 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:44 smithi107 ceph-mon[102221]: pgmap v1037: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 434 op/s 2024-09-21T10:13:44.757 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:44 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1687334681' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:44.758 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:44 smithi107 ceph-mon[105022]: pgmap v1037: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 434 op/s 2024-09-21T10:13:44.758 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:44 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1687334681' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:13:44.768 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:44.794 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:44.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:44.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:44.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:44.917 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:44.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:44.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:45.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:45.040 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:45.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:45.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:45.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:45.164 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:45.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:45.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:45.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:45.287 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:45.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:45.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:45.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:45.410 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:45.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:45.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:45.512 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:45.536 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:45.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:45.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:45 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:13:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:45 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1687334681' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:13:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:45 smithi107 ceph-mon[102221]: osdmap e292: 8 total, 8 up, 7 in 2024-09-21T10:13:45.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:45 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:13:45.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:45 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1687334681' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:13:45.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:45 smithi107 ceph-mon[105022]: osdmap e292: 8 total, 8 up, 7 in 2024-09-21T10:13:45.638 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:45.662 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:45.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:45.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:45 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:13:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:45 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1687334681' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:13:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:45 smithi129 ceph-mon[88471]: osdmap e292: 8 total, 8 up, 7 in 2024-09-21T10:13:45.764 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:45.787 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:45.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:45.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:45.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:45.909 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:45.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:45.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:46.011 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:46.035 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:46.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:46.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:46.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:46.159 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:46.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:46.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:46.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:46.281 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:46.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:46.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:46.383 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:46.407 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:46.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:46.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:46.509 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:46.532 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:46.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:46.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:46.542 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:13:46.542 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:13:46.543 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:46.543 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:46.559 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:13:46.560 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:13:46.581 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:13:46.581 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:13:46.582 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:46 smithi107 ceph-mon[105022]: pgmap v1039: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 79 op/s 2024-09-21T10:13:46.582 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:46 smithi107 ceph-mon[102221]: pgmap v1039: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 79 op/s 2024-09-21T10:13:46.598 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:13:46.598 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:13:46.613 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:13:46.614 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:13:46.629 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:13:46.630 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:13:46.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:46.646 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:13:46.646 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:13:46.655 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:46.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:46.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:46.662 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:13:46.662 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:13:46.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:46 smithi129 ceph-mon[88471]: pgmap v1039: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 79 op/s 2024-09-21T10:13:46.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:46.777 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:46.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:46.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:46.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:46.901 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:46.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:46.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:47.002 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:47.028 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:47.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:47.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:47.129 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:47.158 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:47.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:47.190 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:13:47.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:47.281 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:47.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:47.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:47.382 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:47.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:47.408 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:47.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:47.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:47.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:47.532 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:47.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:47.587 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:47 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2663684883' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:47.587 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:47 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2663684883' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:47.633 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:47.659 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:47.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:47.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:47 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2663684883' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:13:47.761 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:47.787 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:47.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:47.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:47.889 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:47.910 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:47.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:47.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:48.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:48.032 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:48.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:48.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:48.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:48.156 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:48.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:48.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:48.257 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:48.283 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:48.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:48.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:48.385 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:48.411 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:48.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:48.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:48.437 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:48 smithi107 ceph-mon[102221]: pgmap v1040: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 1 op/s 2024-09-21T10:13:48.437 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:48 smithi107 ceph-mon[105022]: pgmap v1040: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 1 op/s 2024-09-21T10:13:48.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:48.535 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:48.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:48.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:48.636 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:48.662 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:48.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:48.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:48.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:48 smithi129 ceph-mon[88471]: pgmap v1040: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 1 op/s 2024-09-21T10:13:48.763 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:48.785 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:48.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:48.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:48.887 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:48.913 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:48.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:48.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:49.015 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:49.040 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:49.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:49.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:49.141 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:49.166 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:49.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:49.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:49.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:49.291 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:49.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:49.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:49.393 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:49.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:49.420 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:49.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:49.521 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:49.546 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:49.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:49.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:49.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:13:49.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:49.678 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:49.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:49.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:49.780 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:49.805 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:49.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:49.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:49.907 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:49.929 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:49.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:49.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:50.031 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:50.052 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:50.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:50.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:50.154 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:50.179 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:50.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:50.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:50.281 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:50.306 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:50.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:50.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:50.404 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_6' removed 2024-09-21T10:13:50.407 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:50.448 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:50.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:50.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:50.549 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:50.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:50 smithi107 ceph-mon[102221]: pgmap v1041: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:13:50.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:50 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2193070287' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:13:50.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:50 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:13:50.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:50 smithi107 ceph-mon[105022]: pgmap v1041: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:13:50.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:50 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2193070287' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:13:50.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:50 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:13:50.571 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:50.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:50.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:50 smithi129 ceph-mon[88471]: pgmap v1041: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:13:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:50 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2193070287' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:13:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:50 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:13:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:13:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:13:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:13:50.673 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:50.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:50.695 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:50.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:50.797 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:50.830 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:50.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:50.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:50.932 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:50.953 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:50.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:50.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:51.055 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:51.076 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:51.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:51.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:51.178 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:51.203 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:51.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:51.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:51.297 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:13:51.298 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:13:51.298 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_7 2024-09-21T10:13:51.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_7 16 2024-09-21T10:13:51.305 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:51.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:51.329 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:51.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:51.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:51.451 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:51.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:51.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:51.553 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:51.575 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:51.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:51.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:51.591 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:51 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:13:51.591 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:51 smithi107 ceph-mon[102221]: osdmap e293: 8 total, 8 up, 7 in 2024-09-21T10:13:51.591 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:51 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:13:51.591 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:51 smithi107 ceph-mon[105022]: osdmap e293: 8 total, 8 up, 7 in 2024-09-21T10:13:51.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:51 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:13:51.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:51 smithi129 ceph-mon[88471]: osdmap e293: 8 total, 8 up, 7 in 2024-09-21T10:13:51.676 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:51.698 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:51.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:51.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:51.799 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:51.820 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:51.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:51.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:51.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:13:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:13:51] "GET /metrics HTTP/1.1" 200 36777 "" "Prometheus/2.51.0" 2024-09-21T10:13:51.923 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:51.945 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:51.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:51.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:52.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:52.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:52.071 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:52.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:52.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:52.188 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:13:52.189 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:13:52.195 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:52.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:52.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:52.287 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:13:52.288 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-09-21T10:13:52.288 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-09-21T10:13:52.288 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:13:52.288 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.288 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:246850 10.2 10:40049bc0:::benchmark_data_smithi107_184135_object115284:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-09-21T10:13:52.288 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:48.368909+0000", 2024-09-21T10:13:52.288 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 303.914968714, 2024-09-21T10:13:52.288 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.048257115, 2024-09-21T10:13:52.288 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:52.288 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:52.288 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:13:52.288 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:52.289 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:13:52.289 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:13:52.289 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 246850 2024-09-21T10:13:52.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.289 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:52.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.289 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:52.289 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.368909+0000", 2024-09-21T10:13:52.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.289 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.289 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.289 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:52.289 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.368909+0000", 2024-09-21T10:13:52.289 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:52.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.368911+0000", 2024-09-21T10:13:52.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9069999999999999e-06 2024-09-21T10:13:52.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:52.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.368917+0000", 2024-09-21T10:13:52.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8780000000000003e-06 2024-09-21T10:13:52.290 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.290 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:52.290 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.368935+0000", 2024-09-21T10:13:52.290 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7731999999999999e-05 2024-09-21T10:13:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.368943+0000", 2024-09-21T10:13:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0579999999999998e-06 2024-09-21T10:13:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.369024+0000", 2024-09-21T10:13:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0993000000000002e-05 2024-09-21T10:13:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.369033+0000", 2024-09-21T10:13:52.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8899999999999996e-06 2024-09-21T10:13:52.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.292 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.292 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.408152+0000", 2024-09-21T10:13:52.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.039119447 2024-09-21T10:13:52.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.292 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:52.292 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.408242+0000", 2024-09-21T10:13:52.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0333000000000004e-05 2024-09-21T10:13:52.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:13:52.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.408534+0000", 2024-09-21T10:13:52.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00029153600000000001 2024-09-21T10:13:52.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.416843+0000", 2024-09-21T10:13:52.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0083089470000000006 2024-09-21T10:13:52.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:52.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.416872+0000", 2024-09-21T10:13:52.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9039000000000001e-05 2024-09-21T10:13:52.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.416935+0000", 2024-09-21T10:13:52.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3071999999999995e-05 2024-09-21T10:13:52.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:13:52.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.417027+0000", 2024-09-21T10:13:52.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2173e-05 2024-09-21T10:13:52.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:13:52.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.417166+0000", 2024-09-21T10:13:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013910999999999999 2024-09-21T10:13:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:246851 10.2 10:43f3cf20:::benchmark_data_smithi107_184135_object115285:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-09-21T10:13:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:48.368965+0000", 2024-09-21T10:13:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 303.91491310800001, 2024-09-21T10:13:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0537696480000001, 2024-09-21T10:13:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:13:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:13:52.295 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:13:52.296 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 246851 2024-09-21T10:13:52.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.296 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:52.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.296 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:52.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.368965+0000", 2024-09-21T10:13:52.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.296 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:52.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.368965+0000", 2024-09-21T10:13:52.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.368965+0000", 2024-09-21T10:13:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8500000000000002e-07 2024-09-21T10:13:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.368968+0000", 2024-09-21T10:13:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.514e-06 2024-09-21T10:13:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.368975+0000", 2024-09-21T10:13:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.674e-06 2024-09-21T10:13:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:52.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.368980+0000", 2024-09-21T10:13:52.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2979999999999999e-06 2024-09-21T10:13:52.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.369057+0000", 2024-09-21T10:13:52.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7582000000000004e-05 2024-09-21T10:13:52.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.369064+0000", 2024-09-21T10:13:52.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2840000000000003e-06 2024-09-21T10:13:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.412908+0000", 2024-09-21T10:13:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.043844572 2024-09-21T10:13:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.413036+0000", 2024-09-21T10:13:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000127534 2024-09-21T10:13:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:13:52.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.413346+0000", 2024-09-21T10:13:52.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000310372 2024-09-21T10:13:52.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.422319+0000", 2024-09-21T10:13:52.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0089730729999999998 2024-09-21T10:13:52.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:52.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.422437+0000", 2024-09-21T10:13:52.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011729499999999999 2024-09-21T10:13:52.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.422475+0000", 2024-09-21T10:13:52.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8754000000000002e-05 2024-09-21T10:13:52.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:13:52.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.422544+0000", 2024-09-21T10:13:52.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9157000000000002e-05 2024-09-21T10:13:52.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:13:52.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.422734+0000", 2024-09-21T10:13:52.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019005399999999999 2024-09-21T10:13:52.301 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.302 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.302 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.302 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:246867 10.2 10:43384b5b:::benchmark_data_smithi107_184135_object115301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-09-21T10:13:52.302 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:48.382804+0000", 2024-09-21T10:13:52.302 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 303.90107364699998, 2024-09-21T10:13:52.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0403223690000001, 2024-09-21T10:13:52.302 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:52.302 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:52.302 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:13:52.302 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:52.302 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:13:52.303 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:13:52.303 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 246867 2024-09-21T10:13:52.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.303 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:52.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:52.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.382804+0000", 2024-09-21T10:13:52.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:52.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.382804+0000", 2024-09-21T10:13:52.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:52.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.382806+0000", 2024-09-21T10:13:52.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6109999999999999e-06 2024-09-21T10:13:52.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:52.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.382811+0000", 2024-09-21T10:13:52.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3680000000000001e-06 2024-09-21T10:13:52.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:52.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.382834+0000", 2024-09-21T10:13:52.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2517000000000001e-05 2024-09-21T10:13:52.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:52.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.382841+0000", 2024-09-21T10:13:52.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9240000000000001e-06 2024-09-21T10:13:52.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.382968+0000", 2024-09-21T10:13:52.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012731999999999999 2024-09-21T10:13:52.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.382976+0000", 2024-09-21T10:13:52.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4169999999999992e-06 2024-09-21T10:13:52.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.414878+0000", 2024-09-21T10:13:52.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0319018339999999 2024-09-21T10:13:52.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:52.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.414943+0000", 2024-09-21T10:13:52.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4743e-05 2024-09-21T10:13:52.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:13:52.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.415020+0000", 2024-09-21T10:13:52.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6616e-05 2024-09-21T10:13:52.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:52.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.422744+0000", 2024-09-21T10:13:52.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0077243520000000003 2024-09-21T10:13:52.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.422791+0000", 2024-09-21T10:13:52.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7315999999999999e-05 2024-09-21T10:13:52.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.422892+0000", 2024-09-21T10:13:52.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010027900000000001 2024-09-21T10:13:52.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:13:52.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.422975+0000", 2024-09-21T10:13:52.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3584999999999993e-05 2024-09-21T10:13:52.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:13:52.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.423127+0000", 2024-09-21T10:13:52.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015148700000000001 2024-09-21T10:13:52.308 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.308 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.308 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.309 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36210.0:246869 10.2 10:4dbb55c5:::benchmark_data_smithi107_184135_object115303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-09-21T10:13:52.309 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:08:48.383785+0000", 2024-09-21T10:13:52.309 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 303.90009265100002, 2024-09-21T10:13:52.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0397199509999999, 2024-09-21T10:13:52.309 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:52.309 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:52.309 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:13:52.309 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:52.309 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36210", 2024-09-21T10:13:52.309 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:13:52.309 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 246869 2024-09-21T10:13:52.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.310 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:52.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.310 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:52.310 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.383785+0000", 2024-09-21T10:13:52.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.310 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.310 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:52.310 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.383785+0000", 2024-09-21T10:13:52.310 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.310 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:52.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.383786+0000", 2024-09-21T10:13:52.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1200000000000003e-07 2024-09-21T10:13:52.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:52.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.383789+0000", 2024-09-21T10:13:52.311 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7750000000000001e-06 2024-09-21T10:13:52.311 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.311 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.311 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:52.311 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.383796+0000", 2024-09-21T10:13:52.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.571e-06 2024-09-21T10:13:52.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:52.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.383802+0000", 2024-09-21T10:13:52.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.874e-06 2024-09-21T10:13:52.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.312 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.383856+0000", 2024-09-21T10:13:52.312 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4135000000000003e-05 2024-09-21T10:13:52.312 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.312 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.312 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:48.383865+0000", 2024-09-21T10:13:52.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9050000000000003e-06 2024-09-21T10:13:52.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.417205+0000", 2024-09-21T10:13:52.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0333393470000001 2024-09-21T10:13:52.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.313 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:52.313 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.417318+0000", 2024-09-21T10:13:52.313 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000113383 2024-09-21T10:13:52.313 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.313 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:13:52.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.417452+0000", 2024-09-21T10:13:52.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000133647 2024-09-21T10:13:52.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:52.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.422749+0000", 2024-09-21T10:13:52.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052978849999999996 2024-09-21T10:13:52.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.314 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.422840+0000", 2024-09-21T10:13:52.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0093999999999995e-05 2024-09-21T10:13:52.314 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.314 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.423223+0000", 2024-09-21T10:13:52.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038303600000000001 2024-09-21T10:13:52.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:13:52.315 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:52.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.423384+0000", 2024-09-21T10:13:52.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016191999999999999 2024-09-21T10:13:52.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:13:52.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:08:49.423505+0000", 2024-09-21T10:13:52.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000120767 2024-09-21T10:13:52.317 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.317 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.317 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.317 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91069 11.8 11:1d9ce1cd:::benchmark_data_smithi107_218812_object91068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-21T10:13:52.317 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.369573+0000", 2024-09-21T10:13:52.317 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 175.91430449800001, 2024-09-21T10:13:52.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3066178669999999, 2024-09-21T10:13:52.317 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:52.317 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:52.317 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:13:52.318 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:52.318 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:52.318 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:52.318 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91069 2024-09-21T10:13:52.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.318 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:52.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:52.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.369573+0000", 2024-09-21T10:13:52.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:52.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.369573+0000", 2024-09-21T10:13:52.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:52.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.369575+0000", 2024-09-21T10:13:52.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.612e-06 2024-09-21T10:13:52.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:52.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.369624+0000", 2024-09-21T10:13:52.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9036999999999998e-05 2024-09-21T10:13:52.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:52.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.369645+0000", 2024-09-21T10:13:52.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0599999999999999e-05 2024-09-21T10:13:52.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:52.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.369653+0000", 2024-09-21T10:13:52.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1890000000000007e-06 2024-09-21T10:13:52.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.376149+0000", 2024-09-21T10:13:52.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0064962070000000004 2024-09-21T10:13:52.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:52.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.376301+0000", 2024-09-21T10:13:52.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000151757 2024-09-21T10:13:52.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:13:52.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.376460+0000", 2024-09-21T10:13:52.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000159586 2024-09-21T10:13:52.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:52.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379481+0000", 2024-09-21T10:13:52.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0030210580000000001 2024-09-21T10:13:52.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.386415+0000", 2024-09-21T10:13:52.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069336040000000003 2024-09-21T10:13:52.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.403651+0000", 2024-09-21T10:13:52.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017236089 2024-09-21T10:13:52.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.403662+0000", 2024-09-21T10:13:52.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0421999999999999e-05 2024-09-21T10:13:52.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384390+0000", 2024-09-21T10:13:52.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.98072825299999999 2024-09-21T10:13:52.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384403+0000", 2024-09-21T10:13:52.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3186e-05 2024-09-21T10:13:52.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.676117+0000", 2024-09-21T10:13:52.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.291714217 2024-09-21T10:13:52.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.324 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:13:52.324 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.676191+0000", 2024-09-21T10:13:52.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4049999999999997e-05 2024-09-21T10:13:52.324 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.324 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.324 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.324 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91074 11.a 11:5a4cfa77:::benchmark_data_smithi107_218812_object91073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-21T10:13:52.324 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.379421+0000", 2024-09-21T10:13:52.324 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 175.90445738599999, 2024-09-21T10:13:52.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3240689140000002, 2024-09-21T10:13:52.325 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:52.325 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:52.325 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:13:52.325 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:52.325 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:52.325 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:52.325 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91074 2024-09-21T10:13:52.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.325 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:52.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:52.325 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379421+0000", 2024-09-21T10:13:52.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:52.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379421+0000", 2024-09-21T10:13:52.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:52.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379422+0000", 2024-09-21T10:13:52.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.513e-06 2024-09-21T10:13:52.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:52.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379443+0000", 2024-09-21T10:13:52.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1273000000000001e-05 2024-09-21T10:13:52.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:52.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379455+0000", 2024-09-21T10:13:52.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.203e-05 2024-09-21T10:13:52.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:52.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379463+0000", 2024-09-21T10:13:52.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3329999999999999e-06 2024-09-21T10:13:52.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.386937+0000", 2024-09-21T10:13:52.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0074744290000000003 2024-09-21T10:13:52.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:52.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.387075+0000", 2024-09-21T10:13:52.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013819900000000001 2024-09-21T10:13:52.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:13:52.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.387218+0000", 2024-09-21T10:13:52.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000142705 2024-09-21T10:13:52.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:52.329 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.393122+0000", 2024-09-21T10:13:52.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005903747 2024-09-21T10:13:52.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.329 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.412062+0000", 2024-09-21T10:13:52.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018940354 2024-09-21T10:13:52.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.329 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.412074+0000", 2024-09-21T10:13:52.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2333e-05 2024-09-21T10:13:52.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.330 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385002+0000", 2024-09-21T10:13:52.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.97292794599999999 2024-09-21T10:13:52.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.330 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385014+0000", 2024-09-21T10:13:52.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.182e-05 2024-09-21T10:13:52.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.687683+0000", 2024-09-21T10:13:52.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.302668766 2024-09-21T10:13:52.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for degraded object", 2024-09-21T10:13:52.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.688842+0000", 2024-09-21T10:13:52.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011593440000000001 2024-09-21T10:13:52.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.703390+0000", 2024-09-21T10:13:52.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014547396000000001 2024-09-21T10:13:52.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:13:52.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.703489+0000", 2024-09-21T10:13:52.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9726e-05 2024-09-21T10:13:52.332 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.332 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.332 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.332 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91083 11.8 11:1024c88d:::benchmark_data_smithi107_218812_object91082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:13:52.332 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.401346+0000", 2024-09-21T10:13:52.332 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 175.88253234000001, 2024-09-21T10:13:52.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2836915819999999, 2024-09-21T10:13:52.332 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:52.332 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:52.333 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:13:52.333 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:52.333 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:52.333 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:52.333 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91083 2024-09-21T10:13:52.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.333 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:52.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.333 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:52.333 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401346+0000", 2024-09-21T10:13:52.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.333 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:52.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401346+0000", 2024-09-21T10:13:52.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:52.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401347+0000", 2024-09-21T10:13:52.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7010000000000001e-06 2024-09-21T10:13:52.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:52.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401621+0000", 2024-09-21T10:13:52.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00027367500000000003 2024-09-21T10:13:52.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:52.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401632+0000", 2024-09-21T10:13:52.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0723000000000001e-05 2024-09-21T10:13:52.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:52.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401640+0000", 2024-09-21T10:13:52.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9880000000000004e-06 2024-09-21T10:13:52.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404683+0000", 2024-09-21T10:13:52.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0030434799999999999 2024-09-21T10:13:52.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404691+0000", 2024-09-21T10:13:52.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0099999999999995e-06 2024-09-21T10:13:52.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384431+0000", 2024-09-21T10:13:52.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.97973983200000003 2024-09-21T10:13:52.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384438+0000", 2024-09-21T10:13:52.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8979999999999999e-06 2024-09-21T10:13:52.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.676420+0000", 2024-09-21T10:13:52.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.291981727 2024-09-21T10:13:52.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:52.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.676514+0000", 2024-09-21T10:13:52.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3908999999999998e-05 2024-09-21T10:13:52.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:13:52.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.676614+0000", 2024-09-21T10:13:52.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000100746 2024-09-21T10:13:52.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.681917+0000", 2024-09-21T10:13:52.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0053031550000000004 2024-09-21T10:13:52.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:52.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.682073+0000", 2024-09-21T10:13:52.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015567900000000001 2024-09-21T10:13:52.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.682113+0000", 2024-09-21T10:13:52.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9536999999999998e-05 2024-09-21T10:13:52.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.684892+0000", 2024-09-21T10:13:52.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027794719999999998 2024-09-21T10:13:52.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:13:52.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.684940+0000", 2024-09-21T10:13:52.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7732000000000003e-05 2024-09-21T10:13:52.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:13:52.340 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.685037+0000", 2024-09-21T10:13:52.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7318000000000006e-05 2024-09-21T10:13:52.340 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.340 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.340 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.340 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91086 11.8 11:14b98868:::benchmark_data_smithi107_218812_object91085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:13:52.340 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.401700+0000", 2024-09-21T10:13:52.340 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 175.88217767399999, 2024-09-21T10:13:52.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2935442479999999, 2024-09-21T10:13:52.340 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:52.340 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:52.340 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:13:52.340 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:52.341 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:52.341 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:52.341 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91086 2024-09-21T10:13:52.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.341 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:52.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:52.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401700+0000", 2024-09-21T10:13:52.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:52.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401700+0000", 2024-09-21T10:13:52.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:52.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401701+0000", 2024-09-21T10:13:52.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6899999999999997e-07 2024-09-21T10:13:52.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:52.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401935+0000", 2024-09-21T10:13:52.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000233957 2024-09-21T10:13:52.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:52.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401944+0000", 2024-09-21T10:13:52.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.799e-06 2024-09-21T10:13:52.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:52.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401950+0000", 2024-09-21T10:13:52.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6499999999999999e-06 2024-09-21T10:13:52.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404712+0000", 2024-09-21T10:13:52.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027615949999999999 2024-09-21T10:13:52.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404718+0000", 2024-09-21T10:13:52.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9359999999999996e-06 2024-09-21T10:13:52.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384462+0000", 2024-09-21T10:13:52.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.97974425499999995 2024-09-21T10:13:52.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384470+0000", 2024-09-21T10:13:52.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5209999999999997e-06 2024-09-21T10:13:52.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.679346+0000", 2024-09-21T10:13:52.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2948768799999999 2024-09-21T10:13:52.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:52.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.679498+0000", 2024-09-21T10:13:52.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015129 2024-09-21T10:13:52.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:13:52.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.679670+0000", 2024-09-21T10:13:52.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017223 2024-09-21T10:13:52.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:52.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.684756+0000", 2024-09-21T10:13:52.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0050856470000000004 2024-09-21T10:13:52.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.684829+0000", 2024-09-21T10:13:52.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3808999999999998e-05 2024-09-21T10:13:52.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695008+0000", 2024-09-21T10:13:52.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010178159000000001 2024-09-21T10:13:52.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695080+0000", 2024-09-21T10:13:52.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2880999999999996e-05 2024-09-21T10:13:52.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:13:52.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695139+0000", 2024-09-21T10:13:52.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8390000000000002e-05 2024-09-21T10:13:52.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:13:52.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695245+0000", 2024-09-21T10:13:52.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010568 2024-09-21T10:13:52.348 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.348 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.348 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.348 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91090 11.a 11:532a4ae5:::benchmark_data_smithi107_218812_object91089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:13:52.348 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.404927+0000", 2024-09-21T10:13:52.348 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 175.87895116000001, 2024-09-21T10:13:52.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.299055724, 2024-09-21T10:13:52.348 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:52.348 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:52.348 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:13:52.348 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:52.349 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:52.349 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:52.349 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91090 2024-09-21T10:13:52.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.349 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:52.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:52.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404927+0000", 2024-09-21T10:13:52.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:52.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404927+0000", 2024-09-21T10:13:52.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:52.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404929+0000", 2024-09-21T10:13:52.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9750000000000001e-06 2024-09-21T10:13:52.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:52.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404955+0000", 2024-09-21T10:13:52.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5783e-05 2024-09-21T10:13:52.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:52.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404965+0000", 2024-09-21T10:13:52.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0400999999999999e-05 2024-09-21T10:13:52.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:52.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404973+0000", 2024-09-21T10:13:52.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4479999999999994e-06 2024-09-21T10:13:52.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.413662+0000", 2024-09-21T10:13:52.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0086882260000000003 2024-09-21T10:13:52.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.413672+0000", 2024-09-21T10:13:52.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0008e-05 2024-09-21T10:13:52.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385041+0000", 2024-09-21T10:13:52.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.97136919799999999 2024-09-21T10:13:52.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385049+0000", 2024-09-21T10:13:52.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.49e-06 2024-09-21T10:13:52.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.689104+0000", 2024-09-21T10:13:52.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.304054893 2024-09-21T10:13:52.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:52.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.689228+0000", 2024-09-21T10:13:52.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000124058 2024-09-21T10:13:52.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:13:52.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.689383+0000", 2024-09-21T10:13:52.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000154857 2024-09-21T10:13:52.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.700561+0000", 2024-09-21T10:13:52.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011177678 2024-09-21T10:13:52.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:52.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.700579+0000", 2024-09-21T10:13:52.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7887e-05 2024-09-21T10:13:52.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.701752+0000", 2024-09-21T10:13:52.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011736050000000001 2024-09-21T10:13:52.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.703561+0000", 2024-09-21T10:13:52.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001809045 2024-09-21T10:13:52.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:13:52.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.703884+0000", 2024-09-21T10:13:52.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000322715 2024-09-21T10:13:52.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:13:52.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.703983+0000", 2024-09-21T10:13:52.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8456999999999996e-05 2024-09-21T10:13:52.356 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.356 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.356 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.356 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91107 11.8 11:16a99f3f:::benchmark_data_smithi107_218812_object91106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:13:52.356 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.432894+0000", 2024-09-21T10:13:52.356 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 175.850983595, 2024-09-21T10:13:52.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2626001310000001, 2024-09-21T10:13:52.356 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:52.356 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:52.356 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:13:52.356 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:52.357 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:52.357 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:52.357 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91107 2024-09-21T10:13:52.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.357 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:52.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:52.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.432894+0000", 2024-09-21T10:13:52.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:52.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.432894+0000", 2024-09-21T10:13:52.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:52.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.432896+0000", 2024-09-21T10:13:52.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.302e-06 2024-09-21T10:13:52.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:52.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.432919+0000", 2024-09-21T10:13:52.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2988000000000001e-05 2024-09-21T10:13:52.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:52.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.432937+0000", 2024-09-21T10:13:52.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8147000000000001e-05 2024-09-21T10:13:52.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:52.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.432944+0000", 2024-09-21T10:13:52.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6809999999999998e-06 2024-09-21T10:13:52.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.433008+0000", 2024-09-21T10:13:52.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3972000000000003e-05 2024-09-21T10:13:52.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.433025+0000", 2024-09-21T10:13:52.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6109000000000001e-05 2024-09-21T10:13:52.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384493+0000", 2024-09-21T10:13:52.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.95146804900000004 2024-09-21T10:13:52.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384499+0000", 2024-09-21T10:13:52.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8040000000000004e-06 2024-09-21T10:13:52.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.682149+0000", 2024-09-21T10:13:52.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.297649716 2024-09-21T10:13:52.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:52.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.682241+0000", 2024-09-21T10:13:52.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1515999999999996e-05 2024-09-21T10:13:52.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:13:52.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.682337+0000", 2024-09-21T10:13:52.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6506000000000001e-05 2024-09-21T10:13:52.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.362 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.685192+0000", 2024-09-21T10:13:52.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028547640000000001 2024-09-21T10:13:52.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.362 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:52.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.685203+0000", 2024-09-21T10:13:52.362 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1555e-05 2024-09-21T10:13:52.362 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.362 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.362 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695046+0000", 2024-09-21T10:13:52.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098422700000000002 2024-09-21T10:13:52.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695355+0000", 2024-09-21T10:13:52.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000309005 2024-09-21T10:13:52.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:13:52.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695402+0000", 2024-09-21T10:13:52.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7172999999999997e-05 2024-09-21T10:13:52.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:13:52.364 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695494+0000", 2024-09-21T10:13:52.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2573999999999998e-05 2024-09-21T10:13:52.364 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.364 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.364 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.364 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91117 11.a 11:54639fb7:::benchmark_data_smithi107_218812_object91116:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:13:52.364 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.443321+0000", 2024-09-21T10:13:52.364 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 175.84055686599999, 2024-09-21T10:13:52.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2608621439999999, 2024-09-21T10:13:52.364 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:52.364 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:52.364 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:13:52.365 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:52.365 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:52.365 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:52.365 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91117 2024-09-21T10:13:52.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.365 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:52.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:52.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443321+0000", 2024-09-21T10:13:52.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:52.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443321+0000", 2024-09-21T10:13:52.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.366 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:52.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443323+0000", 2024-09-21T10:13:52.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.604e-06 2024-09-21T10:13:52.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.366 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:52.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443345+0000", 2024-09-21T10:13:52.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2370000000000001e-05 2024-09-21T10:13:52.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.366 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:52.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443358+0000", 2024-09-21T10:13:52.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3047999999999999e-05 2024-09-21T10:13:52.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:52.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443366+0000", 2024-09-21T10:13:52.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1089999999999998e-06 2024-09-21T10:13:52.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443456+0000", 2024-09-21T10:13:52.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0204999999999994e-05 2024-09-21T10:13:52.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443464+0000", 2024-09-21T10:13:52.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.323e-06 2024-09-21T10:13:52.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385072+0000", 2024-09-21T10:13:52.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.94160824399999998 2024-09-21T10:13:52.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385079+0000", 2024-09-21T10:13:52.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3880000000000002e-06 2024-09-21T10:13:52.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.692299+0000", 2024-09-21T10:13:52.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3072192389999999 2024-09-21T10:13:52.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:52.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.692459+0000", 2024-09-21T10:13:52.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000160686 2024-09-21T10:13:52.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:13:52.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.692664+0000", 2024-09-21T10:13:52.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000204692 2024-09-21T10:13:52.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:52.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.700714+0000", 2024-09-21T10:13:52.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.008050092 2024-09-21T10:13:52.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.700959+0000", 2024-09-21T10:13:52.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024535799999999999 2024-09-21T10:13:52.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.701790+0000", 2024-09-21T10:13:52.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00083083300000000001 2024-09-21T10:13:52.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704027+0000", 2024-09-21T10:13:52.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022363589999999998 2024-09-21T10:13:52.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:13:52.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704086+0000", 2024-09-21T10:13:52.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9502000000000001e-05 2024-09-21T10:13:52.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.371 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:13:52.371 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704183+0000", 2024-09-21T10:13:52.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7091999999999998e-05 2024-09-21T10:13:52.372 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.372 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.372 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.372 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91118 11.a 11:56ad5097:::benchmark_data_smithi107_218812_object91117:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:13:52.372 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.443509+0000", 2024-09-21T10:13:52.372 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 175.84036867899999, 2024-09-21T10:13:52.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2609485060000001, 2024-09-21T10:13:52.372 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:52.372 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:52.372 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:13:52.372 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:52.372 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:52.372 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:52.373 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91118 2024-09-21T10:13:52.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.373 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:52.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:52.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443509+0000", 2024-09-21T10:13:52.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:52.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443509+0000", 2024-09-21T10:13:52.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.374 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:52.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443510+0000", 2024-09-21T10:13:52.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6799999999999999e-07 2024-09-21T10:13:52.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.374 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:52.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443530+0000", 2024-09-21T10:13:52.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9551e-05 2024-09-21T10:13:52.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.374 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:52.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443543+0000", 2024-09-21T10:13:52.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3575e-05 2024-09-21T10:13:52.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:52.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443550+0000", 2024-09-21T10:13:52.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5969999999999997e-06 2024-09-21T10:13:52.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443616+0000", 2024-09-21T10:13:52.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6740000000000001e-05 2024-09-21T10:13:52.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443624+0000", 2024-09-21T10:13:52.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8220000000000007e-06 2024-09-21T10:13:52.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385103+0000", 2024-09-21T10:13:52.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.94147888199999996 2024-09-21T10:13:52.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385111+0000", 2024-09-21T10:13:52.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.486e-06 2024-09-21T10:13:52.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695503+0000", 2024-09-21T10:13:52.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3103919070000001 2024-09-21T10:13:52.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:52.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695648+0000", 2024-09-21T10:13:52.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014547799999999999 2024-09-21T10:13:52.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:13:52.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.696550+0000", 2024-09-21T10:13:52.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00090179799999999999 2024-09-21T10:13:52.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:52.378 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.700722+0000", 2024-09-21T10:13:52.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041717899999999999 2024-09-21T10:13:52.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.378 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.378 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.378 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.701004+0000", 2024-09-21T10:13:52.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00028281900000000002 2024-09-21T10:13:52.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.378 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.378 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.378 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.701834+0000", 2024-09-21T10:13:52.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00082920400000000005 2024-09-21T10:13:52.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.379 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704324+0000", 2024-09-21T10:13:52.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002490632 2024-09-21T10:13:52.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:13:52.379 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704377+0000", 2024-09-21T10:13:52.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2439999999999999e-05 2024-09-21T10:13:52.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:13:52.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704458+0000", 2024-09-21T10:13:52.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1017e-05 2024-09-21T10:13:52.380 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.380 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.380 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.380 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91121 11.8 11:16dab647:::benchmark_data_smithi107_218812_object91120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:13:52.380 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.447395+0000", 2024-09-21T10:13:52.380 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 175.83648249500001, 2024-09-21T10:13:52.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.248324582, 2024-09-21T10:13:52.380 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:52.380 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:52.380 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:13:52.380 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:52.381 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:52.381 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:52.381 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91121 2024-09-21T10:13:52.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.381 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:52.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:52.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447395+0000", 2024-09-21T10:13:52.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:52.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447395+0000", 2024-09-21T10:13:52.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.382 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:52.382 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447397+0000", 2024-09-21T10:13:52.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2330000000000001e-06 2024-09-21T10:13:52.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.382 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:52.382 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447448+0000", 2024-09-21T10:13:52.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1107999999999999e-05 2024-09-21T10:13:52.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.382 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:52.382 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447459+0000", 2024-09-21T10:13:52.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1241e-05 2024-09-21T10:13:52.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:52.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447467+0000", 2024-09-21T10:13:52.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8970000000000008e-06 2024-09-21T10:13:52.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447518+0000", 2024-09-21T10:13:52.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1329999999999998e-05 2024-09-21T10:13:52.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447525+0000", 2024-09-21T10:13:52.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5379999999999999e-06 2024-09-21T10:13:52.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384523+0000", 2024-09-21T10:13:52.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.93699787099999998 2024-09-21T10:13:52.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.385 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.385 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384530+0000", 2024-09-21T10:13:52.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8480000000000003e-06 2024-09-21T10:13:52.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.685237+0000", 2024-09-21T10:13:52.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.300707727 2024-09-21T10:13:52.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:52.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.685376+0000", 2024-09-21T10:13:52.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000139204 2024-09-21T10:13:52.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:13:52.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.685545+0000", 2024-09-21T10:13:52.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000168853 2024-09-21T10:13:52.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695283+0000", 2024-09-21T10:13:52.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097372459999999997 2024-09-21T10:13:52.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695320+0000", 2024-09-21T10:13:52.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7492e-05 2024-09-21T10:13:52.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695535+0000", 2024-09-21T10:13:52.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021545700000000001 2024-09-21T10:13:52.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:52.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695584+0000", 2024-09-21T10:13:52.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8458000000000003e-05 2024-09-21T10:13:52.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:13:52.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695627+0000", 2024-09-21T10:13:52.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3126e-05 2024-09-21T10:13:52.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:13:52.389 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695720+0000", 2024-09-21T10:13:52.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2953000000000003e-05 2024-09-21T10:13:52.389 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.389 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.389 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.389 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:191585 11.8 11:1a907492:::benchmark_data_smithi107_218812_object79098:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-21T10:13:52.389 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:12:53.713246+0000", 2024-09-21T10:13:52.389 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 58.570631591000001, 2024-09-21T10:13:52.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.055687397, 2024-09-21T10:13:52.389 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:52.389 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:52.389 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:13:52.390 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:52.390 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:52.390 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:52.390 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 191585 2024-09-21T10:13:52.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.390 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:52.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:52.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.713246+0000", 2024-09-21T10:13:52.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:52.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.713246+0000", 2024-09-21T10:13:52.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:52.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.713248+0000", 2024-09-21T10:13:52.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6160000000000001e-06 2024-09-21T10:13:52.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:52.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.713254+0000", 2024-09-21T10:13:52.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.494e-06 2024-09-21T10:13:52.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:52.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.713272+0000", 2024-09-21T10:13:52.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7832000000000001e-05 2024-09-21T10:13:52.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:52.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.713281+0000", 2024-09-21T10:13:52.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3170000000000002e-06 2024-09-21T10:13:52.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.714816+0000", 2024-09-21T10:13:52.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0015355309999999999 2024-09-21T10:13:52.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-09-21T10:13:52.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.714830+0000", 2024-09-21T10:13:52.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3747000000000001e-05 2024-09-21T10:13:52.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.725106+0000", 2024-09-21T10:13:52.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010276333 2024-09-21T10:13:52.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.725114+0000", 2024-09-21T10:13:52.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0420000000000003e-06 2024-09-21T10:13:52.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.761920+0000", 2024-09-21T10:13:52.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.036805736 2024-09-21T10:13:52.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:52.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.762052+0000", 2024-09-21T10:13:52.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131796 2024-09-21T10:13:52.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:13:52.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.762202+0000", 2024-09-21T10:13:52.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015021899999999999 2024-09-21T10:13:52.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.766044+0000", 2024-09-21T10:13:52.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038424470000000001 2024-09-21T10:13:52.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.766100+0000", 2024-09-21T10:13:52.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5404e-05 2024-09-21T10:13:52.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:52.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.768825+0000", 2024-09-21T10:13:52.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027253490000000002 2024-09-21T10:13:52.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:13:52.396 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.768891+0000", 2024-09-21T10:13:52.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5691000000000005e-05 2024-09-21T10:13:52.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:13:52.396 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.768934+0000", 2024-09-21T10:13:52.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2843000000000003e-05 2024-09-21T10:13:52.396 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.396 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.396 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.396 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:191588 11.8 11:107bb4c7:::benchmark_data_smithi107_218812_object79101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-21T10:13:52.397 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:12:53.715005+0000", 2024-09-21T10:13:52.397 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 58.568872452000001, 2024-09-21T10:13:52.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.05875348, 2024-09-21T10:13:52.397 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:52.397 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:52.397 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:13:52.397 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:52.397 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:52.397 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:52.397 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 191588 2024-09-21T10:13:52.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.397 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:52.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:52.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.715005+0000", 2024-09-21T10:13:52.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:52.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.715005+0000", 2024-09-21T10:13:52.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:52.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.715006+0000", 2024-09-21T10:13:52.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7099999999999997e-07 2024-09-21T10:13:52.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:52.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.715008+0000", 2024-09-21T10:13:52.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7e-06 2024-09-21T10:13:52.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:52.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.715013+0000", 2024-09-21T10:13:52.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1590000000000001e-06 2024-09-21T10:13:52.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:52.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.715017+0000", 2024-09-21T10:13:52.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3769999999999998e-06 2024-09-21T10:13:52.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.715104+0000", 2024-09-21T10:13:52.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7217000000000002e-05 2024-09-21T10:13:52.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting_for_map not empty", 2024-09-21T10:13:52.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.715109+0000", 2024-09-21T10:13:52.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9949999999999996e-06 2024-09-21T10:13:52.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.725165+0000", 2024-09-21T10:13:52.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010055374000000001 2024-09-21T10:13:52.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.725172+0000", 2024-09-21T10:13:52.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4869999999999996e-06 2024-09-21T10:13:52.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.766126+0000", 2024-09-21T10:13:52.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.040954226 2024-09-21T10:13:52.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:52.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.766195+0000", 2024-09-21T10:13:52.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8657000000000004e-05 2024-09-21T10:13:52.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:13:52.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.766285+0000", 2024-09-21T10:13:52.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0253999999999993e-05 2024-09-21T10:13:52.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.773605+0000", 2024-09-21T10:13:52.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0073198819999999998 2024-09-21T10:13:52.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:52.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.773629+0000", 2024-09-21T10:13:52.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4206000000000002e-05 2024-09-21T10:13:52.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.773668+0000", 2024-09-21T10:13:52.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8436000000000003e-05 2024-09-21T10:13:52.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:13:52.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.773721+0000", 2024-09-21T10:13:52.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3597999999999997e-05 2024-09-21T10:13:52.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:13:52.404 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.773759+0000", 2024-09-21T10:13:52.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7540999999999999e-05 2024-09-21T10:13:52.404 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.404 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.404 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.404 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:191602 11.8 11:17e50690:::benchmark_data_smithi107_218812_object79115:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-21T10:13:52.404 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:12:53.735045+0000", 2024-09-21T10:13:52.404 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 58.548832955999998, 2024-09-21T10:13:52.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0389251159999999, 2024-09-21T10:13:52.404 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:52.404 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:52.404 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:13:52.404 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:52.405 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:52.405 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:52.405 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 191602 2024-09-21T10:13:52.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.405 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:52.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.405 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:52.405 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.735045+0000", 2024-09-21T10:13:52.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.405 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:52.405 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.735045+0000", 2024-09-21T10:13:52.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.406 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:52.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.735047+0000", 2024-09-21T10:13:52.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8020000000000001e-06 2024-09-21T10:13:52.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.406 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:52.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.735052+0000", 2024-09-21T10:13:52.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0289999999999996e-06 2024-09-21T10:13:52.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.406 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:52.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.735059+0000", 2024-09-21T10:13:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3309999999999998e-06 2024-09-21T10:13:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.735067+0000", 2024-09-21T10:13:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4919999999999996e-06 2024-09-21T10:13:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.736048+0000", 2024-09-21T10:13:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00098166700000000009 2024-09-21T10:13:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.407 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.736057+0000", 2024-09-21T10:13:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3040000000000005e-06 2024-09-21T10:13:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.769001+0000", 2024-09-21T10:13:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.032944724 2024-09-21T10:13:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.769132+0000", 2024-09-21T10:13:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013070700000000001 2024-09-21T10:13:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:13:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.769277+0000", 2024-09-21T10:13:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000144907 2024-09-21T10:13:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.773634+0000", 2024-09-21T10:13:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004357461 2024-09-21T10:13:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.773798+0000", 2024-09-21T10:13:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016392699999999999 2024-09-21T10:13:52.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.773873+0000", 2024-09-21T10:13:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4299000000000001e-05 2024-09-21T10:13:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:13:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.773930+0000", 2024-09-21T10:13:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7246e-05 2024-09-21T10:13:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:13:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.773970+0000", 2024-09-21T10:13:52.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0219999999999998e-05 2024-09-21T10:13:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:191603 11.8 11:122a43b4:::benchmark_data_smithi107_218812_object79116:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-21T10:13:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:12:53.736409+0000", 2024-09-21T10:13:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 58.547468791, 2024-09-21T10:13:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.037748688, 2024-09-21T10:13:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:13:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:52.411 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 191603 2024-09-21T10:13:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.736409+0000", 2024-09-21T10:13:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.736409+0000", 2024-09-21T10:13:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.736410+0000", 2024-09-21T10:13:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.015e-06 2024-09-21T10:13:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.736415+0000", 2024-09-21T10:13:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0819999999999998e-06 2024-09-21T10:13:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.736426+0000", 2024-09-21T10:13:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1161999999999999e-05 2024-09-21T10:13:52.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.736433+0000", 2024-09-21T10:13:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9040000000000003e-06 2024-09-21T10:13:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.736498+0000", 2024-09-21T10:13:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4831000000000003e-05 2024-09-21T10:13:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.736506+0000", 2024-09-21T10:13:52.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0350000000000001e-06 2024-09-21T10:13:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.771672+0000", 2024-09-21T10:13:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.035166128 2024-09-21T10:13:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.771786+0000", 2024-09-21T10:13:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000113398 2024-09-21T10:13:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:13:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.771917+0000", 2024-09-21T10:13:52.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131407 2024-09-21T10:13:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.773812+0000", 2024-09-21T10:13:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0018946900000000001 2024-09-21T10:13:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.774017+0000", 2024-09-21T10:13:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00020566200000000001 2024-09-21T10:13:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.774067+0000", 2024-09-21T10:13:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9419000000000002e-05 2024-09-21T10:13:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:13:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.774120+0000", 2024-09-21T10:13:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2976000000000002e-05 2024-09-21T10:13:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:13:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.774158+0000", 2024-09-21T10:13:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7978999999999997e-05 2024-09-21T10:13:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.417 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:191605 11.8 11:13efc3fe:::benchmark_data_smithi107_218812_object79118:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-21T10:13:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:12:53.738260+0000", 2024-09-21T10:13:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 58.545617751999998, 2024-09-21T10:13:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0548225870000001, 2024-09-21T10:13:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:13:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 191605 2024-09-21T10:13:52.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.738260+0000", 2024-09-21T10:13:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.738260+0000", 2024-09-21T10:13:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.738261+0000", 2024-09-21T10:13:52.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8299999999999995e-07 2024-09-21T10:13:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.738265+0000", 2024-09-21T10:13:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.839e-06 2024-09-21T10:13:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.738274+0000", 2024-09-21T10:13:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.738e-06 2024-09-21T10:13:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.738280+0000", 2024-09-21T10:13:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6139999999999997e-06 2024-09-21T10:13:52.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.738335+0000", 2024-09-21T10:13:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5064e-05 2024-09-21T10:13:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.738417+0000", 2024-09-21T10:13:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1631999999999994e-05 2024-09-21T10:13:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.774196+0000", 2024-09-21T10:13:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0357787030000001 2024-09-21T10:13:52.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.774293+0000", 2024-09-21T10:13:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6806999999999997e-05 2024-09-21T10:13:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:13:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.774397+0000", 2024-09-21T10:13:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000104566 2024-09-21T10:13:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.422 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.785271+0000", 2024-09-21T10:13:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01087341 2024-09-21T10:13:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.785319+0000", 2024-09-21T10:13:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8946000000000002e-05 2024-09-21T10:13:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.792989+0000", 2024-09-21T10:13:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0076691659999999998 2024-09-21T10:13:52.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:13:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.793037+0000", 2024-09-21T10:13:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8242999999999999e-05 2024-09-21T10:13:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:13:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.793083+0000", 2024-09-21T10:13:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5875999999999999e-05 2024-09-21T10:13:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.424 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:191609 11.8 11:187e1fde:::benchmark_data_smithi107_218812_object79122:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-21T10:13:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:12:53.738960+0000", 2024-09-21T10:13:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 58.544918191000001, 2024-09-21T10:13:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0542151070000001, 2024-09-21T10:13:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:13:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 191609 2024-09-21T10:13:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.738960+0000", 2024-09-21T10:13:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.738960+0000", 2024-09-21T10:13:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.738961+0000", 2024-09-21T10:13:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0299999999999997e-07 2024-09-21T10:13:52.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.738965+0000", 2024-09-21T10:13:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2039999999999999e-06 2024-09-21T10:13:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.738976+0000", 2024-09-21T10:13:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0956e-05 2024-09-21T10:13:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.738983+0000", 2024-09-21T10:13:52.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8970000000000002e-06 2024-09-21T10:13:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.739032+0000", 2024-09-21T10:13:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9387e-05 2024-09-21T10:13:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.739039+0000", 2024-09-21T10:13:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.832e-06 2024-09-21T10:13:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.428 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.777661+0000", 2024-09-21T10:13:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.03862196 2024-09-21T10:13:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.777825+0000", 2024-09-21T10:13:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000163919 2024-09-21T10:13:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:13:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.777951+0000", 2024-09-21T10:13:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000125883 2024-09-21T10:13:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.429 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.785360+0000", 2024-09-21T10:13:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0074097260000000002 2024-09-21T10:13:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.785402+0000", 2024-09-21T10:13:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2072e-05 2024-09-21T10:13:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.793093+0000", 2024-09-21T10:13:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0076904060000000003 2024-09-21T10:13:52.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:13:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.793134+0000", 2024-09-21T10:13:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1136000000000001e-05 2024-09-21T10:13:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:13:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.793175+0000", 2024-09-21T10:13:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0825999999999998e-05 2024-09-21T10:13:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:191616 11.8 11:1241be03:::benchmark_data_smithi107_218812_object79129:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-21T10:13:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:12:53.745882+0000", 2024-09-21T10:13:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 58.537996270999997, 2024-09-21T10:13:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.047432183, 2024-09-21T10:13:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:13:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:13:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:13:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:13:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:13:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:13:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 191616 2024-09-21T10:13:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:13:52.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:13:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.745882+0000", 2024-09-21T10:13:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:13:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.745882+0000", 2024-09-21T10:13:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:13:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:13:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.745883+0000", 2024-09-21T10:13:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4929999999999999e-06 2024-09-21T10:13:52.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:13:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.745889+0000", 2024-09-21T10:13:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8039999999999998e-06 2024-09-21T10:13:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:13:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.745905+0000", 2024-09-21T10:13:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6464999999999999e-05 2024-09-21T10:13:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.434 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:13:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.745913+0000", 2024-09-21T10:13:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.605e-06 2024-09-21T10:13:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.745969+0000", 2024-09-21T10:13:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5754000000000002e-05 2024-09-21T10:13:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:13:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:53.745977+0000", 2024-09-21T10:13:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8210000000000002e-06 2024-09-21T10:13:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.436 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:13:52.436 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.785435+0000", 2024-09-21T10:13:52.436 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0394586480000001 2024-09-21T10:13:52.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.436 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.436 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:13:52.436 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.785547+0000", 2024-09-21T10:13:52.436 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112094 2024-09-21T10:13:52.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.436 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.436 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:13:52.436 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.785658+0000", 2024-09-21T10:13:52.436 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111067 2024-09-21T10:13:52.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.792970+0000", 2024-09-21T10:13:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0073115899999999998 2024-09-21T10:13:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:13:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.793184+0000", 2024-09-21T10:13:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021382199999999999 2024-09-21T10:13:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:13:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.793222+0000", 2024-09-21T10:13:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7942999999999999e-05 2024-09-21T10:13:52.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:13:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.793274+0000", 2024-09-21T10:13:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2587000000000003e-05 2024-09-21T10:13:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:13:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:13:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:13:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:12:54.793314+0000", 2024-09-21T10:13:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9490000000000003e-05 2024-09-21T10:13:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:13:52.438 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:13:52.439 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:13:52.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:52.440 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:52.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:52.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:52.563 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:52.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:52.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:52.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:52 smithi107 ceph-mon[102221]: pgmap v1043: 93 pgs: 93 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:13:52.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:52 smithi107 ceph-mon[105022]: pgmap v1043: 93 pgs: 93 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:13:52.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:52.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:52 smithi129 ceph-mon[88471]: pgmap v1043: 93 pgs: 93 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:13:52.687 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:52.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:52.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:52.789 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:52.810 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:52.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:52.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:52.912 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:52.932 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:52.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:52.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:53.034 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:53.055 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:53.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:53.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:53.086 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:53.086 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:53.157 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:53.178 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:53.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:53.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:53.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:53.301 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:53.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:53.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:53.402 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:53.424 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:53.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:53.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:53.525 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:53.546 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:53.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:53.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:53.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:53 smithi107 ceph-mon[102221]: from='client.36777 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:13:53.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:53 smithi107 ceph-mon[105022]: from='client.36777 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:13:53.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:53.668 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:53 smithi129 ceph-mon[88471]: from='client.36777 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:13:53.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:53.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:53.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:53.791 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:53.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:53.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:53.892 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:53.914 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:53.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:53.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:54.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:54.037 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:54.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:54.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:54.140 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:54.164 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:54.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:54.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:54.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:54.287 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:54.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:54.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:54.331 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1043,"stamp":"2024-09-21T10:13:51.326476+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":667476,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":35,"num_read_kb":91,"num_write":82,"num_write_kb":229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":680205,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2159,"ondisk_log_size":2159,"up":279,"acting":279,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":329,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":5431432,"kb_used_data":55608,"kb_used_omap":41,"kb_used_meta":5375702,"kb_avail":650784632,"statfs":{"total":671965249536,"available":666403463168,"internally_reserved":0,"allocated":56942592,"data_stored":45290273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42519,"internal_metadata":5504719337},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-224970,"num_write_kb":-7198977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-11576,"num_bytes_recovered":-758644736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001977"},"pg_stats":[{"pgid":"1.11","version":"77'100","reported_seq":100,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454531+0000","last_change":"2024-09-21T10:10:57.397786+0000","last_active":"2024-09-21T10:13:44.454531+0000","last_peered":"2024-09-21T10:13:44.454531+0000","last_clean":"2024-09-21T10:13:44.454531+0000","last_became_active":"2024-09-21T10:10:57.396742+0000","last_became_peered":"2024-09-21T10:10:57.396742+0000","last_unstale":"2024-09-21T10:13:44.454531+0000","last_undegraded":"2024-09-21T10:13:44.454531+0000","last_fullsized":"2024-09-21T10:13:44.454531+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:53.631923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:53.631923+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:33:35.364595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":438,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454539+0000","last_change":"2024-09-21T10:03:38.389577+0000","last_active":"2024-09-21T10:13:44.454539+0000","last_peered":"2024-09-21T10:13:44.454539+0000","last_clean":"2024-09-21T10:13:44.454539+0000","last_became_active":"2024-09-21T10:03:38.389292+0000","last_became_peered":"2024-09-21T10:03:38.389292+0000","last_unstale":"2024-09-21T10:13:44.454539+0000","last_undegraded":"2024-09-21T10:13:44.454539+0000","last_fullsized":"2024-09-21T10:13:44.454539+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:32.113472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":162,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.569402+0000","last_change":"2024-09-21T10:08:38.387871+0000","last_active":"2024-09-21T10:13:30.569402+0000","last_peered":"2024-09-21T10:13:30.569402+0000","last_clean":"2024-09-21T10:13:30.569402+0000","last_became_active":"2024-09-21T10:08:38.376727+0000","last_became_peered":"2024-09-21T10:08:38.376727+0000","last_unstale":"2024-09-21T10:13:30.569402+0000","last_undegraded":"2024-09-21T10:13:30.569402+0000","last_fullsized":"2024-09-21T10:13:30.569402+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:45:54.498950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'100","reported_seq":191,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.568935+0000","last_change":"2024-09-21T10:11:05.438031+0000","last_active":"2024-09-21T10:13:30.568935+0000","last_peered":"2024-09-21T10:13:30.568935+0000","last_clean":"2024-09-21T10:13:30.568935+0000","last_became_active":"2024-09-21T10:11:05.437637+0000","last_became_peered":"2024-09-21T10:11:05.437637+0000","last_unstale":"2024-09-21T10:13:30.568935+0000","last_undegraded":"2024-09-21T10:13:30.568935+0000","last_fullsized":"2024-09-21T10:13:30.568935+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.798577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.798577+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:42.343217+0000","scrub_duration":0.0023622640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":283,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.572880+0000","last_change":"2024-09-21T10:10:57.392782+0000","last_active":"2024-09-21T10:13:30.572880+0000","last_peered":"2024-09-21T10:13:30.572880+0000","last_clean":"2024-09-21T10:13:30.572880+0000","last_became_active":"2024-09-21T10:10:57.392305+0000","last_became_peered":"2024-09-21T10:10:57.392305+0000","last_unstale":"2024-09-21T10:13:30.572880+0000","last_undegraded":"2024-09-21T10:13:30.572880+0000","last_fullsized":"2024-09-21T10:13:30.572880+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:42:01.387622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":359,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454831+0000","last_change":"2024-09-21T10:09:23.846703+0000","last_active":"2024-09-21T10:13:44.454831+0000","last_peered":"2024-09-21T10:13:44.454831+0000","last_clean":"2024-09-21T10:13:44.454831+0000","last_became_active":"2024-09-21T10:09:23.846415+0000","last_became_peered":"2024-09-21T10:09:23.846415+0000","last_unstale":"2024-09-21T10:13:44.454831+0000","last_undegraded":"2024-09-21T10:13:44.454831+0000","last_fullsized":"2024-09-21T10:13:44.454831+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:23.129285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'100","reported_seq":154,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454338+0000","last_change":"2024-09-21T10:09:24.841480+0000","last_active":"2024-09-21T10:13:44.454338+0000","last_peered":"2024-09-21T10:13:44.454338+0000","last_clean":"2024-09-21T10:13:44.454338+0000","last_became_active":"2024-09-21T10:09:24.841145+0000","last_became_peered":"2024-09-21T10:09:24.841145+0000","last_unstale":"2024-09-21T10:13:44.454338+0000","last_undegraded":"2024-09-21T10:13:44.454338+0000","last_fullsized":"2024-09-21T10:13:44.454338+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.433227+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:14:42.886197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17890,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":23,"num_write":5,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":35780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":85,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.455179+0000","last_change":"2024-09-21T10:09:49.922489+0000","last_active":"2024-09-21T10:13:44.455179+0000","last_peered":"2024-09-21T10:13:44.455179+0000","last_clean":"2024-09-21T10:13:44.455179+0000","last_became_active":"2024-09-21T10:09:49.921629+0000","last_became_peered":"2024-09-21T10:09:49.921629+0000","last_unstale":"2024-09-21T10:13:44.455179+0000","last_undegraded":"2024-09-21T10:13:44.455179+0000","last_fullsized":"2024-09-21T10:13:44.455179+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:50.611979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":164,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.569083+0000","last_change":"2024-09-21T10:09:24.836647+0000","last_active":"2024-09-21T10:13:30.569083+0000","last_peered":"2024-09-21T10:13:30.569083+0000","last_clean":"2024-09-21T10:13:30.569083+0000","last_became_active":"2024-09-21T10:09:24.836278+0000","last_became_peered":"2024-09-21T10:09:24.836278+0000","last_unstale":"2024-09-21T10:13:30.569083+0000","last_undegraded":"2024-09-21T10:13:30.569083+0000","last_fullsized":"2024-09-21T10:13:30.569083+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:08:36.535892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"77'100","reported_seq":45,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.565999+0000","last_change":"2024-09-21T10:13:14.804339+0000","last_active":"2024-09-21T10:13:30.565999+0000","last_peered":"2024-09-21T10:13:30.565999+0000","last_clean":"2024-09-21T10:13:30.565999+0000","last_became_active":"2024-09-21T10:10:57.452150+0000","last_became_peered":"2024-09-21T10:10:57.452150+0000","last_unstale":"2024-09-21T10:13:30.565999+0000","last_undegraded":"2024-09-21T10:13:30.565999+0000","last_fullsized":"2024-09-21T10:13:30.565999+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:13:14.804256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:13:14.804256+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:33.744954+0000","scrub_duration":0.002846973,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":438,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.455533+0000","last_change":"2024-09-21T10:03:38.381552+0000","last_active":"2024-09-21T10:13:44.455533+0000","last_peered":"2024-09-21T10:13:44.455533+0000","last_clean":"2024-09-21T10:13:44.455533+0000","last_became_active":"2024-09-21T10:03:38.380898+0000","last_became_peered":"2024-09-21T10:03:38.380898+0000","last_unstale":"2024-09-21T10:13:44.455533+0000","last_undegraded":"2024-09-21T10:13:44.455533+0000","last_fullsized":"2024-09-21T10:13:44.455533+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:53.409606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":351,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.570603+0000","last_change":"2024-09-21T10:10:57.459057+0000","last_active":"2024-09-21T10:13:30.570603+0000","last_peered":"2024-09-21T10:13:30.570603+0000","last_clean":"2024-09-21T10:13:30.570603+0000","last_became_active":"2024-09-21T10:10:57.451776+0000","last_became_peered":"2024-09-21T10:10:57.451776+0000","last_unstale":"2024-09-21T10:13:30.570603+0000","last_undegraded":"2024-09-21T10:13:30.570603+0000","last_fullsized":"2024-09-21T10:13:30.570603+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:00.138317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"77'100","reported_seq":176,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.572859+0000","last_change":"2024-09-21T10:08:38.374573+0000","last_active":"2024-09-21T10:13:30.572859+0000","last_peered":"2024-09-21T10:13:30.572859+0000","last_clean":"2024-09-21T10:13:30.572859+0000","last_became_active":"2024-09-21T10:08:38.374075+0000","last_became_peered":"2024-09-21T10:08:38.374075+0000","last_unstale":"2024-09-21T10:13:30.572859+0000","last_undegraded":"2024-09-21T10:13:30.572859+0000","last_fullsized":"2024-09-21T10:13:30.572859+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:55.259385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:55.259385+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:28:26.784162+0000","scrub_duration":0.0015667260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":207,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.569454+0000","last_change":"2024-09-21T10:09:49.941392+0000","last_active":"2024-09-21T10:13:30.569454+0000","last_peered":"2024-09-21T10:13:30.569454+0000","last_clean":"2024-09-21T10:13:30.569454+0000","last_became_active":"2024-09-21T10:09:49.923535+0000","last_became_peered":"2024-09-21T10:09:49.923535+0000","last_unstale":"2024-09-21T10:13:30.569454+0000","last_undegraded":"2024-09-21T10:13:30.569454+0000","last_fullsized":"2024-09-21T10:13:30.569454+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:16.292163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":45,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.566246+0000","last_change":"2024-09-21T10:11:05.435502+0000","last_active":"2024-09-21T10:13:30.566246+0000","last_peered":"2024-09-21T10:13:30.566246+0000","last_clean":"2024-09-21T10:13:30.566246+0000","last_became_active":"2024-09-21T10:11:05.435216+0000","last_became_peered":"2024-09-21T10:11:05.435216+0000","last_unstale":"2024-09-21T10:13:30.566246+0000","last_undegraded":"2024-09-21T10:13:30.566246+0000","last_fullsized":"2024-09-21T10:13:30.566246+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:24:25.434424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"77'100","reported_seq":245,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454430+0000","last_change":"2024-09-21T10:09:23.839211+0000","last_active":"2024-09-21T10:13:44.454430+0000","last_peered":"2024-09-21T10:13:44.454430+0000","last_clean":"2024-09-21T10:13:44.454430+0000","last_became_active":"2024-09-21T10:09:23.838779+0000","last_became_peered":"2024-09-21T10:09:23.838779+0000","last_unstale":"2024-09-21T10:13:44.454430+0000","last_undegraded":"2024-09-21T10:13:44.454430+0000","last_fullsized":"2024-09-21T10:13:44.454430+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.981336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.981336+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:03:15.566873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":186,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.569773+0000","last_change":"2024-09-21T10:09:49.932631+0000","last_active":"2024-09-21T10:13:30.569773+0000","last_peered":"2024-09-21T10:13:30.569773+0000","last_clean":"2024-09-21T10:13:30.569773+0000","last_became_active":"2024-09-21T10:09:49.922897+0000","last_became_peered":"2024-09-21T10:09:49.922897+0000","last_unstale":"2024-09-21T10:13:30.569773+0000","last_undegraded":"2024-09-21T10:13:30.569773+0000","last_fullsized":"2024-09-21T10:13:30.569773+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:40.795476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":233,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454852+0000","last_change":"2024-09-21T10:08:38.377732+0000","last_active":"2024-09-21T10:13:44.454852+0000","last_peered":"2024-09-21T10:13:44.454852+0000","last_clean":"2024-09-21T10:13:44.454852+0000","last_became_active":"2024-09-21T10:08:38.376601+0000","last_became_peered":"2024-09-21T10:08:38.376601+0000","last_unstale":"2024-09-21T10:13:44.454852+0000","last_undegraded":"2024-09-21T10:13:44.454852+0000","last_fullsized":"2024-09-21T10:13:44.454852+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:02.540158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"77'100","reported_seq":276,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.570943+0000","last_change":"2024-09-21T10:08:38.382100+0000","last_active":"2024-09-21T10:13:30.570943+0000","last_peered":"2024-09-21T10:13:30.570943+0000","last_clean":"2024-09-21T10:13:30.570943+0000","last_became_active":"2024-09-21T10:08:38.381736+0000","last_became_peered":"2024-09-21T10:08:38.381736+0000","last_unstale":"2024-09-21T10:13:30.570943+0000","last_undegraded":"2024-09-21T10:13:30.570943+0000","last_fullsized":"2024-09-21T10:13:30.570943+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.289859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.289859+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:23.739980+0000","scrub_duration":0.001251809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":172,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.572800+0000","last_change":"2024-09-21T10:10:57.460732+0000","last_active":"2024-09-21T10:13:30.572800+0000","last_peered":"2024-09-21T10:13:30.572800+0000","last_clean":"2024-09-21T10:13:30.572800+0000","last_became_active":"2024-09-21T10:10:57.454689+0000","last_became_peered":"2024-09-21T10:10:57.454689+0000","last_unstale":"2024-09-21T10:13:30.572800+0000","last_undegraded":"2024-09-21T10:13:30.572800+0000","last_fullsized":"2024-09-21T10:13:30.572800+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:12.035022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":292,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.570827+0000","last_change":"2024-09-21T10:03:38.378384+0000","last_active":"2024-09-21T10:13:30.570827+0000","last_peered":"2024-09-21T10:13:30.570827+0000","last_clean":"2024-09-21T10:13:30.570827+0000","last_became_active":"2024-09-21T10:03:38.378072+0000","last_became_peered":"2024-09-21T10:03:38.378072+0000","last_unstale":"2024-09-21T10:13:30.570827+0000","last_undegraded":"2024-09-21T10:13:30.570827+0000","last_fullsized":"2024-09-21T10:13:30.570827+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:54:37.039622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"77'100","reported_seq":144,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.570084+0000","last_change":"2024-09-21T10:07:26.188298+0000","last_active":"2024-09-21T10:13:30.570084+0000","last_peered":"2024-09-21T10:13:30.570084+0000","last_clean":"2024-09-21T10:13:30.570084+0000","last_became_active":"2024-09-21T10:07:24.345265+0000","last_became_peered":"2024-09-21T10:07:24.345265+0000","last_unstale":"2024-09-21T10:13:30.570084+0000","last_undegraded":"2024-09-21T10:13:30.570084+0000","last_fullsized":"2024-09-21T10:13:30.570084+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:07:26.188220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:07:26.188220+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:46.381710+0000","scrub_duration":0.021884888000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":242,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454669+0000","last_change":"2024-09-21T10:07:57.551556+0000","last_active":"2024-09-21T10:13:44.454669+0000","last_peered":"2024-09-21T10:13:44.454669+0000","last_clean":"2024-09-21T10:13:44.454669+0000","last_became_active":"2024-09-21T10:07:57.551249+0000","last_became_peered":"2024-09-21T10:07:57.551249+0000","last_unstale":"2024-09-21T10:13:44.454669+0000","last_undegraded":"2024-09-21T10:13:44.454669+0000","last_fullsized":"2024-09-21T10:13:44.454669+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:57.557152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":339,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454805+0000","last_change":"2024-09-21T10:09:24.839665+0000","last_active":"2024-09-21T10:13:44.454805+0000","last_peered":"2024-09-21T10:13:44.454805+0000","last_clean":"2024-09-21T10:13:44.454805+0000","last_became_active":"2024-09-21T10:09:24.839353+0000","last_became_peered":"2024-09-21T10:09:24.839353+0000","last_unstale":"2024-09-21T10:13:44.454805+0000","last_undegraded":"2024-09-21T10:13:44.454805+0000","last_fullsized":"2024-09-21T10:13:44.454805+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:51.760073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":166,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.569161+0000","last_change":"2024-09-21T10:04:53.355975+0000","last_active":"2024-09-21T10:13:30.569161+0000","last_peered":"2024-09-21T10:13:30.569161+0000","last_clean":"2024-09-21T10:13:30.569161+0000","last_became_active":"2024-09-21T10:04:42.547505+0000","last_became_peered":"2024-09-21T10:04:42.547505+0000","last_unstale":"2024-09-21T10:13:30.569161+0000","last_undegraded":"2024-09-21T10:13:30.569161+0000","last_fullsized":"2024-09-21T10:13:30.569161+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.355912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.355912+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:43.456830+0000","scrub_duration":0.0017980030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":438,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454552+0000","last_change":"2024-09-21T10:09:49.939553+0000","last_active":"2024-09-21T10:13:44.454552+0000","last_peered":"2024-09-21T10:13:44.454552+0000","last_clean":"2024-09-21T10:13:44.454552+0000","last_became_active":"2024-09-21T10:09:49.925094+0000","last_became_peered":"2024-09-21T10:09:49.925094+0000","last_unstale":"2024-09-21T10:13:44.454552+0000","last_undegraded":"2024-09-21T10:13:44.454552+0000","last_fullsized":"2024-09-21T10:13:44.454552+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:12:12.933089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":216,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454759+0000","last_change":"2024-09-21T10:08:38.379747+0000","last_active":"2024-09-21T10:13:44.454759+0000","last_peered":"2024-09-21T10:13:44.454759+0000","last_clean":"2024-09-21T10:13:44.454759+0000","last_became_active":"2024-09-21T10:08:38.378983+0000","last_became_peered":"2024-09-21T10:08:38.378983+0000","last_unstale":"2024-09-21T10:13:44.454759+0000","last_undegraded":"2024-09-21T10:13:44.454759+0000","last_fullsized":"2024-09-21T10:13:44.454759+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:45:08.003016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":198,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.384419+0000","last_change":"2024-09-21T10:10:57.403722+0000","last_active":"2024-09-21T10:13:44.384419+0000","last_peered":"2024-09-21T10:13:44.384419+0000","last_clean":"2024-09-21T10:13:44.384419+0000","last_became_active":"2024-09-21T10:10:57.394700+0000","last_became_peered":"2024-09-21T10:10:57.394700+0000","last_unstale":"2024-09-21T10:13:44.384419+0000","last_undegraded":"2024-09-21T10:13:44.384419+0000","last_fullsized":"2024-09-21T10:13:44.384419+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:02.152981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:02.152981+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:52:34.887763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":266,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.570972+0000","last_change":"2024-09-21T10:08:38.379849+0000","last_active":"2024-09-21T10:13:30.570972+0000","last_peered":"2024-09-21T10:13:30.570972+0000","last_clean":"2024-09-21T10:13:30.570972+0000","last_became_active":"2024-09-21T10:08:38.379543+0000","last_became_peered":"2024-09-21T10:08:38.379543+0000","last_unstale":"2024-09-21T10:13:30.570972+0000","last_undegraded":"2024-09-21T10:13:30.570972+0000","last_fullsized":"2024-09-21T10:13:30.570972+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:29.041802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":247,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.385801+0000","last_change":"2024-09-21T10:07:24.342950+0000","last_active":"2024-09-21T10:13:44.385801+0000","last_peered":"2024-09-21T10:13:44.385801+0000","last_clean":"2024-09-21T10:13:44.385801+0000","last_became_active":"2024-09-21T10:07:24.342477+0000","last_became_peered":"2024-09-21T10:07:24.342477+0000","last_unstale":"2024-09-21T10:13:44.385801+0000","last_undegraded":"2024-09-21T10:13:44.385801+0000","last_fullsized":"2024-09-21T10:13:44.385801+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:33:13.664050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"77'100","reported_seq":341,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454914+0000","last_change":"2024-09-21T10:10:57.395906+0000","last_active":"2024-09-21T10:13:44.454914+0000","last_peered":"2024-09-21T10:13:44.454914+0000","last_clean":"2024-09-21T10:13:44.454914+0000","last_became_active":"2024-09-21T10:10:57.395366+0000","last_became_peered":"2024-09-21T10:10:57.395366+0000","last_unstale":"2024-09-21T10:13:44.454914+0000","last_undegraded":"2024-09-21T10:13:44.454914+0000","last_fullsized":"2024-09-21T10:13:44.454914+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.540323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.540323+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:04:34.873336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":339,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.385278+0000","last_change":"2024-09-21T10:07:36.464496+0000","last_active":"2024-09-21T10:13:44.385278+0000","last_peered":"2024-09-21T10:13:44.385278+0000","last_clean":"2024-09-21T10:13:44.385278+0000","last_became_active":"2024-09-21T10:07:36.464172+0000","last_became_peered":"2024-09-21T10:07:36.464172+0000","last_unstale":"2024-09-21T10:13:44.385278+0000","last_undegraded":"2024-09-21T10:13:44.385278+0000","last_fullsized":"2024-09-21T10:13:44.385278+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:09.839709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.570620+0000","last_change":"2024-09-21T10:11:05.442332+0000","last_active":"2024-09-21T10:13:30.570620+0000","last_peered":"2024-09-21T10:13:30.570620+0000","last_clean":"2024-09-21T10:13:30.570620+0000","last_became_active":"2024-09-21T10:11:05.442003+0000","last_became_peered":"2024-09-21T10:11:05.442003+0000","last_unstale":"2024-09-21T10:13:30.570620+0000","last_undegraded":"2024-09-21T10:13:30.570620+0000","last_fullsized":"2024-09-21T10:13:30.570620+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:34.986309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":161,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.385398+0000","last_change":"2024-09-21T10:08:04.645700+0000","last_active":"2024-09-21T10:13:44.385398+0000","last_peered":"2024-09-21T10:13:44.385398+0000","last_clean":"2024-09-21T10:13:44.385398+0000","last_became_active":"2024-09-21T10:07:57.548189+0000","last_became_peered":"2024-09-21T10:07:57.548189+0000","last_unstale":"2024-09-21T10:13:44.385398+0000","last_undegraded":"2024-09-21T10:13:44.385398+0000","last_fullsized":"2024-09-21T10:13:44.385398+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:48.318010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:48.318010+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T16:31:28.173059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":440,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454375+0000","last_change":"2024-09-21T10:03:38.386604+0000","last_active":"2024-09-21T10:13:44.454375+0000","last_peered":"2024-09-21T10:13:44.454375+0000","last_clean":"2024-09-21T10:13:44.454375+0000","last_became_active":"2024-09-21T10:03:38.386283+0000","last_became_peered":"2024-09-21T10:03:38.386283+0000","last_unstale":"2024-09-21T10:13:44.454375+0000","last_undegraded":"2024-09-21T10:13:44.454375+0000","last_fullsized":"2024-09-21T10:13:44.454375+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:27:05.944880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":243,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.455570+0000","last_change":"2024-09-21T10:08:38.372486+0000","last_active":"2024-09-21T10:13:44.455570+0000","last_peered":"2024-09-21T10:13:44.455570+0000","last_clean":"2024-09-21T10:13:44.455570+0000","last_became_active":"2024-09-21T10:08:38.370669+0000","last_became_peered":"2024-09-21T10:08:38.370669+0000","last_unstale":"2024-09-21T10:13:44.455570+0000","last_undegraded":"2024-09-21T10:13:44.455570+0000","last_fullsized":"2024-09-21T10:13:44.455570+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:53:23.247341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":183,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.568842+0000","last_change":"2024-09-21T10:09:49.924286+0000","last_active":"2024-09-21T10:13:30.568842+0000","last_peered":"2024-09-21T10:13:30.568842+0000","last_clean":"2024-09-21T10:13:30.568842+0000","last_became_active":"2024-09-21T10:09:49.923897+0000","last_became_peered":"2024-09-21T10:09:49.923897+0000","last_unstale":"2024-09-21T10:13:30.568842+0000","last_undegraded":"2024-09-21T10:13:30.568842+0000","last_fullsized":"2024-09-21T10:13:30.568842+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:47.393146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:47.393146+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:44.898083+0000","scrub_duration":0.001601086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":194,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.384691+0000","last_change":"2024-09-21T10:10:57.402946+0000","last_active":"2024-09-21T10:13:44.384691+0000","last_peered":"2024-09-21T10:13:44.384691+0000","last_clean":"2024-09-21T10:13:44.384691+0000","last_became_active":"2024-09-21T10:10:57.395450+0000","last_became_peered":"2024-09-21T10:10:57.395450+0000","last_unstale":"2024-09-21T10:13:44.384691+0000","last_undegraded":"2024-09-21T10:13:44.384691+0000","last_fullsized":"2024-09-21T10:13:44.384691+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:08.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":424,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454371+0000","last_change":"2024-09-21T10:10:57.390595+0000","last_active":"2024-09-21T10:13:44.454371+0000","last_peered":"2024-09-21T10:13:44.454371+0000","last_clean":"2024-09-21T10:13:44.454371+0000","last_became_active":"2024-09-21T10:10:57.390265+0000","last_became_peered":"2024-09-21T10:10:57.390265+0000","last_unstale":"2024-09-21T10:13:44.454371+0000","last_undegraded":"2024-09-21T10:13:44.454371+0000","last_fullsized":"2024-09-21T10:13:44.454371+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:37.485063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":299,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.572814+0000","last_change":"2024-09-21T10:09:24.837651+0000","last_active":"2024-09-21T10:13:30.572814+0000","last_peered":"2024-09-21T10:13:30.572814+0000","last_clean":"2024-09-21T10:13:30.572814+0000","last_became_active":"2024-09-21T10:09:24.837226+0000","last_became_peered":"2024-09-21T10:09:24.837226+0000","last_unstale":"2024-09-21T10:13:30.572814+0000","last_undegraded":"2024-09-21T10:13:30.572814+0000","last_fullsized":"2024-09-21T10:13:30.572814+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:16.649601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":32,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.570480+0000","last_change":"2024-09-21T10:11:05.443167+0000","last_active":"2024-09-21T10:13:30.570480+0000","last_peered":"2024-09-21T10:13:30.570480+0000","last_clean":"2024-09-21T10:13:30.570480+0000","last_became_active":"2024-09-21T10:11:05.442805+0000","last_became_peered":"2024-09-21T10:11:05.442805+0000","last_unstale":"2024-09-21T10:13:30.570480+0000","last_undegraded":"2024-09-21T10:13:30.570480+0000","last_fullsized":"2024-09-21T10:13:30.570480+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.550551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.550551+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:58:57.998543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":157,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.568849+0000","last_change":"2024-09-21T10:09:23.838944+0000","last_active":"2024-09-21T10:13:30.568849+0000","last_peered":"2024-09-21T10:13:30.568849+0000","last_clean":"2024-09-21T10:13:30.568849+0000","last_became_active":"2024-09-21T10:09:23.837886+0000","last_became_peered":"2024-09-21T10:09:23.837886+0000","last_unstale":"2024-09-21T10:13:30.568849+0000","last_undegraded":"2024-09-21T10:13:30.568849+0000","last_fullsized":"2024-09-21T10:13:30.568849+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:16.928195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":317,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454440+0000","last_change":"2024-09-21T10:09:24.839770+0000","last_active":"2024-09-21T10:13:44.454440+0000","last_peered":"2024-09-21T10:13:44.454440+0000","last_clean":"2024-09-21T10:13:44.454440+0000","last_became_active":"2024-09-21T10:09:24.839479+0000","last_became_peered":"2024-09-21T10:09:24.839479+0000","last_unstale":"2024-09-21T10:13:44.454440+0000","last_undegraded":"2024-09-21T10:13:44.454440+0000","last_fullsized":"2024-09-21T10:13:44.454440+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:11:19.618193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":85,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.455144+0000","last_change":"2024-09-21T10:09:49.923044+0000","last_active":"2024-09-21T10:13:44.455144+0000","last_peered":"2024-09-21T10:13:44.455144+0000","last_clean":"2024-09-21T10:13:44.455144+0000","last_became_active":"2024-09-21T10:09:49.922711+0000","last_became_peered":"2024-09-21T10:09:49.922711+0000","last_unstale":"2024-09-21T10:13:44.455144+0000","last_undegraded":"2024-09-21T10:13:44.455144+0000","last_fullsized":"2024-09-21T10:13:44.455144+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:14.090084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":197,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.569201+0000","last_change":"2024-09-21T10:09:24.838434+0000","last_active":"2024-09-21T10:13:30.569201+0000","last_peered":"2024-09-21T10:13:30.569201+0000","last_clean":"2024-09-21T10:13:30.569201+0000","last_became_active":"2024-09-21T10:09:24.838066+0000","last_became_peered":"2024-09-21T10:09:24.838066+0000","last_unstale":"2024-09-21T10:13:30.569201+0000","last_undegraded":"2024-09-21T10:13:30.569201+0000","last_fullsized":"2024-09-21T10:13:30.569201+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:52.361137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:52.361137+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:19:57.788195+0000","scrub_duration":0.001255358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":477,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.385427+0000","last_change":"2024-09-21T10:09:23.837859+0000","last_active":"2024-09-21T10:13:44.385427+0000","last_peered":"2024-09-21T10:13:44.385427+0000","last_clean":"2024-09-21T10:13:44.385427+0000","last_became_active":"2024-09-21T10:09:23.837488+0000","last_became_peered":"2024-09-21T10:09:23.837488+0000","last_unstale":"2024-09-21T10:13:44.385427+0000","last_undegraded":"2024-09-21T10:13:44.385427+0000","last_fullsized":"2024-09-21T10:13:44.385427+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:25.215501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":41,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.568764+0000","last_change":"2024-09-21T10:10:27.806070+0000","last_active":"2024-09-21T10:13:30.568764+0000","last_peered":"2024-09-21T10:13:30.568764+0000","last_clean":"2024-09-21T10:13:30.568764+0000","last_became_active":"2024-09-21T10:10:27.805718+0000","last_became_peered":"2024-09-21T10:10:27.805718+0000","last_unstale":"2024-09-21T10:13:30.568764+0000","last_undegraded":"2024-09-21T10:13:30.568764+0000","last_fullsized":"2024-09-21T10:13:30.568764+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:31.144118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":471,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.385002+0000","last_change":"2024-09-21T10:10:57.395073+0000","last_active":"2024-09-21T10:13:44.385002+0000","last_peered":"2024-09-21T10:13:44.385002+0000","last_clean":"2024-09-21T10:13:44.385002+0000","last_became_active":"2024-09-21T10:10:57.394674+0000","last_became_peered":"2024-09-21T10:10:57.394674+0000","last_unstale":"2024-09-21T10:13:44.385002+0000","last_undegraded":"2024-09-21T10:13:44.385002+0000","last_fullsized":"2024-09-21T10:13:44.385002+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:48:48.912649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":341,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454530+0000","last_change":"2024-09-21T10:10:57.398437+0000","last_active":"2024-09-21T10:13:44.454530+0000","last_peered":"2024-09-21T10:13:44.454530+0000","last_clean":"2024-09-21T10:13:44.454530+0000","last_became_active":"2024-09-21T10:10:57.397441+0000","last_became_peered":"2024-09-21T10:10:57.397441+0000","last_unstale":"2024-09-21T10:13:44.454530+0000","last_undegraded":"2024-09-21T10:13:44.454530+0000","last_fullsized":"2024-09-21T10:13:44.454530+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:04.043917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:04.043917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:53:31.813612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":96,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.571052+0000","last_change":"2024-09-21T10:08:26.309323+0000","last_active":"2024-09-21T10:13:30.571052+0000","last_peered":"2024-09-21T10:13:30.571052+0000","last_clean":"2024-09-21T10:13:30.571052+0000","last_became_active":"2024-09-21T10:08:26.309004+0000","last_became_peered":"2024-09-21T10:08:26.309004+0000","last_unstale":"2024-09-21T10:13:30.571052+0000","last_undegraded":"2024-09-21T10:13:30.571052+0000","last_fullsized":"2024-09-21T10:13:30.571052+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:37:20.626702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":151,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454904+0000","last_change":"2024-09-21T10:10:28.180093+0000","last_active":"2024-09-21T10:13:44.454904+0000","last_peered":"2024-09-21T10:13:44.454904+0000","last_clean":"2024-09-21T10:13:44.454904+0000","last_became_active":"2024-09-21T10:10:28.179559+0000","last_became_peered":"2024-09-21T10:10:28.179559+0000","last_unstale":"2024-09-21T10:13:44.454904+0000","last_undegraded":"2024-09-21T10:13:44.454904+0000","last_fullsized":"2024-09-21T10:13:44.454904+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:52:46.146392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":247,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.386216+0000","last_change":"2024-09-21T10:08:26.307041+0000","last_active":"2024-09-21T10:13:44.386216+0000","last_peered":"2024-09-21T10:13:44.386216+0000","last_clean":"2024-09-21T10:13:44.386216+0000","last_became_active":"2024-09-21T10:08:26.306715+0000","last_became_peered":"2024-09-21T10:08:26.306715+0000","last_unstale":"2024-09-21T10:13:44.386216+0000","last_undegraded":"2024-09-21T10:13:44.386216+0000","last_fullsized":"2024-09-21T10:13:44.386216+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:19:26.192398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":266,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.570680+0000","last_change":"2024-09-21T10:04:45.242148+0000","last_active":"2024-09-21T10:13:30.570680+0000","last_peered":"2024-09-21T10:13:30.570680+0000","last_clean":"2024-09-21T10:13:30.570680+0000","last_became_active":"2024-09-21T10:04:42.549844+0000","last_became_peered":"2024-09-21T10:04:42.549844+0000","last_unstale":"2024-09-21T10:13:30.570680+0000","last_undegraded":"2024-09-21T10:13:30.570680+0000","last_fullsized":"2024-09-21T10:13:30.570680+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_deep_scrub":"77'100","last_deep_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.242073+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:03:42.250480+0000","scrub_duration":0.0023466239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":35,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.567458+0000","last_change":"2024-09-21T10:10:57.392880+0000","last_active":"2024-09-21T10:13:30.567458+0000","last_peered":"2024-09-21T10:13:30.567458+0000","last_clean":"2024-09-21T10:13:30.567458+0000","last_became_active":"2024-09-21T10:10:57.392539+0000","last_became_peered":"2024-09-21T10:10:57.392539+0000","last_unstale":"2024-09-21T10:13:30.567458+0000","last_undegraded":"2024-09-21T10:13:30.567458+0000","last_fullsized":"2024-09-21T10:13:30.567458+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:49.754955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":361,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454279+0000","last_change":"2024-09-21T10:09:23.865342+0000","last_active":"2024-09-21T10:13:44.454279+0000","last_peered":"2024-09-21T10:13:44.454279+0000","last_clean":"2024-09-21T10:13:44.454279+0000","last_became_active":"2024-09-21T10:09:23.865008+0000","last_became_peered":"2024-09-21T10:09:23.865008+0000","last_unstale":"2024-09-21T10:13:44.454279+0000","last_undegraded":"2024-09-21T10:13:44.454279+0000","last_fullsized":"2024-09-21T10:13:44.454279+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:03.356101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":35,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.565907+0000","last_change":"2024-09-21T10:10:57.394707+0000","last_active":"2024-09-21T10:13:30.565907+0000","last_peered":"2024-09-21T10:13:30.565907+0000","last_clean":"2024-09-21T10:13:30.565907+0000","last_became_active":"2024-09-21T10:10:57.394363+0000","last_became_peered":"2024-09-21T10:10:57.394363+0000","last_unstale":"2024-09-21T10:13:30.565907+0000","last_undegraded":"2024-09-21T10:13:30.565907+0000","last_fullsized":"2024-09-21T10:13:30.565907+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:38:21.141052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":283,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.570713+0000","last_change":"2024-09-21T10:09:23.839563+0000","last_active":"2024-09-21T10:13:30.570713+0000","last_peered":"2024-09-21T10:13:30.570713+0000","last_clean":"2024-09-21T10:13:30.570713+0000","last_became_active":"2024-09-21T10:09:23.839207+0000","last_became_peered":"2024-09-21T10:09:23.839207+0000","last_unstale":"2024-09-21T10:13:30.570713+0000","last_undegraded":"2024-09-21T10:13:30.570713+0000","last_fullsized":"2024-09-21T10:13:30.570713+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:49.226496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:49.226496+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:03.960125+0000","scrub_duration":0.001656865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":212,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.386124+0000","last_change":"2024-09-21T10:10:57.395167+0000","last_active":"2024-09-21T10:13:44.386124+0000","last_peered":"2024-09-21T10:13:44.386124+0000","last_clean":"2024-09-21T10:13:44.386124+0000","last_became_active":"2024-09-21T10:10:57.394818+0000","last_became_peered":"2024-09-21T10:10:57.394818+0000","last_unstale":"2024-09-21T10:13:44.386124+0000","last_undegraded":"2024-09-21T10:13:44.386124+0000","last_fullsized":"2024-09-21T10:13:44.386124+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:57.224113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":177,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.572762+0000","last_change":"2024-09-21T10:09:49.922782+0000","last_active":"2024-09-21T10:13:30.572762+0000","last_peered":"2024-09-21T10:13:30.572762+0000","last_clean":"2024-09-21T10:13:30.572762+0000","last_became_active":"2024-09-21T10:09:49.922465+0000","last_became_peered":"2024-09-21T10:09:49.922465+0000","last_unstale":"2024-09-21T10:13:30.572762+0000","last_undegraded":"2024-09-21T10:13:30.572762+0000","last_fullsized":"2024-09-21T10:13:30.572762+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:06.151245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":355,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.455006+0000","last_change":"2024-09-21T10:10:57.397251+0000","last_active":"2024-09-21T10:13:44.455006+0000","last_peered":"2024-09-21T10:13:44.455006+0000","last_clean":"2024-09-21T10:13:44.455006+0000","last_became_active":"2024-09-21T10:10:57.396271+0000","last_became_peered":"2024-09-21T10:10:57.396271+0000","last_unstale":"2024-09-21T10:13:44.455006+0000","last_undegraded":"2024-09-21T10:13:44.455006+0000","last_fullsized":"2024-09-21T10:13:44.455006+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:17.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":158,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.385626+0000","last_change":"2024-09-21T10:11:05.455170+0000","last_active":"2024-09-21T10:13:44.385626+0000","last_peered":"2024-09-21T10:13:44.385626+0000","last_clean":"2024-09-21T10:13:44.385626+0000","last_became_active":"2024-09-21T10:11:05.454607+0000","last_became_peered":"2024-09-21T10:11:05.454607+0000","last_unstale":"2024-09-21T10:13:44.385626+0000","last_undegraded":"2024-09-21T10:13:44.385626+0000","last_fullsized":"2024-09-21T10:13:44.385626+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:59.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:59.099852+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:14.788975+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":446,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.455614+0000","last_change":"2024-09-21T10:09:23.841186+0000","last_active":"2024-09-21T10:13:44.455614+0000","last_peered":"2024-09-21T10:13:44.455614+0000","last_clean":"2024-09-21T10:13:44.455614+0000","last_became_active":"2024-09-21T10:09:23.840854+0000","last_became_peered":"2024-09-21T10:09:23.840854+0000","last_unstale":"2024-09-21T10:13:44.455614+0000","last_undegraded":"2024-09-21T10:13:44.455614+0000","last_fullsized":"2024-09-21T10:13:44.455614+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:29.666358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":114,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.570791+0000","last_change":"2024-09-21T10:08:38.380867+0000","last_active":"2024-09-21T10:13:30.570791+0000","last_peered":"2024-09-21T10:13:30.570791+0000","last_clean":"2024-09-21T10:13:30.570791+0000","last_became_active":"2024-09-21T10:08:38.379599+0000","last_became_peered":"2024-09-21T10:08:38.379599+0000","last_unstale":"2024-09-21T10:13:30.570791+0000","last_undegraded":"2024-09-21T10:13:30.570791+0000","last_fullsized":"2024-09-21T10:13:30.570791+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:50.499470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":442,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454300+0000","last_change":"2024-09-21T10:09:49.926548+0000","last_active":"2024-09-21T10:13:44.454300+0000","last_peered":"2024-09-21T10:13:44.454300+0000","last_clean":"2024-09-21T10:13:44.454300+0000","last_became_active":"2024-09-21T10:09:49.926255+0000","last_became_peered":"2024-09-21T10:09:49.926255+0000","last_unstale":"2024-09-21T10:13:44.454300+0000","last_undegraded":"2024-09-21T10:13:44.454300+0000","last_fullsized":"2024-09-21T10:13:44.454300+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:49.855878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":244,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454291+0000","last_change":"2024-09-21T10:10:57.390804+0000","last_active":"2024-09-21T10:13:44.454291+0000","last_peered":"2024-09-21T10:13:44.454291+0000","last_clean":"2024-09-21T10:13:44.454291+0000","last_became_active":"2024-09-21T10:10:57.390323+0000","last_became_peered":"2024-09-21T10:10:57.390323+0000","last_unstale":"2024-09-21T10:13:44.454291+0000","last_undegraded":"2024-09-21T10:13:44.454291+0000","last_fullsized":"2024-09-21T10:13:44.454291+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.005477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.005477+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:29:57.702046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":89,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.455249+0000","last_change":"2024-09-21T10:09:49.930280+0000","last_active":"2024-09-21T10:13:44.455249+0000","last_peered":"2024-09-21T10:13:44.455249+0000","last_clean":"2024-09-21T10:13:44.455249+0000","last_became_active":"2024-09-21T10:09:49.919101+0000","last_became_peered":"2024-09-21T10:09:49.919101+0000","last_unstale":"2024-09-21T10:13:44.455249+0000","last_undegraded":"2024-09-21T10:13:44.455249+0000","last_fullsized":"2024-09-21T10:13:44.455249+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:23.998475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":305,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.570905+0000","last_change":"2024-09-21T10:03:38.392767+0000","last_active":"2024-09-21T10:13:30.570905+0000","last_peered":"2024-09-21T10:13:30.570905+0000","last_clean":"2024-09-21T10:13:30.570905+0000","last_became_active":"2024-09-21T10:03:38.392424+0000","last_became_peered":"2024-09-21T10:03:38.392424+0000","last_unstale":"2024-09-21T10:13:30.570905+0000","last_undegraded":"2024-09-21T10:13:30.570905+0000","last_fullsized":"2024-09-21T10:13:30.570905+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:25.358291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":86,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.565932+0000","last_change":"2024-09-21T10:09:24.836360+0000","last_active":"2024-09-21T10:13:30.565932+0000","last_peered":"2024-09-21T10:13:30.565932+0000","last_clean":"2024-09-21T10:13:30.565932+0000","last_became_active":"2024-09-21T10:09:24.836033+0000","last_became_peered":"2024-09-21T10:09:24.836033+0000","last_unstale":"2024-09-21T10:13:30.565932+0000","last_undegraded":"2024-09-21T10:13:30.565932+0000","last_fullsized":"2024-09-21T10:13:30.565932+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:28:58.368582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":143,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.570826+0000","last_change":"2024-09-21T10:08:38.380189+0000","last_active":"2024-09-21T10:13:30.570826+0000","last_peered":"2024-09-21T10:13:30.570826+0000","last_clean":"2024-09-21T10:13:30.570826+0000","last_became_active":"2024-09-21T10:08:38.379891+0000","last_became_peered":"2024-09-21T10:08:38.379891+0000","last_unstale":"2024-09-21T10:13:30.570826+0000","last_undegraded":"2024-09-21T10:13:30.570826+0000","last_fullsized":"2024-09-21T10:13:30.570826+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:26:35.820921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":223,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.569946+0000","last_change":"2024-09-21T10:08:38.392563+0000","last_active":"2024-09-21T10:13:30.569946+0000","last_peered":"2024-09-21T10:13:30.569946+0000","last_clean":"2024-09-21T10:13:30.569946+0000","last_became_active":"2024-09-21T10:08:38.378735+0000","last_became_peered":"2024-09-21T10:08:38.378735+0000","last_unstale":"2024-09-21T10:13:30.569946+0000","last_undegraded":"2024-09-21T10:13:30.569946+0000","last_fullsized":"2024-09-21T10:13:30.569946+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:29:41.423208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":32,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.570898+0000","last_change":"2024-09-21T10:11:05.453286+0000","last_active":"2024-09-21T10:13:30.570898+0000","last_peered":"2024-09-21T10:13:30.570898+0000","last_clean":"2024-09-21T10:13:30.570898+0000","last_became_active":"2024-09-21T10:11:05.452601+0000","last_became_peered":"2024-09-21T10:11:05.452601+0000","last_unstale":"2024-09-21T10:13:30.570898+0000","last_undegraded":"2024-09-21T10:13:30.570898+0000","last_fullsized":"2024-09-21T10:13:30.570898+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:27.532091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":81,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.385751+0000","last_change":"2024-09-21T10:10:27.833762+0000","last_active":"2024-09-21T10:13:44.385751+0000","last_peered":"2024-09-21T10:13:44.385751+0000","last_clean":"2024-09-21T10:13:44.385751+0000","last_became_active":"2024-09-21T10:10:27.819999+0000","last_became_peered":"2024-09-21T10:10:27.819999+0000","last_unstale":"2024-09-21T10:13:44.385751+0000","last_undegraded":"2024-09-21T10:13:44.385751+0000","last_fullsized":"2024-09-21T10:13:44.385751+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:33.839463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'100","reported_seq":236,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.386003+0000","last_change":"2024-09-21T10:07:24.345428+0000","last_active":"2024-09-21T10:13:44.386003+0000","last_peered":"2024-09-21T10:13:44.386003+0000","last_clean":"2024-09-21T10:13:44.386003+0000","last_became_active":"2024-09-21T10:07:24.343529+0000","last_became_peered":"2024-09-21T10:07:24.343529+0000","last_unstale":"2024-09-21T10:13:44.386003+0000","last_undegraded":"2024-09-21T10:13:44.386003+0000","last_fullsized":"2024-09-21T10:13:44.386003+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.014993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.014993+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:35.677888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":35,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.566387+0000","last_change":"2024-09-21T10:10:57.394885+0000","last_active":"2024-09-21T10:13:30.566387+0000","last_peered":"2024-09-21T10:13:30.566387+0000","last_clean":"2024-09-21T10:13:30.566387+0000","last_became_active":"2024-09-21T10:10:57.394584+0000","last_became_peered":"2024-09-21T10:10:57.394584+0000","last_unstale":"2024-09-21T10:13:30.566387+0000","last_undegraded":"2024-09-21T10:13:30.566387+0000","last_fullsized":"2024-09-21T10:13:30.566387+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:52:16.853713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":428,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454655+0000","last_change":"2024-09-21T10:03:38.425111+0000","last_active":"2024-09-21T10:13:44.454655+0000","last_peered":"2024-09-21T10:13:44.454655+0000","last_clean":"2024-09-21T10:13:44.454655+0000","last_became_active":"2024-09-21T10:03:38.424704+0000","last_became_peered":"2024-09-21T10:03:38.424704+0000","last_unstale":"2024-09-21T10:13:44.454655+0000","last_undegraded":"2024-09-21T10:13:44.454655+0000","last_fullsized":"2024-09-21T10:13:44.454655+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:35.157076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":329,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.455274+0000","last_change":"2024-09-21T10:10:57.397755+0000","last_active":"2024-09-21T10:13:44.455274+0000","last_peered":"2024-09-21T10:13:44.455274+0000","last_clean":"2024-09-21T10:13:44.455274+0000","last_became_active":"2024-09-21T10:10:57.397249+0000","last_became_peered":"2024-09-21T10:10:57.397249+0000","last_unstale":"2024-09-21T10:13:44.455274+0000","last_undegraded":"2024-09-21T10:13:44.455274+0000","last_fullsized":"2024-09-21T10:13:44.455274+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:40.813503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":81,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.570364+0000","last_change":"2024-09-21T10:13:28.546311+0000","last_active":"2024-09-21T10:13:30.570364+0000","last_peered":"2024-09-21T10:13:30.570364+0000","last_clean":"2024-09-21T10:13:30.570364+0000","last_became_active":"2024-09-21T10:13:28.545887+0000","last_became_peered":"2024-09-21T10:13:28.545887+0000","last_unstale":"2024-09-21T10:13:30.570364+0000","last_undegraded":"2024-09-21T10:13:30.570364+0000","last_fullsized":"2024-09-21T10:13:30.570364+0000","mapping_epoch":289,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:32:06.883624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":255,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.570838+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T10:13:30.570838+0000","last_peered":"2024-09-21T10:13:30.570838+0000","last_clean":"2024-09-21T10:13:30.570838+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T10:13:30.570838+0000","last_undegraded":"2024-09-21T10:13:30.570838+0000","last_fullsized":"2024-09-21T10:13:30.570838+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":449,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454142+0000","last_change":"2024-09-21T10:09:49.928053+0000","last_active":"2024-09-21T10:13:44.454142+0000","last_peered":"2024-09-21T10:13:44.454142+0000","last_clean":"2024-09-21T10:13:44.454142+0000","last_became_active":"2024-09-21T10:09:49.927231+0000","last_became_peered":"2024-09-21T10:09:49.927231+0000","last_unstale":"2024-09-21T10:13:44.454142+0000","last_undegraded":"2024-09-21T10:13:44.454142+0000","last_fullsized":"2024-09-21T10:13:44.454142+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:26.987553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":362,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.455441+0000","last_change":"2024-09-21T10:09:23.840717+0000","last_active":"2024-09-21T10:13:44.455441+0000","last_peered":"2024-09-21T10:13:44.455441+0000","last_clean":"2024-09-21T10:13:44.455441+0000","last_became_active":"2024-09-21T10:09:23.840323+0000","last_became_peered":"2024-09-21T10:09:23.840323+0000","last_unstale":"2024-09-21T10:13:44.455441+0000","last_undegraded":"2024-09-21T10:13:44.455441+0000","last_fullsized":"2024-09-21T10:13:44.455441+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:55.769770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":75,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.385141+0000","last_change":"2024-09-21T10:10:27.816526+0000","last_active":"2024-09-21T10:13:44.385141+0000","last_peered":"2024-09-21T10:13:44.385141+0000","last_clean":"2024-09-21T10:13:44.385141+0000","last_became_active":"2024-09-21T10:10:27.816221+0000","last_became_peered":"2024-09-21T10:10:27.816221+0000","last_unstale":"2024-09-21T10:13:44.385141+0000","last_undegraded":"2024-09-21T10:13:44.385141+0000","last_fullsized":"2024-09-21T10:13:44.385141+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:08.331228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":420,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454864+0000","last_change":"2024-09-21T10:08:38.379994+0000","last_active":"2024-09-21T10:13:44.454864+0000","last_peered":"2024-09-21T10:13:44.454864+0000","last_clean":"2024-09-21T10:13:44.454864+0000","last_became_active":"2024-09-21T10:08:38.378729+0000","last_became_peered":"2024-09-21T10:08:38.378729+0000","last_unstale":"2024-09-21T10:13:44.454864+0000","last_undegraded":"2024-09-21T10:13:44.454864+0000","last_fullsized":"2024-09-21T10:13:44.454864+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:13:16.940173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":140,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.385642+0000","last_change":"2024-09-21T10:09:24.844561+0000","last_active":"2024-09-21T10:13:44.385642+0000","last_peered":"2024-09-21T10:13:44.385642+0000","last_clean":"2024-09-21T10:13:44.385642+0000","last_became_active":"2024-09-21T10:09:24.844190+0000","last_became_peered":"2024-09-21T10:09:24.844190+0000","last_unstale":"2024-09-21T10:13:44.385642+0000","last_undegraded":"2024-09-21T10:13:44.385642+0000","last_fullsized":"2024-09-21T10:13:44.385642+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:46:04.775344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":466,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.386182+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:13:44.386182+0000","last_peered":"2024-09-21T10:13:44.386182+0000","last_clean":"2024-09-21T10:13:44.386182+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:13:44.386182+0000","last_undegraded":"2024-09-21T10:13:44.386182+0000","last_fullsized":"2024-09-21T10:13:44.386182+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":85,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454689+0000","last_change":"2024-09-21T10:09:49.918246+0000","last_active":"2024-09-21T10:13:44.454689+0000","last_peered":"2024-09-21T10:13:44.454689+0000","last_clean":"2024-09-21T10:13:44.454689+0000","last_became_active":"2024-09-21T10:09:49.917911+0000","last_became_peered":"2024-09-21T10:09:49.917911+0000","last_unstale":"2024-09-21T10:13:44.454689+0000","last_undegraded":"2024-09-21T10:13:44.454689+0000","last_fullsized":"2024-09-21T10:13:44.454689+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:30:10.567650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":275,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454443+0000","last_change":"2024-09-21T10:07:24.347937+0000","last_active":"2024-09-21T10:13:44.454443+0000","last_peered":"2024-09-21T10:13:44.454443+0000","last_clean":"2024-09-21T10:13:44.454443+0000","last_became_active":"2024-09-21T10:07:24.347376+0000","last_became_peered":"2024-09-21T10:07:24.347376+0000","last_unstale":"2024-09-21T10:13:44.454443+0000","last_undegraded":"2024-09-21T10:13:44.454443+0000","last_fullsized":"2024-09-21T10:13:44.454443+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:48:05.644824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":74,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.570984+0000","last_change":"2024-09-21T10:08:38.382518+0000","last_active":"2024-09-21T10:13:30.570984+0000","last_peered":"2024-09-21T10:13:30.570984+0000","last_clean":"2024-09-21T10:13:30.570984+0000","last_became_active":"2024-09-21T10:08:38.372051+0000","last_became_peered":"2024-09-21T10:08:38.372051+0000","last_unstale":"2024-09-21T10:13:30.570984+0000","last_undegraded":"2024-09-21T10:13:30.570984+0000","last_fullsized":"2024-09-21T10:13:30.570984+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:54:21.360435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":366,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454650+0000","last_change":"2024-09-21T10:09:24.840173+0000","last_active":"2024-09-21T10:13:44.454650+0000","last_peered":"2024-09-21T10:13:44.454650+0000","last_clean":"2024-09-21T10:13:44.454650+0000","last_became_active":"2024-09-21T10:09:24.839762+0000","last_became_peered":"2024-09-21T10:09:24.839762+0000","last_unstale":"2024-09-21T10:13:44.454650+0000","last_undegraded":"2024-09-21T10:13:44.454650+0000","last_fullsized":"2024-09-21T10:13:44.454650+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:12:53.465203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":88,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.572903+0000","last_change":"2024-09-21T10:08:38.388561+0000","last_active":"2024-09-21T10:13:30.572903+0000","last_peered":"2024-09-21T10:13:30.572903+0000","last_clean":"2024-09-21T10:13:30.572903+0000","last_became_active":"2024-09-21T10:08:38.374558+0000","last_became_peered":"2024-09-21T10:08:38.374558+0000","last_unstale":"2024-09-21T10:13:30.572903+0000","last_undegraded":"2024-09-21T10:13:30.572903+0000","last_fullsized":"2024-09-21T10:13:30.572903+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:24.111438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":450,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.454985+0000","last_change":"2024-09-21T10:09:23.840467+0000","last_active":"2024-09-21T10:13:44.454985+0000","last_peered":"2024-09-21T10:13:44.454985+0000","last_clean":"2024-09-21T10:13:44.454985+0000","last_became_active":"2024-09-21T10:09:23.840177+0000","last_became_peered":"2024-09-21T10:09:23.840177+0000","last_unstale":"2024-09-21T10:13:44.454985+0000","last_undegraded":"2024-09-21T10:13:44.454985+0000","last_fullsized":"2024-09-21T10:13:44.454985+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:17.887307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":439,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.455393+0000","last_change":"2024-09-21T10:11:05.451613+0000","last_active":"2024-09-21T10:13:44.455393+0000","last_peered":"2024-09-21T10:13:44.455393+0000","last_clean":"2024-09-21T10:13:44.455393+0000","last_became_active":"2024-09-21T10:11:05.451075+0000","last_became_peered":"2024-09-21T10:11:05.451075+0000","last_unstale":"2024-09-21T10:13:44.455393+0000","last_undegraded":"2024-09-21T10:13:44.455393+0000","last_fullsized":"2024-09-21T10:13:44.455393+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:18.505045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":124,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-21T10:13:30.570365+0000","last_change":"2024-09-21T10:08:38.380508+0000","last_active":"2024-09-21T10:13:30.570365+0000","last_peered":"2024-09-21T10:13:30.570365+0000","last_clean":"2024-09-21T10:13:30.570365+0000","last_became_active":"2024-09-21T10:08:38.380226+0000","last_became_peered":"2024-09-21T10:08:38.380226+0000","last_unstale":"2024-09-21T10:13:30.570365+0000","last_undegraded":"2024-09-21T10:13:30.570365+0000","last_fullsized":"2024-09-21T10:13:30.570365+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:50:38.190118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":104,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-09-21T10:13:44.455172+0000","last_change":"2024-09-21T10:10:57.401588+0000","last_active":"2024-09-21T10:13:44.455172+0000","last_peered":"2024-09-21T10:13:44.455172+0000","last_clean":"2024-09-21T10:13:44.455172+0000","last_became_active":"2024-09-21T10:10:57.397724+0000","last_became_peered":"2024-09-21T10:10:57.397724+0000","last_unstale":"2024-09-21T10:13:44.455172+0000","last_undegraded":"2024-09-21T10:13:44.455172+0000","last_fullsized":"2024-09-21T10:13:44.455172+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:47.843630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":11710,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26580,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":661927,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":91,"num_write":18,"num_write_kb":197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":668495,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":904504,"kb_used_data":4140,"kb_used_omap":1,"kb_used_meta":900350,"kb_avail":92840648,"statfs":{"total":95995035648,"available":95068823552,"internally_reserved":0,"allocated":4239360,"data_stored":2568344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":921958744},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019315,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":245,"seq":1052266987574,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":220396,"kb_used_data":4128,"kb_used_omap":8,"kb_used_meta":216247,"kb_avail":93524756,"statfs":{"total":95995035648,"available":95769350144,"internally_reserved":0,"allocated":4227072,"data_stored":2564187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8940,"internal_metadata":221437204},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":175,"seq":751619276923,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":369144,"kb_used_data":26220,"kb_used_omap":12,"kb_used_meta":342899,"kb_avail":93376008,"statfs":{"total":95995035648,"available":95617032192,"internally_reserved":0,"allocated":26849280,"data_stored":25178267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12845,"internal_metadata":351129043},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823086,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1510184,"kb_used_data":4060,"kb_used_omap":1,"kb_used_meta":1506110,"kb_avail":92234968,"statfs":{"total":95995035648,"available":94448607232,"internally_reserved":0,"allocated":4157440,"data_stored":2498514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1542256984},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879509,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":821480,"kb_used_data":4380,"kb_used_omap":4,"kb_used_meta":817083,"kb_avail":92923672,"statfs":{"total":95995035648,"available":95153840128,"internally_reserved":0,"allocated":4485120,"data_stored":2826275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":836693436},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566350,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":710460,"kb_used_data":4080,"kb_used_omap":6,"kb_used_meta":706361,"kb_avail":93034692,"statfs":{"total":95995035648,"available":95267524608,"internally_reserved":0,"allocated":4177920,"data_stored":2503951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":723314087},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860695,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":895264,"kb_used_data":8600,"kb_used_omap":5,"kb_used_meta":886650,"kb_avail":92849888,"statfs":{"total":95995035648,"available":95078285312,"internally_reserved":0,"allocated":8806400,"data_stored":7150735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":907929839},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7236,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9976,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:13:54.335 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T10:13:54.335 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:13:54.335 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:13:54.335 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-09-21T10:13:54.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 1 1 2024-09-21T10:13:54.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:54.410 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:54.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:54.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:54.513 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:54.537 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:54.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:54.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:54.569 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:54.580 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:54 smithi107 ceph-mon[102221]: pgmap v1044: 93 pgs: 93 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:13:54.580 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:54 smithi107 ceph-mon[105022]: pgmap v1044: 93 pgs: 93 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:13:54.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:54.660 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:54.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:54.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:54 smithi129 ceph-mon[88471]: pgmap v1044: 93 pgs: 93 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:13:54.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:54.784 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:54.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:54.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:54.886 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:54.912 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:54.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:54.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:55.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:55.037 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:55.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:55.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:55.042 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:13:55.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:55.160 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:55.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:55.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:55.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:55.284 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:55.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:55.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:55.300 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:13:55.385 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:55.407 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:55.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:55.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:55.508 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:55.530 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:55.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:55.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:55.631 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:55.653 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:55.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:55.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:55.754 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:55.776 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:55.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:55.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:55.878 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:55.899 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:55.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:55.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:56.000 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:56.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:56.024 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:56.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:56.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:56.146 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:56.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:56.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:56.247 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:56.269 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:56.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:56.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:56.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:56.392 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:56.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:56.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:56.493 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:56 smithi107 ceph-mon[102221]: pgmap v1045: 93 pgs: 93 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:13:56.493 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:56.493 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:56 smithi107 ceph-mon[105022]: pgmap v1045: 93 pgs: 93 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:13:56.494 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:56.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:13:56.510 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:56 smithi129 ceph-mon[88471]: pgmap v1045: 93 pgs: 93 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:13:56.510 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:56.514 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:56.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:56.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:13:56.616 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:56.637 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:56.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:56.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:56.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:56.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:56.761 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:56.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:56.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:56.883 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:56.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:56.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:56.985 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:57.007 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:57.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:57.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:57.064 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:13:57.064 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:13:57.064 INFO:teuthology.orchestra.run.smithi129.stdout: "num_ops": 0 2024-09-21T10:13:57.064 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:13:57.108 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:57.130 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:57.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:57.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:57.231 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:57.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:57.253 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:57.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:57.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:57.376 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:57.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:57.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:57.477 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:57.500 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:57.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:57.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:57.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:57.624 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:57.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:57.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:57.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:13:57.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:57.746 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:57.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:57.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:57.847 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:13:57.870 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:57.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:13:57.919 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:13:57.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:57.992 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:57.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:57.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:58.094 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:58.116 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:58.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:58.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:58.217 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:58.239 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:58.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:58.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:58.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:58.363 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:58.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:58.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:58.419 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_7' created 2024-09-21T10:13:58.465 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:58.486 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:58.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:58.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:58.588 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:58.609 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:58.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:58.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:58.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:58 smithi129 ceph-mon[88471]: pgmap v1046: 93 pgs: 93 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:13:58.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:58 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1106372489' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16} : dispatch 2024-09-21T10:13:58.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:58 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:58.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:58 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:58.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:58.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:58.735 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:58.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:58.836 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:13:58.857 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:58.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:58.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:13:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:58 smithi107 ceph-mon[102221]: pgmap v1046: 93 pgs: 93 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:13:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:58 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1106372489' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16} : dispatch 2024-09-21T10:13:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:58 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:58 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:58 smithi107 ceph-mon[105022]: pgmap v1046: 93 pgs: 93 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:13:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:58 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1106372489' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16} : dispatch 2024-09-21T10:13:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:58 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:58 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:58.959 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:13:58.979 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:58.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:58.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:13:59.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:59.102 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:59.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:59.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:59.203 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_7 rados --yes-i-really-mean-it || true 2024-09-21T10:13:59.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:59.224 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:59.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:59.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:13:59.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:13:59.347 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:59.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:59.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:13:59.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:59 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1106372489' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-09-21T10:13:59.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:59 smithi107 ceph-mon[102221]: osdmap e294: 8 total, 8 up, 7 in 2024-09-21T10:13:59.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:59 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:13:59.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:59 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:13:59.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:13:59 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:59.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:59 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1106372489' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-09-21T10:13:59.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:59 smithi107 ceph-mon[105022]: osdmap e294: 8 total, 8 up, 7 in 2024-09-21T10:13:59.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:59 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:13:59.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:59 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:13:59.431 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:13:59 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:59.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:59.470 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:59.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:59.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:59.534 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:13:59.534 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:13:59.534 INFO:teuthology.orchestra.run.smithi129.stdout: "complaint_time": 30, 2024-09-21T10:13:59.534 INFO:teuthology.orchestra.run.smithi129.stdout: "num_blocked_ops": 0 2024-09-21T10:13:59.534 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:13:59.545 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:13:59.572 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:13:59.593 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:13:59.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:59.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:13:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:59 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1106372489' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-09-21T10:13:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:59 smithi129 ceph-mon[88471]: osdmap e294: 8 total, 8 up, 7 in 2024-09-21T10:13:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:59 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:13:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:59 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:13:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:13:59 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:13:59.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:59.717 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:59.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:59.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:59.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:13:59.840 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:59.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:59.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:13:59.941 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:13:59.963 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:13:59.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:13:59.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:00.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:00.085 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:00.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:00.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:00.133 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:14:00.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:00.209 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:00.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:00.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:00.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:00.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:00.333 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:00.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:00.378 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:14:00.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:14:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:14:00.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:14:00.433 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:00.455 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:00.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:00.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:00.557 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:00.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:00.579 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:00.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:00.594 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:00 smithi107 ceph-mon[102221]: pgmap v1048: 109 pgs: 2 creating+peering, 14 unknown, 93 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:14:00.594 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:00 smithi107 ceph-mon[102221]: osdmap e295: 8 total, 8 up, 7 in 2024-09-21T10:14:00.594 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:00 smithi107 ceph-mon[102221]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:14:00.594 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:00 smithi107 ceph-mon[105022]: pgmap v1048: 109 pgs: 2 creating+peering, 14 unknown, 93 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:14:00.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:00 smithi107 ceph-mon[105022]: osdmap e295: 8 total, 8 up, 7 in 2024-09-21T10:14:00.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:00 smithi107 ceph-mon[105022]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:14:00.680 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:00.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:00.757 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:00.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:00.769 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:00 smithi129 ceph-mon[88471]: pgmap v1048: 109 pgs: 2 creating+peering, 14 unknown, 93 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:14:00.769 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:00 smithi129 ceph-mon[88471]: osdmap e295: 8 total, 8 up, 7 in 2024-09-21T10:14:00.769 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:00 smithi129 ceph-mon[88471]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:14:00.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:00.880 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:00.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:00.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:00.889 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:14:00.889 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:14:00.898 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:14:00.899 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:14:00.914 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:14:00.915 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:14:00.930 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:14:00.930 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:14:00.946 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:14:00.946 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:14:00.962 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:14:00.962 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:14:00.977 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:14:00.978 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:14:00.982 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:00.992 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:14:00.992 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:14:01.004 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:01.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:01.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:01.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:01.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:01.129 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:01.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:01.230 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:01.251 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:01.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:01.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:01.353 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:01.374 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:01.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:01.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:01.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:01.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:01.499 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:01.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:01.500 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:01 smithi129 ceph-mon[88471]: osdmap e296: 8 total, 8 up, 7 in 2024-09-21T10:14:01.500 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:01 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3266483342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:01.600 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:01.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:01 smithi107 ceph-mon[105022]: osdmap e296: 8 total, 8 up, 7 in 2024-09-21T10:14:01.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:01 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3266483342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:01.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:01 smithi107 ceph-mon[102221]: osdmap e296: 8 total, 8 up, 7 in 2024-09-21T10:14:01.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:01 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3266483342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:01.621 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:01.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:01.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:01.723 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:01.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:01.745 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:01.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:01.846 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:01.867 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:01.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:01.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:01.969 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:01.991 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:01.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:01.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:02.024 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:14:02.024 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:14:02.065 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:14:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:14:01] "GET /metrics HTTP/1.1" 200 34371 "" "Prometheus/2.51.0" 2024-09-21T10:14:02.093 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:02.114 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:02.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:02.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:02.216 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:02.237 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:02.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:02.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:02.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:02.362 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:02.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:02.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:02.463 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:02.485 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:02.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:02.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:02.587 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:02.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:02.610 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:02.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:02.632 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:02 smithi129 ceph-mon[88471]: pgmap v1051: 109 pgs: 2 creating+peering, 14 unknown, 93 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:14:02.655 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:14:02.711 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:02.732 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:02.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:02.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:02.787 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:02 smithi107 ceph-mon[102221]: pgmap v1051: 109 pgs: 2 creating+peering, 14 unknown, 93 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:14:02.788 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:02 smithi107 ceph-mon[105022]: pgmap v1051: 109 pgs: 2 creating+peering, 14 unknown, 93 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:14:02.823 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:02.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:02.855 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:02.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:02.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:02.900 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:14:02.957 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:02.978 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:02.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:02.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:03.079 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:03.101 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:03.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:03.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:03.202 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:03.225 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:03.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:03.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:03.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:03.347 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:03.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:03.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:03.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:03.453 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:14:03.474 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:03.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:03.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:03.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:03.596 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:03.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:03.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:03.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:03 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/907106968' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:14:03.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:03 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/907106968' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:14:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:03 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/907106968' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:14:03.699 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:03.721 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:03.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:03.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:03.822 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:03.844 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:03.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:03.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:03.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:03.967 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:03.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:03.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:04.068 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:04.090 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:04.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:04.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:04.192 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:04.213 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:04.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:04.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:04.315 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:04.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:04.338 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:04.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:04.439 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:04.454 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-21T10:14:04.462 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:04.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:04.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:04.571 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:04 smithi129 ceph-mon[88471]: pgmap v1052: 109 pgs: 2 creating+peering, 107 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:14:04.572 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:04 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:14:04.572 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:04 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/907106968' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:14:04.572 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:04 smithi129 ceph-mon[88471]: osdmap e297: 8 total, 8 up, 7 in 2024-09-21T10:14:04.572 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:04 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4223625253' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:14:04.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:04.586 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:04.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:04.687 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:04.707 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:04.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:04.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:04.796 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:14:04.796 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:14:04.809 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:04.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:14:04.831 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:04.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:04.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:04.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:04 smithi107 ceph-mon[105022]: pgmap v1052: 109 pgs: 2 creating+peering, 107 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:14:04.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:04 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:14:04.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:04 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/907106968' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:14:04.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:04 smithi107 ceph-mon[105022]: osdmap e297: 8 total, 8 up, 7 in 2024-09-21T10:14:04.852 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:04 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4223625253' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:14:04.852 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:04 smithi107 ceph-mon[102221]: pgmap v1052: 109 pgs: 2 creating+peering, 107 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:14:04.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:04 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:14:04.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:04 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/907106968' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:14:04.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:04 smithi107 ceph-mon[102221]: osdmap e297: 8 total, 8 up, 7 in 2024-09-21T10:14:04.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:04 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4223625253' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:14:04.933 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:04.957 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:04.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:04.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:05.058 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:05.080 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:05.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:05.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:05.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:05.203 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:05.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:05.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:05.304 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:05.326 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:05.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:05.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:05.428 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:05.452 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:05.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:05.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:05.457 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:14:05.554 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:05.575 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:05.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:05.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:05.677 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:05.700 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:05.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:05.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:05.745 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:14:05.802 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:05.823 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:05.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:05.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:05.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:05 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4223625253' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:14:05.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:05 smithi107 ceph-mon[102221]: osdmap e298: 8 total, 8 up, 7 in 2024-09-21T10:14:05.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:05 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4223625253' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:14:05.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:05 smithi107 ceph-mon[105022]: osdmap e298: 8 total, 8 up, 7 in 2024-09-21T10:14:05.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:05 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4223625253' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:14:05.866 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:05 smithi129 ceph-mon[88471]: osdmap e298: 8 total, 8 up, 7 in 2024-09-21T10:14:05.925 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:05.947 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:05.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:05.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:06.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:06.070 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:06.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:06.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:06.172 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:06.188 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:14:06.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:06.195 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:06.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:06.296 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:06.318 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:06.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:06.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:06.419 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:06.440 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:06.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:06.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:06.468 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:06.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:06.563 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:06.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:06.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:06.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:06.686 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:06.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:06.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:06.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:06 smithi107 ceph-mon[102221]: pgmap v1055: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:14:06.777 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:06 smithi107 ceph-mon[102221]: osdmap e299: 8 total, 8 up, 7 in 2024-09-21T10:14:06.778 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:06 smithi107 ceph-mon[105022]: pgmap v1055: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:14:06.778 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:06 smithi107 ceph-mon[105022]: osdmap e299: 8 total, 8 up, 7 in 2024-09-21T10:14:06.788 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:06.810 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:06.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:06.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:06.905 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:06 smithi129 ceph-mon[88471]: pgmap v1055: 109 pgs: 109 active+clean; 652 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:14:06.905 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:06 smithi129 ceph-mon[88471]: osdmap e299: 8 total, 8 up, 7 in 2024-09-21T10:14:06.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:06.934 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:06.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:06.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:07.035 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:07.057 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:07.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:07.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:07.159 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:07.181 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:07.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:07.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:07.251 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:07.283 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:07.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:07.305 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:07.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:07.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:07.427 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:07.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:07.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:07.433 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:14:07.433 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:14:07.470 INFO:teuthology.orchestra.run.smithi107.stderr:enabled application 'rados' on pool 'unique_pool_7' 2024-09-21T10:14:07.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:07.551 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:07.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:07.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:07.653 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:07.674 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:07.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:07.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:07.728 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:07 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3514054131' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:14:07.728 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:07 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:14:07.728 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:07 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3514054131' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:14:07.728 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:07 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:14:07.754 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:07 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3514054131' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:14:07.754 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:07 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:14:07.776 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:07.797 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:07.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:07.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:07.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:07.920 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:07.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:07.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:08.021 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:08.043 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:08.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:08.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:08.070 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:14:08.145 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:08.174 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:08.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:08.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:08.276 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:08.298 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:08.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:08.357 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:14:08.401 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:08.421 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:08.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:08.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:08.476 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:14:08.476 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:14:08.476 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:14:08.476 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:14:08.476 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:14:08.476 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:14:08.476 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:14:08.477 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:14:08.477 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:14:08.477 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:14:08.477 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:14:08.477 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:14:08.486 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:14:08.487 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:14:08.503 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:14:08.503 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:14:08.522 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:08.544 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:08.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:08.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:08.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:08 smithi107 ceph-mon[102221]: pgmap v1057: 109 pgs: 109 active+clean; 634 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:14:08.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:08 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:14:08.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:08 smithi107 ceph-mon[102221]: osdmap e300: 8 total, 8 up, 7 in 2024-09-21T10:14:08.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:08 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3245764908' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:08.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:08 smithi107 ceph-mon[105022]: pgmap v1057: 109 pgs: 109 active+clean; 634 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:14:08.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:08 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:14:08.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:08 smithi107 ceph-mon[105022]: osdmap e300: 8 total, 8 up, 7 in 2024-09-21T10:14:08.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:08 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3245764908' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:08.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:08.666 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:08.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:08.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:08.741 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:08 smithi129 ceph-mon[88471]: pgmap v1057: 109 pgs: 109 active+clean; 634 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:14:08.741 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:08 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:14:08.741 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:08 smithi129 ceph-mon[88471]: osdmap e300: 8 total, 8 up, 7 in 2024-09-21T10:14:08.741 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:08 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3245764908' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:08.768 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:08.789 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:08.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:08.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:08.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:08.912 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:08.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:08.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:09.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:09.035 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:09.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:09.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:09.137 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:09.160 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:09.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:09.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:09.262 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:09.284 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:09.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:09.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:09.386 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:09.407 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:09.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:09.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:09.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:09.531 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:09.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:09.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:09.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:09.655 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:09.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:09.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:09.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:09.758 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:09 smithi129 ceph-mon[88471]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:14:09.776 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:09 smithi107 ceph-mon[102221]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:14:09.776 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:09 smithi107 ceph-mon[105022]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:14:09.778 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:09.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:09.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:09.787 DEBUG:teuthology.orchestra.run.smithi107:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_7 bench 90 write' 2024-09-21T10:14:09.821 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:14:09.821 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:14:09.865 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:hints = 1 2024-09-21T10:14:09.865 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:14:09.865 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object prefix: benchmark_data_smithi107_250110 2024-09-21T10:14:09.866 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:14:09.866 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:14:09.879 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:09.913 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:09.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:09.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:10.013 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:14:10.013 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 20, 2024-09-21T10:14:10.014 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 600, 2024-09-21T10:14:10.014 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:14:10.014 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.014 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95099 10.6 10:64ab5024:::benchmark_data_smithi107_184135_object95098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T10:14:10.014 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.388487+0000", 2024-09-21T10:14:10.014 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 483.62089555799997, 2024-09-21T10:14:10.014 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.019649673, 2024-09-21T10:14:10.014 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:10.014 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:14:10.014 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:10.014 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:14:10.014 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:14:10.014 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95099 2024-09-21T10:14:10.014 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.014 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:10.015 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.015 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:10.015 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388487+0000", 2024-09-21T10:14:10.015 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.015 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.015 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.015 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:10.015 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388487+0000", 2024-09-21T10:14:10.015 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.015 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.015 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.015 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:10.015 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388485+0000", 2024-09-21T10:14:10.016 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:14:10.016 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.016 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.016 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:10.016 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388545+0000", 2024-09-21T10:14:10.016 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0622999999999999e-05 2024-09-21T10:14:10.016 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.016 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.016 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:10.016 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388561+0000", 2024-09-21T10:14:10.016 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5667000000000001e-05 2024-09-21T10:14:10.016 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.016 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.017 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:10.017 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388567+0000", 2024-09-21T10:14:10.017 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8640000000000001e-06 2024-09-21T10:14:10.017 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.017 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.017 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.017 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388616+0000", 2024-09-21T10:14:10.017 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9168000000000001e-05 2024-09-21T10:14:10.017 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.017 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.017 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:10.017 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388773+0000", 2024-09-21T10:14:10.017 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000157465 2024-09-21T10:14:10.017 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.018 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.018 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:14:10.018 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.388903+0000", 2024-09-21T10:14:10.018 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012996399999999999 2024-09-21T10:14:10.018 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.018 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.018 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:10.018 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.400728+0000", 2024-09-21T10:14:10.018 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.01182456 2024-09-21T10:14:10.018 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.018 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.018 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.018 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407770+0000", 2024-09-21T10:14:10.019 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0070416530000000001 2024-09-21T10:14:10.019 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.019 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.019 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:10.019 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.407778+0000", 2024-09-21T10:14:10.019 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9549999999999992e-06 2024-09-21T10:14:10.019 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.019 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.019 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.019 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.408044+0000", 2024-09-21T10:14:10.019 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.000266069 2024-09-21T10:14:10.019 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.019 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.020 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:10.020 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.408136+0000", 2024-09-21T10:14:10.020 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.2651000000000005e-05 2024-09-21T10:14:10.020 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.020 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.020 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.020 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.020 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.020 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95101 10.6 10:6a273935:::benchmark_data_smithi107_184135_object95100:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T10:14:10.020 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.389198+0000", 2024-09-21T10:14:10.020 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 483.62018396600001, 2024-09-21T10:14:10.020 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0191866650000001, 2024-09-21T10:14:10.020 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:10.020 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:14:10.020 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:10.021 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:14:10.021 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:14:10.021 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95101 2024-09-21T10:14:10.021 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.021 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:10.021 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.021 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:10.021 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.389198+0000", 2024-09-21T10:14:10.021 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.021 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.021 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.021 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:10.021 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.389198+0000", 2024-09-21T10:14:10.021 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.021 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.022 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.022 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:10.022 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.389197+0000", 2024-09-21T10:14:10.022 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:14:10.022 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.022 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.022 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:10.022 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.389230+0000", 2024-09-21T10:14:10.022 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3234000000000001e-05 2024-09-21T10:14:10.022 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.022 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.022 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:10.022 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.389246+0000", 2024-09-21T10:14:10.022 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5705999999999998e-05 2024-09-21T10:14:10.023 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.023 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.023 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:10.023 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.389253+0000", 2024-09-21T10:14:10.023 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8990000000000004e-06 2024-09-21T10:14:10.023 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.023 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.023 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.023 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.393452+0000", 2024-09-21T10:14:10.023 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.004198757 2024-09-21T10:14:10.023 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.023 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.023 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:10.023 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.393568+0000", 2024-09-21T10:14:10.023 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011586699999999999 2024-09-21T10:14:10.024 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.024 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.024 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:14:10.024 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.393687+0000", 2024-09-21T10:14:10.024 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000119133 2024-09-21T10:14:10.024 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.024 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.024 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.024 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.408391+0000", 2024-09-21T10:14:10.024 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.014703842 2024-09-21T10:14:10.024 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.024 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.024 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:10.024 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.408398+0000", 2024-09-21T10:14:10.025 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9349999999999997e-06 2024-09-21T10:14:10.025 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.025 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.025 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.025 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.408337+0000", 2024-09-21T10:14:10.025 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.999939103 2024-09-21T10:14:10.025 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.025 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.025 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:10.025 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.408385+0000", 2024-09-21T10:14:10.025 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8242000000000003e-05 2024-09-21T10:14:10.025 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.025 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.025 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.025 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.026 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.026 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247256 10.6 10:6298720f:::benchmark_data_smithi107_184135_object115690:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:14:10.026 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:50.433070+0000", 2024-09-21T10:14:10.026 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 319.576312105, 2024-09-21T10:14:10.026 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.99909955800000005, 2024-09-21T10:14:10.026 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:10.026 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:10.026 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:10.026 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:14:10.026 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:14:10.026 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247256 2024-09-21T10:14:10.026 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.026 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:10.026 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.026 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:10.027 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433070+0000", 2024-09-21T10:14:10.027 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.027 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.027 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.027 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:10.028 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:10.028 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433070+0000", 2024-09-21T10:14:10.028 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.028 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.028 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.028 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:10.028 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433067+0000", 2024-09-21T10:14:10.028 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999971 2024-09-21T10:14:10.028 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.028 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.029 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:10.029 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433079+0000", 2024-09-21T10:14:10.029 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1345000000000001e-05 2024-09-21T10:14:10.029 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.029 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.029 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:10.029 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433099+0000", 2024-09-21T10:14:10.029 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.0852e-05 2024-09-21T10:14:10.029 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.029 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.029 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:10.029 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433108+0000", 2024-09-21T10:14:10.029 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.8270000000000004e-06 2024-09-21T10:14:10.029 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.029 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.030 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.030 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433168+0000", 2024-09-21T10:14:10.030 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9737000000000001e-05 2024-09-21T10:14:10.030 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.030 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.030 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:10.030 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433179+0000", 2024-09-21T10:14:10.030 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0577000000000001e-05 2024-09-21T10:14:10.030 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.030 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.030 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.030 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.417734+0000", 2024-09-21T10:14:10.030 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.98455554300000003 2024-09-21T10:14:10.030 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.030 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.031 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:10.031 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.417856+0000", 2024-09-21T10:14:10.031 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000121657 2024-09-21T10:14:10.031 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.031 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.031 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:14:10.031 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.417974+0000", 2024-09-21T10:14:10.031 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000118348 2024-09-21T10:14:10.031 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.031 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.031 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:10.031 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.421532+0000", 2024-09-21T10:14:10.031 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003558205 2024-09-21T10:14:10.031 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.031 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.032 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:10.032 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432010+0000", 2024-09-21T10:14:10.032 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.010477765999999999 2024-09-21T10:14:10.032 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.032 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.032 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:10.032 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432019+0000", 2024-09-21T10:14:10.032 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.1379999999999996e-06 2024-09-21T10:14:10.032 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.032 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.032 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:10.032 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432070+0000", 2024-09-21T10:14:10.032 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0719e-05 2024-09-21T10:14:10.033 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.033 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.033 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:10.033 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432170+0000", 2024-09-21T10:14:10.033 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.9630000000000007e-05 2024-09-21T10:14:10.033 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.033 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.033 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.033 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.033 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.033 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247262 10.6 10:67e2f4ef:::benchmark_data_smithi107_184135_object115696:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:14:10.033 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:50.439088+0000", 2024-09-21T10:14:10.033 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 319.570294214, 2024-09-21T10:14:10.033 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.99353488599999995, 2024-09-21T10:14:10.033 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:10.033 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:10.034 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:10.034 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:14:10.034 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:14:10.034 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247262 2024-09-21T10:14:10.034 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.034 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:10.034 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.034 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:10.034 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439088+0000", 2024-09-21T10:14:10.034 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.034 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.034 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.034 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:10.034 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439088+0000", 2024-09-21T10:14:10.034 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.035 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.035 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.035 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:10.035 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439086+0000", 2024-09-21T10:14:10.035 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:14:10.035 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.035 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.035 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:10.035 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439094+0000", 2024-09-21T10:14:10.035 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.7840000000000004e-06 2024-09-21T10:14:10.035 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.035 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.035 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:10.035 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439111+0000", 2024-09-21T10:14:10.035 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.7059e-05 2024-09-21T10:14:10.036 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.036 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.036 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:10.036 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439118+0000", 2024-09-21T10:14:10.036 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5239999999999997e-06 2024-09-21T10:14:10.036 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.036 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.036 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.036 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439172+0000", 2024-09-21T10:14:10.036 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4327999999999999e-05 2024-09-21T10:14:10.036 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.036 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.036 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:10.036 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439185+0000", 2024-09-21T10:14:10.036 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3502999999999999e-05 2024-09-21T10:14:10.037 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.037 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.037 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.037 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.421553+0000", 2024-09-21T10:14:10.037 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.98236765199999998 2024-09-21T10:14:10.037 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.037 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.037 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:10.037 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.421626+0000", 2024-09-21T10:14:10.037 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3146999999999996e-05 2024-09-21T10:14:10.037 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.037 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.037 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:14:10.037 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.421685+0000", 2024-09-21T10:14:10.037 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8621e-05 2024-09-21T10:14:10.038 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.038 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.038 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:10.038 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432293+0000", 2024-09-21T10:14:10.038 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.010608332 2024-09-21T10:14:10.038 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.038 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.038 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:10.038 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432344+0000", 2024-09-21T10:14:10.038 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1279999999999997e-05 2024-09-21T10:14:10.038 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.038 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.038 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:10.039 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432459+0000", 2024-09-21T10:14:10.039 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000114418 2024-09-21T10:14:10.039 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.039 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.039 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:10.039 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432518+0000", 2024-09-21T10:14:10.039 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9299000000000003e-05 2024-09-21T10:14:10.039 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.039 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.039 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:10.039 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432623+0000", 2024-09-21T10:14:10.039 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010468599999999999 2024-09-21T10:14:10.039 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.039 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.039 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.039 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.040 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.040 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247267 10.6 10:615d1e96:::benchmark_data_smithi107_184135_object115701:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:14:10.040 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:50.444059+0000", 2024-09-21T10:14:10.040 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 319.56532334500002, 2024-09-21T10:14:10.040 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.98879305900000003, 2024-09-21T10:14:10.040 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:10.040 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:10.040 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:10.040 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:14:10.040 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:14:10.040 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247267 2024-09-21T10:14:10.040 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.040 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:10.040 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.040 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:10.041 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444059+0000", 2024-09-21T10:14:10.041 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.041 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.041 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.041 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:10.041 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444059+0000", 2024-09-21T10:14:10.041 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.041 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.041 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.041 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:10.041 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444057+0000", 2024-09-21T10:14:10.041 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:14:10.041 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.041 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.041 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:10.042 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444064+0000", 2024-09-21T10:14:10.042 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.5349999999999999e-06 2024-09-21T10:14:10.042 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.042 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.042 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:10.042 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444080+0000", 2024-09-21T10:14:10.042 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5478999999999999e-05 2024-09-21T10:14:10.042 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.042 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.042 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:10.042 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444086+0000", 2024-09-21T10:14:10.042 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.631e-06 2024-09-21T10:14:10.042 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.042 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.043 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.043 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444247+0000", 2024-09-21T10:14:10.043 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00016138 2024-09-21T10:14:10.043 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.043 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.043 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:10.043 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444256+0000", 2024-09-21T10:14:10.043 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.3510000000000008e-06 2024-09-21T10:14:10.043 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.043 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.043 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.043 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.424581+0000", 2024-09-21T10:14:10.043 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.98032498800000001 2024-09-21T10:14:10.043 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.043 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.044 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:10.044 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.424692+0000", 2024-09-21T10:14:10.044 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000111113 2024-09-21T10:14:10.044 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.044 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.044 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:14:10.044 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.424815+0000", 2024-09-21T10:14:10.044 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012274900000000001 2024-09-21T10:14:10.044 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.044 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.044 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:10.044 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432299+0000", 2024-09-21T10:14:10.044 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0074834740000000004 2024-09-21T10:14:10.044 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.045 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.045 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:10.045 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432681+0000", 2024-09-21T10:14:10.045 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00038213 2024-09-21T10:14:10.045 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.045 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.045 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:10.045 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432722+0000", 2024-09-21T10:14:10.045 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.138e-05 2024-09-21T10:14:10.045 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.045 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.045 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:10.045 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432765+0000", 2024-09-21T10:14:10.045 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2769000000000003e-05 2024-09-21T10:14:10.045 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.045 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.046 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:10.046 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432852+0000", 2024-09-21T10:14:10.046 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.7026000000000003e-05 2024-09-21T10:14:10.046 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.046 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.046 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.046 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.046 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.046 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247272 10.6 10:699abf74:::benchmark_data_smithi107_184135_object115706:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:14:10.046 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:50.447705+0000", 2024-09-21T10:14:10.046 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 319.56167750600002, 2024-09-21T10:14:10.046 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.985396304, 2024-09-21T10:14:10.046 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:10.046 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:10.047 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:10.047 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:14:10.047 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:14:10.047 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247272 2024-09-21T10:14:10.047 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.047 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:10.047 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.047 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:10.047 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447705+0000", 2024-09-21T10:14:10.047 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.047 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.047 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.047 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:10.047 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447705+0000", 2024-09-21T10:14:10.047 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.048 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.048 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.048 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:10.048 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447703+0000", 2024-09-21T10:14:10.048 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:14:10.048 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.048 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.048 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:10.048 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447711+0000", 2024-09-21T10:14:10.048 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.5229999999999998e-06 2024-09-21T10:14:10.048 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.048 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.048 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:10.048 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447721+0000", 2024-09-21T10:14:10.048 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0040999999999999e-05 2024-09-21T10:14:10.049 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.049 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.049 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:10.049 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447725+0000", 2024-09-21T10:14:10.049 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2679999999999996e-06 2024-09-21T10:14:10.049 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.049 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.049 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.049 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447767+0000", 2024-09-21T10:14:10.049 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1897999999999998e-05 2024-09-21T10:14:10.049 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.049 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.049 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:10.049 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447776+0000", 2024-09-21T10:14:10.049 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.2089999999999994e-06 2024-09-21T10:14:10.050 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.050 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.050 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.050 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.427698+0000", 2024-09-21T10:14:10.050 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.97992116600000001 2024-09-21T10:14:10.050 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.050 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.050 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:10.050 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.427813+0000", 2024-09-21T10:14:10.050 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000115301 2024-09-21T10:14:10.050 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.050 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.050 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:14:10.050 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.427925+0000", 2024-09-21T10:14:10.050 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000112324 2024-09-21T10:14:10.051 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.051 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.051 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:10.051 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432357+0000", 2024-09-21T10:14:10.051 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0044316570000000003 2024-09-21T10:14:10.051 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.051 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.051 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:10.051 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432891+0000", 2024-09-21T10:14:10.051 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00053454999999999996 2024-09-21T10:14:10.051 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.051 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.051 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:10.051 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432945+0000", 2024-09-21T10:14:10.051 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3307999999999999e-05 2024-09-21T10:14:10.051 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.052 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.052 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:10.052 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.433014+0000", 2024-09-21T10:14:10.052 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8949999999999995e-05 2024-09-21T10:14:10.052 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.052 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.052 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:10.052 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.433101+0000", 2024-09-21T10:14:10.052 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.7322000000000002e-05 2024-09-21T10:14:10.052 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.052 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.052 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.052 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.052 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.052 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:96818 11.5 e269/267 11:ad788e1e:::benchmark_data_smithi107_218812_object96817:head v 269'5821, mlcod=257'161)", 2024-09-21T10:14:10.053 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:11:04.703675+0000", 2024-09-21T10:14:10.053 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 185.30570699099999, 2024-09-21T10:14:10.053 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.046626753999999999, 2024-09-21T10:14:10.053 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:10.053 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:10.053 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:10.053 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.053 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:10.053 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:04.703675+0000", 2024-09-21T10:14:10.053 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.053 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.053 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.053 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:10.054 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:04.703675+0000", 2024-09-21T10:14:10.054 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.054 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.054 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.054 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:10.054 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:04.703674+0000", 2024-09-21T10:14:10.054 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:14:10.055 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.055 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.055 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:10.055 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:04.703699+0000", 2024-09-21T10:14:10.055 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.457e-05 2024-09-21T10:14:10.055 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.055 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.055 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:10.055 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:04.703713+0000", 2024-09-21T10:14:10.055 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4326e-05 2024-09-21T10:14:10.055 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.055 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.055 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:10.055 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:04.703719+0000", 2024-09-21T10:14:10.056 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4659999999999998e-06 2024-09-21T10:14:10.056 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.056 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.056 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.056 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:04.706197+0000", 2024-09-21T10:14:10.056 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0024780190000000001 2024-09-21T10:14:10.056 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.056 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.056 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:10.056 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:04.706220+0000", 2024-09-21T10:14:10.056 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.3298e-05 2024-09-21T10:14:10.056 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.056 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.056 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:10.056 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:04.750268+0000", 2024-09-21T10:14:10.057 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.044048205999999999 2024-09-21T10:14:10.057 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.057 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.057 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:10.057 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:04.750302+0000", 2024-09-21T10:14:10.057 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3538999999999999e-05 2024-09-21T10:14:10.057 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.057 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.057 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.057 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.057 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.057 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:97127 11.5 e269/267 11:a0ed3385:::benchmark_data_smithi107_218812_object97126:head v 269'5846, mlcod=257'167)", 2024-09-21T10:14:10.057 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:11:05.012651+0000", 2024-09-21T10:14:10.057 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 184.99673091899999, 2024-09-21T10:14:10.057 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.052333383999999997, 2024-09-21T10:14:10.057 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:10.058 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:10.058 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:10.058 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.058 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:10.058 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.012651+0000", 2024-09-21T10:14:10.058 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.058 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.058 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.058 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:10.058 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.012651+0000", 2024-09-21T10:14:10.058 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.058 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.058 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.058 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:10.059 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.012651+0000", 2024-09-21T10:14:10.059 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:14:10.059 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.059 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.059 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:10.059 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.012689+0000", 2024-09-21T10:14:10.059 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8645e-05 2024-09-21T10:14:10.059 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.059 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.059 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:10.059 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.012702+0000", 2024-09-21T10:14:10.059 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3049e-05 2024-09-21T10:14:10.059 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.059 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.059 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:10.060 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.012707+0000", 2024-09-21T10:14:10.060 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.079e-06 2024-09-21T10:14:10.060 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.060 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.060 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.060 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.015154+0000", 2024-09-21T10:14:10.060 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002446466 2024-09-21T10:14:10.060 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.060 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.060 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:10.060 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.015180+0000", 2024-09-21T10:14:10.060 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.6118999999999998e-05 2024-09-21T10:14:10.060 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.060 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.060 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:10.061 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.064954+0000", 2024-09-21T10:14:10.061 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.049773533000000002 2024-09-21T10:14:10.061 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.061 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.061 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:10.061 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.064985+0000", 2024-09-21T10:14:10.061 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1096000000000002e-05 2024-09-21T10:14:10.061 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.061 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.061 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.061 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.061 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.061 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:97129 11.5 e269/267 11:a054297d:::benchmark_data_smithi107_218812_object97128:head v 269'5847, mlcod=257'167)", 2024-09-21T10:14:10.061 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:11:05.017578+0000", 2024-09-21T10:14:10.061 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 184.991803906, 2024-09-21T10:14:10.062 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.047423854000000001, 2024-09-21T10:14:10.062 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:10.062 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:10.062 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:10.062 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.062 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:10.062 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.017578+0000", 2024-09-21T10:14:10.062 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.062 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.062 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.062 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:10.062 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.017578+0000", 2024-09-21T10:14:10.062 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.062 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.063 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.063 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:10.063 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.017576+0000", 2024-09-21T10:14:10.063 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:14:10.063 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.063 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.063 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:10.063 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.017606+0000", 2024-09-21T10:14:10.063 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0020000000000001e-05 2024-09-21T10:14:10.063 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.063 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.063 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:10.063 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.017626+0000", 2024-09-21T10:14:10.063 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.9415e-05 2024-09-21T10:14:10.063 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.064 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.064 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:10.064 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.017633+0000", 2024-09-21T10:14:10.064 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.6040000000000004e-06 2024-09-21T10:14:10.064 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.064 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.064 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.064 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.023067+0000", 2024-09-21T10:14:10.064 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.005433284 2024-09-21T10:14:10.064 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.064 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.064 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:10.064 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.023096+0000", 2024-09-21T10:14:10.064 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.9111999999999998e-05 2024-09-21T10:14:10.065 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.065 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.065 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:10.065 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.064991+0000", 2024-09-21T10:14:10.065 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.041895108 2024-09-21T10:14:10.065 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.065 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.065 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:10.065 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.065002+0000", 2024-09-21T10:14:10.065 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1280999999999999e-05 2024-09-21T10:14:10.065 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.065 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.065 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.066 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.066 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.066 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:97142 11.5 e269/267 11:aa3cf3e1:::benchmark_data_smithi107_218812_object97141:head v 269'5848, mlcod=257'167)", 2024-09-21T10:14:10.066 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:11:05.019829+0000", 2024-09-21T10:14:10.066 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 184.98955304699999, 2024-09-21T10:14:10.066 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.045184593000000002, 2024-09-21T10:14:10.066 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:10.066 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:10.066 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:10.066 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.066 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:10.066 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.019829+0000", 2024-09-21T10:14:10.066 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.066 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.067 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.067 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:10.067 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.019829+0000", 2024-09-21T10:14:10.067 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.067 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.067 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.067 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:10.067 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.019828+0000", 2024-09-21T10:14:10.067 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:14:10.067 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.067 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.067 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:10.067 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.019852+0000", 2024-09-21T10:14:10.067 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.3932000000000001e-05 2024-09-21T10:14:10.067 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.067 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.068 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:10.068 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.019868+0000", 2024-09-21T10:14:10.068 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5569e-05 2024-09-21T10:14:10.068 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.068 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.068 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:10.068 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.019873+0000", 2024-09-21T10:14:10.068 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4940000000000003e-06 2024-09-21T10:14:10.068 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.068 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.068 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.068 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.031721+0000", 2024-09-21T10:14:10.068 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.011848152000000001 2024-09-21T10:14:10.068 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.069 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.069 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:10.069 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.031751+0000", 2024-09-21T10:14:10.069 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0111999999999999e-05 2024-09-21T10:14:10.069 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.069 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.069 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:10.069 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.065004+0000", 2024-09-21T10:14:10.069 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.033252219999999999 2024-09-21T10:14:10.069 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.069 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.069 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:10.069 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.065014+0000", 2024-09-21T10:14:10.070 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0039e-05 2024-09-21T10:14:10.070 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.070 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.070 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.070 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.070 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.070 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:108027 11.5 e272/267 11:a08c65b5:::benchmark_data_smithi107_218812_object108026:head v 272'6477, mlcod=257'547)", 2024-09-21T10:14:10.070 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:11:17.585710+0000", 2024-09-21T10:14:10.070 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 172.423672063, 2024-09-21T10:14:10.070 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.044484076999999997, 2024-09-21T10:14:10.070 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:10.070 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:10.070 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:10.070 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.071 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:10.071 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.585710+0000", 2024-09-21T10:14:10.071 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.071 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.071 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.071 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:10.071 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.585710+0000", 2024-09-21T10:14:10.071 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.071 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.071 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.071 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:10.071 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.585709+0000", 2024-09-21T10:14:10.071 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:14:10.071 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.072 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.072 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:10.072 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.585754+0000", 2024-09-21T10:14:10.072 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4891000000000001e-05 2024-09-21T10:14:10.072 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.072 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.072 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:10.072 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.585770+0000", 2024-09-21T10:14:10.072 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5889e-05 2024-09-21T10:14:10.072 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.072 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.072 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:10.072 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.585777+0000", 2024-09-21T10:14:10.072 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8199999999999999e-06 2024-09-21T10:14:10.073 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.073 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.073 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.073 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.591815+0000", 2024-09-21T10:14:10.073 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0060383640000000001 2024-09-21T10:14:10.073 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.073 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.073 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:10.073 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.591838+0000", 2024-09-21T10:14:10.073 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.2725999999999999e-05 2024-09-21T10:14:10.073 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.073 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.073 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:10.073 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.630151+0000", 2024-09-21T10:14:10.074 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.038313543999999998 2024-09-21T10:14:10.074 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.074 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.074 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:10.074 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.630194+0000", 2024-09-21T10:14:10.074 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2932999999999997e-05 2024-09-21T10:14:10.074 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.074 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.074 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.074 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.074 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.074 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:108070 11.5 e272/267 11:a28615af:::benchmark_data_smithi107_218812_object108069:head v 272'6484, mlcod=257'549)", 2024-09-21T10:14:10.074 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:11:17.662139+0000", 2024-09-21T10:14:10.074 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 172.347242838, 2024-09-21T10:14:10.075 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.042853980999999999, 2024-09-21T10:14:10.075 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:10.075 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:10.075 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:10.075 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.075 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:10.075 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.662139+0000", 2024-09-21T10:14:10.075 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.075 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.075 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.075 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:10.075 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.662139+0000", 2024-09-21T10:14:10.075 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.075 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.075 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.076 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:10.076 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.662138+0000", 2024-09-21T10:14:10.076 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:14:10.076 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.076 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.076 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:10.076 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.662163+0000", 2024-09-21T10:14:10.076 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.4703e-05 2024-09-21T10:14:10.076 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.076 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.076 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:10.076 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.662178+0000", 2024-09-21T10:14:10.076 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5407e-05 2024-09-21T10:14:10.076 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.077 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.077 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:10.077 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.662184+0000", 2024-09-21T10:14:10.077 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7420000000000003e-06 2024-09-21T10:14:10.077 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.077 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.077 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.077 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.662256+0000", 2024-09-21T10:14:10.077 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2564000000000005e-05 2024-09-21T10:14:10.077 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.077 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.077 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:10.077 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.662276+0000", 2024-09-21T10:14:10.077 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.9579999999999999e-05 2024-09-21T10:14:10.078 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.078 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.078 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:10.078 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.704954+0000", 2024-09-21T10:14:10.078 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.042678649999999999 2024-09-21T10:14:10.078 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.078 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.078 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:10.078 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.704993+0000", 2024-09-21T10:14:10.078 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8804000000000003e-05 2024-09-21T10:14:10.078 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.078 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.078 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.078 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.079 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.079 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:128494 11.5 e275/267 11:a159820e:::benchmark_data_smithi107_218812_object16007:head v 275'7663, mlcod=258'1538)", 2024-09-21T10:14:10.079 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:11:41.955295+0000", 2024-09-21T10:14:10.079 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 148.05408726300001, 2024-09-21T10:14:10.079 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.046634125999999998, 2024-09-21T10:14:10.079 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:10.079 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:10.079 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:10.079 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.079 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:10.079 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:41.955295+0000", 2024-09-21T10:14:10.079 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.080 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.080 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.080 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:10.080 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:41.955295+0000", 2024-09-21T10:14:10.080 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.080 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.080 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.080 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:10.080 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:41.955294+0000", 2024-09-21T10:14:10.080 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:14:10.080 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.080 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.080 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:10.080 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:41.955301+0000", 2024-09-21T10:14:10.080 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.1069999999999999e-06 2024-09-21T10:14:10.081 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.081 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.081 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:10.081 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:41.955315+0000", 2024-09-21T10:14:10.081 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3594000000000001e-05 2024-09-21T10:14:10.081 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.081 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.081 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:10.081 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:41.955320+0000", 2024-09-21T10:14:10.081 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3159999999999996e-06 2024-09-21T10:14:10.081 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.081 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.081 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.081 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:41.958211+0000", 2024-09-21T10:14:10.082 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0028911850000000001 2024-09-21T10:14:10.082 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.082 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.082 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:10.082 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:41.958239+0000", 2024-09-21T10:14:10.082 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7653000000000001e-05 2024-09-21T10:14:10.082 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.082 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.082 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:10.082 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:42.001875+0000", 2024-09-21T10:14:10.082 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.043635707000000003 2024-09-21T10:14:10.082 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.082 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.083 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:10.083 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:42.001929+0000", 2024-09-21T10:14:10.083 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4194000000000003e-05 2024-09-21T10:14:10.083 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.083 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.083 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.083 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.083 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.083 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:162164 11.5 e277/267 11:a601c716:::benchmark_data_smithi107_218812_object49677:head v 277'9743, mlcod=261'3752)", 2024-09-21T10:14:10.083 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:12:19.752607+0000", 2024-09-21T10:14:10.083 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 110.256775359, 2024-09-21T10:14:10.083 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.020691927999999998, 2024-09-21T10:14:10.083 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:10.083 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:10.084 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:10.084 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.084 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:10.084 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:19.752607+0000", 2024-09-21T10:14:10.084 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.084 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.084 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.084 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:10.084 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:19.752607+0000", 2024-09-21T10:14:10.084 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.084 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.084 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.084 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:10.084 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:19.752606+0000", 2024-09-21T10:14:10.084 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:14:10.085 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.085 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.085 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:10.085 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:19.752613+0000", 2024-09-21T10:14:10.085 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.0639999999999996e-06 2024-09-21T10:14:10.085 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.085 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.085 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:10.085 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:19.752625+0000", 2024-09-21T10:14:10.085 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1663e-05 2024-09-21T10:14:10.085 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.085 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.085 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:10.085 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:19.752630+0000", 2024-09-21T10:14:10.085 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4999999999999999e-06 2024-09-21T10:14:10.085 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.086 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.086 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.086 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:19.757109+0000", 2024-09-21T10:14:10.086 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0044790840000000004 2024-09-21T10:14:10.086 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.086 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.086 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:10.086 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:19.757134+0000", 2024-09-21T10:14:10.086 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.4734000000000001e-05 2024-09-21T10:14:10.086 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.086 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.086 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:10.086 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:19.773270+0000", 2024-09-21T10:14:10.086 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.016136177000000002 2024-09-21T10:14:10.086 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.087 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.087 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:10.087 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:19.773299+0000", 2024-09-21T10:14:10.087 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.8343999999999998e-05 2024-09-21T10:14:10.087 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.087 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.087 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.087 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.087 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.087 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:164615 11.5 e277/267 11:af1c5d77:::benchmark_data_smithi107_218812_object52128:head v 277'9902, mlcod=263'3963)", 2024-09-21T10:14:10.087 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:12:22.259364+0000", 2024-09-21T10:14:10.087 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 107.750018459, 2024-09-21T10:14:10.087 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.021990651, 2024-09-21T10:14:10.087 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:10.088 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:10.088 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:10.088 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.088 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:10.088 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:22.259364+0000", 2024-09-21T10:14:10.088 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.088 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.088 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.088 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:10.088 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:22.259364+0000", 2024-09-21T10:14:10.088 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.088 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.088 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.088 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:10.088 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:22.259363+0000", 2024-09-21T10:14:10.088 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:14:10.089 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.089 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.089 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:10.089 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:22.259379+0000", 2024-09-21T10:14:10.089 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.6022999999999999e-05 2024-09-21T10:14:10.089 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.089 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.089 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:10.089 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:22.259391+0000", 2024-09-21T10:14:10.089 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2668e-05 2024-09-21T10:14:10.089 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.089 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.089 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:10.089 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:22.259397+0000", 2024-09-21T10:14:10.090 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7039999999999999e-06 2024-09-21T10:14:10.090 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.090 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.090 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.090 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:22.260834+0000", 2024-09-21T10:14:10.090 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0014373680000000001 2024-09-21T10:14:10.090 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.090 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.090 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:10.090 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:22.260858+0000", 2024-09-21T10:14:10.090 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.3541999999999999e-05 2024-09-21T10:14:10.090 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.090 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.090 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:10.090 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:22.281299+0000", 2024-09-21T10:14:10.091 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.020440934000000001 2024-09-21T10:14:10.091 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.091 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.091 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:10.091 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:22.281354+0000", 2024-09-21T10:14:10.091 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5466000000000001e-05 2024-09-21T10:14:10.091 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.091 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.091 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.091 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.091 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.091 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:166890 11.5 e277/267 11:a2a28049:::benchmark_data_smithi107_218812_object54403:head v 277'10052, mlcod=263'4227)", 2024-09-21T10:14:10.091 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:12:24.889827+0000", 2024-09-21T10:14:10.091 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 105.119555144, 2024-09-21T10:14:10.091 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.020944009999999999, 2024-09-21T10:14:10.092 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:10.092 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:10.092 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:10.092 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.092 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:10.092 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:24.889827+0000", 2024-09-21T10:14:10.092 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.092 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.092 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.092 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:10.092 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:24.889827+0000", 2024-09-21T10:14:10.092 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.092 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.092 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.092 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:10.092 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:24.889826+0000", 2024-09-21T10:14:10.092 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:14:10.093 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.093 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.093 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:10.093 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:24.889832+0000", 2024-09-21T10:14:10.093 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0140000000000004e-06 2024-09-21T10:14:10.093 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.093 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.093 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:10.093 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:24.889844+0000", 2024-09-21T10:14:10.093 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1253999999999999e-05 2024-09-21T10:14:10.093 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.093 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.093 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:10.093 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:24.889849+0000", 2024-09-21T10:14:10.093 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.254e-06 2024-09-21T10:14:10.094 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.094 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.094 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.094 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:24.895099+0000", 2024-09-21T10:14:10.094 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0052501270000000003 2024-09-21T10:14:10.094 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.094 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.094 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:10.094 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:24.895129+0000", 2024-09-21T10:14:10.095 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.9550999999999999e-05 2024-09-21T10:14:10.095 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.096 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.096 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:10.096 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:24.910748+0000", 2024-09-21T10:14:10.096 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.015618884 2024-09-21T10:14:10.096 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.096 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.096 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:10.096 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:24.910771+0000", 2024-09-21T10:14:10.096 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.3499999999999999e-05 2024-09-21T10:14:10.096 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.096 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.096 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.096 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.096 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.096 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:171102 11.5 e277/267 11:aebf1087:::benchmark_data_smithi107_218812_object58615:head v 277'10315, mlcod=264'4829)", 2024-09-21T10:14:10.097 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:12:29.786388+0000", 2024-09-21T10:14:10.097 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 100.22299419700001, 2024-09-21T10:14:10.097 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.025079976, 2024-09-21T10:14:10.097 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:10.097 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:10.097 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:10.097 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.097 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:10.097 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:29.786388+0000", 2024-09-21T10:14:10.097 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.097 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.097 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.097 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:10.097 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:29.786388+0000", 2024-09-21T10:14:10.097 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.098 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.098 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.098 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:10.098 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:29.786387+0000", 2024-09-21T10:14:10.098 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:14:10.098 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.098 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.098 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:10.098 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:29.786395+0000", 2024-09-21T10:14:10.098 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.1969999999999996e-06 2024-09-21T10:14:10.098 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.098 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.098 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:10.098 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:29.786416+0000", 2024-09-21T10:14:10.098 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1404e-05 2024-09-21T10:14:10.099 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.099 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.099 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:10.099 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:29.786423+0000", 2024-09-21T10:14:10.099 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5590000000000002e-06 2024-09-21T10:14:10.099 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.099 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.099 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.099 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:29.790715+0000", 2024-09-21T10:14:10.099 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0042925109999999997 2024-09-21T10:14:10.099 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.099 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.099 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:10.099 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:29.790740+0000", 2024-09-21T10:14:10.099 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.4454999999999999e-05 2024-09-21T10:14:10.099 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.100 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.100 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:10.100 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:29.811383+0000", 2024-09-21T10:14:10.100 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.020643043999999999 2024-09-21T10:14:10.100 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.100 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.100 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:10.100 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:12:29.811468+0000", 2024-09-21T10:14:10.100 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.5018999999999994e-05 2024-09-21T10:14:10.100 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.100 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.100 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.100 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.100 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.100 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:220980 11.8 e290/281 11:19fac2b4:::benchmark_data_smithi107_218812_object108493:head v 290'13991, mlcod=290'13988)", 2024-09-21T10:14:10.100 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:13:28.857694+0000", 2024-09-21T10:14:10.101 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 41.151688053999997, 2024-09-21T10:14:10.101 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.013265406, 2024-09-21T10:14:10.101 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:10.101 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:10.101 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:10.101 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.101 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:10.101 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:28.857694+0000", 2024-09-21T10:14:10.101 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.101 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.101 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.101 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:10.101 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:28.857694+0000", 2024-09-21T10:14:10.101 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.101 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.101 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.102 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:10.102 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:28.857693+0000", 2024-09-21T10:14:10.102 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:14:10.102 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.102 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.102 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:10.102 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:28.857701+0000", 2024-09-21T10:14:10.102 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.7100000000000007e-06 2024-09-21T10:14:10.102 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.102 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.102 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:10.102 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:28.857715+0000", 2024-09-21T10:14:10.102 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3604999999999999e-05 2024-09-21T10:14:10.102 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.102 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.103 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:10.103 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:28.857724+0000", 2024-09-21T10:14:10.103 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.9770000000000006e-06 2024-09-21T10:14:10.103 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.103 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.103 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.103 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:28.857855+0000", 2024-09-21T10:14:10.103 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013100000000000001 2024-09-21T10:14:10.103 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.103 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.103 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:10.103 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:28.857898+0000", 2024-09-21T10:14:10.103 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2991999999999998e-05 2024-09-21T10:14:10.103 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.103 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.104 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:10.104 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:28.870924+0000", 2024-09-21T10:14:10.104 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.013026421 2024-09-21T10:14:10.104 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.104 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.104 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:10.104 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:28.870960+0000", 2024-09-21T10:14:10.104 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5586000000000001e-05 2024-09-21T10:14:10.104 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.104 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.104 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.104 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.104 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.104 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:223070 11.a e291/286 11:5f5569f2:::benchmark_data_smithi107_218812_object110583:head v 291'13816, mlcod=291'13813)", 2024-09-21T10:14:10.105 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:13:31.097528+0000", 2024-09-21T10:14:10.105 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 38.911854368, 2024-09-21T10:14:10.105 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.012205449, 2024-09-21T10:14:10.105 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:10.105 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:10.105 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:10.105 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.105 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:10.105 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:31.097528+0000", 2024-09-21T10:14:10.105 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.105 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.105 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.105 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:10.105 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:31.097528+0000", 2024-09-21T10:14:10.105 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.106 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.106 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.106 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:10.106 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:31.097527+0000", 2024-09-21T10:14:10.106 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:14:10.106 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.106 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.106 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:10.106 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:31.097533+0000", 2024-09-21T10:14:10.106 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0580000000000003e-06 2024-09-21T10:14:10.106 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.106 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.106 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:10.107 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:31.097543+0000", 2024-09-21T10:14:10.107 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.7929999999999992e-06 2024-09-21T10:14:10.107 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.107 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.107 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:10.107 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:31.097548+0000", 2024-09-21T10:14:10.107 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1830000000000002e-06 2024-09-21T10:14:10.107 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.107 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.107 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.107 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:31.101623+0000", 2024-09-21T10:14:10.107 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0040748269999999996 2024-09-21T10:14:10.107 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.107 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.107 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:10.108 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:31.101644+0000", 2024-09-21T10:14:10.108 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1239000000000001e-05 2024-09-21T10:14:10.108 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.108 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.108 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:10.108 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:31.109692+0000", 2024-09-21T10:14:10.108 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0080478679999999997 2024-09-21T10:14:10.108 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.108 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.108 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:10.108 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:31.109733+0000", 2024-09-21T10:14:10.108 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1013999999999997e-05 2024-09-21T10:14:10.108 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.108 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.109 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.109 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.109 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.109 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:223291 11.5 e291/278 11:a38770f3:::benchmark_data_smithi107_218812_object110804:head v 291'13407, mlcod=291'13405)", 2024-09-21T10:14:10.109 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:13:31.322135+0000", 2024-09-21T10:14:10.109 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 38.687247638999999, 2024-09-21T10:14:10.109 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.013260160999999999, 2024-09-21T10:14:10.109 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:10.109 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:10.109 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:10.109 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.109 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:10.109 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:31.322135+0000", 2024-09-21T10:14:10.109 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.109 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.110 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.110 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:10.110 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:31.322135+0000", 2024-09-21T10:14:10.110 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:10.110 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.110 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.110 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:10.110 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:31.322133+0000", 2024-09-21T10:14:10.110 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:14:10.110 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.110 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.110 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:10.110 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:31.322143+0000", 2024-09-21T10:14:10.110 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0010000000000001e-05 2024-09-21T10:14:10.111 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.111 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.111 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:10.111 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:31.322164+0000", 2024-09-21T10:14:10.111 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.0775999999999999e-05 2024-09-21T10:14:10.111 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.111 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.111 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:10.111 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:31.322171+0000", 2024-09-21T10:14:10.111 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7190000000000004e-06 2024-09-21T10:14:10.111 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.111 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.111 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:10.111 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:31.322235+0000", 2024-09-21T10:14:10.111 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3843999999999994e-05 2024-09-21T10:14:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:31.322261+0000", 2024-09-21T10:14:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.6075999999999999e-05 2024-09-21T10:14:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:31.335212+0000", 2024-09-21T10:14:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.012951677 2024-09-21T10:14:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:13:31.335395+0000", 2024-09-21T10:14:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000182394 2024-09-21T10:14:10.112 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.113 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.113 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.113 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:10.113 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:10.113 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:14:10.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:10.114 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:10.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:10.215 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:10.248 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:10.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:10.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:10.349 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:10.380 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:10.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:10.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:10.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:14:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:14:10.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:14:10.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:10.520 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:10.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:10.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:10.562 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:14:10.562 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:14:10.562 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-21T10:14:10.562 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:14:10.562 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:14:10.562 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-09-21T10:14:10.622 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:10.658 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:10.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:10.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:10.759 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:10.780 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:10 smithi129 ceph-mon[88471]: pgmap v1059: 109 pgs: 109 active+clean; 634 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:14:10.783 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:10 smithi107 ceph-mon[105022]: pgmap v1059: 109 pgs: 109 active+clean; 634 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:14:10.783 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:10 smithi107 ceph-mon[102221]: pgmap v1059: 109 pgs: 109 active+clean; 634 KiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:14:10.798 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:10.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:10.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:10.866 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1 16 3436 3420 213.737 213.75 0.00144362 0.00456483 2024-09-21T10:14:10.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:10.940 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:10.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:10.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:11.042 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:11.075 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:11.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:11.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:11.177 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:11.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:11.207 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:11.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:11.308 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:11.328 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:11.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:11.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:11.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:11.451 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:11.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:11.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:11.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:14:11.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:11.597 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:11.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:11.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:11.698 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:11.727 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:11.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:11.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:11.779 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:11.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:11.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:11.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:14:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:14:11] "GET /metrics HTTP/1.1" 200 36718 "" "Prometheus/2.51.0" 2024-09-21T10:14:11.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:11.867 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2 16 5733 5717 178.553 143.562 0.00519607 0.00558762 2024-09-21T10:14:11.872 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:11.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:11.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:11.875 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:14:11.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:11.987 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:14:12.015 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:12.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:12.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:12.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:12.157 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:12.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:12.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:12.168 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:12.260 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:12.293 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:12.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:12.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:12.394 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:12.425 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:12.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:12.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:12.527 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:12.551 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:14:12.555 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:12.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:12.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:12.658 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:12.687 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:12.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:12.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:12.781 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:14:12.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:12.831 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:12.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:12.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:12.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:12 smithi107 ceph-mon[102221]: pgmap v1060: 109 pgs: 109 active+clean; 17 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 2.4 MiB/s wr, 38 op/s 2024-09-21T10:14:12.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:14:12.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:12 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1881551879' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:14:12.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:12 smithi107 ceph-mon[105022]: pgmap v1060: 109 pgs: 109 active+clean; 17 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 2.4 MiB/s wr, 38 op/s 2024-09-21T10:14:12.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:14:12.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:12 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1881551879' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:14:12.867 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3 16 8759 8743 182.071 189.125 0.00224014 0.00548313 2024-09-21T10:14:12.908 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:12 smithi129 ceph-mon[88471]: pgmap v1060: 109 pgs: 109 active+clean; 17 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 2.4 MiB/s wr, 38 op/s 2024-09-21T10:14:12.908 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:14:12.908 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:12 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1881551879' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:14:12.933 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:12.962 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:12.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:13.064 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:13.093 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:13.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:13.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:13.195 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:13.224 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:13.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:13.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:13.326 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:13.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:13.355 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:13.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:13.456 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:13.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:13.486 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:13.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:13.587 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:13.625 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:13.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:13.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:13.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:13.767 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:13 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:14:13.767 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:13 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:14:13.767 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:13 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1881551879' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:14:13.767 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:13 smithi129 ceph-mon[88471]: osdmap e301: 8 total, 8 up, 7 in 2024-09-21T10:14:13.775 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:13.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:13.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:13.848 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:13 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:14:13.848 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:13 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:14:13.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:13 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1881551879' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:14:13.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:13 smithi107 ceph-mon[102221]: osdmap e301: 8 total, 8 up, 7 in 2024-09-21T10:14:13.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:13 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:14:13.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:13 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:14:13.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:13 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1881551879' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:14:13.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:13 smithi107 ceph-mon[105022]: osdmap e301: 8 total, 8 up, 7 in 2024-09-21T10:14:13.867 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 4 16 11818 11802 184.345 191.188 0.0107621 0.00541767 2024-09-21T10:14:13.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:13.914 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:13.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:13.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:14.016 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:14.048 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:14.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:14.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:14.169 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:14:14.170 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:14:14.170 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:14:14.186 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:14.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:14.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:14.288 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:14.321 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:14.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:14.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:14.423 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:14.445 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:14.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:14.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:14.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:14.581 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:14.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:14.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:14.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:14 smithi107 ceph-mon[102221]: pgmap v1062: 109 pgs: 109 active+clean; 355 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 45 MiB/s wr, 720 op/s 2024-09-21T10:14:14.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:14 smithi107 ceph-mon[105022]: pgmap v1062: 109 pgs: 109 active+clean; 355 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 45 MiB/s wr, 720 op/s 2024-09-21T10:14:14.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:14.719 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:14.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:14.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:14.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:14.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:14.866 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:14.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:14.867 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 5 16 14021 14005 175.01 137.688 0.00388295 0.00570662 2024-09-21T10:14:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:14 smithi129 ceph-mon[88471]: pgmap v1062: 109 pgs: 109 active+clean; 355 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 45 MiB/s wr, 720 op/s 2024-09-21T10:14:14.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:15.005 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:15.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:15.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:15.107 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:15.139 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:15.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:15.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:15.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:15.279 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:15.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:15.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:15.381 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:15.409 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:15.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:15.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:15.511 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:15.525 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:14:15.526 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:14:15.540 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:15.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:15.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:15.565 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:14:15.565 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:14:15.601 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:14:15.601 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:14:15.602 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:15 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/701411515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:15.602 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:15 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/701411515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:15.640 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:14:15.640 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:14:15.642 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:15.672 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:14:15.672 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:14:15.672 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:15.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:15.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:15.702 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:14:15.702 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:14:15.733 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:14:15.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:14:15.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:14:15.762 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:14:15.763 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:14:15.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:15.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:15.819 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:15.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:15.867 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 6 16 16605 16589 172.757 161.5 0.00777845 0.00578194 2024-09-21T10:14:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:15 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/701411515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:15.920 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:15.950 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:15.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:15.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:16.053 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:16.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:16.095 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:16.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:16.196 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:16.227 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:16.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:16.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:16.328 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:16.359 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:16.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:16.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:16.461 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:16.492 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:16.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:16.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:16.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:16.640 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:16.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:16.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:16.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:16 smithi129 ceph-mon[88471]: pgmap v1063: 109 pgs: 109 active+clean; 458 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 57 MiB/s wr, 914 op/s 2024-09-21T10:14:16.670 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:16.743 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:16.770 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:16.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:16.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:16 smithi107 ceph-mon[102221]: pgmap v1063: 109 pgs: 109 active+clean; 458 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 57 MiB/s wr, 914 op/s 2024-09-21T10:14:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:16 smithi107 ceph-mon[105022]: pgmap v1063: 109 pgs: 109 active+clean; 458 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 57 MiB/s wr, 914 op/s 2024-09-21T10:14:16.867 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 7 16 19131 19115 170.629 157.875 0.00815962 0.00585446 2024-09-21T10:14:16.872 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:16.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:16.904 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:16.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:16.988 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:14:16.988 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:14:17.005 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:17.036 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:17.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:17.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:17.137 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:17.166 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:17.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:17.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:17.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:17.305 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:17.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:17.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:17.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:17.444 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:17.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:17.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:17.546 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:17.569 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:17.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:17.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:17.670 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:17.707 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:17.709 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:14:17.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:17.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:17.810 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:17.843 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:17.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:17.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:17.868 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 8 16 21381 21365 166.877 140.625 0.00427764 0.00598632 2024-09-21T10:14:17.944 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:17.979 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:17.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:17.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:18.060 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:14:18.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:18.118 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:18.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:18.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:18.220 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:18.249 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:18.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:18.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:18.351 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:18.387 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:18.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:18.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:18.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:18.528 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:18.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:18.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:18.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:14:18.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:18.708 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:18.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:18.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:18.810 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:18.831 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:18.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:18.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:18.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:18 smithi107 ceph-mon[102221]: pgmap v1064: 109 pgs: 109 active+clean; 928 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T10:14:18.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:18 smithi107 ceph-mon[105022]: pgmap v1064: 109 pgs: 109 active+clean; 928 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T10:14:18.868 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 9 16 23240 23224 161.244 116.188 0.00170728 0.00612491 2024-09-21T10:14:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:18 smithi129 ceph-mon[88471]: pgmap v1064: 109 pgs: 109 active+clean; 928 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T10:14:18.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:18.954 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:18.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:18.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:19.056 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:19.084 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:19.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:19.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:19.186 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:19.217 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:19.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:19.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:19.319 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:19.345 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:19.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:19.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:19.447 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:19.470 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:19.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:19.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:19.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:19.609 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:19.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:19.710 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:19.735 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:19.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:19.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:19.837 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:19.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:19.867 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:19.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:19.868 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 10 16 25147 25131 157.037 119.188 0.00159482 0.0063518 2024-09-21T10:14:19.969 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:20.004 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:20.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:20.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:20.106 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:20.134 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:20.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:20.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:20.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:20.257 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:20.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:20.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:20.352 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:20.359 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:20.385 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:20.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:20.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:20.487 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:20.515 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:20.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:20.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:20.604 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:14:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:14:20.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:14:20.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:20.652 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:20.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:20.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:20.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:20.790 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:20.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:20.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:20.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:20 smithi107 ceph-mon[102221]: pgmap v1065: 109 pgs: 109 active+clean; 1.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-09-21T10:14:20.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:20 smithi107 ceph-mon[105022]: pgmap v1065: 109 pgs: 109 active+clean; 1.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-09-21T10:14:20.868 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 11 16 27169 27153 154.249 126.375 0.00397238 0.00647737 2024-09-21T10:14:20.892 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:20 smithi129 ceph-mon[88471]: pgmap v1065: 109 pgs: 109 active+clean; 1.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-09-21T10:14:20.920 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:20.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:20.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:20.964 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:14:20.964 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:14:20.964 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-21T10:14:20.964 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd in 4 2024-09-21T10:14:21.022 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:21.045 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:21.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:21.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:21.147 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:21.172 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:21.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:21.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:21.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:21.356 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:21.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:21.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:21.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:21.495 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:21.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:21.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:21.597 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:21.624 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:14:21.627 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:21.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:21.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:21.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:21 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1220508902' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:14:21.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:21 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:14:21.725 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:21 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1220508902' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:14:21.725 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:21 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:14:21.725 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:21 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1220508902' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:14:21.725 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:21 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:14:21.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:21.768 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:21.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:21.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:21.868 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 12 16 29009 28993 150.978 115 0.00590741 0.0066179 2024-09-21T10:14:21.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:21.906 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:21.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:21.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:22.007 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:22.010 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:14:22.010 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:14:22.030 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:22.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:22.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:22.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:14:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:14:21] "GET /metrics HTTP/1.1" 200 36718 "" "Prometheus/2.51.0" 2024-09-21T10:14:22.131 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:22.159 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:22.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:22.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:22.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:22.300 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:22.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:22.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:22.402 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:22.427 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:22.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:22.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:22.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:22.567 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:22.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:22.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:22.661 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:14:22.668 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:22.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:22.736 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:22.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:22.836 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:22.868 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 13 16 30959 30943 148.733 121.875 0.00808655 0.00671662 2024-09-21T10:14:22.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:22.881 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:22.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:22 smithi129 ceph-mon[88471]: pgmap v1066: 109 pgs: 109 active+clean; 1.3 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 132 MiB/s wr, 2.12k op/s 2024-09-21T10:14:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:22 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:14:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:22 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:14:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:22 smithi129 ceph-mon[88471]: osdmap e302: 8 total, 8 up, 7 in 2024-09-21T10:14:22.982 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:23.019 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:23.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:23.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:23.023 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:14:23.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:22 smithi107 ceph-mon[105022]: pgmap v1066: 109 pgs: 109 active+clean; 1.3 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 132 MiB/s wr, 2.12k op/s 2024-09-21T10:14:23.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:22 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:14:23.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:22 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:14:23.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:22 smithi107 ceph-mon[105022]: osdmap e302: 8 total, 8 up, 7 in 2024-09-21T10:14:23.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:22 smithi107 ceph-mon[102221]: pgmap v1066: 109 pgs: 109 active+clean; 1.3 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 132 MiB/s wr, 2.12k op/s 2024-09-21T10:14:23.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:22 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:14:23.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:22 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:14:23.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:22 smithi107 ceph-mon[102221]: osdmap e302: 8 total, 8 up, 7 in 2024-09-21T10:14:23.122 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:23.151 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:23.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:23.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:23.253 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:23.285 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:23.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:23.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:23.387 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:23.412 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:23.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:23.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:23.514 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:23.544 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:23.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:23.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:23.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:23.683 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:23.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:23.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:23.785 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:23.797 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:23.836 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:23.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:23.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:23.869 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 14 16 32924 32908 146.88 122.812 0.0098022 0.00680237 2024-09-21T10:14:23.941 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:23.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:23.987 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:23.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:24.088 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:24.116 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:24.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:24.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:24.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:24.255 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:24.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:24.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:24.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:24.390 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:24.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:24.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:24.492 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:24.519 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:24.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:24.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:24.621 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:24.649 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:24.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:24.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:24.733 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:24 smithi107 ceph-mon[102221]: pgmap v1068: 109 pgs: 109 active+clean; 1.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-09-21T10:14:24.733 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:24 smithi107 ceph-mon[105022]: pgmap v1068: 109 pgs: 109 active+clean; 1.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-09-21T10:14:24.751 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:24.779 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:24.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:24.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:24.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:14:24.839 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:14:24.839 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:14:24.839 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:24 smithi129 ceph-mon[88471]: pgmap v1068: 109 pgs: 109 active+clean; 1.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 144 MiB/s wr, 2.30k op/s 2024-09-21T10:14:24.869 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 15 16 34513 34497 143.709 99.3125 0.00672582 0.00695152 2024-09-21T10:14:24.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:24.924 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:24.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:24.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:25.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:25.048 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:25.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:25.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:25.150 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:25.177 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:25.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:25.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:25.279 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:25.318 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:25.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:25.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:25.421 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:25.450 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:25.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:25.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:25.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:25.590 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:25.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:25.692 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:25.718 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:25.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:25.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:25.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:25.855 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:25.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:25.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:25.870 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 16 16 36027 36011 140.634 94.625 0.00873004 0.00710343 2024-09-21T10:14:25.912 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:14:25.912 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:14:25.955 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:14:25.955 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:14:25.957 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:25.984 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:25.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:25.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:25.988 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:14:25.988 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:14:26.020 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:14:26.020 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:14:26.052 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:14:26.053 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:14:26.070 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:14:26.084 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:14:26.084 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:14:26.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:26.114 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:14:26.114 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:14:26.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:26.131 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:26.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:26.149 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:14:26.149 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:14:26.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:26.253 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:26.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:26.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:26.355 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:26.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:26.391 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:26.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:26.402 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:14:26.492 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:26.521 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:26.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:26.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:26.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:26.665 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:26.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:26.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:26.767 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:26.795 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:26.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:26.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:26.870 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 17 16 37791 37775 138.847 110.25 0.00704077 0.00719704 2024-09-21T10:14:26.874 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:26 smithi129 ceph-mon[88471]: pgmap v1069: 109 pgs: 109 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-09-21T10:14:26.874 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:26 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/6730909' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:26.874 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:26.897 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:26.928 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:26.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:26.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:27.030 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:27.063 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:26 smithi107 ceph-mon[102221]: pgmap v1069: 109 pgs: 109 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-09-21T10:14:27.063 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:26 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/6730909' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:27.063 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:27.066 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:26 smithi107 ceph-mon[105022]: pgmap v1069: 109 pgs: 109 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-09-21T10:14:27.067 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:26 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/6730909' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:27.067 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:27.068 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:27.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:27.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:27.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:27.214 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:27.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:27.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:27.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:27.356 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:27.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:27.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:27.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:27.502 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:27.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:27.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:27.604 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:27.648 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:27.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:27.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:27.749 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:27.790 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:27.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:27.870 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 18 16 39586 39570 137.365 112.188 0.0115115 0.00727471 2024-09-21T10:14:27.892 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:27.930 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:27.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:27.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:28.032 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:28.062 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:28.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:28.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:28.164 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:28.193 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:28.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:28.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:28.292 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:14:28.292 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:14:28.292 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:28.293 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36822.0:40327 12.6 e302/298 12:67c2534a:::benchmark_data_smithi107_250110_object40326:head v 302'2543, mlcod=302'2541)", 2024-09-21T10:14:28.293 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:14:28.287640+0000", 2024-09-21T10:14:28.293 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.00129352, 2024-09-21T10:14:28.293 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0013192670000000001, 2024-09-21T10:14:28.293 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:28.293 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "started", 2024-09-21T10:14:28.293 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:28.293 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:28.293 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:28.293 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:28.287640+0000", 2024-09-21T10:14:28.293 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:28.293 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:28.293 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:28.293 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:28.293 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:28.287640+0000", 2024-09-21T10:14:28.293 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:28.294 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:28.294 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:28.294 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:28.294 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:28.287639+0000", 2024-09-21T10:14:28.294 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:14:28.294 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:28.294 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:28.294 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:28.294 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:28.287702+0000", 2024-09-21T10:14:28.294 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3120000000000006e-05 2024-09-21T10:14:28.294 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:28.294 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:28.294 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:28.294 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:28.287711+0000", 2024-09-21T10:14:28.294 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.8289999999999997e-06 2024-09-21T10:14:28.295 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:28.295 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:28.295 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:28.295 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:28.287715+0000", 2024-09-21T10:14:28.295 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.9759999999999997e-06 2024-09-21T10:14:28.295 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:28.295 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:28.295 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:28.295 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:28.287750+0000", 2024-09-21T10:14:28.295 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4848999999999998e-05 2024-09-21T10:14:28.295 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:28.295 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:28.295 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:28.295 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:28.287768+0000", 2024-09-21T10:14:28.296 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.8383999999999999e-05 2024-09-21T10:14:28.296 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:28.296 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:28.296 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:28.296 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:28.296 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-21T10:14:28.296 INFO:teuthology.orchestra.run.smithi129.stdout: "num_ops": 1 2024-09-21T10:14:28.296 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:14:28.296 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:28.330 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:28.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:28.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:28.431 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:28.457 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:28.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:28.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:28.499 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:28.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:28.601 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:28.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:28.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:28.703 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:28.730 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:28.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:28.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:28.831 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:28.857 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:28.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:28.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:28.871 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 19 16 41234 41218 135.545 103 0.012031 0.00737148 2024-09-21T10:14:28.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:28 smithi129 ceph-mon[88471]: pgmap v1070: 109 pgs: 109 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-21T10:14:28.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:28.969 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:14:28.994 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:28.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:28.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:29.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:29.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:28 smithi107 ceph-mon[105022]: pgmap v1070: 109 pgs: 109 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-21T10:14:29.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:28 smithi107 ceph-mon[102221]: pgmap v1070: 109 pgs: 109 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 126 MiB/s wr, 2.01k op/s 2024-09-21T10:14:29.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:28 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a[102198]: 2024-09-21T10:14:28.747+0000 7f3189124640 -1 mon.a@0(leader).osd e302 definitely_dead 0 2024-09-21T10:14:29.133 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:29.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:29.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:29.234 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:29.268 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:29.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:29.369 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:29.373 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:14:29.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:29.405 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:29.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:29.505 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:29.545 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:29.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:29.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:29.645 INFO:teuthology.orchestra.run.smithi107.stderr:marked in osd.4. 2024-09-21T10:14:29.647 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:29.681 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:29.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:29.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:29.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:29.803 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:29.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:29.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:29.810 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:29 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1329097685' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:14:29.872 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:14:29.870667+0000 min lat: 0.00109013 max lat: 0.228858 avg lat: 0.00745354 2024-09-21T10:14:29.872 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:14:29.872 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 20 16 42643 42627 133.171 88.0625 0.00207295 0.00745354 2024-09-21T10:14:29.905 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:29.926 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:29.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:30.028 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:30.049 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:30.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:30.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:30.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:29 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1329097685' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:14:30.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:29 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1329097685' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:14:30.140 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:14:30.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:30.204 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:30.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:30.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:30.306 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:30.328 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:30.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:30.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:30.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:30.450 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:30.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:30.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:30.532 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:14:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:14:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:14:30.552 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:30.573 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:30.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:30.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:30.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:30.699 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:30.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:30.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:30.801 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:30.823 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:30.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:30.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:30.855 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:30 smithi129 ceph-mon[88471]: pgmap v1071: 109 pgs: 109 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-09-21T10:14:30.855 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:30 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1329097685' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T10:14:30.855 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:30 smithi129 ceph-mon[88471]: osdmap e303: 8 total, 8 up, 8 in 2024-09-21T10:14:30.872 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 21 16 42643 42627 126.831 0 - 0.00745354 2024-09-21T10:14:30.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:30.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:30.946 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:30.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:31.047 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:31.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:31.071 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:31.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:31.075 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:14:31.075 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:14:31.075 INFO:teuthology.orchestra.run.smithi129.stdout: "complaint_time": 30, 2024-09-21T10:14:31.075 INFO:teuthology.orchestra.run.smithi129.stdout: "num_blocked_ops": 0 2024-09-21T10:14:31.075 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:14:31.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:30 smithi107 ceph-mon[102221]: pgmap v1071: 109 pgs: 109 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-09-21T10:14:31.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:30 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1329097685' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T10:14:31.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:30 smithi107 ceph-mon[102221]: osdmap e303: 8 total, 8 up, 8 in 2024-09-21T10:14:31.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:30 smithi107 ceph-mon[105022]: pgmap v1071: 109 pgs: 109 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 118 MiB/s wr, 1.88k op/s 2024-09-21T10:14:31.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:30 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1329097685' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T10:14:31.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:30 smithi107 ceph-mon[105022]: osdmap e303: 8 total, 8 up, 8 in 2024-09-21T10:14:31.172 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:31.193 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:31.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:31.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:31.295 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:31.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:31.316 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:31.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:31.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:31.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:31.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:31.543 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:31.567 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:31.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:31.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:31.656 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:14:31.668 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:31.721 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:31.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:31.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:31.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:31.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:31.858 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:31.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:31.872 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 22 16 42716 42700 121.274 2.28125 0.0139447 0.00823994 2024-09-21T10:14:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:31 smithi129 ceph-mon[88471]: osdmap e304: 8 total, 8 up, 8 in 2024-09-21T10:14:31.959 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:31.973 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:14:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:31.996 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:31.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:32.097 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:32.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:31 smithi107 ceph-mon[102221]: osdmap e304: 8 total, 8 up, 8 in 2024-09-21T10:14:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:14:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:14:31] "GET /metrics HTTP/1.1" 200 36783 "" "Prometheus/2.51.0" 2024-09-21T10:14:32.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:31 smithi107 ceph-mon[105022]: osdmap e304: 8 total, 8 up, 8 in 2024-09-21T10:14:32.129 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:32.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:32.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:32.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:32.271 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:32.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:32.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:32.373 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:32.403 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:32.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:32.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:32.505 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:32.531 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:32.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:32.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:32.632 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:32.662 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:32.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:32.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:32.763 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:32.787 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:32.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:32.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:32.872 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 23 16 44164 44148 119.936 90.5 0.00608336 0.00833212 2024-09-21T10:14:32.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:32.929 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:32.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:32.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:33.031 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:33.052 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:33.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:33.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:33.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:32 smithi107 ceph-mon[102221]: pgmap v1074: 109 pgs: 3 peering, 106 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-09-21T10:14:33.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:32 smithi107 ceph-mon[102221]: osdmap e305: 8 total, 8 up, 8 in 2024-09-21T10:14:33.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:32 smithi107 ceph-mon[105022]: pgmap v1074: 109 pgs: 3 peering, 106 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-09-21T10:14:33.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:32 smithi107 ceph-mon[105022]: osdmap e305: 8 total, 8 up, 8 in 2024-09-21T10:14:33.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:33.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:32 smithi129 ceph-mon[88471]: pgmap v1074: 109 pgs: 3 peering, 106 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-09-21T10:14:33.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:32 smithi129 ceph-mon[88471]: osdmap e305: 8 total, 8 up, 8 in 2024-09-21T10:14:33.188 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:33.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:33.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:33.290 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:33.327 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:33.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:33.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:33.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:33.464 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:33.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:33.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:33.566 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:33.601 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:33.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:33.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:33.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:33.737 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:33.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:33.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:33.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:33 smithi129 ceph-mon[88471]: Health check failed: Degraded data redundancy: 2681/129531 objects degraded (2.070%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:33.838 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:33.859 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:14:33.859 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 20, 2024-09-21T10:14:33.859 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 600, 2024-09-21T10:14:33.859 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:14:33.859 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.860 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95081 10.c 10:3ce718bd:::benchmark_data_smithi107_184135_object95080:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T10:14:33.860 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.369602+0000", 2024-09-21T10:14:33.860 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 507.48347202299999, 2024-09-21T10:14:33.860 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0566048509999999, 2024-09-21T10:14:33.860 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:33.860 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:14:33.860 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:33.860 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:14:33.860 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:14:33.860 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95081 2024-09-21T10:14:33.860 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.860 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:33.860 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.861 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:33.861 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.369602+0000", 2024-09-21T10:14:33.861 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.861 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.861 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.861 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:33.861 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.369602+0000", 2024-09-21T10:14:33.861 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.861 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.861 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.861 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:33.861 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.369601+0000", 2024-09-21T10:14:33.861 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:14:33.861 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.862 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.862 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:33.862 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.369721+0000", 2024-09-21T10:14:33.862 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012002000000000001 2024-09-21T10:14:33.862 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.862 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.862 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:33.862 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.369736+0000", 2024-09-21T10:14:33.862 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5016e-05 2024-09-21T10:14:33.862 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.862 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.862 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:33.863 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.369742+0000", 2024-09-21T10:14:33.863 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7150000000000003e-06 2024-09-21T10:14:33.863 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.863 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.863 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.863 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.369790+0000", 2024-09-21T10:14:33.863 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8238000000000001e-05 2024-09-21T10:14:33.863 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.863 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.863 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:33.863 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.369927+0000", 2024-09-21T10:14:33.863 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013691699999999999 2024-09-21T10:14:33.863 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.864 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.864 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:14:33.864 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.370082+0000", 2024-09-21T10:14:33.864 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000155616 2024-09-21T10:14:33.864 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.864 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.864 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:33.864 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.373661+0000", 2024-09-21T10:14:33.864 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0035782660000000001 2024-09-21T10:14:33.864 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.864 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.864 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.864 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.377955+0000", 2024-09-21T10:14:33.864 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0042938689999999996 2024-09-21T10:14:33.865 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.865 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.865 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.865 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.398356+0000", 2024-09-21T10:14:33.865 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.020401263999999999 2024-09-21T10:14:33.865 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.865 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.865 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.865 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.398364+0000", 2024-09-21T10:14:33.865 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9160000000000001e-06 2024-09-21T10:14:33.865 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.865 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.865 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.865 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.426137+0000", 2024-09-21T10:14:33.866 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0277736980000001 2024-09-21T10:14:33.866 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.866 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.866 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:33.866 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.426207+0000", 2024-09-21T10:14:33.866 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9596000000000002e-05 2024-09-21T10:14:33.866 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.866 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.866 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.866 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.866 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.866 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:95085 10.c 10:33c119cf:::benchmark_data_smithi107_184135_object95084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-21T10:14:33.866 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:06:06.374530+0000", 2024-09-21T10:14:33.867 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 507.47854467899998, 2024-09-21T10:14:33.867 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.051753323, 2024-09-21T10:14:33.867 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:33.867 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:14:33.867 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:33.867 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:14:33.867 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:14:33.867 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 95085 2024-09-21T10:14:33.867 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.867 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:33.867 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.867 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:33.867 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.374530+0000", 2024-09-21T10:14:33.867 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.867 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.868 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.868 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:33.868 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.374530+0000", 2024-09-21T10:14:33.868 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.868 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.868 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.868 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:33.868 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.374528+0000", 2024-09-21T10:14:33.868 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:14:33.868 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.868 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.868 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:33.868 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.374594+0000", 2024-09-21T10:14:33.868 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6126999999999999e-05 2024-09-21T10:14:33.868 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.868 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.869 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:33.869 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.374607+0000", 2024-09-21T10:14:33.869 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3688e-05 2024-09-21T10:14:33.869 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.869 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.869 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:33.869 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.374613+0000", 2024-09-21T10:14:33.869 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.242e-06 2024-09-21T10:14:33.869 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.869 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.869 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.869 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.374662+0000", 2024-09-21T10:14:33.869 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9156000000000002e-05 2024-09-21T10:14:33.869 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.869 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.870 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:33.870 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.374797+0000", 2024-09-21T10:14:33.870 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000135329 2024-09-21T10:14:33.870 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.870 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.870 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:14:33.870 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.374898+0000", 2024-09-21T10:14:33.870 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000101185 2024-09-21T10:14:33.870 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.870 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.870 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:33.870 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.377741+0000", 2024-09-21T10:14:33.870 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0028426800000000002 2024-09-21T10:14:33.871 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.871 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.871 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.871 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.378262+0000", 2024-09-21T10:14:33.871 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00052137799999999997 2024-09-21T10:14:33.871 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.871 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.871 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.871 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.398390+0000", 2024-09-21T10:14:33.871 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.020127546 2024-09-21T10:14:33.871 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.871 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.871 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.872 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:06.398395+0000", 2024-09-21T10:14:33.872 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.741e-06 2024-09-21T10:14:33.872 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.872 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.872 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.872 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.426235+0000", 2024-09-21T10:14:33.872 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.027840436 2024-09-21T10:14:33.872 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.872 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.872 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:33.872 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:06:07.426283+0000", 2024-09-21T10:14:33.872 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7763000000000003e-05 2024-09-21T10:14:33.872 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.872 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.872 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.873 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.873 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.873 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247135 10.1 10:8bb6709e:::benchmark_data_smithi107_184135_object115569:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:14:33.873 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:49.452878+0000", 2024-09-21T10:14:33.873 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 344.40019601900002, 2024-09-21T10:14:33.873 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.958185693, 2024-09-21T10:14:33.873 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:33.873 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:33.873 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:33.873 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:14:33.873 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:14:33.873 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247135 2024-09-21T10:14:33.873 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.873 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:33.873 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.873 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:33.874 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.452878+0000", 2024-09-21T10:14:33.874 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.874 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.874 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.874 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:33.874 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.452878+0000", 2024-09-21T10:14:33.874 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.874 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.874 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.874 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:33.874 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.452876+0000", 2024-09-21T10:14:33.874 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:14:33.874 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.875 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.875 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:33.875 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.452884+0000", 2024-09-21T10:14:33.875 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.5839999999999997e-06 2024-09-21T10:14:33.875 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.875 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.875 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:33.875 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.452897+0000", 2024-09-21T10:14:33.875 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3644e-05 2024-09-21T10:14:33.875 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.875 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.875 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:33.875 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.452904+0000", 2024-09-21T10:14:33.875 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8070000000000002e-06 2024-09-21T10:14:33.876 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.876 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.876 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.876 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.452950+0000", 2024-09-21T10:14:33.876 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5348e-05 2024-09-21T10:14:33.876 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.876 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.876 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.876 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.452959+0000", 2024-09-21T10:14:33.876 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.0620000000000007e-06 2024-09-21T10:14:33.876 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.876 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.876 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.876 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.396167+0000", 2024-09-21T10:14:33.876 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.94320812600000004 2024-09-21T10:14:33.877 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.877 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.877 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:33.877 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.396257+0000", 2024-09-21T10:14:33.877 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.0739e-05 2024-09-21T10:14:33.877 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.877 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.877 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:14:33.877 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.396336+0000", 2024-09-21T10:14:33.877 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.8697999999999998e-05 2024-09-21T10:14:33.877 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.877 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.877 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:33.877 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.405056+0000", 2024-09-21T10:14:33.877 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0087199270000000006 2024-09-21T10:14:33.878 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.878 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.878 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.878 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.410930+0000", 2024-09-21T10:14:33.878 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0058740019999999997 2024-09-21T10:14:33.878 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.878 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.878 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.878 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.410985+0000", 2024-09-21T10:14:33.878 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4898000000000002e-05 2024-09-21T10:14:33.878 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.878 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.878 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:33.878 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411015+0000", 2024-09-21T10:14:33.879 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.9828999999999999e-05 2024-09-21T10:14:33.879 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.879 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.879 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:33.879 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411064+0000", 2024-09-21T10:14:33.879 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9058999999999999e-05 2024-09-21T10:14:33.879 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.879 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.879 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.879 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.879 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.879 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247136 10.1 10:864af190:::benchmark_data_smithi107_184135_object115570:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:14:33.879 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:49.454413+0000", 2024-09-21T10:14:33.879 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 344.39866111200001, 2024-09-21T10:14:33.880 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.95676656000000004, 2024-09-21T10:14:33.880 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:33.880 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:33.880 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:33.880 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:14:33.880 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:14:33.880 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247136 2024-09-21T10:14:33.880 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.880 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:33.880 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.880 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:33.880 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.454413+0000", 2024-09-21T10:14:33.880 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.880 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.880 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.881 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:33.881 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.454413+0000", 2024-09-21T10:14:33.881 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.881 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.881 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.881 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:33.881 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.454412+0000", 2024-09-21T10:14:33.881 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:14:33.881 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.881 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.881 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:33.881 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.454421+0000", 2024-09-21T10:14:33.881 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6030000000000005e-06 2024-09-21T10:14:33.881 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.881 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.882 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:33.882 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.454436+0000", 2024-09-21T10:14:33.882 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5727999999999999e-05 2024-09-21T10:14:33.882 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.882 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.882 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:33.882 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.454442+0000", 2024-09-21T10:14:33.882 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4809999999999997e-06 2024-09-21T10:14:33.882 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.882 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.882 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.882 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.455319+0000", 2024-09-21T10:14:33.882 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00087677499999999999 2024-09-21T10:14:33.882 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.883 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.883 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.883 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.455325+0000", 2024-09-21T10:14:33.883 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6989999999999997e-06 2024-09-21T10:14:33.883 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.883 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.883 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.883 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.398618+0000", 2024-09-21T10:14:33.883 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.94329290099999996 2024-09-21T10:14:33.883 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.883 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.883 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:33.883 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.398701+0000", 2024-09-21T10:14:33.884 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.2421000000000003e-05 2024-09-21T10:14:33.884 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.884 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.884 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:14:33.884 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.398770+0000", 2024-09-21T10:14:33.884 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9162e-05 2024-09-21T10:14:33.884 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.884 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.884 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:33.884 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.408712+0000", 2024-09-21T10:14:33.884 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0099424209999999999 2024-09-21T10:14:33.884 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.884 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.884 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.885 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.410967+0000", 2024-09-21T10:14:33.885 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0022543630000000001 2024-09-21T10:14:33.885 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.885 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.885 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.885 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411099+0000", 2024-09-21T10:14:33.885 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013281999999999999 2024-09-21T10:14:33.885 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.885 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.885 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:33.885 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411130+0000", 2024-09-21T10:14:33.885 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0416000000000002e-05 2024-09-21T10:14:33.885 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.885 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.885 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:33.885 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411180+0000", 2024-09-21T10:14:33.886 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9802999999999997e-05 2024-09-21T10:14:33.886 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.886 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.886 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.886 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.886 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.887 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:33 smithi107 ceph-mon[102221]: Health check failed: Degraded data redundancy: 2681/129531 objects degraded (2.070%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:33.887 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247143 10.1 10:86c515c8:::benchmark_data_smithi107_184135_object115577:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:14:33.887 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:49.459691+0000", 2024-09-21T10:14:33.887 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 344.39338321000002, 2024-09-21T10:14:33.887 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.95170592899999995, 2024-09-21T10:14:33.887 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:33.887 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:33.887 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:33.887 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:14:33.887 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:14:33.888 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247143 2024-09-21T10:14:33.888 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.888 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:33.888 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.888 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:33.888 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.459691+0000", 2024-09-21T10:14:33.888 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.888 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.888 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.888 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:33.888 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.459691+0000", 2024-09-21T10:14:33.888 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.888 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.889 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.889 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:33.889 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.459689+0000", 2024-09-21T10:14:33.889 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:14:33.889 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.889 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.889 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:33.889 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.459697+0000", 2024-09-21T10:14:33.889 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2899999999999997e-06 2024-09-21T10:14:33.889 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.889 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.889 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:33.889 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.459709+0000", 2024-09-21T10:14:33.889 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2218000000000001e-05 2024-09-21T10:14:33.890 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.890 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.890 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:33.890 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.459715+0000", 2024-09-21T10:14:33.890 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5140000000000001e-06 2024-09-21T10:14:33.890 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.890 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.890 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.890 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.462126+0000", 2024-09-21T10:14:33.890 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0024117359999999998 2024-09-21T10:14:33.890 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.890 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.890 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.890 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.462135+0000", 2024-09-21T10:14:33.891 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.132e-06 2024-09-21T10:14:33.891 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.891 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.891 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.891 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.400841+0000", 2024-09-21T10:14:33.891 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.93870596299999998 2024-09-21T10:14:33.891 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.891 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.891 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:33.891 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.400922+0000", 2024-09-21T10:14:33.891 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.0877000000000005e-05 2024-09-21T10:14:33.891 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.892 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.892 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:14:33.892 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.400994+0000", 2024-09-21T10:14:33.892 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.1768999999999997e-05 2024-09-21T10:14:33.892 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.892 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.892 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:33.892 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.408719+0000", 2024-09-21T10:14:33.892 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0077246820000000001 2024-09-21T10:14:33.892 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.892 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.892 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.892 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411217+0000", 2024-09-21T10:14:33.892 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0024978880000000002 2024-09-21T10:14:33.893 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.893 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.893 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.893 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411282+0000", 2024-09-21T10:14:33.893 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4969999999999996e-05 2024-09-21T10:14:33.893 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.893 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.893 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:33.893 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411310+0000", 2024-09-21T10:14:33.893 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.8738000000000002e-05 2024-09-21T10:14:33.893 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.893 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.893 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:33.893 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411397+0000", 2024-09-21T10:14:33.894 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6639999999999997e-05 2024-09-21T10:14:33.894 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.894 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.894 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.894 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.894 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.894 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247190 10.1 10:8be1d16c:::benchmark_data_smithi107_184135_object115624:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:14:33.894 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:49.511817+0000", 2024-09-21T10:14:33.894 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 344.341257063, 2024-09-21T10:14:33.894 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.89981461900000004, 2024-09-21T10:14:33.894 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:33.894 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:33.894 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:33.894 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:14:33.895 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:14:33.895 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247190 2024-09-21T10:14:33.895 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.895 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:33.895 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.895 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:33.895 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.511817+0000", 2024-09-21T10:14:33.895 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.895 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.895 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.895 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:33.895 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.511817+0000", 2024-09-21T10:14:33.895 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.895 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.896 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.896 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:33.896 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.511816+0000", 2024-09-21T10:14:33.896 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:14:33.896 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.896 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.896 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:33.896 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.511822+0000", 2024-09-21T10:14:33.896 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.054e-06 2024-09-21T10:14:33.896 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.896 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.896 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:33.896 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.511836+0000", 2024-09-21T10:14:33.896 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4123e-05 2024-09-21T10:14:33.897 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.897 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.897 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:33.897 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.511842+0000", 2024-09-21T10:14:33.897 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4439999999999996e-06 2024-09-21T10:14:33.897 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.897 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.897 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.897 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.511907+0000", 2024-09-21T10:14:33.897 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4968000000000005e-05 2024-09-21T10:14:33.897 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.897 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.897 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.897 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.511915+0000", 2024-09-21T10:14:33.897 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9189999999999999e-06 2024-09-21T10:14:33.898 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.898 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.898 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.898 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.402760+0000", 2024-09-21T10:14:33.898 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.89084475100000005 2024-09-21T10:14:33.898 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.898 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.898 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:33.898 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.402877+0000", 2024-09-21T10:14:33.898 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000116815 2024-09-21T10:14:33.898 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.898 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.898 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:14:33.898 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.403008+0000", 2024-09-21T10:14:33.899 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013146100000000001 2024-09-21T10:14:33.899 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.899 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.899 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:33.899 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.410523+0000", 2024-09-21T10:14:33.899 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0075145780000000001 2024-09-21T10:14:33.899 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.899 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.899 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.899 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411257+0000", 2024-09-21T10:14:33.899 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00073459800000000004 2024-09-21T10:14:33.899 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.899 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.899 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.899 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411548+0000", 2024-09-21T10:14:33.900 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00029048799999999999 2024-09-21T10:14:33.900 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.900 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.900 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:33.900 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411576+0000", 2024-09-21T10:14:33.900 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7807999999999999e-05 2024-09-21T10:14:33.900 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.900 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.900 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:33.900 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411632+0000", 2024-09-21T10:14:33.900 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6016e-05 2024-09-21T10:14:33.900 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.900 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.900 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.901 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.901 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.901 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:33 smithi107 ceph-mon[105022]: Health check failed: Degraded data redundancy: 2681/129531 objects degraded (2.070%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:33.901 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:33.901 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 24 16 45475 45459 118.353 81.9375 0.0113837 0.00844312 2024-09-21T10:14:33.902 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247201 10.1 10:8dbea996:::benchmark_data_smithi107_184135_object115635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:14:33.902 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:49.523792+0000", 2024-09-21T10:14:33.902 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 344.32928244599998, 2024-09-21T10:14:33.902 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.88794138099999997, 2024-09-21T10:14:33.902 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:33.902 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:33.902 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:33.902 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:14:33.902 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:14:33.902 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247201 2024-09-21T10:14:33.902 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.902 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:33.903 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.903 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:33.903 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.523792+0000", 2024-09-21T10:14:33.903 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.903 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.903 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.903 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:33.903 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.523792+0000", 2024-09-21T10:14:33.903 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.903 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.903 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.903 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:33.903 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.523791+0000", 2024-09-21T10:14:33.904 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:14:33.904 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.904 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.904 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:33.904 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.523797+0000", 2024-09-21T10:14:33.904 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0249999999999999e-06 2024-09-21T10:14:33.904 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.904 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.904 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:33.904 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.523824+0000", 2024-09-21T10:14:33.904 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7591999999999999e-05 2024-09-21T10:14:33.904 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.904 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.904 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:33.905 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.523832+0000", 2024-09-21T10:14:33.905 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.259e-06 2024-09-21T10:14:33.905 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.905 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.905 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.905 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.523879+0000", 2024-09-21T10:14:33.905 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6149999999999997e-05 2024-09-21T10:14:33.905 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.905 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.905 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.905 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.523888+0000", 2024-09-21T10:14:33.905 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.696e-06 2024-09-21T10:14:33.905 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.905 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.906 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.906 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.405090+0000", 2024-09-21T10:14:33.906 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.88120210799999998 2024-09-21T10:14:33.906 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.906 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.906 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:33.906 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.405162+0000", 2024-09-21T10:14:33.906 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.1599000000000003e-05 2024-09-21T10:14:33.906 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.906 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.906 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:14:33.906 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.405242+0000", 2024-09-21T10:14:33.906 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.0344000000000002e-05 2024-09-21T10:14:33.906 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.907 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.907 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:33.907 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.410527+0000", 2024-09-21T10:14:33.907 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0052849510000000004 2024-09-21T10:14:33.907 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.907 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.907 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.907 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411519+0000", 2024-09-21T10:14:33.907 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00099159300000000003 2024-09-21T10:14:33.907 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.907 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.907 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.907 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411661+0000", 2024-09-21T10:14:33.907 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014172300000000001 2024-09-21T10:14:33.907 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.908 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.908 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:33.908 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411686+0000", 2024-09-21T10:14:33.908 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.5471000000000001e-05 2024-09-21T10:14:33.908 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.908 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.908 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:33.908 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411733+0000", 2024-09-21T10:14:33.908 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7036000000000002e-05 2024-09-21T10:14:33.908 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.908 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.908 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.908 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.908 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.909 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247221 10.1 10:8404822e:::benchmark_data_smithi107_184135_object115655:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:14:33.909 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:49.557178+0000", 2024-09-21T10:14:33.909 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 344.29589604300003, 2024-09-21T10:14:33.909 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.85504601800000002, 2024-09-21T10:14:33.909 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:33.909 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:33.909 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:33.909 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:14:33.909 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:14:33.909 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247221 2024-09-21T10:14:33.909 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.909 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:33.909 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.909 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:33.910 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.557178+0000", 2024-09-21T10:14:33.910 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.910 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.910 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.910 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:33.910 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.557178+0000", 2024-09-21T10:14:33.910 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.910 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.910 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.910 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:33.910 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.557177+0000", 2024-09-21T10:14:33.910 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:14:33.910 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.911 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.911 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:33.911 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.557183+0000", 2024-09-21T10:14:33.911 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4799999999999998e-06 2024-09-21T10:14:33.911 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.911 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.911 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:33.911 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.557194+0000", 2024-09-21T10:14:33.911 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1338000000000001e-05 2024-09-21T10:14:33.911 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.911 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.911 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:33.911 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.557200+0000", 2024-09-21T10:14:33.911 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9699999999999996e-06 2024-09-21T10:14:33.912 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.912 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.912 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.912 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.557239+0000", 2024-09-21T10:14:33.912 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8871000000000001e-05 2024-09-21T10:14:33.912 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.912 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.912 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.912 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.557245+0000", 2024-09-21T10:14:33.912 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.908e-06 2024-09-21T10:14:33.912 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.912 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.912 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.912 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.406907+0000", 2024-09-21T10:14:33.912 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.84966192699999998 2024-09-21T10:14:33.913 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.913 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.913 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:33.913 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.406986+0000", 2024-09-21T10:14:33.913 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9424000000000004e-05 2024-09-21T10:14:33.913 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.913 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.913 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:14:33.913 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.407109+0000", 2024-09-21T10:14:33.913 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012250799999999999 2024-09-21T10:14:33.913 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.913 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.913 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:33.913 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.410564+0000", 2024-09-21T10:14:33.913 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0034551999999999999 2024-09-21T10:14:33.913 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.914 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.914 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.914 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411769+0000", 2024-09-21T10:14:33.914 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0012049719999999999 2024-09-21T10:14:33.914 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.914 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.914 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.914 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.412083+0000", 2024-09-21T10:14:33.914 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00031341399999999998 2024-09-21T10:14:33.914 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.914 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.914 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:33.914 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.412130+0000", 2024-09-21T10:14:33.914 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7654999999999997e-05 2024-09-21T10:14:33.914 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.915 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.915 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:33.915 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.412224+0000", 2024-09-21T10:14:33.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:33.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:33.916 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.3914999999999998e-05 2024-09-21T10:14:33.916 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.916 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.917 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.917 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.917 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.917 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247224 10.1 10:8b2831c2:::benchmark_data_smithi107_184135_object115658:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T10:14:33.917 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:49.565141+0000", 2024-09-21T10:14:33.917 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 344.28793268700002, 2024-09-21T10:14:33.917 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.84724627799999996, 2024-09-21T10:14:33.917 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:33.917 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:33.917 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:33.917 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:14:33.917 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:14:33.917 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247224 2024-09-21T10:14:33.917 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.917 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:33.918 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.918 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:33.918 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.565141+0000", 2024-09-21T10:14:33.918 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.918 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.918 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.918 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:33.918 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.565141+0000", 2024-09-21T10:14:33.918 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.918 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.918 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.918 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:33.918 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.565140+0000", 2024-09-21T10:14:33.918 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:14:33.918 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.919 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.919 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:33.919 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.565146+0000", 2024-09-21T10:14:33.919 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3810000000000004e-06 2024-09-21T10:14:33.919 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.919 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.919 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:33.919 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.565157+0000", 2024-09-21T10:14:33.919 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0248000000000001e-05 2024-09-21T10:14:33.919 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.919 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.919 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:33.919 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.565160+0000", 2024-09-21T10:14:33.919 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7120000000000002e-06 2024-09-21T10:14:33.919 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.920 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.920 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.920 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.565194+0000", 2024-09-21T10:14:33.920 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4010999999999997e-05 2024-09-21T10:14:33.920 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.920 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.920 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.920 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:49.565200+0000", 2024-09-21T10:14:33.920 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6319999999999996e-06 2024-09-21T10:14:33.920 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.920 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.920 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.920 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.408744+0000", 2024-09-21T10:14:33.921 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.84354435000000005 2024-09-21T10:14:33.921 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.921 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.921 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:33.921 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.408810+0000", 2024-09-21T10:14:33.921 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5333000000000006e-05 2024-09-21T10:14:33.921 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.921 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.921 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:14:33.921 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.408877+0000", 2024-09-21T10:14:33.921 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7806999999999996e-05 2024-09-21T10:14:33.921 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.921 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.921 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:33.922 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411187+0000", 2024-09-21T10:14:33.922 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002309256 2024-09-21T10:14:33.922 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.922 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.922 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.922 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.411812+0000", 2024-09-21T10:14:33.922 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00062509600000000001 2024-09-21T10:14:33.922 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.922 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.922 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.922 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.412265+0000", 2024-09-21T10:14:33.922 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000453516 2024-09-21T10:14:33.922 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.922 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.922 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:33.923 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.412295+0000", 2024-09-21T10:14:33.923 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.9938999999999999e-05 2024-09-21T10:14:33.923 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.923 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.923 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:33.923 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.412388+0000", 2024-09-21T10:14:33.923 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.2521000000000004e-05 2024-09-21T10:14:33.923 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.923 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.923 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.923 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.923 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.923 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247247 10.c 10:3f0c3473:::benchmark_data_smithi107_184135_object115681:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:14:33.923 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:50.425960+0000", 2024-09-21T10:14:33.924 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 343.427114518, 2024-09-21T10:14:33.924 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0036370640000001, 2024-09-21T10:14:33.924 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:33.924 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:33.924 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:33.924 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:14:33.924 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:14:33.924 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247247 2024-09-21T10:14:33.924 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.924 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:33.924 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.924 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:33.924 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.425960+0000", 2024-09-21T10:14:33.924 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.924 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.924 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.925 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:33.925 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.425960+0000", 2024-09-21T10:14:33.925 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.925 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.925 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.925 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:33.925 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.425958+0000", 2024-09-21T10:14:33.925 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:14:33.925 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.925 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.925 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:33.925 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.425965+0000", 2024-09-21T10:14:33.925 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9349999999999997e-06 2024-09-21T10:14:33.925 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.925 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.925 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:33.926 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.425979+0000", 2024-09-21T10:14:33.926 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4382000000000001e-05 2024-09-21T10:14:33.926 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.926 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.926 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:33.926 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.425986+0000", 2024-09-21T10:14:33.926 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2589999999999997e-06 2024-09-21T10:14:33.926 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.926 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.926 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.926 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.426038+0000", 2024-09-21T10:14:33.926 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2216000000000003e-05 2024-09-21T10:14:33.926 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.926 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.926 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.927 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.426048+0000", 2024-09-21T10:14:33.927 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0292000000000001e-05 2024-09-21T10:14:33.927 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.927 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.927 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.927 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.416064+0000", 2024-09-21T10:14:33.927 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.99001581500000002 2024-09-21T10:14:33.927 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.927 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.927 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:33.927 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.416149+0000", 2024-09-21T10:14:33.927 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.4993000000000004e-05 2024-09-21T10:14:33.927 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.927 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.927 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:14:33.927 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.416565+0000", 2024-09-21T10:14:33.928 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00041579900000000001 2024-09-21T10:14:33.928 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.928 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.928 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:33.928 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.428755+0000", 2024-09-21T10:14:33.928 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.012190256 2024-09-21T10:14:33.928 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.928 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.928 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.928 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.429344+0000", 2024-09-21T10:14:33.928 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000589399 2024-09-21T10:14:33.928 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.928 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.928 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.928 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.429447+0000", 2024-09-21T10:14:33.928 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010212700000000001 2024-09-21T10:14:33.929 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.929 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.929 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:33.929 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.429493+0000", 2024-09-21T10:14:33.929 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6418000000000002e-05 2024-09-21T10:14:33.929 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.929 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.929 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:33.929 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.429597+0000", 2024-09-21T10:14:33.929 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000103751 2024-09-21T10:14:33.929 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.929 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.929 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.929 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.929 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.930 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247250 10.c 10:36c8fb82:::benchmark_data_smithi107_184135_object115684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:14:33.930 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:50.427099+0000", 2024-09-21T10:14:33.930 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 343.42597564900001, 2024-09-21T10:14:33.930 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.003121208, 2024-09-21T10:14:33.930 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:33.930 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:33.930 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:33.930 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:14:33.930 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:14:33.930 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247250 2024-09-21T10:14:33.930 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.930 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:33.930 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.930 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:33.931 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.427099+0000", 2024-09-21T10:14:33.931 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.931 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.931 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.931 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:33.931 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.427099+0000", 2024-09-21T10:14:33.931 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.931 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.931 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.931 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:33.931 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.427097+0000", 2024-09-21T10:14:33.931 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:14:33.931 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.931 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.931 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:33.932 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.427103+0000", 2024-09-21T10:14:33.932 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7629999999999997e-06 2024-09-21T10:14:33.932 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.932 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.932 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:33.932 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.427117+0000", 2024-09-21T10:14:33.932 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3387000000000001e-05 2024-09-21T10:14:33.932 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.932 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.932 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:33.932 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.427122+0000", 2024-09-21T10:14:33.932 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7679999999999997e-06 2024-09-21T10:14:33.932 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.932 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.932 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.933 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.427170+0000", 2024-09-21T10:14:33.933 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7683999999999999e-05 2024-09-21T10:14:33.933 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.933 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.933 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.933 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.427178+0000", 2024-09-21T10:14:33.933 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.1089999999999998e-06 2024-09-21T10:14:33.933 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.933 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.933 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.933 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.420448+0000", 2024-09-21T10:14:33.933 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.99326971600000002 2024-09-21T10:14:33.933 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.933 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.933 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:33.933 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.420567+0000", 2024-09-21T10:14:33.934 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011879000000000001 2024-09-21T10:14:33.934 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.934 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.934 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:14:33.934 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.420690+0000", 2024-09-21T10:14:33.934 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012281600000000001 2024-09-21T10:14:33.934 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.934 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.934 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:33.934 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.428762+0000", 2024-09-21T10:14:33.934 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0080721899999999999 2024-09-21T10:14:33.934 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.934 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.934 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.934 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.429402+0000", 2024-09-21T10:14:33.935 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00063987799999999997 2024-09-21T10:14:33.935 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.935 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.935 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.935 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.430063+0000", 2024-09-21T10:14:33.935 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00066114500000000003 2024-09-21T10:14:33.935 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.935 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.935 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:33.935 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.430126+0000", 2024-09-21T10:14:33.935 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2983000000000003e-05 2024-09-21T10:14:33.935 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.935 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.935 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:33.935 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.430220+0000", 2024-09-21T10:14:33.936 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.4023000000000004e-05 2024-09-21T10:14:33.936 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.936 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.936 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.936 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.936 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.936 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247271 10.c 10:33c1c3c0:::benchmark_data_smithi107_184135_object115705:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:14:33.936 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:50.447124+0000", 2024-09-21T10:14:33.936 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 343.40595005, 2024-09-21T10:14:33.936 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.98329654799999999, 2024-09-21T10:14:33.936 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:33.936 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:33.936 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:33.936 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:14:33.936 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:14:33.936 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247271 2024-09-21T10:14:33.937 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.937 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:33.937 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.937 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:33.937 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447124+0000", 2024-09-21T10:14:33.937 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.937 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.937 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.937 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:33.937 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447124+0000", 2024-09-21T10:14:33.937 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.937 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.937 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.937 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:33.937 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447123+0000", 2024-09-21T10:14:33.937 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:14:33.938 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.938 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.938 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:33.938 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447130+0000", 2024-09-21T10:14:33.938 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9640000000000003e-06 2024-09-21T10:14:33.938 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.938 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.938 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:33.938 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447145+0000", 2024-09-21T10:14:33.938 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4280999999999999e-05 2024-09-21T10:14:33.938 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.938 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.938 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:33.938 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447152+0000", 2024-09-21T10:14:33.938 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.1169999999999998e-06 2024-09-21T10:14:33.939 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.939 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.939 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.939 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447213+0000", 2024-09-21T10:14:33.939 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1270000000000001e-05 2024-09-21T10:14:33.939 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.939 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.939 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.939 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447224+0000", 2024-09-21T10:14:33.939 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0813e-05 2024-09-21T10:14:33.939 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.939 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.939 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.939 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.423482+0000", 2024-09-21T10:14:33.939 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.97625846900000002 2024-09-21T10:14:33.939 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.940 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.940 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:33.940 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.423596+0000", 2024-09-21T10:14:33.940 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000114015 2024-09-21T10:14:33.940 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.940 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.940 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:14:33.940 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.423713+0000", 2024-09-21T10:14:33.940 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000116908 2024-09-21T10:14:33.940 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.940 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.940 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:33.940 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.428767+0000", 2024-09-21T10:14:33.940 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0050531769999999998 2024-09-21T10:14:33.940 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.940 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.941 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.941 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.429646+0000", 2024-09-21T10:14:33.941 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00087916000000000003 2024-09-21T10:14:33.941 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.941 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.941 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.941 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.430271+0000", 2024-09-21T10:14:33.941 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00062482399999999995 2024-09-21T10:14:33.941 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.941 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.941 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:33.941 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.430315+0000", 2024-09-21T10:14:33.941 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4449000000000001e-05 2024-09-21T10:14:33.941 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.942 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.942 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:33.942 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.430421+0000", 2024-09-21T10:14:33.942 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010572499999999999 2024-09-21T10:14:33.942 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.942 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.942 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.942 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.942 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.942 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247278 10.c 10:300615d8:::benchmark_data_smithi107_184135_object115712:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:14:33.942 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:50.460998+0000", 2024-09-21T10:14:33.942 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 343.39207641100001, 2024-09-21T10:14:33.942 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.96960406300000002, 2024-09-21T10:14:33.942 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:33.942 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:33.943 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:33.943 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:14:33.943 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:14:33.943 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247278 2024-09-21T10:14:33.943 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.943 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:33.943 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.943 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:33.943 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.460998+0000", 2024-09-21T10:14:33.943 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.943 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.943 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.943 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:33.943 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.460998+0000", 2024-09-21T10:14:33.943 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.943 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.944 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.944 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:33.944 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.460997+0000", 2024-09-21T10:14:33.944 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:14:33.944 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.944 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.944 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:33.944 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.461002+0000", 2024-09-21T10:14:33.944 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8620000000000002e-06 2024-09-21T10:14:33.944 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.944 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.944 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:33.944 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.461012+0000", 2024-09-21T10:14:33.944 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0616e-05 2024-09-21T10:14:33.944 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.944 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.945 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:33.945 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.461017+0000", 2024-09-21T10:14:33.945 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4129999999999999e-06 2024-09-21T10:14:33.945 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.945 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.945 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.945 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.461081+0000", 2024-09-21T10:14:33.945 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4398000000000002e-05 2024-09-21T10:14:33.945 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.945 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.945 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.945 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.461090+0000", 2024-09-21T10:14:33.945 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6629999999999999e-06 2024-09-21T10:14:33.945 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.945 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.945 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.946 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.426105+0000", 2024-09-21T10:14:33.946 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.96501570000000003 2024-09-21T10:14:33.946 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.946 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.946 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:33.946 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.426178+0000", 2024-09-21T10:14:33.946 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2175000000000006e-05 2024-09-21T10:14:33.946 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.946 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.946 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,4", 2024-09-21T10:14:33.946 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.426247+0000", 2024-09-21T10:14:33.946 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9805e-05 2024-09-21T10:14:33.946 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.947 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.947 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:33.947 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.429282+0000", 2024-09-21T10:14:33.947 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0030342699999999999 2024-09-21T10:14:33.947 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.947 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.947 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.947 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.429675+0000", 2024-09-21T10:14:33.947 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00039293900000000003 2024-09-21T10:14:33.947 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.947 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.947 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.947 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.430460+0000", 2024-09-21T10:14:33.947 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00078493499999999995 2024-09-21T10:14:33.947 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.948 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.948 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:33.948 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.430514+0000", 2024-09-21T10:14:33.948 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4901999999999997e-05 2024-09-21T10:14:33.948 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.948 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.948 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:33.948 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.430602+0000", 2024-09-21T10:14:33.948 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.7385000000000004e-05 2024-09-21T10:14:33.948 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.948 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.948 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.948 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.948 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.948 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36561.0:50688 11.e 11:7867821a:::benchmark_data_smithi107_218812_object50687:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-21T10:14:33.949 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:10:19.583524+0000", 2024-09-21T10:14:33.949 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 254.26955061800001, 2024-09-21T10:14:33.949 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.245396748, 2024-09-21T10:14:33.949 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:33.949 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:33.949 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:33.949 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36561", 2024-09-21T10:14:33.949 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:14:33.949 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 50688 2024-09-21T10:14:33.949 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.949 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:33.949 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.949 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:33.949 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.583524+0000", 2024-09-21T10:14:33.949 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.950 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.950 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.950 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:33.950 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.583524+0000", 2024-09-21T10:14:33.950 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.950 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.950 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.950 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:33.950 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.583522+0000", 2024-09-21T10:14:33.950 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:14:33.950 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.950 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.950 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:33.951 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.583579+0000", 2024-09-21T10:14:33.951 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6925999999999997e-05 2024-09-21T10:14:33.951 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.951 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.951 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:33.951 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.583591+0000", 2024-09-21T10:14:33.951 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2527999999999999e-05 2024-09-21T10:14:33.951 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.951 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.951 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:33.951 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.583595+0000", 2024-09-21T10:14:33.951 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.019e-06 2024-09-21T10:14:33.951 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.951 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.951 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.951 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.583637+0000", 2024-09-21T10:14:33.952 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1841000000000001e-05 2024-09-21T10:14:33.952 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.952 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.952 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:33.952 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.583720+0000", 2024-09-21T10:14:33.952 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.2999999999999998e-05 2024-09-21T10:14:33.952 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.952 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.952 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:14:33.952 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.583806+0000", 2024-09-21T10:14:33.952 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6076999999999996e-05 2024-09-21T10:14:33.952 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.952 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.952 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:33.952 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.585516+0000", 2024-09-21T10:14:33.952 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0017102949999999999 2024-09-21T10:14:33.953 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.953 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.953 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.953 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.587893+0000", 2024-09-21T10:14:33.953 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0023768470000000001 2024-09-21T10:14:33.953 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.953 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.953 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.953 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.828840+0000", 2024-09-21T10:14:33.953 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.24094675800000001 2024-09-21T10:14:33.953 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.953 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.953 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:33.953 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.828892+0000", 2024-09-21T10:14:33.953 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1731999999999999e-05 2024-09-21T10:14:33.953 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.954 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.954 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:33.954 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.828920+0000", 2024-09-21T10:14:33.954 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.8649999999999998e-05 2024-09-21T10:14:33.954 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.954 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.954 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.954 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.954 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.954 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36561.0:50706 11.f 11:f8bf77a7:::benchmark_data_smithi107_218812_object50705:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-21T10:14:33.954 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:10:19.592848+0000", 2024-09-21T10:14:33.954 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 254.26022656200001, 2024-09-21T10:14:33.954 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.24061939600000001, 2024-09-21T10:14:33.954 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:33.954 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:33.955 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:33.955 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36561", 2024-09-21T10:14:33.955 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:14:33.955 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 50706 2024-09-21T10:14:33.955 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.955 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:33.955 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.955 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:33.955 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.592848+0000", 2024-09-21T10:14:33.955 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.955 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.955 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.955 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:33.955 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.592848+0000", 2024-09-21T10:14:33.956 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.956 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.956 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.956 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:33.956 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.592846+0000", 2024-09-21T10:14:33.956 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:14:33.956 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.956 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.956 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:33.956 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.592887+0000", 2024-09-21T10:14:33.956 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0278000000000003e-05 2024-09-21T10:14:33.956 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.956 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.956 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:33.956 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.592901+0000", 2024-09-21T10:14:33.956 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4299e-05 2024-09-21T10:14:33.957 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.957 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.957 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:33.957 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.592907+0000", 2024-09-21T10:14:33.957 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2910000000000004e-06 2024-09-21T10:14:33.957 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.957 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.957 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.957 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.592959+0000", 2024-09-21T10:14:33.957 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1397000000000003e-05 2024-09-21T10:14:33.957 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.957 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.957 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:33.957 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.593065+0000", 2024-09-21T10:14:33.957 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010652100000000001 2024-09-21T10:14:33.957 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.958 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.958 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:14:33.958 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.593190+0000", 2024-09-21T10:14:33.958 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012517 2024-09-21T10:14:33.958 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.958 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.958 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:33.958 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.594917+0000", 2024-09-21T10:14:33.958 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001727042 2024-09-21T10:14:33.958 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.958 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.958 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.958 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.595965+0000", 2024-09-21T10:14:33.958 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0010480909999999999 2024-09-21T10:14:33.959 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.959 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.959 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.959 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.833370+0000", 2024-09-21T10:14:33.959 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23740465799999999 2024-09-21T10:14:33.959 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.959 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.959 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:33.959 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.833426+0000", 2024-09-21T10:14:33.959 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6198e-05 2024-09-21T10:14:33.959 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.959 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.959 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:33.959 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:19.833467+0000", 2024-09-21T10:14:33.959 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0781000000000001e-05 2024-09-21T10:14:33.960 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.960 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.960 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.960 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.960 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.960 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36561.0:91073 11.5 11:aba05e3e:::benchmark_data_smithi107_218812_object91072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-21T10:14:33.960 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:10:56.374738+0000", 2024-09-21T10:14:33.960 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 217.478335797, 2024-09-21T10:14:33.960 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.371354148, 2024-09-21T10:14:33.960 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:33.960 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:14:33.960 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:33.960 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36561", 2024-09-21T10:14:33.960 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:14:33.960 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 91073 2024-09-21T10:14:33.961 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.961 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:33.961 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.961 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:33.961 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.374738+0000", 2024-09-21T10:14:33.961 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.961 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.961 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.961 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:33.961 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.374738+0000", 2024-09-21T10:14:33.961 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.961 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.961 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.961 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:33.961 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.374737+0000", 2024-09-21T10:14:33.961 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:14:33.962 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.962 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.962 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:33.962 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.374836+0000", 2024-09-21T10:14:33.962 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.9283000000000005e-05 2024-09-21T10:14:33.962 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.962 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.962 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:33.962 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.374848+0000", 2024-09-21T10:14:33.962 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1473e-05 2024-09-21T10:14:33.962 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.962 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.962 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:33.962 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.374853+0000", 2024-09-21T10:14:33.963 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2719999999999997e-06 2024-09-21T10:14:33.963 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.963 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.963 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.963 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.374934+0000", 2024-09-21T10:14:33.963 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.0448e-05 2024-09-21T10:14:33.963 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.963 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.963 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:33.963 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.375050+0000", 2024-09-21T10:14:33.963 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000116227 2024-09-21T10:14:33.963 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.963 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.963 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:14:33.963 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.375205+0000", 2024-09-21T10:14:33.963 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015526200000000001 2024-09-21T10:14:33.964 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.964 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.964 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:33.964 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.378707+0000", 2024-09-21T10:14:33.964 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003502158 2024-09-21T10:14:33.964 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.964 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.964 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.964 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.378907+0000", 2024-09-21T10:14:33.964 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000199326 2024-09-21T10:14:33.964 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.964 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.964 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.964 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.384513+0000", 2024-09-21T10:14:33.964 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0056069120000000004 2024-09-21T10:14:33.964 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.965 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.965 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.965 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.384522+0000", 2024-09-21T10:14:33.965 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.5169999999999999e-06 2024-09-21T10:14:33.965 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.965 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.965 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.965 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:57.387352+0000", 2024-09-21T10:14:33.965 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0028299380000001 2024-09-21T10:14:33.965 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.965 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.965 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.965 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:57.387363+0000", 2024-09-21T10:14:33.965 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1212999999999999e-05 2024-09-21T10:14:33.965 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.965 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.966 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.966 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.746017+0000", 2024-09-21T10:14:33.966 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3586540110000001 2024-09-21T10:14:33.966 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.966 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.966 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:33.966 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.746093+0000", 2024-09-21T10:14:33.966 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.5432000000000006e-05 2024-09-21T10:14:33.966 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.966 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.966 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.966 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.966 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.966 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36561.0:91079 11.5 11:a8488685:::benchmark_data_smithi107_218812_object91078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-21T10:14:33.967 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:10:56.383599+0000", 2024-09-21T10:14:33.967 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 217.469475101, 2024-09-21T10:14:33.967 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.4063422349999999, 2024-09-21T10:14:33.967 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:33.967 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:33.967 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:33.967 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36561", 2024-09-21T10:14:33.967 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:14:33.967 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 91079 2024-09-21T10:14:33.967 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.967 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:33.967 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.967 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:33.967 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.383599+0000", 2024-09-21T10:14:33.967 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.967 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.968 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.968 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:33.968 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.383599+0000", 2024-09-21T10:14:33.968 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.968 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.968 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.968 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:33.968 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.383597+0000", 2024-09-21T10:14:33.968 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:14:33.968 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.968 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.968 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:33.968 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.383690+0000", 2024-09-21T10:14:33.969 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.2906999999999997e-05 2024-09-21T10:14:33.969 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.969 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.969 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:33.969 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.383705+0000", 2024-09-21T10:14:33.969 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4817999999999999e-05 2024-09-21T10:14:33.969 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.969 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.969 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:33.969 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.383713+0000", 2024-09-21T10:14:33.969 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.6599999999999995e-06 2024-09-21T10:14:33.969 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.969 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.969 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.969 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.384546+0000", 2024-09-21T10:14:33.969 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00083367200000000004 2024-09-21T10:14:33.970 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.970 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.970 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.970 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.384553+0000", 2024-09-21T10:14:33.970 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3099999999999997e-06 2024-09-21T10:14:33.970 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.970 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.970 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.970 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:57.387384+0000", 2024-09-21T10:14:33.970 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.00283181 2024-09-21T10:14:33.970 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.970 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.970 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.970 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:57.387391+0000", 2024-09-21T10:14:33.970 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5719999999999999e-06 2024-09-21T10:14:33.970 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.971 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.971 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.971 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.746128+0000", 2024-09-21T10:14:33.971 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3587367399999999 2024-09-21T10:14:33.971 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.971 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.971 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:33.971 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.746197+0000", 2024-09-21T10:14:33.971 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9661999999999998e-05 2024-09-21T10:14:33.971 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.971 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.971 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:14:33.971 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.746285+0000", 2024-09-21T10:14:33.971 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.7943999999999997e-05 2024-09-21T10:14:33.972 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.972 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.972 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:33.972 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.767420+0000", 2024-09-21T10:14:33.972 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.021134673999999999 2024-09-21T10:14:33.972 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.972 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.972 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.972 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.774710+0000", 2024-09-21T10:14:33.972 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0072896499999999999 2024-09-21T10:14:33.972 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.972 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.972 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.972 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.784581+0000", 2024-09-21T10:14:33.972 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0098717130000000007 2024-09-21T10:14:33.972 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.973 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.973 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:33.973 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.784639+0000", 2024-09-21T10:14:33.973 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8146999999999997e-05 2024-09-21T10:14:33.973 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.973 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.973 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:33.973 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.789941+0000", 2024-09-21T10:14:33.973 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0053018570000000001 2024-09-21T10:14:33.973 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.973 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.973 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.973 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.973 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.973 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36561.0:91098 11.5 11:ab6631c2:::benchmark_data_smithi107_218812_object91097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:14:33.973 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:10:56.419733+0000", 2024-09-21T10:14:33.974 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 217.43334070399999, 2024-09-21T10:14:33.974 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.3754622799999998, 2024-09-21T10:14:33.974 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:33.974 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:33.974 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:33.974 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36561", 2024-09-21T10:14:33.974 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:14:33.974 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 91098 2024-09-21T10:14:33.974 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.974 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:33.974 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.974 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:33.974 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.419733+0000", 2024-09-21T10:14:33.974 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.974 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.975 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.975 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:33.975 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.419733+0000", 2024-09-21T10:14:33.975 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.975 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.975 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.975 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:33.975 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.419733+0000", 2024-09-21T10:14:33.975 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:14:33.975 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.975 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.975 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:33.975 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.419765+0000", 2024-09-21T10:14:33.975 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1600000000000002e-05 2024-09-21T10:14:33.975 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.975 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.976 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:33.976 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.419772+0000", 2024-09-21T10:14:33.976 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3080000000000002e-06 2024-09-21T10:14:33.976 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.976 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.976 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:33.976 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.419776+0000", 2024-09-21T10:14:33.976 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.946e-06 2024-09-21T10:14:33.976 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.976 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.976 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.976 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.419829+0000", 2024-09-21T10:14:33.976 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3248999999999998e-05 2024-09-21T10:14:33.976 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.976 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.976 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.977 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.419835+0000", 2024-09-21T10:14:33.977 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4049999999999997e-06 2024-09-21T10:14:33.977 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.977 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.977 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.977 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:57.387420+0000", 2024-09-21T10:14:33.977 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.96758447299999994 2024-09-21T10:14:33.977 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.977 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.977 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.977 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:57.387426+0000", 2024-09-21T10:14:33.977 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5880000000000003e-06 2024-09-21T10:14:33.977 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.977 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.977 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.977 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.748878+0000", 2024-09-21T10:14:33.978 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.361451953 2024-09-21T10:14:33.978 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.978 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.978 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:33.978 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.749001+0000", 2024-09-21T10:14:33.978 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000122318 2024-09-21T10:14:33.978 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.978 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.978 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:14:33.978 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.749153+0000", 2024-09-21T10:14:33.978 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015237000000000001 2024-09-21T10:14:33.979 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.979 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.979 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:33.979 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.767428+0000", 2024-09-21T10:14:33.979 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.018274420999999999 2024-09-21T10:14:33.979 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.979 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.979 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.979 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.774770+0000", 2024-09-21T10:14:33.979 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0073419870000000003 2024-09-21T10:14:33.979 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.979 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.979 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.979 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.789992+0000", 2024-09-21T10:14:33.979 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.015222037000000001 2024-09-21T10:14:33.980 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.980 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.980 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:33.980 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.790046+0000", 2024-09-21T10:14:33.980 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4761e-05 2024-09-21T10:14:33.980 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.980 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.980 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:33.980 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.795196+0000", 2024-09-21T10:14:33.980 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0051494139999999997 2024-09-21T10:14:33.980 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.980 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.980 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.980 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.981 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.981 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36561.0:91120 11.5 11:a9d90972:::benchmark_data_smithi107_218812_object91119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:14:33.981 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:10:56.447451+0000", 2024-09-21T10:14:33.981 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 217.40562305399999, 2024-09-21T10:14:33.981 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.3534111200000001, 2024-09-21T10:14:33.981 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:33.981 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:33.981 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:33.981 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36561", 2024-09-21T10:14:33.981 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:14:33.981 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 91120 2024-09-21T10:14:33.981 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.981 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:33.981 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.981 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:33.982 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.447451+0000", 2024-09-21T10:14:33.982 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.982 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.982 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.982 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:33.982 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.447451+0000", 2024-09-21T10:14:33.982 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.982 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.982 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.982 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:33.982 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.447450+0000", 2024-09-21T10:14:33.982 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:14:33.982 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.982 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.982 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:33.982 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.447477+0000", 2024-09-21T10:14:33.983 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.6923e-05 2024-09-21T10:14:33.983 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.983 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.983 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:33.983 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.447489+0000", 2024-09-21T10:14:33.983 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1941e-05 2024-09-21T10:14:33.983 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.983 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.983 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:33.983 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.447493+0000", 2024-09-21T10:14:33.983 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4499999999999997e-06 2024-09-21T10:14:33.983 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.983 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.983 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.983 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.447538+0000", 2024-09-21T10:14:33.984 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5040000000000002e-05 2024-09-21T10:14:33.984 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.984 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.984 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.984 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.447545+0000", 2024-09-21T10:14:33.984 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.951e-06 2024-09-21T10:14:33.984 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.984 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.984 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.984 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:57.387446+0000", 2024-09-21T10:14:33.984 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.93990089499999996 2024-09-21T10:14:33.984 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.984 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.984 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.984 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:57.387452+0000", 2024-09-21T10:14:33.984 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3110000000000002e-06 2024-09-21T10:14:33.985 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.985 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.985 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.985 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.752150+0000", 2024-09-21T10:14:33.985 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3646975020000001 2024-09-21T10:14:33.985 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.985 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.985 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:33.985 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.752281+0000", 2024-09-21T10:14:33.985 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013172699999999999 2024-09-21T10:14:33.985 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.985 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.985 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:14:33.985 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.752463+0000", 2024-09-21T10:14:33.985 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00018151500000000001 2024-09-21T10:14:33.985 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.986 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.986 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:33.986 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.767434+0000", 2024-09-21T10:14:33.986 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.014971029 2024-09-21T10:14:33.986 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.986 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.986 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.986 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.774832+0000", 2024-09-21T10:14:33.986 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0073980349999999999 2024-09-21T10:14:33.986 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.986 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.986 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.986 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.795273+0000", 2024-09-21T10:14:33.986 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.020440732999999999 2024-09-21T10:14:33.987 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.987 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.987 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:33.987 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.795327+0000", 2024-09-21T10:14:33.987 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4500000000000003e-05 2024-09-21T10:14:33.987 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.987 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.987 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:33.987 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.800862+0000", 2024-09-21T10:14:33.987 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0055349400000000003 2024-09-21T10:14:33.987 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.987 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.987 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.987 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.987 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.987 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36561.0:91129 11.5 11:afb7745b:::benchmark_data_smithi107_218812_object91128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:14:33.988 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:10:56.470467+0000", 2024-09-21T10:14:33.988 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 217.38260677599999, 2024-09-21T10:14:33.988 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.3404116099999999, 2024-09-21T10:14:33.988 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:14:33.988 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:14:33.988 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:14:33.988 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36561", 2024-09-21T10:14:33.988 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:14:33.988 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 91129 2024-09-21T10:14:33.988 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.988 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:14:33.988 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.988 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:14:33.988 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.470467+0000", 2024-09-21T10:14:33.988 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.988 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.989 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.989 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:14:33.989 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.470467+0000", 2024-09-21T10:14:33.989 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:14:33.989 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.989 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.989 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:14:33.989 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.470466+0000", 2024-09-21T10:14:33.989 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:14:33.989 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.989 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.989 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:14:33.989 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.470518+0000", 2024-09-21T10:14:33.989 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1638999999999998e-05 2024-09-21T10:14:33.989 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.989 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.990 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:14:33.990 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.470530+0000", 2024-09-21T10:14:33.990 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1764e-05 2024-09-21T10:14:33.990 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.990 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.990 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:14:33.990 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.470534+0000", 2024-09-21T10:14:33.990 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4240000000000003e-06 2024-09-21T10:14:33.990 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.990 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.990 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.990 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.470575+0000", 2024-09-21T10:14:33.990 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1199000000000003e-05 2024-09-21T10:14:33.990 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.990 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.991 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.991 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:56.470583+0000", 2024-09-21T10:14:33.991 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.0099999999999995e-06 2024-09-21T10:14:33.991 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.991 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.991 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.991 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:57.387473+0000", 2024-09-21T10:14:33.991 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.91688940900000004 2024-09-21T10:14:33.991 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.991 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.991 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:14:33.991 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:57.387479+0000", 2024-09-21T10:14:33.991 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4509999999999997e-06 2024-09-21T10:14:33.991 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.991 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.991 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:14:33.992 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.755649+0000", 2024-09-21T10:14:33.992 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.36817038 2024-09-21T10:14:33.992 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.992 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.992 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:14:33.992 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.755801+0000", 2024-09-21T10:14:33.992 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015123599999999999 2024-09-21T10:14:33.992 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.992 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.992 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:14:33.992 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.756034+0000", 2024-09-21T10:14:33.992 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00023330700000000001 2024-09-21T10:14:33.992 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.992 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.992 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:14:33.993 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.774649+0000", 2024-09-21T10:14:33.993 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.018614616 2024-09-21T10:14:33.993 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.993 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.993 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.993 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.774875+0000", 2024-09-21T10:14:33.993 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000226624 2024-09-21T10:14:33.993 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.993 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.993 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:14:33.993 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.800924+0000", 2024-09-21T10:14:33.993 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.026048737999999998 2024-09-21T10:14:33.993 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.993 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.993 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:14:33.993 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.800975+0000", 2024-09-21T10:14:33.994 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0770999999999999e-05 2024-09-21T10:14:33.994 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:14:33.994 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:14:33.994 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:14:33.994 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:10:58.810879+0000", 2024-09-21T10:14:33.994 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0099042560000000002 2024-09-21T10:14:33.994 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.994 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.994 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.994 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:14:33.994 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:14:33.994 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:14:34.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:34.048 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:34.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:34.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:34.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:34.187 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:34.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:34.289 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:34.317 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:34.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:34.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:34.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:34.440 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:34.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:34.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:34.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:34.562 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:34.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:34.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:34.664 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:34.692 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:34.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:34.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:34.794 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:34.834 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:34.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:34.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:34.872 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 25 16 46471 46455 116.109 62.25 0.011211 0.00860576 2024-09-21T10:14:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:34 smithi129 ceph-mon[88471]: pgmap v1076: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 peering, 104 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s; 2681/129531 objects degraded (2.070%); 2600/129531 objects misplaced (2.007%); 107 KiB/s, 1 keys/s, 2 objects/s recovering 2024-09-21T10:14:34.936 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:34.964 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:34.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:34.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:35.065 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:35.097 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:35.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:35.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:34 smithi107 ceph-mon[102221]: pgmap v1076: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 peering, 104 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s; 2681/129531 objects degraded (2.070%); 2600/129531 objects misplaced (2.007%); 107 KiB/s, 1 keys/s, 2 objects/s recovering 2024-09-21T10:14:35.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:34 smithi107 ceph-mon[105022]: pgmap v1076: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 peering, 104 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s; 2681/129531 objects degraded (2.070%); 2600/129531 objects misplaced (2.007%); 107 KiB/s, 1 keys/s, 2 objects/s recovering 2024-09-21T10:14:35.111 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:14:35.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:35.260 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:35.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:35.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:35.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:35.405 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:35.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:35.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:35.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:35.545 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:35.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:35.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:35.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:35.677 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:35.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:35.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:35.779 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:35.805 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:35.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:35.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:35.872 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 26 16 47533 47517 114.196 66.375 0.0047089 0.00875132 2024-09-21T10:14:35.906 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:35.934 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:35.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:35.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:36.036 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:36.062 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:36.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:36.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:36.164 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:36.192 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:36.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:36.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:36.294 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:36.319 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:36.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:36.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:36.420 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:36.448 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:36.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:36.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:36.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:36.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:36.628 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:36.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:36.729 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:36.767 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:36.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:36.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:36.869 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:36.872 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 27 16 48777 48761 112.846 77.75 0.00873048 0.00885477 2024-09-21T10:14:36.900 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:36.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:36.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:37.001 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:37.032 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:37.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:37.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:37.048 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:36 smithi107 ceph-mon[102221]: pgmap v1077: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 peering, 104 active+clean; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 2681/131139 objects degraded (2.044%); 2600/131139 objects misplaced (1.983%); 107 KiB/s, 1 keys/s, 2 objects/s recovering 2024-09-21T10:14:37.048 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:36 smithi107 ceph-mon[105022]: pgmap v1077: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 peering, 104 active+clean; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 2681/131139 objects degraded (2.044%); 2600/131139 objects misplaced (1.983%); 107 KiB/s, 1 keys/s, 2 objects/s recovering 2024-09-21T10:14:37.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:37.166 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:37.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:37.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:37.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:36 smithi129 ceph-mon[88471]: pgmap v1077: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 peering, 104 active+clean; 2.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s; 2681/131139 objects degraded (2.044%); 2600/131139 objects misplaced (1.983%); 107 KiB/s, 1 keys/s, 2 objects/s recovering 2024-09-21T10:14:37.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:37.296 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:37.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:37.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:37.398 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:37.427 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:37.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:37.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:37.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:37.571 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:37.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:37.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:37.673 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:37.705 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:37.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:37.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:37.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:37.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:37.848 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:37.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:37.872 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 28 16 50189 50173 111.967 88.25 0.00798633 0.00892644 2024-09-21T10:14:37.949 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:37.992 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:37.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:37.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:38.094 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:38.125 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:38.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:38.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:38.227 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:38.252 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:38.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:38.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:38.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:38.390 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:38.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:38.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:38.491 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:38.523 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:38.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:38.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:38.565 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-21T10:14:38.625 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:38.651 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:38.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:38.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:38.753 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:38.780 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:38.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:38 smithi107 ceph-mon[102221]: pgmap v1078: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 2681/140685 objects degraded (1.906%); 2360/140685 objects misplaced (1.678%); 2.0 MiB/s, 1 keys/s, 33 objects/s recovering 2024-09-21T10:14:38.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:38 smithi107 ceph-mon[105022]: pgmap v1078: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 2681/140685 objects degraded (1.906%); 2360/140685 objects misplaced (1.678%); 2.0 MiB/s, 1 keys/s, 33 objects/s recovering 2024-09-21T10:14:38.872 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 29 16 51616 51600 111.182 89.1875 0.0167985 0.00898861 2024-09-21T10:14:38.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:38.930 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:38.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:38.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:39.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:39.082 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:39.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:39.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:39.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:38 smithi129 ceph-mon[88471]: pgmap v1078: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 2681/140685 objects degraded (1.906%); 2360/140685 objects misplaced (1.678%); 2.0 MiB/s, 1 keys/s, 33 objects/s recovering 2024-09-21T10:14:39.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:39.227 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:39.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:39.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:39.241 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:39.329 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:39.353 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:39.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:39.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:39.454 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:39.497 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:39.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:39.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:39.599 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:39.622 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:39.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:39.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:39.723 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:39.750 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:39.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:39.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:39.786 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:14:39.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:39 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2681/140685 objects degraded (1.906%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:39 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3389534935' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:14:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:39 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:14:39.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:39 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2681/140685 objects degraded (1.906%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:39.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:39 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3389534935' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:14:39.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:39 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:14:39.873 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 30 16 52805 52789 109.953 74.3125 0.0100744 0.00908751 2024-09-21T10:14:39.900 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:39.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:39.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:40.002 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:40.032 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:40.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:40.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:40.070 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:39 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2681/140685 objects degraded (1.906%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:40.070 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:39 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3389534935' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:14:40.070 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:39 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:14:40.135 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:40.168 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:40.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:40.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:40.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:40.291 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:40.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:40.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:40.392 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:40.420 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:14:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:14:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:14:40.424 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:40.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:40.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:40.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:40.558 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:40.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:40.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:40.659 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:40.691 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:40.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:40.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:40.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:40.826 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:40.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:40.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:40.873 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 31 16 54145 54129 109.107 83.75 0.0126071 0.0091601 2024-09-21T10:14:40.928 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:40.962 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:40.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:40.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:41.064 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:41.092 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:41.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:41.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:40 smithi107 ceph-mon[102221]: pgmap v1079: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 2681/146376 objects degraded (1.832%); 2360/146376 objects misplaced (1.612%); 1.8 MiB/s, 1 keys/s, 29 objects/s recovering 2024-09-21T10:14:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:40 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:14:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:40 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:14:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:40 smithi107 ceph-mon[102221]: osdmap e306: 8 total, 8 up, 8 in 2024-09-21T10:14:41.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:40 smithi107 ceph-mon[105022]: pgmap v1079: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 2681/146376 objects degraded (1.832%); 2360/146376 objects misplaced (1.612%); 1.8 MiB/s, 1 keys/s, 29 objects/s recovering 2024-09-21T10:14:41.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:40 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:14:41.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:40 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:14:41.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:40 smithi107 ceph-mon[105022]: osdmap e306: 8 total, 8 up, 8 in 2024-09-21T10:14:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:40 smithi129 ceph-mon[88471]: pgmap v1079: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 2681/146376 objects degraded (1.832%); 2360/146376 objects misplaced (1.612%); 1.8 MiB/s, 1 keys/s, 29 objects/s recovering 2024-09-21T10:14:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:40 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:14:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:40 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:14:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:40 smithi129 ceph-mon[88471]: osdmap e306: 8 total, 8 up, 8 in 2024-09-21T10:14:41.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:41.232 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:41.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:41.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:41.334 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:41.348 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:41.365 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:41.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:41.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:41.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:41.509 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:41.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:41.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:41.611 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:41.640 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:41.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:41.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:41.742 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:41.775 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:41.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:41.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:41.873 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 32 16 55491 55475 108.326 84.125 0.00300798 0.00921244 2024-09-21T10:14:41.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:41.897 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:41.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:41.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:41.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:42.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:42.033 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:42.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:42.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:14:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:14:41] "GET /metrics HTTP/1.1" 200 36823 "" "Prometheus/2.51.0" 2024-09-21T10:14:42.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:42.170 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:42.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:42.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:42.271 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:42.304 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:42.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:42.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:42.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:42.440 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:42.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:42.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:42.542 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:42.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:42.576 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:42.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:14:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:42 smithi107 ceph-mon[102221]: pgmap v1081: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s; 2681/150993 objects degraded (1.776%); 2360/150993 objects misplaced (1.563%); 1.6 MiB/s, 1 keys/s, 26 objects/s recovering 2024-09-21T10:14:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:14:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:14:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:42 smithi107 ceph-mon[105022]: pgmap v1081: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s; 2681/150993 objects degraded (1.776%); 2360/150993 objects misplaced (1.563%); 1.6 MiB/s, 1 keys/s, 26 objects/s recovering 2024-09-21T10:14:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:14:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:14:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:42 smithi129 ceph-mon[88471]: pgmap v1081: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s; 2681/150993 objects degraded (1.776%); 2360/150993 objects misplaced (1.563%); 1.6 MiB/s, 1 keys/s, 26 objects/s recovering 2024-09-21T10:14:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:14:42.676 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:42.703 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:42.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:42.805 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:42.833 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:42.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:42.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:42.874 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 33 16 56704 56688 107.338 75.8125 0.00625054 0.00931078 2024-09-21T10:14:42.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:42.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:42.971 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:42.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:43.071 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:43.100 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:43.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:43.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:43.202 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:43.240 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:43.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:43.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:43.386 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:43.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:43.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:43.488 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:43.514 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:43.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:43.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:43.566 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:14:43.566 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:14:43.566 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:14:43.566 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.000000 2024-09-21T10:14:43.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 5 0 2024-09-21T10:14:43.616 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:43.644 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:43.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:43.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:43.746 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:43.773 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:43.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:43.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:43.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:43.875 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 34 16 57991 57975 106.545 80.4375 0.0241725 0.00937865 2024-09-21T10:14:43.919 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:43.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:43.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:44.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:44.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:44.060 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:44.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:44.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:44.188 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:44.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:44.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:44.289 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:44.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:44.321 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:44.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:44.422 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:44.462 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:44.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:44.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:44.563 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:44.590 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:44.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:44.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:44 smithi129 ceph-mon[88471]: pgmap v1082: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s; 2681/165882 objects degraded (1.616%); 2160/165882 objects misplaced (1.302%); 2.7 MiB/s, 44 objects/s recovering 2024-09-21T10:14:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:44 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/893920425' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:44.692 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:44.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:44.722 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:44.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:44.823 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:44.851 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:44.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:44.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:44.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:44 smithi107 ceph-mon[102221]: pgmap v1082: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s; 2681/165882 objects degraded (1.616%); 2160/165882 objects misplaced (1.302%); 2.7 MiB/s, 44 objects/s recovering 2024-09-21T10:14:44.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:44 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/893920425' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:44.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:44 smithi107 ceph-mon[105022]: pgmap v1082: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s; 2681/165882 objects degraded (1.616%); 2160/165882 objects misplaced (1.302%); 2.7 MiB/s, 44 objects/s recovering 2024-09-21T10:14:44.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:44 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/893920425' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:14:44.874 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 35 16 59169 59153 105.604 73.625 0.00450424 0.00946353 2024-09-21T10:14:44.953 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:44.979 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:44.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:44.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:45.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:45.123 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:45.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:45.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:45.225 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:45.245 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:14:45.245 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:14:45.245 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:14:45.246 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:14:45.246 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:14:45.246 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:14:45.246 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:14:45.246 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:14:45.246 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:14:45.246 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:14:45.246 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:14:45.246 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:14:45.246 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:14:45.247 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:14:45.247 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:14:45.247 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:14:45.255 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:45.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:45.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:45.356 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:45.387 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:45.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:45.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:45.489 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:45.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:14:45.511 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:45.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:45.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:45.516 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:45 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2681/165882 objects degraded (1.616%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:45.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:45 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2681/165882 objects degraded (1.616%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:45.612 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:45.652 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:45.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:45.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:45 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2681/165882 objects degraded (1.616%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:45.754 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:45.779 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:45.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:45.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:45.875 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 36 16 60302 60286 104.637 70.8125 0.00562341 0.00955108 2024-09-21T10:14:45.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:45.924 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:45.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:45.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:46.026 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:46.056 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:46.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:46.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:46.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:46.195 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:46.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:46.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:46.297 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:46.334 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:46.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:46.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:46.436 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:46.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:46.474 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:46.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:46.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:46.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:46 smithi107 ceph-mon[102221]: pgmap v1083: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 77 MiB/s wr, 1.24k op/s; 2681/168291 objects degraded (1.593%); 2160/168291 objects misplaced (1.283%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:14:46.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:46 smithi107 ceph-mon[105022]: pgmap v1083: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 77 MiB/s wr, 1.24k op/s; 2681/168291 objects degraded (1.593%); 2160/168291 objects misplaced (1.283%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:14:46.613 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:46.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:46.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:46.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:46 smithi129 ceph-mon[88471]: pgmap v1083: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 77 MiB/s wr, 1.24k op/s; 2681/168291 objects degraded (1.593%); 2160/168291 objects misplaced (1.283%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:14:46.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:46.750 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:46.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:46.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:46.851 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:46.875 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 37 16 61533 61517 103.889 76.9375 0.0165525 0.00962043 2024-09-21T10:14:46.879 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:46.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:46.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:46.980 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:47.004 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:47.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:47.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:47.106 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:47.135 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:47.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:47.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:47.237 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:47.278 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:47.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:47.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:47.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:47.414 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:47.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:47.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:47.516 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:47.544 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:47.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:47.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:47.646 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:47.677 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:47.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:47.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:47.779 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:47.800 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:47.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:47.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:47.875 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 38 16 62527 62511 102.79 62.125 0.00401637 0.00969286 2024-09-21T10:14:47.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:47.922 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:47.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:47.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:48.024 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:48.051 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:48.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:48.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:48.153 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:48.178 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:48.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:48.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:48.280 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:48.302 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:48.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:48.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:48.404 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:48.439 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:48.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:48.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:48.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:48.592 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:48.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:48.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:48 smithi129 ceph-mon[88471]: pgmap v1084: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s; 2681/178731 objects degraded (1.500%); 1965/178731 objects misplaced (1.099%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T10:14:48.694 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:48.733 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:48.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:48.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:48.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:48.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:48 smithi107 ceph-mon[102221]: pgmap v1084: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s; 2681/178731 objects degraded (1.500%); 1965/178731 objects misplaced (1.099%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T10:14:48.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:48 smithi107 ceph-mon[105022]: pgmap v1084: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s; 2681/178731 objects degraded (1.500%); 1965/178731 objects misplaced (1.099%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T10:14:48.875 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 39 16 63527 63511 101.757 62.5 0.00803075 0.00982108 2024-09-21T10:14:48.882 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:48.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:48.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:48.984 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:49.010 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:49.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:49.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:49.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:49.148 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:49.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:49.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:49.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:49.295 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:49.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:49.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:49.397 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:49.414 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:49.415 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:49.472 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:49.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:49.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:49.576 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:49.607 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:49.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:49.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:49.709 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:49.735 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:49.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:49.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:49.837 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:49.863 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:49.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:49.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:49.875 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:14:49.874190+0000 min lat: 0.00109013 max lat: 2.1623 avg lat: 0.0098813 2024-09-21T10:14:49.875 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:14:49.875 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 40 16 64762 64746 101.142 77.1875 0.00658541 0.0098813 2024-09-21T10:14:49.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:49.999 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:49.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:50.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:50.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:50.121 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:50.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:50.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:50.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:50.254 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:50.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:50.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:50.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:50.391 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:50.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:50.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:50.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:50.532 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:50.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:50.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:50.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:14:50.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:50 smithi129 ceph-mon[88471]: pgmap v1085: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s; 2681/184251 objects degraded (1.455%); 1965/184251 objects misplaced (1.066%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T10:14:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:50 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2681/184251 objects degraded (1.455%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:14:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:14:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:14:50.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:50.683 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:50.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:50.784 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:50.811 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:50.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:50.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:50.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:50 smithi107 ceph-mon[102221]: pgmap v1085: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s; 2681/184251 objects degraded (1.455%); 1965/184251 objects misplaced (1.066%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T10:14:50.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:50 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2681/184251 objects degraded (1.455%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:50.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:50 smithi107 ceph-mon[105022]: pgmap v1085: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s; 2681/184251 objects degraded (1.455%); 1965/184251 objects misplaced (1.066%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T10:14:50.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:50 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2681/184251 objects degraded (1.455%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:50.875 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:14:50.876 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:14:50.876 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 41 16 65787 65771 100.237 64.0625 0.00873449 0.00997016 2024-09-21T10:14:50.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:50.957 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:50.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:50.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:51.059 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:51.086 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:51.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:51.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:51.188 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:51.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:51.227 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:51.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:51.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:51.366 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:51.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:51.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:51.468 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:51.497 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:51.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:51.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:51.599 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:51.628 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:51.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:51.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:51.730 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:51.757 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:51.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:51.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:51.858 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:51.875 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 42 16 66960 66944 99.596 73.3125 0.021366 0.0100343 2024-09-21T10:14:51.887 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:51.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:51.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:51.945 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 10:14:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=cleanup t=2024-09-21T10:14:51.435784196Z level=info msg="Completed cleanup jobs" duration=13.392756ms 2024-09-21T10:14:52.041 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:14:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:14:51] "GET /metrics HTTP/1.1" 200 36823 "" "Prometheus/2.51.0" 2024-09-21T10:14:52.046 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:52.072 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:52.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:52.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:52.201 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:52.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:52.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:52.303 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:52.328 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:52.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:52.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:52.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:52.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:52.475 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:52.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:52.576 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:52.602 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:52.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:52.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:52.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:52 smithi107 ceph-mon[102221]: pgmap v1086: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 2681/188484 objects degraded (1.422%); 1965/188484 objects misplaced (1.043%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T10:14:52.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:52 smithi107 ceph-mon[105022]: pgmap v1086: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 2681/188484 objects degraded (1.422%); 1965/188484 objects misplaced (1.043%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T10:14:52.704 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:52.733 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:52.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:52.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:52.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:52.871 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:52.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:52.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:52.876 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 43 16 68184 68168 99.0588 76.5 0.0118701 0.0100887 2024-09-21T10:14:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:52 smithi129 ceph-mon[88471]: pgmap v1086: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s; 2681/188484 objects degraded (1.422%); 1965/188484 objects misplaced (1.043%); 2.1 MiB/s, 34 objects/s recovering 2024-09-21T10:14:52.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:53.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:14:53.030 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:53.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:53.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:53.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:53.172 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:53.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:53.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:53.273 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:53.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:53.318 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:53.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:53.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:53.470 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:53.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:53.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:53.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:53.613 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:53.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:53.714 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:53.755 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:53.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:53.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:53.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:53.876 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 44 16 69314 69298 98.4117 70.625 0.0048796 0.0101558 2024-09-21T10:14:53.904 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:53.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:53.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:54.005 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:54.033 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:54.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:54.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:54.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:54.168 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:54.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:54.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:54.270 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:54.297 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:54.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:54.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:54.399 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:54.429 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:54.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:54.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:54.475 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:14:54.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:54.571 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:54.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:54.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:54.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:54.706 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:54.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:54.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:54.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:54.844 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:54.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:54.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:54 smithi107 ceph-mon[102221]: pgmap v1087: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s; 2681/200733 objects degraded (1.336%); 1686/200733 objects misplaced (0.840%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:14:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:54 smithi107 ceph-mon[105022]: pgmap v1087: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s; 2681/200733 objects degraded (1.336%); 1686/200733 objects misplaced (0.840%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:14:54.876 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 45 15 70511 70496 97.8886 74.875 0.0159817 0.01021 2024-09-21T10:14:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:54 smithi129 ceph-mon[88471]: pgmap v1087: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s; 2681/200733 objects degraded (1.336%); 1686/200733 objects misplaced (0.840%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:14:54.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:55.019 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:55.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:55.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:55.121 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:55.162 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:55.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:55.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:55.263 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:55.291 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:55.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:55.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:55.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:55.431 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:55.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:55.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:55.533 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:55.562 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:55.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:55.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:55.663 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:55.691 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:55.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:55.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:55.793 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:55.820 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:55.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:55.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:55.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:55 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2681/200733 objects degraded (1.336%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:55.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:55 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2681/200733 objects degraded (1.336%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:55.876 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 46 16 71448 71432 97.0324 58.5 0.0345173 0.0102995 2024-09-21T10:14:55.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:55 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2681/200733 objects degraded (1.336%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:14:55.922 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:55.950 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:55.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:55.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:56.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:56.087 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:56.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:56.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:56.189 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:56.217 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:56.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:56.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:56.319 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:56.347 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:56.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:56.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:56.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:56.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:56.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:56.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:56.614 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:56.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:56.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:56.716 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:56.744 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:56.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:56.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:56.825 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:56 smithi107 ceph-mon[102221]: pgmap v1088: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 2681/202902 objects degraded (1.321%); 1686/202902 objects misplaced (0.831%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T10:14:56.825 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:56.826 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:56 smithi107 ceph-mon[105022]: pgmap v1088: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 2681/202902 objects degraded (1.321%); 1686/202902 objects misplaced (0.831%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T10:14:56.826 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:56.845 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:56.867 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:56.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:56.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:56.876 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 47 16 72391 72375 96.2218 58.9375 0.0529465 0.0103863 2024-09-21T10:14:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:56 smithi129 ceph-mon[88471]: pgmap v1088: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 2681/202902 objects degraded (1.321%); 1686/202902 objects misplaced (0.831%); 2.5 MiB/s, 39 objects/s recovering 2024-09-21T10:14:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:56.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:57.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:57.011 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:57.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:57.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:14:57.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:57.160 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:57.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:14:57.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:57.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:57.284 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:57.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:57.385 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:57.417 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:57.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:57.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:57.518 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:57.550 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:57.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:57.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:57.653 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:57.674 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:57.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:57.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:57.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:57.805 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:57.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:57.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:57.876 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 48 16 73127 73111 95.1756 46 0.00724003 0.010501 2024-09-21T10:14:57.907 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:57.935 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:57.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:57.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:58.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:58.077 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:58.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:58.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:14:58.179 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:58.212 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:58.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:58.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:58.314 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:58.335 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:58.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:58.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:58.437 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:58.467 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:58.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:58.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:58.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:58.603 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:58.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:58.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:58.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:14:58.721 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:58 smithi107 ceph-mon[105022]: pgmap v1089: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s; 2681/212190 objects degraded (1.263%); 1463/212190 objects misplaced (0.689%); 3.6 MiB/s, 58 objects/s recovering 2024-09-21T10:14:58.723 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:58 smithi107 ceph-mon[102221]: pgmap v1089: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s; 2681/212190 objects degraded (1.263%); 1463/212190 objects misplaced (0.689%); 3.6 MiB/s, 58 objects/s recovering 2024-09-21T10:14:58.741 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:58.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:58.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:14:58.843 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:58.869 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:58.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:58.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:58.876 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 49 16 74223 74207 94.6311 68.5 0.0142596 0.0105622 2024-09-21T10:14:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:58 smithi129 ceph-mon[88471]: pgmap v1089: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s; 2681/212190 objects degraded (1.263%); 1463/212190 objects misplaced (0.689%); 3.6 MiB/s, 58 objects/s recovering 2024-09-21T10:14:58.971 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:58.997 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:58.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:58.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:59.099 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:14:59.126 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:59.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:59.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:14:59.227 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:14:59.251 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:59.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:59.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:14:59.355 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:14:59.396 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:59.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:59.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:14:59.498 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:14:59.531 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:59.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:59.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:14:59.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:14:59 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:14:59.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:14:59 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:14:59.633 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:59.660 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:59.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:59.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:59.729 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:14:59 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:14:59.762 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:14:59.796 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:14:59.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:59.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:14:59.877 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 50 16 75232 75216 93.9997 63.0625 0.0201809 0.0106284 2024-09-21T10:14:59.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:14:59.938 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:14:59.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:14:59.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:00.040 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:00.077 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:00.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:00.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:00.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:00.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:00.215 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:00.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:00.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:00.357 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:00.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:00.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:00.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:15:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:15:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:15:00.460 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:00.464 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.5 primary-affinity to 0 (802) 2024-09-21T10:15:00.489 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:00.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:00.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:00.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:00.632 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:00.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:00.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:00.779 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:00.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:00.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:00.818 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:00 smithi107 ceph-mon[102221]: pgmap v1090: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 2681/216873 objects degraded (1.236%); 1463/216873 objects misplaced (0.675%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:15:00.818 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:00 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2681/216873 objects degraded (1.236%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:15:00.818 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:00 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2113588762' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.0} : dispatch 2024-09-21T10:15:00.818 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:00 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.0} : dispatch 2024-09-21T10:15:00.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:00 smithi107 ceph-mon[105022]: pgmap v1090: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 2681/216873 objects degraded (1.236%); 1463/216873 objects misplaced (0.675%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:15:00.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:00 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2681/216873 objects degraded (1.236%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:15:00.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:00 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2113588762' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.0} : dispatch 2024-09-21T10:15:00.819 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:00 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.0} : dispatch 2024-09-21T10:15:00.824 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:00 smithi129 ceph-mon[88471]: pgmap v1090: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s; 2681/216873 objects degraded (1.236%); 1463/216873 objects misplaced (0.675%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:15:00.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:00 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2681/216873 objects degraded (1.236%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:15:00.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:00 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2113588762' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.0} : dispatch 2024-09-21T10:15:00.825 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:00 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.0} : dispatch 2024-09-21T10:15:00.877 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 51 16 76319 76303 93.4887 67.9375 0.00627456 0.0106918 2024-09-21T10:15:00.881 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:00.908 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:00.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:00.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:01.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:01.054 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:01.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:01.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:01.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:01.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:01.194 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:01.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:01.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:01.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:01.338 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:01.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:01.439 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:01.472 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:01.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:01.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:01.520 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:01 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-09-21T10:15:01.521 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:01 smithi107 ceph-mon[102221]: osdmap e307: 8 total, 8 up, 8 in 2024-09-21T10:15:01.521 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:01 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-09-21T10:15:01.521 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:01 smithi107 ceph-mon[105022]: osdmap e307: 8 total, 8 up, 8 in 2024-09-21T10:15:01.574 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:01.595 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:01.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:01.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:01.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:01.734 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:01.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:01.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:01.778 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:15:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:15:01] "GET /metrics HTTP/1.1" 200 36820 "" "Prometheus/2.51.0" 2024-09-21T10:15:01.836 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:01.865 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:01.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:01.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:01.877 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 52 16 77291 77275 92.8591 60.75 0.0152004 0.0107633 2024-09-21T10:15:01.882 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:01 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-09-21T10:15:01.882 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:01 smithi129 ceph-mon[88471]: osdmap e307: 8 total, 8 up, 8 in 2024-09-21T10:15:01.967 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:01.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:01.999 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:02.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:02.100 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:02.128 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:02.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:02.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:02.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:02.302 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:02.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:02.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:02.404 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:02.425 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:02.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:02.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:02.500 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:15:02.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:02.549 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:02.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:02.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:02.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:02.685 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:02.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:02.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:02.787 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:02.792 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:02 smithi129 ceph-mon[88471]: pgmap v1092: 109 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 3.4 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 2681/220458 objects degraded (1.216%); 1463/220458 objects misplaced (0.664%); 3.1 MiB/s, 50 objects/s recovering 2024-09-21T10:15:02.792 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:02 smithi129 ceph-mon[88471]: osdmap e308: 8 total, 8 up, 8 in 2024-09-21T10:15:02.792 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:02 smithi129 ceph-mon[88471]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:15:02.792 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:02 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1342878202' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:02.792 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:02 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/783420231' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:15:02.792 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:02 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:15:02.811 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:15:02.811 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:15:02.811 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:15:02.811 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:15:02.811 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:15:02.811 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:15:02.811 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:15:02.811 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:15:02.811 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:15:02.812 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:15:02.812 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:15:02.812 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:15:02.812 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:15:02.812 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:15:02.812 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:15:02.812 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:15:02.813 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:02 smithi107 ceph-mon[102221]: pgmap v1092: 109 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 3.4 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 2681/220458 objects degraded (1.216%); 1463/220458 objects misplaced (0.664%); 3.1 MiB/s, 50 objects/s recovering 2024-09-21T10:15:02.813 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:02 smithi107 ceph-mon[102221]: osdmap e308: 8 total, 8 up, 8 in 2024-09-21T10:15:02.813 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:02 smithi107 ceph-mon[102221]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:15:02.813 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:02 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1342878202' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:02.813 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:02 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/783420231' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:15:02.813 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:02 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:15:02.813 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:02 smithi107 ceph-mon[105022]: pgmap v1092: 109 pgs: 1 peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 3.4 KiB/s rd, 67 MiB/s wr, 1.07k op/s; 2681/220458 objects degraded (1.216%); 1463/220458 objects misplaced (0.664%); 3.1 MiB/s, 50 objects/s recovering 2024-09-21T10:15:02.813 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:02 smithi107 ceph-mon[105022]: osdmap e308: 8 total, 8 up, 8 in 2024-09-21T10:15:02.813 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:02 smithi107 ceph-mon[105022]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:15:02.813 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:02 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1342878202' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:02.814 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:02 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/783420231' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:15:02.814 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:02 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:15:02.820 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:02.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:02.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:02.877 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 53 16 77897 77881 91.8217 37.875 0.0187073 0.0108853 2024-09-21T10:15:02.921 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:02.967 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:02.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:02.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:03.068 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:03.089 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:03.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:03.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:03.190 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:03.217 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:03.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:03.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:03.318 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:03.352 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:03.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:03.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:03.454 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:03.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:03.482 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:03.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:03.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:03.617 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:03.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:03.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:03.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:03.754 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:03.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:03.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:03.765 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:03 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:15:03.765 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:03 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:15:03.766 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:03 smithi129 ceph-mon[88471]: osdmap e309: 8 total, 8 up, 8 in 2024-09-21T10:15:03.766 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:03 smithi129 ceph-mon[88471]: 1.12 scrub starts 2024-09-21T10:15:03.766 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:03 smithi129 ceph-mon[88471]: 1.12 scrub ok 2024-09-21T10:15:03.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:03.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:03 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:15:03.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:03 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:15:03.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:03 smithi107 ceph-mon[102221]: osdmap e309: 8 total, 8 up, 8 in 2024-09-21T10:15:03.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:03 smithi107 ceph-mon[102221]: 1.12 scrub starts 2024-09-21T10:15:03.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:03 smithi107 ceph-mon[102221]: 1.12 scrub ok 2024-09-21T10:15:03.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:03 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:15:03.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:03 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:15:03.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:03 smithi107 ceph-mon[105022]: osdmap e309: 8 total, 8 up, 8 in 2024-09-21T10:15:03.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:03 smithi107 ceph-mon[105022]: 1.12 scrub starts 2024-09-21T10:15:03.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:03 smithi107 ceph-mon[105022]: 1.12 scrub ok 2024-09-21T10:15:03.879 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 54 16 78937 78921 91.322 65 0.0188142 0.010945 2024-09-21T10:15:03.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:03.892 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:03.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:03.993 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:04.021 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:04.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:04.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:04.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:04.158 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:04.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:04.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:04.260 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:04.289 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:04.289 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:04.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:04.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:04.391 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:04.420 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:04.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:04.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:04.560 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:04.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:04.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:04.661 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:04.702 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:04.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:04.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:04.804 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:04.836 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:04.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:04.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:04.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:04 smithi107 ceph-mon[105022]: pgmap v1095: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 106 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 5.7 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 2681/231333 objects degraded (1.159%); 1123/231333 objects misplaced (0.485%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:15:04.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:04 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:04.849 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:04 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:04.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:04 smithi107 ceph-mon[102221]: pgmap v1095: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 106 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 5.7 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 2681/231333 objects degraded (1.159%); 1123/231333 objects misplaced (0.485%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:15:04.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:04 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:04.849 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:04 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:04.879 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 55 16 80024 80008 90.8968 67.9375 0.0151375 0.0109954 2024-09-21T10:15:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:04 smithi129 ceph-mon[88471]: pgmap v1095: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 106 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 5.7 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 2681/231333 objects degraded (1.159%); 1123/231333 objects misplaced (0.485%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:15:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:04 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:04 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:04.938 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:04.964 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:04.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:04.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:05.066 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:05.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:05.103 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:05.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:05.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:05.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:05.245 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:05.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:05.346 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:05.374 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:05.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:05.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:05.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:05.514 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:05.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:05.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:05.615 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:05.656 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:05.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:05.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:05.733 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:05 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2681/231333 objects degraded (1.159%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:15:05.733 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:05 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2681/231333 objects degraded (1.159%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:15:05.758 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:05.785 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:05.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:05.879 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 56 16 81082 81066 90.4543 66.125 0.00908266 0.011049 2024-09-21T10:15:05.885 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:05.913 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:05.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:05.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:05 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2681/231333 objects degraded (1.159%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:15:06.015 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:06.044 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:06.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:06.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:06.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:06.186 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:06.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:06.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:06.287 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:06.316 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:06.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:06.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:06.418 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:06.446 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:06.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:06.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:06.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:06.590 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:06.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:06.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:06.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:06.743 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:06.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:06.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:06.845 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:06.857 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:06 smithi107 ceph-mon[102221]: pgmap v1096: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 106 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 5.7 KiB/s rd, 56 MiB/s wr, 902 op/s; 2681/233064 objects degraded (1.150%); 1123/233064 objects misplaced (0.482%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:15:06.857 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:06 smithi107 ceph-mon[105022]: pgmap v1096: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 106 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 5.7 KiB/s rd, 56 MiB/s wr, 902 op/s; 2681/233064 objects degraded (1.150%); 1123/233064 objects misplaced (0.482%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:15:06.870 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:06.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:06.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:06.880 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 57 15 82144 82129 90.0314 66.4375 0.0208546 0.0111012 2024-09-21T10:15:06.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:15:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:06 smithi129 ceph-mon[88471]: pgmap v1096: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 106 active+clean; 4.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 5.7 KiB/s rd, 56 MiB/s wr, 902 op/s; 2681/233064 objects degraded (1.150%); 1123/233064 objects misplaced (0.482%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:15:06.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:07.021 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:07.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:07.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:07.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:07.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:07.187 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:07.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:07.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:07.335 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:07.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:07.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:07.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:07.479 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:07.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:07.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:07.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:07.623 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:07.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:07.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:07.662 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:07 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:07.663 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:07 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:07.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:07.757 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:07.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:07.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:07.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:15:07.859 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:07.883 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 58 16 83033 83017 89.432 55.5 0.0137656 0.0111763 2024-09-21T10:15:07.887 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:07.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:07.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:07 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:07.989 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:08.015 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:08.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:08.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:08.042 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:15:08.042 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:15:08.042 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-09-21T10:15:08.042 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-09-21T10:15:08.042 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:15:08.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:08.156 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:08.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:08.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:08.260 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:08.286 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:08.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:08.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:08.387 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:08.420 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:08.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:08.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:08.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:08.561 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:08.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:08.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:08.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:08 smithi107 ceph-mon[105022]: pgmap v1097: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 2681/241290 objects degraded (1.111%); 772/241290 objects misplaced (0.320%); 6.3 MiB/s, 100 objects/s recovering 2024-09-21T10:15:08.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:08 smithi107 ceph-mon[105022]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:15:08.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:08 smithi107 ceph-mon[102221]: pgmap v1097: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 2681/241290 objects degraded (1.111%); 772/241290 objects misplaced (0.320%); 6.3 MiB/s, 100 objects/s recovering 2024-09-21T10:15:08.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:08 smithi107 ceph-mon[102221]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:15:08.663 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:08.690 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:08.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:08.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:08.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:15:08.789 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:08.792 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:08.794 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:08.818 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:08.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:08.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:08.883 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 59 16 84013 83997 88.9543 61.25 0.0196862 0.0112354 2024-09-21T10:15:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:08 smithi129 ceph-mon[88471]: pgmap v1097: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 4.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 2681/241290 objects degraded (1.111%); 772/241290 objects misplaced (0.320%); 6.3 MiB/s, 100 objects/s recovering 2024-09-21T10:15:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:08 smithi129 ceph-mon[88471]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:15:08.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:08.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:08.972 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:08.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:09.027 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:15:09.027 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:15:09.027 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-21T10:15:09.027 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:15:09.027 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:15:09.027 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-09-21T10:15:09.072 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:09.102 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:09.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:09.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:09.204 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:09.237 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:09.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:09.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:09.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:09.376 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:15:09.378 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:09.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:09.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:09.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:09.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:09.532 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:09.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:09.633 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:09.660 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:09.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:09.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:09.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:09.804 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:09.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:09.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:09.837 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:09 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2681/241290 objects degraded (1.111%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:15:09.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:09 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2681/241290 objects degraded (1.111%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:15:09.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:09 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2681/241290 objects degraded (1.111%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:15:09.883 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:15:09.882009+0000 min lat: 0.00109013 max lat: 2.1623 avg lat: 0.0112988 2024-09-21T10:15:09.883 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:15:09.883 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 60 16 84966 84950 88.4645 59.5625 0.0120349 0.0112988 2024-09-21T10:15:09.907 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:09.947 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:09.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:09.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:10.049 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:10.081 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:10.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:10.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:10.183 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:10.220 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:10.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:10.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:10.322 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:10.358 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:10.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:10.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:10.460 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:10.493 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:10.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:10.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:10.595 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:10.616 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:15:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:15:10.215Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:15:10.622 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:10.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:10.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:10.724 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:10.762 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:10.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:10.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:10.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:10 smithi107 ceph-mon[105022]: pgmap v1098: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 2681/246072 objects degraded (1.090%); 772/246072 objects misplaced (0.314%); 5.4 MiB/s, 86 objects/s recovering 2024-09-21T10:15:10.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:10 smithi107 ceph-mon[102221]: pgmap v1098: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 2681/246072 objects degraded (1.090%); 772/246072 objects misplaced (0.314%); 5.4 MiB/s, 86 objects/s recovering 2024-09-21T10:15:10.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:10.870 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:10 smithi129 ceph-mon[88471]: pgmap v1098: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 2681/246072 objects degraded (1.090%); 772/246072 objects misplaced (0.314%); 5.4 MiB/s, 86 objects/s recovering 2024-09-21T10:15:10.883 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 61 16 86046 86030 88.1204 67.5 0.00932129 0.0113425 2024-09-21T10:15:10.909 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:10.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:10.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:11.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:11.061 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:11.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:11.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:11.163 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:11.208 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:11.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:11.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:11.303 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:15:11.303 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "" 2024-09-21T10:15:11.303 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:15:11.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:11.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:11.357 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:11.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:11.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:11.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:11.519 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:11.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:11.620 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:11.654 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:11.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:11.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:11.715 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:11.757 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:11.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:11.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:11.785 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:11.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:11.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:11.883 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 62 16 87040 87024 87.7012 62.125 0.011601 0.011397 2024-09-21T10:15:11.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:11.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:11.935 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:11.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:12.036 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:12.066 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:15:12.067 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:15:12.067 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:15:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:15:11] "GET /metrics HTTP/1.1" 200 36820 "" "Prometheus/2.51.0" 2024-09-21T10:15:12.067 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:12.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:12.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:12.099 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:15:12.099 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:15:12.146 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:15:12.147 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:15:12.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:12.186 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:15:12.186 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:15:12.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:12.221 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:12.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:12.226 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:15:12.226 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:15:12.264 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:15:12.264 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:15:12.298 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:15:12.298 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:15:12.322 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:12.336 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:15:12.336 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:15:12.362 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:12.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:12.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:12.466 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:12.497 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:12.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:12.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:12.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:12.638 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:12.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:12.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:12.740 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:12.768 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:12.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:12.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:12.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:12.883 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 63 16 87973 87957 87.2345 58.3125 0.00534534 0.0114569 2024-09-21T10:15:12.913 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:12.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:12.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:12.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:12 smithi129 ceph-mon[88471]: pgmap v1099: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 62 MiB/s wr, 990 op/s; 2681/249885 objects degraded (1.073%); 772/249885 objects misplaced (0.309%); 4.4 MiB/s, 69 objects/s recovering 2024-09-21T10:15:12.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:12 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/25624868' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:13.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:13.053 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:13.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:13.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:13.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:12 smithi107 ceph-mon[102221]: pgmap v1099: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 62 MiB/s wr, 990 op/s; 2681/249885 objects degraded (1.073%); 772/249885 objects misplaced (0.309%); 4.4 MiB/s, 69 objects/s recovering 2024-09-21T10:15:13.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:12 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/25624868' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:13.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:12 smithi107 ceph-mon[105022]: pgmap v1099: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 62 MiB/s wr, 990 op/s; 2681/249885 objects degraded (1.073%); 772/249885 objects misplaced (0.309%); 4.4 MiB/s, 69 objects/s recovering 2024-09-21T10:15:13.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:12 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/25624868' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:13.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:13.189 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:13.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:13.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:13.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:13.331 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:13.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:13.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:13.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:13.473 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:13.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:13.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:13.575 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:13.602 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:13.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:13.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:13.656 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:15:13.703 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:13.730 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:13.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:13.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:13.831 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:13.869 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:13.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:13.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:13.884 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 64 16 89022 89006 86.8959 65.5625 0.00790152 0.0115029 2024-09-21T10:15:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:13 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2363339430' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:15:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:13 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:15:13.970 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:14.000 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:14.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:14.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:14.101 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:14.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:13 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2363339430' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:15:14.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:13 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:15:14.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:13 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2363339430' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:15:14.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:13 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:15:14.128 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:14.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:14.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:14.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:14.265 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:14.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:14.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:14.367 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:14.394 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:14.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:14.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:14.417 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:14.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:14.547 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:14.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:14.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:14.649 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:14.674 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:14.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:14.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:14.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:14.814 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:14.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:14.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:14.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:14 smithi107 ceph-mon[105022]: pgmap v1100: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 2681/261015 objects degraded (1.027%); 440/261015 objects misplaced (0.169%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T10:15:14.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:14 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:15:14.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:14 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:15:14.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:14 smithi107 ceph-mon[105022]: osdmap e310: 8 total, 8 up, 8 in 2024-09-21T10:15:14.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:14 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2681/261015 objects degraded (1.027%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:15:14.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:14 smithi107 ceph-mon[102221]: pgmap v1100: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 2681/261015 objects degraded (1.027%); 440/261015 objects misplaced (0.169%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T10:15:14.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:14 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:15:14.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:14 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:15:14.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:14 smithi107 ceph-mon[102221]: osdmap e310: 8 total, 8 up, 8 in 2024-09-21T10:15:14.831 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:14 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2681/261015 objects degraded (1.027%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:15:14.884 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 65 16 90026 90010 86.5243 62.75 0.0174848 0.0115516 2024-09-21T10:15:14.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:14 smithi129 ceph-mon[88471]: pgmap v1100: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 2681/261015 objects degraded (1.027%); 440/261015 objects misplaced (0.169%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T10:15:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:14 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:15:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:14 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:15:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:14 smithi129 ceph-mon[88471]: osdmap e310: 8 total, 8 up, 8 in 2024-09-21T10:15:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:14 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2681/261015 objects degraded (1.027%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:15:14.952 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:14.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:14.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:15.054 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:15.081 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:15.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:15.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:15.183 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:15.209 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:15.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:15.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:15.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:15.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:15.356 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:15.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:15.457 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:15.484 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:15.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:15.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:15.586 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:15.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:15.617 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:15.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:15.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:15.762 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:15.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:15.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:15.863 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:15.885 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 66 16 91047 91031 86.179 63.8125 0.0187474 0.0115979 2024-09-21T10:15:15.899 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:15.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:15.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:16.000 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:16.027 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:16.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:16.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:16.129 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:16.157 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:16.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:15:16.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:16.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:16.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:16.297 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:16.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:16.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:16.399 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:16.428 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:16.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:16.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:16.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:16.571 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:16.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:16.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:16.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:16.719 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:16.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:16.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:16.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:16.857 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:16.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:16.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:16 smithi107 ceph-mon[102221]: pgmap v1102: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 994 op/s; 2681/262920 objects degraded (1.020%); 440/262920 objects misplaced (0.167%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T10:15:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:16 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:16 smithi107 ceph-mon[105022]: pgmap v1102: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 994 op/s; 2681/262920 objects degraded (1.020%); 440/262920 objects misplaced (0.167%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T10:15:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:16 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:16.867 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:15:16.885 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 67 16 92046 92030 85.824 62.4375 0.029657 0.0116459 2024-09-21T10:15:16.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:16 smithi129 ceph-mon[88471]: pgmap v1102: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 994 op/s; 2681/262920 objects degraded (1.020%); 440/262920 objects misplaced (0.167%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T10:15:16.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:16 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:16.958 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:16.985 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:16.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:16.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:17.087 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:17.113 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:17.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:17.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:17.159 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:15:17.160 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:15:17.160 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-21T10:15:17.160 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:15:17.160 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:15:17.160 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-09-21T10:15:17.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:17.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:17.267 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:17.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:17.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:17.417 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:17.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:17.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:17.505 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:15:17.519 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:17.554 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:17.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:17.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:17.657 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:17.681 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:17.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:17.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:17.783 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:17.820 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:17.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:17.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:17.885 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 68 16 92932 92916 85.3762 55.375 0.00413123 0.0116923 2024-09-21T10:15:17.922 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:17.944 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:17.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:17.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:18.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:18.067 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:18.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:18.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:18.169 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:18.190 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:18.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:18.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:18.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:18.313 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:18.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:18.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:18.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:18.436 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:18.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:18.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:18.538 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:18.559 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:18.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:18.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:18.661 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:18.682 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:18.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:18.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:18.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:18.824 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:18.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:18.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:18.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:18 smithi107 ceph-mon[102221]: pgmap v1103: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 990 op/s; 2681/271020 objects degraded (0.989%); 5/271020 objects misplaced (0.002%); 4.8 MiB/s, 76 objects/s recovering 2024-09-21T10:15:18.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:18 smithi107 ceph-mon[102221]: osdmap e311: 8 total, 8 up, 8 in 2024-09-21T10:15:18.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:18 smithi107 ceph-mon[105022]: pgmap v1103: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 990 op/s; 2681/271020 objects degraded (0.989%); 5/271020 objects misplaced (0.002%); 4.8 MiB/s, 76 objects/s recovering 2024-09-21T10:15:18.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:18 smithi107 ceph-mon[105022]: osdmap e311: 8 total, 8 up, 8 in 2024-09-21T10:15:18.885 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 69 16 93125 93109 84.3139 12.0625 0.0304739 0.011855 2024-09-21T10:15:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:18 smithi129 ceph-mon[88471]: pgmap v1103: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 990 op/s; 2681/271020 objects degraded (0.989%); 5/271020 objects misplaced (0.002%); 4.8 MiB/s, 76 objects/s recovering 2024-09-21T10:15:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:18 smithi129 ceph-mon[88471]: osdmap e311: 8 total, 8 up, 8 in 2024-09-21T10:15:18.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:18.969 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:18.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:18.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:19.071 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:19.099 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:19.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:19.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:19.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:19.239 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:19.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:19.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:19.341 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:19.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:19.383 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:19.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:19.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:19.519 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:19.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:19.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:19.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:19.658 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:19.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:19.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:19.760 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:19.788 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:19.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:19.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:19.885 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 70 16 94124 94108 84.0014 62.4375 0.00426813 0.0118981 2024-09-21T10:15:19.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:19 smithi129 ceph-mon[88471]: osdmap e312: 8 total, 8 up, 8 in 2024-09-21T10:15:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:19 smithi129 ceph-mon[88471]: Health check update: Degraded data redundancy: 2681/271020 objects degraded (0.989%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:15:19.921 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:19.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:19.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:20.023 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:20.055 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:20.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:20.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:20.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:19 smithi107 ceph-mon[102221]: osdmap e312: 8 total, 8 up, 8 in 2024-09-21T10:15:20.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:19 smithi107 ceph-mon[102221]: Health check update: Degraded data redundancy: 2681/271020 objects degraded (0.989%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:15:20.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:19 smithi107 ceph-mon[105022]: osdmap e312: 8 total, 8 up, 8 in 2024-09-21T10:15:20.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:19 smithi107 ceph-mon[105022]: Health check update: Degraded data redundancy: 2681/271020 objects degraded (0.989%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:15:20.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:20.187 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:20.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:20.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:20.288 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:20.318 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:20.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:20.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:20.420 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:20.461 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:20.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:20.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:20.487 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:15:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:15:20.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:15:20.563 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:20.596 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:20.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:20.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:20.698 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:20.728 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:20.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:20.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:20.816 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:20 smithi129 ceph-mon[88471]: pgmap v1106: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 52 MiB/s wr, 827 op/s; 2681/275925 objects degraded (0.972%); 5/275925 objects misplaced (0.002%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:15:20.830 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:20.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:20.862 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:20.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:20.885 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 71 16 95098 95082 83.6757 60.875 0.00557242 0.0119445 2024-09-21T10:15:20.963 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:20.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:20.989 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:20.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:21.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:21.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:20 smithi107 ceph-mon[102221]: pgmap v1106: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 52 MiB/s wr, 827 op/s; 2681/275925 objects degraded (0.972%); 5/275925 objects misplaced (0.002%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:15:21.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:20 smithi107 ceph-mon[105022]: pgmap v1106: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.6 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 52 MiB/s wr, 827 op/s; 2681/275925 objects degraded (0.972%); 5/275925 objects misplaced (0.002%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:15:21.127 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:21.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:21.228 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:21.262 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:21.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:21.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:21.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:21.408 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:21.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:21.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:21.511 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:21.542 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:21.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:21.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:21.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:21.685 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:21.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:21.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:21.786 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:21.799 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:15:21.799 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "" 2024-09-21T10:15:21.799 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:15:21.816 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:21.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:21.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:21.886 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 72 16 96078 96062 83.3642 61.25 0.0120836 0.0119904 2024-09-21T10:15:21.918 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:21.940 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:21.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:21.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:22.041 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:22.077 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:22.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:22.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:22.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:15:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:15:21] "GET /metrics HTTP/1.1" 200 36820 "" "Prometheus/2.51.0" 2024-09-21T10:15:22.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:22.219 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:22.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:22.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:22.321 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:22.348 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:22.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:22.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:22.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:22.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:22.491 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:22.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:22.592 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:22.620 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:22.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:22.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:22.742 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:22.768 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:22.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:22.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:22.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:22 smithi107 ceph-mon[102221]: pgmap v1107: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 48 MiB/s wr, 774 op/s; 2681/278943 objects degraded (0.961%); 5/278943 objects misplaced (0.002%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:15:22.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:22 smithi107 ceph-mon[105022]: pgmap v1107: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 48 MiB/s wr, 774 op/s; 2681/278943 objects degraded (0.961%); 5/278943 objects misplaced (0.002%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:15:22.870 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:22.886 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 73 16 96949 96933 82.9679 54.4375 0.0129567 0.0120472 2024-09-21T10:15:22.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:22.905 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:22.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:22 smithi129 ceph-mon[88471]: pgmap v1107: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 107 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 48 MiB/s wr, 774 op/s; 2681/278943 objects degraded (0.961%); 5/278943 objects misplaced (0.002%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:15:23.007 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:23.030 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:23.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:23.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:23.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:23.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:23.164 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:23.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:23.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:23.304 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:23.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:23.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:23.406 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:23.433 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:23.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:23.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:23.534 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:23.555 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:23.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:23.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:23.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:23.692 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:23.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:23.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:23.794 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:23.823 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:23.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:23.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:23.834 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:15:23.834 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:15:23.886 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 74 16 97827 97811 82.5883 54.875 0.0180082 0.0121032 2024-09-21T10:15:23.924 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:23.956 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:23.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:23.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:24.022 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:23 smithi107 ceph-mon[102221]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2681/278943 objects degraded (0.961%), 1 pg degraded) 2024-09-21T10:15:24.023 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:23 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:15:24.023 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:23 smithi107 ceph-mon[105022]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2681/278943 objects degraded (0.961%), 1 pg degraded) 2024-09-21T10:15:24.023 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:23 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:15:24.057 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:24.082 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:24.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:24.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:24.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:23 smithi129 ceph-mon[88471]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2681/278943 objects degraded (0.961%), 1 pg degraded) 2024-09-21T10:15:24.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:23 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:15:24.183 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:24.203 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:24.205 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:24.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:24.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:24.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:24.356 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:24.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:24.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:24.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:24.494 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:24.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:24.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:24.595 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:24.625 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:24.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:24.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:24.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:24.766 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:24.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:24.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:24.868 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:24.886 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 75 16 98683 98667 82.2005 53.5 0.0134547 0.0121595 2024-09-21T10:15:24.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:24.896 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:24.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:24.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:25.002 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:24 smithi107 ceph-mon[102221]: pgmap v1108: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 5.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 2420/288117 objects misplaced (0.840%); 5.5 MiB/s, 87 objects/s recovering 2024-09-21T10:15:25.002 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:24 smithi107 ceph-mon[105022]: pgmap v1108: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 5.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 2420/288117 objects misplaced (0.840%); 5.5 MiB/s, 87 objects/s recovering 2024-09-21T10:15:25.032 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:25.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:25.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:25.133 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:25.159 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:25.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:25.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:24 smithi129 ceph-mon[88471]: pgmap v1108: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 5.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 2420/288117 objects misplaced (0.840%); 5.5 MiB/s, 87 objects/s recovering 2024-09-21T10:15:25.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:25.299 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:25.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:25.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:25.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:25.434 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:25.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:25.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:25.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:25.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:25.566 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:25.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:25.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:25.693 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:25.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:25.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:25.795 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:25.832 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:25.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:25.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:25.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:15:25.889 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 76 16 99421 99405 81.7229 46.125 0.0198198 0.0122298 2024-09-21T10:15:25.891 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:25.934 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:25.958 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:25.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:25.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:26.059 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:26.085 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:26.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:26.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:26.187 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:26.211 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:26.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:26.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:26.312 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:26.339 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:26.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:26.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:26.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:26.492 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:26.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:26.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:26.597 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:26.624 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:26.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:26.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:26.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:26.766 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:26.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:26.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:26.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:26.889 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 77 15 100235 100220 81.3232 50.9375 0.0087824 0.0122915 2024-09-21T10:15:26.905 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:26.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:26.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:27.007 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:27.033 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:27.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:27.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:27.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:26 smithi107 ceph-mon[102221]: pgmap v1109: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 5.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 49 MiB/s wr, 780 op/s; 2420/289767 objects misplaced (0.835%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T10:15:27.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:27.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:27.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:26 smithi107 ceph-mon[105022]: pgmap v1109: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 5.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 49 MiB/s wr, 780 op/s; 2420/289767 objects misplaced (0.835%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T10:15:27.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:27.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:27.135 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:27.162 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:27.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:27.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:26 smithi129 ceph-mon[88471]: pgmap v1109: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 5.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 49 MiB/s wr, 780 op/s; 2420/289767 objects misplaced (0.835%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T10:15:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:27.265 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:27.294 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:27.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:27.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:27.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:27.436 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:27.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:27.533 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:15:27.534 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:15:27.534 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_7 2024-09-21T10:15:27.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:15:27.537 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:27.563 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:27.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:27.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:27.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:27.709 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:27.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:27.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:27.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:27.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:27.876 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:27.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:27.889 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 78 16 101091 101075 80.965 53.4375 0.0298249 0.0123459 2024-09-21T10:15:27.977 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:28.002 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:28.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:28.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:28.151 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:28.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:28.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:28.253 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:28.280 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:28.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:28.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:28.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:28.441 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:28.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:28.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:28.543 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:28.568 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:28.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:28.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:28.670 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:28.696 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:28.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:28.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:28.798 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:28.825 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:28.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:28.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:28.890 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 79 16 101992 101976 80.6529 56.3125 0.0192189 0.0123933 2024-09-21T10:15:28.926 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:28.938 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:28 smithi107 ceph-mon[102221]: pgmap v1110: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 55 MiB/s wr, 886 op/s; 2236/296847 objects misplaced (0.753%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:15:28.939 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:28 smithi107 ceph-mon[105022]: pgmap v1110: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 55 MiB/s wr, 886 op/s; 2236/296847 objects misplaced (0.753%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:15:28.953 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:28.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:29.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:29.090 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:29.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:29.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:29.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:28 smithi129 ceph-mon[88471]: pgmap v1110: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 55 MiB/s wr, 886 op/s; 2236/296847 objects misplaced (0.753%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:15:29.191 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:29.220 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:29.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:29.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:29.322 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:29.347 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:29.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:29.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:29.449 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:29.477 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:29.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:29.579 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:29.606 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:29.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:29.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:29.748 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:29.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:29.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:29.850 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:29.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:29.881 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:29.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:29.882 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:29.890 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:15:29.889443+0000 min lat: 0.00109013 max lat: 2.1623 avg lat: 0.012436 2024-09-21T10:15:29.890 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:15:29.890 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 80 16 102926 102910 80.3739 58.375 0.0109151 0.012436 2024-09-21T10:15:29.982 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:30.010 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:30.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:30.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:30.111 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:30.145 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:30.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:30.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:30.247 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:30.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:30.282 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:30.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:30.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:30.416 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:30.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:30.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:30.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:30.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:30.556 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:30.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:30.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:15:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:15:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:15:30.692 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:30.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:30.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:30.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:30.830 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:30.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:30.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:30.890 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 81 16 103825 103809 80.0752 56.1875 0.0128294 0.0124822 2024-09-21T10:15:30.932 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:30.956 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:30.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:30.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:30.963 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:30 smithi107 ceph-mon[102221]: pgmap v1111: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 48 MiB/s wr, 772 op/s; 2236/300750 objects misplaced (0.743%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T10:15:30.963 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:30 smithi107 ceph-mon[105022]: pgmap v1111: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 48 MiB/s wr, 772 op/s; 2236/300750 objects misplaced (0.743%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T10:15:31.058 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:31.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:31.082 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:31.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:31.122 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:31.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:30 smithi129 ceph-mon[88471]: pgmap v1111: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 48 MiB/s wr, 772 op/s; 2236/300750 objects misplaced (0.743%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T10:15:31.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:31.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:31.238 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:31.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:31.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:31.406 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:31.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:31.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:31.507 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:31.535 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:31.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:31.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:31.636 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:31.664 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:31.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:31.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:31.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:31.811 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:31.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:31.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:31.891 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 82 16 104589 104573 79.6811 47.75 0.0161586 0.012544 2024-09-21T10:15:31.914 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:31.938 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:31.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:31.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:32.040 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:32.047 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:15:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:15:31] "GET /metrics HTTP/1.1" 200 36816 "" "Prometheus/2.51.0" 2024-09-21T10:15:32.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:32.078 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:32.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:32.167 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:15:32.179 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:32.204 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:32.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:32.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:32.302 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:32 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:32.302 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:32 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/394609812' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:15:32.302 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:32 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:15:32.302 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:32 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:32.302 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:32 smithi107 ceph-mon[102221]: pgmap v1112: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.2 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 49 MiB/s wr, 780 op/s; 2236/304023 objects misplaced (0.735%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T10:15:32.302 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:32 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:15:32.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:32 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:15:32.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:32 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:32.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:32 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1196226138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:32.303 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:32 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:32.303 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:32 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/394609812' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:15:32.303 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:32 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:15:32.303 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:32 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:32.303 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:32 smithi107 ceph-mon[105022]: pgmap v1112: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.2 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 49 MiB/s wr, 780 op/s; 2236/304023 objects misplaced (0.735%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T10:15:32.303 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:32 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:15:32.303 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:32 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:15:32.303 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:32 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:32.304 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:32 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1196226138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:32.305 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:32.331 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:32.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:32.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:32.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:32 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:32.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:32 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/394609812' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:15:32.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:32 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:15:32.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:32 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:32.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:32 smithi129 ceph-mon[88471]: pgmap v1112: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.2 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 49 MiB/s wr, 780 op/s; 2236/304023 objects misplaced (0.735%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T10:15:32.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:32 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:15:32.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:32 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:15:32.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:32 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:32.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:32 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1196226138' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:32.433 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:32.460 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:32.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:32.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:32.562 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:32.586 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:32.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:32.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:32.688 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:32.714 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:32.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:32.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:32.816 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:32.843 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:32.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:32.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:32.891 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 83 16 105458 105442 79.3754 54.3125 0.0150777 0.0125939 2024-09-21T10:15:32.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:32.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:32.980 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:32.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:33.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:33.128 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:33.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:33.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:33.169 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:15:33.170 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:15:33.170 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:15:33.170 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:15:33.170 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:15:33.170 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:15:33.170 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:15:33.170 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:15:33.170 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:15:33.170 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:15:33.170 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:15:33.170 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:15:33.171 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:15:33.171 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:15:33.171 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:15:33.171 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:15:33.230 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:33.255 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:33.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:33.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:33.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:33.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:33 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:15:33.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:33 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:15:33.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:33 smithi107 ceph-mon[102221]: osdmap e313: 8 total, 8 up, 8 in 2024-09-21T10:15:33.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:33 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:15:33.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:33 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:15:33.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:33 smithi107 ceph-mon[105022]: osdmap e313: 8 total, 8 up, 8 in 2024-09-21T10:15:33.404 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:33.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:33.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:33.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:33 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:15:33.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:33 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:15:33.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:33 smithi129 ceph-mon[88471]: osdmap e313: 8 total, 8 up, 8 in 2024-09-21T10:15:33.505 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:33.534 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:33.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:33.636 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:33.662 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:33.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:33.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:33.763 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:33.796 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:33.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:33.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:33.891 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 84 16 106347 106331 79.0919 55.5625 0.0172638 0.012638 2024-09-21T10:15:33.897 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:33.923 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:33.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:33.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:34.025 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:34.052 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:34.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:34.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:34.154 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:34.184 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:34.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:34.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:34.286 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:34.315 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:34.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:34.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:34.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:34.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:34 smithi129 ceph-mon[88471]: pgmap v1114: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 53 MiB/s wr, 855 op/s; 2062/313788 objects misplaced (0.657%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:15:34.455 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:34.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:34.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:34.480 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:34 smithi107 ceph-mon[102221]: pgmap v1114: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 53 MiB/s wr, 855 op/s; 2062/313788 objects misplaced (0.657%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:15:34.480 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:34 smithi107 ceph-mon[105022]: pgmap v1114: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 53 MiB/s wr, 855 op/s; 2062/313788 objects misplaced (0.657%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:15:34.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:34.598 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:34.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:34.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:34.699 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:34.722 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:34.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:34.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:34.824 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:34.847 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:34.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:34.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:34.891 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 85 16 107246 107230 78.8225 56.1875 0.00867273 0.0126813 2024-09-21T10:15:34.949 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:34.976 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:34.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:34.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:35.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:15:35.078 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:35.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:35.105 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:35.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:35.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:35.258 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:35.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:35.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:35.360 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:35.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:35 smithi107 ceph-mon[102221]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-21T10:15:35.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:35 smithi107 ceph-mon[105022]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-21T10:15:35.413 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:35.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:35.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:35.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:35 smithi129 ceph-mon[88471]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-21T10:15:35.515 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:35.541 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:35.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:35.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:35.643 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:35.673 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:35.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:35.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:35.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:35.817 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:35.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:35.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:35.891 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 86 16 108106 108090 78.5309 53.75 0.0060318 0.0127276 2024-09-21T10:15:35.919 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:35.950 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:35.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:35.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:36.052 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:36.081 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:36.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:36.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:36.183 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:36.211 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:36.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:36.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:36.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:36.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:36.366 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:36.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:36.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:36 smithi129 ceph-mon[88471]: pgmap v1115: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s; 2062/315366 objects misplaced (0.654%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:15:36.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:36.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:36.502 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:36.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:36.506 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:36 smithi107 ceph-mon[102221]: pgmap v1115: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s; 2062/315366 objects misplaced (0.654%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:15:36.507 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:36 smithi107 ceph-mon[105022]: pgmap v1115: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 53 MiB/s wr, 852 op/s; 2062/315366 objects misplaced (0.654%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:15:36.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:15:36.604 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:36.630 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:36.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:36.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:36.731 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:36.756 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:36.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:36.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:36.857 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:36.884 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:36.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:36.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:36.892 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 87 16 108954 108938 78.2368 53 0.0111067 0.0127762 2024-09-21T10:15:36.963 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:36.985 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:37.008 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:37.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:37.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:37.110 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:37.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:37.139 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:37.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:37.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:37.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:37.300 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:37.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:37.401 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:37.428 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:37.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:37.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:37.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:37.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:37.575 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:37.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:37.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:37.718 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:37.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:37.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:37.773 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:15:37.773 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:15:37.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.774 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.36822.0:109667 12.5 e313/294 12:a0e1a04e:::benchmark_data_smithi107_250110_object109666:head v 313'6945, mlcod=313'6943)", 2024-09-21T10:15:37.774 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:15:37.753639+0000", 2024-09-21T10:15:37.774 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.016935111999999999, 2024-09-21T10:15:37.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016960695000000001, 2024-09-21T10:15:37.774 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:15:37.774 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:15:37.774 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:15:37.774 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:15:37.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:15:37.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.753639+0000", 2024-09-21T10:15:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:15:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.753639+0000", 2024-09-21T10:15:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:15:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.753640+0000", 2024-09-21T10:15:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9999999999999996e-07 2024-09-21T10:15:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:15:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.753744+0000", 2024-09-21T10:15:37.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010353600000000001 2024-09-21T10:15:37.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:15:37.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.753752+0000", 2024-09-21T10:15:37.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5280000000000003e-06 2024-09-21T10:15:37.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:15:37.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.753758+0000", 2024-09-21T10:15:37.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4229999999999996e-06 2024-09-21T10:15:37.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:15:37.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.753817+0000", 2024-09-21T10:15:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9070000000000002e-05 2024-09-21T10:15:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:15:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.753845+0000", 2024-09-21T10:15:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8736999999999999e-05 2024-09-21T10:15:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:15:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.36822.0:109668 12.5 e313/294 12:a08263e8:::benchmark_data_smithi107_250110_object109667:head v 313'6946, mlcod=313'6943)", 2024-09-21T10:15:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:15:37.757082+0000", 2024-09-21T10:15:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.013492495, 2024-09-21T10:15:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013570186, 2024-09-21T10:15:37.777 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:15:37.778 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:15:37.778 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:15:37.778 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:15:37.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:15:37.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.757082+0000", 2024-09-21T10:15:37.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:15:37.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.757082+0000", 2024-09-21T10:15:37.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:15:37.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.757083+0000", 2024-09-21T10:15:37.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8400000000000003e-07 2024-09-21T10:15:37.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:15:37.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.757112+0000", 2024-09-21T10:15:37.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9447000000000001e-05 2024-09-21T10:15:37.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:15:37.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.757127+0000", 2024-09-21T10:15:37.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4436999999999999e-05 2024-09-21T10:15:37.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:15:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.757133+0000", 2024-09-21T10:15:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.601e-06 2024-09-21T10:15:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:15:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.763075+0000", 2024-09-21T10:15:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0059417080000000004 2024-09-21T10:15:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:15:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.763094+0000", 2024-09-21T10:15:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8804000000000002e-05 2024-09-21T10:15:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:15:37.780 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.781 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.36822.0:109671 12.4 e313/294 12:24b0dfc0:::benchmark_data_smithi107_250110_object109670:head v 313'6787, mlcod=313'6784)", 2024-09-21T10:15:37.781 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:15:37.758572+0000", 2024-09-21T10:15:37.781 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.012002703, 2024-09-21T10:15:37.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.01212654, 2024-09-21T10:15:37.781 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:15:37.781 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:15:37.781 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:15:37.781 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:15:37.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:15:37.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.758572+0000", 2024-09-21T10:15:37.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:15:37.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.758572+0000", 2024-09-21T10:15:37.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:15:37.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.758572+0000", 2024-09-21T10:15:37.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9599999999999999e-07 2024-09-21T10:15:37.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:15:37.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.758601+0000", 2024-09-21T10:15:37.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8523000000000001e-05 2024-09-21T10:15:37.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:15:37.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.758610+0000", 2024-09-21T10:15:37.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6109999999999994e-06 2024-09-21T10:15:37.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:15:37.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.758615+0000", 2024-09-21T10:15:37.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6799999999999998e-06 2024-09-21T10:15:37.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:15:37.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.763518+0000", 2024-09-21T10:15:37.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0049025070000000004 2024-09-21T10:15:37.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:15:37.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.763579+0000", 2024-09-21T10:15:37.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1328000000000006e-05 2024-09-21T10:15:37.784 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.784 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:15:37.784 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.784 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:109674 12.7 12:edde3e56:::benchmark_data_smithi107_250110_object109673:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T10:15:37.784 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:15:37.758774+0000", 2024-09-21T10:15:37.784 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.011800224, 2024-09-21T10:15:37.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011976524000000001, 2024-09-21T10:15:37.785 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:15:37.785 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:15:37.785 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:15:37.785 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:15:37.785 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:15:37.785 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:15:37.785 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 109674 2024-09-21T10:15:37.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.785 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:15:37.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:15:37.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.758774+0000", 2024-09-21T10:15:37.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:15:37.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.758774+0000", 2024-09-21T10:15:37.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:15:37.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.758775+0000", 2024-09-21T10:15:37.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2160000000000001e-06 2024-09-21T10:15:37.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:15:37.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.758796+0000", 2024-09-21T10:15:37.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0251999999999999e-05 2024-09-21T10:15:37.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:15:37.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.758806+0000", 2024-09-21T10:15:37.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0540000000000001e-05 2024-09-21T10:15:37.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:15:37.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.758813+0000", 2024-09-21T10:15:37.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1389999999999998e-06 2024-09-21T10:15:37.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:15:37.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.758866+0000", 2024-09-21T10:15:37.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3000000000000001e-05 2024-09-21T10:15:37.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:15:37.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.759018+0000", 2024-09-21T10:15:37.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015215699999999999 2024-09-21T10:15:37.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:15:37.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.759160+0000", 2024-09-21T10:15:37.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000141545 2024-09-21T10:15:37.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:37.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.764980+0000", 2024-09-21T10:15:37.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0058202979999999998 2024-09-21T10:15:37.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:15:37.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.769718+0000", 2024-09-21T10:15:37.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0047376639999999999 2024-09-21T10:15:37.790 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.790 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:15:37.790 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.790 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.36822.0:109673 12.4 e313/294)", 2024-09-21T10:15:37.790 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:15:37.761709+0000", 2024-09-21T10:15:37.790 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0088653489999999998, 2024-09-21T10:15:37.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00910475, 2024-09-21T10:15:37.791 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:15:37.791 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:15:37.791 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2024-09-21T10:15:37.791 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:15:37.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:15:37.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.761709+0000", 2024-09-21T10:15:37.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:15:37.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.761709+0000", 2024-09-21T10:15:37.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:15:37.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.761710+0000", 2024-09-21T10:15:37.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7599999999999996e-07 2024-09-21T10:15:37.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:15:37.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.761741+0000", 2024-09-21T10:15:37.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0818999999999997e-05 2024-09-21T10:15:37.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:15:37.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.761754+0000", 2024-09-21T10:15:37.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3553999999999999e-05 2024-09-21T10:15:37.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:15:37.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.761761+0000", 2024-09-21T10:15:37.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5760000000000002e-06 2024-09-21T10:15:37.793 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.793 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:15:37.793 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.793 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:109677 12.6 12:6749799c:::benchmark_data_smithi107_250110_object109676:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T10:15:37.794 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:15:37.764736+0000", 2024-09-21T10:15:37.794 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.005837961, 2024-09-21T10:15:37.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0061284269999999997, 2024-09-21T10:15:37.794 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:15:37.794 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:15:37.794 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:15:37.794 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:15:37.794 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:15:37.794 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:15:37.794 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 109677 2024-09-21T10:15:37.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.794 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:15:37.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:15:37.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.764736+0000", 2024-09-21T10:15:37.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:15:37.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.764736+0000", 2024-09-21T10:15:37.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:15:37.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.764738+0000", 2024-09-21T10:15:37.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8700000000000001e-06 2024-09-21T10:15:37.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:15:37.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.764759+0000", 2024-09-21T10:15:37.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0322999999999998e-05 2024-09-21T10:15:37.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:15:37.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.764774+0000", 2024-09-21T10:15:37.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5653000000000001e-05 2024-09-21T10:15:37.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:15:37.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.764782+0000", 2024-09-21T10:15:37.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3429999999999998e-06 2024-09-21T10:15:37.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:15:37.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.765149+0000", 2024-09-21T10:15:37.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036742399999999998 2024-09-21T10:15:37.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:15:37.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.765502+0000", 2024-09-21T10:15:37.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035283299999999999 2024-09-21T10:15:37.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:15:37.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.767995+0000", 2024-09-21T10:15:37.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002493224 2024-09-21T10:15:37.797 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.798 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:15:37.798 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.798 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.36822.0:109657 12.6 e313/298)", 2024-09-21T10:15:37.798 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:15:37.768228+0000", 2024-09-21T10:15:37.798 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.002346856, 2024-09-21T10:15:37.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0026970689999999999, 2024-09-21T10:15:37.798 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:15:37.798 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:15:37.798 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2024-09-21T10:15:37.798 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:15:37.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:15:37.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.768228+0000", 2024-09-21T10:15:37.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:15:37.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.768228+0000", 2024-09-21T10:15:37.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:15:37.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.768229+0000", 2024-09-21T10:15:37.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1209999999999999e-06 2024-09-21T10:15:37.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:15:37.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.768235+0000", 2024-09-21T10:15:37.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7680000000000003e-06 2024-09-21T10:15:37.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:15:37.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.768255+0000", 2024-09-21T10:15:37.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9160999999999998e-05 2024-09-21T10:15:37.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:15:37.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.768276+0000", 2024-09-21T10:15:37.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1107999999999998e-05 2024-09-21T10:15:37.801 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.801 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:15:37.801 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.801 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.36822.0:109666 12.6 e313/298)", 2024-09-21T10:15:37.801 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:15:37.768293+0000", 2024-09-21T10:15:37.801 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0022815130000000002, 2024-09-21T10:15:37.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0026668460000000001, 2024-09-21T10:15:37.801 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:15:37.801 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:15:37.801 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2024-09-21T10:15:37.801 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:15:37.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:15:37.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.768293+0000", 2024-09-21T10:15:37.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:15:37.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.768293+0000", 2024-09-21T10:15:37.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:15:37.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.768293+0000", 2024-09-21T10:15:37.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4699999999999998e-07 2024-09-21T10:15:37.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:15:37.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.768295+0000", 2024-09-21T10:15:37.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2390000000000001e-06 2024-09-21T10:15:37.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:15:37.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.768304+0000", 2024-09-21T10:15:37.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4190000000000002e-06 2024-09-21T10:15:37.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:15:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.768308+0000", 2024-09-21T10:15:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2560000000000004e-06 2024-09-21T10:15:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:15:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:109657 12.6 12:6b3d4546:::benchmark_data_smithi107_250110_object109656:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T10:15:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:15:37.741148+0000", 2024-09-21T10:15:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.029425975, 2024-09-21T10:15:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029855129000000001, 2024-09-21T10:15:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:15:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:15:37.804 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:15:37.805 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:15:37.805 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:15:37.805 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:15:37.805 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 109657 2024-09-21T10:15:37.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.805 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:15:37.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:15:37.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.741148+0000", 2024-09-21T10:15:37.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:15:37.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.741148+0000", 2024-09-21T10:15:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:15:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.741150+0000", 2024-09-21T10:15:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9549999999999999e-06 2024-09-21T10:15:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:15:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.741204+0000", 2024-09-21T10:15:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3433999999999997e-05 2024-09-21T10:15:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:15:37.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.741223+0000", 2024-09-21T10:15:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8705000000000001e-05 2024-09-21T10:15:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:15:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.741231+0000", 2024-09-21T10:15:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2060000000000007e-06 2024-09-21T10:15:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:15:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.745110+0000", 2024-09-21T10:15:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038791920000000001 2024-09-21T10:15:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:15:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.745433+0000", 2024-09-21T10:15:37.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000323441 2024-09-21T10:15:37.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:15:37.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.745833+0000", 2024-09-21T10:15:37.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00039935900000000001 2024-09-21T10:15:37.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:37.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.760223+0000", 2024-09-21T10:15:37.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014390254 2024-09-21T10:15:37.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:15:37.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.760295+0000", 2024-09-21T10:15:37.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2249999999999994e-05 2024-09-21T10:15:37.809 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.809 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:15:37.809 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.809 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:109664 12.7 12:eeea3820:::benchmark_data_smithi107_250110_object109663:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T10:15:37.809 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:15:37.741934+0000", 2024-09-21T10:15:37.809 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.028640500999999999, 2024-09-21T10:15:37.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.029136744999999999, 2024-09-21T10:15:37.809 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:15:37.809 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:15:37.809 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:15:37.809 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:15:37.809 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:15:37.810 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:15:37.810 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 109664 2024-09-21T10:15:37.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.810 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:15:37.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:15:37.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.741934+0000", 2024-09-21T10:15:37.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:15:37.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.741934+0000", 2024-09-21T10:15:37.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:15:37.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.741935+0000", 2024-09-21T10:15:37.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6799999999999999e-07 2024-09-21T10:15:37.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:15:37.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.741982+0000", 2024-09-21T10:15:37.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7181000000000001e-05 2024-09-21T10:15:37.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:15:37.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.741991+0000", 2024-09-21T10:15:37.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2369999999999998e-06 2024-09-21T10:15:37.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:15:37.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.741997+0000", 2024-09-21T10:15:37.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2360000000000001e-06 2024-09-21T10:15:37.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:15:37.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.745974+0000", 2024-09-21T10:15:37.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003976645 2024-09-21T10:15:37.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.812 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:15:37.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.746088+0000", 2024-09-21T10:15:37.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000113987 2024-09-21T10:15:37.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:15:37.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.746209+0000", 2024-09-21T10:15:37.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012097800000000001 2024-09-21T10:15:37.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:15:37.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.751099+0000", 2024-09-21T10:15:37.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048898429999999996 2024-09-21T10:15:37.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:37.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.753627+0000", 2024-09-21T10:15:37.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002528069 2024-09-21T10:15:37.814 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.814 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:15:37.814 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.814 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:109666 12.6 12:619bd49d:::benchmark_data_smithi107_250110_object109665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e313)", 2024-09-21T10:15:37.814 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:15:37.749848+0000", 2024-09-21T10:15:37.814 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.020726596999999999, 2024-09-21T10:15:37.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021290221000000002, 2024-09-21T10:15:37.815 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:15:37.815 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:15:37.815 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:15:37.815 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:15:37.815 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:15:37.815 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:15:37.815 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 109666 2024-09-21T10:15:37.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.815 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:15:37.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:15:37.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.749848+0000", 2024-09-21T10:15:37.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:15:37.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.749848+0000", 2024-09-21T10:15:37.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:15:37.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:15:37.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.749849+0000", 2024-09-21T10:15:37.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5460000000000001e-06 2024-09-21T10:15:37.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:15:37.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.749875+0000", 2024-09-21T10:15:37.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5973e-05 2024-09-21T10:15:37.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:15:37.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.749888+0000", 2024-09-21T10:15:37.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2989999999999999e-05 2024-09-21T10:15:37.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:15:37.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.749897+0000", 2024-09-21T10:15:37.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3720000000000005e-06 2024-09-21T10:15:37.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:15:37.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.754040+0000", 2024-09-21T10:15:37.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041429070000000004 2024-09-21T10:15:37.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:15:37.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.754190+0000", 2024-09-21T10:15:37.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015080700000000001 2024-09-21T10:15:37.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:15:37.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.754338+0000", 2024-09-21T10:15:37.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000147637 2024-09-21T10:15:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:15:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.760312+0000", 2024-09-21T10:15:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0059740840000000002 2024-09-21T10:15:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:15:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:15:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:15:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:37.760440+0000", 2024-09-21T10:15:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012774300000000001 2024-09-21T10:15:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:15:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:15:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-09-21T10:15:37.819 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 11 2024-09-21T10:15:37.820 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:15:37.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:37.854 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:37.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:37.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:37.892 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 88 16 109770 109754 77.9273 51 0.0104704 0.0128271 2024-09-21T10:15:37.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:37.993 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:37.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:37.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:38.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:38.138 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:38.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:38.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:38.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:38.282 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:38.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:38.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:38.372 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:15:38.375 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:15:38.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:38.427 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:38.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:38.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:38.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:38.571 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:38.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:38.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:38.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:38.692 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:38 smithi107 ceph-mon[102221]: pgmap v1116: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 54 MiB/s wr, 863 op/s; 1883/322764 objects misplaced (0.583%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:15:38.692 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:38 smithi107 ceph-mon[105022]: pgmap v1116: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 54 MiB/s wr, 863 op/s; 1883/322764 objects misplaced (0.583%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:15:38.712 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:38.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:38.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:38.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:38.854 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:38.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:38.892 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 89 16 110570 110554 77.6135 50 0.014364 0.0128786 2024-09-21T10:15:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:38 smithi129 ceph-mon[88471]: pgmap v1116: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 54 MiB/s wr, 863 op/s; 1883/322764 objects misplaced (0.583%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:15:38.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:38.998 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:38.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:38.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:39.099 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:39.126 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:39.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:39.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:39.228 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:39.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:39.256 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:39.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:39.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:39.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:39.394 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:39.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:39.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:15:39.495 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:39.521 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:39.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:39.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:39.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:39.685 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:39.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:39.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:39.786 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:39.811 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:39.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:39.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:39.855 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:39 smithi107 ceph-mon[102221]: from='client.36876 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:15:39.856 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:39 smithi107 ceph-mon[105022]: from='client.36876 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:15:39.893 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total time run: 90.0089 2024-09-21T10:15:39.893 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total writes made: 111407 2024-09-21T10:15:39.893 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Write size: 65536 2024-09-21T10:15:39.894 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object size: 65536 2024-09-21T10:15:39.894 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Bandwidth (MB/sec): 77.3584 2024-09-21T10:15:39.894 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Bandwidth: 36.9721 2024-09-21T10:15:39.894 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max bandwidth (MB/sec): 213.75 2024-09-21T10:15:39.894 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min bandwidth (MB/sec): 0 2024-09-21T10:15:39.894 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average IOPS: 1237 2024-09-21T10:15:39.894 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev IOPS: 591.564 2024-09-21T10:15:39.894 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max IOPS: 3420 2024-09-21T10:15:39.894 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min IOPS: 0 2024-09-21T10:15:39.894 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average Latency(s): 0.0129225 2024-09-21T10:15:39.894 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Latency(s): 0.0300592 2024-09-21T10:15:39.894 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max latency(s): 2.1623 2024-09-21T10:15:39.894 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min latency(s): 0.00109013 2024-09-21T10:15:39.897 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:15:39.912 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:39.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:39 smithi129 ceph-mon[88471]: from='client.36876 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:15:39.940 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:39.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:39.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:40.041 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:40.088 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:40.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:40.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:40.189 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:40.211 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:40.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:40.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:40.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:40.367 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:40.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:40.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:40.459 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:40.468 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:40.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:40.495 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:40.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:40.596 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:40.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:40.630 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:40.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:40.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:40 smithi129 ceph-mon[88471]: pgmap v1117: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 54 MiB/s wr, 868 op/s; 1883/326829 objects misplaced (0.576%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:15:40.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:15:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:15:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:15:40.731 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:40.760 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:40.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:40 smithi107 ceph-mon[102221]: pgmap v1117: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 54 MiB/s wr, 868 op/s; 1883/326829 objects misplaced (0.576%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:15:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:40 smithi107 ceph-mon[105022]: pgmap v1117: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.6 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 54 MiB/s wr, 868 op/s; 1883/326829 objects misplaced (0.576%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:15:40.861 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:40.886 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:40.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:40.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:40.900 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1116,"stamp":"2024-09-21T10:15:37.371150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7049832818,"num_objects":107588,"num_object_clones":0,"num_object_copies":322764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1883,"num_objects_unfound":0,"num_objects_dirty":107588,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":107639,"num_write_kb":6884148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3474,"num_bytes_recovered":223991259,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108321,"ondisk_log_size":108321,"up":327,"acting":326,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":329,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":28077016,"kb_used_data":20820556,"kb_used_omap":43,"kb_used_meta":7256340,"kb_avail":721884200,"statfs":{"total":767960285184,"available":739209420800,"internally_reserved":0,"allocated":21320249344,"data_stored":21305632366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44223,"internal_metadata":7430492993},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,4,8,7,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":566165504,"num_objects":8639,"num_object_clones":0,"num_object_copies":25917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-353,"num_objects_unfound":0,"num_objects_dirty":8639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8639,"num_write_kb":552896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":353,"num_bytes_recovered":23134208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004777"},"pg_stats":[{"pgid":"1.11","version":"77'100","reported_seq":142,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.168751+0000","last_change":"2024-09-21T10:10:57.397786+0000","last_active":"2024-09-21T10:15:32.168751+0000","last_peered":"2024-09-21T10:15:32.168751+0000","last_clean":"2024-09-21T10:15:32.168751+0000","last_became_active":"2024-09-21T10:10:57.396742+0000","last_became_peered":"2024-09-21T10:10:57.396742+0000","last_unstale":"2024-09-21T10:15:32.168751+0000","last_undegraded":"2024-09-21T10:15:32.168751+0000","last_fullsized":"2024-09-21T10:15:32.168751+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:53.631923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:53.631923+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:33:35.364595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":481,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.164308+0000","last_change":"2024-09-21T10:03:38.389577+0000","last_active":"2024-09-21T10:15:32.164308+0000","last_peered":"2024-09-21T10:15:32.164308+0000","last_clean":"2024-09-21T10:15:32.164308+0000","last_became_active":"2024-09-21T10:03:38.389292+0000","last_became_peered":"2024-09-21T10:03:38.389292+0000","last_unstale":"2024-09-21T10:15:32.164308+0000","last_undegraded":"2024-09-21T10:15:32.164308+0000","last_fullsized":"2024-09-21T10:15:32.164308+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:32.113472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":198,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.709665+0000","last_change":"2024-09-21T10:14:30.717695+0000","last_active":"2024-09-21T10:15:18.709665+0000","last_peered":"2024-09-21T10:15:18.709665+0000","last_clean":"2024-09-21T10:15:18.709665+0000","last_became_active":"2024-09-21T10:14:30.705952+0000","last_became_peered":"2024-09-21T10:14:30.705952+0000","last_unstale":"2024-09-21T10:15:18.709665+0000","last_undegraded":"2024-09-21T10:15:18.709665+0000","last_fullsized":"2024-09-21T10:15:18.709665+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:08:18.863430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'100","reported_seq":211,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.710072+0000","last_change":"2024-09-21T10:11:05.438031+0000","last_active":"2024-09-21T10:15:18.710072+0000","last_peered":"2024-09-21T10:15:18.710072+0000","last_clean":"2024-09-21T10:15:18.710072+0000","last_became_active":"2024-09-21T10:11:05.437637+0000","last_became_peered":"2024-09-21T10:11:05.437637+0000","last_unstale":"2024-09-21T10:15:18.710072+0000","last_undegraded":"2024-09-21T10:15:18.710072+0000","last_fullsized":"2024-09-21T10:15:18.710072+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.798577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.798577+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:42.343217+0000","scrub_duration":0.0023622640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":303,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.710312+0000","last_change":"2024-09-21T10:10:57.392782+0000","last_active":"2024-09-21T10:15:18.710312+0000","last_peered":"2024-09-21T10:15:18.710312+0000","last_clean":"2024-09-21T10:15:18.710312+0000","last_became_active":"2024-09-21T10:10:57.392305+0000","last_became_peered":"2024-09-21T10:10:57.392305+0000","last_unstale":"2024-09-21T10:15:18.710312+0000","last_undegraded":"2024-09-21T10:15:18.710312+0000","last_fullsized":"2024-09-21T10:15:18.710312+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:42:01.387622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":401,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.169575+0000","last_change":"2024-09-21T10:09:23.846703+0000","last_active":"2024-09-21T10:15:32.169575+0000","last_peered":"2024-09-21T10:15:32.169575+0000","last_clean":"2024-09-21T10:15:32.169575+0000","last_became_active":"2024-09-21T10:09:23.846415+0000","last_became_peered":"2024-09-21T10:09:23.846415+0000","last_unstale":"2024-09-21T10:15:32.169575+0000","last_undegraded":"2024-09-21T10:15:32.169575+0000","last_fullsized":"2024-09-21T10:15:32.169575+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:23.129285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'100","reported_seq":136,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.161744+0000","last_change":"2024-09-21T10:14:05.475181+0000","last_active":"2024-09-21T10:15:32.161744+0000","last_peered":"2024-09-21T10:15:32.161744+0000","last_clean":"2024-09-21T10:15:32.161744+0000","last_became_active":"2024-09-21T10:14:05.474523+0000","last_became_peered":"2024-09-21T10:14:05.474523+0000","last_unstale":"2024-09-21T10:15:32.161744+0000","last_undegraded":"2024-09-21T10:15:32.161744+0000","last_fullsized":"2024-09-21T10:15:32.161744+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.433227+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:40.087104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":127,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.169724+0000","last_change":"2024-09-21T10:09:49.922489+0000","last_active":"2024-09-21T10:15:32.169724+0000","last_peered":"2024-09-21T10:15:32.169724+0000","last_clean":"2024-09-21T10:15:32.169724+0000","last_became_active":"2024-09-21T10:09:49.921629+0000","last_became_peered":"2024-09-21T10:09:49.921629+0000","last_unstale":"2024-09-21T10:15:32.169724+0000","last_undegraded":"2024-09-21T10:15:32.169724+0000","last_fullsized":"2024-09-21T10:15:32.169724+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:50.611979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":194,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.709962+0000","last_change":"2024-09-21T10:14:30.709707+0000","last_active":"2024-09-21T10:15:18.709962+0000","last_peered":"2024-09-21T10:15:18.709962+0000","last_clean":"2024-09-21T10:15:18.709962+0000","last_became_active":"2024-09-21T10:14:30.709353+0000","last_became_peered":"2024-09-21T10:14:30.709353+0000","last_unstale":"2024-09-21T10:15:18.709962+0000","last_undegraded":"2024-09-21T10:15:18.709962+0000","last_fullsized":"2024-09-21T10:15:18.709962+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:12.822947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"77'100","reported_seq":32,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.161744+0000","last_change":"2024-09-21T10:15:02.842044+0000","last_active":"2024-09-21T10:15:32.161744+0000","last_peered":"2024-09-21T10:15:32.161744+0000","last_clean":"2024-09-21T10:15:32.161744+0000","last_became_active":"2024-09-21T10:15:01.478273+0000","last_became_peered":"2024-09-21T10:15:01.478273+0000","last_unstale":"2024-09-21T10:15:32.161744+0000","last_undegraded":"2024-09-21T10:15:32.161744+0000","last_fullsized":"2024-09-21T10:15:32.161744+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:15:02.841968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:15:02.841968+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:38:23.910899+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":481,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.163833+0000","last_change":"2024-09-21T10:03:38.381552+0000","last_active":"2024-09-21T10:15:32.163833+0000","last_peered":"2024-09-21T10:15:32.163833+0000","last_clean":"2024-09-21T10:15:32.163833+0000","last_became_active":"2024-09-21T10:03:38.380898+0000","last_became_peered":"2024-09-21T10:03:38.380898+0000","last_unstale":"2024-09-21T10:15:32.163833+0000","last_undegraded":"2024-09-21T10:15:32.163833+0000","last_fullsized":"2024-09-21T10:15:32.163833+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:53.409606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":371,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.718491+0000","last_change":"2024-09-21T10:10:57.459057+0000","last_active":"2024-09-21T10:15:18.718491+0000","last_peered":"2024-09-21T10:15:18.718491+0000","last_clean":"2024-09-21T10:15:18.718491+0000","last_became_active":"2024-09-21T10:10:57.451776+0000","last_became_peered":"2024-09-21T10:10:57.451776+0000","last_unstale":"2024-09-21T10:15:18.718491+0000","last_undegraded":"2024-09-21T10:15:18.718491+0000","last_fullsized":"2024-09-21T10:15:18.718491+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:00.138317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"313'6713","reported_seq":6739,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.870096+0000","last_change":"2024-09-21T10:13:59.438100+0000","last_active":"2024-09-21T10:15:32.870096+0000","last_peered":"2024-09-21T10:15:32.870096+0000","last_clean":"2024-09-21T10:15:32.870096+0000","last_became_active":"2024-09-21T10:13:59.437802+0000","last_became_peered":"2024-09-21T10:13:59.437802+0000","last_unstale":"2024-09-21T10:15:32.870096+0000","last_undegraded":"2024-09-21T10:15:32.870096+0000","last_fullsized":"2024-09-21T10:15:32.870096+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":294,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_clean_scrub_stamp":"2024-09-21T10:13:58.411930+0000","objects_scrubbed":0,"log_size":6713,"log_dups_size":0,"ondisk_log_size":6713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:09.342171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439943168,"num_objects":6713,"num_object_clones":0,"num_object_copies":20139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6713,"num_write_kb":429632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"77'100","reported_seq":206,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.709688+0000","last_change":"2024-09-21T10:14:30.708275+0000","last_active":"2024-09-21T10:15:18.709688+0000","last_peered":"2024-09-21T10:15:18.709688+0000","last_clean":"2024-09-21T10:15:18.709688+0000","last_became_active":"2024-09-21T10:14:30.707763+0000","last_became_peered":"2024-09-21T10:14:30.707763+0000","last_unstale":"2024-09-21T10:15:18.709688+0000","last_undegraded":"2024-09-21T10:15:18.709688+0000","last_fullsized":"2024-09-21T10:15:18.709688+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:55.259385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:55.259385+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:41.322447+0000","scrub_duration":0.0015667260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":241,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.710175+0000","last_change":"2024-09-21T10:14:30.720250+0000","last_active":"2024-09-21T10:15:18.710175+0000","last_peered":"2024-09-21T10:15:18.710175+0000","last_clean":"2024-09-21T10:15:18.710175+0000","last_became_active":"2024-09-21T10:14:30.707036+0000","last_became_peered":"2024-09-21T10:14:30.707036+0000","last_unstale":"2024-09-21T10:15:18.710175+0000","last_undegraded":"2024-09-21T10:15:18.710175+0000","last_fullsized":"2024-09-21T10:15:18.710175+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:17.438534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":284,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.709893+0000","last_change":"2024-09-21T10:15:01.476521+0000","last_active":"2024-09-21T10:15:18.709893+0000","last_peered":"2024-09-21T10:15:18.709893+0000","last_clean":"2024-09-21T10:15:18.709893+0000","last_became_active":"2024-09-21T10:15:01.476083+0000","last_became_peered":"2024-09-21T10:15:01.476083+0000","last_unstale":"2024-09-21T10:15:18.709893+0000","last_undegraded":"2024-09-21T10:15:18.709893+0000","last_fullsized":"2024-09-21T10:15:18.709893+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:50.230443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"313'6579","reported_seq":6605,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.860528+0000","last_change":"2024-09-21T10:13:59.437648+0000","last_active":"2024-09-21T10:15:32.860528+0000","last_peered":"2024-09-21T10:15:32.860528+0000","last_clean":"2024-09-21T10:15:32.860528+0000","last_became_active":"2024-09-21T10:13:59.437351+0000","last_became_peered":"2024-09-21T10:13:59.437351+0000","last_unstale":"2024-09-21T10:15:32.860528+0000","last_undegraded":"2024-09-21T10:15:32.860528+0000","last_fullsized":"2024-09-21T10:15:32.860528+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":294,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_clean_scrub_stamp":"2024-09-21T10:13:58.411930+0000","objects_scrubbed":0,"log_size":6579,"log_dups_size":0,"ondisk_log_size":6579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:56:55.960002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431161344,"num_objects":6579,"num_object_clones":0,"num_object_copies":19737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6579,"num_write_kb":421056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"77'100","reported_seq":287,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.164192+0000","last_change":"2024-09-21T10:09:23.839211+0000","last_active":"2024-09-21T10:15:32.164192+0000","last_peered":"2024-09-21T10:15:32.164192+0000","last_clean":"2024-09-21T10:15:32.164192+0000","last_became_active":"2024-09-21T10:09:23.838779+0000","last_became_peered":"2024-09-21T10:09:23.838779+0000","last_unstale":"2024-09-21T10:15:32.164192+0000","last_undegraded":"2024-09-21T10:15:32.164192+0000","last_fullsized":"2024-09-21T10:15:32.164192+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.981336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.981336+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:03:15.566873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":206,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.710514+0000","last_change":"2024-09-21T10:09:49.932631+0000","last_active":"2024-09-21T10:15:18.710514+0000","last_peered":"2024-09-21T10:15:18.710514+0000","last_clean":"2024-09-21T10:15:18.710514+0000","last_became_active":"2024-09-21T10:09:49.922897+0000","last_became_peered":"2024-09-21T10:09:49.922897+0000","last_unstale":"2024-09-21T10:15:18.710514+0000","last_undegraded":"2024-09-21T10:15:18.710514+0000","last_fullsized":"2024-09-21T10:15:18.710514+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:16:40.795476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":19,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.709158+0000","last_change":"2024-09-21T10:14:30.706497+0000","last_active":"2024-09-21T10:15:18.709158+0000","last_peered":"2024-09-21T10:15:18.709158+0000","last_clean":"2024-09-21T10:15:18.709158+0000","last_became_active":"2024-09-21T10:14:30.706202+0000","last_became_peered":"2024-09-21T10:14:30.706202+0000","last_unstale":"2024-09-21T10:15:18.709158+0000","last_undegraded":"2024-09-21T10:15:18.709158+0000","last_fullsized":"2024-09-21T10:15:18.709158+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:42:57.912707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2","version":"313'6709","reported_seq":8357,"reported_epoch":313,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T10:15:37.159848+0000","last_change":"2024-09-21T10:15:17.390869+0000","last_active":"2024-09-21T10:15:37.159848+0000","last_peered":"2024-09-21T10:15:37.159848+0000","last_clean":"2024-09-21T10:14:29.633017+0000","last_became_active":"2024-09-21T10:14:31.799003+0000","last_became_peered":"2024-09-21T10:14:31.799003+0000","last_unstale":"2024-09-21T10:15:37.159848+0000","last_undegraded":"2024-09-21T10:15:37.159848+0000","last_fullsized":"2024-09-21T10:14:31.697866+0000","mapping_epoch":304,"log_start":"300'300","ondisk_log_start":"300'300","created":294,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_clean_scrub_stamp":"2024-09-21T10:13:58.411930+0000","objects_scrubbed":0,"log_size":6409,"log_dups_size":0,"ondisk_log_size":6409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439681024,"num_objects":6709,"num_object_clones":0,"num_object_copies":20127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1883,"num_objects_unfound":0,"num_objects_dirty":6709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6709,"num_write_kb":429376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":798,"num_bytes_recovered":52297728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[3,7],"avail_no_missing":["3","6","7"],"object_location_counts":[{"shards":"3,6,7","objects":1883},{"shards":"3,7","objects":4826}],"blocked_by":[],"up_primary":4,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"77'100","reported_seq":306,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.718227+0000","last_change":"2024-09-21T10:14:30.709997+0000","last_active":"2024-09-21T10:15:18.718227+0000","last_peered":"2024-09-21T10:15:18.718227+0000","last_clean":"2024-09-21T10:15:18.718227+0000","last_became_active":"2024-09-21T10:14:30.709580+0000","last_became_peered":"2024-09-21T10:14:30.709580+0000","last_unstale":"2024-09-21T10:15:18.718227+0000","last_undegraded":"2024-09-21T10:15:18.718227+0000","last_fullsized":"2024-09-21T10:15:18.718227+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.289859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.289859+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:27:50.110602+0000","scrub_duration":0.001251809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":192,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.710206+0000","last_change":"2024-09-21T10:10:57.460732+0000","last_active":"2024-09-21T10:15:18.710206+0000","last_peered":"2024-09-21T10:15:18.710206+0000","last_clean":"2024-09-21T10:15:18.710206+0000","last_became_active":"2024-09-21T10:10:57.454689+0000","last_became_peered":"2024-09-21T10:10:57.454689+0000","last_unstale":"2024-09-21T10:15:18.710206+0000","last_undegraded":"2024-09-21T10:15:18.710206+0000","last_fullsized":"2024-09-21T10:15:18.710206+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:12.035022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":312,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.718584+0000","last_change":"2024-09-21T10:03:38.378384+0000","last_active":"2024-09-21T10:15:18.718584+0000","last_peered":"2024-09-21T10:15:18.718584+0000","last_clean":"2024-09-21T10:15:18.718584+0000","last_became_active":"2024-09-21T10:03:38.378072+0000","last_became_peered":"2024-09-21T10:03:38.378072+0000","last_unstale":"2024-09-21T10:15:18.718584+0000","last_undegraded":"2024-09-21T10:15:18.718584+0000","last_fullsized":"2024-09-21T10:15:18.718584+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:54:37.039622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.3","version":"313'6908","reported_seq":6955,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:37.326331+0000","last_change":"2024-09-21T10:13:59.437741+0000","last_active":"2024-09-21T10:15:37.326331+0000","last_peered":"2024-09-21T10:15:37.326331+0000","last_clean":"2024-09-21T10:15:37.326331+0000","last_became_active":"2024-09-21T10:13:59.437450+0000","last_became_peered":"2024-09-21T10:13:59.437450+0000","last_unstale":"2024-09-21T10:15:37.326331+0000","last_undegraded":"2024-09-21T10:15:37.326331+0000","last_fullsized":"2024-09-21T10:15:37.326331+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":294,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_clean_scrub_stamp":"2024-09-21T10:13:58.411930+0000","objects_scrubbed":0,"log_size":6908,"log_dups_size":0,"ondisk_log_size":6908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:02:02.877405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452722688,"num_objects":6908,"num_object_clones":0,"num_object_copies":20724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6908,"num_write_kb":442112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"77'100","reported_seq":164,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.710390+0000","last_change":"2024-09-21T10:07:26.188298+0000","last_active":"2024-09-21T10:15:18.710390+0000","last_peered":"2024-09-21T10:15:18.710390+0000","last_clean":"2024-09-21T10:15:18.710390+0000","last_became_active":"2024-09-21T10:07:24.345265+0000","last_became_peered":"2024-09-21T10:07:24.345265+0000","last_unstale":"2024-09-21T10:15:18.710390+0000","last_undegraded":"2024-09-21T10:15:18.710390+0000","last_fullsized":"2024-09-21T10:15:18.710390+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:07:26.188220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:07:26.188220+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:46.381710+0000","scrub_duration":0.021884888000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":285,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.164417+0000","last_change":"2024-09-21T10:07:57.551556+0000","last_active":"2024-09-21T10:15:32.164417+0000","last_peered":"2024-09-21T10:15:32.164417+0000","last_clean":"2024-09-21T10:15:32.164417+0000","last_became_active":"2024-09-21T10:07:57.551249+0000","last_became_peered":"2024-09-21T10:07:57.551249+0000","last_unstale":"2024-09-21T10:15:32.164417+0000","last_undegraded":"2024-09-21T10:15:32.164417+0000","last_fullsized":"2024-09-21T10:15:32.164417+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:57.557152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":388,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.175588+0000","last_change":"2024-09-21T10:14:30.711105+0000","last_active":"2024-09-21T10:15:32.175588+0000","last_peered":"2024-09-21T10:15:32.175588+0000","last_clean":"2024-09-21T10:15:32.175588+0000","last_became_active":"2024-09-21T10:14:30.710759+0000","last_became_peered":"2024-09-21T10:14:30.710759+0000","last_unstale":"2024-09-21T10:15:32.175588+0000","last_undegraded":"2024-09-21T10:15:32.175588+0000","last_fullsized":"2024-09-21T10:15:32.175588+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:40.690042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.4","version":"313'6532","reported_seq":6558,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.879064+0000","last_change":"2024-09-21T10:13:59.473741+0000","last_active":"2024-09-21T10:15:32.879064+0000","last_peered":"2024-09-21T10:15:32.879064+0000","last_clean":"2024-09-21T10:15:32.879064+0000","last_became_active":"2024-09-21T10:13:59.473443+0000","last_became_peered":"2024-09-21T10:13:59.473443+0000","last_unstale":"2024-09-21T10:15:32.879064+0000","last_undegraded":"2024-09-21T10:15:32.879064+0000","last_fullsized":"2024-09-21T10:15:32.879064+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":294,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_clean_scrub_stamp":"2024-09-21T10:13:58.411930+0000","objects_scrubbed":0,"log_size":6532,"log_dups_size":0,"ondisk_log_size":6532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:02:32.358588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428081152,"num_objects":6532,"num_object_clones":0,"num_object_copies":19596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6532,"num_write_kb":418048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":186,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.710176+0000","last_change":"2024-09-21T10:04:53.355975+0000","last_active":"2024-09-21T10:15:18.710176+0000","last_peered":"2024-09-21T10:15:18.710176+0000","last_clean":"2024-09-21T10:15:18.710176+0000","last_became_active":"2024-09-21T10:04:42.547505+0000","last_became_peered":"2024-09-21T10:04:42.547505+0000","last_unstale":"2024-09-21T10:15:18.710176+0000","last_undegraded":"2024-09-21T10:15:18.710176+0000","last_fullsized":"2024-09-21T10:15:18.710176+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.355912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.355912+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:43.456830+0000","scrub_duration":0.0017980030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":492,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.170607+0000","last_change":"2024-09-21T10:14:30.715303+0000","last_active":"2024-09-21T10:15:32.170607+0000","last_peered":"2024-09-21T10:15:32.170607+0000","last_clean":"2024-09-21T10:15:32.170607+0000","last_became_active":"2024-09-21T10:14:30.703664+0000","last_became_peered":"2024-09-21T10:14:30.703664+0000","last_unstale":"2024-09-21T10:15:32.170607+0000","last_undegraded":"2024-09-21T10:15:32.170607+0000","last_fullsized":"2024-09-21T10:15:32.170607+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:28.967906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.710915+0000","last_change":"2024-09-21T10:14:30.705927+0000","last_active":"2024-09-21T10:15:18.710915+0000","last_peered":"2024-09-21T10:15:18.710915+0000","last_clean":"2024-09-21T10:15:18.710915+0000","last_became_active":"2024-09-21T10:14:30.705112+0000","last_became_peered":"2024-09-21T10:14:30.705112+0000","last_unstale":"2024-09-21T10:15:18.710915+0000","last_undegraded":"2024-09-21T10:15:18.710915+0000","last_fullsized":"2024-09-21T10:15:18.710915+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:38:33.962199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.5","version":"313'6624","reported_seq":6650,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.888921+0000","last_change":"2024-09-21T10:13:59.474803+0000","last_active":"2024-09-21T10:15:32.888921+0000","last_peered":"2024-09-21T10:15:32.888921+0000","last_clean":"2024-09-21T10:15:32.888921+0000","last_became_active":"2024-09-21T10:13:59.474507+0000","last_became_peered":"2024-09-21T10:13:59.474507+0000","last_unstale":"2024-09-21T10:15:32.888921+0000","last_undegraded":"2024-09-21T10:15:32.888921+0000","last_fullsized":"2024-09-21T10:15:32.888921+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":294,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_clean_scrub_stamp":"2024-09-21T10:13:58.411930+0000","objects_scrubbed":0,"log_size":6624,"log_dups_size":0,"ondisk_log_size":6624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:54:50.121747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434110464,"num_objects":6624,"num_object_clones":0,"num_object_copies":19872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6624,"num_write_kb":423936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":241,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.158166+0000","last_change":"2024-09-21T10:10:57.403722+0000","last_active":"2024-09-21T10:15:32.158166+0000","last_peered":"2024-09-21T10:15:32.158166+0000","last_clean":"2024-09-21T10:15:32.158166+0000","last_became_active":"2024-09-21T10:10:57.394700+0000","last_became_peered":"2024-09-21T10:10:57.394700+0000","last_unstale":"2024-09-21T10:15:32.158166+0000","last_undegraded":"2024-09-21T10:15:32.158166+0000","last_fullsized":"2024-09-21T10:15:32.158166+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:02.152981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:02.152981+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:52:34.887763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":296,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.718123+0000","last_change":"2024-09-21T10:14:30.707046+0000","last_active":"2024-09-21T10:15:18.718123+0000","last_peered":"2024-09-21T10:15:18.718123+0000","last_clean":"2024-09-21T10:15:18.718123+0000","last_became_active":"2024-09-21T10:14:30.706753+0000","last_became_peered":"2024-09-21T10:14:30.706753+0000","last_unstale":"2024-09-21T10:15:18.718123+0000","last_undegraded":"2024-09-21T10:15:18.718123+0000","last_fullsized":"2024-09-21T10:15:18.718123+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:58.136096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":290,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.167945+0000","last_change":"2024-09-21T10:07:24.342950+0000","last_active":"2024-09-21T10:15:32.167945+0000","last_peered":"2024-09-21T10:15:32.167945+0000","last_clean":"2024-09-21T10:15:32.167945+0000","last_became_active":"2024-09-21T10:07:24.342477+0000","last_became_peered":"2024-09-21T10:07:24.342477+0000","last_unstale":"2024-09-21T10:15:32.167945+0000","last_undegraded":"2024-09-21T10:15:32.167945+0000","last_fullsized":"2024-09-21T10:15:32.167945+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:33:13.664050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"313'6705","reported_seq":6747,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:35.535312+0000","last_change":"2024-09-21T10:14:05.474157+0000","last_active":"2024-09-21T10:15:35.535312+0000","last_peered":"2024-09-21T10:15:35.535312+0000","last_clean":"2024-09-21T10:15:35.535312+0000","last_became_active":"2024-09-21T10:14:05.473850+0000","last_became_peered":"2024-09-21T10:14:05.473850+0000","last_unstale":"2024-09-21T10:15:35.535312+0000","last_undegraded":"2024-09-21T10:15:35.535312+0000","last_fullsized":"2024-09-21T10:15:35.535312+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":294,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_clean_scrub_stamp":"2024-09-21T10:13:58.411930+0000","objects_scrubbed":0,"log_size":6705,"log_dups_size":0,"ondisk_log_size":6705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:09:31.971041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439418880,"num_objects":6705,"num_object_clones":0,"num_object_copies":20115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6705,"num_write_kb":429120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"77'100","reported_seq":133,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.161643+0000","last_change":"2024-09-21T10:14:05.474100+0000","last_active":"2024-09-21T10:15:32.161643+0000","last_peered":"2024-09-21T10:15:32.161643+0000","last_clean":"2024-09-21T10:15:32.161643+0000","last_became_active":"2024-09-21T10:14:05.473336+0000","last_became_peered":"2024-09-21T10:14:05.473336+0000","last_unstale":"2024-09-21T10:15:32.161643+0000","last_undegraded":"2024-09-21T10:15:32.161643+0000","last_fullsized":"2024-09-21T10:15:32.161643+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.540323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.540323+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:13:30.035766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":382,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.159011+0000","last_change":"2024-09-21T10:07:36.464496+0000","last_active":"2024-09-21T10:15:32.159011+0000","last_peered":"2024-09-21T10:15:32.159011+0000","last_clean":"2024-09-21T10:15:32.159011+0000","last_became_active":"2024-09-21T10:07:36.464172+0000","last_became_peered":"2024-09-21T10:07:36.464172+0000","last_unstale":"2024-09-21T10:15:32.159011+0000","last_undegraded":"2024-09-21T10:15:32.159011+0000","last_fullsized":"2024-09-21T10:15:32.159011+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:09.839709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":19,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.709642+0000","last_change":"2024-09-21T10:14:30.706905+0000","last_active":"2024-09-21T10:15:18.709642+0000","last_peered":"2024-09-21T10:15:18.709642+0000","last_clean":"2024-09-21T10:15:18.709642+0000","last_became_active":"2024-09-21T10:14:30.706614+0000","last_became_peered":"2024-09-21T10:14:30.706614+0000","last_unstale":"2024-09-21T10:15:18.709642+0000","last_undegraded":"2024-09-21T10:15:18.709642+0000","last_fullsized":"2024-09-21T10:15:18.709642+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:26:33.014993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"313'6792","reported_seq":6834,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:35.507384+0000","last_change":"2024-09-21T10:14:05.476580+0000","last_active":"2024-09-21T10:15:35.507384+0000","last_peered":"2024-09-21T10:15:35.507384+0000","last_clean":"2024-09-21T10:15:35.507384+0000","last_became_active":"2024-09-21T10:14:05.476304+0000","last_became_peered":"2024-09-21T10:14:05.476304+0000","last_unstale":"2024-09-21T10:15:35.507384+0000","last_undegraded":"2024-09-21T10:15:35.507384+0000","last_fullsized":"2024-09-21T10:15:35.507384+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":294,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_clean_scrub_stamp":"2024-09-21T10:13:58.411930+0000","objects_scrubbed":0,"log_size":6792,"log_dups_size":0,"ondisk_log_size":6792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:44.027603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445120512,"num_objects":6792,"num_object_clones":0,"num_object_copies":20376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6792,"num_write_kb":434688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":204,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.174445+0000","last_change":"2024-09-21T10:08:04.645700+0000","last_active":"2024-09-21T10:15:32.174445+0000","last_peered":"2024-09-21T10:15:32.174445+0000","last_clean":"2024-09-21T10:15:32.174445+0000","last_became_active":"2024-09-21T10:07:57.548189+0000","last_became_peered":"2024-09-21T10:07:57.548189+0000","last_unstale":"2024-09-21T10:15:32.174445+0000","last_undegraded":"2024-09-21T10:15:32.174445+0000","last_fullsized":"2024-09-21T10:15:32.174445+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:48.318010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:48.318010+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T16:31:28.173059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":482,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.167254+0000","last_change":"2024-09-21T10:03:38.386604+0000","last_active":"2024-09-21T10:15:32.167254+0000","last_peered":"2024-09-21T10:15:32.167254+0000","last_clean":"2024-09-21T10:15:32.167254+0000","last_became_active":"2024-09-21T10:03:38.386283+0000","last_became_peered":"2024-09-21T10:03:38.386283+0000","last_unstale":"2024-09-21T10:15:32.167254+0000","last_undegraded":"2024-09-21T10:15:32.167254+0000","last_fullsized":"2024-09-21T10:15:32.167254+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:27:05.944880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":341,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.162920+0000","last_change":"2024-09-21T10:14:30.709615+0000","last_active":"2024-09-21T10:15:32.162920+0000","last_peered":"2024-09-21T10:15:32.162920+0000","last_clean":"2024-09-21T10:15:32.162920+0000","last_became_active":"2024-09-21T10:14:30.709184+0000","last_became_peered":"2024-09-21T10:14:30.709184+0000","last_unstale":"2024-09-21T10:15:32.162920+0000","last_undegraded":"2024-09-21T10:15:32.162920+0000","last_fullsized":"2024-09-21T10:15:32.162920+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:55:27.220232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"313'6767","reported_seq":6793,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:36.877317+0000","last_change":"2024-09-21T10:13:59.475251+0000","last_active":"2024-09-21T10:15:36.877317+0000","last_peered":"2024-09-21T10:15:36.877317+0000","last_clean":"2024-09-21T10:15:36.877317+0000","last_became_active":"2024-09-21T10:13:59.474624+0000","last_became_peered":"2024-09-21T10:13:59.474624+0000","last_unstale":"2024-09-21T10:15:36.877317+0000","last_undegraded":"2024-09-21T10:15:36.877317+0000","last_fullsized":"2024-09-21T10:15:36.877317+0000","mapping_epoch":294,"log_start":"300'200","ondisk_log_start":"300'200","created":294,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_clean_scrub_stamp":"2024-09-21T10:13:58.411930+0000","objects_scrubbed":0,"log_size":6567,"log_dups_size":0,"ondisk_log_size":6567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:38:07.234445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443482112,"num_objects":6767,"num_object_clones":0,"num_object_copies":20301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6767,"num_write_kb":433088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":203,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.710008+0000","last_change":"2024-09-21T10:09:49.924286+0000","last_active":"2024-09-21T10:15:18.710008+0000","last_peered":"2024-09-21T10:15:18.710008+0000","last_clean":"2024-09-21T10:15:18.710008+0000","last_became_active":"2024-09-21T10:09:49.923897+0000","last_became_peered":"2024-09-21T10:09:49.923897+0000","last_unstale":"2024-09-21T10:15:18.710008+0000","last_undegraded":"2024-09-21T10:15:18.710008+0000","last_fullsized":"2024-09-21T10:15:18.710008+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:47.393146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:47.393146+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:44.898083+0000","scrub_duration":0.001601086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":237,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.158851+0000","last_change":"2024-09-21T10:10:57.402946+0000","last_active":"2024-09-21T10:15:32.158851+0000","last_peered":"2024-09-21T10:15:32.158851+0000","last_clean":"2024-09-21T10:15:32.158851+0000","last_became_active":"2024-09-21T10:10:57.395450+0000","last_became_peered":"2024-09-21T10:10:57.395450+0000","last_unstale":"2024-09-21T10:15:32.158851+0000","last_undegraded":"2024-09-21T10:15:32.158851+0000","last_fullsized":"2024-09-21T10:15:32.158851+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:08.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":467,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.171597+0000","last_change":"2024-09-21T10:10:57.390595+0000","last_active":"2024-09-21T10:15:32.171597+0000","last_peered":"2024-09-21T10:15:32.171597+0000","last_clean":"2024-09-21T10:15:32.171597+0000","last_became_active":"2024-09-21T10:10:57.390265+0000","last_became_peered":"2024-09-21T10:10:57.390265+0000","last_unstale":"2024-09-21T10:15:32.171597+0000","last_undegraded":"2024-09-21T10:15:32.171597+0000","last_fullsized":"2024-09-21T10:15:32.171597+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:37.485063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":329,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.718167+0000","last_change":"2024-09-21T10:14:30.711719+0000","last_active":"2024-09-21T10:15:18.718167+0000","last_peered":"2024-09-21T10:15:18.718167+0000","last_clean":"2024-09-21T10:15:18.718167+0000","last_became_active":"2024-09-21T10:14:30.711452+0000","last_became_peered":"2024-09-21T10:14:30.711452+0000","last_unstale":"2024-09-21T10:15:18.718167+0000","last_undegraded":"2024-09-21T10:15:18.718167+0000","last_fullsized":"2024-09-21T10:15:18.718167+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:46.907528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"313'6745","reported_seq":12024,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:37.345418+0000","last_change":"2024-09-21T10:15:18.716809+0000","last_active":"2024-09-21T10:15:37.345418+0000","last_peered":"2024-09-21T10:15:37.345418+0000","last_clean":"2024-09-21T10:15:37.345418+0000","last_became_active":"2024-09-21T10:15:18.711139+0000","last_became_peered":"2024-09-21T10:15:18.711139+0000","last_unstale":"2024-09-21T10:15:37.345418+0000","last_undegraded":"2024-09-21T10:15:37.345418+0000","last_fullsized":"2024-09-21T10:15:37.345418+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":294,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_clean_scrub_stamp":"2024-09-21T10:13:58.411930+0000","objects_scrubbed":0,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:35:38.464686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442040320,"num_objects":6745,"num_object_clones":0,"num_object_copies":20235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6745,"num_write_kb":431680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2610,"num_bytes_recovered":171048960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":143,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.161866+0000","last_change":"2024-09-21T10:14:30.701644+0000","last_active":"2024-09-21T10:15:32.161866+0000","last_peered":"2024-09-21T10:15:32.161866+0000","last_clean":"2024-09-21T10:15:32.161866+0000","last_became_active":"2024-09-21T10:14:30.701301+0000","last_became_peered":"2024-09-21T10:14:30.701301+0000","last_unstale":"2024-09-21T10:15:32.161866+0000","last_undegraded":"2024-09-21T10:15:32.161866+0000","last_fullsized":"2024-09-21T10:15:32.161866+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.550551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.550551+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:23.637695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":177,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.709989+0000","last_change":"2024-09-21T10:09:23.838944+0000","last_active":"2024-09-21T10:15:18.709989+0000","last_peered":"2024-09-21T10:15:18.709989+0000","last_clean":"2024-09-21T10:15:18.709989+0000","last_became_active":"2024-09-21T10:09:23.837886+0000","last_became_peered":"2024-09-21T10:09:23.837886+0000","last_unstale":"2024-09-21T10:15:18.709989+0000","last_undegraded":"2024-09-21T10:15:18.709989+0000","last_fullsized":"2024-09-21T10:15:18.709989+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:16.928195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":359,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.169173+0000","last_change":"2024-09-21T10:09:24.839770+0000","last_active":"2024-09-21T10:15:32.169173+0000","last_peered":"2024-09-21T10:15:32.169173+0000","last_clean":"2024-09-21T10:15:32.169173+0000","last_became_active":"2024-09-21T10:09:24.839479+0000","last_became_peered":"2024-09-21T10:09:24.839479+0000","last_unstale":"2024-09-21T10:15:32.169173+0000","last_undegraded":"2024-09-21T10:15:32.169173+0000","last_fullsized":"2024-09-21T10:15:32.169173+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:11:19.618193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":127,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.169681+0000","last_change":"2024-09-21T10:09:49.923044+0000","last_active":"2024-09-21T10:15:32.169681+0000","last_peered":"2024-09-21T10:15:32.169681+0000","last_clean":"2024-09-21T10:15:32.169681+0000","last_became_active":"2024-09-21T10:09:49.922711+0000","last_became_peered":"2024-09-21T10:09:49.922711+0000","last_unstale":"2024-09-21T10:15:32.169681+0000","last_undegraded":"2024-09-21T10:15:32.169681+0000","last_fullsized":"2024-09-21T10:15:32.169681+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:14.090084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.f","version":"313'6650","reported_seq":6676,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.903555+0000","last_change":"2024-09-21T10:13:59.474119+0000","last_active":"2024-09-21T10:15:32.903555+0000","last_peered":"2024-09-21T10:15:32.903555+0000","last_clean":"2024-09-21T10:15:32.903555+0000","last_became_active":"2024-09-21T10:13:59.473666+0000","last_became_peered":"2024-09-21T10:13:59.473666+0000","last_unstale":"2024-09-21T10:15:32.903555+0000","last_undegraded":"2024-09-21T10:15:32.903555+0000","last_fullsized":"2024-09-21T10:15:32.903555+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":294,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_clean_scrub_stamp":"2024-09-21T10:13:58.411930+0000","objects_scrubbed":0,"log_size":6650,"log_dups_size":0,"ondisk_log_size":6650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:01.029758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435814400,"num_objects":6650,"num_object_clones":0,"num_object_copies":19950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6650,"num_write_kb":425600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":217,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.710417+0000","last_change":"2024-09-21T10:09:24.838434+0000","last_active":"2024-09-21T10:15:18.710417+0000","last_peered":"2024-09-21T10:15:18.710417+0000","last_clean":"2024-09-21T10:15:18.710417+0000","last_became_active":"2024-09-21T10:09:24.838066+0000","last_became_peered":"2024-09-21T10:09:24.838066+0000","last_unstale":"2024-09-21T10:15:18.710417+0000","last_undegraded":"2024-09-21T10:15:18.710417+0000","last_fullsized":"2024-09-21T10:15:18.710417+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:52.361137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:52.361137+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:19:57.788195+0000","scrub_duration":0.001255358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":520,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.159127+0000","last_change":"2024-09-21T10:09:23.837859+0000","last_active":"2024-09-21T10:15:32.159127+0000","last_peered":"2024-09-21T10:15:32.159127+0000","last_clean":"2024-09-21T10:15:32.159127+0000","last_became_active":"2024-09-21T10:09:23.837488+0000","last_became_peered":"2024-09-21T10:09:23.837488+0000","last_unstale":"2024-09-21T10:15:32.159127+0000","last_undegraded":"2024-09-21T10:15:32.159127+0000","last_fullsized":"2024-09-21T10:15:32.159127+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:25.215501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":61,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.709739+0000","last_change":"2024-09-21T10:10:27.806070+0000","last_active":"2024-09-21T10:15:18.709739+0000","last_peered":"2024-09-21T10:15:18.709739+0000","last_clean":"2024-09-21T10:15:18.709739+0000","last_became_active":"2024-09-21T10:10:27.805718+0000","last_became_peered":"2024-09-21T10:10:27.805718+0000","last_unstale":"2024-09-21T10:15:18.709739+0000","last_undegraded":"2024-09-21T10:15:18.709739+0000","last_fullsized":"2024-09-21T10:15:18.709739+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:31.144118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":514,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.157832+0000","last_change":"2024-09-21T10:10:57.395073+0000","last_active":"2024-09-21T10:15:32.157832+0000","last_peered":"2024-09-21T10:15:32.157832+0000","last_clean":"2024-09-21T10:15:32.157832+0000","last_became_active":"2024-09-21T10:10:57.394674+0000","last_became_peered":"2024-09-21T10:10:57.394674+0000","last_unstale":"2024-09-21T10:15:32.157832+0000","last_undegraded":"2024-09-21T10:15:32.157832+0000","last_fullsized":"2024-09-21T10:15:32.157832+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:48:48.912649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.c","version":"313'6702","reported_seq":6750,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:37.338657+0000","last_change":"2024-09-21T10:13:59.474663+0000","last_active":"2024-09-21T10:15:37.338657+0000","last_peered":"2024-09-21T10:15:37.338657+0000","last_clean":"2024-09-21T10:15:37.338657+0000","last_became_active":"2024-09-21T10:13:59.474400+0000","last_became_peered":"2024-09-21T10:13:59.474400+0000","last_unstale":"2024-09-21T10:15:37.338657+0000","last_undegraded":"2024-09-21T10:15:37.338657+0000","last_fullsized":"2024-09-21T10:15:37.338657+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":294,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_clean_scrub_stamp":"2024-09-21T10:13:58.411930+0000","objects_scrubbed":0,"log_size":6702,"log_dups_size":0,"ondisk_log_size":6702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:10:47.276866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439222272,"num_objects":6702,"num_object_clones":0,"num_object_copies":20106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6702,"num_write_kb":428928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":382,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.168640+0000","last_change":"2024-09-21T10:10:57.398437+0000","last_active":"2024-09-21T10:15:32.168640+0000","last_peered":"2024-09-21T10:15:32.168640+0000","last_clean":"2024-09-21T10:15:32.168640+0000","last_became_active":"2024-09-21T10:10:57.397441+0000","last_became_peered":"2024-09-21T10:10:57.397441+0000","last_unstale":"2024-09-21T10:15:32.168640+0000","last_undegraded":"2024-09-21T10:15:32.168640+0000","last_fullsized":"2024-09-21T10:15:32.168640+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:04.043917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:04.043917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:53:31.813612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":75,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.161593+0000","last_change":"2024-09-21T10:14:05.472790+0000","last_active":"2024-09-21T10:15:32.161593+0000","last_peered":"2024-09-21T10:15:32.161593+0000","last_clean":"2024-09-21T10:15:32.161593+0000","last_became_active":"2024-09-21T10:14:05.472238+0000","last_became_peered":"2024-09-21T10:14:05.472238+0000","last_unstale":"2024-09-21T10:15:32.161593+0000","last_undegraded":"2024-09-21T10:15:32.161593+0000","last_fullsized":"2024-09-21T10:15:32.161593+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:56.561032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.709394+0000","last_change":"2024-09-21T10:14:30.709652+0000","last_active":"2024-09-21T10:15:18.709394+0000","last_peered":"2024-09-21T10:15:18.709394+0000","last_clean":"2024-09-21T10:15:18.709394+0000","last_became_active":"2024-09-21T10:14:30.705706+0000","last_became_peered":"2024-09-21T10:14:30.705706+0000","last_unstale":"2024-09-21T10:15:18.709394+0000","last_undegraded":"2024-09-21T10:15:18.709394+0000","last_fullsized":"2024-09-21T10:15:18.709394+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:24:25.749923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":346,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.161699+0000","last_change":"2024-09-21T10:14:05.477875+0000","last_active":"2024-09-21T10:15:32.161699+0000","last_peered":"2024-09-21T10:15:32.161699+0000","last_clean":"2024-09-21T10:15:32.161699+0000","last_became_active":"2024-09-21T10:14:05.477192+0000","last_became_peered":"2024-09-21T10:14:05.477192+0000","last_unstale":"2024-09-21T10:15:32.161699+0000","last_undegraded":"2024-09-21T10:15:32.161699+0000","last_fullsized":"2024-09-21T10:15:32.161699+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:19:47.345673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.a","version":"313'6697","reported_seq":6746,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:34.456858+0000","last_change":"2024-09-21T10:13:59.431959+0000","last_active":"2024-09-21T10:15:34.456858+0000","last_peered":"2024-09-21T10:15:34.456858+0000","last_clean":"2024-09-21T10:15:34.456858+0000","last_became_active":"2024-09-21T10:13:59.431631+0000","last_became_peered":"2024-09-21T10:13:59.431631+0000","last_unstale":"2024-09-21T10:15:34.456858+0000","last_undegraded":"2024-09-21T10:15:34.456858+0000","last_fullsized":"2024-09-21T10:15:34.456858+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":294,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_clean_scrub_stamp":"2024-09-21T10:13:58.411930+0000","objects_scrubbed":0,"log_size":6697,"log_dups_size":0,"ondisk_log_size":6697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:07:04.427934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438894592,"num_objects":6697,"num_object_clones":0,"num_object_copies":20091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6697,"num_write_kb":428608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":286,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.738543+0000","last_change":"2024-09-21T10:04:45.242148+0000","last_active":"2024-09-21T10:15:18.738543+0000","last_peered":"2024-09-21T10:15:18.738543+0000","last_clean":"2024-09-21T10:15:18.738543+0000","last_became_active":"2024-09-21T10:04:42.549844+0000","last_became_peered":"2024-09-21T10:04:42.549844+0000","last_unstale":"2024-09-21T10:15:18.738543+0000","last_undegraded":"2024-09-21T10:15:18.738543+0000","last_fullsized":"2024-09-21T10:15:18.738543+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_deep_scrub":"77'100","last_deep_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.242073+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:03:42.250480+0000","scrub_duration":0.0023466239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":189,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.163692+0000","last_change":"2024-09-21T10:15:01.489326+0000","last_active":"2024-09-21T10:15:32.163692+0000","last_peered":"2024-09-21T10:15:32.163692+0000","last_clean":"2024-09-21T10:15:32.163692+0000","last_became_active":"2024-09-21T10:15:01.489070+0000","last_became_peered":"2024-09-21T10:15:01.489070+0000","last_unstale":"2024-09-21T10:15:32.163692+0000","last_undegraded":"2024-09-21T10:15:32.163692+0000","last_fullsized":"2024-09-21T10:15:32.163692+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:27:41.788029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.161893+0000","last_change":"2024-09-21T10:14:05.475970+0000","last_active":"2024-09-21T10:15:32.161893+0000","last_peered":"2024-09-21T10:15:32.161893+0000","last_clean":"2024-09-21T10:15:32.161893+0000","last_became_active":"2024-09-21T10:14:05.475331+0000","last_became_peered":"2024-09-21T10:14:05.475331+0000","last_unstale":"2024-09-21T10:15:32.161893+0000","last_undegraded":"2024-09-21T10:15:32.161893+0000","last_fullsized":"2024-09-21T10:15:32.161893+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:17.069736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":331,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.161847+0000","last_change":"2024-09-21T10:14:05.473630+0000","last_active":"2024-09-21T10:15:32.161847+0000","last_peered":"2024-09-21T10:15:32.161847+0000","last_clean":"2024-09-21T10:15:32.161847+0000","last_became_active":"2024-09-21T10:14:05.473346+0000","last_became_peered":"2024-09-21T10:14:05.473346+0000","last_unstale":"2024-09-21T10:15:32.161847+0000","last_undegraded":"2024-09-21T10:15:32.161847+0000","last_fullsized":"2024-09-21T10:15:32.161847+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:52:53.276888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.b","version":"313'7047","reported_seq":7096,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:37.093045+0000","last_change":"2024-09-21T10:13:59.431657+0000","last_active":"2024-09-21T10:15:37.093045+0000","last_peered":"2024-09-21T10:15:37.093045+0000","last_clean":"2024-09-21T10:15:37.093045+0000","last_became_active":"2024-09-21T10:13:59.431370+0000","last_became_peered":"2024-09-21T10:13:59.431370+0000","last_unstale":"2024-09-21T10:15:37.093045+0000","last_undegraded":"2024-09-21T10:15:37.093045+0000","last_fullsized":"2024-09-21T10:15:37.093045+0000","mapping_epoch":294,"log_start":"301'600","ondisk_log_start":"301'600","created":294,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_clean_scrub_stamp":"2024-09-21T10:13:58.411930+0000","objects_scrubbed":0,"log_size":6447,"log_dups_size":0,"ondisk_log_size":6447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:20:24.780959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":461832192,"num_objects":7047,"num_object_clones":0,"num_object_copies":21141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7047,"num_write_kb":451008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":303,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.734940+0000","last_change":"2024-09-21T10:09:23.839563+0000","last_active":"2024-09-21T10:15:18.734940+0000","last_peered":"2024-09-21T10:15:18.734940+0000","last_clean":"2024-09-21T10:15:18.734940+0000","last_became_active":"2024-09-21T10:09:23.839207+0000","last_became_peered":"2024-09-21T10:09:23.839207+0000","last_unstale":"2024-09-21T10:15:18.734940+0000","last_undegraded":"2024-09-21T10:15:18.734940+0000","last_fullsized":"2024-09-21T10:15:18.734940+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:49.226496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:49.226496+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:03.960125+0000","scrub_duration":0.001656865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":334,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.161787+0000","last_change":"2024-09-21T10:14:05.478152+0000","last_active":"2024-09-21T10:15:32.161787+0000","last_peered":"2024-09-21T10:15:32.161787+0000","last_clean":"2024-09-21T10:15:32.161787+0000","last_became_active":"2024-09-21T10:14:05.477547+0000","last_became_peered":"2024-09-21T10:14:05.477547+0000","last_unstale":"2024-09-21T10:15:32.161787+0000","last_undegraded":"2024-09-21T10:15:32.161787+0000","last_fullsized":"2024-09-21T10:15:32.161787+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:34.646046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":207,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.709782+0000","last_change":"2024-09-21T10:14:30.708090+0000","last_active":"2024-09-21T10:15:18.709782+0000","last_peered":"2024-09-21T10:15:18.709782+0000","last_clean":"2024-09-21T10:15:18.709782+0000","last_became_active":"2024-09-21T10:14:30.707266+0000","last_became_peered":"2024-09-21T10:14:30.707266+0000","last_unstale":"2024-09-21T10:15:18.709782+0000","last_undegraded":"2024-09-21T10:15:18.709782+0000","last_fullsized":"2024-09-21T10:15:18.709782+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:38.474537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":397,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.177280+0000","last_change":"2024-09-21T10:10:57.397251+0000","last_active":"2024-09-21T10:15:32.177280+0000","last_peered":"2024-09-21T10:15:32.177280+0000","last_clean":"2024-09-21T10:15:32.177280+0000","last_became_active":"2024-09-21T10:10:57.396271+0000","last_became_peered":"2024-09-21T10:10:57.396271+0000","last_unstale":"2024-09-21T10:15:32.177280+0000","last_undegraded":"2024-09-21T10:15:32.177280+0000","last_fullsized":"2024-09-21T10:15:32.177280+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:17.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"313'6743","reported_seq":6792,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:37.149051+0000","last_change":"2024-09-21T10:13:59.429041+0000","last_active":"2024-09-21T10:15:37.149051+0000","last_peered":"2024-09-21T10:15:37.149051+0000","last_clean":"2024-09-21T10:15:37.149051+0000","last_became_active":"2024-09-21T10:13:59.428723+0000","last_became_peered":"2024-09-21T10:13:59.428723+0000","last_unstale":"2024-09-21T10:15:37.149051+0000","last_undegraded":"2024-09-21T10:15:37.149051+0000","last_fullsized":"2024-09-21T10:15:37.149051+0000","mapping_epoch":294,"log_start":"300'300","ondisk_log_start":"300'300","created":294,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_clean_scrub_stamp":"2024-09-21T10:13:58.411930+0000","objects_scrubbed":0,"log_size":6443,"log_dups_size":0,"ondisk_log_size":6443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:27:11.790424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441909248,"num_objects":6743,"num_object_clones":0,"num_object_copies":20229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6743,"num_write_kb":431552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":19,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.709786+0000","last_change":"2024-09-21T10:14:30.709001+0000","last_active":"2024-09-21T10:15:18.709786+0000","last_peered":"2024-09-21T10:15:18.709786+0000","last_clean":"2024-09-21T10:15:18.709786+0000","last_became_active":"2024-09-21T10:14:30.708636+0000","last_became_peered":"2024-09-21T10:14:30.708636+0000","last_unstale":"2024-09-21T10:15:18.709786+0000","last_undegraded":"2024-09-21T10:15:18.709786+0000","last_fullsized":"2024-09-21T10:15:18.709786+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:59.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:59.099852+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:20:01.949185+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":488,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.163951+0000","last_change":"2024-09-21T10:09:23.841186+0000","last_active":"2024-09-21T10:15:32.163951+0000","last_peered":"2024-09-21T10:15:32.163951+0000","last_clean":"2024-09-21T10:15:32.163951+0000","last_became_active":"2024-09-21T10:09:23.840854+0000","last_became_peered":"2024-09-21T10:09:23.840854+0000","last_unstale":"2024-09-21T10:15:32.163951+0000","last_undegraded":"2024-09-21T10:15:32.163951+0000","last_fullsized":"2024-09-21T10:15:32.163951+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:29.666358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":341,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.163672+0000","last_change":"2024-09-21T10:14:30.708385+0000","last_active":"2024-09-21T10:15:32.163672+0000","last_peered":"2024-09-21T10:15:32.163672+0000","last_clean":"2024-09-21T10:15:32.163672+0000","last_became_active":"2024-09-21T10:14:30.708046+0000","last_became_peered":"2024-09-21T10:14:30.708046+0000","last_unstale":"2024-09-21T10:15:32.163672+0000","last_undegraded":"2024-09-21T10:15:32.163672+0000","last_fullsized":"2024-09-21T10:15:32.163672+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:16:45.563095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":485,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.166854+0000","last_change":"2024-09-21T10:09:49.926548+0000","last_active":"2024-09-21T10:15:32.166854+0000","last_peered":"2024-09-21T10:15:32.166854+0000","last_clean":"2024-09-21T10:15:32.166854+0000","last_became_active":"2024-09-21T10:09:49.926255+0000","last_became_peered":"2024-09-21T10:09:49.926255+0000","last_unstale":"2024-09-21T10:15:32.166854+0000","last_undegraded":"2024-09-21T10:15:32.166854+0000","last_fullsized":"2024-09-21T10:15:32.166854+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:49.855878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.9","version":"313'6649","reported_seq":6698,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:34.475006+0000","last_change":"2024-09-21T10:13:59.431732+0000","last_active":"2024-09-21T10:15:34.475006+0000","last_peered":"2024-09-21T10:15:34.475006+0000","last_clean":"2024-09-21T10:15:34.475006+0000","last_became_active":"2024-09-21T10:13:59.431430+0000","last_became_peered":"2024-09-21T10:13:59.431430+0000","last_unstale":"2024-09-21T10:15:34.475006+0000","last_undegraded":"2024-09-21T10:15:34.475006+0000","last_fullsized":"2024-09-21T10:15:34.475006+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":294,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:13:58.411930+0000","last_clean_scrub_stamp":"2024-09-21T10:13:58.411930+0000","objects_scrubbed":0,"log_size":6649,"log_dups_size":0,"ondisk_log_size":6649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:17:30.295388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435748864,"num_objects":6649,"num_object_clones":0,"num_object_copies":19947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6649,"num_write_kb":425536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":287,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.164108+0000","last_change":"2024-09-21T10:10:57.390804+0000","last_active":"2024-09-21T10:15:32.164108+0000","last_peered":"2024-09-21T10:15:32.164108+0000","last_clean":"2024-09-21T10:15:32.164108+0000","last_became_active":"2024-09-21T10:10:57.390323+0000","last_became_peered":"2024-09-21T10:10:57.390323+0000","last_unstale":"2024-09-21T10:15:32.164108+0000","last_undegraded":"2024-09-21T10:15:32.164108+0000","last_fullsized":"2024-09-21T10:15:32.164108+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.005477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.005477+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:29:57.702046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":142,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.169776+0000","last_change":"2024-09-21T10:14:30.718942+0000","last_active":"2024-09-21T10:15:32.169776+0000","last_peered":"2024-09-21T10:15:32.169776+0000","last_clean":"2024-09-21T10:15:32.169776+0000","last_became_active":"2024-09-21T10:14:30.709129+0000","last_became_peered":"2024-09-21T10:14:30.709129+0000","last_unstale":"2024-09-21T10:15:32.169776+0000","last_undegraded":"2024-09-21T10:15:32.169776+0000","last_fullsized":"2024-09-21T10:15:32.169776+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:18.171064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":325,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.738792+0000","last_change":"2024-09-21T10:03:38.392767+0000","last_active":"2024-09-21T10:15:18.738792+0000","last_peered":"2024-09-21T10:15:18.738792+0000","last_clean":"2024-09-21T10:15:18.738792+0000","last_became_active":"2024-09-21T10:03:38.392424+0000","last_became_peered":"2024-09-21T10:03:38.392424+0000","last_unstale":"2024-09-21T10:15:18.738792+0000","last_undegraded":"2024-09-21T10:15:18.738792+0000","last_fullsized":"2024-09-21T10:15:18.738792+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:25.358291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.709241+0000","last_change":"2024-09-21T10:14:30.703352+0000","last_active":"2024-09-21T10:15:18.709241+0000","last_peered":"2024-09-21T10:15:18.709241+0000","last_clean":"2024-09-21T10:15:18.709241+0000","last_became_active":"2024-09-21T10:14:30.703068+0000","last_became_peered":"2024-09-21T10:14:30.703068+0000","last_unstale":"2024-09-21T10:15:18.709241+0000","last_undegraded":"2024-09-21T10:15:18.709241+0000","last_fullsized":"2024-09-21T10:15:18.709241+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:00:26.966633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.709304+0000","last_change":"2024-09-21T10:14:30.702750+0000","last_active":"2024-09-21T10:15:18.709304+0000","last_peered":"2024-09-21T10:15:18.709304+0000","last_clean":"2024-09-21T10:15:18.709304+0000","last_became_active":"2024-09-21T10:14:30.702443+0000","last_became_peered":"2024-09-21T10:14:30.702443+0000","last_unstale":"2024-09-21T10:15:18.709304+0000","last_undegraded":"2024-09-21T10:15:18.709304+0000","last_fullsized":"2024-09-21T10:15:18.709304+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:38:19.353586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":29,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.710859+0000","last_change":"2024-09-21T10:14:30.716754+0000","last_active":"2024-09-21T10:15:18.710859+0000","last_peered":"2024-09-21T10:15:18.710859+0000","last_clean":"2024-09-21T10:15:18.710859+0000","last_became_active":"2024-09-21T10:14:30.704683+0000","last_became_peered":"2024-09-21T10:14:30.704683+0000","last_unstale":"2024-09-21T10:15:18.710859+0000","last_undegraded":"2024-09-21T10:15:18.710859+0000","last_fullsized":"2024-09-21T10:15:18.710859+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:45.993877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":393,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.161594+0000","last_change":"2024-09-21T10:14:05.477429+0000","last_active":"2024-09-21T10:15:32.161594+0000","last_peered":"2024-09-21T10:15:32.161594+0000","last_clean":"2024-09-21T10:15:32.161594+0000","last_became_active":"2024-09-21T10:14:05.476794+0000","last_became_peered":"2024-09-21T10:14:05.476794+0000","last_unstale":"2024-09-21T10:15:32.161594+0000","last_undegraded":"2024-09-21T10:15:32.161594+0000","last_fullsized":"2024-09-21T10:15:32.161594+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:18:54.963704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":137,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.167842+0000","last_change":"2024-09-21T10:14:30.713873+0000","last_active":"2024-09-21T10:15:32.167842+0000","last_peered":"2024-09-21T10:15:32.167842+0000","last_clean":"2024-09-21T10:15:32.167842+0000","last_became_active":"2024-09-21T10:14:30.703149+0000","last_became_peered":"2024-09-21T10:14:30.703149+0000","last_unstale":"2024-09-21T10:15:32.167842+0000","last_undegraded":"2024-09-21T10:15:32.167842+0000","last_fullsized":"2024-09-21T10:15:32.167842+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:26.453902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'100","reported_seq":279,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.157662+0000","last_change":"2024-09-21T10:07:24.345428+0000","last_active":"2024-09-21T10:15:32.157662+0000","last_peered":"2024-09-21T10:15:32.157662+0000","last_clean":"2024-09-21T10:15:32.157662+0000","last_became_active":"2024-09-21T10:07:24.343529+0000","last_became_peered":"2024-09-21T10:07:24.343529+0000","last_unstale":"2024-09-21T10:15:32.157662+0000","last_undegraded":"2024-09-21T10:15:32.157662+0000","last_fullsized":"2024-09-21T10:15:32.157662+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.014993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.014993+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:35.677888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":226,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.710520+0000","last_change":"2024-09-21T10:15:01.483814+0000","last_active":"2024-09-21T10:15:18.710520+0000","last_peered":"2024-09-21T10:15:18.710520+0000","last_clean":"2024-09-21T10:15:18.710520+0000","last_became_active":"2024-09-21T10:15:01.483522+0000","last_became_peered":"2024-09-21T10:15:01.483522+0000","last_unstale":"2024-09-21T10:15:18.710520+0000","last_undegraded":"2024-09-21T10:15:18.710520+0000","last_fullsized":"2024-09-21T10:15:18.710520+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:13.632066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":471,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.167651+0000","last_change":"2024-09-21T10:03:38.425111+0000","last_active":"2024-09-21T10:15:32.167651+0000","last_peered":"2024-09-21T10:15:32.167651+0000","last_clean":"2024-09-21T10:15:32.167651+0000","last_became_active":"2024-09-21T10:03:38.424704+0000","last_became_peered":"2024-09-21T10:03:38.424704+0000","last_unstale":"2024-09-21T10:15:32.167651+0000","last_undegraded":"2024-09-21T10:15:32.167651+0000","last_fullsized":"2024-09-21T10:15:32.167651+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:35.157076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":371,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.169607+0000","last_change":"2024-09-21T10:10:57.397755+0000","last_active":"2024-09-21T10:15:32.169607+0000","last_peered":"2024-09-21T10:15:32.169607+0000","last_clean":"2024-09-21T10:15:32.169607+0000","last_became_active":"2024-09-21T10:10:57.397249+0000","last_became_peered":"2024-09-21T10:10:57.397249+0000","last_unstale":"2024-09-21T10:15:32.169607+0000","last_undegraded":"2024-09-21T10:15:32.169607+0000","last_fullsized":"2024-09-21T10:15:32.169607+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:40.813503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":111,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.161254+0000","last_change":"2024-09-21T10:14:05.471573+0000","last_active":"2024-09-21T10:15:32.161254+0000","last_peered":"2024-09-21T10:15:32.161254+0000","last_clean":"2024-09-21T10:15:32.161254+0000","last_became_active":"2024-09-21T10:14:05.471270+0000","last_became_peered":"2024-09-21T10:14:05.471270+0000","last_unstale":"2024-09-21T10:15:32.161254+0000","last_undegraded":"2024-09-21T10:15:32.161254+0000","last_fullsized":"2024-09-21T10:15:32.161254+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:56:13.468768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":275,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.738726+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T10:15:18.738726+0000","last_peered":"2024-09-21T10:15:18.738726+0000","last_clean":"2024-09-21T10:15:18.738726+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T10:15:18.738726+0000","last_undegraded":"2024-09-21T10:15:18.738726+0000","last_fullsized":"2024-09-21T10:15:18.738726+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":492,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.167171+0000","last_change":"2024-09-21T10:09:49.928053+0000","last_active":"2024-09-21T10:15:32.167171+0000","last_peered":"2024-09-21T10:15:32.167171+0000","last_clean":"2024-09-21T10:15:32.167171+0000","last_became_active":"2024-09-21T10:09:49.927231+0000","last_became_peered":"2024-09-21T10:09:49.927231+0000","last_unstale":"2024-09-21T10:15:32.167171+0000","last_undegraded":"2024-09-21T10:15:32.167171+0000","last_fullsized":"2024-09-21T10:15:32.167171+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:26.987553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":404,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.169921+0000","last_change":"2024-09-21T10:09:23.840717+0000","last_active":"2024-09-21T10:15:32.169921+0000","last_peered":"2024-09-21T10:15:32.169921+0000","last_clean":"2024-09-21T10:15:32.169921+0000","last_became_active":"2024-09-21T10:09:23.840323+0000","last_became_peered":"2024-09-21T10:09:23.840323+0000","last_unstale":"2024-09-21T10:15:32.169921+0000","last_undegraded":"2024-09-21T10:15:32.169921+0000","last_fullsized":"2024-09-21T10:15:32.169921+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:55.769770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":118,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.158877+0000","last_change":"2024-09-21T10:10:27.816526+0000","last_active":"2024-09-21T10:15:32.158877+0000","last_peered":"2024-09-21T10:15:32.158877+0000","last_clean":"2024-09-21T10:15:32.158877+0000","last_became_active":"2024-09-21T10:10:27.816221+0000","last_became_peered":"2024-09-21T10:10:27.816221+0000","last_unstale":"2024-09-21T10:15:32.158877+0000","last_undegraded":"2024-09-21T10:15:32.158877+0000","last_fullsized":"2024-09-21T10:15:32.158877+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:08.331228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":469,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.163550+0000","last_change":"2024-09-21T10:14:30.702101+0000","last_active":"2024-09-21T10:15:32.163550+0000","last_peered":"2024-09-21T10:15:32.163550+0000","last_clean":"2024-09-21T10:15:32.163550+0000","last_became_active":"2024-09-21T10:14:30.701783+0000","last_became_peered":"2024-09-21T10:14:30.701783+0000","last_unstale":"2024-09-21T10:15:32.163550+0000","last_undegraded":"2024-09-21T10:15:32.163550+0000","last_fullsized":"2024-09-21T10:15:32.163550+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:13.087010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":19,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.709294+0000","last_change":"2024-09-21T10:14:30.707503+0000","last_active":"2024-09-21T10:15:18.709294+0000","last_peered":"2024-09-21T10:15:18.709294+0000","last_clean":"2024-09-21T10:15:18.709294+0000","last_became_active":"2024-09-21T10:14:30.706881+0000","last_became_peered":"2024-09-21T10:14:30.706881+0000","last_unstale":"2024-09-21T10:15:18.709294+0000","last_undegraded":"2024-09-21T10:15:18.709294+0000","last_fullsized":"2024-09-21T10:15:18.709294+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:23:07.485461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":509,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.157754+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:15:32.157754+0000","last_peered":"2024-09-21T10:15:32.157754+0000","last_clean":"2024-09-21T10:15:32.157754+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:15:32.157754+0000","last_undegraded":"2024-09-21T10:15:32.157754+0000","last_fullsized":"2024-09-21T10:15:32.157754+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":134,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.174782+0000","last_change":"2024-09-21T10:14:30.702414+0000","last_active":"2024-09-21T10:15:32.174782+0000","last_peered":"2024-09-21T10:15:32.174782+0000","last_clean":"2024-09-21T10:15:32.174782+0000","last_became_active":"2024-09-21T10:14:30.702009+0000","last_became_peered":"2024-09-21T10:14:30.702009+0000","last_unstale":"2024-09-21T10:15:32.174782+0000","last_undegraded":"2024-09-21T10:15:32.174782+0000","last_fullsized":"2024-09-21T10:15:32.174782+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:50:48.396082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":318,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.170514+0000","last_change":"2024-09-21T10:07:24.347937+0000","last_active":"2024-09-21T10:15:32.170514+0000","last_peered":"2024-09-21T10:15:32.170514+0000","last_clean":"2024-09-21T10:15:32.170514+0000","last_became_active":"2024-09-21T10:07:24.347376+0000","last_became_peered":"2024-09-21T10:07:24.347376+0000","last_unstale":"2024-09-21T10:15:32.170514+0000","last_undegraded":"2024-09-21T10:15:32.170514+0000","last_fullsized":"2024-09-21T10:15:32.170514+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:48:05.644824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":23,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.710384+0000","last_change":"2024-09-21T10:14:30.720787+0000","last_active":"2024-09-21T10:15:18.710384+0000","last_peered":"2024-09-21T10:15:18.710384+0000","last_clean":"2024-09-21T10:15:18.710384+0000","last_became_active":"2024-09-21T10:14:30.706001+0000","last_became_peered":"2024-09-21T10:14:30.706001+0000","last_unstale":"2024-09-21T10:15:18.710384+0000","last_undegraded":"2024-09-21T10:15:18.710384+0000","last_fullsized":"2024-09-21T10:15:18.710384+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:40:11.987108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":409,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.171731+0000","last_change":"2024-09-21T10:09:24.840173+0000","last_active":"2024-09-21T10:15:32.171731+0000","last_peered":"2024-09-21T10:15:32.171731+0000","last_clean":"2024-09-21T10:15:32.171731+0000","last_became_active":"2024-09-21T10:09:24.839762+0000","last_became_peered":"2024-09-21T10:09:24.839762+0000","last_unstale":"2024-09-21T10:15:32.171731+0000","last_undegraded":"2024-09-21T10:15:32.171731+0000","last_fullsized":"2024-09-21T10:15:32.171731+0000","mapping_epoch":246,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":247,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:12:53.465203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":23,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.709506+0000","last_change":"2024-09-21T10:14:30.722338+0000","last_active":"2024-09-21T10:15:18.709506+0000","last_peered":"2024-09-21T10:15:18.709506+0000","last_clean":"2024-09-21T10:15:18.709506+0000","last_became_active":"2024-09-21T10:14:30.708164+0000","last_became_peered":"2024-09-21T10:14:30.708164+0000","last_unstale":"2024-09-21T10:15:18.709506+0000","last_undegraded":"2024-09-21T10:15:18.709506+0000","last_fullsized":"2024-09-21T10:15:18.709506+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:32:13.825114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":493,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.168000+0000","last_change":"2024-09-21T10:09:23.840467+0000","last_active":"2024-09-21T10:15:32.168000+0000","last_peered":"2024-09-21T10:15:32.168000+0000","last_clean":"2024-09-21T10:15:32.168000+0000","last_became_active":"2024-09-21T10:09:23.840177+0000","last_became_peered":"2024-09-21T10:09:23.840177+0000","last_unstale":"2024-09-21T10:15:32.168000+0000","last_undegraded":"2024-09-21T10:15:32.168000+0000","last_fullsized":"2024-09-21T10:15:32.168000+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:17.887307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":482,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.164104+0000","last_change":"2024-09-21T10:11:05.451613+0000","last_active":"2024-09-21T10:15:32.164104+0000","last_peered":"2024-09-21T10:15:32.164104+0000","last_clean":"2024-09-21T10:15:32.164104+0000","last_became_active":"2024-09-21T10:11:05.451075+0000","last_became_peered":"2024-09-21T10:11:05.451075+0000","last_unstale":"2024-09-21T10:15:32.164104+0000","last_undegraded":"2024-09-21T10:15:32.164104+0000","last_fullsized":"2024-09-21T10:15:32.164104+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:18.505045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-21T10:15:18.709142+0000","last_change":"2024-09-21T10:14:30.707548+0000","last_active":"2024-09-21T10:15:18.709142+0000","last_peered":"2024-09-21T10:15:18.709142+0000","last_clean":"2024-09-21T10:15:18.709142+0000","last_became_active":"2024-09-21T10:14:30.707232+0000","last_became_peered":"2024-09-21T10:14:30.707232+0000","last_unstale":"2024-09-21T10:15:18.709142+0000","last_undegraded":"2024-09-21T10:15:18.709142+0000","last_fullsized":"2024-09-21T10:15:18.709142+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:11:38.905767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":146,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-21T10:15:32.169548+0000","last_change":"2024-09-21T10:10:57.401588+0000","last_active":"2024-09-21T10:15:32.169548+0000","last_peered":"2024-09-21T10:15:32.169548+0000","last_clean":"2024-09-21T10:15:32.169548+0000","last_became_active":"2024-09-21T10:10:57.397724+0000","last_became_peered":"2024-09-21T10:10:57.397724+0000","last_unstale":"2024-09-21T10:15:32.169548+0000","last_undegraded":"2024-09-21T10:15:32.169548+0000","last_fullsized":"2024-09-21T10:15:32.169548+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:47.843630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":16,"stat_sum":{"num_bytes":7049183232,"num_objects":107562,"num_object_clones":0,"num_object_copies":322686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1883,"num_objects_unfound":0,"num_objects_dirty":107562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107562,"num_write_kb":6883968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3408,"num_bytes_recovered":223346688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21387411456,"data_stored":21387411456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106162,"ondisk_log_size":106162,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":11856,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26580,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":644037,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":632715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626878,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4045952,"kb_used_data":3137272,"kb_used_omap":1,"kb_used_meta":908670,"kb_avail":89699200,"statfs":{"total":95995035648,"available":91851980800,"internally_reserved":0,"allocated":3212566528,"data_stored":3210735407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":930478424},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019336,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1598920,"kb_used_data":699644,"kb_used_omap":1,"kb_used_meta":899262,"kb_avail":92146232,"statfs":{"total":95995035648,"available":94357741568,"internally_reserved":0,"allocated":716435456,"data_stored":714600280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920844632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":245,"seq":1052266987596,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2929844,"kb_used_data":2596204,"kb_used_omap":8,"kb_used_meta":333623,"kb_avail":90815308,"statfs":{"total":95995035648,"available":92994875392,"internally_reserved":0,"allocated":2658512896,"data_stored":2656689850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8940,"internal_metadata":341630228},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":175,"seq":751619276945,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4400260,"kb_used_data":3467132,"kb_used_omap":12,"kb_used_meta":933107,"kb_avail":89344892,"statfs":{"total":95995035648,"available":91489169408,"internally_reserved":0,"allocated":3550343168,"data_stored":3548507988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12845,"internal_metadata":955502035},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823107,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2334392,"kb_used_data":840748,"kb_used_omap":1,"kb_used_meta":1493630,"kb_avail":91410760,"statfs":{"total":95995035648,"available":93604618240,"internally_reserved":0,"allocated":860925952,"data_stored":859102859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1529477464},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879531,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4585332,"kb_used_data":3662444,"kb_used_omap":4,"kb_used_meta":922875,"kb_avail":89159820,"statfs":{"total":95995035648,"available":91299655680,"internally_reserved":0,"allocated":3750342656,"data_stored":3748519644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":945024444},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,5,5,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566371,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5085312,"kb_used_data":4265076,"kb_used_omap":6,"kb_used_meta":820217,"kb_avail":88659840,"statfs":{"total":95995035648,"available":90787676160,"internally_reserved":0,"allocated":4367437824,"data_stored":4365607232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":839902631},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860716,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3097004,"kb_used_data":2152036,"kb_used_omap":5,"kb_used_meta":944954,"kb_avail":90648148,"statfs":{"total":95995035648,"available":92823703552,"internally_reserved":0,"allocated":2203684864,"data_stored":2201869106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":967633135},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7236,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9976,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2671902720,"data_stored":2671902720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2208038912,"data_stored":2208038912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4393467904,"data_stored":4393467904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3748528128,"data_stored":3748528128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":716963840,"data_stored":716963840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":859897856,"data_stored":859897856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3222667264,"data_stored":3222667264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3565944832,"data_stored":3565944832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:15:40.904 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T10:15:40.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_7 pg_num 26 2024-09-21T10:15:40.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:41.042 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:41.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:41.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:41.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:41.208 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:41.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:41.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:41.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:41.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:41.358 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:41.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:41.459 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:41.488 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:41.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:41.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:41.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:41.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:41.625 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:41.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:41.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:41.774 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:41.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:41.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:41.876 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:41.906 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:41.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:41.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:42.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:42.037 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:42.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:42.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:42.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:15:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:15:41] "GET /metrics HTTP/1.1" 200 36818 "" "Prometheus/2.51.0" 2024-09-21T10:15:42.141 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:42.160 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:42.187 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:42.192 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:15:42.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:42.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:42.293 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:42.319 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:42.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:42.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:42.420 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:42.451 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:42.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:42.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:42.553 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:42.579 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:42.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:42.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:42 smithi107 ceph-mon[102221]: pgmap v1118: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 53 MiB/s wr, 863 op/s; 1883/329496 objects misplaced (0.571%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:15:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:42 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3078995891' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:15:42.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:42.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:42.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:42 smithi107 ceph-mon[105022]: pgmap v1118: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 53 MiB/s wr, 863 op/s; 1883/329496 objects misplaced (0.571%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:15:42.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:42 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3078995891' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:15:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:15:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:42 smithi129 ceph-mon[88471]: pgmap v1118: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 53 MiB/s wr, 863 op/s; 1883/329496 objects misplaced (0.571%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:15:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:42 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3078995891' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:15:42.682 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:42.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:42.710 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:42.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:42.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:42.857 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:42.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:42.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:42.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:43.003 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:43.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:43.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:43.105 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:43.219 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:43.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:43.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:43.267 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:43 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:15:43.267 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:43 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3078995891' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:15:43.267 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:43 smithi107 ceph-mon[102221]: osdmap e314: 8 total, 8 up, 8 in 2024-09-21T10:15:43.267 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:43 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:15:43.267 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:43 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3078995891' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:15:43.268 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:43 smithi107 ceph-mon[105022]: osdmap e314: 8 total, 8 up, 8 in 2024-09-21T10:15:43.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:43.357 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:43.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:43.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:43.459 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:43.487 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:43.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:43.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:43.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:43.626 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:43.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:43.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:43 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:15:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:43 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3078995891' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:15:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:43 smithi129 ceph-mon[88471]: osdmap e314: 8 total, 8 up, 8 in 2024-09-21T10:15:43.729 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:43.760 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:43.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:43.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:43.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:43.906 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:43.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:43.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:44.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:44.012 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:15:44.013 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:15:44.037 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:44.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:44.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:44.047 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:15:44.047 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:15:44.078 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:15:44.079 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:15:44.110 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:15:44.111 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:15:44.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:44.140 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:15:44.140 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:15:44.172 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:15:44.172 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:15:44.177 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:44.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:44.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:44.195 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:15:44.195 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:15:44.216 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:44.227 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:15:44.227 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:15:44.275 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:44 smithi107 ceph-mon[102221]: pgmap v1120: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 42 MiB/s wr, 856 op/s; 1685/328893 objects misplaced (0.512%); 2.4 MiB/s, 37 objects/s recovering 2024-09-21T10:15:44.275 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:44 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/908911069' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:44.275 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:44 smithi107 ceph-mon[105022]: pgmap v1120: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 42 MiB/s wr, 856 op/s; 1685/328893 objects misplaced (0.512%); 2.4 MiB/s, 37 objects/s recovering 2024-09-21T10:15:44.275 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:44 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/908911069' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:44.279 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:44.302 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:44.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:44.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:44.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:44.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:44.442 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:44.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:44.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:44.578 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:44.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:44.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:44 smithi129 ceph-mon[88471]: pgmap v1120: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 42 MiB/s wr, 856 op/s; 1685/328893 objects misplaced (0.512%); 2.4 MiB/s, 37 objects/s recovering 2024-09-21T10:15:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:44 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/908911069' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:15:44.680 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:44.709 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:44.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:44.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:44.811 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:44.838 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:44.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:44.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:44.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:44.979 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:44.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:44.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:45.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:45.106 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:45.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:45.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:45.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:45.242 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:45.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:45.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:45.344 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:45.381 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:45.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:45.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:45.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:45.522 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:45.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:45.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:45.624 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:45.649 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:45.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:45.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:45.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:15:45.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:45.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:45.819 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:45.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:45.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:45.982 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:45.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:45.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:46.084 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:46.111 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:46.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:46.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:46.213 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:46.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:46.242 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:46.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:46.343 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:46.367 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:46.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:46.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:46.469 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:46.495 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:46.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:46.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:46.597 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:46.634 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:46.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:46.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:46.736 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:46.766 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:46.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:46.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:46.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:46 smithi107 ceph-mon[102221]: pgmap v1121: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 39 MiB/s wr, 855 op/s; 1685/327576 objects misplaced (0.514%); 2.4 MiB/s, 37 objects/s recovering 2024-09-21T10:15:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:46 smithi107 ceph-mon[105022]: pgmap v1121: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 39 MiB/s wr, 855 op/s; 1685/327576 objects misplaced (0.514%); 2.4 MiB/s, 37 objects/s recovering 2024-09-21T10:15:46.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:46.900 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:46.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:46.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:46 smithi129 ceph-mon[88471]: pgmap v1121: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 39 MiB/s wr, 855 op/s; 1685/327576 objects misplaced (0.514%); 2.4 MiB/s, 37 objects/s recovering 2024-09-21T10:15:47.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:47.038 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:47.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:47.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:47.140 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:47.177 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:47.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:47.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:47.279 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:47.303 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:47.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:47.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:47.404 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:47.458 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:47.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:47.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:47.560 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:47.589 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:47.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:47.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:47.690 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:47.714 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:47.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:47.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:47.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:47.862 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:47.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:47.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:47.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:48.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:48.005 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:48.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:48.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:48.155 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:48.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:48.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:48.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:48.294 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:48.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:48.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:48.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:48.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:48.453 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:48.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:48.546 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:15:48.546 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:15:48.554 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:48.580 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:48.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:48.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:48.657 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:48 smithi107 ceph-mon[102221]: pgmap v1122: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 24 MiB/s wr, 851 op/s; 1462/320295 objects misplaced (0.456%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T10:15:48.658 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:48 smithi107 ceph-mon[105022]: pgmap v1122: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 24 MiB/s wr, 851 op/s; 1462/320295 objects misplaced (0.456%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T10:15:48.681 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:48.719 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:48.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:48.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:48.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:48.856 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:48.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:48.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:48.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:48 smithi129 ceph-mon[88471]: pgmap v1122: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 24 MiB/s wr, 851 op/s; 1462/320295 objects misplaced (0.456%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T10:15:48.957 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:48.986 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:48.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:48.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:49.088 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:49.115 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:49.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:49.217 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:49.245 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:49.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:49.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:49.347 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:49.376 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:49.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:49.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:49.477 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:49.506 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:49.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:49.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:49.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:49.653 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:49.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:49.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:49.756 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:49.788 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:49.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:49.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:49.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:15:49.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:15:49.889 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:49.914 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:49.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:50.016 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:50.040 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:50.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:50.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:50.142 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:50.165 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:50.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:50.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:50.267 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:50.292 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:50.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:50.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:50.356 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:50.394 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:50.451 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:50.465 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:15:50.476 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:50.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:50.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:50.561 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:50 smithi107 ceph-mon[102221]: pgmap v1123: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 16 MiB/s wr, 855 op/s; 1462/316113 objects misplaced (0.462%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T10:15:50.562 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:50 smithi107 ceph-mon[105022]: pgmap v1123: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 16 MiB/s wr, 855 op/s; 1462/316113 objects misplaced (0.462%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T10:15:50.578 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:50.629 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:50.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:50.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:50 smithi129 ceph-mon[88471]: pgmap v1123: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.4 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 16 MiB/s wr, 855 op/s; 1462/316113 objects misplaced (0.462%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T10:15:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:15:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:15:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:15:50.731 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:50.756 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:50.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:50.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:50.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:50.903 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:50.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:50.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:51.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:51.048 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:51.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:51.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:51.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:51.184 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:51.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:51.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:51.286 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:51.313 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:51.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:51.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:51.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:51.450 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:51.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:51.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:51.552 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:51.578 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:51.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:51.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:51.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:51.716 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:51.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:51.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:51.817 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:51.848 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:51.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:51.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:51.950 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:51.977 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:51.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:51.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:52.010 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:15:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:15:51] "GET /metrics HTTP/1.1" 200 36818 "" "Prometheus/2.51.0" 2024-09-21T10:15:52.079 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:52.108 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:52.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:52.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:52.209 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:52.233 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:52.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:52.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:52.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:52.363 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:52.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:52.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:52.465 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:52.490 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:52.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:52.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:52.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:52.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:52.626 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:52.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:52.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:52.767 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:52.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:52.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:52.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:52 smithi107 ceph-mon[102221]: pgmap v1124: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 9.5 MiB/s wr, 864 op/s; 1462/312726 objects misplaced (0.468%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T10:15:52.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:52 smithi107 ceph-mon[105022]: pgmap v1124: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 9.5 MiB/s wr, 864 op/s; 1462/312726 objects misplaced (0.468%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T10:15:52.868 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:52.898 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:52.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:52.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:52 smithi129 ceph-mon[88471]: pgmap v1124: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 9.5 MiB/s wr, 864 op/s; 1462/312726 objects misplaced (0.468%); 2.6 MiB/s, 42 objects/s recovering 2024-09-21T10:15:53.000 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:53.027 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:53.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:53.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:53.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:53.163 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:53.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:53.265 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:53.292 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:53.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:53.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:53.393 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:53.421 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:53.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:53.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:53.523 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:53.548 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:53.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:53.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:53.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:53.688 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:53.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:53.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:53.790 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:53.819 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:53.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:53.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:53.921 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:53.957 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:53.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:53.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:54.059 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:54.086 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:54.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:54.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:54.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:54.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:54.225 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:54.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:54.326 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:54.352 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:54.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:54.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:54.454 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:54.477 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:54.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:54.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:54.580 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:54.608 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:54.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:54.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:54.709 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:54.744 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:54.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:54.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:54.845 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:54 smithi107 ceph-mon[102221]: pgmap v1125: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 7.7 MiB/s wr, 1.04k op/s; 1225/302787 objects misplaced (0.405%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:15:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:54 smithi107 ceph-mon[105022]: pgmap v1125: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 7.7 MiB/s wr, 1.04k op/s; 1225/302787 objects misplaced (0.405%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:15:54.872 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:54.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:54.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:54 smithi129 ceph-mon[88471]: pgmap v1125: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 7.7 MiB/s wr, 1.04k op/s; 1225/302787 objects misplaced (0.405%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:15:54.974 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:55.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:55.007 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:55.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:55.108 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:55.135 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:55.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:55.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:55.237 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:55.263 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:55.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:55.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:55.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:55.399 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:55.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:55.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:55.501 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:55.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:55.527 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:55.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:55.628 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:55.663 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:55.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:55.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:55.764 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:55.791 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:55.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:55.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:55.892 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:55.919 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:55.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:55.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:56.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:56.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:56.066 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:56.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:56.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:56.202 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:56.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:56.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:56.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:56.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:56.344 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:56.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:56.444 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:56.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:56.469 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:56.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:56.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:56.607 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:56.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:56.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:56.709 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:56.737 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:56.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:56.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:56.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:56 smithi107 ceph-mon[102221]: pgmap v1126: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 187 KiB/s wr, 777 op/s; 1225/301101 objects misplaced (0.407%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:15:56.775 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:56.775 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:56 smithi107 ceph-mon[105022]: pgmap v1126: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 187 KiB/s wr, 777 op/s; 1225/301101 objects misplaced (0.407%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:15:56.775 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:56.838 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:56.866 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:56.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:56.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:56 smithi129 ceph-mon[88471]: pgmap v1126: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 187 KiB/s wr, 777 op/s; 1225/301101 objects misplaced (0.407%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:15:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:56.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:57.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:57.018 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:57.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:57.119 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:57.144 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:57.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:57.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:57.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:57.290 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:57.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:57.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:57.391 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:57.417 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:57.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:57.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:57.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:57.563 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:57.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:57.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:57.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:15:57.707 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:57.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:57.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:15:57.809 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:57.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:57.844 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:57.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:57.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:57.986 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:57.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:57.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:58.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:58.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:58.123 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:58.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:58.224 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:58.263 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:58.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:58.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:58.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:15:58.403 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:58.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:58.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:15:58.505 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:58.530 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:58.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:58.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:58.632 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:58.657 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:58.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:58.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:58.759 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:58.785 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:58.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:58.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:58 smithi107 ceph-mon[102221]: pgmap v1127: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 943 op/s; 971/293610 objects misplaced (0.331%); 3.7 MiB/s, 59 objects/s recovering 2024-09-21T10:15:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:58 smithi107 ceph-mon[105022]: pgmap v1127: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 943 op/s; 971/293610 objects misplaced (0.331%); 3.7 MiB/s, 59 objects/s recovering 2024-09-21T10:15:58.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:15:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:58 smithi129 ceph-mon[88471]: pgmap v1127: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 943 op/s; 971/293610 objects misplaced (0.331%); 3.7 MiB/s, 59 objects/s recovering 2024-09-21T10:15:58.945 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:58.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:58.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:15:59.047 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:59.071 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:59.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:59.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:59.172 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:59.195 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:59.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:59.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:59.298 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:15:59.327 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:59.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:59.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:15:59.428 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:59.455 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:59.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:59.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:15:59.482 INFO:teuthology.orchestra.run.smithi107.stderr:set pool 12 pg_num to 26 2024-09-21T10:15:59.556 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:15:59.579 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:59.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:59.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:15:59.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:15:59.723 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:15:59.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:59.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:15:59.825 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:15:59.850 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:59.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:59.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:15:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:59 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3162584332' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:15:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:15:59 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:15:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:59 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3162584332' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:15:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:15:59 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:15:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:59 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3162584332' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:15:59.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:15:59 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:15:59.951 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:15:59.975 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:15:59.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:15:59.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:00.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:00.121 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:00.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:00.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:00.222 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:00.250 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:00.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:00.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:00.351 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:00.388 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:00.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:00.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:00.480 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:16:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:16:00.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:16:00.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:00.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:00.533 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:00.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:00.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:00.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:00.712 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:00.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:00.756 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:00 smithi107 ceph-mon[102221]: pgmap v1128: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 5.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 852 op/s; 971/289605 objects misplaced (0.335%); 2.6 MiB/s, 40 objects/s recovering 2024-09-21T10:16:00.757 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:00 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:16:00.757 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:00 smithi107 ceph-mon[102221]: osdmap e315: 8 total, 8 up, 8 in 2024-09-21T10:16:00.757 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:00 smithi107 ceph-mon[105022]: pgmap v1128: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 5.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 852 op/s; 971/289605 objects misplaced (0.335%); 2.6 MiB/s, 40 objects/s recovering 2024-09-21T10:16:00.757 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:00 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:16:00.757 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:00 smithi107 ceph-mon[105022]: osdmap e315: 8 total, 8 up, 8 in 2024-09-21T10:16:00.814 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:00.843 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:00.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:00.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:00.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:00 smithi129 ceph-mon[88471]: pgmap v1128: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 5.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 852 op/s; 971/289605 objects misplaced (0.335%); 2.6 MiB/s, 40 objects/s recovering 2024-09-21T10:16:00.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:00 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:16:00.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:00 smithi129 ceph-mon[88471]: osdmap e315: 8 total, 8 up, 8 in 2024-09-21T10:16:00.944 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:00.970 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:00.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:00.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:01.072 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:01.100 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:01.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:01.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:01.202 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:01.251 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:01.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:01.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:01.353 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:01.379 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:01.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:01.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:01.481 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:01.529 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:01.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:01.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:01.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:01.689 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:01.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:01.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:01.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:01 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:16:01.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:01 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:16:01.792 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:01.813 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:01.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:01.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:01.915 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:01.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:01 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:16:01.936 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:01.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:01.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:02.038 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:02.058 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:02.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:02.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:02.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:16:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:16:01] "GET /metrics HTTP/1.1" 200 36822 "" "Prometheus/2.51.0" 2024-09-21T10:16:02.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:02.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:02.183 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:02.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:02.283 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:02.307 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:02.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:02.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:02.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:02.430 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:02.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:02.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:02.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:02.575 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:02.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:02.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:02.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:02.710 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:02.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:02.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:02.811 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:02.835 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:16:02.835 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:02 smithi107 ceph-mon[102221]: pgmap v1130: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 5.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 880 op/s; 971/286293 objects misplaced (0.339%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:16:02.835 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:02 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:16:02.835 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:02 smithi107 ceph-mon[102221]: osdmap e316: 8 total, 8 up, 8 in 2024-09-21T10:16:02.835 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:02 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/898846809' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:16:02.836 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:02 smithi107 ceph-mon[105022]: pgmap v1130: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 5.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 880 op/s; 971/286293 objects misplaced (0.339%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:16:02.836 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:02 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:16:02.836 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:02 smithi107 ceph-mon[105022]: osdmap e316: 8 total, 8 up, 8 in 2024-09-21T10:16:02.836 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:02 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/898846809' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:16:02.839 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:02.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:02.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:02.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:02 smithi129 ceph-mon[88471]: pgmap v1130: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 5.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 880 op/s; 971/286293 objects misplaced (0.339%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:16:02.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:02 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:16:02.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:02 smithi129 ceph-mon[88471]: osdmap e316: 8 total, 8 up, 8 in 2024-09-21T10:16:02.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:02 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/898846809' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:16:02.941 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:02.967 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:02.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:02.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:03.069 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:03.091 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:03.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:03.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:03.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:03.215 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:03.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:03.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:03.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:03.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:03.338 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:03.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:03.439 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:03.460 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:03.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:03.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:03.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:03.584 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:03.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:03.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:03.686 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:03.707 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:03.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:03.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:03.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:03 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:16:03.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:03 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/898846809' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:16:03.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:03 smithi107 ceph-mon[102221]: osdmap e317: 8 total, 8 up, 8 in 2024-09-21T10:16:03.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:03 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:16:03.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:03 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/898846809' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:16:03.771 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:03 smithi107 ceph-mon[105022]: osdmap e317: 8 total, 8 up, 8 in 2024-09-21T10:16:03.772 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:16:03.772 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:16:03.809 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:03.831 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:03.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:03.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:03.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:03 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:16:03.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:03 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/898846809' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:16:03.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:03 smithi129 ceph-mon[88471]: osdmap e317: 8 total, 8 up, 8 in 2024-09-21T10:16:03.933 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:03.955 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:03.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:03.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:04.057 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:04.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:04.079 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:04.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:04.181 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:04.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:04.203 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:04.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:04.304 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:04.325 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:04.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:04.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:04.427 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:04.449 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:04.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:04.551 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:04.578 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:04.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:04.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:04.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:16:04.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:04.715 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:04.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:04.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:04.750 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:04 smithi107 ceph-mon[102221]: pgmap v1133: 119 pgs: 1 creating+peering, 1 active+recovery_wait, 2 peering, 1 remapped+peering, 5 unknown, 109 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:16:04.750 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:04 smithi107 ceph-mon[102221]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:16:04.750 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:04 smithi107 ceph-mon[102221]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-21T10:16:04.750 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:04 smithi107 ceph-mon[102221]: osdmap e318: 8 total, 8 up, 8 in 2024-09-21T10:16:04.750 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:04 smithi107 ceph-mon[102221]: 12.6 scrub starts 2024-09-21T10:16:04.750 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:04 smithi107 ceph-mon[102221]: 12.6 scrub ok 2024-09-21T10:16:04.750 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:04 smithi107 ceph-mon[105022]: pgmap v1133: 119 pgs: 1 creating+peering, 1 active+recovery_wait, 2 peering, 1 remapped+peering, 5 unknown, 109 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:16:04.751 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:04 smithi107 ceph-mon[105022]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:16:04.751 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:04 smithi107 ceph-mon[105022]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-21T10:16:04.751 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:04 smithi107 ceph-mon[105022]: osdmap e318: 8 total, 8 up, 8 in 2024-09-21T10:16:04.751 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:04 smithi107 ceph-mon[105022]: 12.6 scrub starts 2024-09-21T10:16:04.751 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:04 smithi107 ceph-mon[105022]: 12.6 scrub ok 2024-09-21T10:16:04.817 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:04.839 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:04.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:04.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:04 smithi129 ceph-mon[88471]: pgmap v1133: 119 pgs: 1 creating+peering, 1 active+recovery_wait, 2 peering, 1 remapped+peering, 5 unknown, 109 active+clean; 5.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 865 op/s; 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:16:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:04 smithi129 ceph-mon[88471]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:16:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:04 smithi129 ceph-mon[88471]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-21T10:16:04.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:04 smithi129 ceph-mon[88471]: osdmap e318: 8 total, 8 up, 8 in 2024-09-21T10:16:04.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:04 smithi129 ceph-mon[88471]: 12.6 scrub starts 2024-09-21T10:16:04.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:04 smithi129 ceph-mon[88471]: 12.6 scrub ok 2024-09-21T10:16:04.941 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:04.965 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:16:04.965 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:16:04.965 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:16:04.965 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:16:04.975 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:04.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:04.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:04.987 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:16:04.987 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:16:05.007 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:16:05.007 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:16:05.033 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:16:05.033 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:16:05.057 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:16:05.057 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:16:05.077 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:05.082 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:16:05.083 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:16:05.100 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:05.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:05.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:05.107 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:16:05.107 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:16:05.202 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:05.224 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:05.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:05.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:05.325 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:05.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:05.347 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:05.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:05.449 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:05.470 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:05.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:05.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:05.572 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:05.598 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:05.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:05.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:05.699 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:05.725 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:05.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:05.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:05.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:05 smithi107 ceph-mon[102221]: osdmap e319: 8 total, 8 up, 8 in 2024-09-21T10:16:05.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:05 smithi107 ceph-mon[102221]: 12.9 scrub starts 2024-09-21T10:16:05.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:05 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2865760934' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:05.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:05 smithi107 ceph-mon[102221]: 12.9 scrub ok 2024-09-21T10:16:05.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:05 smithi107 ceph-mon[105022]: osdmap e319: 8 total, 8 up, 8 in 2024-09-21T10:16:05.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:05 smithi107 ceph-mon[105022]: 12.9 scrub starts 2024-09-21T10:16:05.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:05 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2865760934' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:05.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:05 smithi107 ceph-mon[105022]: 12.9 scrub ok 2024-09-21T10:16:05.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:05.864 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:05.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:05.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:05 smithi129 ceph-mon[88471]: osdmap e319: 8 total, 8 up, 8 in 2024-09-21T10:16:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:05 smithi129 ceph-mon[88471]: 12.9 scrub starts 2024-09-21T10:16:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:05 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2865760934' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:05.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:05 smithi129 ceph-mon[88471]: 12.9 scrub ok 2024-09-21T10:16:05.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:06.010 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:06.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:06.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:06.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:06.114 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:16:06.114 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:16:06.114 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:16:06.114 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.990743 2024-09-21T10:16:06.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 4 0.9907425776904465 2024-09-21T10:16:06.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:06.147 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:06.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:06.233 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:06.248 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:06.274 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:06.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:06.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:06.376 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:06.402 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:06.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:06.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:06.504 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:06.531 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:06 smithi107 ceph-mon[105022]: pgmap v1136: 119 pgs: 1 creating+peering, 1 active+recovery_wait, 2 peering, 1 remapped+peering, 4 unknown, 110 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 754 op/s; 3.7 MiB/s, 60 objects/s recovering 2024-09-21T10:16:06.531 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:06 smithi107 ceph-mon[105022]: osdmap e320: 8 total, 8 up, 8 in 2024-09-21T10:16:06.531 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:06 smithi107 ceph-mon[105022]: 12.19 scrub starts 2024-09-21T10:16:06.533 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:06.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:06.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:06.634 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:06.659 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:06.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:06.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:06.761 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:06.788 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:06.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:06.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:06.798 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:06 smithi107 ceph-mon[102221]: pgmap v1136: 119 pgs: 1 creating+peering, 1 active+recovery_wait, 2 peering, 1 remapped+peering, 4 unknown, 110 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 754 op/s; 3.7 MiB/s, 60 objects/s recovering 2024-09-21T10:16:06.798 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:06 smithi107 ceph-mon[102221]: osdmap e320: 8 total, 8 up, 8 in 2024-09-21T10:16:06.798 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:06 smithi107 ceph-mon[102221]: 12.19 scrub starts 2024-09-21T10:16:06.889 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:06.911 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:06.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:06.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:06 smithi129 ceph-mon[88471]: pgmap v1136: 119 pgs: 1 creating+peering, 1 active+recovery_wait, 2 peering, 1 remapped+peering, 4 unknown, 110 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 754 op/s; 3.7 MiB/s, 60 objects/s recovering 2024-09-21T10:16:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:06 smithi129 ceph-mon[88471]: osdmap e320: 8 total, 8 up, 8 in 2024-09-21T10:16:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:06 smithi129 ceph-mon[88471]: 12.19 scrub starts 2024-09-21T10:16:07.013 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:07.034 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:07.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:07.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:07.136 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:07.158 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:07.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:07.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:07.261 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:07.283 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:07.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:07.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:07.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:07.406 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:07.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:07.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:07.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:16:07.509 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:07.530 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:07.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:07.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:07.632 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:07.655 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:07.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:07.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:07.757 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:07.786 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:07.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:07.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:07 smithi107 ceph-mon[102221]: osdmap e321: 8 total, 8 up, 8 in 2024-09-21T10:16:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:07 smithi107 ceph-mon[102221]: 12.19 scrub ok 2024-09-21T10:16:07.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:07 smithi107 ceph-mon[105022]: osdmap e321: 8 total, 8 up, 8 in 2024-09-21T10:16:07.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:07 smithi107 ceph-mon[105022]: 12.19 scrub ok 2024-09-21T10:16:07.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:07 smithi129 ceph-mon[88471]: osdmap e321: 8 total, 8 up, 8 in 2024-09-21T10:16:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:07 smithi129 ceph-mon[88471]: 12.19 scrub ok 2024-09-21T10:16:07.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:07.928 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:07.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:08.029 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:08.074 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:08.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:08.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:08.176 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:08.200 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:16:08.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:08.201 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:08.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:08.302 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:08.328 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:08.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:08.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:08.430 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:08.456 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:08.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:08.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:08.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:08.603 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:08.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:08.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:08.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:08.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:08.784 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:08.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:08.827 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:08 smithi107 ceph-mon[102221]: 12.1 scrub starts 2024-09-21T10:16:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:08 smithi107 ceph-mon[102221]: pgmap v1139: 119 pgs: 1 creating+peering, 1 active+recovery_wait, 1 remapped+peering, 116 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:16:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:08 smithi107 ceph-mon[102221]: 12.1 scrub ok 2024-09-21T10:16:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:08 smithi107 ceph-mon[102221]: osdmap e322: 8 total, 8 up, 8 in 2024-09-21T10:16:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:08 smithi107 ceph-mon[102221]: 12.13 scrub starts 2024-09-21T10:16:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:08 smithi107 ceph-mon[102221]: 12.16 scrub starts 2024-09-21T10:16:08.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:08 smithi107 ceph-mon[105022]: 12.1 scrub starts 2024-09-21T10:16:08.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:08 smithi107 ceph-mon[105022]: pgmap v1139: 119 pgs: 1 creating+peering, 1 active+recovery_wait, 1 remapped+peering, 116 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:16:08.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:08 smithi107 ceph-mon[105022]: 12.1 scrub ok 2024-09-21T10:16:08.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:08 smithi107 ceph-mon[105022]: osdmap e322: 8 total, 8 up, 8 in 2024-09-21T10:16:08.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:08 smithi107 ceph-mon[105022]: 12.13 scrub starts 2024-09-21T10:16:08.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:08 smithi107 ceph-mon[105022]: 12.16 scrub starts 2024-09-21T10:16:08.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:08 smithi129 ceph-mon[88471]: 12.1 scrub starts 2024-09-21T10:16:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:08 smithi129 ceph-mon[88471]: pgmap v1139: 119 pgs: 1 creating+peering, 1 active+recovery_wait, 1 remapped+peering, 116 active+clean; 5.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:16:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:08 smithi129 ceph-mon[88471]: 12.1 scrub ok 2024-09-21T10:16:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:08 smithi129 ceph-mon[88471]: osdmap e322: 8 total, 8 up, 8 in 2024-09-21T10:16:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:08 smithi129 ceph-mon[88471]: 12.13 scrub starts 2024-09-21T10:16:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:08 smithi129 ceph-mon[88471]: 12.16 scrub starts 2024-09-21T10:16:08.933 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:08.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:08.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:09.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:09.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:09.072 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:09.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:09.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:09.211 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:09.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:09.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:09.313 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:09.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:09.343 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:09.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:09.444 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:09.468 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:09.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:09.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:09.570 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:09.594 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:09.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:09.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:09.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:09.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:09.742 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:09.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:09.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:09 smithi107 ceph-mon[102221]: 12.5 scrub starts 2024-09-21T10:16:09.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:09 smithi107 ceph-mon[102221]: 12.13 scrub ok 2024-09-21T10:16:09.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:09 smithi107 ceph-mon[102221]: 12.16 scrub ok 2024-09-21T10:16:09.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:09 smithi107 ceph-mon[102221]: 12.17 scrub starts 2024-09-21T10:16:09.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:09 smithi107 ceph-mon[102221]: 12.18 scrub starts 2024-09-21T10:16:09.772 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:09 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-21T10:16:09.772 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:09 smithi107 ceph-mon[105022]: 12.5 scrub starts 2024-09-21T10:16:09.772 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:09 smithi107 ceph-mon[105022]: 12.13 scrub ok 2024-09-21T10:16:09.772 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:09 smithi107 ceph-mon[105022]: 12.16 scrub ok 2024-09-21T10:16:09.772 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:09 smithi107 ceph-mon[105022]: 12.17 scrub starts 2024-09-21T10:16:09.772 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:09 smithi107 ceph-mon[105022]: 12.18 scrub starts 2024-09-21T10:16:09.772 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:09 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-21T10:16:09.843 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:09.868 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:09.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:09.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:09 smithi129 ceph-mon[88471]: 12.5 scrub starts 2024-09-21T10:16:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:09 smithi129 ceph-mon[88471]: 12.13 scrub ok 2024-09-21T10:16:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:09 smithi129 ceph-mon[88471]: 12.16 scrub ok 2024-09-21T10:16:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:09 smithi129 ceph-mon[88471]: 12.17 scrub starts 2024-09-21T10:16:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:09 smithi129 ceph-mon[88471]: 12.18 scrub starts 2024-09-21T10:16:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:09 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-21T10:16:09.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:09.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:09.997 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:09.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:10.098 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:10.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:10.121 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:10.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:10.222 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:10.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:10.244 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:10.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:10.345 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:10.366 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:10.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:10.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:10.468 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:10.489 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:10.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:10.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:10.566 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:16:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:16:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:16:10.591 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:10.612 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:10.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:10.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:10.714 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:10.738 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:10.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:10.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:10.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[102221]: 12.5 scrub ok 2024-09-21T10:16:10.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[102221]: 12.4 scrub starts 2024-09-21T10:16:10.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[102221]: pgmap v1141: 119 pgs: 119 active+clean; 5.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 9.4 MiB/s, 151 objects/s recovering 2024-09-21T10:16:10.816 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[102221]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T10:16:10.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-21T10:16:10.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[102221]: osdmap e323: 8 total, 8 up, 8 in 2024-09-21T10:16:10.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[102221]: 12.3 scrub starts 2024-09-21T10:16:10.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[102221]: 12.17 scrub ok 2024-09-21T10:16:10.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[102221]: 12.4 scrub ok 2024-09-21T10:16:10.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[102221]: 12.18 scrub ok 2024-09-21T10:16:10.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[102221]: 12.7 scrub starts 2024-09-21T10:16:10.817 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[102221]: 12.8 scrub starts 2024-09-21T10:16:10.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[105022]: 12.5 scrub ok 2024-09-21T10:16:10.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[105022]: 12.4 scrub starts 2024-09-21T10:16:10.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[105022]: pgmap v1141: 119 pgs: 119 active+clean; 5.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 9.4 MiB/s, 151 objects/s recovering 2024-09-21T10:16:10.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[105022]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T10:16:10.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-21T10:16:10.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[105022]: osdmap e323: 8 total, 8 up, 8 in 2024-09-21T10:16:10.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[105022]: 12.3 scrub starts 2024-09-21T10:16:10.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[105022]: 12.17 scrub ok 2024-09-21T10:16:10.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[105022]: 12.4 scrub ok 2024-09-21T10:16:10.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[105022]: 12.18 scrub ok 2024-09-21T10:16:10.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[105022]: 12.7 scrub starts 2024-09-21T10:16:10.818 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:10 smithi107 ceph-mon[105022]: 12.8 scrub starts 2024-09-21T10:16:10.819 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:16:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:16:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135241 12.10 12.514f79d0 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:16:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581333+0000", 2024-09-21T10:16:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.23138724199999999, 2024-09-21T10:16:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.23140845099999999, 2024-09-21T10:16:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:16:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:16:10.819 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for peered", 2024-09-21T10:16:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:16:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:16:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:16:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135241 2024-09-21T10:16:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:16:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:16:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581333+0000", 2024-09-21T10:16:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:16:10.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581333+0000", 2024-09-21T10:16:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:16:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581334+0000", 2024-09-21T10:16:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.536e-06 2024-09-21T10:16:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:16:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581341+0000", 2024-09-21T10:16:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1330000000000004e-06 2024-09-21T10:16:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:16:10.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581353+0000", 2024-09-21T10:16:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2646999999999999e-05 2024-09-21T10:16:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:16:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581362+0000", 2024-09-21T10:16:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9819999999999997e-06 2024-09-21T10:16:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:16:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581415+0000", 2024-09-21T10:16:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2926000000000001e-05 2024-09-21T10:16:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.822 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:16:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581426+0000", 2024-09-21T10:16:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0421999999999999e-05 2024-09-21T10:16:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:16:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135271 12.10 12.ae4ac6b0 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:16:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581377+0000", 2024-09-21T10:16:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.231343095, 2024-09-21T10:16:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.23140312599999999, 2024-09-21T10:16:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:16:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:16:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for peered", 2024-09-21T10:16:10.823 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:16:10.824 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:16:10.824 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:16:10.824 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135271 2024-09-21T10:16:10.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.824 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:16:10.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:16:10.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581377+0000", 2024-09-21T10:16:10.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:16:10.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581377+0000", 2024-09-21T10:16:10.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:16:10.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581378+0000", 2024-09-21T10:16:10.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0800000000000005e-07 2024-09-21T10:16:10.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:16:10.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581380+0000", 2024-09-21T10:16:10.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4729999999999999e-06 2024-09-21T10:16:10.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:16:10.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581386+0000", 2024-09-21T10:16:10.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7250000000000002e-06 2024-09-21T10:16:10.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:16:10.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581389+0000", 2024-09-21T10:16:10.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3139999999999999e-06 2024-09-21T10:16:10.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:16:10.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581518+0000", 2024-09-21T10:16:10.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000129374 2024-09-21T10:16:10.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:16:10.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581526+0000", 2024-09-21T10:16:10.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7009999999999996e-06 2024-09-21T10:16:10.827 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.827 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:16:10.827 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.827 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135282 12.10 12.125efe10 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:16:10.827 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581404+0000", 2024-09-21T10:16:10.827 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.23131634500000001, 2024-09-21T10:16:10.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.23140458799999999, 2024-09-21T10:16:10.827 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:16:10.827 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:16:10.827 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for peered", 2024-09-21T10:16:10.827 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:16:10.827 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:16:10.827 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:16:10.828 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135282 2024-09-21T10:16:10.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.828 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:16:10.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:16:10.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581404+0000", 2024-09-21T10:16:10.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:16:10.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581404+0000", 2024-09-21T10:16:10.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:16:10.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581404+0000", 2024-09-21T10:16:10.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4499999999999998e-07 2024-09-21T10:16:10.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:16:10.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581406+0000", 2024-09-21T10:16:10.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.857e-06 2024-09-21T10:16:10.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:16:10.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581412+0000", 2024-09-21T10:16:10.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9150000000000001e-06 2024-09-21T10:16:10.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:16:10.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581416+0000", 2024-09-21T10:16:10.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.822e-06 2024-09-21T10:16:10.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:16:10.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581574+0000", 2024-09-21T10:16:10.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015840799999999999 2024-09-21T10:16:10.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:16:10.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581580+0000", 2024-09-21T10:16:10.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1469999999999998e-06 2024-09-21T10:16:10.830 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.830 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:16:10.830 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.831 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135289 12.10 12.79179630 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:16:10.831 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581420+0000", 2024-09-21T10:16:10.831 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.231299963, 2024-09-21T10:16:10.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.23144476999999999, 2024-09-21T10:16:10.831 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:16:10.831 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:16:10.831 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for peered", 2024-09-21T10:16:10.831 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:16:10.831 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:16:10.831 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:16:10.831 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135289 2024-09-21T10:16:10.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.831 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:16:10.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:16:10.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581420+0000", 2024-09-21T10:16:10.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:16:10.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581420+0000", 2024-09-21T10:16:10.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:16:10.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581420+0000", 2024-09-21T10:16:10.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6399999999999998e-07 2024-09-21T10:16:10.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:16:10.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581422+0000", 2024-09-21T10:16:10.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2890000000000001e-06 2024-09-21T10:16:10.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:16:10.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581426+0000", 2024-09-21T10:16:10.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1949999999999996e-06 2024-09-21T10:16:10.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:16:10.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581429+0000", 2024-09-21T10:16:10.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7590000000000002e-06 2024-09-21T10:16:10.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:16:10.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581607+0000", 2024-09-21T10:16:10.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017814600000000001 2024-09-21T10:16:10.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:16:10.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581612+0000", 2024-09-21T10:16:10.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0509999999999996e-06 2024-09-21T10:16:10.834 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.834 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:16:10.834 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.834 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135317 12.10 12.484f34d0 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:16:10.834 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581449+0000", 2024-09-21T10:16:10.834 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.23127150499999999, 2024-09-21T10:16:10.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.231445491, 2024-09-21T10:16:10.835 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:16:10.835 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:16:10.835 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for peered", 2024-09-21T10:16:10.835 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:16:10.835 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:16:10.835 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:16:10.835 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135317 2024-09-21T10:16:10.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.835 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:16:10.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:16:10.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581449+0000", 2024-09-21T10:16:10.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:16:10.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581449+0000", 2024-09-21T10:16:10.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:16:10.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581449+0000", 2024-09-21T10:16:10.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3099999999999998e-07 2024-09-21T10:16:10.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:16:10.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581451+0000", 2024-09-21T10:16:10.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0360000000000001e-06 2024-09-21T10:16:10.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:16:10.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581458+0000", 2024-09-21T10:16:10.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5169999999999996e-06 2024-09-21T10:16:10.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:16:10.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581462+0000", 2024-09-21T10:16:10.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6670000000000004e-06 2024-09-21T10:16:10.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:16:10.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581639+0000", 2024-09-21T10:16:10.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017724200000000001 2024-09-21T10:16:10.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:16:10.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581651+0000", 2024-09-21T10:16:10.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1234e-05 2024-09-21T10:16:10.838 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.838 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:16:10.838 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.838 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135369 12.10 12.1c186750 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:16:10.838 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581471+0000", 2024-09-21T10:16:10.838 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.231248975, 2024-09-21T10:16:10.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.23145168499999999, 2024-09-21T10:16:10.838 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:16:10.839 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:16:10.839 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for peered", 2024-09-21T10:16:10.839 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:16:10.839 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:16:10.839 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:16:10.839 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135369 2024-09-21T10:16:10.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.839 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:16:10.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:16:10.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581471+0000", 2024-09-21T10:16:10.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:16:10.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581471+0000", 2024-09-21T10:16:10.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:16:10.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581472+0000", 2024-09-21T10:16:10.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1600000000000002e-07 2024-09-21T10:16:10.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:16:10.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581474+0000", 2024-09-21T10:16:10.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1430000000000001e-06 2024-09-21T10:16:10.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:16:10.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581481+0000", 2024-09-21T10:16:10.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1999999999999997e-06 2024-09-21T10:16:10.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:16:10.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581486+0000", 2024-09-21T10:16:10.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9219999999999996e-06 2024-09-21T10:16:10.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:16:10.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581675+0000", 2024-09-21T10:16:10.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018961 2024-09-21T10:16:10.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:16:10.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581679+0000", 2024-09-21T10:16:10.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5250000000000001e-06 2024-09-21T10:16:10.842 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.842 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:16:10.842 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.842 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135398 12.10 12.75803510 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:16:10.842 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581512+0000", 2024-09-21T10:16:10.842 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.23120822899999999, 2024-09-21T10:16:10.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.23143934399999999, 2024-09-21T10:16:10.842 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:16:10.842 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:16:10.842 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for peered", 2024-09-21T10:16:10.843 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:16:10.843 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:16:10.843 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:16:10.843 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135398 2024-09-21T10:16:10.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.843 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:16:10.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:16:10.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581512+0000", 2024-09-21T10:16:10.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:16:10.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581512+0000", 2024-09-21T10:16:10.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.844 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:16:10.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581512+0000", 2024-09-21T10:16:10.844 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6899999999999998e-07 2024-09-21T10:16:10.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.844 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:16:10.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581515+0000", 2024-09-21T10:16:10.844 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.244e-06 2024-09-21T10:16:10.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.844 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:16:10.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581521+0000", 2024-09-21T10:16:10.844 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.268e-06 2024-09-21T10:16:10.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:16:10.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581526+0000", 2024-09-21T10:16:10.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4870000000000002e-06 2024-09-21T10:16:10.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:16:10.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581926+0000", 2024-09-21T10:16:10.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040013100000000002 2024-09-21T10:16:10.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:16:10.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581932+0000", 2024-09-21T10:16:10.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9529999999999996e-06 2024-09-21T10:16:10.845 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.846 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:16:10.846 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.846 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135454 12.10 12.96b50e30 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:16:10.846 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581549+0000", 2024-09-21T10:16:10.846 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.231171242, 2024-09-21T10:16:10.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.23143151100000001, 2024-09-21T10:16:10.846 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:16:10.846 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:16:10.846 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for peered", 2024-09-21T10:16:10.846 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:16:10.846 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:16:10.846 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:16:10.847 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135454 2024-09-21T10:16:10.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.847 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:16:10.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:16:10.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581549+0000", 2024-09-21T10:16:10.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:16:10.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581549+0000", 2024-09-21T10:16:10.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:16:10.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581549+0000", 2024-09-21T10:16:10.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3999999999999997e-07 2024-09-21T10:16:10.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:16:10.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581551+0000", 2024-09-21T10:16:10.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.725e-06 2024-09-21T10:16:10.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:16:10.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581555+0000", 2024-09-21T10:16:10.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2459999999999997e-06 2024-09-21T10:16:10.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:16:10.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581559+0000", 2024-09-21T10:16:10.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1919999999999998e-06 2024-09-21T10:16:10.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:16:10.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581965+0000", 2024-09-21T10:16:10.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040525199999999999 2024-09-21T10:16:10.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:16:10.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581969+0000", 2024-09-21T10:16:10.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.673e-06 2024-09-21T10:16:10.849 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.849 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:16:10.849 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.850 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135465 12.10 12.f8fcdb70 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:16:10.850 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581578+0000", 2024-09-21T10:16:10.850 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.23114179500000001, 2024-09-21T10:16:10.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.23143023300000001, 2024-09-21T10:16:10.850 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:16:10.850 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:16:10.850 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for peered", 2024-09-21T10:16:10.850 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:16:10.850 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:16:10.850 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:16:10.850 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135465 2024-09-21T10:16:10.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.851 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:16:10.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:16:10.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581578+0000", 2024-09-21T10:16:10.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:16:10.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581578+0000", 2024-09-21T10:16:10.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:16:10.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581579+0000", 2024-09-21T10:16:10.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.34e-07 2024-09-21T10:16:10.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:16:10.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581581+0000", 2024-09-21T10:16:10.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9520000000000001e-06 2024-09-21T10:16:10.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:16:10.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581586+0000", 2024-09-21T10:16:10.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7230000000000001e-06 2024-09-21T10:16:10.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:16:10.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581591+0000", 2024-09-21T10:16:10.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7060000000000003e-06 2024-09-21T10:16:10.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:16:10.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581992+0000", 2024-09-21T10:16:10.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040077299999999999 2024-09-21T10:16:10.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:16:10.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581996+0000", 2024-09-21T10:16:10.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8039999999999999e-06 2024-09-21T10:16:10.853 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.853 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:16:10.853 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.853 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135467 12.10 12.d68ee550 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:16:10.853 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581602+0000", 2024-09-21T10:16:10.854 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.23111810199999999, 2024-09-21T10:16:10.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.23143482100000001, 2024-09-21T10:16:10.854 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:16:10.854 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:16:10.854 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for peered", 2024-09-21T10:16:10.854 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:16:10.854 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:16:10.854 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:16:10.854 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135467 2024-09-21T10:16:10.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.854 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:16:10.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:16:10.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581602+0000", 2024-09-21T10:16:10.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:16:10.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581602+0000", 2024-09-21T10:16:10.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:16:10.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581602+0000", 2024-09-21T10:16:10.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3099999999999999e-07 2024-09-21T10:16:10.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:16:10.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581604+0000", 2024-09-21T10:16:10.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5030000000000001e-06 2024-09-21T10:16:10.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:16:10.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581608+0000", 2024-09-21T10:16:10.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0219999999999998e-06 2024-09-21T10:16:10.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:16:10.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581611+0000", 2024-09-21T10:16:10.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9239999999999999e-06 2024-09-21T10:16:10.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:16:10.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582015+0000", 2024-09-21T10:16:10.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000404688 2024-09-21T10:16:10.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:16:10.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582019+0000", 2024-09-21T10:16:10.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4379999999999999e-06 2024-09-21T10:16:10.857 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.857 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:16:10.857 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.857 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135469 12.10 12.ed302490 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:16:10.857 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581614+0000", 2024-09-21T10:16:10.858 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.23110565199999999, 2024-09-21T10:16:10.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.23145054600000001, 2024-09-21T10:16:10.858 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:16:10.858 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:16:10.858 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for peered", 2024-09-21T10:16:10.858 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:16:10.858 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:16:10.858 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:16:10.858 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135469 2024-09-21T10:16:10.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.858 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:16:10.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:16:10.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:10.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581614+0000", 2024-09-21T10:16:10.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:16:10.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581614+0000", 2024-09-21T10:16:10.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:16:10.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581615+0000", 2024-09-21T10:16:10.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0600000000000001e-07 2024-09-21T10:16:10.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:16:10.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581616+0000", 2024-09-21T10:16:10.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4500000000000001e-06 2024-09-21T10:16:10.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:16:10.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581622+0000", 2024-09-21T10:16:10.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5289999999999999e-06 2024-09-21T10:16:10.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:16:10.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581627+0000", 2024-09-21T10:16:10.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5459999999999999e-06 2024-09-21T10:16:10.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:16:10.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582038+0000", 2024-09-21T10:16:10.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041066600000000001 2024-09-21T10:16:10.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:16:10.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582041+0000", 2024-09-21T10:16:10.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3419999999999999e-06 2024-09-21T10:16:10.862 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.862 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:16:10.862 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.863 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135559 12.10 12.9ec8a690 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:16:10.863 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581633+0000", 2024-09-21T10:16:10.863 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.23108701600000001, 2024-09-21T10:16:10.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.23146017899999999, 2024-09-21T10:16:10.863 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:16:10.863 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:16:10.863 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for peered", 2024-09-21T10:16:10.863 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:16:10.863 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:16:10.863 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:16:10.863 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135559 2024-09-21T10:16:10.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.863 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:16:10.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:16:10.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581633+0000", 2024-09-21T10:16:10.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:16:10.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581633+0000", 2024-09-21T10:16:10.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:16:10.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581633+0000", 2024-09-21T10:16:10.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5999999999999999e-07 2024-09-21T10:16:10.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:16:10.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581639+0000", 2024-09-21T10:16:10.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.587e-06 2024-09-21T10:16:10.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:16:10.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581648+0000", 2024-09-21T10:16:10.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2490000000000007e-06 2024-09-21T10:16:10.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:16:10.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581653+0000", 2024-09-21T10:16:10.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1080000000000001e-06 2024-09-21T10:16:10.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:16:10.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582064+0000", 2024-09-21T10:16:10.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041026000000000002 2024-09-21T10:16:10.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:16:10.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582067+0000", 2024-09-21T10:16:10.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.32e-06 2024-09-21T10:16:10.866 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.866 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:16:10.866 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135564 12.10 12.ba04a5f0 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:16:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581672+0000", 2024-09-21T10:16:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.23104817899999999, 2024-09-21T10:16:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.231469797, 2024-09-21T10:16:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:16:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:16:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for peered", 2024-09-21T10:16:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:16:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:16:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:16:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135564 2024-09-21T10:16:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.867 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:16:10.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:16:10.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581672+0000", 2024-09-21T10:16:10.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:16:10.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581672+0000", 2024-09-21T10:16:10.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:16:10.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581672+0000", 2024-09-21T10:16:10.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.41e-07 2024-09-21T10:16:10.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:16:10.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581674+0000", 2024-09-21T10:16:10.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7179999999999999e-06 2024-09-21T10:16:10.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:16:10.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581678+0000", 2024-09-21T10:16:10.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.189e-06 2024-09-21T10:16:10.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:16:10.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581682+0000", 2024-09-21T10:16:10.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.382e-06 2024-09-21T10:16:10.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:16:10.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582087+0000", 2024-09-21T10:16:10.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040544499999999999 2024-09-21T10:16:10.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:16:10.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582092+0000", 2024-09-21T10:16:10.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5000000000000001e-06 2024-09-21T10:16:10.870 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.870 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:16:10.871 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.871 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135576 12.10 12.7a83bb90 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:16:10.871 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581687+0000", 2024-09-21T10:16:10.871 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.23103265000000001, 2024-09-21T10:16:10.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.23148170800000001, 2024-09-21T10:16:10.871 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:16:10.871 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:16:10.871 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for peered", 2024-09-21T10:16:10.871 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:16:10.871 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:16:10.871 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:16:10.871 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135576 2024-09-21T10:16:10.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.872 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:16:10.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:16:10.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581687+0000", 2024-09-21T10:16:10.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:16:10.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581687+0000", 2024-09-21T10:16:10.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:16:10.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581688+0000", 2024-09-21T10:16:10.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6100000000000002e-07 2024-09-21T10:16:10.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:16:10.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581689+0000", 2024-09-21T10:16:10.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5889999999999999e-06 2024-09-21T10:16:10.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:16:10.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581695+0000", 2024-09-21T10:16:10.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6779999999999997e-06 2024-09-21T10:16:10.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:16:10.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581700+0000", 2024-09-21T10:16:10.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8080000000000004e-06 2024-09-21T10:16:10.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:16:10.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582111+0000", 2024-09-21T10:16:10.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041119699999999997 2024-09-21T10:16:10.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:16:10.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582114+0000", 2024-09-21T10:16:10.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2440000000000002e-06 2024-09-21T10:16:10.874 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.874 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:16:10.874 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.875 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135577 12.10 12.817256f0 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:16:10.875 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581706+0000", 2024-09-21T10:16:10.875 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.231014368, 2024-09-21T10:16:10.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.23149269, 2024-09-21T10:16:10.875 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:16:10.875 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:16:10.875 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for peered", 2024-09-21T10:16:10.875 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:16:10.875 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:16:10.875 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:16:10.875 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135577 2024-09-21T10:16:10.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.876 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:16:10.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:16:10.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581706+0000", 2024-09-21T10:16:10.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:16:10.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581706+0000", 2024-09-21T10:16:10.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:16:10.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581706+0000", 2024-09-21T10:16:10.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0499999999999999e-07 2024-09-21T10:16:10.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:16:10.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581708+0000", 2024-09-21T10:16:10.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5820000000000001e-06 2024-09-21T10:16:10.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:16:10.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581717+0000", 2024-09-21T10:16:10.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.901e-06 2024-09-21T10:16:10.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:16:10.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581722+0000", 2024-09-21T10:16:10.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8510000000000004e-06 2024-09-21T10:16:10.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:16:10.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582136+0000", 2024-09-21T10:16:10.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041303599999999998 2024-09-21T10:16:10.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:16:10.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582139+0000", 2024-09-21T10:16:10.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7850000000000002e-06 2024-09-21T10:16:10.879 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.879 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:16:10.879 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.879 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135632 12.10 12.8abdd870 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:16:10.879 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581728+0000", 2024-09-21T10:16:10.880 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.230991641, 2024-09-21T10:16:10.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.23149624399999999, 2024-09-21T10:16:10.880 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:16:10.880 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:16:10.880 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for peered", 2024-09-21T10:16:10.880 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:16:10.880 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:16:10.880 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:16:10.880 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135632 2024-09-21T10:16:10.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.880 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:16:10.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:16:10.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581728+0000", 2024-09-21T10:16:10.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:16:10.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581728+0000", 2024-09-21T10:16:10.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:16:10.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:16:10.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581729+0000", 2024-09-21T10:16:10.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.58e-07 2024-09-21T10:16:10.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:16:10.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581731+0000", 2024-09-21T10:16:10.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.689e-06 2024-09-21T10:16:10.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:16:10.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581736+0000", 2024-09-21T10:16:10.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8309999999999997e-06 2024-09-21T10:16:10.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:16:10.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581746+0000", 2024-09-21T10:16:10.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8179999999999999e-06 2024-09-21T10:16:10.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:16:10.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582153+0000", 2024-09-21T10:16:10.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040682499999999998 2024-09-21T10:16:10.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:16:10.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:16:10.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:16:10.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582158+0000", 2024-09-21T10:16:10.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9350000000000002e-06 2024-09-21T10:16:10.884 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.884 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:16:10.884 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.884 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:16:10.884 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-09-21T10:16:10.884 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 16 2024-09-21T10:16:10.884 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:16:10.906 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:10.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:10.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:10 smithi129 ceph-mon[88471]: 12.5 scrub ok 2024-09-21T10:16:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:10 smithi129 ceph-mon[88471]: 12.4 scrub starts 2024-09-21T10:16:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:10 smithi129 ceph-mon[88471]: pgmap v1141: 119 pgs: 119 active+clean; 5.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 9.4 MiB/s, 151 objects/s recovering 2024-09-21T10:16:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:10 smithi129 ceph-mon[88471]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T10:16:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:10 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-21T10:16:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:10 smithi129 ceph-mon[88471]: osdmap e323: 8 total, 8 up, 8 in 2024-09-21T10:16:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:10 smithi129 ceph-mon[88471]: 12.3 scrub starts 2024-09-21T10:16:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:10 smithi129 ceph-mon[88471]: 12.17 scrub ok 2024-09-21T10:16:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:10 smithi129 ceph-mon[88471]: 12.4 scrub ok 2024-09-21T10:16:10.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:10 smithi129 ceph-mon[88471]: 12.18 scrub ok 2024-09-21T10:16:10.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:10 smithi129 ceph-mon[88471]: 12.7 scrub starts 2024-09-21T10:16:10.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:10 smithi129 ceph-mon[88471]: 12.8 scrub starts 2024-09-21T10:16:11.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:11.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:11.029 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:11.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:11.130 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:11.155 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:11.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:11.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:11.257 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:11.278 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:11.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:11.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:11.380 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:11.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:11.402 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:11.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:11.502 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:11.524 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:11.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:11.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:11.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:11.648 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:11.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:11.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:11.675 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:11 smithi107 ceph-mon[102221]: 12.3 scrub ok 2024-09-21T10:16:11.675 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:11 smithi107 ceph-mon[102221]: 12.7 scrub ok 2024-09-21T10:16:11.675 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:11 smithi107 ceph-mon[102221]: 12.8 scrub ok 2024-09-21T10:16:11.675 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:11 smithi107 ceph-mon[102221]: osdmap e324: 8 total, 8 up, 8 in 2024-09-21T10:16:11.675 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:11 smithi107 ceph-mon[102221]: 12.2 scrub starts 2024-09-21T10:16:11.675 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:11.675 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:11.675 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:11 smithi107 ceph-mon[102221]: 12.2 scrub ok 2024-09-21T10:16:11.676 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-21T10:16:11.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:11 smithi107 ceph-mon[105022]: 12.3 scrub ok 2024-09-21T10:16:11.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:11 smithi107 ceph-mon[105022]: 12.7 scrub ok 2024-09-21T10:16:11.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:11 smithi107 ceph-mon[105022]: 12.8 scrub ok 2024-09-21T10:16:11.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:11 smithi107 ceph-mon[105022]: osdmap e324: 8 total, 8 up, 8 in 2024-09-21T10:16:11.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:11 smithi107 ceph-mon[105022]: 12.2 scrub starts 2024-09-21T10:16:11.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:11.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:11.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:11 smithi107 ceph-mon[105022]: 12.2 scrub ok 2024-09-21T10:16:11.676 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-21T10:16:11.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:11.772 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:11.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:11.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:11.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:11.896 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:11.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:11.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:11 smithi129 ceph-mon[88471]: 12.3 scrub ok 2024-09-21T10:16:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:11 smithi129 ceph-mon[88471]: 12.7 scrub ok 2024-09-21T10:16:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:11 smithi129 ceph-mon[88471]: 12.8 scrub ok 2024-09-21T10:16:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:11 smithi129 ceph-mon[88471]: osdmap e324: 8 total, 8 up, 8 in 2024-09-21T10:16:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:11 smithi129 ceph-mon[88471]: 12.2 scrub starts 2024-09-21T10:16:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:11 smithi129 ceph-mon[88471]: 12.2 scrub ok 2024-09-21T10:16:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-21T10:16:11.989 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:16:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:16:11] "GET /metrics HTTP/1.1" 200 36819 "" "Prometheus/2.51.0" 2024-09-21T10:16:11.998 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:12.019 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:12.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:12.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:12.121 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:12.145 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:12.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:12.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:12.247 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:12.270 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:12.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:12.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:12.275 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:12.372 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:12.395 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:12.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:12.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:12.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:12.522 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:12.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:12.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:12.623 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:12.645 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:12.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:12.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:12.746 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:12.751 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:12 smithi107 ceph-mon[102221]: pgmap v1144: 119 pgs: 119 active+clean; 5.4 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 9.4 MiB/s, 151 objects/s recovering 2024-09-21T10:16:12.751 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-21T10:16:12.751 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:12 smithi107 ceph-mon[102221]: 12.10 starting backfill to osd.0 from (0'0,0'0] MAX to 322'8439 2024-09-21T10:16:12.751 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:12 smithi107 ceph-mon[102221]: osdmap e325: 8 total, 8 up, 8 in 2024-09-21T10:16:12.752 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:12 smithi107 ceph-mon[102221]: 12.10 starting backfill to osd.1 from (0'0,0'0] MAX to 322'8439 2024-09-21T10:16:12.752 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:12 smithi107 ceph-mon[105022]: pgmap v1144: 119 pgs: 119 active+clean; 5.4 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 9.4 MiB/s, 151 objects/s recovering 2024-09-21T10:16:12.752 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-21T10:16:12.752 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:12 smithi107 ceph-mon[105022]: 12.10 starting backfill to osd.0 from (0'0,0'0] MAX to 322'8439 2024-09-21T10:16:12.752 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:12 smithi107 ceph-mon[105022]: osdmap e325: 8 total, 8 up, 8 in 2024-09-21T10:16:12.752 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:12 smithi107 ceph-mon[105022]: 12.10 starting backfill to osd.1 from (0'0,0'0] MAX to 322'8439 2024-09-21T10:16:12.768 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:12.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:12.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:12.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:16:12.861 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:12.869 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:12.891 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:12.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:12.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:12.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:12 smithi129 ceph-mon[88471]: pgmap v1144: 119 pgs: 119 active+clean; 5.4 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 9.4 MiB/s, 151 objects/s recovering 2024-09-21T10:16:12.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-21T10:16:12.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:12 smithi129 ceph-mon[88471]: 12.10 starting backfill to osd.0 from (0'0,0'0] MAX to 322'8439 2024-09-21T10:16:12.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:12 smithi129 ceph-mon[88471]: osdmap e325: 8 total, 8 up, 8 in 2024-09-21T10:16:12.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:12 smithi129 ceph-mon[88471]: 12.10 starting backfill to osd.1 from (0'0,0'0] MAX to 322'8439 2024-09-21T10:16:12.993 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:13.018 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:13.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:13.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:13.119 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:13.141 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:13.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:13.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:13.243 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:13.265 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:13.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:13.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:13.366 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:13.387 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:13.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:13.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:13.489 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:13.514 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:13.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:13.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:13.584 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.4 primary-affinity to 0.990743 (8649292) 2024-09-21T10:16:13.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:13.640 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:13.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:13.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:13.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:13.765 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:13.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:13.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:13 smithi107 ceph-mon[102221]: osdmap e326: 8 total, 8 up, 8 in 2024-09-21T10:16:13.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:13 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2052819422' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.9907425776904465} : dispatch 2024-09-21T10:16:13.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:13 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.9907425776904465} : dispatch 2024-09-21T10:16:13.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:13 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.9907425776904465}]': finished 2024-09-21T10:16:13.831 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:13 smithi107 ceph-mon[105022]: osdmap e326: 8 total, 8 up, 8 in 2024-09-21T10:16:13.831 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:13 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2052819422' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.9907425776904465} : dispatch 2024-09-21T10:16:13.831 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:13 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.9907425776904465} : dispatch 2024-09-21T10:16:13.831 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:13 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.9907425776904465}]': finished 2024-09-21T10:16:13.868 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:13.895 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:13.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:13.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:13 smithi129 ceph-mon[88471]: osdmap e326: 8 total, 8 up, 8 in 2024-09-21T10:16:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:13 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2052819422' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.9907425776904465} : dispatch 2024-09-21T10:16:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:13 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.9907425776904465} : dispatch 2024-09-21T10:16:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:13 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.9907425776904465}]': finished 2024-09-21T10:16:13.997 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:14.037 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:14.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:14.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:14.139 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:14.166 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:14.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:14.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:14.268 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:14.296 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:14.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:14.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:14.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:14.443 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:14.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:14.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:14.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:14.589 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:14.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:14.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:14.690 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:14.720 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:14.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:14.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:14.822 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:14.849 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:14.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:14.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:14 smithi107 ceph-mon[102221]: pgmap v1147: 119 pgs: 1 unknown, 118 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail 2024-09-21T10:16:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:14 smithi107 ceph-mon[102221]: 12.11 starting backfill to osd.2 from (0'0,0'0] MAX to 323'8179 2024-09-21T10:16:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:14 smithi107 ceph-mon[102221]: osdmap e327: 8 total, 8 up, 8 in 2024-09-21T10:16:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:14 smithi107 ceph-mon[102221]: 12.11 starting backfill to osd.5 from (0'0,0'0] MAX to 323'8179 2024-09-21T10:16:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:14 smithi107 ceph-mon[105022]: pgmap v1147: 119 pgs: 1 unknown, 118 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail 2024-09-21T10:16:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:14 smithi107 ceph-mon[105022]: 12.11 starting backfill to osd.2 from (0'0,0'0] MAX to 323'8179 2024-09-21T10:16:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:14 smithi107 ceph-mon[105022]: osdmap e327: 8 total, 8 up, 8 in 2024-09-21T10:16:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:14 smithi107 ceph-mon[105022]: 12.11 starting backfill to osd.5 from (0'0,0'0] MAX to 323'8179 2024-09-21T10:16:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:14 smithi129 ceph-mon[88471]: pgmap v1147: 119 pgs: 1 unknown, 118 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail 2024-09-21T10:16:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:14 smithi129 ceph-mon[88471]: 12.11 starting backfill to osd.2 from (0'0,0'0] MAX to 323'8179 2024-09-21T10:16:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:14 smithi129 ceph-mon[88471]: osdmap e327: 8 total, 8 up, 8 in 2024-09-21T10:16:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:14 smithi129 ceph-mon[88471]: 12.11 starting backfill to osd.5 from (0'0,0'0] MAX to 323'8179 2024-09-21T10:16:14.952 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:14.990 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:14.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:14.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:15.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:15.127 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:15.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:15.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:15.229 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:15.255 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:15.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:15.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:15.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:15.396 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:15.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:15.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:15.497 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:15.522 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:15.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:15.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:15.556 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:15.624 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:15.649 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:15.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:15.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:15.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:15.786 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:15.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:15.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:15.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:15 smithi107 ceph-mon[102221]: 12.0 scrub starts 2024-09-21T10:16:15.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:15 smithi107 ceph-mon[105022]: 12.0 scrub starts 2024-09-21T10:16:15.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:15 smithi129 ceph-mon[88471]: 12.0 scrub starts 2024-09-21T10:16:15.922 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:15.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:15.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:16.025 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:16.049 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:16.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:16.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:16.151 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:16.179 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:16.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:16.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:16.280 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:16.307 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:16.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:16.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:16.409 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:16.434 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:16.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:16.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:16.536 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:16.565 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:16.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:16.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:16.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:16.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:16.702 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:16.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:16.803 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:16.829 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:16.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:16.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:16 smithi107 ceph-mon[102221]: 12.0 scrub ok 2024-09-21T10:16:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:16 smithi107 ceph-mon[102221]: pgmap v1149: 119 pgs: 1 active+remapped+backfilling, 1 unknown, 117 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 3904/260985 objects misplaced (1.496%) 2024-09-21T10:16:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:16 smithi107 ceph-mon[102221]: 12.12 scrub starts 2024-09-21T10:16:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:16 smithi107 ceph-mon[105022]: 12.0 scrub ok 2024-09-21T10:16:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:16 smithi107 ceph-mon[105022]: pgmap v1149: 119 pgs: 1 active+remapped+backfilling, 1 unknown, 117 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 3904/260985 objects misplaced (1.496%) 2024-09-21T10:16:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:16 smithi107 ceph-mon[105022]: 12.12 scrub starts 2024-09-21T10:16:16.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:16 smithi129 ceph-mon[88471]: 12.0 scrub ok 2024-09-21T10:16:16.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:16 smithi129 ceph-mon[88471]: pgmap v1149: 119 pgs: 1 active+remapped+backfilling, 1 unknown, 117 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 3904/260985 objects misplaced (1.496%) 2024-09-21T10:16:16.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:16 smithi129 ceph-mon[88471]: 12.12 scrub starts 2024-09-21T10:16:16.930 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:16.958 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:16.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:16.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:17.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:17.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:17.103 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:17.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:17.204 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:17.230 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:17.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:17.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:17.301 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:16:17.301 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:16:17.331 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:17.343 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:16:17.343 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:16:17.354 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:17.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:17.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:17.376 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:16:17.376 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:16:17.421 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:16:17.421 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:16:17.453 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:16:17.453 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:16:17.456 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:17.481 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:17.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:17.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:17.486 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:16:17.486 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:16:17.515 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:16:17.515 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:16:17.546 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:16:17.546 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:16:17.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:17.622 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:17.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:17.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:17.668 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:16:17.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:17.760 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:17.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:17.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:17.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:17 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:17.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:17 smithi107 ceph-mon[102221]: 12.12 scrub ok 2024-09-21T10:16:17.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:17 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/924787275' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:17.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:17 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/837510302' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:16:17.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:17 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:16:17.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:17 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-21T10:16:17.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:17 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:17.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:17 smithi107 ceph-mon[105022]: 12.12 scrub ok 2024-09-21T10:16:17.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:17 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/924787275' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:17.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:17 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/837510302' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:16:17.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:17 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:16:17.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:17 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-21T10:16:17.862 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:17.884 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:17.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:17.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:17 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:17 smithi129 ceph-mon[88471]: 12.12 scrub ok 2024-09-21T10:16:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:17 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/924787275' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:17 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/837510302' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:16:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:17 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:16:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:17 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-21T10:16:17.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:18.013 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:18.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:18.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:18.115 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:18.137 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:18.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:18.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:18.238 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:18.260 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:18.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:18.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:18.362 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:18.383 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:18.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:18.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:18.484 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:18.507 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:18.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:18.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:18.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:18.635 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:18.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:18.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:18.737 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:18.763 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:18.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:18.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:18.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:18 smithi107 ceph-mon[102221]: pgmap v1150: 119 pgs: 2 active+remapped+backfilling, 117 active+clean; 5.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 8588/255033 objects misplaced (3.367%) 2024-09-21T10:16:18.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:18 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:16:18.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:18 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:16:18.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:18 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:16:18.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:18 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-21T10:16:18.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:18 smithi107 ceph-mon[102221]: osdmap e328: 8 total, 8 up, 8 in 2024-09-21T10:16:18.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:18 smithi107 ceph-mon[105022]: pgmap v1150: 119 pgs: 2 active+remapped+backfilling, 117 active+clean; 5.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 8588/255033 objects misplaced (3.367%) 2024-09-21T10:16:18.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:18 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:16:18.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:18 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:16:18.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:18 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:16:18.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:18 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-21T10:16:18.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:18 smithi107 ceph-mon[105022]: osdmap e328: 8 total, 8 up, 8 in 2024-09-21T10:16:18.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:18.894 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:18.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:18.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:18 smithi129 ceph-mon[88471]: pgmap v1150: 119 pgs: 2 active+remapped+backfilling, 117 active+clean; 5.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 8588/255033 objects misplaced (3.367%) 2024-09-21T10:16:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:18 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:16:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:18 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:16:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:18 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:16:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:18 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-21T10:16:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:18 smithi129 ceph-mon[88471]: osdmap e328: 8 total, 8 up, 8 in 2024-09-21T10:16:18.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:19.024 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:19.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:19.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:19.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:19.151 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:19.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:19.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:19.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:19.279 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:19.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:19.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:19.381 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:19.403 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:19.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:19.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:19.505 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:19.527 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:19.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:19.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:19.580 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:16:19.580 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T10:16:19.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:19.655 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:19.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:19.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:19.757 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:19.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:19.784 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:19.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:19.792 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:19 smithi107 ceph-mon[102221]: osdmap e329: 8 total, 8 up, 8 in 2024-09-21T10:16:19.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:19 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-21T10:16:19.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:19 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-21T10:16:19.793 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:19 smithi107 ceph-mon[102221]: osdmap e330: 8 total, 8 up, 8 in 2024-09-21T10:16:19.793 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:19 smithi107 ceph-mon[105022]: osdmap e329: 8 total, 8 up, 8 in 2024-09-21T10:16:19.794 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:19 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-21T10:16:19.794 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:19 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-21T10:16:19.794 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:19 smithi107 ceph-mon[105022]: osdmap e330: 8 total, 8 up, 8 in 2024-09-21T10:16:19.885 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:19.911 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:19.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:19.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:20.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:20.042 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:20.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:20.143 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:20.166 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:20.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:20.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:20.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:19 smithi129 ceph-mon[88471]: osdmap e329: 8 total, 8 up, 8 in 2024-09-21T10:16:20.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:19 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-21T10:16:20.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:19 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-21T10:16:20.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:19 smithi129 ceph-mon[88471]: osdmap e330: 8 total, 8 up, 8 in 2024-09-21T10:16:20.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:20.297 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:20.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:20.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:20.399 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:20.422 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:20.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:20.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:20.524 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:20.546 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:20.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:20.632 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:16:20.632 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:16:20.648 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:16:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:16:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:16:20.670 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:20.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:20.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:20.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:20.798 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:20.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:20.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:20.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:20 smithi107 ceph-mon[102221]: 12.15 scrub starts 2024-09-21T10:16:20.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:20 smithi107 ceph-mon[102221]: pgmap v1153: 119 pgs: 2 active+remapped+backfilling, 117 active+clean; 5.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 8588/250968 objects misplaced (3.422%); 11 MiB/s, 177 objects/s recovering 2024-09-21T10:16:20.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:20 smithi107 ceph-mon[102221]: 12.15 scrub ok 2024-09-21T10:16:20.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:20 smithi107 ceph-mon[102221]: 12.12 starting backfill to osd.1 from (0'0,0'0] MAX to 327'8328 2024-09-21T10:16:20.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:20 smithi107 ceph-mon[102221]: 12.12 starting backfill to osd.5 from (0'0,0'0] MAX to 327'8328 2024-09-21T10:16:20.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:20 smithi107 ceph-mon[102221]: osdmap e331: 8 total, 8 up, 8 in 2024-09-21T10:16:20.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:20 smithi107 ceph-mon[105022]: 12.15 scrub starts 2024-09-21T10:16:20.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:20 smithi107 ceph-mon[105022]: pgmap v1153: 119 pgs: 2 active+remapped+backfilling, 117 active+clean; 5.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 8588/250968 objects misplaced (3.422%); 11 MiB/s, 177 objects/s recovering 2024-09-21T10:16:20.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:20 smithi107 ceph-mon[105022]: 12.15 scrub ok 2024-09-21T10:16:20.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:20 smithi107 ceph-mon[105022]: 12.12 starting backfill to osd.1 from (0'0,0'0] MAX to 327'8328 2024-09-21T10:16:20.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:20 smithi107 ceph-mon[105022]: 12.12 starting backfill to osd.5 from (0'0,0'0] MAX to 327'8328 2024-09-21T10:16:20.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:20 smithi107 ceph-mon[105022]: osdmap e331: 8 total, 8 up, 8 in 2024-09-21T10:16:20.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:20.927 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:20.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:20.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:21.029 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:21.050 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:21.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:21.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:21.152 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:21.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:20 smithi129 ceph-mon[88471]: 12.15 scrub starts 2024-09-21T10:16:21.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:20 smithi129 ceph-mon[88471]: pgmap v1153: 119 pgs: 2 active+remapped+backfilling, 117 active+clean; 5.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 8588/250968 objects misplaced (3.422%); 11 MiB/s, 177 objects/s recovering 2024-09-21T10:16:21.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:20 smithi129 ceph-mon[88471]: 12.15 scrub ok 2024-09-21T10:16:21.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:20 smithi129 ceph-mon[88471]: 12.12 starting backfill to osd.1 from (0'0,0'0] MAX to 327'8328 2024-09-21T10:16:21.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:20 smithi129 ceph-mon[88471]: 12.12 starting backfill to osd.5 from (0'0,0'0] MAX to 327'8328 2024-09-21T10:16:21.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:20 smithi129 ceph-mon[88471]: osdmap e331: 8 total, 8 up, 8 in 2024-09-21T10:16:21.173 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:21.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:21.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:21.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:21.303 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:21.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:21.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:21.330 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:16:21.333 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:21.406 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:21.427 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:21.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:21.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:21.517 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:16:21.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:21.591 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:21.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:21.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:21.693 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:21.715 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:21.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:21.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:21.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:21.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:16:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:16:21] "GET /metrics HTTP/1.1" 200 36819 "" "Prometheus/2.51.0" 2024-09-21T10:16:21.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:21.871 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:21.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:21.972 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:21.999 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:22.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:22.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:22.101 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:22.126 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:22.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:22.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:22.185 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:22.228 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:22.290 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:22.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:22.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:22.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:16:22.392 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:22.419 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:22.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:22.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:22.521 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:22.547 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:22.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:22.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:22.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:22.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:22.704 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:22.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:22.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:22.849 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:22.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:22.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:22 smithi129 ceph-mon[88471]: pgmap v1156: 119 pgs: 1 unknown, 2 active+remapped+backfilling, 116 active+clean; 4.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 6420/239892 objects misplaced (2.676%); 16 MiB/s, 251 objects/s recovering 2024-09-21T10:16:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:22 smithi129 ceph-mon[88471]: 12.13 starting backfill to osd.1 from (0'0,0'0] MAX to 328'8563 2024-09-21T10:16:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:22 smithi129 ceph-mon[88471]: osdmap e332: 8 total, 8 up, 8 in 2024-09-21T10:16:22.950 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:22.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:22 smithi107 ceph-mon[102221]: pgmap v1156: 119 pgs: 1 unknown, 2 active+remapped+backfilling, 116 active+clean; 4.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 6420/239892 objects misplaced (2.676%); 16 MiB/s, 251 objects/s recovering 2024-09-21T10:16:22.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:22 smithi107 ceph-mon[102221]: 12.13 starting backfill to osd.1 from (0'0,0'0] MAX to 328'8563 2024-09-21T10:16:22.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:22 smithi107 ceph-mon[102221]: osdmap e332: 8 total, 8 up, 8 in 2024-09-21T10:16:22.952 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:22 smithi107 ceph-mon[105022]: pgmap v1156: 119 pgs: 1 unknown, 2 active+remapped+backfilling, 116 active+clean; 4.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 6420/239892 objects misplaced (2.676%); 16 MiB/s, 251 objects/s recovering 2024-09-21T10:16:22.953 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:22 smithi107 ceph-mon[105022]: 12.13 starting backfill to osd.1 from (0'0,0'0] MAX to 328'8563 2024-09-21T10:16:22.953 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:22 smithi107 ceph-mon[105022]: osdmap e332: 8 total, 8 up, 8 in 2024-09-21T10:16:22.975 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:22.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:22.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:23.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:23.102 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:23.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:23.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:23.204 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:23.225 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:23.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:23.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:23.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:23.356 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:23.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:23.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:23.458 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:23.480 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:23.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:23.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:23.582 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:23.603 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:23.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:23.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:23.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:23.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:23.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:23.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:23.876 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:23.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:23.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:23.978 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:24.003 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:24.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:24.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:24.021 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:23 smithi107 ceph-mon[102221]: osdmap e333: 8 total, 8 up, 8 in 2024-09-21T10:16:24.021 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:23 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T10:16:24.021 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:23 smithi107 ceph-mon[105022]: osdmap e333: 8 total, 8 up, 8 in 2024-09-21T10:16:24.021 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:23 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T10:16:24.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:24.139 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:24.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:24.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:24.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:23 smithi129 ceph-mon[88471]: osdmap e333: 8 total, 8 up, 8 in 2024-09-21T10:16:24.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:23 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T10:16:24.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:24.275 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:24.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:24.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:24.377 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:24.399 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:24.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:24.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:24.501 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:24.521 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:24.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:24.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:24.623 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:24.645 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:24.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:24.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:24.747 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:24.768 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:24.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:24.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:24.869 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:24.890 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:24.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:24.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:24.992 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:25.003 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:24 smithi107 ceph-mon[105022]: 12.14 deep-scrub starts 2024-09-21T10:16:25.003 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:24 smithi107 ceph-mon[105022]: pgmap v1159: 119 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 115 active+clean; 5.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 10457/243573 objects misplaced (4.293%); 21 MiB/s, 337 objects/s recovering 2024-09-21T10:16:25.004 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:24 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T10:16:25.004 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:24 smithi107 ceph-mon[105022]: osdmap e334: 8 total, 8 up, 8 in 2024-09-21T10:16:25.004 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:24 smithi107 ceph-mon[105022]: osdmap e335: 8 total, 8 up, 8 in 2024-09-21T10:16:25.004 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:24 smithi107 ceph-mon[102221]: 12.14 deep-scrub starts 2024-09-21T10:16:25.004 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:24 smithi107 ceph-mon[102221]: pgmap v1159: 119 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 115 active+clean; 5.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 10457/243573 objects misplaced (4.293%); 21 MiB/s, 337 objects/s recovering 2024-09-21T10:16:25.004 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:24 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T10:16:25.004 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:24 smithi107 ceph-mon[102221]: osdmap e334: 8 total, 8 up, 8 in 2024-09-21T10:16:25.004 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:24 smithi107 ceph-mon[102221]: osdmap e335: 8 total, 8 up, 8 in 2024-09-21T10:16:25.013 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:25.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:25.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:25.114 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:25.135 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:25.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:25.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:24 smithi129 ceph-mon[88471]: 12.14 deep-scrub starts 2024-09-21T10:16:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:24 smithi129 ceph-mon[88471]: pgmap v1159: 119 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 115 active+clean; 5.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 10457/243573 objects misplaced (4.293%); 21 MiB/s, 337 objects/s recovering 2024-09-21T10:16:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:24 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T10:16:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:24 smithi129 ceph-mon[88471]: osdmap e334: 8 total, 8 up, 8 in 2024-09-21T10:16:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:24 smithi129 ceph-mon[88471]: osdmap e335: 8 total, 8 up, 8 in 2024-09-21T10:16:25.237 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:25.258 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:25.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:25.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:25.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:25.382 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:25.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:25.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:25.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:25.506 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:25.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:25.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:25.608 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:25.630 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:25.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:25.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:25.732 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:25.758 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:25.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:25.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:25 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-21T10:16:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:25 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-21T10:16:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:25 smithi107 ceph-mon[102221]: osdmap e336: 8 total, 8 up, 8 in 2024-09-21T10:16:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:25 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-21T10:16:25.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:25 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-21T10:16:25.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:25 smithi107 ceph-mon[105022]: osdmap e336: 8 total, 8 up, 8 in 2024-09-21T10:16:25.860 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:25.882 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:25.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:25.984 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:26.007 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:26.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:26.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:26.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:26.130 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:26.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:26.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:25 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-21T10:16:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:25 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-21T10:16:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:25 smithi129 ceph-mon[88471]: osdmap e336: 8 total, 8 up, 8 in 2024-09-21T10:16:26.231 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:26.256 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:26.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:26.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:26.389 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:26.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:26.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:26.491 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:26.514 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:26.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:26.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:26.615 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:26.641 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:26.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:26.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:26.743 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:26.771 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:26.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:26.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:26.873 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:26.898 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:26.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:26.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:27.000 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:27.023 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:27.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:27.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:27.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:26 smithi107 ceph-mon[102221]: pgmap v1162: 119 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 115 active+clean; 4.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 10457/242748 objects misplaced (4.308%); 5.4 MiB/s, 85 objects/s recovering 2024-09-21T10:16:27.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:26 smithi107 ceph-mon[102221]: 12.14 starting backfill to osd.5 from (0'0,0'0] MAX to 333'8672 2024-09-21T10:16:27.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:26 smithi107 ceph-mon[102221]: 12.14 starting backfill to osd.7 from (0'0,0'0] MAX to 333'8672 2024-09-21T10:16:27.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:27.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:27.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:26 smithi107 ceph-mon[102221]: osdmap e337: 8 total, 8 up, 8 in 2024-09-21T10:16:27.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:27.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:26 smithi107 ceph-mon[105022]: pgmap v1162: 119 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 115 active+clean; 4.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 10457/242748 objects misplaced (4.308%); 5.4 MiB/s, 85 objects/s recovering 2024-09-21T10:16:27.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:26 smithi107 ceph-mon[105022]: 12.14 starting backfill to osd.5 from (0'0,0'0] MAX to 333'8672 2024-09-21T10:16:27.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:26 smithi107 ceph-mon[105022]: 12.14 starting backfill to osd.7 from (0'0,0'0] MAX to 333'8672 2024-09-21T10:16:27.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:27.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:27.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:26 smithi107 ceph-mon[105022]: osdmap e337: 8 total, 8 up, 8 in 2024-09-21T10:16:27.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:27.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:27.168 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:27.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:27.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:26 smithi129 ceph-mon[88471]: pgmap v1162: 119 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 115 active+clean; 4.9 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 10457/242748 objects misplaced (4.308%); 5.4 MiB/s, 85 objects/s recovering 2024-09-21T10:16:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:26 smithi129 ceph-mon[88471]: 12.14 starting backfill to osd.5 from (0'0,0'0] MAX to 333'8672 2024-09-21T10:16:27.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:26 smithi129 ceph-mon[88471]: 12.14 starting backfill to osd.7 from (0'0,0'0] MAX to 333'8672 2024-09-21T10:16:27.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:27.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:27.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:26 smithi129 ceph-mon[88471]: osdmap e337: 8 total, 8 up, 8 in 2024-09-21T10:16:27.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:27.269 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:27.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:27.299 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:27.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:27.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:27.420 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:27.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:27.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:27.522 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:27.547 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:27.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:27.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:27.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:27.690 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:27.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:27.791 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:27.826 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:27.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:27.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:27.928 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:27.957 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:27.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:27.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:28.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:28.095 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:28.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:28.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:28.197 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:28.223 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:28.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:28.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:28.325 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:28.352 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:28.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:28.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:28.454 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:28.479 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:28.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:28.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:28.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:28.621 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:28.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:28.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:28.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:28.745 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:28.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:28.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:28.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:28.887 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:28.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:28.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:28.988 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:29.017 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:29.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:29.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:29.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:28 smithi107 ceph-mon[102221]: pgmap v1165: 119 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 115 active+clean; 4.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 446 op/s; 13529/238215 objects misplaced (5.679%); 11 MiB/s, 183 objects/s recovering 2024-09-21T10:16:29.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:28 smithi107 ceph-mon[105022]: pgmap v1165: 119 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 115 active+clean; 4.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 446 op/s; 13529/238215 objects misplaced (5.679%); 11 MiB/s, 183 objects/s recovering 2024-09-21T10:16:29.118 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:29.146 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:29.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:29.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:29.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:28 smithi129 ceph-mon[88471]: pgmap v1165: 119 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 115 active+clean; 4.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 446 op/s; 13529/238215 objects misplaced (5.679%); 11 MiB/s, 183 objects/s recovering 2024-09-21T10:16:29.248 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:29.287 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:29.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:29.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:29.388 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:29.430 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:29.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:29.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:29.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:29.567 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:29.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:29.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:29.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:29.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:29.708 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:29.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:29.809 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:29.836 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:29.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:29.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:29.938 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:29.964 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:29.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:29.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:30.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:30.098 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:30.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:30.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:30.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:29 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-21T10:16:30.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:29 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-21T10:16:30.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:29 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-21T10:16:30.200 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:30.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:30.227 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:30.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:30.251 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:30.252 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:30.328 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:30.350 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:30.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:30.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:30.452 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:30.474 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:30.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:30.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:30.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:30.599 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:30.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:30.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:16:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:16:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:16:30.700 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:30.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:30.728 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:30.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:30.745 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:30 smithi107 ceph-mon[102221]: pgmap v1166: 119 pgs: 2 active+remapped+backfill_wait, 1 active+remapped+backfilling, 116 active+clean; 4.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 428 op/s; 10465/236247 objects misplaced (4.430%); 31 MiB/s, 504 objects/s recovering 2024-09-21T10:16:30.745 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:30 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-21T10:16:30.745 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:30 smithi107 ceph-mon[102221]: osdmap e338: 8 total, 8 up, 8 in 2024-09-21T10:16:30.746 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:30 smithi107 ceph-mon[105022]: pgmap v1166: 119 pgs: 2 active+remapped+backfill_wait, 1 active+remapped+backfilling, 116 active+clean; 4.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 428 op/s; 10465/236247 objects misplaced (4.430%); 31 MiB/s, 504 objects/s recovering 2024-09-21T10:16:30.746 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:30 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-21T10:16:30.746 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:30 smithi107 ceph-mon[105022]: osdmap e338: 8 total, 8 up, 8 in 2024-09-21T10:16:30.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:30.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:30.853 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:30.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:30.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:30.978 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:30.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:30.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:31.036 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:16:31.037 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:16:31.080 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:31.102 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:31.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:31.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:31.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:30 smithi129 ceph-mon[88471]: pgmap v1166: 119 pgs: 2 active+remapped+backfill_wait, 1 active+remapped+backfilling, 116 active+clean; 4.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 428 op/s; 10465/236247 objects misplaced (4.430%); 31 MiB/s, 504 objects/s recovering 2024-09-21T10:16:31.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:30 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-21T10:16:31.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:30 smithi129 ceph-mon[88471]: osdmap e338: 8 total, 8 up, 8 in 2024-09-21T10:16:31.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:31.225 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:31.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:31.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:31.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:31.349 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:31.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:31.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:31.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:31.472 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:31.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:31.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:31.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:31.596 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:31.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:31.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:31.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:31.720 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:31.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:31.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:31.736 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T10:16:31.822 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:31.847 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:31.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:31.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:31.949 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:31.992 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:31.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:31.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:32.094 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:32.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:31 smithi107 ceph-mon[102221]: osdmap e339: 8 total, 8 up, 8 in 2024-09-21T10:16:32.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:31 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2298737884' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:16:32.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:31 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:16:32.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:31 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "24"} : dispatch 2024-09-21T10:16:32.110 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:16:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:16:31] "GET /metrics HTTP/1.1" 200 36823 "" "Prometheus/2.51.0" 2024-09-21T10:16:32.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:31 smithi107 ceph-mon[105022]: osdmap e339: 8 total, 8 up, 8 in 2024-09-21T10:16:32.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:31 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2298737884' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:16:32.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:31 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:16:32.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:31 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "24"} : dispatch 2024-09-21T10:16:32.138 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:32.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:32.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:32.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:31 smithi129 ceph-mon[88471]: osdmap e339: 8 total, 8 up, 8 in 2024-09-21T10:16:32.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:31 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2298737884' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:16:32.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:31 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:16:32.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:31 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "24"} : dispatch 2024-09-21T10:16:32.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:32.276 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:32.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:32.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:32.378 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:32.407 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:32.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:32.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:32.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:32.536 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:32.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:32.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:32.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:32.660 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:32.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:32.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:32.762 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:32.785 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:32.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:32.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:32.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:32.909 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:32.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:32.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:33.010 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:33.033 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:33.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:33.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:33.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:32 smithi107 ceph-mon[102221]: pgmap v1169: 119 pgs: 2 active+remapped+backfill_wait, 1 active+remapped+backfilling, 116 active+clean; 4.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 541 op/s; 10465/233235 objects misplaced (4.487%); 30 MiB/s, 491 objects/s recovering 2024-09-21T10:16:33.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:32 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:16:33.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:32 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "24"}]': finished 2024-09-21T10:16:33.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:32 smithi107 ceph-mon[102221]: osdmap e340: 8 total, 8 up, 8 in 2024-09-21T10:16:33.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:32 smithi107 ceph-mon[102221]: 12.16 starting backfill to osd.2 from (0'0,0'0] MAX to 337'8643 2024-09-21T10:16:33.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:32 smithi107 ceph-mon[102221]: 12.16 starting backfill to osd.6 from (0'0,0'0] MAX to 337'8643 2024-09-21T10:16:33.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:32 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:16:33.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:32 smithi107 ceph-mon[105022]: pgmap v1169: 119 pgs: 2 active+remapped+backfill_wait, 1 active+remapped+backfilling, 116 active+clean; 4.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 541 op/s; 10465/233235 objects misplaced (4.487%); 30 MiB/s, 491 objects/s recovering 2024-09-21T10:16:33.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:32 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:16:33.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:32 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "24"}]': finished 2024-09-21T10:16:33.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:32 smithi107 ceph-mon[105022]: osdmap e340: 8 total, 8 up, 8 in 2024-09-21T10:16:33.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:32 smithi107 ceph-mon[105022]: 12.16 starting backfill to osd.2 from (0'0,0'0] MAX to 337'8643 2024-09-21T10:16:33.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:32 smithi107 ceph-mon[105022]: 12.16 starting backfill to osd.6 from (0'0,0'0] MAX to 337'8643 2024-09-21T10:16:33.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:32 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:16:33.112 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:32 smithi129 ceph-mon[88471]: pgmap v1169: 119 pgs: 2 active+remapped+backfill_wait, 1 active+remapped+backfilling, 116 active+clean; 4.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 541 op/s; 10465/233235 objects misplaced (4.487%); 30 MiB/s, 491 objects/s recovering 2024-09-21T10:16:33.112 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:32 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:16:33.112 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:32 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "24"}]': finished 2024-09-21T10:16:33.112 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:32 smithi129 ceph-mon[88471]: osdmap e340: 8 total, 8 up, 8 in 2024-09-21T10:16:33.112 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:32 smithi129 ceph-mon[88471]: 12.16 starting backfill to osd.2 from (0'0,0'0] MAX to 337'8643 2024-09-21T10:16:33.112 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:32 smithi129 ceph-mon[88471]: 12.16 starting backfill to osd.6 from (0'0,0'0] MAX to 337'8643 2024-09-21T10:16:33.112 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:32 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:16:33.135 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:33.158 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:33.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:33.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T10:16:33.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:33.282 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:33.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:33.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:33.384 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:33.407 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:33.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:33.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:33.509 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:33.533 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:33.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:33.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:33.635 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:33.657 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:33.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:33.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:33.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:33.784 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:33.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:33.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:33.885 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:33.912 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:33.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:33.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:34.013 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:34.017 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:33 smithi129 ceph-mon[88471]: osdmap e341: 8 total, 8 up, 8 in 2024-09-21T10:16:34.050 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:34.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:34.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:34.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:33 smithi107 ceph-mon[102221]: osdmap e341: 8 total, 8 up, 8 in 2024-09-21T10:16:34.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:33 smithi107 ceph-mon[105022]: osdmap e341: 8 total, 8 up, 8 in 2024-09-21T10:16:34.152 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:34.174 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:34.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:34.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:34.277 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:34.304 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:34.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:34.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:34.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:34.429 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:34.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:34.531 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:34.557 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:34.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:34.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:34.659 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:34.685 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:34.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:34.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:34.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:34.824 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:34.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:34.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:34.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:34.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:34.950 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:34.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:35.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:34 smithi129 ceph-mon[88471]: pgmap v1172: 119 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 115 active+clean; 4.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 984 op/s; 11653/224436 objects misplaced (5.192%); 22 MiB/s, 354 objects/s recovering 2024-09-21T10:16:35.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:34 smithi129 ceph-mon[88471]: 12.17 starting backfill to osd.3 from (0'0,0'0] MAX to 338'8773 2024-09-21T10:16:35.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:34 smithi129 ceph-mon[88471]: osdmap e342: 8 total, 8 up, 8 in 2024-09-21T10:16:35.007 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:34 smithi129 ceph-mon[88471]: 12.17 starting backfill to osd.6 from (0'0,0'0] MAX to 338'8773 2024-09-21T10:16:35.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:35.092 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:35.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:35.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:35.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:34 smithi107 ceph-mon[105022]: pgmap v1172: 119 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 115 active+clean; 4.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 984 op/s; 11653/224436 objects misplaced (5.192%); 22 MiB/s, 354 objects/s recovering 2024-09-21T10:16:35.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:34 smithi107 ceph-mon[105022]: 12.17 starting backfill to osd.3 from (0'0,0'0] MAX to 338'8773 2024-09-21T10:16:35.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:34 smithi107 ceph-mon[105022]: osdmap e342: 8 total, 8 up, 8 in 2024-09-21T10:16:35.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:34 smithi107 ceph-mon[105022]: 12.17 starting backfill to osd.6 from (0'0,0'0] MAX to 338'8773 2024-09-21T10:16:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:34 smithi107 ceph-mon[102221]: pgmap v1172: 119 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 115 active+clean; 4.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 984 op/s; 11653/224436 objects misplaced (5.192%); 22 MiB/s, 354 objects/s recovering 2024-09-21T10:16:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:34 smithi107 ceph-mon[102221]: 12.17 starting backfill to osd.3 from (0'0,0'0] MAX to 338'8773 2024-09-21T10:16:35.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:34 smithi107 ceph-mon[102221]: osdmap e342: 8 total, 8 up, 8 in 2024-09-21T10:16:35.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:34 smithi107 ceph-mon[102221]: 12.17 starting backfill to osd.6 from (0'0,0'0] MAX to 338'8773 2024-09-21T10:16:35.194 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:35.222 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:35.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:35.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:35.324 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:35.358 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:35.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:35.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:35.459 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:35.493 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:35.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:35.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:35.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:35.640 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:35.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:35.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:35.741 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:35.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:35.769 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:35.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:35.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:35.909 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:35.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:35.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:36.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:36.044 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:36.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:36.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:36.075 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:35 smithi129 ceph-mon[88471]: osdmap e343: 8 total, 8 up, 8 in 2024-09-21T10:16:36.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:35 smithi107 ceph-mon[102221]: osdmap e343: 8 total, 8 up, 8 in 2024-09-21T10:16:36.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:35 smithi107 ceph-mon[105022]: osdmap e343: 8 total, 8 up, 8 in 2024-09-21T10:16:36.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:36.178 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:36.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:36.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:36.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:36.318 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:36.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:36.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:36.420 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:36.454 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:36.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:36.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:36.555 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:36.582 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:36.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:36.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:36.683 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:36.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:36.707 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:36.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:36.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:36.830 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:36.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:36.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:36.931 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:36.960 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:36.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:36.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:37.061 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:37.087 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:37.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:37.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:36 smithi107 ceph-mon[102221]: pgmap v1175: 119 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 115 active+clean; 4.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 789 op/s; 11653/223758 objects misplaced (5.208%); 22 MiB/s, 354 objects/s recovering 2024-09-21T10:16:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:36 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:37.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:36 smithi107 ceph-mon[105022]: pgmap v1175: 119 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 115 active+clean; 4.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 789 op/s; 11653/223758 objects misplaced (5.208%); 22 MiB/s, 354 objects/s recovering 2024-09-21T10:16:37.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:36 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:37.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:36 smithi129 ceph-mon[88471]: pgmap v1175: 119 pgs: 1 active+remapped+backfilling, 3 active+remapped+backfill_wait, 115 active+clean; 4.6 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 789 op/s; 11653/223758 objects misplaced (5.208%); 22 MiB/s, 354 objects/s recovering 2024-09-21T10:16:37.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:36 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:37.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:37.231 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:37.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:37.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:37.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:37.375 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:37.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:37.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:37.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:37.515 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:37.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:37.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:37.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:37.652 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:37.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:37.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:37.754 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:37.779 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:37.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:37.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:37.881 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:37.909 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:37.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:37.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:38.011 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:38.038 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:38.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:38.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:38.139 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:38.166 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:38.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:38.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:38.268 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:38.290 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:38.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:38.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:38.392 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:38.418 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:38.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:38.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:38.520 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:38.550 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:38.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:38.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:38 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:38 smithi107 ceph-mon[105022]: pgmap v1176: 119 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 115 active+clean; 4.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 888 op/s; 12737/218133 objects misplaced (5.839%); 33 MiB/s, 527 objects/s recovering 2024-09-21T10:16:38.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:38.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:38 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:38.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:38 smithi107 ceph-mon[102221]: pgmap v1176: 119 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 115 active+clean; 4.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 888 op/s; 12737/218133 objects misplaced (5.839%); 33 MiB/s, 527 objects/s recovering 2024-09-21T10:16:38.652 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:38.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:38.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:38 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:38.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:38 smithi129 ceph-mon[88471]: pgmap v1176: 119 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 115 active+clean; 4.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 888 op/s; 12737/218133 objects misplaced (5.839%); 33 MiB/s, 527 objects/s recovering 2024-09-21T10:16:38.677 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:38.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:38.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:38.779 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:38.818 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:38.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:38.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:38.940 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:38.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:38.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:39.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:39.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:39.079 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:39.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:39.180 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:39.205 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:39.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:39.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:39.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:39.339 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:39.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:39.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:39.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:39.481 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:39.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:39.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:39.583 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:39.611 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:39.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:39.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:39.713 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:39.755 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:39.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:39.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:39.857 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:39.884 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:39.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:39.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:39.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:40.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:40.020 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:40.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:40.121 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:40.157 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:40.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:40.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:40.258 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:40.283 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:40.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:40.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:40.384 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:40.405 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:40.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:40.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:40.471 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:40 smithi129 ceph-mon[88471]: pgmap v1177: 119 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 115 active+clean; 4.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 431 op/s; 12480/215820 objects misplaced (5.783%); 15 MiB/s, 235 objects/s recovering 2024-09-21T10:16:40.471 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:16:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:16:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:16:40.507 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:40.529 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:40.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:40.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:40.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:40.666 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:40.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:40.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:40.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:40.805 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:40.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:40.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:40 smithi107 ceph-mon[102221]: pgmap v1177: 119 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 115 active+clean; 4.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 431 op/s; 12480/215820 objects misplaced (5.783%); 15 MiB/s, 235 objects/s recovering 2024-09-21T10:16:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:40 smithi107 ceph-mon[105022]: pgmap v1177: 119 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 115 active+clean; 4.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 431 op/s; 12480/215820 objects misplaced (5.783%); 15 MiB/s, 235 objects/s recovering 2024-09-21T10:16:40.907 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:40.971 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:40.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:40.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:41.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:41.124 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:41.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:41.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:41.225 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:41.255 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:41.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:41.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:41.356 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:41.388 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:41.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:41.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:41.489 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:41.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:41.528 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:41.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:41.629 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:41.660 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:41.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:41.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:41.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:41.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:41.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:41.800 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:41.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:41.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:41.902 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:41.943 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:41.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:41.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:42.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:42.080 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:42.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:42.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:42.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:16:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:16:41] "GET /metrics HTTP/1.1" 200 36825 "" "Prometheus/2.51.0" 2024-09-21T10:16:42.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:42.213 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:42.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:42.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:42.314 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:42.341 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:42.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:42.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:42.442 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:42.469 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:42.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:42.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:42.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:42.606 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:42.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:42.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:42 smithi107 ceph-mon[102221]: pgmap v1178: 119 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 115 active+clean; 4.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 492 op/s; 11446/212622 objects misplaced (5.383%); 16 MiB/s, 248 objects/s recovering 2024-09-21T10:16:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:42 smithi107 ceph-mon[105022]: pgmap v1178: 119 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 115 active+clean; 4.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 492 op/s; 11446/212622 objects misplaced (5.383%); 16 MiB/s, 248 objects/s recovering 2024-09-21T10:16:42.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:42.740 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:42.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:42.842 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:42.872 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:42.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:42.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:42 smithi129 ceph-mon[88471]: pgmap v1178: 119 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 115 active+clean; 4.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 492 op/s; 11446/212622 objects misplaced (5.383%); 16 MiB/s, 248 objects/s recovering 2024-09-21T10:16:42.974 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:43.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:43.002 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:43.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:43.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:43.143 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:43.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:43.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:43.245 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:43.274 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:43.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:43.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:43.376 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:43.402 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:43.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:43.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:43.503 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:43.529 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:43.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:43.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:43.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:43.664 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:43.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:43.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:43.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:43.798 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:43.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:43.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:43.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:43 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "25"} : dispatch 2024-09-21T10:16:43.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:43 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "25"} : dispatch 2024-09-21T10:16:43.899 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:43.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:43 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "25"} : dispatch 2024-09-21T10:16:43.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:43.921 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:43.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:44.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:44.052 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:44.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:44.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:44.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:44.182 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:44.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:44.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:44.284 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:44.307 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:44.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:44.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:44.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:44.437 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:44.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:44.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:44.539 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:44.562 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:44.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:44.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:44.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:44.690 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:44.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:44.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:44.791 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:44.814 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:44.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:44.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:44.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:44 smithi107 ceph-mon[102221]: pgmap v1179: 119 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 115 active+clean; 4.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 801 op/s; 10051/201237 objects misplaced (4.995%); 15 MiB/s, 247 objects/s recovering 2024-09-21T10:16:44.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:44 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "25"}]': finished 2024-09-21T10:16:44.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:44 smithi107 ceph-mon[102221]: osdmap e344: 8 total, 8 up, 8 in 2024-09-21T10:16:44.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:44 smithi107 ceph-mon[105022]: pgmap v1179: 119 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 115 active+clean; 4.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 801 op/s; 10051/201237 objects misplaced (4.995%); 15 MiB/s, 247 objects/s recovering 2024-09-21T10:16:44.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:44 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "25"}]': finished 2024-09-21T10:16:44.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:44 smithi107 ceph-mon[105022]: osdmap e344: 8 total, 8 up, 8 in 2024-09-21T10:16:44.915 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:44 smithi129 ceph-mon[88471]: pgmap v1179: 119 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 115 active+clean; 4.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 801 op/s; 10051/201237 objects misplaced (4.995%); 15 MiB/s, 247 objects/s recovering 2024-09-21T10:16:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:44 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "25"}]': finished 2024-09-21T10:16:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:44 smithi129 ceph-mon[88471]: osdmap e344: 8 total, 8 up, 8 in 2024-09-21T10:16:44.941 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:44.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:44.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:45.043 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:45.064 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:45.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:45.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:45.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:45.192 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:45.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:45.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:45.294 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:45.319 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:45.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:45.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:45.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:45.446 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:45.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:45.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:45.548 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:45.574 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:45.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:45.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:45.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:45.702 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:45.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:45.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:45.804 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:45.825 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:45.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:45.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:45.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:45 smithi107 ceph-mon[102221]: osdmap e345: 8 total, 8 up, 8 in 2024-09-21T10:16:45.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:45 smithi107 ceph-mon[105022]: osdmap e345: 8 total, 8 up, 8 in 2024-09-21T10:16:45.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:45 smithi129 ceph-mon[88471]: osdmap e345: 8 total, 8 up, 8 in 2024-09-21T10:16:45.927 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:45.955 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:45.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:45.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:46.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:46.096 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:46.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:46.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:46.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:46.242 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:46.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:46.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:46.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:46.379 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:46.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:46.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:46.481 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:46.509 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:46.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:46.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:46.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:46.650 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:46.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:46.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:46.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:46.787 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:46.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:46.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:46.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:46 smithi107 ceph-mon[102221]: pgmap v1182: 119 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 115 active+clean; 4.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 775 op/s; 10051/199509 objects misplaced (5.038%); 6.4 MiB/s, 102 objects/s recovering 2024-09-21T10:16:46.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:46 smithi107 ceph-mon[102221]: 12.18 starting backfill to osd.1 from (0'0,0'0] MAX to 343'8983 2024-09-21T10:16:46.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:46 smithi107 ceph-mon[102221]: osdmap e346: 8 total, 8 up, 8 in 2024-09-21T10:16:46.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:46 smithi107 ceph-mon[102221]: 12.18 starting backfill to osd.4 from (0'0,0'0] MAX to 343'8983 2024-09-21T10:16:46.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:46 smithi107 ceph-mon[102221]: 12.18 starting backfill to osd.7 from (0'0,0'0] MAX to 343'8983 2024-09-21T10:16:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:46 smithi107 ceph-mon[105022]: pgmap v1182: 119 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 115 active+clean; 4.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 775 op/s; 10051/199509 objects misplaced (5.038%); 6.4 MiB/s, 102 objects/s recovering 2024-09-21T10:16:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:46 smithi107 ceph-mon[105022]: 12.18 starting backfill to osd.1 from (0'0,0'0] MAX to 343'8983 2024-09-21T10:16:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:46 smithi107 ceph-mon[105022]: osdmap e346: 8 total, 8 up, 8 in 2024-09-21T10:16:46.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:46 smithi107 ceph-mon[105022]: 12.18 starting backfill to osd.4 from (0'0,0'0] MAX to 343'8983 2024-09-21T10:16:46.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:46 smithi107 ceph-mon[105022]: 12.18 starting backfill to osd.7 from (0'0,0'0] MAX to 343'8983 2024-09-21T10:16:46.889 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:46 smithi129 ceph-mon[88471]: pgmap v1182: 119 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 115 active+clean; 4.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 775 op/s; 10051/199509 objects misplaced (5.038%); 6.4 MiB/s, 102 objects/s recovering 2024-09-21T10:16:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:46 smithi129 ceph-mon[88471]: 12.18 starting backfill to osd.1 from (0'0,0'0] MAX to 343'8983 2024-09-21T10:16:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:46 smithi129 ceph-mon[88471]: osdmap e346: 8 total, 8 up, 8 in 2024-09-21T10:16:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:46 smithi129 ceph-mon[88471]: 12.18 starting backfill to osd.4 from (0'0,0'0] MAX to 343'8983 2024-09-21T10:16:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:46 smithi129 ceph-mon[88471]: 12.18 starting backfill to osd.7 from (0'0,0'0] MAX to 343'8983 2024-09-21T10:16:46.919 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:46.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:46.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:47.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:47.061 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:47.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:47.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:47.163 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:47.192 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:47.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:47.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:47.294 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:47.322 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:47.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:47.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:47.424 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:47.452 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:47.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:47.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:47.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:47.596 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:47.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:47.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:47.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:47.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:47.737 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:47.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:47.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:47 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T10:16:47.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:47 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T10:16:47.763 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:16:47.838 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:47.890 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:47.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:47.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:47.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:47 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-21T10:16:47.992 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:48.018 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:48.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:48.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:48.119 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:48.143 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:48.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:48.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:48.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:48.270 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:48.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:48.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:48.372 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:48.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:48.394 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:48.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:48.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:48.522 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:48.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:48.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:48 smithi107 ceph-mon[102221]: pgmap v1184: 119 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 115 active+clean; 3.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 5981/192879 objects misplaced (3.101%); 24 MiB/s, 388 objects/s recovering 2024-09-21T10:16:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:48 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-21T10:16:48.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:48 smithi107 ceph-mon[102221]: osdmap e347: 8 total, 8 up, 8 in 2024-09-21T10:16:48.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:48 smithi107 ceph-mon[105022]: pgmap v1184: 119 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 115 active+clean; 3.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 5981/192879 objects misplaced (3.101%); 24 MiB/s, 388 objects/s recovering 2024-09-21T10:16:48.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:48 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-21T10:16:48.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:48 smithi107 ceph-mon[105022]: osdmap e347: 8 total, 8 up, 8 in 2024-09-21T10:16:48.625 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:48.646 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:48.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:48.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:48.748 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:48.769 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:48.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:48.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:48.870 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:48.892 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:48.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:48.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:48.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:48 smithi129 ceph-mon[88471]: pgmap v1184: 119 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 115 active+clean; 3.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 5981/192879 objects misplaced (3.101%); 24 MiB/s, 388 objects/s recovering 2024-09-21T10:16:48.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:48 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-21T10:16:48.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:48 smithi129 ceph-mon[88471]: osdmap e347: 8 total, 8 up, 8 in 2024-09-21T10:16:48.939 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:48.939 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:16:48.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:49.019 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:49.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:49.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:49.120 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:49.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:49.143 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:49.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:49.244 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:49.271 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:49.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:49.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:49.373 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:49.394 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:49.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:49.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:49.496 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:49.509 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:16:49.520 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:49.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:49.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:49.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:49.663 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:49.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:49.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:49.765 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:49.795 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:49.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:49.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:49.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:49 smithi107 ceph-mon[102221]: osdmap e348: 8 total, 8 up, 8 in 2024-09-21T10:16:49.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:49 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/300430439' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:16:49.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:49 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:16:49.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:49 smithi107 ceph-mon[102221]: 12.14 scrub starts 2024-09-21T10:16:49.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:49 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:16:49.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:49 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:16:49.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:49 smithi107 ceph-mon[102221]: osdmap e349: 8 total, 8 up, 8 in 2024-09-21T10:16:49.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:49 smithi107 ceph-mon[105022]: osdmap e348: 8 total, 8 up, 8 in 2024-09-21T10:16:49.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:49 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/300430439' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:16:49.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:49 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:16:49.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:49 smithi107 ceph-mon[105022]: 12.14 scrub starts 2024-09-21T10:16:49.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:49 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:16:49.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:49 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:16:49.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:49 smithi107 ceph-mon[105022]: osdmap e349: 8 total, 8 up, 8 in 2024-09-21T10:16:49.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:49 smithi129 ceph-mon[88471]: osdmap e348: 8 total, 8 up, 8 in 2024-09-21T10:16:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:49 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/300430439' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:16:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:49 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:16:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:49 smithi129 ceph-mon[88471]: 12.14 scrub starts 2024-09-21T10:16:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:49 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:16:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:49 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:16:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:49 smithi129 ceph-mon[88471]: osdmap e349: 8 total, 8 up, 8 in 2024-09-21T10:16:49.923 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:49.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:49.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:50.025 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:50.051 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:50.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:50.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:50.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:50.192 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:50.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:50.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:50.294 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:50.336 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:50.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:50.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:50.438 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:50.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:50.466 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:50.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:50.522 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:16:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:16:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:16:50.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:50.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:50.602 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:50.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:50.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:50.746 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:50.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:50.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:50.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:50.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:50 smithi107 ceph-mon[102221]: pgmap v1187: 119 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_wait, 114 active+clean; 3.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 11484/189762 objects misplaced (6.052%); 25 MiB/s, 392 objects/s recovering 2024-09-21T10:16:50.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:50 smithi107 ceph-mon[102221]: 12.19 starting backfill to osd.0 from (0'0,0'0] MAX to 346'9041 2024-09-21T10:16:50.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:50 smithi107 ceph-mon[102221]: 12.19 starting backfill to osd.5 from (0'0,0'0] MAX to 346'9041 2024-09-21T10:16:50.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:50 smithi107 ceph-mon[105022]: pgmap v1187: 119 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_wait, 114 active+clean; 3.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 11484/189762 objects misplaced (6.052%); 25 MiB/s, 392 objects/s recovering 2024-09-21T10:16:50.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:50 smithi107 ceph-mon[105022]: 12.19 starting backfill to osd.0 from (0'0,0'0] MAX to 346'9041 2024-09-21T10:16:50.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:50 smithi107 ceph-mon[105022]: 12.19 starting backfill to osd.5 from (0'0,0'0] MAX to 346'9041 2024-09-21T10:16:50.887 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:50.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:50.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:50.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:50 smithi129 ceph-mon[88471]: pgmap v1187: 119 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_wait, 114 active+clean; 3.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 11484/189762 objects misplaced (6.052%); 25 MiB/s, 392 objects/s recovering 2024-09-21T10:16:50.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:50 smithi129 ceph-mon[88471]: 12.19 starting backfill to osd.0 from (0'0,0'0] MAX to 346'9041 2024-09-21T10:16:50.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:50 smithi129 ceph-mon[88471]: 12.19 starting backfill to osd.5 from (0'0,0'0] MAX to 346'9041 2024-09-21T10:16:50.988 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:51.012 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:51.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:51.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:51.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:51.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:51.153 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:51.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:51.254 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:51.291 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:51.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:51.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:51.392 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:51.422 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:51.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:51.524 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:51.549 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:51.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:51.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:51.650 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:51.677 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:51.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:51.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:51.779 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:51.822 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:51.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:51.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:51.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:51.967 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:51.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:51.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:52.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:52.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:52.106 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:52.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:52.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:16:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:16:51] "GET /metrics HTTP/1.1" 200 36825 "" "Prometheus/2.51.0" 2024-09-21T10:16:52.207 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:52.231 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:52.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:52.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:52.332 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:52.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:52.361 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:52.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:52.463 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:52.501 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:52.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:52.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:52.603 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:52.632 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:52.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:52.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:52.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:52.769 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:52.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:52.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:52.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:52 smithi107 ceph-mon[102221]: pgmap v1189: 119 pgs: 1 activating+remapped, 2 active+remapped+backfilling, 3 active+remapped+backfill_wait, 113 active+clean; 3.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 646 op/s; 15226/188043 objects misplaced (8.097%); 20 MiB/s, 327 objects/s recovering 2024-09-21T10:16:52.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:52 smithi107 ceph-mon[105022]: pgmap v1189: 119 pgs: 1 activating+remapped, 2 active+remapped+backfilling, 3 active+remapped+backfill_wait, 113 active+clean; 3.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 646 op/s; 15226/188043 objects misplaced (8.097%); 20 MiB/s, 327 objects/s recovering 2024-09-21T10:16:52.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:52.905 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:52.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:52.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:52 smithi129 ceph-mon[88471]: pgmap v1189: 119 pgs: 1 activating+remapped, 2 active+remapped+backfilling, 3 active+remapped+backfill_wait, 113 active+clean; 3.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 646 op/s; 15226/188043 objects misplaced (8.097%); 20 MiB/s, 327 objects/s recovering 2024-09-21T10:16:53.007 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:53.034 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:53.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:53.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:53.136 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:53.158 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:53.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:53.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:53.260 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:53.281 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:53.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:53.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:53.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:53.404 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:53.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:53.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:53.505 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:53.527 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:53.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:53.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:53.629 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:53.657 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:53.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:53.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:53.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:53.795 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:53.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:53.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:53.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:53 smithi107 ceph-mon[102221]: osdmap e350: 8 total, 8 up, 8 in 2024-09-21T10:16:53.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:53 smithi107 ceph-mon[102221]: osdmap e351: 8 total, 8 up, 8 in 2024-09-21T10:16:53.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:53 smithi107 ceph-mon[105022]: osdmap e350: 8 total, 8 up, 8 in 2024-09-21T10:16:53.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:53 smithi107 ceph-mon[105022]: osdmap e351: 8 total, 8 up, 8 in 2024-09-21T10:16:53.897 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:53 smithi129 ceph-mon[88471]: osdmap e350: 8 total, 8 up, 8 in 2024-09-21T10:16:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:53 smithi129 ceph-mon[88471]: osdmap e351: 8 total, 8 up, 8 in 2024-09-21T10:16:53.923 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:53.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:53.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:54.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:54.065 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:54.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:54.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:54.166 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:54.197 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:54.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:54.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:54.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:54.322 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:54.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:54.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:54.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:54.447 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:54.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:54.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:54.549 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:54.575 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:54.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:54.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:54.677 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:54.698 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:54.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:54.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:54.800 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:54.840 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:54.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:54.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:54 smithi107 ceph-mon[102221]: pgmap v1191: 119 pgs: 1 active+remapped+backfill_wait, 1 active+clean+remapped, 1 activating+remapped, 1 active+remapped+backfilling, 115 active+clean; 3.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 634 op/s; 12554/178545 objects misplaced (7.031%); 16 MiB/s, 257 objects/s recovering 2024-09-21T10:16:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:54 smithi107 ceph-mon[105022]: pgmap v1191: 119 pgs: 1 active+remapped+backfill_wait, 1 active+clean+remapped, 1 activating+remapped, 1 active+remapped+backfilling, 115 active+clean; 3.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 634 op/s; 12554/178545 objects misplaced (7.031%); 16 MiB/s, 257 objects/s recovering 2024-09-21T10:16:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:54 smithi129 ceph-mon[88471]: pgmap v1191: 119 pgs: 1 active+remapped+backfill_wait, 1 active+clean+remapped, 1 activating+remapped, 1 active+remapped+backfilling, 115 active+clean; 3.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 634 op/s; 12554/178545 objects misplaced (7.031%); 16 MiB/s, 257 objects/s recovering 2024-09-21T10:16:54.941 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:54.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:54.986 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:54.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:55.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:55.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:55.122 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:55.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:55.223 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:55.251 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:55.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:55.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:55.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:55.392 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:55.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:55.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:55.494 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:55.520 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:55.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:55.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:55.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:55.664 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:55.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:55.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:55.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:55.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:55.795 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:55.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:55.895 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:55.917 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:55.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:55.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:56.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:56.047 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:16:56.047 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:16:56.047 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:16:56.047 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:16:56.047 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:16:56.047 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:16:56.047 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:16:56.047 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:16:56.047 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:16:56.047 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:16:56.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:56.064 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:56.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:56.071 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:16:56.071 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:16:56.102 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:16:56.102 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:16:56.150 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:16:56.150 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:16:56.165 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:56.189 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:56.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:56.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:56.290 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:56.315 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:56.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:56.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:56.417 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:56.440 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:56.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:56.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:56.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:56.593 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:56.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:56.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:56.696 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:16:56.722 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:56.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:56.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:16:56.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:56.827 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:56 smithi107 ceph-mon[102221]: pgmap v1193: 119 pgs: 1 active+remapped+backfill_wait, 1 active+clean+remapped, 2 active+remapped+backfilling, 115 active+clean; 3.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 11010/176826 objects misplaced (6.226%); 23 MiB/s, 361 objects/s recovering 2024-09-21T10:16:56.827 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:56 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2736237693' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:56.827 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:56.827 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:56.827 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:56 smithi107 ceph-mon[105022]: pgmap v1193: 119 pgs: 1 active+remapped+backfill_wait, 1 active+clean+remapped, 2 active+remapped+backfilling, 115 active+clean; 3.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 11010/176826 objects misplaced (6.226%); 23 MiB/s, 361 objects/s recovering 2024-09-21T10:16:56.827 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:56 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2736237693' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:56.828 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:56.828 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:56.868 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:56.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:56.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:56 smithi129 ceph-mon[88471]: pgmap v1193: 119 pgs: 1 active+remapped+backfill_wait, 1 active+clean+remapped, 2 active+remapped+backfilling, 115 active+clean; 3.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 11010/176826 objects misplaced (6.226%); 23 MiB/s, 361 objects/s recovering 2024-09-21T10:16:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:56 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2736237693' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:16:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:16:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:56.970 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:56.996 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:56.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:56.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:57.097 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:57.141 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:57.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:57.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:57.243 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:57.282 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:57.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:57.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:57.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:57.427 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:57.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:57.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:57.529 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:57.559 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:57.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:57.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:57.661 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:57.709 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:57.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:57.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:57.812 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:57.842 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:57.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:57.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:57.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:57.982 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:57.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:58.083 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:58.122 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:58.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:58.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:58.224 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:58.253 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:58.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:58.355 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:58.384 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:58.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:58.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:58.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:58.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:58.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:58.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:58.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:58.679 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:58.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:58.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:16:58.816 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:58.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:58.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:16:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:16:58 smithi107 ceph-mon[102221]: pgmap v1194: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 3.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 849 op/s; 11010/169641 objects misplaced (6.490%); 17 MiB/s, 274 objects/s recovering 2024-09-21T10:16:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:16:58 smithi107 ceph-mon[105022]: pgmap v1194: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 3.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 849 op/s; 11010/169641 objects misplaced (6.490%); 17 MiB/s, 274 objects/s recovering 2024-09-21T10:16:58.918 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:16:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:16:58 smithi129 ceph-mon[88471]: pgmap v1194: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 3.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 849 op/s; 11010/169641 objects misplaced (6.490%); 17 MiB/s, 274 objects/s recovering 2024-09-21T10:16:58.945 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:58.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:58.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:16:59.046 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:59.072 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:59.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:59.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:59.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:59.238 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:59.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:59.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:16:59.340 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:16:59.367 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:59.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:59.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:16:59.469 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:16:59.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:59.516 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:59.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:16:59.601 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:16:59.617 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:16:59.642 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:16:59.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:59.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:16:59.744 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:16:59.801 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:59.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:59.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:16:59.902 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:16:59.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:16:59.977 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:16:59.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:00.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:00.129 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:00.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:00.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:00.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:00.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:00.283 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:00.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:00.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:00.422 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:00.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:00.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:00.524 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:00.549 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:00.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:00.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:00.576 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:17:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:17:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:17:00.577 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:17:00.577 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:17:00.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.577 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:171118 12.b 12:d10d88bd:::benchmark_data_smithi107_250110_object59708:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-21T10:17:00.577 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:17:00.516847+0000", 2024-09-21T10:17:00.577 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.009137549, 2024-09-21T10:17:00.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0091703670000000004, 2024-09-21T10:17:00.578 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:00.578 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:00.578 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:17:00.578 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:00.578 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:00.578 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:00.578 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 171118 2024-09-21T10:17:00.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.578 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:00.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:00.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.516847+0000", 2024-09-21T10:17:00.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:00.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.516847+0000", 2024-09-21T10:17:00.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:00.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.516849+0000", 2024-09-21T10:17:00.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7430000000000001e-06 2024-09-21T10:17:00.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:00.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.516853+0000", 2024-09-21T10:17:00.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8759999999999998e-06 2024-09-21T10:17:00.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:00.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.516863+0000", 2024-09-21T10:17:00.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0079e-05 2024-09-21T10:17:00.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:00.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.516870+0000", 2024-09-21T10:17:00.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0759999999999997e-06 2024-09-21T10:17:00.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:00.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.519391+0000", 2024-09-21T10:17:00.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025211130000000002 2024-09-21T10:17:00.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:00.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.519488+0000", 2024-09-21T10:17:00.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6985999999999996e-05 2024-09-21T10:17:00.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:17:00.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.519590+0000", 2024-09-21T10:17:00.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101427 2024-09-21T10:17:00.581 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.581 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:00.582 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.582 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:171120 12.b 12:d44bef6d:::benchmark_data_smithi107_250110_object59710:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-21T10:17:00.582 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:17:00.517285+0000", 2024-09-21T10:17:00.582 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0086997020000000001, 2024-09-21T10:17:00.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0088082079999999997, 2024-09-21T10:17:00.582 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:00.582 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:00.582 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:17:00.582 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:00.582 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:00.582 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:00.583 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 171120 2024-09-21T10:17:00.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.583 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:00.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:00.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.517285+0000", 2024-09-21T10:17:00.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:00.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.517285+0000", 2024-09-21T10:17:00.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:00.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.517286+0000", 2024-09-21T10:17:00.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5600000000000005e-07 2024-09-21T10:17:00.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:00.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.517290+0000", 2024-09-21T10:17:00.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9099999999999998e-06 2024-09-21T10:17:00.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:00.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.517300+0000", 2024-09-21T10:17:00.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0339e-05 2024-09-21T10:17:00.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:00.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.517308+0000", 2024-09-21T10:17:00.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5290000000000003e-06 2024-09-21T10:17:00.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:00.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.523202+0000", 2024-09-21T10:17:00.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0058941999999999996 2024-09-21T10:17:00.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:00.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.523299+0000", 2024-09-21T10:17:00.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7165999999999998e-05 2024-09-21T10:17:00.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:17:00.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.523400+0000", 2024-09-21T10:17:00.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000100992 2024-09-21T10:17:00.586 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.586 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:00.586 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.586 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:171121 12.b 12.6841120b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-21T10:17:00.586 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:17:00.517577+0000", 2024-09-21T10:17:00.586 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0084084520000000003, 2024-09-21T10:17:00.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085843029999999997, 2024-09-21T10:17:00.587 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:00.587 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:00.587 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2024-09-21T10:17:00.587 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:00.587 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:00.587 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:00.587 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 171121 2024-09-21T10:17:00.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.587 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:00.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:00.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.517577+0000", 2024-09-21T10:17:00.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:00.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.517577+0000", 2024-09-21T10:17:00.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:00.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.517577+0000", 2024-09-21T10:17:00.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9500000000000001e-07 2024-09-21T10:17:00.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:00.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.517581+0000", 2024-09-21T10:17:00.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.087e-06 2024-09-21T10:17:00.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:00.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.517595+0000", 2024-09-21T10:17:00.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.329e-05 2024-09-21T10:17:00.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:00.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.517601+0000", 2024-09-21T10:17:00.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4330000000000001e-06 2024-09-21T10:17:00.589 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.589 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:00.589 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.590 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:171123 12.8 12:12b205d4:::benchmark_data_smithi107_250110_object59713:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-21T10:17:00.590 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:17:00.517974+0000", 2024-09-21T10:17:00.590 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0080105130000000004, 2024-09-21T10:17:00.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0082385679999999999, 2024-09-21T10:17:00.590 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:00.590 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:00.590 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:17:00.590 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:00.590 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:00.590 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:00.590 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 171123 2024-09-21T10:17:00.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.591 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:00.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:00.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.517974+0000", 2024-09-21T10:17:00.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:00.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.517974+0000", 2024-09-21T10:17:00.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:00.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.517975+0000", 2024-09-21T10:17:00.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6899999999999996e-07 2024-09-21T10:17:00.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:00.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.517979+0000", 2024-09-21T10:17:00.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9820000000000002e-06 2024-09-21T10:17:00.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:00.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.517993+0000", 2024-09-21T10:17:00.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3888e-05 2024-09-21T10:17:00.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:00.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.518000+0000", 2024-09-21T10:17:00.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6810000000000001e-06 2024-09-21T10:17:00.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:00.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.518071+0000", 2024-09-21T10:17:00.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1009999999999999e-05 2024-09-21T10:17:00.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:00.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.518220+0000", 2024-09-21T10:17:00.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014930600000000001 2024-09-21T10:17:00.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,2", 2024-09-21T10:17:00.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.518627+0000", 2024-09-21T10:17:00.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040665299999999998 2024-09-21T10:17:00.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:00.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.524629+0000", 2024-09-21T10:17:00.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0060017719999999998 2024-09-21T10:17:00.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:00.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.524685+0000", 2024-09-21T10:17:00.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6818999999999999e-05 2024-09-21T10:17:00.595 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.595 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:00.595 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.595 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:171124 12.b 12.7e1f656b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-21T10:17:00.595 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:17:00.518018+0000", 2024-09-21T10:17:00.595 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0079674849999999998, 2024-09-21T10:17:00.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.008268081, 2024-09-21T10:17:00.595 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:00.595 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:00.595 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2024-09-21T10:17:00.596 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:00.596 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:00.596 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:00.596 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 171124 2024-09-21T10:17:00.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.596 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:00.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:00.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.518018+0000", 2024-09-21T10:17:00.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:00.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.518018+0000", 2024-09-21T10:17:00.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:00.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.518018+0000", 2024-09-21T10:17:00.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2e-07 2024-09-21T10:17:00.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:00.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.518020+0000", 2024-09-21T10:17:00.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1040000000000001e-06 2024-09-21T10:17:00.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:00.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.518026+0000", 2024-09-21T10:17:00.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9950000000000002e-06 2024-09-21T10:17:00.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:00.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.518030+0000", 2024-09-21T10:17:00.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4329999999999997e-06 2024-09-21T10:17:00.598 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.598 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:00.598 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.598 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.36822.0:171118 12.b e351/294)", 2024-09-21T10:17:00.598 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:17:00.524680+0000", 2024-09-21T10:17:00.599 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.00130456, 2024-09-21T10:17:00.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0016521260000000001, 2024-09-21T10:17:00.599 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:00.599 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:00.599 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2024-09-21T10:17:00.599 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:00.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:00.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.524680+0000", 2024-09-21T10:17:00.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:00.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.524680+0000", 2024-09-21T10:17:00.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:00.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.524681+0000", 2024-09-21T10:17:00.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1299999999999999e-07 2024-09-21T10:17:00.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:00.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.524686+0000", 2024-09-21T10:17:00.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8339999999999998e-06 2024-09-21T10:17:00.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:00.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.524699+0000", 2024-09-21T10:17:00.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.259e-05 2024-09-21T10:17:00.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:00.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.524705+0000", 2024-09-21T10:17:00.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8789999999999997e-06 2024-09-21T10:17:00.601 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.601 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:00.601 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.601 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:171130 12.11 12.34910731 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-21T10:17:00.602 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:17:00.525838+0000", 2024-09-21T10:17:00.602 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.00014690800000000001, 2024-09-21T10:17:00.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00053558299999999996, 2024-09-21T10:17:00.602 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:00.602 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:00.602 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2024-09-21T10:17:00.602 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:00.602 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:00.603 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:00.603 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 171130 2024-09-21T10:17:00.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.603 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:00.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:00.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.525838+0000", 2024-09-21T10:17:00.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:00.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.525838+0000", 2024-09-21T10:17:00.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:00.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.525839+0000", 2024-09-21T10:17:00.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.359e-06 2024-09-21T10:17:00.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:00.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.525844+0000", 2024-09-21T10:17:00.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.262e-06 2024-09-21T10:17:00.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:00.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.525859+0000", 2024-09-21T10:17:00.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5712000000000001e-05 2024-09-21T10:17:00.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:00.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.525866+0000", 2024-09-21T10:17:00.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6200000000000001e-06 2024-09-21T10:17:00.605 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.605 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:00.605 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.605 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:171131 12.12 12:4da06521:::benchmark_data_smithi107_250110_object59721:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-21T10:17:00.605 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:17:00.525883+0000", 2024-09-21T10:17:00.605 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.000101795, 2024-09-21T10:17:00.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00053938800000000002, 2024-09-21T10:17:00.606 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:00.606 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:00.606 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:17:00.606 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:00.606 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:00.606 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:00.606 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 171131 2024-09-21T10:17:00.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.606 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:00.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:00.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.525883+0000", 2024-09-21T10:17:00.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:00.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.525883+0000", 2024-09-21T10:17:00.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:00.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.525884+0000", 2024-09-21T10:17:00.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7399999999999998e-07 2024-09-21T10:17:00.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:00.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.525886+0000", 2024-09-21T10:17:00.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9810000000000002e-06 2024-09-21T10:17:00.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:00.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.525892+0000", 2024-09-21T10:17:00.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9719999999999997e-06 2024-09-21T10:17:00.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:00.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.525896+0000", 2024-09-21T10:17:00.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3719999999999998e-06 2024-09-21T10:17:00.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:00.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.525969+0000", 2024-09-21T10:17:00.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3122999999999998e-05 2024-09-21T10:17:00.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:00.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.526073+0000", 2024-09-21T10:17:00.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000104044 2024-09-21T10:17:00.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:17:00.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.526433+0000", 2024-09-21T10:17:00.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035971199999999999 2024-09-21T10:17:00.609 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.609 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:00.609 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.610 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:171104 12.b 12:d905d04a:::benchmark_data_smithi107_250110_object59694:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-21T10:17:00.610 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:17:00.500731+0000", 2024-09-21T10:17:00.610 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.025254002000000001, 2024-09-21T10:17:00.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.025758829, 2024-09-21T10:17:00.610 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:00.610 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:00.610 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:17:00.610 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:00.610 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:00.610 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:00.610 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 171104 2024-09-21T10:17:00.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.611 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:00.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:00.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.500731+0000", 2024-09-21T10:17:00.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:00.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.500731+0000", 2024-09-21T10:17:00.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:00.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.500732+0000", 2024-09-21T10:17:00.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.299e-06 2024-09-21T10:17:00.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:00.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.500737+0000", 2024-09-21T10:17:00.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5079999999999999e-06 2024-09-21T10:17:00.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:00.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.500948+0000", 2024-09-21T10:17:00.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021148799999999999 2024-09-21T10:17:00.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:00.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.500957+0000", 2024-09-21T10:17:00.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2169999999999994e-06 2024-09-21T10:17:00.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:00.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.501091+0000", 2024-09-21T10:17:00.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013458 2024-09-21T10:17:00.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:00.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.501269+0000", 2024-09-21T10:17:00.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000177837 2024-09-21T10:17:00.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:17:00.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.502115+0000", 2024-09-21T10:17:00.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00084621100000000003 2024-09-21T10:17:00.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:00.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.519330+0000", 2024-09-21T10:17:00.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017215290000000001 2024-09-21T10:17:00.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:00.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.519365+0000", 2024-09-21T10:17:00.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4554999999999997e-05 2024-09-21T10:17:00.615 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.615 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:00.615 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.615 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:171111 12.b 12:dbf6f072:::benchmark_data_smithi107_250110_object59701:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-21T10:17:00.615 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:17:00.508208+0000", 2024-09-21T10:17:00.615 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.017777122999999999, 2024-09-21T10:17:00.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018362584000000001, 2024-09-21T10:17:00.615 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:00.615 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:00.615 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:17:00.616 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:00.616 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:00.616 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:00.616 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 171111 2024-09-21T10:17:00.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.616 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:00.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:00.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.508208+0000", 2024-09-21T10:17:00.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:00.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.508208+0000", 2024-09-21T10:17:00.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:00.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.508209+0000", 2024-09-21T10:17:00.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3400000000000001e-06 2024-09-21T10:17:00.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:00.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.508213+0000", 2024-09-21T10:17:00.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4410000000000002e-06 2024-09-21T10:17:00.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:00.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.508223+0000", 2024-09-21T10:17:00.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8660000000000001e-06 2024-09-21T10:17:00.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:00.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.508229+0000", 2024-09-21T10:17:00.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5610000000000004e-06 2024-09-21T10:17:00.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:00.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.511216+0000", 2024-09-21T10:17:00.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0029865809999999999 2024-09-21T10:17:00.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:00.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.511327+0000", 2024-09-21T10:17:00.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111574 2024-09-21T10:17:00.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:17:00.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.511444+0000", 2024-09-21T10:17:00.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000116669 2024-09-21T10:17:00.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:00.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.523144+0000", 2024-09-21T10:17:00.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011699952 2024-09-21T10:17:00.619 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.620 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:00.620 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.620 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:171113 12.b 12:d9fde123:::benchmark_data_smithi107_250110_object59703:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-09-21T10:17:00.620 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:17:00.508699+0000", 2024-09-21T10:17:00.620 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.017286337999999998, 2024-09-21T10:17:00.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017942771999999999, 2024-09-21T10:17:00.620 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:00.620 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:00.620 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:17:00.620 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:00.620 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:00.621 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:00.621 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 171113 2024-09-21T10:17:00.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.621 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:00.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:00.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.508699+0000", 2024-09-21T10:17:00.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:00.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.508699+0000", 2024-09-21T10:17:00.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:00.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:00.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.508700+0000", 2024-09-21T10:17:00.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.78e-07 2024-09-21T10:17:00.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:00.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.508703+0000", 2024-09-21T10:17:00.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.788e-06 2024-09-21T10:17:00.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:00.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.508713+0000", 2024-09-21T10:17:00.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9250000000000007e-06 2024-09-21T10:17:00.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:00.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.508769+0000", 2024-09-21T10:17:00.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5294000000000003e-05 2024-09-21T10:17:00.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:00.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.514860+0000", 2024-09-21T10:17:00.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.006091674 2024-09-21T10:17:00.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:00.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.515879+0000", 2024-09-21T10:17:00.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010186660000000001 2024-09-21T10:17:00.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:17:00.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.516020+0000", 2024-09-21T10:17:00.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014067100000000001 2024-09-21T10:17:00.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:00.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:00.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:00.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:00.523176+0000", 2024-09-21T10:17:00.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071567820000000004 2024-09-21T10:17:00.624 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.624 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:00.625 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.625 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:00.625 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-09-21T10:17:00.625 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 11 2024-09-21T10:17:00.625 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:17:00.651 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:00.679 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:00.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:00.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:00.781 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:00.808 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:00.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:00.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:00.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:00 smithi107 ceph-mon[102221]: pgmap v1195: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 3.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 882 op/s; 9125/166869 objects misplaced (5.468%); 22 MiB/s, 356 objects/s recovering 2024-09-21T10:17:00.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:00 smithi107 ceph-mon[105022]: pgmap v1195: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 3.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 882 op/s; 9125/166869 objects misplaced (5.468%); 22 MiB/s, 356 objects/s recovering 2024-09-21T10:17:00.910 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:00.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:00 smithi129 ceph-mon[88471]: pgmap v1195: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 3.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 882 op/s; 9125/166869 objects misplaced (5.468%); 22 MiB/s, 356 objects/s recovering 2024-09-21T10:17:00.937 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:00.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:00.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:01.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:01.077 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:01.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:01.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:01.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:01.213 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:01.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:01.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:01.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:01.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:01.358 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:01.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:01.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:01.496 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:01.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:01.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:01.597 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T10:17:01.598 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:01.623 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:01.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:01.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:01.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:01.748 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:01.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:01.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:01.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:01.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:01 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/193827548' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:17:01.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:01 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:17:01.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:01 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/193827548' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:17:01.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:01 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:17:01.860 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:17:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:17:01] "GET /metrics HTTP/1.1" 200 36826 "" "Prometheus/2.51.0" 2024-09-21T10:17:01.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:01.885 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:01.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:01.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:01 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/193827548' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:17:01.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:01 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:17:01.986 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:02.013 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:02.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:02.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:02.115 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:02.143 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:02.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:02.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:02.245 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:02.272 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:02.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:02.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:02.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:02.411 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:02.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:02.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:02.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:02.554 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:02.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:02.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:02.655 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:02.682 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:02.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:02.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:02.784 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:02.812 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:02.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:02.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:02 smithi107 ceph-mon[102221]: pgmap v1196: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 3.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 557 op/s; 7981/163725 objects misplaced (4.875%); 13 MiB/s, 207 objects/s recovering 2024-09-21T10:17:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:02 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:17:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:02 smithi107 ceph-mon[102221]: osdmap e352: 8 total, 8 up, 8 in 2024-09-21T10:17:02.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:02 smithi107 ceph-mon[105022]: pgmap v1196: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 3.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 557 op/s; 7981/163725 objects misplaced (4.875%); 13 MiB/s, 207 objects/s recovering 2024-09-21T10:17:02.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:02 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:17:02.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:02 smithi107 ceph-mon[105022]: osdmap e352: 8 total, 8 up, 8 in 2024-09-21T10:17:02.913 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:02.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:02 smithi129 ceph-mon[88471]: pgmap v1196: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 3.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 557 op/s; 7981/163725 objects misplaced (4.875%); 13 MiB/s, 207 objects/s recovering 2024-09-21T10:17:02.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:02 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:17:02.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:02 smithi129 ceph-mon[88471]: osdmap e352: 8 total, 8 up, 8 in 2024-09-21T10:17:02.952 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:02.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:02.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:03.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:03.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:03.093 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:03.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:03.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:03.241 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:03.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:03.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:03.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:03.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:03.382 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:03.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:03.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:03.523 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:03.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:03.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:03.625 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:03.650 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:03.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:03.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:03.753 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:03.780 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:03.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:03.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:03.881 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:03.906 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:03.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:03.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:04.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:04.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:04.037 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:04.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:04.138 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:04.164 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:04.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:04.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:04.266 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:04.294 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:04.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:04.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:04.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:04.432 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:04.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:04.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:04.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:04.572 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:04.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:04.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:04.674 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:04.699 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:04.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:04.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:04.800 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:04.828 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:04.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:04.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:04 smithi107 ceph-mon[102221]: pgmap v1198: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 3.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 902 op/s; 6606/151833 objects misplaced (4.351%); 14 MiB/s, 224 objects/s recovering 2024-09-21T10:17:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:04 smithi107 ceph-mon[105022]: pgmap v1198: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 3.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 902 op/s; 6606/151833 objects misplaced (4.351%); 14 MiB/s, 224 objects/s recovering 2024-09-21T10:17:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:04 smithi129 ceph-mon[88471]: pgmap v1198: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 3.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 902 op/s; 6606/151833 objects misplaced (4.351%); 14 MiB/s, 224 objects/s recovering 2024-09-21T10:17:04.929 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:04.957 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:04.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:04.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:05.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:05.099 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:05.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:05.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:05.202 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:05.227 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:05.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:05.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:05.329 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:05.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:05.355 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:05.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:05.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:17:05.456 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:05.497 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:05.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:05.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:05.599 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:05.625 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:05.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:05.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:05.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:05.764 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T10:17:05.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:05.789 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:05.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:05.890 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:05.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:05.920 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:05.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:06.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:06.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:06.077 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:06.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:06.178 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:06.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:06.230 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:06.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:06.331 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:06.357 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:06.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:06.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:06.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:17:06.449 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:06.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:06.537 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:06.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:06.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:06.638 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:06.662 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:06.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:06.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:06.758 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:06.763 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:06.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:06.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:06.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:06 smithi107 ceph-mon[102221]: pgmap v1199: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 3.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 892 op/s; 5708/150018 objects misplaced (3.805%); 12 MiB/s, 198 objects/s recovering 2024-09-21T10:17:06.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:06 smithi107 ceph-mon[105022]: pgmap v1199: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 3.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 892 op/s; 5708/150018 objects misplaced (3.805%); 12 MiB/s, 198 objects/s recovering 2024-09-21T10:17:06.910 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:06 smithi129 ceph-mon[88471]: pgmap v1199: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 3.1 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 892 op/s; 5708/150018 objects misplaced (3.805%); 12 MiB/s, 198 objects/s recovering 2024-09-21T10:17:06.947 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:06.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:06.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:07.049 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:07.073 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:07.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:07.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:07.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:07.200 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:07.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:07.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:07.303 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:07.311 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:07.325 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:07.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:07.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:07.426 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:07.448 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:07.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:07.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:07.550 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:07.573 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:07.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:07.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:07.674 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:07.703 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:07.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:07.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:07.740 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:07 smithi107 ceph-mon[102221]: osdmap e353: 8 total, 8 up, 8 in 2024-09-21T10:17:07.740 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:07 smithi107 ceph-mon[105022]: osdmap e353: 8 total, 8 up, 8 in 2024-09-21T10:17:07.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:07.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:07.855 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:07.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:07 smithi129 ceph-mon[88471]: osdmap e353: 8 total, 8 up, 8 in 2024-09-21T10:17:07.956 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:07.981 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:07.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:07.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:08.083 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:08.107 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:08.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:08.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:08.209 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:08.232 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:08.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:08.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:08.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:08.358 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:08.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:08.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:08.460 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:08.483 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:08.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:08.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:08.585 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:08.607 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:08.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:08.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:08.709 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:08.736 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:08.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:08.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:08.838 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:08.875 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:08.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:08.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:08 smithi129 ceph-mon[88471]: pgmap v1201: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 2.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 5708/141072 objects misplaced (4.046%); 10 MiB/s, 163 objects/s recovering 2024-09-21T10:17:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:08 smithi129 ceph-mon[88471]: osdmap e354: 8 total, 8 up, 8 in 2024-09-21T10:17:08.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:09.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:09.013 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:09.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:08 smithi107 ceph-mon[102221]: pgmap v1201: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 2.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 5708/141072 objects misplaced (4.046%); 10 MiB/s, 163 objects/s recovering 2024-09-21T10:17:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:08 smithi107 ceph-mon[102221]: osdmap e354: 8 total, 8 up, 8 in 2024-09-21T10:17:09.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:08 smithi107 ceph-mon[105022]: pgmap v1201: 119 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 116 active+clean; 2.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s; 5708/141072 objects misplaced (4.046%); 10 MiB/s, 163 objects/s recovering 2024-09-21T10:17:09.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:08 smithi107 ceph-mon[105022]: osdmap e354: 8 total, 8 up, 8 in 2024-09-21T10:17:09.114 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:09.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:09.145 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:09.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:09.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:09.276 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:09.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:09.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:09.377 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:09.414 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:09.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:09.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:09.516 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:09.538 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:09.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:09.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:09.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:09.679 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:09.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:09.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:09.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:09.821 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:09.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:09.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:09.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:09 smithi107 ceph-mon[102221]: osdmap e355: 8 total, 8 up, 8 in 2024-09-21T10:17:09.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:09 smithi107 ceph-mon[105022]: osdmap e355: 8 total, 8 up, 8 in 2024-09-21T10:17:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:09 smithi129 ceph-mon[88471]: osdmap e355: 8 total, 8 up, 8 in 2024-09-21T10:17:09.922 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:09.949 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:09.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:09.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:10.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:10.091 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:10.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:10.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:10.193 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:10.224 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:10.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:10.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:10.361 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:10.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:10.463 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:10.507 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:10.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:10.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:10.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:10.638 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:17:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:17:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:17:10.647 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:10.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:10.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:10.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:10.784 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:10.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:10.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:10.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:10.907 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:10.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:10.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:10 smithi129 ceph-mon[88471]: pgmap v1204: 119 pgs: 1 peering, 1 active+clean+remapped, 1 active+remapped+backfilling, 116 active+clean; 2.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 3924/137655 objects misplaced (2.851%); 11 MiB/s, 172 objects/s recovering 2024-09-21T10:17:11.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:11.050 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:11.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:11.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:11.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:10 smithi107 ceph-mon[102221]: pgmap v1204: 119 pgs: 1 peering, 1 active+clean+remapped, 1 active+remapped+backfilling, 116 active+clean; 2.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 3924/137655 objects misplaced (2.851%); 11 MiB/s, 172 objects/s recovering 2024-09-21T10:17:11.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:10 smithi107 ceph-mon[105022]: pgmap v1204: 119 pgs: 1 peering, 1 active+clean+remapped, 1 active+remapped+backfilling, 116 active+clean; 2.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 787 op/s; 3924/137655 objects misplaced (2.851%); 11 MiB/s, 172 objects/s recovering 2024-09-21T10:17:11.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:11.196 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:11.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:11.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:11.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:11.338 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:11.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:11.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:11.440 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:11.473 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:11.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:11.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:11.575 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:11.613 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:11.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:11.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:11.714 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:11.743 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:11.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:11.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:11.823 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:11.845 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:11.892 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:11.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:11.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:11.966 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:17:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:17:11] "GET /metrics HTTP/1.1" 200 36823 "" "Prometheus/2.51.0" 2024-09-21T10:17:11.967 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:11.967 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:11.993 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:12.024 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:12.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:12.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:12.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:12.164 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:12.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:12.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:12.265 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:12.294 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:12.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:12.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:12.396 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:12.423 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:12.423 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:12.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:12.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:12.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:12.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:12.589 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:12.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:12.690 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:12 smithi107 ceph-mon[102221]: pgmap v1205: 119 pgs: 1 peering, 1 active+clean+remapped, 1 active+remapped+backfilling, 116 active+clean; 2.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 3924/135555 objects misplaced (2.895%); 6.4 MiB/s, 103 objects/s recovering 2024-09-21T10:17:12.690 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:12 smithi107 ceph-mon[105022]: pgmap v1205: 119 pgs: 1 peering, 1 active+clean+remapped, 1 active+remapped+backfilling, 116 active+clean; 2.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 3924/135555 objects misplaced (2.895%); 6.4 MiB/s, 103 objects/s recovering 2024-09-21T10:17:12.690 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:12.721 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:12.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:12.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:12.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:12.866 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:12.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:12.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:12.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:12 smithi129 ceph-mon[88471]: pgmap v1205: 119 pgs: 1 peering, 1 active+clean+remapped, 1 active+remapped+backfilling, 116 active+clean; 2.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 3924/135555 objects misplaced (2.895%); 6.4 MiB/s, 103 objects/s recovering 2024-09-21T10:17:12.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:13.003 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:13.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:13.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:13.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:13.152 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:13.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:13.254 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:13.265 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:17:13.266 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:17:13.284 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:13.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:13.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:13.306 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:17:13.306 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:17:13.353 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:17:13.354 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:17:13.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:13.396 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:17:13.396 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:17:13.424 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:13.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:13.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:13.428 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:17:13.428 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:17:13.457 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:17:13.458 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:17:13.495 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:17:13.496 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:17:13.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:13.532 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:17:13.532 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:17:13.566 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:13.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:13.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:13.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:13.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:13.706 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:13.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:13.806 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:13.833 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:13.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:13.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:13.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:13 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1520441903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:13.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:13 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1520441903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:13 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1520441903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:13.935 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:13.960 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:13.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:13.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:14.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:14.111 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:14.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:14.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:14.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:14.254 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:14.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:14.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:14.355 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:14.383 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:14.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:14.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:14.485 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:14.509 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:14.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:14.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:14.611 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:14.638 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:14.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:14.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:14.681 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:17:14.740 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:14.761 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:14 smithi107 ceph-mon[105022]: pgmap v1206: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 2.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s; 1446/125433 objects misplaced (1.153%); 13 MiB/s, 200 objects/s recovering 2024-09-21T10:17:14.761 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:14 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2509642972' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:17:14.761 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:14 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:17:14.761 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:14 smithi107 ceph-mon[102221]: pgmap v1206: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 2.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s; 1446/125433 objects misplaced (1.153%); 13 MiB/s, 200 objects/s recovering 2024-09-21T10:17:14.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:14 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2509642972' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:17:14.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:14 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:17:14.764 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:14.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:14.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:14.866 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:14.892 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:14.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:14.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:14 smithi129 ceph-mon[88471]: pgmap v1206: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 2.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s; 1446/125433 objects misplaced (1.153%); 13 MiB/s, 200 objects/s recovering 2024-09-21T10:17:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:14 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2509642972' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:17:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:14 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:17:14.995 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:15.025 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:15.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:15.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:15.127 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:15.155 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:15.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:15.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:15.257 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:15.291 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:15.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:15.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:15.443 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:15.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:15.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:15.544 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:15.572 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:15.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:15.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:15.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:15.715 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:15.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:15.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:15.816 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:15.843 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:15.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:15.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:15.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:15 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:17:15.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:15 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:17:15.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:15 smithi107 ceph-mon[102221]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T10:17:15.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:15 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:17:15.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:15 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:17:15.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:15 smithi107 ceph-mon[105022]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T10:17:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:15 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:17:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:15 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:17:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:15 smithi129 ceph-mon[88471]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T10:17:15.946 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:15.970 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:15.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:15.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:16.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:16.112 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:16.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:16.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:16.214 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:16.241 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:16.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:16.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:16.343 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:16.373 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:16.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:16.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:16.474 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:16.515 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:16.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:16.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:16.544 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:17:16.617 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:16.641 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:16.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:16.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:16.742 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:16.767 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:16.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:16.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:16.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:16 smithi107 ceph-mon[102221]: pgmap v1208: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 2.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 587 op/s; 1446/123930 objects misplaced (1.167%); 6.0 MiB/s, 95 objects/s recovering 2024-09-21T10:17:16.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:16 smithi107 ceph-mon[105022]: pgmap v1208: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 2.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 587 op/s; 1446/123930 objects misplaced (1.167%); 6.0 MiB/s, 95 objects/s recovering 2024-09-21T10:17:16.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:16.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:16.915 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:16.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:17.016 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:17.038 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:17.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:17.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:17.140 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:16 smithi129 ceph-mon[88471]: pgmap v1208: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 2.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 587 op/s; 1446/123930 objects misplaced (1.167%); 6.0 MiB/s, 95 objects/s recovering 2024-09-21T10:17:17.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:17.172 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:17.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:17.273 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:17.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:17.299 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:17.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:17.400 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:17.428 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:17.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:17.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:17.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:17.573 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:17.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:17.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:17.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:17.707 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:17.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:17.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:17.809 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:17.836 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:17.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:17.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:17.938 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:17.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:17.964 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:17.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:18.065 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:18.087 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:18.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:18.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:18.188 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:18.209 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:18.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:18.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:18.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:18.333 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:18.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:18.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:18.434 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:18.456 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:18.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:18.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:18.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:18.580 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:18.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:18.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:18.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:18.707 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:18.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:18.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:18.809 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:18.837 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:18.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:18.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:18.939 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:18.966 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:18.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:18.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:19.067 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:19.090 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:19.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:19.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:19.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:18 smithi107 ceph-mon[102221]: pgmap v1209: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 2.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 921 op/s; 1446/113361 objects misplaced (1.276%); 5.3 MiB/s, 84 objects/s recovering 2024-09-21T10:17:19.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:18 smithi107 ceph-mon[102221]: osdmap e357: 8 total, 8 up, 8 in 2024-09-21T10:17:19.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:18 smithi107 ceph-mon[105022]: pgmap v1209: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 2.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 921 op/s; 1446/113361 objects misplaced (1.276%); 5.3 MiB/s, 84 objects/s recovering 2024-09-21T10:17:19.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:18 smithi107 ceph-mon[105022]: osdmap e357: 8 total, 8 up, 8 in 2024-09-21T10:17:19.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:18 smithi129 ceph-mon[88471]: pgmap v1209: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 2.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 921 op/s; 1446/113361 objects misplaced (1.276%); 5.3 MiB/s, 84 objects/s recovering 2024-09-21T10:17:19.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:18 smithi129 ceph-mon[88471]: osdmap e357: 8 total, 8 up, 8 in 2024-09-21T10:17:19.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:19.228 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:19.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:19.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:19.330 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:19.369 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:19.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:19.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:19.471 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:19.499 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:19.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:19.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:19.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:19.658 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:19.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:19.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:19.760 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:19.788 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:19.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:19.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:19.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:19 smithi107 ceph-mon[102221]: osdmap e358: 8 total, 8 up, 8 in 2024-09-21T10:17:19.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:19 smithi107 ceph-mon[105022]: osdmap e358: 8 total, 8 up, 8 in 2024-09-21T10:17:19.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:19.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:19.934 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:19.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:20.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:20.071 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:20.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:20.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:20.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:19 smithi129 ceph-mon[88471]: osdmap e358: 8 total, 8 up, 8 in 2024-09-21T10:17:20.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:20.216 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:20.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:20.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:20.318 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:20.347 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:20.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:20.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:20.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:20.485 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:20.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:20.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:20.587 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:20.627 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:20.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:20.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:17:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:17:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:17:20.729 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:20.777 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:20.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:20.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:20.879 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:20.903 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:20.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:20.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:20.937 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:20 smithi107 ceph-mon[102221]: pgmap v1212: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 2.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 15/109215 objects misplaced (0.014%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:17:20.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:20 smithi107 ceph-mon[105022]: pgmap v1212: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 2.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 15/109215 objects misplaced (0.014%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:17:20.989 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:17:20.989 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:17:21.004 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:21.030 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:21.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:21.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:21.133 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:21.158 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:21.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:21.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:21.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:20 smithi129 ceph-mon[88471]: pgmap v1212: 119 pgs: 1 active+remapped+backfilling, 118 active+clean; 2.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 900 op/s; 15/109215 objects misplaced (0.014%); 4.5 MiB/s, 72 objects/s recovering 2024-09-21T10:17:21.260 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:21.287 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:21.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:21.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:21.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:21.427 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:21.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:21.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:21.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:21.565 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:21.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:21.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:21.693 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:21.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:21.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:21.795 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:21.822 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:21.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:21.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:21.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:21.972 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:21.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:21.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:22.004 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:21 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3150108284' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:17:22.004 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:21 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:17:22.004 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:21 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3150108284' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:17:22.004 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:21 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:17:22.005 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:17:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:17:21] "GET /metrics HTTP/1.1" 200 36823 "" "Prometheus/2.51.0" 2024-09-21T10:17:22.018 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T10:17:22.074 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:22.102 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:22.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:22.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:22.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:21 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3150108284' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:17:22.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:21 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:17:22.204 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:22.232 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:22.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:22.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:22.334 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:22.364 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:22.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:22.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:22.466 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:22.492 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:22.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:22.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:22.594 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:22.624 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:22.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:22.714 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:17:22.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:22.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:22.808 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:22.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:22.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:22 smithi129 ceph-mon[88471]: pgmap v1213: 119 pgs: 119 active+clean; 2.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 955 op/s; 4.1 MiB/s, 65 objects/s recovering 2024-09-21T10:17:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:22 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:17:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:22 smithi129 ceph-mon[88471]: osdmap e359: 8 total, 8 up, 8 in 2024-09-21T10:17:22.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:22.947 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:22.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:23.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:23.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:22 smithi107 ceph-mon[102221]: pgmap v1213: 119 pgs: 119 active+clean; 2.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 955 op/s; 4.1 MiB/s, 65 objects/s recovering 2024-09-21T10:17:23.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:22 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:17:23.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:22 smithi107 ceph-mon[102221]: osdmap e359: 8 total, 8 up, 8 in 2024-09-21T10:17:23.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:22 smithi107 ceph-mon[105022]: pgmap v1213: 119 pgs: 119 active+clean; 2.2 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 955 op/s; 4.1 MiB/s, 65 objects/s recovering 2024-09-21T10:17:23.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:22 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:17:23.092 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:22 smithi107 ceph-mon[105022]: osdmap e359: 8 total, 8 up, 8 in 2024-09-21T10:17:23.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:23.097 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:23.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:23.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:23.237 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:23.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:23.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:23.338 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:23.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:23.377 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:23.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:23.478 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:23.503 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:23.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:23.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:23.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T10:17:23.605 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:23.629 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:23.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:23.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:23.730 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:23.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:23.771 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:23.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:23.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:23.911 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:23.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:23.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:24.017 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:24.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:17:24.040 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:24.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:24.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:24.045 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:24.142 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:24.164 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:24.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:24.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:24.266 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:24.290 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:24.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:24.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:24.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:24.472 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:24.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:24.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:24.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:24.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:24.612 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:24.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:24.713 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:24.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:24.741 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:24.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:24.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:24.879 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:24.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:24.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:24.981 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:25.007 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:25.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:25.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:25.068 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:24 smithi107 ceph-mon[102221]: pgmap v1215: 119 pgs: 119 active+clean; 1.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 4.6 MiB/s, 73 objects/s recovering 2024-09-21T10:17:25.068 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:24 smithi107 ceph-mon[105022]: pgmap v1215: 119 pgs: 119 active+clean; 1.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 4.6 MiB/s, 73 objects/s recovering 2024-09-21T10:17:25.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:25.149 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:25.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:25.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:24 smithi129 ceph-mon[88471]: pgmap v1215: 119 pgs: 119 active+clean; 1.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 4.6 MiB/s, 73 objects/s recovering 2024-09-21T10:17:25.251 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:25.293 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:25.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:25.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:25.395 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:25.423 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:25.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:25.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:25.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:25.605 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:25.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:25.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:25.707 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:25.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:25.734 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:25.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:25.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:25.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:25.902 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:25.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:26.003 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:26.029 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:26.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:26.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:26.133 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:26.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:26.181 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:26.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:26.281 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:26.322 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:26.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:26.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:26.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:26.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:26.454 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:26.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:26.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:26.597 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:26.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:26.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:26.665 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:17:26.666 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:17:26.699 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:26.708 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:17:26.709 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:17:26.737 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:26.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:26.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:26.762 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:17:26.763 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:17:26.794 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:17:26.794 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:17:26.824 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:17:26.824 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:17:26.839 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:26.858 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:17:26.858 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:17:26.878 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:26.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:26.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:26.886 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:17:26.886 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:17:26.918 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:17:26.919 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:17:26.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:26 smithi107 ceph-mon[102221]: pgmap v1216: 119 pgs: 119 active+clean; 1.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 41 KiB/s, 0 objects/s recovering 2024-09-21T10:17:26.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:26.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:26 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/134196854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:26.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:17:26.920 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:26 smithi107 ceph-mon[105022]: pgmap v1216: 119 pgs: 119 active+clean; 1.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 41 KiB/s, 0 objects/s recovering 2024-09-21T10:17:26.920 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:26.920 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:26 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/134196854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:26.920 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:17:26.980 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:27.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:27.008 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:27.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:27.109 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:27.140 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:27.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:27.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:26 smithi129 ceph-mon[88471]: pgmap v1216: 119 pgs: 119 active+clean; 1.9 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 691 op/s; 41 KiB/s, 0 objects/s recovering 2024-09-21T10:17:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:26 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/134196854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:17:27.242 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:27.281 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:27.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:27.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:27.383 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:27.408 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:27.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:27.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:27.510 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:27.548 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:27.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:27.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:27.650 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:27.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:27.676 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:27.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:27.777 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:27.811 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:27.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:27.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:27.913 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:27.946 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:27.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:27.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:28.048 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:28.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:28.077 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:28.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:28.178 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:28.206 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:28.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:28.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:28.308 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:28.339 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:28.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:28.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:28.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:28.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:28.476 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:28.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:28.577 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:28.606 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:28.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:28.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:28.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:28.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:28.747 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:28.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:28.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:28.885 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:28.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:28.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:28.986 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:29.013 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:29.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:29.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:29.070 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:28 smithi107 ceph-mon[102221]: pgmap v1217: 119 pgs: 119 active+clean; 1.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 37 KiB/s, 0 objects/s recovering 2024-09-21T10:17:29.070 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:28 smithi107 ceph-mon[105022]: pgmap v1217: 119 pgs: 119 active+clean; 1.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 37 KiB/s, 0 objects/s recovering 2024-09-21T10:17:29.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:29.162 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:29.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:29.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:29.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:28 smithi129 ceph-mon[88471]: pgmap v1217: 119 pgs: 119 active+clean; 1.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.03k op/s; 37 KiB/s, 0 objects/s recovering 2024-09-21T10:17:29.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:29.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:29.312 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:29.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:29.413 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:29.423 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:29.423 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:29.493 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:29.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:29.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:29.595 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:29.623 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:29.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:29.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:29.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:29.758 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:29.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:29.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:29.860 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:29.888 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:29.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:29.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:29.990 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:30.016 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:30.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:30.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:30.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:30.159 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:30.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:30.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:30.260 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:30.289 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:30.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:30.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:30.390 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:30.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:30 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:17:30.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:30 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:17:30.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:30 smithi129 ceph-mon[88471]: pgmap v1218: 119 pgs: 119 active+clean; 1.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 32 KiB/s, 0 objects/s recovering 2024-09-21T10:17:30.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:17:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:17:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:17:30.420 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:30.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:30.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:30.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:30 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:17:30.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:30 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:17:30.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:30 smithi107 ceph-mon[102221]: pgmap v1218: 119 pgs: 119 active+clean; 1.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 32 KiB/s, 0 objects/s recovering 2024-09-21T10:17:30.452 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:30 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:17:30.452 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:30 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:17:30.453 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:30 smithi107 ceph-mon[105022]: pgmap v1218: 119 pgs: 119 active+clean; 1.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 32 KiB/s, 0 objects/s recovering 2024-09-21T10:17:30.522 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:30.548 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:30.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:30.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:30.650 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:30.675 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:30.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:30.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:30.777 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:30.801 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:30.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:30.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:30.893 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:30.902 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:30.927 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:30.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:30.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:31.029 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:31.053 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:31.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:31.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:31.155 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:31.179 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:31.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:31.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:31.280 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:31.305 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:31.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:31.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:31.407 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:31.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:31 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:17:31.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:31 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:17:31.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:31 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:17:31.438 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:31.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:31.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:31.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:31 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:17:31.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:31 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:17:31.531 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:31 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:17:31.532 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:31 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:17:31.532 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:31 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:17:31.532 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:31 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:17:31.540 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:31.584 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:31.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:31.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:31.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:31.723 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:31.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:31.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:31.824 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:31.851 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:31.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:31.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:31.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:17:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:17:31] "GET /metrics HTTP/1.1" 200 36809 "" "Prometheus/2.51.0" 2024-09-21T10:17:31.952 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:31.988 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:31.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:32.090 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:32.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:32.125 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:32.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:32.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:32.261 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:32.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:32.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:32.274 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:32 smithi107 ceph-mon[102221]: pgmap v1219: 119 pgs: 119 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-21T10:17:32.274 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:32 smithi107 ceph-mon[105022]: pgmap v1219: 119 pgs: 119 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-21T10:17:32.363 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:32.404 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:32.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:32.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:32.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:32 smithi129 ceph-mon[88471]: pgmap v1219: 119 pgs: 119 active+clean; 1.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-21T10:17:32.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:17:32.506 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:32.535 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:32.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:32.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:32.636 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:32.661 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:32.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:32.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:32.763 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:32.811 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:32.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:32.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:32.913 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:32.940 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:32.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:32.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:33.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:33.082 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:33.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:33.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:33.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:33.229 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:33.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:33.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:33.330 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:33.362 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:33.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:33.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:33.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:33.510 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:33.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:33.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:33.611 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:33.640 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:33.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:33.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:33.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:33.778 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:33.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:33.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:33.880 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:33.905 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:33.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:33.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:34.007 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:34.033 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:34.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:34.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:34.135 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:34.159 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:34.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:34.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:34.260 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:34.286 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:34.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:34.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:34.388 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:34.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:34.428 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:34.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:34.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:34.576 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:34.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:34.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:34.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:34.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:34.722 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:34.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:34.756 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:34 smithi107 ceph-mon[102221]: pgmap v1220: 119 pgs: 119 active+clean; 1.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-21T10:17:34.756 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:34 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2343838679' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:17:34.757 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:34 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:17:34.757 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:34 smithi107 ceph-mon[105022]: pgmap v1220: 119 pgs: 119 active+clean; 1.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-21T10:17:34.757 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:34 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2343838679' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:17:34.757 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:34 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:17:34.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:34.858 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:34.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:34.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:34 smithi129 ceph-mon[88471]: pgmap v1220: 119 pgs: 119 active+clean; 1.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.23k op/s 2024-09-21T10:17:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:34 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2343838679' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:17:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:34 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:17:34.960 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:34.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:34.993 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:34.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:35.094 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:35.124 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:35.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:35.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:35.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:35.260 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:35.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:35.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:35.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:35.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:35.402 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:35.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:35.440 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T10:17:35.503 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:35.527 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:35.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:35.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:35.628 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:35.656 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:35.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:35.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:35.758 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:35.784 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:35.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:35.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:35.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:35 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:17:35.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:35 smithi107 ceph-mon[102221]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T10:17:35.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:35 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:17:35.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:35 smithi107 ceph-mon[105022]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T10:17:35.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:35.918 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:35.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:35.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:35.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:35 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:17:35.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:35 smithi129 ceph-mon[88471]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T10:17:36.020 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:36.045 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:36.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:36.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:36.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:36.187 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:36.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:36.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:36.288 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:36.329 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:36.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:36.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:36.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:36.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:36.472 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:36.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:36.573 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:36.598 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:36.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:36.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:36.700 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:36.727 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:36.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:36.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:36.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:36 smithi107 ceph-mon[102221]: pgmap v1222: 119 pgs: 119 active+clean; 1.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T10:17:36.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:36 smithi107 ceph-mon[105022]: pgmap v1222: 119 pgs: 119 active+clean; 1.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T10:17:36.873 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:36.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:36.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:36 smithi129 ceph-mon[88471]: pgmap v1222: 119 pgs: 119 active+clean; 1.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-21T10:17:36.975 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:37.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:37.007 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:37.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:37.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:37.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:37.153 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:37.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:37.254 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:37.279 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:37.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:37.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:37.380 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:37.412 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:37.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:37.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:37.552 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:37.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:37.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:37.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:37.691 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:37.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:37.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:37.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:37.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:37.836 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:37.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:37.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:37.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:37.977 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:37.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:38.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:38.115 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:38.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:38.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:38.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:38.257 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:38.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:38.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:38.359 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:38.388 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:38.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:38.470 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T10:17:38.490 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:38.518 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:38.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:38.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:38.619 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:38.644 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:38.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:38.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:38.716 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:38 smithi107 ceph-mon[102221]: pgmap v1223: 119 pgs: 119 active+clean; 1.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-21T10:17:38.716 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:38 smithi107 ceph-mon[105022]: pgmap v1223: 119 pgs: 119 active+clean; 1.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-21T10:17:38.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:38.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:38.802 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:38.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:38.903 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:38 smithi129 ceph-mon[88471]: pgmap v1223: 119 pgs: 119 active+clean; 1.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.07k op/s 2024-09-21T10:17:38.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:38.949 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:38.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:39.022 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:39.050 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:39.068 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:17:39.068 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-09-21T10:17:39.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-09-21T10:17:39.069 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:17:39.069 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.069 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91069 11.8 11:1d9ce1cd:::benchmark_data_smithi107_218812_object91068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-21T10:17:39.069 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.369573+0000", 2024-09-21T10:17:39.069 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 402.691810675, 2024-09-21T10:17:39.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3066178669999999, 2024-09-21T10:17:39.069 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:39.069 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:39.070 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:17:39.070 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:39.070 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:17:39.070 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:17:39.070 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91069 2024-09-21T10:17:39.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.070 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:39.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:39.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.369573+0000", 2024-09-21T10:17:39.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:39.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.369573+0000", 2024-09-21T10:17:39.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:39.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.369575+0000", 2024-09-21T10:17:39.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.612e-06 2024-09-21T10:17:39.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:39.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.369624+0000", 2024-09-21T10:17:39.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9036999999999998e-05 2024-09-21T10:17:39.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:39.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.369645+0000", 2024-09-21T10:17:39.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0599999999999999e-05 2024-09-21T10:17:39.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.073 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:39.073 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.369653+0000", 2024-09-21T10:17:39.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1890000000000007e-06 2024-09-21T10:17:39.073 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.073 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.073 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.376149+0000", 2024-09-21T10:17:39.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0064962070000000004 2024-09-21T10:17:39.073 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.073 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:39.073 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.376301+0000", 2024-09-21T10:17:39.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000151757 2024-09-21T10:17:39.073 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.074 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.074 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:17:39.074 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.376460+0000", 2024-09-21T10:17:39.074 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000159586 2024-09-21T10:17:39.074 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.074 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.074 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:39.074 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379481+0000", 2024-09-21T10:17:39.074 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0030210580000000001 2024-09-21T10:17:39.074 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.074 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.074 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.386415+0000", 2024-09-21T10:17:39.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069336040000000003 2024-09-21T10:17:39.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.403651+0000", 2024-09-21T10:17:39.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017236089 2024-09-21T10:17:39.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.403662+0000", 2024-09-21T10:17:39.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0421999999999999e-05 2024-09-21T10:17:39.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384390+0000", 2024-09-21T10:17:39.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.98072825299999999 2024-09-21T10:17:39.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384403+0000", 2024-09-21T10:17:39.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3186e-05 2024-09-21T10:17:39.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.676117+0000", 2024-09-21T10:17:39.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.291714217 2024-09-21T10:17:39.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.077 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:17:39.077 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.676191+0000", 2024-09-21T10:17:39.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4049999999999997e-05 2024-09-21T10:17:39.077 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.077 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.077 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.077 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91074 11.a 11:5a4cfa77:::benchmark_data_smithi107_218812_object91073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-21T10:17:39.077 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.379421+0000", 2024-09-21T10:17:39.077 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 402.68196356300001, 2024-09-21T10:17:39.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3240689140000002, 2024-09-21T10:17:39.077 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:39.078 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:39.078 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:17:39.078 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:39.078 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:17:39.078 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:17:39.078 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91074 2024-09-21T10:17:39.078 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.078 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:39.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.078 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:39.078 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379421+0000", 2024-09-21T10:17:39.078 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.078 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:39.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379421+0000", 2024-09-21T10:17:39.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:39.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379422+0000", 2024-09-21T10:17:39.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.513e-06 2024-09-21T10:17:39.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:39.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379443+0000", 2024-09-21T10:17:39.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1273000000000001e-05 2024-09-21T10:17:39.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:39.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379455+0000", 2024-09-21T10:17:39.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.203e-05 2024-09-21T10:17:39.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:39.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379463+0000", 2024-09-21T10:17:39.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3329999999999999e-06 2024-09-21T10:17:39.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.386937+0000", 2024-09-21T10:17:39.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0074744290000000003 2024-09-21T10:17:39.081 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.081 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.081 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:39.081 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.387075+0000", 2024-09-21T10:17:39.081 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013819900000000001 2024-09-21T10:17:39.081 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.081 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.081 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:17:39.081 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.387218+0000", 2024-09-21T10:17:39.081 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000142705 2024-09-21T10:17:39.081 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.081 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.081 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:39.081 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.393122+0000", 2024-09-21T10:17:39.082 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005903747 2024-09-21T10:17:39.082 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.082 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.082 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.082 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.412062+0000", 2024-09-21T10:17:39.082 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018940354 2024-09-21T10:17:39.082 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.082 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.412074+0000", 2024-09-21T10:17:39.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2333e-05 2024-09-21T10:17:39.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385002+0000", 2024-09-21T10:17:39.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.97292794599999999 2024-09-21T10:17:39.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385014+0000", 2024-09-21T10:17:39.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.182e-05 2024-09-21T10:17:39.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.084 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.084 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.084 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.687683+0000", 2024-09-21T10:17:39.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.302668766 2024-09-21T10:17:39.084 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.084 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.084 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for degraded object", 2024-09-21T10:17:39.084 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.688842+0000", 2024-09-21T10:17:39.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011593440000000001 2024-09-21T10:17:39.084 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.084 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.084 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.084 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.703390+0000", 2024-09-21T10:17:39.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014547396000000001 2024-09-21T10:17:39.085 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.085 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.085 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:17:39.085 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.703489+0000", 2024-09-21T10:17:39.085 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9726e-05 2024-09-21T10:17:39.085 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.085 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.085 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.085 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.085 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.085 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91083 11.8 11:1024c88d:::benchmark_data_smithi107_218812_object91082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:17:39.085 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.401346+0000", 2024-09-21T10:17:39.085 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 402.66003851699998, 2024-09-21T10:17:39.085 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2836915819999999, 2024-09-21T10:17:39.085 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:39.086 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:39.086 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:39.086 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:39.086 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:17:39.086 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:17:39.086 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91083 2024-09-21T10:17:39.086 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.086 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:39.086 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.086 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:39.086 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401346+0000", 2024-09-21T10:17:39.086 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.086 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.086 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.087 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:39.087 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401346+0000", 2024-09-21T10:17:39.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.087 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.087 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:39.087 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401347+0000", 2024-09-21T10:17:39.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7010000000000001e-06 2024-09-21T10:17:39.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.087 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.087 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:39.087 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401621+0000", 2024-09-21T10:17:39.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00027367500000000003 2024-09-21T10:17:39.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:39.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401632+0000", 2024-09-21T10:17:39.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0723000000000001e-05 2024-09-21T10:17:39.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:39.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401640+0000", 2024-09-21T10:17:39.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9880000000000004e-06 2024-09-21T10:17:39.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404683+0000", 2024-09-21T10:17:39.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0030434799999999999 2024-09-21T10:17:39.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404691+0000", 2024-09-21T10:17:39.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0099999999999995e-06 2024-09-21T10:17:39.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384431+0000", 2024-09-21T10:17:39.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.97973983200000003 2024-09-21T10:17:39.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384438+0000", 2024-09-21T10:17:39.090 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8979999999999999e-06 2024-09-21T10:17:39.090 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.090 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.090 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.090 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.676420+0000", 2024-09-21T10:17:39.090 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.291981727 2024-09-21T10:17:39.090 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.090 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.090 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:39.090 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.676514+0000", 2024-09-21T10:17:39.090 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3908999999999998e-05 2024-09-21T10:17:39.090 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.090 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.090 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:17:39.091 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.676614+0000", 2024-09-21T10:17:39.091 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000100746 2024-09-21T10:17:39.091 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.091 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.091 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.091 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.681917+0000", 2024-09-21T10:17:39.091 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0053031550000000004 2024-09-21T10:17:39.091 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.091 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.091 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:39.091 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.682073+0000", 2024-09-21T10:17:39.091 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015567900000000001 2024-09-21T10:17:39.091 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.091 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.092 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.682113+0000", 2024-09-21T10:17:39.092 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9536999999999998e-05 2024-09-21T10:17:39.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.092 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.092 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.684892+0000", 2024-09-21T10:17:39.092 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027794719999999998 2024-09-21T10:17:39.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.092 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:17:39.092 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.684940+0000", 2024-09-21T10:17:39.092 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7732000000000003e-05 2024-09-21T10:17:39.093 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.093 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.093 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:17:39.093 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.685037+0000", 2024-09-21T10:17:39.093 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7318000000000006e-05 2024-09-21T10:17:39.093 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.093 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.093 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.093 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.093 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.093 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91086 11.8 11:14b98868:::benchmark_data_smithi107_218812_object91085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:17:39.093 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.401700+0000", 2024-09-21T10:17:39.093 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 402.65968385100001, 2024-09-21T10:17:39.093 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2935442479999999, 2024-09-21T10:17:39.094 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:39.094 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:39.094 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:39.094 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:39.094 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:17:39.094 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:17:39.094 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91086 2024-09-21T10:17:39.094 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.094 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:39.094 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.094 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:39.094 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401700+0000", 2024-09-21T10:17:39.094 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.095 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.095 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.095 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:39.095 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401700+0000", 2024-09-21T10:17:39.095 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.095 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.095 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.095 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:39.095 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401701+0000", 2024-09-21T10:17:39.095 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6899999999999997e-07 2024-09-21T10:17:39.095 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.095 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.095 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:39.095 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401935+0000", 2024-09-21T10:17:39.095 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000233957 2024-09-21T10:17:39.096 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.096 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.096 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:39.096 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401944+0000", 2024-09-21T10:17:39.096 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.799e-06 2024-09-21T10:17:39.096 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.096 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.096 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:39.096 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401950+0000", 2024-09-21T10:17:39.096 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6499999999999999e-06 2024-09-21T10:17:39.096 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.096 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.096 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.096 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404712+0000", 2024-09-21T10:17:39.097 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027615949999999999 2024-09-21T10:17:39.097 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.097 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.097 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.097 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404718+0000", 2024-09-21T10:17:39.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:39.098 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9359999999999996e-06 2024-09-21T10:17:39.098 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.098 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.099 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.099 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384462+0000", 2024-09-21T10:17:39.099 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.97974425499999995 2024-09-21T10:17:39.099 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.099 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.099 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.099 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384470+0000", 2024-09-21T10:17:39.099 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5209999999999997e-06 2024-09-21T10:17:39.099 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.099 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.099 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.099 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.679346+0000", 2024-09-21T10:17:39.100 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2948768799999999 2024-09-21T10:17:39.100 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.100 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.100 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:39.100 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.679498+0000", 2024-09-21T10:17:39.100 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015129 2024-09-21T10:17:39.100 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.100 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.100 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:17:39.100 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.679670+0000", 2024-09-21T10:17:39.100 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017223 2024-09-21T10:17:39.100 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.100 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.100 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:39.101 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.684756+0000", 2024-09-21T10:17:39.101 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0050856470000000004 2024-09-21T10:17:39.101 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.101 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.101 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.101 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.684829+0000", 2024-09-21T10:17:39.101 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3808999999999998e-05 2024-09-21T10:17:39.101 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.101 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.101 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.101 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695008+0000", 2024-09-21T10:17:39.101 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010178159000000001 2024-09-21T10:17:39.101 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.101 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.101 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.102 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695080+0000", 2024-09-21T10:17:39.102 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2880999999999996e-05 2024-09-21T10:17:39.102 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.102 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.102 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:17:39.102 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695139+0000", 2024-09-21T10:17:39.102 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8390000000000002e-05 2024-09-21T10:17:39.102 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.102 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.102 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:17:39.102 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695245+0000", 2024-09-21T10:17:39.102 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010568 2024-09-21T10:17:39.102 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.103 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.103 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.103 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.103 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.103 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91090 11.a 11:532a4ae5:::benchmark_data_smithi107_218812_object91089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:17:39.103 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.404927+0000", 2024-09-21T10:17:39.103 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 402.65645733700001, 2024-09-21T10:17:39.103 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.299055724, 2024-09-21T10:17:39.103 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:39.103 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:39.103 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:39.103 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:39.103 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:17:39.103 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:17:39.103 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91090 2024-09-21T10:17:39.104 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.104 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:39.104 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.104 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:39.104 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404927+0000", 2024-09-21T10:17:39.104 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.104 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.104 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.104 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:39.104 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404927+0000", 2024-09-21T10:17:39.104 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.104 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.104 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.105 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:39.105 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404929+0000", 2024-09-21T10:17:39.105 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9750000000000001e-06 2024-09-21T10:17:39.105 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.105 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.105 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:39.105 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404955+0000", 2024-09-21T10:17:39.105 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5783e-05 2024-09-21T10:17:39.105 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.105 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.105 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:39.105 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404965+0000", 2024-09-21T10:17:39.105 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0400999999999999e-05 2024-09-21T10:17:39.105 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404973+0000", 2024-09-21T10:17:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4479999999999994e-06 2024-09-21T10:17:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.413662+0000", 2024-09-21T10:17:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0086882260000000003 2024-09-21T10:17:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.106 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.413672+0000", 2024-09-21T10:17:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0008e-05 2024-09-21T10:17:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385041+0000", 2024-09-21T10:17:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.97136919799999999 2024-09-21T10:17:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385049+0000", 2024-09-21T10:17:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.49e-06 2024-09-21T10:17:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.107 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.689104+0000", 2024-09-21T10:17:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.304054893 2024-09-21T10:17:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.689228+0000", 2024-09-21T10:17:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000124058 2024-09-21T10:17:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:17:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.689383+0000", 2024-09-21T10:17:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000154857 2024-09-21T10:17:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.108 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.700561+0000", 2024-09-21T10:17:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011177678 2024-09-21T10:17:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.700579+0000", 2024-09-21T10:17:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7887e-05 2024-09-21T10:17:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.701752+0000", 2024-09-21T10:17:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011736050000000001 2024-09-21T10:17:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.109 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.703561+0000", 2024-09-21T10:17:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001809045 2024-09-21T10:17:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:17:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.703884+0000", 2024-09-21T10:17:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000322715 2024-09-21T10:17:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:17:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.703983+0000", 2024-09-21T10:17:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8456999999999996e-05 2024-09-21T10:17:39.110 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91107 11.8 11:16a99f3f:::benchmark_data_smithi107_218812_object91106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:17:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.432894+0000", 2024-09-21T10:17:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 402.62848977200002, 2024-09-21T10:17:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2626001310000001, 2024-09-21T10:17:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:39.111 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:17:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:17:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91107 2024-09-21T10:17:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.432894+0000", 2024-09-21T10:17:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.432894+0000", 2024-09-21T10:17:39.112 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.432896+0000", 2024-09-21T10:17:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.302e-06 2024-09-21T10:17:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.432919+0000", 2024-09-21T10:17:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2988000000000001e-05 2024-09-21T10:17:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:39.113 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.432937+0000", 2024-09-21T10:17:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8147000000000001e-05 2024-09-21T10:17:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.432944+0000", 2024-09-21T10:17:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6809999999999998e-06 2024-09-21T10:17:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.433008+0000", 2024-09-21T10:17:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3972000000000003e-05 2024-09-21T10:17:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.114 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.433025+0000", 2024-09-21T10:17:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6109000000000001e-05 2024-09-21T10:17:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384493+0000", 2024-09-21T10:17:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.95146804900000004 2024-09-21T10:17:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.115 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384499+0000", 2024-09-21T10:17:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8040000000000004e-06 2024-09-21T10:17:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.682149+0000", 2024-09-21T10:17:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.297649716 2024-09-21T10:17:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:39.116 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.682241+0000", 2024-09-21T10:17:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1515999999999996e-05 2024-09-21T10:17:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:17:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.682337+0000", 2024-09-21T10:17:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6506000000000001e-05 2024-09-21T10:17:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.685192+0000", 2024-09-21T10:17:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028547640000000001 2024-09-21T10:17:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.685203+0000", 2024-09-21T10:17:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1555e-05 2024-09-21T10:17:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695046+0000", 2024-09-21T10:17:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098422700000000002 2024-09-21T10:17:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695355+0000", 2024-09-21T10:17:39.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000309005 2024-09-21T10:17:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:17:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695402+0000", 2024-09-21T10:17:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7172999999999997e-05 2024-09-21T10:17:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:17:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695494+0000", 2024-09-21T10:17:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2573999999999998e-05 2024-09-21T10:17:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91117 11.a 11:54639fb7:::benchmark_data_smithi107_218812_object91116:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:17:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.443321+0000", 2024-09-21T10:17:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 402.61806304300001, 2024-09-21T10:17:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2608621439999999, 2024-09-21T10:17:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:17:39.120 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:17:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91117 2024-09-21T10:17:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443321+0000", 2024-09-21T10:17:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443321+0000", 2024-09-21T10:17:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443323+0000", 2024-09-21T10:17:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.604e-06 2024-09-21T10:17:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443345+0000", 2024-09-21T10:17:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2370000000000001e-05 2024-09-21T10:17:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443358+0000", 2024-09-21T10:17:39.122 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3047999999999999e-05 2024-09-21T10:17:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443366+0000", 2024-09-21T10:17:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1089999999999998e-06 2024-09-21T10:17:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443456+0000", 2024-09-21T10:17:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0204999999999994e-05 2024-09-21T10:17:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.123 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443464+0000", 2024-09-21T10:17:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.323e-06 2024-09-21T10:17:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385072+0000", 2024-09-21T10:17:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.94160824399999998 2024-09-21T10:17:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385079+0000", 2024-09-21T10:17:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3880000000000002e-06 2024-09-21T10:17:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.692299+0000", 2024-09-21T10:17:39.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3072192389999999 2024-09-21T10:17:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.692459+0000", 2024-09-21T10:17:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000160686 2024-09-21T10:17:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:17:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.692664+0000", 2024-09-21T10:17:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000204692 2024-09-21T10:17:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.700714+0000", 2024-09-21T10:17:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.008050092 2024-09-21T10:17:39.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.700959+0000", 2024-09-21T10:17:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024535799999999999 2024-09-21T10:17:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.701790+0000", 2024-09-21T10:17:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00083083300000000001 2024-09-21T10:17:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.126 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704027+0000", 2024-09-21T10:17:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022363589999999998 2024-09-21T10:17:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:17:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704086+0000", 2024-09-21T10:17:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9502000000000001e-05 2024-09-21T10:17:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:17:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704183+0000", 2024-09-21T10:17:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7091999999999998e-05 2024-09-21T10:17:39.127 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91118 11.a 11:56ad5097:::benchmark_data_smithi107_218812_object91117:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:17:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.443509+0000", 2024-09-21T10:17:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 402.61787485600001, 2024-09-21T10:17:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2609485060000001, 2024-09-21T10:17:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:17:39.128 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:17:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91118 2024-09-21T10:17:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443509+0000", 2024-09-21T10:17:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443509+0000", 2024-09-21T10:17:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443510+0000", 2024-09-21T10:17:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6799999999999999e-07 2024-09-21T10:17:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443530+0000", 2024-09-21T10:17:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9551e-05 2024-09-21T10:17:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443543+0000", 2024-09-21T10:17:39.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3575e-05 2024-09-21T10:17:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443550+0000", 2024-09-21T10:17:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5969999999999997e-06 2024-09-21T10:17:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443616+0000", 2024-09-21T10:17:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6740000000000001e-05 2024-09-21T10:17:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.131 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443624+0000", 2024-09-21T10:17:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8220000000000007e-06 2024-09-21T10:17:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385103+0000", 2024-09-21T10:17:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.94147888199999996 2024-09-21T10:17:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385111+0000", 2024-09-21T10:17:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.486e-06 2024-09-21T10:17:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695503+0000", 2024-09-21T10:17:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3103919070000001 2024-09-21T10:17:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695648+0000", 2024-09-21T10:17:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014547799999999999 2024-09-21T10:17:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:17:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.696550+0000", 2024-09-21T10:17:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00090179799999999999 2024-09-21T10:17:39.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.700722+0000", 2024-09-21T10:17:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041717899999999999 2024-09-21T10:17:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.701004+0000", 2024-09-21T10:17:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00028281900000000002 2024-09-21T10:17:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.701834+0000", 2024-09-21T10:17:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00082920400000000005 2024-09-21T10:17:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704324+0000", 2024-09-21T10:17:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002490632 2024-09-21T10:17:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:17:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704377+0000", 2024-09-21T10:17:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2439999999999999e-05 2024-09-21T10:17:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:17:39.135 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704458+0000", 2024-09-21T10:17:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1017e-05 2024-09-21T10:17:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91121 11.8 11:16dab647:::benchmark_data_smithi107_218812_object91120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:17:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.447395+0000", 2024-09-21T10:17:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 402.613988672, 2024-09-21T10:17:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.248324582, 2024-09-21T10:17:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:39.136 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:17:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:17:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91121 2024-09-21T10:17:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447395+0000", 2024-09-21T10:17:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447395+0000", 2024-09-21T10:17:39.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447397+0000", 2024-09-21T10:17:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2330000000000001e-06 2024-09-21T10:17:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447448+0000", 2024-09-21T10:17:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1107999999999999e-05 2024-09-21T10:17:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:39.138 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447459+0000", 2024-09-21T10:17:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1241e-05 2024-09-21T10:17:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447467+0000", 2024-09-21T10:17:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8970000000000008e-06 2024-09-21T10:17:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447518+0000", 2024-09-21T10:17:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1329999999999998e-05 2024-09-21T10:17:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447525+0000", 2024-09-21T10:17:39.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5379999999999999e-06 2024-09-21T10:17:39.140 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.140 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.140 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384523+0000", 2024-09-21T10:17:39.140 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:39.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:39.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.93699787099999998 2024-09-21T10:17:39.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384530+0000", 2024-09-21T10:17:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8480000000000003e-06 2024-09-21T10:17:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.685237+0000", 2024-09-21T10:17:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.300707727 2024-09-21T10:17:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.685376+0000", 2024-09-21T10:17:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000139204 2024-09-21T10:17:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:17:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.685545+0000", 2024-09-21T10:17:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000168853 2024-09-21T10:17:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695283+0000", 2024-09-21T10:17:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097372459999999997 2024-09-21T10:17:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695320+0000", 2024-09-21T10:17:39.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7492e-05 2024-09-21T10:17:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695535+0000", 2024-09-21T10:17:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021545700000000001 2024-09-21T10:17:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695584+0000", 2024-09-21T10:17:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8458000000000003e-05 2024-09-21T10:17:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:17:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695627+0000", 2024-09-21T10:17:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3126e-05 2024-09-21T10:17:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:17:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695720+0000", 2024-09-21T10:17:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2953000000000003e-05 2024-09-21T10:17:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135668 12.11 12:88ec77bf:::benchmark_data_smithi107_250110_object24258:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:17:39.145 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.091435+0000", 2024-09-21T10:17:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.969948893999998, 2024-09-21T10:17:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7784578739999999, 2024-09-21T10:17:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135668 2024-09-21T10:17:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.091435+0000", 2024-09-21T10:17:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.091435+0000", 2024-09-21T10:17:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.091437+0000", 2024-09-21T10:17:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.341e-06 2024-09-21T10:17:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.091441+0000", 2024-09-21T10:17:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8210000000000001e-06 2024-09-21T10:17:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.091460+0000", 2024-09-21T10:17:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8746999999999999e-05 2024-09-21T10:17:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.091469+0000", 2024-09-21T10:17:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.388e-06 2024-09-21T10:17:39.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.091551+0000", 2024-09-21T10:17:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2844999999999997e-05 2024-09-21T10:17:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.091560+0000", 2024-09-21T10:17:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0659999999999993e-06 2024-09-21T10:17:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.580580+0000", 2024-09-21T10:17:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.489019859 2024-09-21T10:17:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.580591+0000", 2024-09-21T10:17:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1009000000000001e-05 2024-09-21T10:17:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.865360+0000", 2024-09-21T10:17:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2847688399999999 2024-09-21T10:17:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.865459+0000", 2024-09-21T10:17:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8939999999999998e-05 2024-09-21T10:17:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:17:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.865645+0000", 2024-09-21T10:17:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018592599999999999 2024-09-21T10:17:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869202+0000", 2024-09-21T10:17:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035568499999999999 2024-09-21T10:17:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869418+0000", 2024-09-21T10:17:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021635299999999999 2024-09-21T10:17:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869438+0000", 2024-09-21T10:17:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.942e-05 2024-09-21T10:17:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869617+0000", 2024-09-21T10:17:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000179703 2024-09-21T10:17:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869674+0000", 2024-09-21T10:17:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6777000000000002e-05 2024-09-21T10:17:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:17:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869734+0000", 2024-09-21T10:17:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0359000000000003e-05 2024-09-21T10:17:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:17:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869893+0000", 2024-09-21T10:17:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015862999999999999 2024-09-21T10:17:39.153 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135696 12.11 12:8cd3885c:::benchmark_data_smithi107_250110_object24286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:17:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.109943+0000", 2024-09-21T10:17:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.951440992000002, 2024-09-21T10:17:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.761551254, 2024-09-21T10:17:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:39.154 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135696 2024-09-21T10:17:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.109943+0000", 2024-09-21T10:17:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.109943+0000", 2024-09-21T10:17:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.109944+0000", 2024-09-21T10:17:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.89e-07 2024-09-21T10:17:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.109946+0000", 2024-09-21T10:17:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3590000000000002e-06 2024-09-21T10:17:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.109952+0000", 2024-09-21T10:17:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7830000000000004e-06 2024-09-21T10:17:39.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.109957+0000", 2024-09-21T10:17:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9679999999999997e-06 2024-09-21T10:17:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.110899+0000", 2024-09-21T10:17:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00094190099999999998 2024-09-21T10:17:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.110908+0000", 2024-09-21T10:17:39.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9099999999999994e-06 2024-09-21T10:17:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581020+0000", 2024-09-21T10:17:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.470112478 2024-09-21T10:17:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581029+0000", 2024-09-21T10:17:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1130000000000007e-06 2024-09-21T10:17:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.866702+0000", 2024-09-21T10:17:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.285672996 2024-09-21T10:17:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.866775+0000", 2024-09-21T10:17:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2391000000000005e-05 2024-09-21T10:17:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:17:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.866946+0000", 2024-09-21T10:17:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017150299999999999 2024-09-21T10:17:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869480+0000", 2024-09-21T10:17:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002534002 2024-09-21T10:17:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869506+0000", 2024-09-21T10:17:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6100000000000001e-05 2024-09-21T10:17:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869569+0000", 2024-09-21T10:17:39.160 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3360000000000003e-05 2024-09-21T10:17:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871245+0000", 2024-09-21T10:17:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001675775 2024-09-21T10:17:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871299+0000", 2024-09-21T10:17:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3761999999999998e-05 2024-09-21T10:17:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:17:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871360+0000", 2024-09-21T10:17:39.161 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1044999999999996e-05 2024-09-21T10:17:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:17:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871494+0000", 2024-09-21T10:17:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013431900000000001 2024-09-21T10:17:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135783 12.11 12:8ba445ab:::benchmark_data_smithi107_250110_object24373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:17:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.184409+0000", 2024-09-21T10:17:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.876974931000007, 2024-09-21T10:17:39.162 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.687499938, 2024-09-21T10:17:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135783 2024-09-21T10:17:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.184409+0000", 2024-09-21T10:17:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.163 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.184409+0000", 2024-09-21T10:17:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.184411+0000", 2024-09-21T10:17:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.455e-06 2024-09-21T10:17:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.184416+0000", 2024-09-21T10:17:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3059999999999997e-06 2024-09-21T10:17:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.184427+0000", 2024-09-21T10:17:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1483e-05 2024-09-21T10:17:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.184434+0000", 2024-09-21T10:17:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0310000000000001e-06 2024-09-21T10:17:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.184495+0000", 2024-09-21T10:17:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0273999999999999e-05 2024-09-21T10:17:39.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.184505+0000", 2024-09-21T10:17:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8349999999999999e-06 2024-09-21T10:17:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581054+0000", 2024-09-21T10:17:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.39654935299999999 2024-09-21T10:17:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.166 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581061+0000", 2024-09-21T10:17:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5299999999999999e-06 2024-09-21T10:17:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.867976+0000", 2024-09-21T10:17:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.286914812 2024-09-21T10:17:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.868095+0000", 2024-09-21T10:17:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011861799999999999 2024-09-21T10:17:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.167 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:17:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.868306+0000", 2024-09-21T10:17:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021122400000000001 2024-09-21T10:17:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869515+0000", 2024-09-21T10:17:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001208885 2024-09-21T10:17:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871185+0000", 2024-09-21T10:17:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0016699849999999999 2024-09-21T10:17:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.168 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871567+0000", 2024-09-21T10:17:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038179499999999999 2024-09-21T10:17:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871683+0000", 2024-09-21T10:17:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011604600000000001 2024-09-21T10:17:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871741+0000", 2024-09-21T10:17:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8096000000000001e-05 2024-09-21T10:17:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:17:39.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871791+0000", 2024-09-21T10:17:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0105999999999998e-05 2024-09-21T10:17:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:17:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871909+0000", 2024-09-21T10:17:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011810400000000001 2024-09-21T10:17:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135785 12.11 12:8b794dd9:::benchmark_data_smithi107_250110_object24375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:17:39.170 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.186985+0000", 2024-09-21T10:17:39.171 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.874398986000003, 2024-09-21T10:17:39.171 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6870419459999999, 2024-09-21T10:17:39.171 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:39.171 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:39.171 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:39.171 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:39.171 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:39.171 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:39.171 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135785 2024-09-21T10:17:39.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.172 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:39.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.172 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:39.172 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.186985+0000", 2024-09-21T10:17:39.172 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.173 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:39.173 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.186985+0000", 2024-09-21T10:17:39.173 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.173 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:39.173 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.186987+0000", 2024-09-21T10:17:39.173 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.508e-06 2024-09-21T10:17:39.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.174 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:39.174 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.186992+0000", 2024-09-21T10:17:39.174 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7049999999999996e-06 2024-09-21T10:17:39.174 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.174 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.174 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:39.174 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.187005+0000", 2024-09-21T10:17:39.174 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2259000000000001e-05 2024-09-21T10:17:39.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.175 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:39.175 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.187012+0000", 2024-09-21T10:17:39.175 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5190000000000003e-06 2024-09-21T10:17:39.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.175 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.176 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.188537+0000", 2024-09-21T10:17:39.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0015246579999999999 2024-09-21T10:17:39.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.176 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.176 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.176 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.188545+0000", 2024-09-21T10:17:39.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6959999999999994e-06 2024-09-21T10:17:39.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.176 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581086+0000", 2024-09-21T10:17:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.39254024199999998 2024-09-21T10:17:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581093+0000", 2024-09-21T10:17:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5850000000000002e-06 2024-09-21T10:17:39.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.178 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.178 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869950+0000", 2024-09-21T10:17:39.178 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2888568010000001 2024-09-21T10:17:39.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.178 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:39.178 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.870065+0000", 2024-09-21T10:17:39.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114547 2024-09-21T10:17:39.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.179 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:17:39.179 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.870201+0000", 2024-09-21T10:17:39.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013678099999999999 2024-09-21T10:17:39.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871587+0000", 2024-09-21T10:17:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001385266 2024-09-21T10:17:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.873496+0000", 2024-09-21T10:17:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001909315 2024-09-21T10:17:39.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.873546+0000", 2024-09-21T10:17:39.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9809999999999999e-05 2024-09-21T10:17:39.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.873607+0000", 2024-09-21T10:17:39.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1660000000000003e-05 2024-09-21T10:17:39.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.182 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.182 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.873848+0000", 2024-09-21T10:17:39.182 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024019500000000001 2024-09-21T10:17:39.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:17:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.873905+0000", 2024-09-21T10:17:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6950000000000002e-05 2024-09-21T10:17:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:17:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.874027+0000", 2024-09-21T10:17:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012244900000000001 2024-09-21T10:17:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135795 12.11 12:8de107fc:::benchmark_data_smithi107_250110_object24385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:17:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.194829+0000", 2024-09-21T10:17:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.866555539999993, 2024-09-21T10:17:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6837946939999999, 2024-09-21T10:17:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135795 2024-09-21T10:17:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:39.184 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.194829+0000", 2024-09-21T10:17:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.194829+0000", 2024-09-21T10:17:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.194830+0000", 2024-09-21T10:17:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.544e-06 2024-09-21T10:17:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.185 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.194838+0000", 2024-09-21T10:17:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8259999999999993e-06 2024-09-21T10:17:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.194849+0000", 2024-09-21T10:17:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1112e-05 2024-09-21T10:17:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.194856+0000", 2024-09-21T10:17:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0190000000000001e-06 2024-09-21T10:17:39.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.194912+0000", 2024-09-21T10:17:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5736999999999998e-05 2024-09-21T10:17:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.194921+0000", 2024-09-21T10:17:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6600000000000007e-06 2024-09-21T10:17:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581119+0000", 2024-09-21T10:17:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.38619732899999998 2024-09-21T10:17:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581126+0000", 2024-09-21T10:17:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.092e-06 2024-09-21T10:17:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871943+0000", 2024-09-21T10:17:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2908169979999999 2024-09-21T10:17:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.872055+0000", 2024-09-21T10:17:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111839 2024-09-21T10:17:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:17:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.872274+0000", 2024-09-21T10:17:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021912799999999999 2024-09-21T10:17:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.876414+0000", 2024-09-21T10:17:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004140477 2024-09-21T10:17:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878212+0000", 2024-09-21T10:17:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017976890000000001 2024-09-21T10:17:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.190 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878270+0000", 2024-09-21T10:17:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7739999999999999e-05 2024-09-21T10:17:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878329+0000", 2024-09-21T10:17:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9713999999999998e-05 2024-09-21T10:17:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878413+0000", 2024-09-21T10:17:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3217e-05 2024-09-21T10:17:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.191 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:17:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878508+0000", 2024-09-21T10:17:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5184000000000002e-05 2024-09-21T10:17:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:17:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878623+0000", 2024-09-21T10:17:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000115389 2024-09-21T10:17:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135809 12.11 12:8db42dc0:::benchmark_data_smithi107_250110_object24399:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:17:39.192 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.207496+0000", 2024-09-21T10:17:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.853888483000006, 2024-09-21T10:17:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6738327019999999, 2024-09-21T10:17:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135809 2024-09-21T10:17:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207496+0000", 2024-09-21T10:17:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207496+0000", 2024-09-21T10:17:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207497+0000", 2024-09-21T10:17:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.263e-06 2024-09-21T10:17:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:39.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207501+0000", 2024-09-21T10:17:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0890000000000002e-06 2024-09-21T10:17:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207510+0000", 2024-09-21T10:17:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5620000000000004e-06 2024-09-21T10:17:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207516+0000", 2024-09-21T10:17:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4779999999999996e-06 2024-09-21T10:17:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207580+0000", 2024-09-21T10:17:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4138000000000002e-05 2024-09-21T10:17:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207589+0000", 2024-09-21T10:17:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0510000000000002e-06 2024-09-21T10:17:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581151+0000", 2024-09-21T10:17:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.37356210099999998 2024-09-21T10:17:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581228+0000", 2024-09-21T10:17:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7077999999999996e-05 2024-09-21T10:17:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.874074+0000", 2024-09-21T10:17:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2928457309999999 2024-09-21T10:17:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.874194+0000", 2024-09-21T10:17:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000120315 2024-09-21T10:17:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.197 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:17:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.874495+0000", 2024-09-21T10:17:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030059999999999999 2024-09-21T10:17:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878342+0000", 2024-09-21T10:17:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038473410000000002 2024-09-21T10:17:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878373+0000", 2024-09-21T10:17:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0340000000000001e-05 2024-09-21T10:17:39.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878659+0000", 2024-09-21T10:17:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00028606999999999998 2024-09-21T10:17:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878766+0000", 2024-09-21T10:17:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000107095 2024-09-21T10:17:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.199 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.881101+0000", 2024-09-21T10:17:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023348700000000002 2024-09-21T10:17:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:17:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.881202+0000", 2024-09-21T10:17:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000100945 2024-09-21T10:17:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:17:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.881328+0000", 2024-09-21T10:17:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126635 2024-09-21T10:17:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.200 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135810 12.11 12:8861f427:::benchmark_data_smithi107_250110_object24400:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:17:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.207613+0000", 2024-09-21T10:17:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.853771464000005, 2024-09-21T10:17:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6877408169999999, 2024-09-21T10:17:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:39.201 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135810 2024-09-21T10:17:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207613+0000", 2024-09-21T10:17:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207613+0000", 2024-09-21T10:17:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.202 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207613+0000", 2024-09-21T10:17:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5700000000000002e-07 2024-09-21T10:17:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207616+0000", 2024-09-21T10:17:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8650000000000001e-06 2024-09-21T10:17:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207623+0000", 2024-09-21T10:17:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.199e-06 2024-09-21T10:17:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207628+0000", 2024-09-21T10:17:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1900000000000003e-06 2024-09-21T10:17:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207671+0000", 2024-09-21T10:17:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2518999999999997e-05 2024-09-21T10:17:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207678+0000", 2024-09-21T10:17:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4830000000000002e-06 2024-09-21T10:17:39.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581308+0000", 2024-09-21T10:17:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.37362985799999998 2024-09-21T10:17:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581318+0000", 2024-09-21T10:17:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2380000000000003e-06 2024-09-21T10:17:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.876458+0000", 2024-09-21T10:17:39.205 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295139985 2024-09-21T10:17:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.876571+0000", 2024-09-21T10:17:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000113198 2024-09-21T10:17:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:17:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.876799+0000", 2024-09-21T10:17:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022833099999999999 2024-09-21T10:17:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.206 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878384+0000", 2024-09-21T10:17:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001584785 2024-09-21T10:17:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.880888+0000", 2024-09-21T10:17:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025038119999999998 2024-09-21T10:17:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.880931+0000", 2024-09-21T10:17:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3463e-05 2024-09-21T10:17:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.207 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.881056+0000", 2024-09-21T10:17:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012447000000000001 2024-09-21T10:17:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.895044+0000", 2024-09-21T10:17:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013988377999999999 2024-09-21T10:17:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:17:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.895100+0000", 2024-09-21T10:17:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5686000000000002e-05 2024-09-21T10:17:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:17:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.895353+0000", 2024-09-21T10:17:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00025379999999999999 2024-09-21T10:17:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135845 12.11 12:8c5bb777:::benchmark_data_smithi107_250110_object24435:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:17:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.237799+0000", 2024-09-21T10:17:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.823584920000002, 2024-09-21T10:17:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6577566849999998, 2024-09-21T10:17:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:39.209 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135845 2024-09-21T10:17:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.237799+0000", 2024-09-21T10:17:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:39.210 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.237799+0000", 2024-09-21T10:17:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.237801+0000", 2024-09-21T10:17:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7099999999999999e-06 2024-09-21T10:17:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.237807+0000", 2024-09-21T10:17:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7969999999999997e-06 2024-09-21T10:17:39.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.237829+0000", 2024-09-21T10:17:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2353000000000001e-05 2024-09-21T10:17:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.237837+0000", 2024-09-21T10:17:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5190000000000003e-06 2024-09-21T10:17:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.237953+0000", 2024-09-21T10:17:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000115948 2024-09-21T10:17:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.237962+0000", 2024-09-21T10:17:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9690000000000006e-06 2024-09-21T10:17:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581347+0000", 2024-09-21T10:17:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.34338489700000002 2024-09-21T10:17:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581355+0000", 2024-09-21T10:17:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9780000000000005e-06 2024-09-21T10:17:39.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878792+0000", 2024-09-21T10:17:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2974364190000001 2024-09-21T10:17:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878909+0000", 2024-09-21T10:17:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011755 2024-09-21T10:17:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:17:39.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.879130+0000", 2024-09-21T10:17:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022026599999999999 2024-09-21T10:17:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.881376+0000", 2024-09-21T10:17:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022460739999999998 2024-09-21T10:17:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.881415+0000", 2024-09-21T10:17:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9267999999999997e-05 2024-09-21T10:17:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.215 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.216 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.881456+0000", 2024-09-21T10:17:39.216 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1029000000000003e-05 2024-09-21T10:17:39.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.216 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.216 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:39.216 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.886954+0000", 2024-09-21T10:17:39.216 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054981810000000004 2024-09-21T10:17:39.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.895400+0000", 2024-09-21T10:17:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0084454530000000003 2024-09-21T10:17:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:17:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.895449+0000", 2024-09-21T10:17:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8937999999999998e-05 2024-09-21T10:17:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:17:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.895556+0000", 2024-09-21T10:17:39.217 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000107336 2024-09-21T10:17:39.218 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.218 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.218 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.218 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.218 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135861 12.11 12:895efe19:::benchmark_data_smithi107_250110_object24451:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:17:39.218 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.255163+0000", 2024-09-21T10:17:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.806221543000007, 2024-09-21T10:17:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6406458740000001, 2024-09-21T10:17:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135861 2024-09-21T10:17:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.219 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.255163+0000", 2024-09-21T10:17:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.255163+0000", 2024-09-21T10:17:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:39.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.255164+0000", 2024-09-21T10:17:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8199999999999999e-06 2024-09-21T10:17:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.255171+0000", 2024-09-21T10:17:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9099999999999999e-06 2024-09-21T10:17:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.255188+0000", 2024-09-21T10:17:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6798000000000001e-05 2024-09-21T10:17:39.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.255197+0000", 2024-09-21T10:17:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4020000000000002e-06 2024-09-21T10:17:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.255267+0000", 2024-09-21T10:17:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0282000000000002e-05 2024-09-21T10:17:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.255278+0000", 2024-09-21T10:17:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0736e-05 2024-09-21T10:17:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581433+0000", 2024-09-21T10:17:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.3261558 2024-09-21T10:17:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581443+0000", 2024-09-21T10:17:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5300000000000002e-06 2024-09-21T10:17:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.881496+0000", 2024-09-21T10:17:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3000530180000001 2024-09-21T10:17:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.881665+0000", 2024-09-21T10:17:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016944099999999999 2024-09-21T10:17:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:17:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.881969+0000", 2024-09-21T10:17:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030320500000000002 2024-09-21T10:17:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.224 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.886854+0000", 2024-09-21T10:17:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004885375 2024-09-21T10:17:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.886941+0000", 2024-09-21T10:17:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6817000000000006e-05 2024-09-21T10:17:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.886970+0000", 2024-09-21T10:17:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8867e-05 2024-09-21T10:17:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.225 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.887019+0000", 2024-09-21T10:17:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9780000000000001e-05 2024-09-21T10:17:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.895654+0000", 2024-09-21T10:17:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0086348669999999992 2024-09-21T10:17:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:17:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.895705+0000", 2024-09-21T10:17:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0757000000000002e-05 2024-09-21T10:17:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.226 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:17:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.895808+0000", 2024-09-21T10:17:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010346899999999999 2024-09-21T10:17:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135947 12.11 12:8e645876:::benchmark_data_smithi107_250110_object24537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:17:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.353234+0000", 2024-09-21T10:17:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.708149917, 2024-09-21T10:17:39.227 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.543056319, 2024-09-21T10:17:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135947 2024-09-21T10:17:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.353234+0000", 2024-09-21T10:17:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.228 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.353234+0000", 2024-09-21T10:17:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.353236+0000", 2024-09-21T10:17:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7099999999999999e-06 2024-09-21T10:17:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.353241+0000", 2024-09-21T10:17:39.229 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8049999999999997e-06 2024-09-21T10:17:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.353258+0000", 2024-09-21T10:17:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7113e-05 2024-09-21T10:17:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.353266+0000", 2024-09-21T10:17:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4300000000000006e-06 2024-09-21T10:17:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.230 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.353325+0000", 2024-09-21T10:17:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9212999999999997e-05 2024-09-21T10:17:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.353332+0000", 2024-09-21T10:17:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5799999999999997e-06 2024-09-21T10:17:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581471+0000", 2024-09-21T10:17:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.22813892099999999 2024-09-21T10:17:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.231 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581480+0000", 2024-09-21T10:17:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1789999999999997e-06 2024-09-21T10:17:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.883441+0000", 2024-09-21T10:17:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.301960647 2024-09-21T10:17:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.883578+0000", 2024-09-21T10:17:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013736 2024-09-21T10:17:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.232 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:17:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.883807+0000", 2024-09-21T10:17:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000228414 2024-09-21T10:17:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.886974+0000", 2024-09-21T10:17:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031678969999999998 2024-09-21T10:17:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.887061+0000", 2024-09-21T10:17:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6939000000000002e-05 2024-09-21T10:17:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.233 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.887102+0000", 2024-09-21T10:17:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0284999999999998e-05 2024-09-21T10:17:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.887190+0000", 2024-09-21T10:17:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8722999999999997e-05 2024-09-21T10:17:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.895851+0000", 2024-09-21T10:17:39.234 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0086606219999999998 2024-09-21T10:17:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:17:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.896174+0000", 2024-09-21T10:17:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00032276800000000002 2024-09-21T10:17:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:17:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.896291+0000", 2024-09-21T10:17:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011671300000000001 2024-09-21T10:17:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.235 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135988 12.11 12:8e060443:::benchmark_data_smithi107_250110_object24578:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:17:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.397279+0000", 2024-09-21T10:17:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 86.664105382000002, 2024-09-21T10:17:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4995267540000001, 2024-09-21T10:17:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:17:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:17:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:17:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:17:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:17:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:17:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135988 2024-09-21T10:17:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.236 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:17:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:17:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.397279+0000", 2024-09-21T10:17:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:17:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.397279+0000", 2024-09-21T10:17:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:17:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:17:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.397280+0000", 2024-09-21T10:17:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.378e-06 2024-09-21T10:17:39.237 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:17:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.397316+0000", 2024-09-21T10:17:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5926000000000001e-05 2024-09-21T10:17:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:17:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.397330+0000", 2024-09-21T10:17:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.413e-05 2024-09-21T10:17:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.238 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:17:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.397338+0000", 2024-09-21T10:17:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9349999999999994e-06 2024-09-21T10:17:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.397406+0000", 2024-09-21T10:17:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7875999999999999e-05 2024-09-21T10:17:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.397416+0000", 2024-09-21T10:17:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0361e-05 2024-09-21T10:17:39.239 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581531+0000", 2024-09-21T10:17:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.184114578 2024-09-21T10:17:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:17:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581542+0000", 2024-09-21T10:17:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1333e-05 2024-09-21T10:17:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:17:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.885172+0000", 2024-09-21T10:17:39.240 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.303630093 2024-09-21T10:17:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:17:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.885311+0000", 2024-09-21T10:17:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013817500000000001 2024-09-21T10:17:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:17:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.885506+0000", 2024-09-21T10:17:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019548600000000001 2024-09-21T10:17:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.241 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.887151+0000", 2024-09-21T10:17:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001644572 2024-09-21T10:17:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.887231+0000", 2024-09-21T10:17:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0036000000000004e-05 2024-09-21T10:17:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.887272+0000", 2024-09-21T10:17:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.155e-05 2024-09-21T10:17:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.242 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:17:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.891257+0000", 2024-09-21T10:17:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0039846200000000003 2024-09-21T10:17:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:17:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.896475+0000", 2024-09-21T10:17:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052183869999999997 2024-09-21T10:17:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:17:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.896698+0000", 2024-09-21T10:17:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022309700000000001 2024-09-21T10:17:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:17:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:17:39.243 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:17:39.244 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.896805+0000", 2024-09-21T10:17:39.244 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000107221 2024-09-21T10:17:39.244 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.244 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.244 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.244 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:17:39.244 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:17:39.244 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:17:39.244 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:39.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:39.290 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:39.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:39.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:39.427 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:39.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:39.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:39.523 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:17:39.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:39.576 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:39.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:39.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:39.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:39.720 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:39.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:39.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:39.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:39.855 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:39.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:39 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/151962826' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:17:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:39 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:17:39.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:39 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/151962826' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:17:39.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:39 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:17:39.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:39 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/151962826' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:17:39.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:39 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:17:39.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:39.995 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:39.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:39.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:40.097 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:40.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:40.132 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:40.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:40.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:40.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:40.269 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:40.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:40.370 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:40.394 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:40.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:40.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:40.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:40.518 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:17:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:17:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:17:40.532 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:40.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:40.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:40 smithi107 ceph-mon[102221]: pgmap v1224: 119 pgs: 119 active+clean; 968 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T10:17:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:40 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:17:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:40 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:17:40.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:40 smithi107 ceph-mon[102221]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T10:17:40.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:40 smithi107 ceph-mon[105022]: pgmap v1224: 119 pgs: 119 active+clean; 968 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T10:17:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:40 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:17:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:40 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:17:40.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:40 smithi107 ceph-mon[105022]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T10:17:40.633 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:40.664 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:40.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:40.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:40.766 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:40.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:40.793 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:40.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:40.894 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:40.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:40 smithi129 ceph-mon[88471]: pgmap v1224: 119 pgs: 119 active+clean; 968 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T10:17:40.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:40 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:17:40.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:40 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:17:40.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:40 smithi129 ceph-mon[88471]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T10:17:40.942 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:40.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:40.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:41.044 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:41.068 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:41.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:41.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:41.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:41.209 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:41.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:41.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:41.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:41.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:41.351 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:41.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:41.452 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:41.479 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:41.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:41.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:41.581 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:41.607 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:41.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:41.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:41.709 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:41.744 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:41.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:41.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:41.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:41.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:41.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:41.888 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:41.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:41.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:41.986 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:17:41.990 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:42.072 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:42.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:42.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:42.082 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:17:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:17:41] "GET /metrics HTTP/1.1" 200 36801 "" "Prometheus/2.51.0" 2024-09-21T10:17:42.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:42.209 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:42.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:42.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:42.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:42.349 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:42.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:42.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:42.372 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:17:42.382 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:42.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:42.499 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:42.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:42.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:42.601 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:42.639 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:42.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:42.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:42.741 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:42.766 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:42.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:42.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:42.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:42 smithi107 ceph-mon[102221]: pgmap v1226: 119 pgs: 119 active+clean; 896 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 833 op/s 2024-09-21T10:17:42.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:42 smithi107 ceph-mon[105022]: pgmap v1226: 119 pgs: 119 active+clean; 896 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 833 op/s 2024-09-21T10:17:42.867 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:42.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:42.913 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:42.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:42 smithi129 ceph-mon[88471]: pgmap v1226: 119 pgs: 119 active+clean; 896 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 833 op/s 2024-09-21T10:17:43.014 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:43.046 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:43.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:43.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:43.148 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:43.176 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:43.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:43.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:43.278 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:43.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:43.314 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:43.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:43.415 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:43.449 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:43.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:43.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:43.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:17:43.551 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:43.577 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:43.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:43.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:43.678 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:43.705 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:43.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:43.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:43.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:43.875 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:43.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:43.976 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:44.008 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:44.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:44.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:44.144 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:44.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:44.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:44.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:44.279 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:44.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:44.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:44.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:44.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:44.432 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:44.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:44.533 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:44.571 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:44.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:44.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:44 smithi129 ceph-mon[88471]: pgmap v1227: 119 pgs: 119 active+clean; 636 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-09-21T10:17:44.673 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:44.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:44.703 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:44.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:44.728 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:44.794 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:44 smithi107 ceph-mon[102221]: pgmap v1227: 119 pgs: 119 active+clean; 636 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-09-21T10:17:44.794 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:44 smithi107 ceph-mon[105022]: pgmap v1227: 119 pgs: 119 active+clean; 636 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s 2024-09-21T10:17:44.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:44.863 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:44.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:44.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:44.965 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:45.000 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:45.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:45.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:45.063 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:17:45.063 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:17:45.064 INFO:teuthology.orchestra.run.smithi129.stdout: "num_ops": 0 2024-09-21T10:17:45.064 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:17:45.102 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:45.131 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:45.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:45.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:45.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:45.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:45.268 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:45.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:45.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:45.401 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:45.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:45.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:45.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:45.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:45.551 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:45.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:45.652 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:45.686 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:45.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:45.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:45.769 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:17:45.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:45.823 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:45.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:45.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:45.924 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:45.964 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:45.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:45.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:46.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:46.102 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:46.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:46.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:46.132 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:17:46.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:46.247 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:46.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:46.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:46.349 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:46.372 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:46.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:46.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:46.474 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:46.513 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:46.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:46.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:46.615 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:46.643 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:46.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:46.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:46.745 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:46.772 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:46.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:46.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:46.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:46 smithi107 ceph-mon[102221]: pgmap v1228: 119 pgs: 119 active+clean; 603 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T10:17:46.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:46 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3520782591' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:46 smithi107 ceph-mon[105022]: pgmap v1228: 119 pgs: 119 active+clean; 603 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T10:17:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:46 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3520782591' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:46.874 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:46.900 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:46.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:46.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:46 smithi129 ceph-mon[88471]: pgmap v1228: 119 pgs: 119 active+clean; 603 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T10:17:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:46 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3520782591' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:47.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:47.044 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:47.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:47.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:47.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:47.177 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:47.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:47.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:47.280 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:47.306 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:47.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:47.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:47.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:47.441 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:47.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:47.543 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:47.572 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:47.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:47.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:47.674 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:47.702 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:47.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:47.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:47.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:47.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:47.840 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:47.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:47.941 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:47.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:47.982 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:47.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:48.083 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:48.110 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:48.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:48.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:48.212 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:48.241 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:48.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:48.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:48.343 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:48.371 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:48.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:48.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:48.469 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:17:48.469 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:17:48.472 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:48.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:48.506 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:48.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:48.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:48 smithi107 ceph-mon[102221]: pgmap v1229: 119 pgs: 119 active+clean; 364 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T10:17:48.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:48 smithi107 ceph-mon[105022]: pgmap v1229: 119 pgs: 119 active+clean; 364 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T10:17:48.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:48.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:48.649 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:48.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:48.672 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:48 smithi129 ceph-mon[88471]: pgmap v1229: 119 pgs: 119 active+clean; 364 MiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T10:17:48.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:48.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:48.795 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:48.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:48.896 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:48.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:48.918 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:48.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:49.019 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:49.046 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:49.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:49.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:49.148 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:49.177 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:49.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:49.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:49.278 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:49.312 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:49.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:49.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:49.414 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:49.444 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:49.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:49.545 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:49.570 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:49.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:49.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:49.577 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T10:17:49.671 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:49.701 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:49.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:49.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:49.803 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:49.831 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:49.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:49.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:49.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:49 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3195354446' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:17:49.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:49 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:17:49.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:49 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3195354446' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:17:49.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:49 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:17:49.905 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:17:49.905 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:17:49.905 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:17:49.905 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:17:49.905 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:17:49.905 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:17:49.906 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:17:49.906 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:17:49.906 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:17:49.906 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:17:49.906 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:17:49.906 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:17:49.906 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:17:49.906 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:17:49.906 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:17:49.906 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:17:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:49 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3195354446' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:17:49.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:49 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:17:49.932 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:49.955 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:49.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:49.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:50.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:50.094 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:50.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:50.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:50.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:50.239 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:50.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:50.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:50.341 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:50.370 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:50.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:50.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:50.472 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:50.498 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:17:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:17:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:17:50.502 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:17:50.508 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:50.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:50.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:50.557 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:50 smithi107 ceph-mon[102221]: pgmap v1230: 119 pgs: 119 active+clean; 293 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T10:17:50.610 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:50.643 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:50.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:50.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:50.745 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:50.779 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:50.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:50.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:50.813 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:50 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:17:50.813 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:50 smithi107 ceph-mon[102221]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T10:17:50.813 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:50 smithi107 ceph-mon[105022]: pgmap v1230: 119 pgs: 119 active+clean; 293 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T10:17:50.813 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:50 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:17:50.813 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:50 smithi107 ceph-mon[105022]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T10:17:50.862 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:17:50.881 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:50.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:50 smithi129 ceph-mon[88471]: pgmap v1230: 119 pgs: 119 active+clean; 293 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T10:17:50.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:50 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:17:50.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:50 smithi129 ceph-mon[88471]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T10:17:50.922 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:50.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:50.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:51.024 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:51.049 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:51.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:51.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:51.102 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Removed 111407 objects 2024-09-21T10:17:51.102 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Clean up completed and total clean up time :131.204 2024-09-21T10:17:51.151 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:51.173 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:51.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:51.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:51.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T10:17:51.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:51.297 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:51.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:51.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:51.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:51.420 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:51.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:51.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:51.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:51.544 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:51.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:51.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:51.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:51.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:51.667 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:51.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:51.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:51.790 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:51.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:51.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:51.891 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:51.914 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:51.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:51.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:51.931 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:17:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:17:51] "GET /metrics HTTP/1.1" 200 36801 "" "Prometheus/2.51.0" 2024-09-21T10:17:52.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:52.037 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:52.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:52.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:52.139 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:52.160 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:52.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:52.198 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:52.262 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:52.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:52.284 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:52.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:52.385 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:52.407 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:52.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:52.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:52.509 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:52.530 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:52.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:52.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:52.560 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:17:52.574 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:52 smithi107 ceph-mon[102221]: pgmap v1232: 119 pgs: 119 active+clean; 221 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T10:17:52.574 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:52 smithi107 ceph-mon[102221]: 12.11 scrub starts 2024-09-21T10:17:52.574 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:52 smithi107 ceph-mon[102221]: 12.11 scrub ok 2024-09-21T10:17:52.574 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:52 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/409896113' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:17:52.575 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:52 smithi107 ceph-mon[105022]: pgmap v1232: 119 pgs: 119 active+clean; 221 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T10:17:52.575 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:52 smithi107 ceph-mon[105022]: 12.11 scrub starts 2024-09-21T10:17:52.575 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:52 smithi107 ceph-mon[105022]: 12.11 scrub ok 2024-09-21T10:17:52.575 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:52 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/409896113' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:17:52.632 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:52.653 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:52.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:52.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:52.659 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:17:52.659 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:17:52.756 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:52.777 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:52.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:52.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:52.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:17:52.879 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:52.900 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:52.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:52.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:52 smithi129 ceph-mon[88471]: pgmap v1232: 119 pgs: 119 active+clean; 221 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-21T10:17:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:52 smithi129 ceph-mon[88471]: 12.11 scrub starts 2024-09-21T10:17:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:52 smithi129 ceph-mon[88471]: 12.11 scrub ok 2024-09-21T10:17:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:52 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/409896113' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:17:53.002 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:53.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:53.025 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:53.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:53.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:53.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:53.148 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:53.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:53.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:53.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:53.271 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:53.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:53.373 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:53.395 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:53.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:53.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:53.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:53.517 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:53.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:53.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:53.619 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:53.641 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:53.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:53.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:53.743 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:53.763 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:53.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:53.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:53 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:17:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:53 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:17:53.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:53 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/409896113' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:17:53.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:53 smithi107 ceph-mon[102221]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T10:17:53.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:53 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:17:53.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:53 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:17:53.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:53 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/409896113' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:17:53.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:53 smithi107 ceph-mon[105022]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T10:17:53.864 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:53.885 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:53.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:53.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:53 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:17:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:53 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:17:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:53 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/409896113' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:17:53.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:53 smithi129 ceph-mon[88471]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T10:17:53.987 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:54.008 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:54.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:54.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:54.083 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:17:54.110 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:54.148 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:54.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:54.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:54.250 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:54.272 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:54.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:54.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:54.327 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:17:54.374 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:54.395 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:54.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:54.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:54.499 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:54.522 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:54.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:54.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:54.624 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:54.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:54.645 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:54.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:54.739 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:54 smithi129 ceph-mon[88471]: pgmap v1234: 119 pgs: 119 active+clean; 15 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 311 op/s 2024-09-21T10:17:54.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:54.769 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:54.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:54.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:54 smithi107 ceph-mon[102221]: pgmap v1234: 119 pgs: 119 active+clean; 15 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 311 op/s 2024-09-21T10:17:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:54 smithi107 ceph-mon[105022]: pgmap v1234: 119 pgs: 119 active+clean; 15 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 311 op/s 2024-09-21T10:17:54.870 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:17:54.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:54.892 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:54.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:17:54.993 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:55.013 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:55.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:55.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:55.115 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:55.136 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:55.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:55.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:55.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:55.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:55.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:55.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:55.370 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:55.387 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:55.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:55.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:55.489 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:55.509 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:55.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:55.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:55.569 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_7 2024-09-21T10:17:55.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_7 unique_pool_7 --yes-i-really-really-mean-it 2024-09-21T10:17:55.578 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:17:55.582 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T10:17:55.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:55.646 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:55.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:55.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:55.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:55.754 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:55 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2992517633' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:17:55.754 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:55 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:17:55.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:55 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2992517633' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:17:55.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:55 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:17:55.768 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:55 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2992517633' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:17:55.768 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:55 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:17:55.771 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:55.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:55.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:55.873 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:55.894 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:55.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:55.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:55.963 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:17:55.963 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:17:55.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:56.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:56.018 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:56.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:56.119 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:56.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:56.141 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:56.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:56.242 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:56.265 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:56.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:56.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:56.367 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:56.388 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:56.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:56.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:56.490 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:56.512 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:56.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:56.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:56.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T10:17:56.610 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:17:56.614 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:56.674 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:56.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:56.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:56.775 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:17:56.797 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:56.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:56.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:17:56.858 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:17:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:56 smithi107 ceph-mon[102221]: pgmap v1235: 119 pgs: 119 active+clean; 6.0 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:17:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:56 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:17:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:56 smithi107 ceph-mon[102221]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T10:17:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:17:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:56.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:56 smithi107 ceph-mon[105022]: pgmap v1235: 119 pgs: 119 active+clean; 6.0 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:17:56.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:56 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:17:56.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:56 smithi107 ceph-mon[105022]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T10:17:56.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:17:56.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:56.899 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:56 smithi129 ceph-mon[88471]: pgmap v1235: 119 pgs: 119 active+clean; 6.0 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:17:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:56 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:17:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:56 smithi129 ceph-mon[88471]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T10:17:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:17:56.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:56.921 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:56.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:56.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:57.007 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:57.008 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:57.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:57.049 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:57.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:57.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:57.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:57.172 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:57.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:57.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:57.274 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:57.296 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:57.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:57.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:57.397 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:57.418 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:57.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:57.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:57.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:57.541 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:57.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:57.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:57.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:57.664 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:57.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:57.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:57.766 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:57.788 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:57.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:57.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:57.889 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:57.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:57.912 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:57.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:58.013 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:58.017 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:17:58.036 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:58.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:58.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:58.137 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:58.158 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:58.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:58.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:58.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:17:58.282 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:58.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:58.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:17:58.384 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:58.406 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:58.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:58.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:58.496 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:17:58.496 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:17:58.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:58.529 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:58.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:58.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:58.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:58.653 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:58.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:58.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:58.686 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:17:58.687 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:17:58.703 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:17:58.703 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:17:58.720 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:17:58.720 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:17:58.736 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:17:58.736 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:17:58.752 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:17:58.752 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:17:58.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:58.767 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:17:58.767 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:17:58.775 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:58.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:58.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:58.783 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:17:58.783 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:17:58.799 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:17:58.799 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:17:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:58 smithi107 ceph-mon[102221]: pgmap v1237: 119 pgs: 119 active+clean; 2.3 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:17:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:58 smithi107 ceph-mon[105022]: pgmap v1237: 119 pgs: 119 active+clean; 2.3 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:17:58.860 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:58 smithi129 ceph-mon[88471]: pgmap v1237: 119 pgs: 119 active+clean; 2.3 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:17:58.877 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:58.901 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:58.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:58.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:59.003 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:17:59.025 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:59.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:59.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:17:59.127 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:59.150 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:59.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:59.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:59.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:17:59.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:17:59.274 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:59.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:59.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:17:59.375 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:59.397 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:59.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:59.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:59.443 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:17:59.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:59.519 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:59.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:59.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:59.621 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:17:59.643 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:17:59.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:59.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:17:59.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:17:59.766 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:59.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:59.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:17:59.821 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:17:59 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1926715889' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:17:59 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1926715889' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:17:59 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1926715889' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:17:59.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:17:59.890 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:17:59.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:17:59.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:17:59.991 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:00.013 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:00.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:00.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:00.114 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:00.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:00.137 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:00.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:00.238 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:00.259 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:00.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:00.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:00.360 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:00.382 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:00.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:00.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:00.484 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:00.506 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:00.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:00.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:00.585 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:18:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:18:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:18:00.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:00.630 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:00.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:00.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:00.731 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:00.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:00.754 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:00.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:00.844 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:00 smithi129 ceph-mon[88471]: pgmap v1238: 119 pgs: 119 active+clean; 2.3 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:00.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:00 smithi107 ceph-mon[105022]: pgmap v1238: 119 pgs: 119 active+clean; 2.3 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:00.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:00 smithi107 ceph-mon[102221]: pgmap v1238: 119 pgs: 119 active+clean; 2.3 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:00.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:00.878 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:00.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:00.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:00.979 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:01.001 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:01.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:01.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:01.102 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:01.123 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:01.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:01.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:01.196 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:18:01.196 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 20, 2024-09-21T10:18:01.196 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 600, 2024-09-21T10:18:01.196 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:18:01.196 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.196 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247256 10.6 10:6298720f:::benchmark_data_smithi107_184135_object115690:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:18:01.197 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:50.433070+0000", 2024-09-21T10:18:01.197 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 550.75680909200003, 2024-09-21T10:18:01.197 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.99909955800000005, 2024-09-21T10:18:01.197 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:18:01.197 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:01.197 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:18:01.197 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:18:01.197 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:18:01.197 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247256 2024-09-21T10:18:01.197 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.197 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:18:01.197 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.197 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:18:01.197 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433070+0000", 2024-09-21T10:18:01.198 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.198 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.198 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.198 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:18:01.198 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433070+0000", 2024-09-21T10:18:01.198 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.198 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.198 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.198 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:18:01.198 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433067+0000", 2024-09-21T10:18:01.198 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999971 2024-09-21T10:18:01.198 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.198 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.198 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:18:01.199 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433079+0000", 2024-09-21T10:18:01.199 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1345000000000001e-05 2024-09-21T10:18:01.199 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.199 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.199 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:18:01.199 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433099+0000", 2024-09-21T10:18:01.199 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.0852e-05 2024-09-21T10:18:01.199 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.199 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.199 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:18:01.199 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433108+0000", 2024-09-21T10:18:01.199 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.8270000000000004e-06 2024-09-21T10:18:01.199 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.200 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.200 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.200 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433168+0000", 2024-09-21T10:18:01.200 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9737000000000001e-05 2024-09-21T10:18:01.200 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.200 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.200 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:18:01.200 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.433179+0000", 2024-09-21T10:18:01.200 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0577000000000001e-05 2024-09-21T10:18:01.200 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.200 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.200 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.200 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.417734+0000", 2024-09-21T10:18:01.200 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.98455554300000003 2024-09-21T10:18:01.201 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.201 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.201 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:18:01.201 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.417856+0000", 2024-09-21T10:18:01.201 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000121657 2024-09-21T10:18:01.201 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.201 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.201 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:18:01.201 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.417974+0000", 2024-09-21T10:18:01.201 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000118348 2024-09-21T10:18:01.201 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.201 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.201 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:01.201 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.421532+0000", 2024-09-21T10:18:01.202 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003558205 2024-09-21T10:18:01.202 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.202 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.202 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:01.202 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432010+0000", 2024-09-21T10:18:01.202 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.010477765999999999 2024-09-21T10:18:01.202 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.202 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.202 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:18:01.202 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432019+0000", 2024-09-21T10:18:01.202 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.1379999999999996e-06 2024-09-21T10:18:01.202 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.202 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:18:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432070+0000", 2024-09-21T10:18:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0719e-05 2024-09-21T10:18:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:18:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432170+0000", 2024-09-21T10:18:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.9630000000000007e-05 2024-09-21T10:18:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.203 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247262 10.6 10:67e2f4ef:::benchmark_data_smithi107_184135_object115696:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:18:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:50.439088+0000", 2024-09-21T10:18:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 550.75079120099997, 2024-09-21T10:18:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.99353488599999995, 2024-09-21T10:18:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:18:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:18:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:18:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:18:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247262 2024-09-21T10:18:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:18:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.204 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:18:01.205 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439088+0000", 2024-09-21T10:18:01.205 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.205 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.205 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.205 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:18:01.205 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439088+0000", 2024-09-21T10:18:01.205 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.205 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.205 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.205 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:18:01.205 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439086+0000", 2024-09-21T10:18:01.205 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:18:01.205 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.205 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.206 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:18:01.206 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439094+0000", 2024-09-21T10:18:01.206 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.7840000000000004e-06 2024-09-21T10:18:01.206 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.206 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.206 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:18:01.206 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439111+0000", 2024-09-21T10:18:01.206 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.7059e-05 2024-09-21T10:18:01.206 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.206 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.206 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:18:01.206 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439118+0000", 2024-09-21T10:18:01.206 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5239999999999997e-06 2024-09-21T10:18:01.207 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.207 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.207 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.207 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439172+0000", 2024-09-21T10:18:01.207 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4327999999999999e-05 2024-09-21T10:18:01.207 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.207 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.207 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:18:01.207 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.439185+0000", 2024-09-21T10:18:01.207 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3502999999999999e-05 2024-09-21T10:18:01.207 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.207 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.207 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.207 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.421553+0000", 2024-09-21T10:18:01.208 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.98236765199999998 2024-09-21T10:18:01.208 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.208 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.208 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:18:01.208 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.421626+0000", 2024-09-21T10:18:01.208 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3146999999999996e-05 2024-09-21T10:18:01.208 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.208 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.208 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:18:01.208 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.421685+0000", 2024-09-21T10:18:01.208 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8621e-05 2024-09-21T10:18:01.208 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.208 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.208 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:18:01.209 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432293+0000", 2024-09-21T10:18:01.209 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.010608332 2024-09-21T10:18:01.209 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.209 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.209 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:01.209 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432344+0000", 2024-09-21T10:18:01.209 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1279999999999997e-05 2024-09-21T10:18:01.209 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.209 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.209 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:01.209 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432459+0000", 2024-09-21T10:18:01.209 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000114418 2024-09-21T10:18:01.210 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.210 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.210 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:18:01.210 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432518+0000", 2024-09-21T10:18:01.210 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9299000000000003e-05 2024-09-21T10:18:01.210 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.210 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.210 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:18:01.210 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432623+0000", 2024-09-21T10:18:01.210 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010468599999999999 2024-09-21T10:18:01.210 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.210 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.211 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.211 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.211 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.211 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247267 10.6 10:615d1e96:::benchmark_data_smithi107_184135_object115701:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:18:01.211 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:50.444059+0000", 2024-09-21T10:18:01.211 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 550.74582033199999, 2024-09-21T10:18:01.211 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.98879305900000003, 2024-09-21T10:18:01.211 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:18:01.211 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:01.211 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:18:01.211 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:18:01.211 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:18:01.211 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247267 2024-09-21T10:18:01.212 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.212 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:18:01.212 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.212 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:18:01.212 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444059+0000", 2024-09-21T10:18:01.212 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.212 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.212 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.212 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:18:01.212 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444059+0000", 2024-09-21T10:18:01.212 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.212 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.212 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.212 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:18:01.212 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444057+0000", 2024-09-21T10:18:01.213 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:18:01.213 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.213 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.213 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:18:01.213 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444064+0000", 2024-09-21T10:18:01.213 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.5349999999999999e-06 2024-09-21T10:18:01.213 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.213 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.213 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:18:01.213 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444080+0000", 2024-09-21T10:18:01.213 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5478999999999999e-05 2024-09-21T10:18:01.213 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.213 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.213 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:18:01.214 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444086+0000", 2024-09-21T10:18:01.214 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.631e-06 2024-09-21T10:18:01.214 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.214 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.214 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.214 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444247+0000", 2024-09-21T10:18:01.214 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00016138 2024-09-21T10:18:01.214 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.214 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.214 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:18:01.214 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.444256+0000", 2024-09-21T10:18:01.214 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.3510000000000008e-06 2024-09-21T10:18:01.214 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.215 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.215 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.215 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.424581+0000", 2024-09-21T10:18:01.215 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.98032498800000001 2024-09-21T10:18:01.215 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.215 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.215 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:18:01.215 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.424692+0000", 2024-09-21T10:18:01.215 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000111113 2024-09-21T10:18:01.215 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.215 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.215 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:18:01.215 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.424815+0000", 2024-09-21T10:18:01.215 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012274900000000001 2024-09-21T10:18:01.215 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.215 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.216 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:18:01.216 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432299+0000", 2024-09-21T10:18:01.216 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0074834740000000004 2024-09-21T10:18:01.216 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.216 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.216 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:01.216 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432681+0000", 2024-09-21T10:18:01.216 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00038213 2024-09-21T10:18:01.216 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.216 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.216 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:01.216 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432722+0000", 2024-09-21T10:18:01.216 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.138e-05 2024-09-21T10:18:01.216 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.217 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.217 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:18:01.217 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432765+0000", 2024-09-21T10:18:01.217 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2769000000000003e-05 2024-09-21T10:18:01.217 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.217 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.217 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:18:01.217 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432852+0000", 2024-09-21T10:18:01.217 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.7026000000000003e-05 2024-09-21T10:18:01.217 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.217 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.217 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.217 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.217 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.218 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36210.0:247272 10.6 10:699abf74:::benchmark_data_smithi107_184135_object115706:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T10:18:01.218 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:08:50.447705+0000", 2024-09-21T10:18:01.218 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 550.74217449299999, 2024-09-21T10:18:01.218 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.985396304, 2024-09-21T10:18:01.218 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:18:01.218 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:01.218 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:18:01.218 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36210", 2024-09-21T10:18:01.218 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/4091374417", 2024-09-21T10:18:01.218 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 247272 2024-09-21T10:18:01.218 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.218 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:18:01.218 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.219 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:18:01.219 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447705+0000", 2024-09-21T10:18:01.219 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.219 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.219 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.219 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:18:01.219 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447705+0000", 2024-09-21T10:18:01.219 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.219 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.219 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.219 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:18:01.219 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447703+0000", 2024-09-21T10:18:01.219 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:18:01.219 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.220 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.220 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:18:01.220 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447711+0000", 2024-09-21T10:18:01.220 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.5229999999999998e-06 2024-09-21T10:18:01.220 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.220 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.220 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:18:01.220 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447721+0000", 2024-09-21T10:18:01.220 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0040999999999999e-05 2024-09-21T10:18:01.220 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.220 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.220 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:18:01.220 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447725+0000", 2024-09-21T10:18:01.220 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2679999999999996e-06 2024-09-21T10:18:01.221 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.221 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.221 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.221 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447767+0000", 2024-09-21T10:18:01.221 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1897999999999998e-05 2024-09-21T10:18:01.221 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.221 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.221 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:18:01.221 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:50.447776+0000", 2024-09-21T10:18:01.221 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.2089999999999994e-06 2024-09-21T10:18:01.221 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.221 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.221 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.221 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.427698+0000", 2024-09-21T10:18:01.222 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.97992116600000001 2024-09-21T10:18:01.222 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.222 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.222 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:18:01.222 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.427813+0000", 2024-09-21T10:18:01.222 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000115301 2024-09-21T10:18:01.222 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.222 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.222 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:18:01.222 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.427925+0000", 2024-09-21T10:18:01.222 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000112324 2024-09-21T10:18:01.222 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.222 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.222 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:18:01.223 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432357+0000", 2024-09-21T10:18:01.223 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0044316570000000003 2024-09-21T10:18:01.223 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.223 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.223 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:01.223 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432891+0000", 2024-09-21T10:18:01.223 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00053454999999999996 2024-09-21T10:18:01.223 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.223 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.223 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:01.223 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.432945+0000", 2024-09-21T10:18:01.223 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3307999999999999e-05 2024-09-21T10:18:01.223 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.223 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.223 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:18:01.224 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.433014+0000", 2024-09-21T10:18:01.224 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8949999999999995e-05 2024-09-21T10:18:01.224 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.224 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.224 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:18:01.224 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:08:51.433101+0000", 2024-09-21T10:18:01.224 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.7322000000000002e-05 2024-09-21T10:18:01.224 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.224 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.224 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.224 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.224 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.225 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:96818 11.5 e269/267 11:ad788e1e:::benchmark_data_smithi107_218812_object96817:head v 269'5821, mlcod=257'161)", 2024-09-21T10:18:01.225 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:11:04.703675+0000", 2024-09-21T10:18:01.225 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 416.48620397799999, 2024-09-21T10:18:01.225 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.046626753999999999, 2024-09-21T10:18:01.225 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:18:01.225 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:01.225 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:18:01.225 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.225 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:18:01.225 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:04.703675+0000", 2024-09-21T10:18:01.225 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.225 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.225 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.225 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:18:01.226 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:04.703675+0000", 2024-09-21T10:18:01.226 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.226 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.226 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.226 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:18:01.226 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:04.703674+0000", 2024-09-21T10:18:01.226 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:18:01.226 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.226 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.226 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:18:01.226 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:04.703699+0000", 2024-09-21T10:18:01.226 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.457e-05 2024-09-21T10:18:01.226 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.227 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.227 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:18:01.227 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:04.703713+0000", 2024-09-21T10:18:01.227 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4326e-05 2024-09-21T10:18:01.227 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.227 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.227 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:18:01.227 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:04.703719+0000", 2024-09-21T10:18:01.227 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4659999999999998e-06 2024-09-21T10:18:01.227 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.227 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.227 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.227 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:04.706197+0000", 2024-09-21T10:18:01.227 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0024780190000000001 2024-09-21T10:18:01.228 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.228 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.228 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:18:01.228 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:04.706220+0000", 2024-09-21T10:18:01.228 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.3298e-05 2024-09-21T10:18:01.228 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.228 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.228 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:18:01.228 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:04.750268+0000", 2024-09-21T10:18:01.228 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.044048205999999999 2024-09-21T10:18:01.228 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.228 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.228 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:18:01.228 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:04.750302+0000", 2024-09-21T10:18:01.229 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3538999999999999e-05 2024-09-21T10:18:01.229 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.229 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.229 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.229 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.229 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.229 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:97127 11.5 e269/267 11:a0ed3385:::benchmark_data_smithi107_218812_object97126:head v 269'5846, mlcod=257'167)", 2024-09-21T10:18:01.229 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:11:05.012651+0000", 2024-09-21T10:18:01.229 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 416.17722790599998, 2024-09-21T10:18:01.229 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.052333383999999997, 2024-09-21T10:18:01.230 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:18:01.230 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:01.230 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:18:01.230 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.230 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:18:01.230 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.012651+0000", 2024-09-21T10:18:01.230 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.230 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.230 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.230 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:18:01.230 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.012651+0000", 2024-09-21T10:18:01.230 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.230 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.231 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.231 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:18:01.231 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.012651+0000", 2024-09-21T10:18:01.231 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:18:01.231 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.231 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.231 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:18:01.231 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.012689+0000", 2024-09-21T10:18:01.231 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8645e-05 2024-09-21T10:18:01.231 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.231 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.231 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:18:01.231 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.012702+0000", 2024-09-21T10:18:01.231 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3049e-05 2024-09-21T10:18:01.232 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.232 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.232 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:18:01.232 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.012707+0000", 2024-09-21T10:18:01.232 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.079e-06 2024-09-21T10:18:01.232 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.232 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.232 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.232 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.015154+0000", 2024-09-21T10:18:01.232 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002446466 2024-09-21T10:18:01.232 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.232 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.232 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:18:01.232 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.015180+0000", 2024-09-21T10:18:01.233 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.6118999999999998e-05 2024-09-21T10:18:01.233 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.233 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.233 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:18:01.233 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.064954+0000", 2024-09-21T10:18:01.233 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.049773533000000002 2024-09-21T10:18:01.233 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.233 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.233 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:18:01.233 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.064985+0000", 2024-09-21T10:18:01.233 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1096000000000002e-05 2024-09-21T10:18:01.233 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.233 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.233 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.234 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.234 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.234 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:97129 11.5 e269/267 11:a054297d:::benchmark_data_smithi107_218812_object97128:head v 269'5847, mlcod=257'167)", 2024-09-21T10:18:01.234 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:11:05.017578+0000", 2024-09-21T10:18:01.234 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 416.172300893, 2024-09-21T10:18:01.234 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.047423854000000001, 2024-09-21T10:18:01.234 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:18:01.234 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:01.234 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:18:01.234 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.234 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:18:01.234 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.017578+0000", 2024-09-21T10:18:01.234 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.234 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.235 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.235 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:18:01.235 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.017578+0000", 2024-09-21T10:18:01.235 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.235 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.235 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.235 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:18:01.235 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.017576+0000", 2024-09-21T10:18:01.235 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:18:01.235 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.235 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.235 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:18:01.235 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.017606+0000", 2024-09-21T10:18:01.236 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0020000000000001e-05 2024-09-21T10:18:01.236 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.236 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.236 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:18:01.236 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.017626+0000", 2024-09-21T10:18:01.236 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.9415e-05 2024-09-21T10:18:01.236 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.236 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.236 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:18:01.236 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.017633+0000", 2024-09-21T10:18:01.236 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.6040000000000004e-06 2024-09-21T10:18:01.236 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.237 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.237 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.237 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.023067+0000", 2024-09-21T10:18:01.237 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.005433284 2024-09-21T10:18:01.237 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.237 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.237 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:18:01.237 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.023096+0000", 2024-09-21T10:18:01.237 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.9111999999999998e-05 2024-09-21T10:18:01.237 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.237 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.237 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:18:01.237 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.064991+0000", 2024-09-21T10:18:01.237 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.041895108 2024-09-21T10:18:01.237 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.238 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.238 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:18:01.238 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.065002+0000", 2024-09-21T10:18:01.238 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1280999999999999e-05 2024-09-21T10:18:01.238 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.238 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.238 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.238 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.238 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.238 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:97142 11.5 e269/267 11:aa3cf3e1:::benchmark_data_smithi107_218812_object97141:head v 269'5848, mlcod=257'167)", 2024-09-21T10:18:01.238 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:11:05.019829+0000", 2024-09-21T10:18:01.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:01.240 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 416.17005003399998, 2024-09-21T10:18:01.240 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.045184593000000002, 2024-09-21T10:18:01.240 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:18:01.240 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:01.240 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:18:01.240 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.240 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:18:01.240 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.019829+0000", 2024-09-21T10:18:01.240 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.241 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.241 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.241 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:18:01.241 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.019829+0000", 2024-09-21T10:18:01.241 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.241 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.241 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.241 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:18:01.241 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.019828+0000", 2024-09-21T10:18:01.241 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:18:01.241 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.241 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.241 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:18:01.241 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.019852+0000", 2024-09-21T10:18:01.242 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.3932000000000001e-05 2024-09-21T10:18:01.242 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.242 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.242 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:18:01.242 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.019868+0000", 2024-09-21T10:18:01.242 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5569e-05 2024-09-21T10:18:01.242 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.242 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.242 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:18:01.242 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.019873+0000", 2024-09-21T10:18:01.242 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4940000000000003e-06 2024-09-21T10:18:01.242 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.242 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.242 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.031721+0000", 2024-09-21T10:18:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.011848152000000001 2024-09-21T10:18:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:18:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.031751+0000", 2024-09-21T10:18:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0111999999999999e-05 2024-09-21T10:18:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:18:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.065004+0000", 2024-09-21T10:18:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.033252219999999999 2024-09-21T10:18:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.243 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.244 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:18:01.244 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:05.065014+0000", 2024-09-21T10:18:01.244 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0039e-05 2024-09-21T10:18:01.244 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.244 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.244 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.244 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.244 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.244 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:108027 11.5 e272/267 11:a08c65b5:::benchmark_data_smithi107_218812_object108026:head v 272'6477, mlcod=257'547)", 2024-09-21T10:18:01.244 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:11:17.585710+0000", 2024-09-21T10:18:01.244 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 403.60416905, 2024-09-21T10:18:01.244 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.044484076999999997, 2024-09-21T10:18:01.244 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:18:01.245 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:01.245 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:18:01.245 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.245 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:18:01.245 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.585710+0000", 2024-09-21T10:18:01.245 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.245 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.245 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.245 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:18:01.245 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.585710+0000", 2024-09-21T10:18:01.245 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.245 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.245 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.245 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:18:01.246 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.585709+0000", 2024-09-21T10:18:01.246 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:18:01.246 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.246 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.246 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:18:01.246 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.585754+0000", 2024-09-21T10:18:01.246 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4891000000000001e-05 2024-09-21T10:18:01.246 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.246 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.246 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:18:01.246 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.585770+0000", 2024-09-21T10:18:01.246 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5889e-05 2024-09-21T10:18:01.246 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.247 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.247 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:18:01.247 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.585777+0000", 2024-09-21T10:18:01.247 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8199999999999999e-06 2024-09-21T10:18:01.247 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.247 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.247 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.247 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.591815+0000", 2024-09-21T10:18:01.247 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0060383640000000001 2024-09-21T10:18:01.247 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.248 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.248 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:18:01.248 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.591838+0000", 2024-09-21T10:18:01.248 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.2725999999999999e-05 2024-09-21T10:18:01.248 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.248 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.248 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:18:01.248 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.630151+0000", 2024-09-21T10:18:01.248 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.038313543999999998 2024-09-21T10:18:01.248 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.248 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.248 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:18:01.248 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.630194+0000", 2024-09-21T10:18:01.248 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2932999999999997e-05 2024-09-21T10:18:01.249 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.249 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.249 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.249 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.249 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.249 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:108070 11.5 e272/267 11:a28615af:::benchmark_data_smithi107_218812_object108069:head v 272'6484, mlcod=257'549)", 2024-09-21T10:18:01.249 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:11:17.662139+0000", 2024-09-21T10:18:01.249 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 403.52773982500003, 2024-09-21T10:18:01.249 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.042853980999999999, 2024-09-21T10:18:01.249 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:18:01.249 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:01.249 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:18:01.249 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.249 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:18:01.250 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.662139+0000", 2024-09-21T10:18:01.250 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.250 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.250 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.250 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:18:01.250 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.662139+0000", 2024-09-21T10:18:01.250 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.250 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.250 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.250 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:18:01.250 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.662138+0000", 2024-09-21T10:18:01.250 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:18:01.250 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.251 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.251 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:18:01.251 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.662163+0000", 2024-09-21T10:18:01.251 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.4703e-05 2024-09-21T10:18:01.251 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.251 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.251 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:18:01.251 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.662178+0000", 2024-09-21T10:18:01.251 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5407e-05 2024-09-21T10:18:01.251 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.251 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.251 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:18:01.251 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.662184+0000", 2024-09-21T10:18:01.251 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7420000000000003e-06 2024-09-21T10:18:01.251 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.252 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.252 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.252 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.662256+0000", 2024-09-21T10:18:01.252 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2564000000000005e-05 2024-09-21T10:18:01.252 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.252 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.252 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:18:01.252 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.662276+0000", 2024-09-21T10:18:01.252 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.9579999999999999e-05 2024-09-21T10:18:01.252 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.252 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.252 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:18:01.252 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.704954+0000", 2024-09-21T10:18:01.252 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.042678649999999999 2024-09-21T10:18:01.253 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.253 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.253 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:18:01.253 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:17.704993+0000", 2024-09-21T10:18:01.253 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8804000000000003e-05 2024-09-21T10:18:01.253 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.253 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.253 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.253 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.253 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.253 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36561.0:128494 11.5 e275/267 11:a159820e:::benchmark_data_smithi107_218812_object16007:head v 275'7663, mlcod=258'1538)", 2024-09-21T10:18:01.253 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:11:41.955295+0000", 2024-09-21T10:18:01.253 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 379.23458425000001, 2024-09-21T10:18:01.253 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.046634125999999998, 2024-09-21T10:18:01.253 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:18:01.254 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:01.254 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:18:01.254 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.254 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:18:01.254 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:41.955295+0000", 2024-09-21T10:18:01.254 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.254 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.254 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.254 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:18:01.254 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:41.955295+0000", 2024-09-21T10:18:01.254 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.254 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.254 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.255 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:18:01.255 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:41.955294+0000", 2024-09-21T10:18:01.255 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:18:01.255 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.255 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.255 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:18:01.255 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:41.955301+0000", 2024-09-21T10:18:01.255 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.1069999999999999e-06 2024-09-21T10:18:01.255 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.255 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.255 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:18:01.255 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:41.955315+0000", 2024-09-21T10:18:01.255 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3594000000000001e-05 2024-09-21T10:18:01.256 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.256 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.256 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:18:01.256 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:41.955320+0000", 2024-09-21T10:18:01.256 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3159999999999996e-06 2024-09-21T10:18:01.256 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.256 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.256 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.256 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:41.958211+0000", 2024-09-21T10:18:01.256 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0028911850000000001 2024-09-21T10:18:01.256 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.256 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.256 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:18:01.257 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:41.958239+0000", 2024-09-21T10:18:01.257 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7653000000000001e-05 2024-09-21T10:18:01.257 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.257 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.257 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:18:01.257 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:42.001875+0000", 2024-09-21T10:18:01.257 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.043635707000000003 2024-09-21T10:18:01.257 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.257 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.257 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:18:01.257 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:11:42.001929+0000", 2024-09-21T10:18:01.257 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4194000000000003e-05 2024-09-21T10:18:01.257 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.257 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.257 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.258 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.258 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.258 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36822.0:4338 12.f e300/294 12:f455ab21:::benchmark_data_smithi107_250110_object4337:head v 300'282, mlcod=300'280)", 2024-09-21T10:18:01.258 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:14:11.149439+0000", 2024-09-21T10:18:01.258 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 230.040440187, 2024-09-21T10:18:01.258 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.032672068999999998, 2024-09-21T10:18:01.258 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:18:01.258 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:01.258 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:18:01.258 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.258 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:18:01.258 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.149439+0000", 2024-09-21T10:18:01.258 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.258 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.259 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.259 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:18:01.259 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.149439+0000", 2024-09-21T10:18:01.259 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.259 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.259 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.259 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:18:01.259 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.149438+0000", 2024-09-21T10:18:01.259 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:18:01.259 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.259 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.259 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:18:01.259 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.149465+0000", 2024-09-21T10:18:01.260 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.6706000000000001e-05 2024-09-21T10:18:01.260 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.260 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.260 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:18:01.260 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.149480+0000", 2024-09-21T10:18:01.260 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4990999999999999e-05 2024-09-21T10:18:01.260 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.260 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.260 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:18:01.260 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.149487+0000", 2024-09-21T10:18:01.260 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2180000000000002e-06 2024-09-21T10:18:01.260 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.260 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.260 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.261 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.149527+0000", 2024-09-21T10:18:01.261 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0339999999999997e-05 2024-09-21T10:18:01.261 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.261 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.261 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:18:01.261 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.149549+0000", 2024-09-21T10:18:01.261 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1171000000000001e-05 2024-09-21T10:18:01.261 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.261 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.261 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:18:01.261 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.150213+0000", 2024-09-21T10:18:01.261 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00066409599999999998 2024-09-21T10:18:01.261 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.261 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.261 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:18:01.262 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.182111+0000", 2024-09-21T10:18:01.262 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.031898414999999999 2024-09-21T10:18:01.262 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.262 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.262 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.262 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.262 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.262 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36822.0:4334 12.c e300/294 12:35058bc8:::benchmark_data_smithi107_250110_object4333:head v 300'301, mlcod=300'299)", 2024-09-21T10:18:01.262 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:14:11.149955+0000", 2024-09-21T10:18:01.262 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 230.03992457699999, 2024-09-21T10:18:01.262 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.033191888000000003, 2024-09-21T10:18:01.262 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:18:01.262 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:01.263 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:18:01.263 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.263 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:18:01.263 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.149955+0000", 2024-09-21T10:18:01.263 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.263 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.263 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.263 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:18:01.263 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.149955+0000", 2024-09-21T10:18:01.263 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.263 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.263 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.263 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:18:01.263 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.149954+0000", 2024-09-21T10:18:01.263 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:18:01.264 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.264 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.264 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:18:01.264 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.149987+0000", 2024-09-21T10:18:01.264 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2965e-05 2024-09-21T10:18:01.264 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.264 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.264 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:18:01.264 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.149998+0000", 2024-09-21T10:18:01.264 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1105e-05 2024-09-21T10:18:01.264 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.264 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.265 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:18:01.265 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.150004+0000", 2024-09-21T10:18:01.265 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5899999999999998e-06 2024-09-21T10:18:01.265 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.265 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.265 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.265 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.150052+0000", 2024-09-21T10:18:01.265 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8625000000000003e-05 2024-09-21T10:18:01.265 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.265 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.265 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:18:01.265 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.150073+0000", 2024-09-21T10:18:01.265 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1058000000000001e-05 2024-09-21T10:18:01.265 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.265 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.266 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:18:01.266 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.183109+0000", 2024-09-21T10:18:01.266 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.033035454999999998 2024-09-21T10:18:01.266 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.266 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.266 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:18:01.266 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.183146+0000", 2024-09-21T10:18:01.266 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7719000000000003e-05 2024-09-21T10:18:01.266 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.266 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.266 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.266 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.266 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.266 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36822.0:4339 12.c e300/294 12:38f253b7:::benchmark_data_smithi107_250110_object4338:head v 300'302, mlcod=300'299)", 2024-09-21T10:18:01.267 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:14:11.150860+0000", 2024-09-21T10:18:01.267 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 230.039019408, 2024-09-21T10:18:01.267 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.032544337, 2024-09-21T10:18:01.267 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:18:01.267 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:01.267 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:18:01.267 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.267 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:18:01.267 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.150860+0000", 2024-09-21T10:18:01.267 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.267 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.267 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.267 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:18:01.268 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.150860+0000", 2024-09-21T10:18:01.268 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.268 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.268 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.268 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:18:01.268 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.150859+0000", 2024-09-21T10:18:01.268 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:18:01.268 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.268 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.268 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:18:01.268 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.150910+0000", 2024-09-21T10:18:01.268 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0584999999999998e-05 2024-09-21T10:18:01.268 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.269 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.269 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:18:01.269 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.182130+0000", 2024-09-21T10:18:01.269 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.031220436000000001 2024-09-21T10:18:01.269 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.269 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.269 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:18:01.269 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.182138+0000", 2024-09-21T10:18:01.269 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.3450000000000006e-06 2024-09-21T10:18:01.269 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.269 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.269 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.269 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.182354+0000", 2024-09-21T10:18:01.269 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000215672 2024-09-21T10:18:01.270 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.270 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.270 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:18:01.270 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.182368+0000", 2024-09-21T10:18:01.270 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3764999999999999e-05 2024-09-21T10:18:01.270 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.270 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.270 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:18:01.270 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.183370+0000", 2024-09-21T10:18:01.270 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0010018329999999999 2024-09-21T10:18:01.270 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.270 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.270 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:18:01.271 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.183404+0000", 2024-09-21T10:18:01.271 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4416999999999999e-05 2024-09-21T10:18:01.271 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.271 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.271 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.271 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.271 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.271 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36822.0:4342 12.f e300/294 12:fe58baaf:::benchmark_data_smithi107_250110_object4341:head v 300'283, mlcod=300'280)", 2024-09-21T10:18:01.271 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:14:11.151868+0000", 2024-09-21T10:18:01.271 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 230.03801163399999, 2024-09-21T10:18:01.271 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.031737911000000001, 2024-09-21T10:18:01.271 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:18:01.271 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:01.271 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:18:01.272 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.272 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:18:01.272 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.151868+0000", 2024-09-21T10:18:01.272 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.272 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.272 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.272 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:18:01.272 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.151868+0000", 2024-09-21T10:18:01.272 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.272 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.272 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.272 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:18:01.272 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.151867+0000", 2024-09-21T10:18:01.272 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:18:01.272 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.273 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.273 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:18:01.273 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.151922+0000", 2024-09-21T10:18:01.273 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5648e-05 2024-09-21T10:18:01.273 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.273 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.273 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:18:01.273 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.182452+0000", 2024-09-21T10:18:01.273 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.030529318999999999 2024-09-21T10:18:01.273 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.273 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.273 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:18:01.273 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.182461+0000", 2024-09-21T10:18:01.274 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.5000000000000005e-06 2024-09-21T10:18:01.274 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.274 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.274 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.274 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.182500+0000", 2024-09-21T10:18:01.274 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8896000000000002e-05 2024-09-21T10:18:01.274 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.274 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.274 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:18:01.274 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.182521+0000", 2024-09-21T10:18:01.274 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.0852e-05 2024-09-21T10:18:01.274 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.275 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.275 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:18:01.275 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.183579+0000", 2024-09-21T10:18:01.275 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001057466 2024-09-21T10:18:01.275 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.275 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.275 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:18:01.275 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:11.183605+0000", 2024-09-21T10:18:01.275 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.6913000000000001e-05 2024-09-21T10:18:01.275 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.275 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.275 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.275 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.276 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.276 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36822.0:12995 12.f e301/294 12:f87583a4:::benchmark_data_smithi107_250110_object12994:head v 301'819, mlcod=301'817)", 2024-09-21T10:18:01.276 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:14:14.288267+0000", 2024-09-21T10:18:01.276 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 226.90161223999999, 2024-09-21T10:18:01.276 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.22758118099999999, 2024-09-21T10:18:01.276 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:18:01.276 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:01.276 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:18:01.276 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.276 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:18:01.276 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.288267+0000", 2024-09-21T10:18:01.276 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.276 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.277 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.277 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:18:01.277 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.288267+0000", 2024-09-21T10:18:01.277 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.277 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.277 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.277 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:18:01.277 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.288265+0000", 2024-09-21T10:18:01.277 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:18:01.277 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.277 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.277 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:18:01.277 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.288294+0000", 2024-09-21T10:18:01.277 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.8861999999999999e-05 2024-09-21T10:18:01.278 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.278 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.278 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:18:01.278 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.288310+0000", 2024-09-21T10:18:01.278 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.6042999999999999e-05 2024-09-21T10:18:01.278 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.278 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.278 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:18:01.278 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.288318+0000", 2024-09-21T10:18:01.278 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.4150000000000002e-06 2024-09-21T10:18:01.278 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.278 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.278 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.279 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.288356+0000", 2024-09-21T10:18:01.279 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8232000000000003e-05 2024-09-21T10:18:01.279 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.279 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.279 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:18:01.279 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.288375+0000", 2024-09-21T10:18:01.279 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.8760999999999999e-05 2024-09-21T10:18:01.279 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.279 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.279 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:18:01.279 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.289050+0000", 2024-09-21T10:18:01.279 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00067560700000000005 2024-09-21T10:18:01.279 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.279 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.280 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:18:01.280 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.515848+0000", 2024-09-21T10:18:01.280 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.22679776600000001 2024-09-21T10:18:01.280 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.280 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.280 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.280 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.280 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.280 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36822.0:13004 12.c e301/294 12:35430c8b:::benchmark_data_smithi107_250110_object13003:head v 301'815, mlcod=301'813)", 2024-09-21T10:18:01.280 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:14:14.293024+0000", 2024-09-21T10:18:01.280 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 226.89685545500001, 2024-09-21T10:18:01.280 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.22461303099999999, 2024-09-21T10:18:01.280 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:18:01.280 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:01.281 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:18:01.281 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.281 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:18:01.281 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.293024+0000", 2024-09-21T10:18:01.281 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.281 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.281 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.281 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:18:01.281 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.293024+0000", 2024-09-21T10:18:01.281 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.281 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.281 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.281 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:18:01.282 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.293023+0000", 2024-09-21T10:18:01.282 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:18:01.282 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.282 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.282 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:18:01.282 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.293083+0000", 2024-09-21T10:18:01.282 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.071e-05 2024-09-21T10:18:01.282 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.282 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.282 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:18:01.282 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.515818+0000", 2024-09-21T10:18:01.282 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.222734617 2024-09-21T10:18:01.282 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.282 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.283 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:18:01.283 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.515828+0000", 2024-09-21T10:18:01.283 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.039e-05 2024-09-21T10:18:01.283 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.283 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.283 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.283 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.515878+0000", 2024-09-21T10:18:01.283 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9342000000000003e-05 2024-09-21T10:18:01.283 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.283 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.283 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:18:01.283 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.515900+0000", 2024-09-21T10:18:01.283 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.2560000000000001e-05 2024-09-21T10:18:01.284 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.284 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.284 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:18:01.284 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.517611+0000", 2024-09-21T10:18:01.284 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001710437 2024-09-21T10:18:01.284 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.284 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.284 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:18:01.284 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.517637+0000", 2024-09-21T10:18:01.284 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.6129e-05 2024-09-21T10:18:01.284 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.284 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.284 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.284 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.284 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.285 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36822.0:13016 12.f e301/294 12:fb99eecf:::benchmark_data_smithi107_250110_object13015:head v 301'820, mlcod=301'817)", 2024-09-21T10:18:01.285 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:14:14.297940+0000", 2024-09-21T10:18:01.285 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 226.89193912600001, 2024-09-21T10:18:01.285 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.219761862, 2024-09-21T10:18:01.285 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:18:01.285 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:01.285 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:18:01.285 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.285 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:18:01.285 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.297940+0000", 2024-09-21T10:18:01.285 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.285 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.285 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.285 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:18:01.286 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.297940+0000", 2024-09-21T10:18:01.286 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.286 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.286 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.286 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:18:01.286 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.297939+0000", 2024-09-21T10:18:01.286 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:18:01.286 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.286 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.286 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:18:01.286 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.297968+0000", 2024-09-21T10:18:01.286 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.898e-05 2024-09-21T10:18:01.286 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.286 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.287 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:18:01.287 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.515832+0000", 2024-09-21T10:18:01.287 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21786397299999999 2024-09-21T10:18:01.287 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.287 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.287 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:18:01.287 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.515840+0000", 2024-09-21T10:18:01.287 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.7139999999999993e-06 2024-09-21T10:18:01.287 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.287 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.287 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.287 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.515881+0000", 2024-09-21T10:18:01.287 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1795999999999998e-05 2024-09-21T10:18:01.287 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.288 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.288 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:18:01.288 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.515903+0000", 2024-09-21T10:18:01.288 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1202000000000001e-05 2024-09-21T10:18:01.288 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.288 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.288 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:18:01.288 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.517662+0000", 2024-09-21T10:18:01.288 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0017596689999999999 2024-09-21T10:18:01.288 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.288 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.288 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:18:01.288 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:14.517702+0000", 2024-09-21T10:18:01.288 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.9546999999999999e-05 2024-09-21T10:18:01.288 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.289 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.289 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.289 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.289 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.289 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36822.0:181455 12.f e355/294 12:f55fe4b6:::benchmark_data_smithi107_250110_object70045:head v 355'11455, mlcod=355'11452)", 2024-09-21T10:18:01.289 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:17:12.081403+0000", 2024-09-21T10:18:01.289 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 49.108475687999999, 2024-09-21T10:18:01.289 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.016062258999999999, 2024-09-21T10:18:01.289 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:18:01.289 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:01.289 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:18:01.289 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.289 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:18:01.289 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:12.081403+0000", 2024-09-21T10:18:01.290 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.290 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.290 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.290 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:18:01.290 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:12.081403+0000", 2024-09-21T10:18:01.290 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.290 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.290 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.290 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:18:01.290 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:12.081402+0000", 2024-09-21T10:18:01.290 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:18:01.290 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.290 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.290 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:18:01.291 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:12.081416+0000", 2024-09-21T10:18:01.291 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3478999999999999e-05 2024-09-21T10:18:01.291 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.291 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.291 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:18:01.291 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:12.081428+0000", 2024-09-21T10:18:01.291 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2427e-05 2024-09-21T10:18:01.291 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.291 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.291 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:18:01.291 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:12.081436+0000", 2024-09-21T10:18:01.291 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2740000000000002e-06 2024-09-21T10:18:01.291 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.292 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.292 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.292 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:12.082563+0000", 2024-09-21T10:18:01.292 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001127255 2024-09-21T10:18:01.292 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.292 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.292 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:18:01.292 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:12.082584+0000", 2024-09-21T10:18:01.292 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.0999999999999999e-05 2024-09-21T10:18:01.292 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.292 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.292 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:18:01.292 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:12.097335+0000", 2024-09-21T10:18:01.292 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.01475161 2024-09-21T10:18:01.292 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.293 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.293 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:18:01.293 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:12.097466+0000", 2024-09-21T10:18:01.293 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013027700000000001 2024-09-21T10:18:01.293 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.293 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.293 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.293 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.293 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.293 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.36822.0:214526 12.c e361/294 12:394d6a32:::benchmark_data_smithi107_250110_object103116:head v 361'13150, mlcod=361'13148)", 2024-09-21T10:18:01.293 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:17:43.283471+0000", 2024-09-21T10:18:01.293 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 17.906407717, 2024-09-21T10:18:01.293 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.015343967999999999, 2024-09-21T10:18:01.294 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:18:01.294 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:01.294 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:18:01.294 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.294 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:18:01.294 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:43.283471+0000", 2024-09-21T10:18:01.294 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.294 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.294 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.294 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:18:01.294 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:43.283471+0000", 2024-09-21T10:18:01.294 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:18:01.294 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.295 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.295 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:18:01.295 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:43.283471+0000", 2024-09-21T10:18:01.295 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967296 2024-09-21T10:18:01.295 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.295 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.295 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:18:01.295 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:43.283475+0000", 2024-09-21T10:18:01.295 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7290000000000002e-06 2024-09-21T10:18:01.295 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.295 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.295 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:18:01.295 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:43.283480+0000", 2024-09-21T10:18:01.296 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9479999999999999e-06 2024-09-21T10:18:01.296 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.296 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.296 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:18:01.296 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:43.283483+0000", 2024-09-21T10:18:01.296 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1020000000000001e-06 2024-09-21T10:18:01.296 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.296 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.296 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:18:01.296 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:43.283537+0000", 2024-09-21T10:18:01.296 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3897999999999998e-05 2024-09-21T10:18:01.296 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.296 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.297 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:18:01.297 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:43.283555+0000", 2024-09-21T10:18:01.297 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.8215000000000001e-05 2024-09-21T10:18:01.297 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.297 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.297 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:18:01.297 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:43.298655+0000", 2024-09-21T10:18:01.297 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.015099585 2024-09-21T10:18:01.297 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:18:01.297 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:18:01.297 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:18:01.297 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:43.298815+0000", 2024-09-21T10:18:01.297 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000160729 2024-09-21T10:18:01.297 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.298 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.298 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.298 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:18:01.298 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:18:01.298 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:18:01.319 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:01.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:01.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:01.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:01.442 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:01.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:01.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:01.543 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:01.564 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:01.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:01.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:01.666 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:01.687 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:01.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:01.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:01.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:18:01.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:01.812 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:01.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:01.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:01.914 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:01.936 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:01.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:01.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:02.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:02.059 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:02.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:02.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:02.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:18:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:18:01] "GET /metrics HTTP/1.1" 200 36787 "" "Prometheus/2.51.0" 2024-09-21T10:18:02.152 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:18:02.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:02.216 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:02.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:02.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:02.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:02.340 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:02.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:02.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:02.397 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:18:02.442 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:02.463 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:02.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:02.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:02.564 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:02.585 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:02.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:02.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:02.602 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_7' removed 2024-09-21T10:18:02.686 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:02.708 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:02.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:02.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:02.801 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:02 smithi129 ceph-mon[88471]: pgmap v1239: 119 pgs: 119 active+clean; 2.3 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 23 op/s 2024-09-21T10:18:02.801 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:02 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3852265693' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:18:02.801 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:02 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:18:02.810 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:02.834 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:02.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:02.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:02 smithi107 ceph-mon[102221]: pgmap v1239: 119 pgs: 119 active+clean; 2.3 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 23 op/s 2024-09-21T10:18:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:02 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3852265693' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:18:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:02 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:18:02.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:02 smithi107 ceph-mon[105022]: pgmap v1239: 119 pgs: 119 active+clean; 2.3 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 23 op/s 2024-09-21T10:18:02.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:02 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3852265693' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:18:02.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:02 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:18:02.936 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:02.957 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:02.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:02.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:02.996 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:03.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:03.079 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:03.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:03.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:03.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:03.210 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:03.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:03.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:03.312 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:03.333 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:03.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:03.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:03.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:03.457 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:03.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:03.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:03.558 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:03.581 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:03.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:03.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:03.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:03.704 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:03.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:03.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:03.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:03.822 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:03 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:18:03.822 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:03 smithi129 ceph-mon[88471]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T10:18:03.826 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:03.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:03.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:03.850 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:03 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:18:03.850 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:03 smithi107 ceph-mon[102221]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T10:18:03.850 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:03 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:18:03.850 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:03 smithi107 ceph-mon[105022]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T10:18:03.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:03.948 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:03.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:03.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:04.050 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:04.070 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:18:04.070 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:18:04.070 INFO:teuthology.orchestra.run.smithi129.stdout: "num_ops": 0 2024-09-21T10:18:04.070 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:18:04.073 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:04.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:04.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:04.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:04.196 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:04.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:04.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:04.297 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:04.319 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:04.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:04.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:04.421 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:04.443 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:04.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:04.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:04.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:04.567 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:04.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:04.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:04.612 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T10:18:04.613 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:18:04.668 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:04 smithi129 ceph-mon[88471]: pgmap v1241: 93 pgs: 93 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:04 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/905408324' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:18:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:04 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:18:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:04 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4276377815' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:18:04.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:04 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:18:04.690 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:04.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:04.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:04.722 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:18:04.723 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:18:04.723 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:18:04.723 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_8 2024-09-21T10:18:04.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_8 16 2024-09-21T10:18:04.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:04.813 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:04.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:04.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:04 smithi107 ceph-mon[102221]: pgmap v1241: 93 pgs: 93 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:04 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/905408324' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:18:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:04 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:18:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:04 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4276377815' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:18:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:04 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:18:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:04 smithi107 ceph-mon[105022]: pgmap v1241: 93 pgs: 93 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:04 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/905408324' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:18:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:04 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:18:04.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:04 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4276377815' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:18:04.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:04 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:18:04.915 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:04.937 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:04.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:04.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:05.008 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:18:05.039 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:05.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:05.062 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:05.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:05.162 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:05.184 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:05.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:05.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:05.285 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:05.307 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:05.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:05.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:05.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:05.431 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:05.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:05.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:05.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T10:18:05.532 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:05.554 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:05.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:05.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:05.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:05.676 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:05.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:05.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:05.778 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:05.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:05.800 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:05.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:05.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:05 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:18:05.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:05 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:18:05.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:05 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:18:05.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:05 smithi107 ceph-mon[102221]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T10:18:05.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:05 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:18:05.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:05 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:18:05.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:05 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:18:05.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:05 smithi107 ceph-mon[105022]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T10:18:05.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:05.911 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:05 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:18:05.911 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:05 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:18:05.911 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:05 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:18:05.911 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:05 smithi129 ceph-mon[88471]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T10:18:05.922 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:05.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:05.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:06.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:06.045 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:06.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:06.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:06.146 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:06.168 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:06.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:06.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:06.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:06.291 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:06.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:06.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:06.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:06.413 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:06.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:06.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:06.514 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:06.535 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:06.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:06.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:06.637 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:06.658 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:06.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:06.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:06.715 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:18:06.715 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:18:06.715 INFO:teuthology.orchestra.run.smithi129.stdout: "complaint_time": 30, 2024-09-21T10:18:06.715 INFO:teuthology.orchestra.run.smithi129.stdout: "num_blocked_ops": 0 2024-09-21T10:18:06.715 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:18:06.760 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:06.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:06.782 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:06.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:06.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:06 smithi107 ceph-mon[105022]: pgmap v1243: 93 pgs: 93 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:06.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:06 smithi107 ceph-mon[102221]: pgmap v1243: 93 pgs: 93 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:06.882 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:06.903 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:06.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:06.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:06 smithi129 ceph-mon[88471]: pgmap v1243: 93 pgs: 93 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:07.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:07.026 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:07.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:07.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:07.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:07.150 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:07.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:07.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:07.252 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:07.275 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:07.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:07.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:07.287 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:18:07.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:07.382 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:07.382 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:07.395 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:18:07.395 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:18:07.401 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:07.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:07.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:07.422 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:18:07.422 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:18:07.438 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:18:07.438 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:18:07.454 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:18:07.454 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:18:07.469 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:18:07.469 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:18:07.485 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:18:07.485 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:18:07.500 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:18:07.500 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:18:07.503 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:07.518 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:18:07.518 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:18:07.526 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:07.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:07.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:07.576 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:18:07.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:07.650 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:07.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:07.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:07.751 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:07.772 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:07.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:07.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:07.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:07.896 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:07.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:07.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:07 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3896156930' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:18:07.963 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:07 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3896156930' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:18:07.963 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:07 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3896156930' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:18:07.997 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:08.018 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:08.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:08.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:08.120 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:08.141 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:08.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:08.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:08.242 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:08.265 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:08.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:08.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:08.367 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:08.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:18:08.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:08.389 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:08.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:08.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:08.512 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:08.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:08.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:08.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:08.634 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:08.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:08.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:08.677 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:08 smithi129 ceph-mon[88471]: pgmap v1244: 93 pgs: 93 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:08.736 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:08.758 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:08.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:08.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:08.859 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:08.881 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:08.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:08.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:08.982 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:09.004 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:09.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:09.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:09.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:08 smithi107 ceph-mon[102221]: pgmap v1244: 93 pgs: 93 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:09.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:08 smithi107 ceph-mon[105022]: pgmap v1244: 93 pgs: 93 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:09.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:09.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:09.128 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:09.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:09.166 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:18:09.166 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:18:09.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:09.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:09.251 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:09.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:09.353 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:09.375 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:09.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:09.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:09.476 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:09.498 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:09.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:09.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:09.599 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:09.621 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:09.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:09.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:09.722 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:09.744 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:09.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:09.759 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:18:09.846 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:09.868 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:09.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:09.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:09.970 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:09.992 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:09.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:09.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:10.045 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:18:10.094 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:10.115 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:10.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:10.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:10.216 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:10.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:10.239 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:10.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:10.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:10.362 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:10.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:10.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:18:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:18:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:18:10.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:10.485 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:10.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:10.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:10.587 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:10.608 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:10.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:10.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:10.710 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:10.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:10.732 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:10.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:10.833 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:10.854 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:10.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:10.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:10.883 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:10 smithi107 ceph-mon[102221]: pgmap v1245: 93 pgs: 93 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:10.883 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:10 smithi107 ceph-mon[105022]: pgmap v1245: 93 pgs: 93 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:10 smithi129 ceph-mon[88471]: pgmap v1245: 93 pgs: 93 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:10.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:10.977 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:10.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:10.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:11.079 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:11.102 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:11.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:11.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:11.203 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:11.226 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:11.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:11.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:11.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:11.349 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:11.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:11.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:11.442 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:11.451 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:11.472 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:11.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:11.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:11.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:11.637 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_8' created 2024-09-21T10:18:11.637 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T10:18:11.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:11 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/358234879' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:18:11.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:11 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:18:11.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:11.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:11.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:11 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2106423143' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16} : dispatch 2024-09-21T10:18:11.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:11 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16} : dispatch 2024-09-21T10:18:11.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:11 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/358234879' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:18:11.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:11 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:18:11.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:11.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:11.648 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:11 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2106423143' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16} : dispatch 2024-09-21T10:18:11.649 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:11 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16} : dispatch 2024-09-21T10:18:11.659 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:11.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:11.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:11.727 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:18:11.727 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:18:11.761 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:11.785 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:11.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:11.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:11.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:11.908 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:11.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:11.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:11 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/358234879' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:18:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:11 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:18:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:11 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2106423143' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16} : dispatch 2024-09-21T10:18:11.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:11 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16} : dispatch 2024-09-21T10:18:11.966 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:18:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:18:11] "GET /metrics HTTP/1.1" 200 34393 "" "Prometheus/2.51.0" 2024-09-21T10:18:12.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:12.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:12.032 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:12.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:12.133 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:12.154 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:12.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:12.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:12.256 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:12.277 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:12.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:12.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:12.379 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:12.395 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:18:12.400 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:12.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:12.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:12.501 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:12.522 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:12.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:12.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:12.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:12.632 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:12 smithi129 ceph-mon[88471]: pgmap v1246: 93 pgs: 93 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:12.632 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:12 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:18:12.632 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:12 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16}]': finished 2024-09-21T10:18:12.655 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:12.655 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:18:12.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:12.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:12.756 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:12.780 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:12.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:12.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:12.881 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:12.902 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:12.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:12.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:12.904 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:12 smithi129 ceph-mon[88471]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T10:18:12.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:18:12.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:12 smithi107 ceph-mon[102221]: pgmap v1246: 93 pgs: 93 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:12.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:12 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:18:12.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:12 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16}]': finished 2024-09-21T10:18:12.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:12 smithi107 ceph-mon[102221]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T10:18:12.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:12 smithi107 ceph-mon[105022]: pgmap v1246: 93 pgs: 93 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:12.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:12 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:18:12.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:12 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_8", "pg_num": 16}]': finished 2024-09-21T10:18:12.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:12 smithi107 ceph-mon[105022]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T10:18:13.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:13.026 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:13.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:13.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:13.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_8 rados --yes-i-really-mean-it || true 2024-09-21T10:18:13.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:13.148 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:13.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:13.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:13.251 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:13.272 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:13.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:13.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:13.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:13.395 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:13.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:13.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:13.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:13.518 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:13.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:13.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:13.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:13.643 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:13.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:13.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:13.745 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:13.768 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:13.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:13.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:13.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:13 smithi107 ceph-mon[102221]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T10:18:13.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:13 smithi107 ceph-mon[105022]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T10:18:13.869 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:13.887 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T10:18:13.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:18:13.891 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:13.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:13.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:13.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:13 smithi129 ceph-mon[88471]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T10:18:13.926 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:13.993 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:14.014 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:14.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:14.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:14.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:14.137 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:14.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:14.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:14.198 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:14.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:14.260 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:14.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:14.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:14.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:14.383 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:14.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:14.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:14.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:14.493 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:18:14.494 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:18:14.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:14.507 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:14.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:14.609 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:14.632 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:14.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:14.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:14.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:14.756 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:14.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:14.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:14.857 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:14 smithi107 ceph-mon[102221]: pgmap v1249: 109 pgs: 13 creating+peering, 8 peering, 3 unknown, 85 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:14 smithi107 ceph-mon[102221]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:18:14.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:14 smithi107 ceph-mon[102221]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T10:18:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:14 smithi107 ceph-mon[105022]: pgmap v1249: 109 pgs: 13 creating+peering, 8 peering, 3 unknown, 85 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:14 smithi107 ceph-mon[105022]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:18:14.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:14 smithi107 ceph-mon[105022]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T10:18:14.879 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:14.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:14.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:14 smithi129 ceph-mon[88471]: pgmap v1249: 109 pgs: 13 creating+peering, 8 peering, 3 unknown, 85 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:14 smithi129 ceph-mon[88471]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:18:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:14 smithi129 ceph-mon[88471]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T10:18:14.980 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:15.002 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:15.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:15.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:15.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:15.125 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:15.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:15.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:15.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:18:15.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:15.248 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:15.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:15.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:15.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:15.371 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:15.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:15.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:15.401 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:18:15.472 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:15.493 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:15.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:15.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:15.595 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:15.617 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:15.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:15.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:15.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:15.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:15.740 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:15.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:15.841 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:15.863 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:15.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:15.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:15.964 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:15.986 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:15.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:15.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:16.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:16.109 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:16.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:16.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:16.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:16.232 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:16.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:16.333 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:16.355 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:16.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:16.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:16.457 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:16.479 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:16.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:16.580 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:16.602 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:16.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:16.645 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:18:16.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:16.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:16.726 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:16.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:16.813 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:16 smithi129 ceph-mon[88471]: pgmap v1251: 109 pgs: 13 creating+peering, 8 peering, 1 unknown, 87 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:16.814 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:16 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1274566901' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:18:16.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:16.849 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:16.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:16.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:16.909 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:16 smithi107 ceph-mon[105022]: pgmap v1251: 109 pgs: 13 creating+peering, 8 peering, 1 unknown, 87 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:16.909 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:16 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1274566901' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:18:16.909 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:16 smithi107 ceph-mon[102221]: pgmap v1251: 109 pgs: 13 creating+peering, 8 peering, 1 unknown, 87 active+clean; 634 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:18:16.909 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:16 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1274566901' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:18:16.922 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:18:16.922 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:18:16.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:16.973 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:16.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:16.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:17.074 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:17.096 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:17.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:17.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:17.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:17.220 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:17.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:17.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:17.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:17.342 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:17.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:17.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:17.444 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:17.468 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:17.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:17.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:17.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:18:17.570 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:17.591 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:17.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:17.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:17.693 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:17.714 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:17.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:17.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:17.817 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:17.839 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:17.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:17.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:17.841 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:17 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:18:17.841 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:17 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1274566901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:18:17.841 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:17 smithi107 ceph-mon[102221]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T10:18:17.841 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:17 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:18:17.841 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:17 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1274566901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:18:17.841 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:17 smithi107 ceph-mon[105022]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T10:18:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:17 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:18:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:17 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1274566901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:18:17.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:17 smithi129 ceph-mon[88471]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T10:18:17.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:17.949 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:18:17.949 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:18:17.954 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:18:17.954 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:18:17.955 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:17.963 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:17.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:17.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:17.978 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:18:17.978 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:18:17.996 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:18:17.996 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:18:18.012 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:18:18.012 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:18:18.027 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:18:18.027 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:18:18.043 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:18:18.043 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:18:18.058 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:18:18.058 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:18:18.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:18.086 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:18.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:18.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:18.188 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:18.213 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:18.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:18.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:18.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:18.335 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:18.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:18.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:18.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:18.458 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:18.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:18.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:18.560 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:18.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:18.587 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:18.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:18.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:18.711 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:18.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:18.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:18.812 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:18.838 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:18.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:18.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:18.898 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:18 smithi107 ceph-mon[102221]: pgmap v1253: 109 pgs: 2 creating+peering, 1 peering, 106 active+clean; 661 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 5.9 KiB/s rd, 13 KiB/s wr, 3 op/s; 9.0 KiB/s, 0 objects/s recovering 2024-09-21T10:18:18.899 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:18 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3288274798' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:18:18.899 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:18 smithi107 ceph-mon[105022]: pgmap v1253: 109 pgs: 2 creating+peering, 1 peering, 106 active+clean; 661 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 5.9 KiB/s rd, 13 KiB/s wr, 3 op/s; 9.0 KiB/s, 0 objects/s recovering 2024-09-21T10:18:18.899 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:18 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3288274798' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:18:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:18 smithi129 ceph-mon[88471]: pgmap v1253: 109 pgs: 2 creating+peering, 1 peering, 106 active+clean; 661 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 5.9 KiB/s rd, 13 KiB/s wr, 3 op/s; 9.0 KiB/s, 0 objects/s recovering 2024-09-21T10:18:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:18 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3288274798' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:18:18.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:18.961 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:18.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:18.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:19.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:19.084 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:19.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:19.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:19.186 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:19.212 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:19.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:19.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:19.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:19.336 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:19.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:19.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:19.437 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:19.463 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:19.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:19.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:19.565 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:19.587 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:19.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:19.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:19.654 INFO:teuthology.orchestra.run.smithi107.stderr:enabled application 'rados' on pool 'unique_pool_8' 2024-09-21T10:18:19.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:19.709 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:19.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:19.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:19.749 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:18:19.773 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:19 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/762861184' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:18:19.773 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:19 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/762861184' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:18:19.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:19.834 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:19.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:19.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:19 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/762861184' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:18:19.936 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:19.962 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:19.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:19.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:20.064 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:20.089 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:20.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:20.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:20.191 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:20.216 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:20.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:20.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:20.317 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:20.339 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:20.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:20.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:20.462 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:20.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:20.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:20.564 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:20.589 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:20.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:20.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:20.620 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:20.655 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:18:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:18:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:18:20.691 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:20.716 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:20.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:20.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:20.818 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:20.843 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:20.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:20.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:20.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:20 smithi107 ceph-mon[105022]: pgmap v1254: 109 pgs: 109 active+clean; 661 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 5.0 KiB/s rd, 11 KiB/s wr, 3 op/s; 7.7 KiB/s, 0 objects/s recovering 2024-09-21T10:18:20.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:20 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/762861184' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:18:20.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:20 smithi107 ceph-mon[105022]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T10:18:20.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:20 smithi107 ceph-mon[102221]: pgmap v1254: 109 pgs: 109 active+clean; 661 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 5.0 KiB/s rd, 11 KiB/s wr, 3 op/s; 7.7 KiB/s, 0 objects/s recovering 2024-09-21T10:18:20.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:20 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/762861184' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:18:20.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:20 smithi107 ceph-mon[102221]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T10:18:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:20 smithi129 ceph-mon[88471]: pgmap v1254: 109 pgs: 109 active+clean; 661 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 5.0 KiB/s rd, 11 KiB/s wr, 3 op/s; 7.7 KiB/s, 0 objects/s recovering 2024-09-21T10:18:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:20 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/762861184' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_8", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:18:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:20 smithi129 ceph-mon[88471]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T10:18:20.944 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:20.970 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:20.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:20.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:21.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:21.085 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:21.097 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:21.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:21.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:21.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:21.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:21.221 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:21.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:21.322 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:21.343 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:21.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:21.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:21.445 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:21.470 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:21.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:21.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:21.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:21.594 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:21.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:21.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:21.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:21.719 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:21.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:21.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:21.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:21.843 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:21.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:21.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:21.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:18:21.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:21 smithi107 ceph-mon[102221]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:18:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:21 smithi107 ceph-mon[105022]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:18:21.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:18:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:18:21] "GET /metrics HTTP/1.1" 200 34393 "" "Prometheus/2.51.0" 2024-09-21T10:18:21.945 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:21.968 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:21.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:21.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:22.008 DEBUG:teuthology.orchestra.run.smithi107:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_8 bench 90 write' 2024-09-21T10:18:22.011 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:18:22.011 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:18:22.057 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:hints = 1 2024-09-21T10:18:22.057 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:18:22.057 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object prefix: benchmark_data_smithi107_278655 2024-09-21T10:18:22.057 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:18:22.057 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:18:22.071 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:22.111 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:22.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:22.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:22.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:21 smithi129 ceph-mon[88471]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:18:22.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:22.248 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:22.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:22.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:22.350 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:22.388 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:22.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:22.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:22.490 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:22.526 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:22.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:22.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:22.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:22.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:22.668 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:22.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:22.769 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:22.804 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:22.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:22.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:22.846 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:22 smithi107 ceph-mon[102221]: pgmap v1256: 109 pgs: 109 active+clean; 661 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 4.4 KiB/s rd, 9.5 KiB/s wr, 2 op/s; 6.7 KiB/s, 0 objects/s recovering 2024-09-21T10:18:22.846 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:22 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:22.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:22 smithi107 ceph-mon[105022]: pgmap v1256: 109 pgs: 109 active+clean; 661 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 4.4 KiB/s rd, 9.5 KiB/s wr, 2 op/s; 6.7 KiB/s, 0 objects/s recovering 2024-09-21T10:18:22.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:22 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:22.906 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:22 smithi129 ceph-mon[88471]: pgmap v1256: 109 pgs: 109 active+clean; 661 KiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 4.4 KiB/s rd, 9.5 KiB/s wr, 2 op/s; 6.7 KiB/s, 0 objects/s recovering 2024-09-21T10:18:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:22 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:22.937 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:22.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:22.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:23.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:23.057 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1 16 3970 3954 247.112 247.125 0.00805184 0.00402932 2024-09-21T10:18:23.078 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:18:23.079 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:18:23.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:23.080 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:23.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:23.181 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:23.215 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:23.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:23.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:23.317 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:23.352 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:23.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:23.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:23.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:23.485 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:23.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:23.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:23.587 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:23.622 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:23.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:23.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:23.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:23.757 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:23.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:23.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:23.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:23 smithi107 ceph-mon[102221]: from='client.46877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:18:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:23 smithi107 ceph-mon[105022]: from='client.46877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:18:23.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:23.896 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:23.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:23.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:23.997 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:24.031 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:24.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:24.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:24.057 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2 16 7813 7797 243.639 240.188 0.0043476 0.00409761 2024-09-21T10:18:24.133 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:24.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:23 smithi129 ceph-mon[88471]: from='client.46877 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:18:24.170 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:24.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:24.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:24.272 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:24.308 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:24.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:24.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:24.410 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:24.441 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:24.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:24.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:24.543 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:24.578 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:24.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:24.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:24.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:24.720 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:24.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:24.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:24.823 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:24.859 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:24.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:24.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:24.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:24.995 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:24.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:24.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:25.057 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3 16 11277 11261 234.587 216.5 0.00412617 0.00425513 2024-09-21T10:18:25.097 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:25.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:24 smithi107 ceph-mon[105022]: pgmap v1257: 109 pgs: 109 active+clean; 29 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.5 MiB/s wr, 59 op/s; 6.5 KiB/s, 0 objects/s recovering 2024-09-21T10:18:25.111 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:24 smithi107 ceph-mon[102221]: pgmap v1257: 109 pgs: 109 active+clean; 29 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.5 MiB/s wr, 59 op/s; 6.5 KiB/s, 0 objects/s recovering 2024-09-21T10:18:25.139 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:25.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:25.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:25.160 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1256,"stamp":"2024-09-21T10:18:21.427002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":676421,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":40,"num_read_kb":102,"num_write":84,"num_write_kb":254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":698241,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2159,"ondisk_log_size":2159,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":329,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7693792,"kb_used_data":42092,"kb_used_omap":43,"kb_used_meta":7651540,"kb_avail":742267424,"statfs":{"total":767960285184,"available":760081842176,"internally_reserved":0,"allocated":43102208,"data_stored":25006975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44223,"internal_metadata":7835177793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.790253"},"pg_stats":[{"pgid":"1.11","version":"77'100","reported_seq":257,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.647295+0000","last_change":"2024-09-21T10:10:57.397786+0000","last_active":"2024-09-21T10:18:16.647295+0000","last_peered":"2024-09-21T10:18:16.647295+0000","last_clean":"2024-09-21T10:18:16.647295+0000","last_became_active":"2024-09-21T10:10:57.396742+0000","last_became_peered":"2024-09-21T10:10:57.396742+0000","last_unstale":"2024-09-21T10:18:16.647295+0000","last_undegraded":"2024-09-21T10:18:16.647295+0000","last_fullsized":"2024-09-21T10:18:16.647295+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:53.631923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:53.631923+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:33:35.364595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":594,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.209774+0000","last_change":"2024-09-21T10:03:38.389577+0000","last_active":"2024-09-21T10:18:14.209774+0000","last_peered":"2024-09-21T10:18:14.209774+0000","last_clean":"2024-09-21T10:18:14.209774+0000","last_became_active":"2024-09-21T10:03:38.389292+0000","last_became_peered":"2024-09-21T10:03:38.389292+0000","last_unstale":"2024-09-21T10:18:14.209774+0000","last_undegraded":"2024-09-21T10:18:14.209774+0000","last_fullsized":"2024-09-21T10:18:14.209774+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:32.113472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":255,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.182120+0000","last_change":"2024-09-21T10:14:30.717695+0000","last_active":"2024-09-21T10:18:14.182120+0000","last_peered":"2024-09-21T10:18:14.182120+0000","last_clean":"2024-09-21T10:18:14.182120+0000","last_became_active":"2024-09-21T10:14:30.705952+0000","last_became_peered":"2024-09-21T10:14:30.705952+0000","last_unstale":"2024-09-21T10:18:14.182120+0000","last_undegraded":"2024-09-21T10:18:14.182120+0000","last_fullsized":"2024-09-21T10:18:14.182120+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:08:18.863430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'100","reported_seq":268,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.182219+0000","last_change":"2024-09-21T10:11:05.438031+0000","last_active":"2024-09-21T10:18:14.182219+0000","last_peered":"2024-09-21T10:18:14.182219+0000","last_clean":"2024-09-21T10:18:14.182219+0000","last_became_active":"2024-09-21T10:11:05.437637+0000","last_became_peered":"2024-09-21T10:11:05.437637+0000","last_unstale":"2024-09-21T10:18:14.182219+0000","last_undegraded":"2024-09-21T10:18:14.182219+0000","last_fullsized":"2024-09-21T10:18:14.182219+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.798577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.798577+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:42.343217+0000","scrub_duration":0.0023622640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":360,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.182256+0000","last_change":"2024-09-21T10:10:57.392782+0000","last_active":"2024-09-21T10:18:14.182256+0000","last_peered":"2024-09-21T10:18:14.182256+0000","last_clean":"2024-09-21T10:18:14.182256+0000","last_became_active":"2024-09-21T10:10:57.392305+0000","last_became_peered":"2024-09-21T10:10:57.392305+0000","last_unstale":"2024-09-21T10:18:14.182256+0000","last_undegraded":"2024-09-21T10:18:14.182256+0000","last_fullsized":"2024-09-21T10:18:14.182256+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:42:01.387622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":516,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.646629+0000","last_change":"2024-09-21T10:09:23.846703+0000","last_active":"2024-09-21T10:18:16.646629+0000","last_peered":"2024-09-21T10:18:16.646629+0000","last_clean":"2024-09-21T10:18:16.646629+0000","last_became_active":"2024-09-21T10:09:23.846415+0000","last_became_peered":"2024-09-21T10:09:23.846415+0000","last_unstale":"2024-09-21T10:18:16.646629+0000","last_undegraded":"2024-09-21T10:18:16.646629+0000","last_fullsized":"2024-09-21T10:18:16.646629+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:23.129285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'100","reported_seq":260,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.654482+0000","last_change":"2024-09-21T10:18:12.637168+0000","last_active":"2024-09-21T10:18:19.654482+0000","last_peered":"2024-09-21T10:18:19.654482+0000","last_clean":"2024-09-21T10:18:19.654482+0000","last_became_active":"2024-09-21T10:18:12.636410+0000","last_became_peered":"2024-09-21T10:18:12.636410+0000","last_unstale":"2024-09-21T10:18:19.654482+0000","last_undegraded":"2024-09-21T10:18:19.654482+0000","last_fullsized":"2024-09-21T10:18:19.654482+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.433227+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:59:07.270049+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":242,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.647539+0000","last_change":"2024-09-21T10:09:49.922489+0000","last_active":"2024-09-21T10:18:16.647539+0000","last_peered":"2024-09-21T10:18:16.647539+0000","last_clean":"2024-09-21T10:18:16.647539+0000","last_became_active":"2024-09-21T10:09:49.921629+0000","last_became_peered":"2024-09-21T10:09:49.921629+0000","last_unstale":"2024-09-21T10:18:16.647539+0000","last_undegraded":"2024-09-21T10:18:16.647539+0000","last_fullsized":"2024-09-21T10:18:16.647539+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:50.611979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":251,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.181846+0000","last_change":"2024-09-21T10:14:30.709707+0000","last_active":"2024-09-21T10:18:14.181846+0000","last_peered":"2024-09-21T10:18:14.181846+0000","last_clean":"2024-09-21T10:18:14.181846+0000","last_became_active":"2024-09-21T10:14:30.709353+0000","last_became_peered":"2024-09-21T10:14:30.709353+0000","last_unstale":"2024-09-21T10:18:14.181846+0000","last_undegraded":"2024-09-21T10:18:14.181846+0000","last_fullsized":"2024-09-21T10:18:14.181846+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:12.822947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"77'100","reported_seq":149,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.654677+0000","last_change":"2024-09-21T10:15:02.842044+0000","last_active":"2024-09-21T10:18:19.654677+0000","last_peered":"2024-09-21T10:18:19.654677+0000","last_clean":"2024-09-21T10:18:19.654677+0000","last_became_active":"2024-09-21T10:15:01.478273+0000","last_became_peered":"2024-09-21T10:15:01.478273+0000","last_unstale":"2024-09-21T10:18:19.654677+0000","last_undegraded":"2024-09-21T10:18:19.654677+0000","last_fullsized":"2024-09-21T10:18:19.654677+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:15:02.841968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:15:02.841968+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:38:23.910899+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":594,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.210390+0000","last_change":"2024-09-21T10:03:38.381552+0000","last_active":"2024-09-21T10:18:14.210390+0000","last_peered":"2024-09-21T10:18:14.210390+0000","last_clean":"2024-09-21T10:18:14.210390+0000","last_became_active":"2024-09-21T10:03:38.380898+0000","last_became_peered":"2024-09-21T10:03:38.380898+0000","last_unstale":"2024-09-21T10:18:14.210390+0000","last_undegraded":"2024-09-21T10:18:14.210390+0000","last_fullsized":"2024-09-21T10:18:14.210390+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:53.409606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":427,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.630164+0000","last_change":"2024-09-21T10:10:57.459057+0000","last_active":"2024-09-21T10:18:12.630164+0000","last_peered":"2024-09-21T10:18:12.630164+0000","last_clean":"2024-09-21T10:18:12.630164+0000","last_became_active":"2024-09-21T10:10:57.451776+0000","last_became_peered":"2024-09-21T10:10:57.451776+0000","last_unstale":"2024-09-21T10:18:12.630164+0000","last_undegraded":"2024-09-21T10:18:12.630164+0000","last_fullsized":"2024-09-21T10:18:12.630164+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:00.138317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":10,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.637982+0000","last_change":"2024-09-21T10:18:12.636805+0000","last_active":"2024-09-21T10:18:12.637982+0000","last_peered":"2024-09-21T10:18:12.637982+0000","last_clean":"2024-09-21T10:18:12.637982+0000","last_became_active":"2024-09-21T10:18:12.636542+0000","last_became_peered":"2024-09-21T10:18:12.636542+0000","last_unstale":"2024-09-21T10:18:12.637982+0000","last_undegraded":"2024-09-21T10:18:12.637982+0000","last_fullsized":"2024-09-21T10:18:12.637982+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:29.211883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"77'100","reported_seq":263,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.182181+0000","last_change":"2024-09-21T10:14:30.708275+0000","last_active":"2024-09-21T10:18:14.182181+0000","last_peered":"2024-09-21T10:18:14.182181+0000","last_clean":"2024-09-21T10:18:14.182181+0000","last_became_active":"2024-09-21T10:14:30.707763+0000","last_became_peered":"2024-09-21T10:14:30.707763+0000","last_unstale":"2024-09-21T10:18:14.182181+0000","last_undegraded":"2024-09-21T10:18:14.182181+0000","last_fullsized":"2024-09-21T10:18:14.182181+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:55.259385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:55.259385+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:41.322447+0000","scrub_duration":0.0015667260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":298,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.182253+0000","last_change":"2024-09-21T10:14:30.720250+0000","last_active":"2024-09-21T10:18:14.182253+0000","last_peered":"2024-09-21T10:18:14.182253+0000","last_clean":"2024-09-21T10:18:14.182253+0000","last_became_active":"2024-09-21T10:14:30.707036+0000","last_became_peered":"2024-09-21T10:14:30.707036+0000","last_unstale":"2024-09-21T10:18:14.182253+0000","last_undegraded":"2024-09-21T10:18:14.182253+0000","last_fullsized":"2024-09-21T10:18:14.182253+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:17.438534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":341,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.182140+0000","last_change":"2024-09-21T10:15:01.476521+0000","last_active":"2024-09-21T10:18:14.182140+0000","last_peered":"2024-09-21T10:18:14.182140+0000","last_clean":"2024-09-21T10:18:14.182140+0000","last_became_active":"2024-09-21T10:15:01.476083+0000","last_became_peered":"2024-09-21T10:15:01.476083+0000","last_unstale":"2024-09-21T10:18:14.182140+0000","last_undegraded":"2024-09-21T10:18:14.182140+0000","last_fullsized":"2024-09-21T10:18:14.182140+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:50.230443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.181756+0000","last_change":"2024-09-21T10:18:12.633194+0000","last_active":"2024-09-21T10:18:14.181756+0000","last_peered":"2024-09-21T10:18:14.181756+0000","last_clean":"2024-09-21T10:18:14.181756+0000","last_became_active":"2024-09-21T10:18:12.632915+0000","last_became_peered":"2024-09-21T10:18:12.632915+0000","last_unstale":"2024-09-21T10:18:14.181756+0000","last_undegraded":"2024-09-21T10:18:14.181756+0000","last_fullsized":"2024-09-21T10:18:14.181756+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:13:26.865830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"77'100","reported_seq":401,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.209668+0000","last_change":"2024-09-21T10:09:23.839211+0000","last_active":"2024-09-21T10:18:14.209668+0000","last_peered":"2024-09-21T10:18:14.209668+0000","last_clean":"2024-09-21T10:18:14.209668+0000","last_became_active":"2024-09-21T10:09:23.838779+0000","last_became_peered":"2024-09-21T10:09:23.838779+0000","last_unstale":"2024-09-21T10:18:14.209668+0000","last_undegraded":"2024-09-21T10:18:14.209668+0000","last_fullsized":"2024-09-21T10:18:14.209668+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.981336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.981336+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:03:15.566873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":277,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.182354+0000","last_change":"2024-09-21T10:18:12.643154+0000","last_active":"2024-09-21T10:18:14.182354+0000","last_peered":"2024-09-21T10:18:14.182354+0000","last_clean":"2024-09-21T10:18:14.182354+0000","last_became_active":"2024-09-21T10:18:12.636287+0000","last_became_peered":"2024-09-21T10:18:12.636287+0000","last_unstale":"2024-09-21T10:18:14.182354+0000","last_undegraded":"2024-09-21T10:18:14.182354+0000","last_fullsized":"2024-09-21T10:18:14.182354+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:45:31.008309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":75,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.625079+0000","last_change":"2024-09-21T10:14:30.706497+0000","last_active":"2024-09-21T10:18:12.625079+0000","last_peered":"2024-09-21T10:18:12.625079+0000","last_clean":"2024-09-21T10:18:12.625079+0000","last_became_active":"2024-09-21T10:14:30.706202+0000","last_became_peered":"2024-09-21T10:14:30.706202+0000","last_unstale":"2024-09-21T10:18:12.625079+0000","last_undegraded":"2024-09-21T10:18:12.625079+0000","last_fullsized":"2024-09-21T10:18:12.625079+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:42:57.912707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":16,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.646807+0000","last_change":"2024-09-21T10:18:12.641462+0000","last_active":"2024-09-21T10:18:16.646807+0000","last_peered":"2024-09-21T10:18:16.646807+0000","last_clean":"2024-09-21T10:18:16.646807+0000","last_became_active":"2024-09-21T10:18:12.641056+0000","last_became_peered":"2024-09-21T10:18:12.641056+0000","last_unstale":"2024-09-21T10:18:16.646807+0000","last_undegraded":"2024-09-21T10:18:16.646807+0000","last_fullsized":"2024-09-21T10:18:16.646807+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:53:18.030279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"77'100","reported_seq":362,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.630887+0000","last_change":"2024-09-21T10:14:30.709997+0000","last_active":"2024-09-21T10:18:12.630887+0000","last_peered":"2024-09-21T10:18:12.630887+0000","last_clean":"2024-09-21T10:18:12.630887+0000","last_became_active":"2024-09-21T10:14:30.709580+0000","last_became_peered":"2024-09-21T10:14:30.709580+0000","last_unstale":"2024-09-21T10:18:12.630887+0000","last_undegraded":"2024-09-21T10:18:12.630887+0000","last_fullsized":"2024-09-21T10:18:12.630887+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.289859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.289859+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:27:50.110602+0000","scrub_duration":0.001251809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":249,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.182180+0000","last_change":"2024-09-21T10:10:57.460732+0000","last_active":"2024-09-21T10:18:14.182180+0000","last_peered":"2024-09-21T10:18:14.182180+0000","last_clean":"2024-09-21T10:18:14.182180+0000","last_became_active":"2024-09-21T10:10:57.454689+0000","last_became_peered":"2024-09-21T10:10:57.454689+0000","last_unstale":"2024-09-21T10:18:14.182180+0000","last_undegraded":"2024-09-21T10:18:14.182180+0000","last_fullsized":"2024-09-21T10:18:14.182180+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:12.035022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":368,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.629688+0000","last_change":"2024-09-21T10:03:38.378384+0000","last_active":"2024-09-21T10:18:12.629688+0000","last_peered":"2024-09-21T10:18:12.629688+0000","last_clean":"2024-09-21T10:18:12.629688+0000","last_became_active":"2024-09-21T10:03:38.378072+0000","last_became_peered":"2024-09-21T10:03:38.378072+0000","last_unstale":"2024-09-21T10:18:12.629688+0000","last_undegraded":"2024-09-21T10:18:12.629688+0000","last_fullsized":"2024-09-21T10:18:12.629688+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:54:37.039622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":14,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.210155+0000","last_change":"2024-09-21T10:18:13.037514+0000","last_active":"2024-09-21T10:18:14.210155+0000","last_peered":"2024-09-21T10:18:14.210155+0000","last_clean":"2024-09-21T10:18:14.210155+0000","last_became_active":"2024-09-21T10:18:13.037222+0000","last_became_peered":"2024-09-21T10:18:13.037222+0000","last_unstale":"2024-09-21T10:18:14.210155+0000","last_undegraded":"2024-09-21T10:18:14.210155+0000","last_fullsized":"2024-09-21T10:18:14.210155+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:02.409288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"77'100","reported_seq":11,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.635567+0000","last_change":"2024-09-21T10:18:12.633625+0000","last_active":"2024-09-21T10:18:12.635567+0000","last_peered":"2024-09-21T10:18:12.635567+0000","last_clean":"2024-09-21T10:18:12.635567+0000","last_became_active":"2024-09-21T10:18:12.633342+0000","last_became_peered":"2024-09-21T10:18:12.633342+0000","last_unstale":"2024-09-21T10:18:12.635567+0000","last_undegraded":"2024-09-21T10:18:12.635567+0000","last_fullsized":"2024-09-21T10:18:12.635567+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:07:26.188220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:07:26.188220+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":398,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.209365+0000","last_change":"2024-09-21T10:07:57.551556+0000","last_active":"2024-09-21T10:18:14.209365+0000","last_peered":"2024-09-21T10:18:14.209365+0000","last_clean":"2024-09-21T10:18:14.209365+0000","last_became_active":"2024-09-21T10:07:57.551249+0000","last_became_peered":"2024-09-21T10:07:57.551249+0000","last_unstale":"2024-09-21T10:18:14.209365+0000","last_undegraded":"2024-09-21T10:18:14.209365+0000","last_fullsized":"2024-09-21T10:18:14.209365+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:57.557152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":503,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.647054+0000","last_change":"2024-09-21T10:14:30.711105+0000","last_active":"2024-09-21T10:18:16.647054+0000","last_peered":"2024-09-21T10:18:16.647054+0000","last_clean":"2024-09-21T10:18:16.647054+0000","last_became_active":"2024-09-21T10:14:30.710759+0000","last_became_peered":"2024-09-21T10:14:30.710759+0000","last_unstale":"2024-09-21T10:18:16.647054+0000","last_undegraded":"2024-09-21T10:18:16.647054+0000","last_fullsized":"2024-09-21T10:18:16.647054+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:40.690042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":10,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:13.038938+0000","last_change":"2024-09-21T10:18:13.037771+0000","last_active":"2024-09-21T10:18:13.038938+0000","last_peered":"2024-09-21T10:18:13.038938+0000","last_clean":"2024-09-21T10:18:13.038938+0000","last_became_active":"2024-09-21T10:18:13.037425+0000","last_became_peered":"2024-09-21T10:18:13.037425+0000","last_unstale":"2024-09-21T10:18:13.038938+0000","last_undegraded":"2024-09-21T10:18:13.038938+0000","last_fullsized":"2024-09-21T10:18:13.038938+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:34:43.436187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":243,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.181929+0000","last_change":"2024-09-21T10:04:53.355975+0000","last_active":"2024-09-21T10:18:14.181929+0000","last_peered":"2024-09-21T10:18:14.181929+0000","last_clean":"2024-09-21T10:18:14.181929+0000","last_became_active":"2024-09-21T10:04:42.547505+0000","last_became_peered":"2024-09-21T10:04:42.547505+0000","last_unstale":"2024-09-21T10:18:14.181929+0000","last_undegraded":"2024-09-21T10:18:14.181929+0000","last_fullsized":"2024-09-21T10:18:14.181929+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.355912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.355912+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:43.456830+0000","scrub_duration":0.0017980030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":605,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.209809+0000","last_change":"2024-09-21T10:14:30.715303+0000","last_active":"2024-09-21T10:18:14.209809+0000","last_peered":"2024-09-21T10:18:14.209809+0000","last_clean":"2024-09-21T10:18:14.209809+0000","last_became_active":"2024-09-21T10:14:30.703664+0000","last_became_peered":"2024-09-21T10:14:30.703664+0000","last_unstale":"2024-09-21T10:18:14.209809+0000","last_undegraded":"2024-09-21T10:18:14.209809+0000","last_fullsized":"2024-09-21T10:18:14.209809+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:28.967906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":75,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.625500+0000","last_change":"2024-09-21T10:14:30.705927+0000","last_active":"2024-09-21T10:18:12.625500+0000","last_peered":"2024-09-21T10:18:12.625500+0000","last_clean":"2024-09-21T10:18:12.625500+0000","last_became_active":"2024-09-21T10:14:30.705112+0000","last_became_peered":"2024-09-21T10:14:30.705112+0000","last_unstale":"2024-09-21T10:18:12.625500+0000","last_undegraded":"2024-09-21T10:18:12.625500+0000","last_fullsized":"2024-09-21T10:18:12.625500+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:38:33.962199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":10,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.639329+0000","last_change":"2024-09-21T10:18:12.639080+0000","last_active":"2024-09-21T10:18:12.639329+0000","last_peered":"2024-09-21T10:18:12.639329+0000","last_clean":"2024-09-21T10:18:12.639329+0000","last_became_active":"2024-09-21T10:18:12.638789+0000","last_became_peered":"2024-09-21T10:18:12.638789+0000","last_unstale":"2024-09-21T10:18:12.639329+0000","last_undegraded":"2024-09-21T10:18:12.639329+0000","last_fullsized":"2024-09-21T10:18:12.639329+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:53.178338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":356,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.925958+0000","last_change":"2024-09-21T10:10:57.403722+0000","last_active":"2024-09-21T10:18:16.925958+0000","last_peered":"2024-09-21T10:18:16.925958+0000","last_clean":"2024-09-21T10:18:16.925958+0000","last_became_active":"2024-09-21T10:10:57.394700+0000","last_became_peered":"2024-09-21T10:10:57.394700+0000","last_unstale":"2024-09-21T10:18:16.925958+0000","last_undegraded":"2024-09-21T10:18:16.925958+0000","last_fullsized":"2024-09-21T10:18:16.925958+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:02.152981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:02.152981+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:52:34.887763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":352,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.629813+0000","last_change":"2024-09-21T10:14:30.707046+0000","last_active":"2024-09-21T10:18:12.629813+0000","last_peered":"2024-09-21T10:18:12.629813+0000","last_clean":"2024-09-21T10:18:12.629813+0000","last_became_active":"2024-09-21T10:14:30.706753+0000","last_became_peered":"2024-09-21T10:14:30.706753+0000","last_unstale":"2024-09-21T10:18:12.629813+0000","last_undegraded":"2024-09-21T10:18:12.629813+0000","last_fullsized":"2024-09-21T10:18:12.629813+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:58.136096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":11,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:13.039034+0000","last_change":"2024-09-21T10:18:13.037945+0000","last_active":"2024-09-21T10:18:13.039034+0000","last_peered":"2024-09-21T10:18:13.039034+0000","last_clean":"2024-09-21T10:18:13.039034+0000","last_became_active":"2024-09-21T10:18:13.037550+0000","last_became_peered":"2024-09-21T10:18:13.037550+0000","last_unstale":"2024-09-21T10:18:13.039034+0000","last_undegraded":"2024-09-21T10:18:13.039034+0000","last_fullsized":"2024-09-21T10:18:13.039034+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:24:24.702636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":11,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.182187+0000","last_change":"2024-09-21T10:18:12.635921+0000","last_active":"2024-09-21T10:18:14.182187+0000","last_peered":"2024-09-21T10:18:14.182187+0000","last_clean":"2024-09-21T10:18:14.182187+0000","last_became_active":"2024-09-21T10:18:12.635624+0000","last_became_peered":"2024-09-21T10:18:12.635624+0000","last_unstale":"2024-09-21T10:18:14.182187+0000","last_undegraded":"2024-09-21T10:18:14.182187+0000","last_fullsized":"2024-09-21T10:18:14.182187+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:08:14.392867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"77'100","reported_seq":250,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.654381+0000","last_change":"2024-09-21T10:14:05.474100+0000","last_active":"2024-09-21T10:18:19.654381+0000","last_peered":"2024-09-21T10:18:19.654381+0000","last_clean":"2024-09-21T10:18:19.654381+0000","last_became_active":"2024-09-21T10:14:05.473336+0000","last_became_peered":"2024-09-21T10:14:05.473336+0000","last_unstale":"2024-09-21T10:18:19.654381+0000","last_undegraded":"2024-09-21T10:18:19.654381+0000","last_fullsized":"2024-09-21T10:18:19.654381+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.540323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.540323+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:13:30.035766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.632399+0000","last_change":"2024-09-21T10:18:12.629311+0000","last_active":"2024-09-21T10:18:12.632399+0000","last_peered":"2024-09-21T10:18:12.632399+0000","last_clean":"2024-09-21T10:18:12.632399+0000","last_became_active":"2024-09-21T10:18:12.629001+0000","last_became_peered":"2024-09-21T10:18:12.629001+0000","last_unstale":"2024-09-21T10:18:12.632399+0000","last_undegraded":"2024-09-21T10:18:12.632399+0000","last_fullsized":"2024-09-21T10:18:12.632399+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:09:17.165751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":75,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.625659+0000","last_change":"2024-09-21T10:14:30.706905+0000","last_active":"2024-09-21T10:18:12.625659+0000","last_peered":"2024-09-21T10:18:12.625659+0000","last_clean":"2024-09-21T10:18:12.625659+0000","last_became_active":"2024-09-21T10:14:30.706614+0000","last_became_peered":"2024-09-21T10:14:30.706614+0000","last_unstale":"2024-09-21T10:18:12.625659+0000","last_undegraded":"2024-09-21T10:18:12.625659+0000","last_fullsized":"2024-09-21T10:18:12.625659+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:26:33.014993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":10,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:13.039087+0000","last_change":"2024-09-21T10:18:13.038010+0000","last_active":"2024-09-21T10:18:13.039087+0000","last_peered":"2024-09-21T10:18:13.039087+0000","last_clean":"2024-09-21T10:18:13.039087+0000","last_became_active":"2024-09-21T10:18:13.037700+0000","last_became_peered":"2024-09-21T10:18:13.037700+0000","last_unstale":"2024-09-21T10:18:13.039087+0000","last_undegraded":"2024-09-21T10:18:13.039087+0000","last_fullsized":"2024-09-21T10:18:13.039087+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:14:58.367606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":319,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.926065+0000","last_change":"2024-09-21T10:08:04.645700+0000","last_active":"2024-09-21T10:18:16.926065+0000","last_peered":"2024-09-21T10:18:16.926065+0000","last_clean":"2024-09-21T10:18:16.926065+0000","last_became_active":"2024-09-21T10:07:57.548189+0000","last_became_peered":"2024-09-21T10:07:57.548189+0000","last_unstale":"2024-09-21T10:18:16.926065+0000","last_undegraded":"2024-09-21T10:18:16.926065+0000","last_fullsized":"2024-09-21T10:18:16.926065+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:48.318010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:48.318010+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T16:31:28.173059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":596,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.209519+0000","last_change":"2024-09-21T10:03:38.386604+0000","last_active":"2024-09-21T10:18:14.209519+0000","last_peered":"2024-09-21T10:18:14.209519+0000","last_clean":"2024-09-21T10:18:14.209519+0000","last_became_active":"2024-09-21T10:03:38.386283+0000","last_became_peered":"2024-09-21T10:03:38.386283+0000","last_unstale":"2024-09-21T10:18:14.209519+0000","last_undegraded":"2024-09-21T10:18:14.209519+0000","last_fullsized":"2024-09-21T10:18:14.209519+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:27:05.944880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":458,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.654979+0000","last_change":"2024-09-21T10:14:30.709615+0000","last_active":"2024-09-21T10:18:19.654979+0000","last_peered":"2024-09-21T10:18:19.654979+0000","last_clean":"2024-09-21T10:18:19.654979+0000","last_became_active":"2024-09-21T10:14:30.709184+0000","last_became_peered":"2024-09-21T10:14:30.709184+0000","last_unstale":"2024-09-21T10:18:19.654979+0000","last_undegraded":"2024-09-21T10:18:19.654979+0000","last_fullsized":"2024-09-21T10:18:19.654979+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:55:27.220232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":19,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.654500+0000","last_change":"2024-09-21T10:18:13.037466+0000","last_active":"2024-09-21T10:18:19.654500+0000","last_peered":"2024-09-21T10:18:19.654500+0000","last_clean":"2024-09-21T10:18:19.654500+0000","last_became_active":"2024-09-21T10:18:13.037151+0000","last_became_peered":"2024-09-21T10:18:13.037151+0000","last_unstale":"2024-09-21T10:18:19.654500+0000","last_undegraded":"2024-09-21T10:18:19.654500+0000","last_fullsized":"2024-09-21T10:18:19.654500+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:35:23.611663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":260,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.181753+0000","last_change":"2024-09-21T10:09:49.924286+0000","last_active":"2024-09-21T10:18:14.181753+0000","last_peered":"2024-09-21T10:18:14.181753+0000","last_clean":"2024-09-21T10:18:14.181753+0000","last_became_active":"2024-09-21T10:09:49.923897+0000","last_became_peered":"2024-09-21T10:09:49.923897+0000","last_unstale":"2024-09-21T10:18:14.181753+0000","last_undegraded":"2024-09-21T10:18:14.181753+0000","last_fullsized":"2024-09-21T10:18:14.181753+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:47.393146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:47.393146+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:44.898083+0000","scrub_duration":0.001601086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":352,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.926197+0000","last_change":"2024-09-21T10:10:57.402946+0000","last_active":"2024-09-21T10:18:16.926197+0000","last_peered":"2024-09-21T10:18:16.926197+0000","last_clean":"2024-09-21T10:18:16.926197+0000","last_became_active":"2024-09-21T10:10:57.395450+0000","last_became_peered":"2024-09-21T10:10:57.395450+0000","last_unstale":"2024-09-21T10:18:16.926197+0000","last_undegraded":"2024-09-21T10:18:16.926197+0000","last_fullsized":"2024-09-21T10:18:16.926197+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:08.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":580,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.209681+0000","last_change":"2024-09-21T10:10:57.390595+0000","last_active":"2024-09-21T10:18:14.209681+0000","last_peered":"2024-09-21T10:18:14.209681+0000","last_clean":"2024-09-21T10:18:14.209681+0000","last_became_active":"2024-09-21T10:10:57.390265+0000","last_became_peered":"2024-09-21T10:10:57.390265+0000","last_unstale":"2024-09-21T10:18:14.209681+0000","last_undegraded":"2024-09-21T10:18:14.209681+0000","last_fullsized":"2024-09-21T10:18:14.209681+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:37.485063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":385,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.629898+0000","last_change":"2024-09-21T10:14:30.711719+0000","last_active":"2024-09-21T10:18:12.629898+0000","last_peered":"2024-09-21T10:18:12.629898+0000","last_clean":"2024-09-21T10:18:12.629898+0000","last_became_active":"2024-09-21T10:14:30.711452+0000","last_became_peered":"2024-09-21T10:14:30.711452+0000","last_unstale":"2024-09-21T10:18:12.629898+0000","last_undegraded":"2024-09-21T10:18:12.629898+0000","last_fullsized":"2024-09-21T10:18:12.629898+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:46.907528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":10,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.639316+0000","last_change":"2024-09-21T10:18:12.639005+0000","last_active":"2024-09-21T10:18:12.639316+0000","last_peered":"2024-09-21T10:18:12.639316+0000","last_clean":"2024-09-21T10:18:12.639316+0000","last_became_active":"2024-09-21T10:18:12.638716+0000","last_became_peered":"2024-09-21T10:18:12.638716+0000","last_unstale":"2024-09-21T10:18:12.639316+0000","last_undegraded":"2024-09-21T10:18:12.639316+0000","last_fullsized":"2024-09-21T10:18:12.639316+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:08.923853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":261,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.654635+0000","last_change":"2024-09-21T10:14:30.701644+0000","last_active":"2024-09-21T10:18:19.654635+0000","last_peered":"2024-09-21T10:18:19.654635+0000","last_clean":"2024-09-21T10:18:19.654635+0000","last_became_active":"2024-09-21T10:14:30.701301+0000","last_became_peered":"2024-09-21T10:14:30.701301+0000","last_unstale":"2024-09-21T10:18:19.654635+0000","last_undegraded":"2024-09-21T10:18:19.654635+0000","last_fullsized":"2024-09-21T10:18:19.654635+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.550551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.550551+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:23.637695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":234,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.181951+0000","last_change":"2024-09-21T10:09:23.838944+0000","last_active":"2024-09-21T10:18:14.181951+0000","last_peered":"2024-09-21T10:18:14.181951+0000","last_clean":"2024-09-21T10:18:14.181951+0000","last_became_active":"2024-09-21T10:09:23.837886+0000","last_became_peered":"2024-09-21T10:09:23.837886+0000","last_unstale":"2024-09-21T10:18:14.181951+0000","last_undegraded":"2024-09-21T10:18:14.181951+0000","last_fullsized":"2024-09-21T10:18:14.181951+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:16.928195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":481,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.647166+0000","last_change":"2024-09-21T10:18:12.637103+0000","last_active":"2024-09-21T10:18:16.647166+0000","last_peered":"2024-09-21T10:18:16.647166+0000","last_clean":"2024-09-21T10:18:16.647166+0000","last_became_active":"2024-09-21T10:18:12.636645+0000","last_became_peered":"2024-09-21T10:18:12.636645+0000","last_unstale":"2024-09-21T10:18:16.647166+0000","last_undegraded":"2024-09-21T10:18:16.647166+0000","last_fullsized":"2024-09-21T10:18:16.647166+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:53:50.745484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":242,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.647462+0000","last_change":"2024-09-21T10:09:49.923044+0000","last_active":"2024-09-21T10:18:16.647462+0000","last_peered":"2024-09-21T10:18:16.647462+0000","last_clean":"2024-09-21T10:18:16.647462+0000","last_became_active":"2024-09-21T10:09:49.922711+0000","last_became_peered":"2024-09-21T10:09:49.922711+0000","last_unstale":"2024-09-21T10:18:16.647462+0000","last_undegraded":"2024-09-21T10:18:16.647462+0000","last_fullsized":"2024-09-21T10:18:16.647462+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:14.090084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":11,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.182113+0000","last_change":"2024-09-21T10:18:13.038029+0000","last_active":"2024-09-21T10:18:14.182113+0000","last_peered":"2024-09-21T10:18:14.182113+0000","last_clean":"2024-09-21T10:18:14.182113+0000","last_became_active":"2024-09-21T10:18:13.037584+0000","last_became_peered":"2024-09-21T10:18:13.037584+0000","last_unstale":"2024-09-21T10:18:14.182113+0000","last_undegraded":"2024-09-21T10:18:14.182113+0000","last_fullsized":"2024-09-21T10:18:14.182113+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:18.430382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":284,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.181924+0000","last_change":"2024-09-21T10:18:12.636851+0000","last_active":"2024-09-21T10:18:14.181924+0000","last_peered":"2024-09-21T10:18:14.181924+0000","last_clean":"2024-09-21T10:18:14.181924+0000","last_became_active":"2024-09-21T10:18:12.636513+0000","last_became_peered":"2024-09-21T10:18:12.636513+0000","last_unstale":"2024-09-21T10:18:14.181924+0000","last_undegraded":"2024-09-21T10:18:14.181924+0000","last_fullsized":"2024-09-21T10:18:14.181924+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:52.361137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:52.361137+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:21.025735+0000","scrub_duration":0.001255358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":635,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.926231+0000","last_change":"2024-09-21T10:09:23.837859+0000","last_active":"2024-09-21T10:18:16.926231+0000","last_peered":"2024-09-21T10:18:16.926231+0000","last_clean":"2024-09-21T10:18:16.926231+0000","last_became_active":"2024-09-21T10:09:23.837488+0000","last_became_peered":"2024-09-21T10:09:23.837488+0000","last_unstale":"2024-09-21T10:18:16.926231+0000","last_undegraded":"2024-09-21T10:18:16.926231+0000","last_fullsized":"2024-09-21T10:18:16.926231+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:25.215501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":118,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.181841+0000","last_change":"2024-09-21T10:10:27.806070+0000","last_active":"2024-09-21T10:18:14.181841+0000","last_peered":"2024-09-21T10:18:14.181841+0000","last_clean":"2024-09-21T10:18:14.181841+0000","last_became_active":"2024-09-21T10:10:27.805718+0000","last_became_peered":"2024-09-21T10:10:27.805718+0000","last_unstale":"2024-09-21T10:18:14.181841+0000","last_undegraded":"2024-09-21T10:18:14.181841+0000","last_fullsized":"2024-09-21T10:18:14.181841+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:31.144118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":629,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.926513+0000","last_change":"2024-09-21T10:10:57.395073+0000","last_active":"2024-09-21T10:18:16.926513+0000","last_peered":"2024-09-21T10:18:16.926513+0000","last_clean":"2024-09-21T10:18:16.926513+0000","last_became_active":"2024-09-21T10:10:57.394674+0000","last_became_peered":"2024-09-21T10:10:57.394674+0000","last_unstale":"2024-09-21T10:18:16.926513+0000","last_undegraded":"2024-09-21T10:18:16.926513+0000","last_fullsized":"2024-09-21T10:18:16.926513+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:48:48.912649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":10,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.635451+0000","last_change":"2024-09-21T10:18:12.632980+0000","last_active":"2024-09-21T10:18:12.635451+0000","last_peered":"2024-09-21T10:18:12.635451+0000","last_clean":"2024-09-21T10:18:12.635451+0000","last_became_active":"2024-09-21T10:18:12.632684+0000","last_became_peered":"2024-09-21T10:18:12.632684+0000","last_unstale":"2024-09-21T10:18:12.635451+0000","last_undegraded":"2024-09-21T10:18:12.635451+0000","last_fullsized":"2024-09-21T10:18:12.635451+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:08.974530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":498,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.647601+0000","last_change":"2024-09-21T10:10:57.398437+0000","last_active":"2024-09-21T10:18:16.647601+0000","last_peered":"2024-09-21T10:18:16.647601+0000","last_clean":"2024-09-21T10:18:16.647601+0000","last_became_active":"2024-09-21T10:10:57.397441+0000","last_became_peered":"2024-09-21T10:10:57.397441+0000","last_unstale":"2024-09-21T10:18:16.647601+0000","last_undegraded":"2024-09-21T10:18:16.647601+0000","last_fullsized":"2024-09-21T10:18:16.647601+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:04.043917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:04.043917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:53:31.813612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":193,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.654503+0000","last_change":"2024-09-21T10:14:05.472790+0000","last_active":"2024-09-21T10:18:19.654503+0000","last_peered":"2024-09-21T10:18:19.654503+0000","last_clean":"2024-09-21T10:18:19.654503+0000","last_became_active":"2024-09-21T10:14:05.472238+0000","last_became_peered":"2024-09-21T10:14:05.472238+0000","last_unstale":"2024-09-21T10:18:19.654503+0000","last_undegraded":"2024-09-21T10:18:19.654503+0000","last_fullsized":"2024-09-21T10:18:19.654503+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:56.561032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":75,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.626474+0000","last_change":"2024-09-21T10:14:30.709652+0000","last_active":"2024-09-21T10:18:12.626474+0000","last_peered":"2024-09-21T10:18:12.626474+0000","last_clean":"2024-09-21T10:18:12.626474+0000","last_became_active":"2024-09-21T10:14:30.705706+0000","last_became_peered":"2024-09-21T10:14:30.705706+0000","last_unstale":"2024-09-21T10:18:12.626474+0000","last_undegraded":"2024-09-21T10:18:12.626474+0000","last_fullsized":"2024-09-21T10:18:12.626474+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:24:25.749923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":464,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655086+0000","last_change":"2024-09-21T10:14:05.477875+0000","last_active":"2024-09-21T10:18:19.655086+0000","last_peered":"2024-09-21T10:18:19.655086+0000","last_clean":"2024-09-21T10:18:19.655086+0000","last_became_active":"2024-09-21T10:14:05.477192+0000","last_became_peered":"2024-09-21T10:14:05.477192+0000","last_unstale":"2024-09-21T10:18:19.655086+0000","last_undegraded":"2024-09-21T10:18:19.655086+0000","last_fullsized":"2024-09-21T10:18:19.655086+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:19:47.345673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":14,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.209681+0000","last_change":"2024-09-21T10:18:13.039421+0000","last_active":"2024-09-21T10:18:14.209681+0000","last_peered":"2024-09-21T10:18:14.209681+0000","last_clean":"2024-09-21T10:18:14.209681+0000","last_became_active":"2024-09-21T10:18:13.039085+0000","last_became_peered":"2024-09-21T10:18:13.039085+0000","last_unstale":"2024-09-21T10:18:14.209681+0000","last_undegraded":"2024-09-21T10:18:14.209681+0000","last_fullsized":"2024-09-21T10:18:14.209681+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:04:27.693872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":342,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.630800+0000","last_change":"2024-09-21T10:04:45.242148+0000","last_active":"2024-09-21T10:18:12.630800+0000","last_peered":"2024-09-21T10:18:12.630800+0000","last_clean":"2024-09-21T10:18:12.630800+0000","last_became_active":"2024-09-21T10:04:42.549844+0000","last_became_peered":"2024-09-21T10:04:42.549844+0000","last_unstale":"2024-09-21T10:18:12.630800+0000","last_undegraded":"2024-09-21T10:18:12.630800+0000","last_fullsized":"2024-09-21T10:18:12.630800+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_deep_scrub":"77'100","last_deep_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.242073+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:03:42.250480+0000","scrub_duration":0.0023466239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":302,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.209872+0000","last_change":"2024-09-21T10:15:01.489326+0000","last_active":"2024-09-21T10:18:14.209872+0000","last_peered":"2024-09-21T10:18:14.209872+0000","last_clean":"2024-09-21T10:18:14.209872+0000","last_became_active":"2024-09-21T10:15:01.489070+0000","last_became_peered":"2024-09-21T10:15:01.489070+0000","last_unstale":"2024-09-21T10:18:14.209872+0000","last_undegraded":"2024-09-21T10:18:14.209872+0000","last_fullsized":"2024-09-21T10:18:14.209872+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:27:41.788029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":520,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655093+0000","last_change":"2024-09-21T10:14:05.475970+0000","last_active":"2024-09-21T10:18:19.655093+0000","last_peered":"2024-09-21T10:18:19.655093+0000","last_clean":"2024-09-21T10:18:19.655093+0000","last_became_active":"2024-09-21T10:14:05.475331+0000","last_became_peered":"2024-09-21T10:14:05.475331+0000","last_unstale":"2024-09-21T10:18:19.655093+0000","last_undegraded":"2024-09-21T10:18:19.655093+0000","last_fullsized":"2024-09-21T10:18:19.655093+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:17.069736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":456,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.654735+0000","last_change":"2024-09-21T10:18:13.039172+0000","last_active":"2024-09-21T10:18:19.654735+0000","last_peered":"2024-09-21T10:18:19.654735+0000","last_clean":"2024-09-21T10:18:19.654735+0000","last_became_active":"2024-09-21T10:18:13.038756+0000","last_became_peered":"2024-09-21T10:18:13.038756+0000","last_unstale":"2024-09-21T10:18:19.654735+0000","last_undegraded":"2024-09-21T10:18:19.654735+0000","last_fullsized":"2024-09-21T10:18:19.654735+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:06.668926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":16,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.926061+0000","last_change":"2024-09-21T10:18:12.633418+0000","last_active":"2024-09-21T10:18:16.926061+0000","last_peered":"2024-09-21T10:18:16.926061+0000","last_clean":"2024-09-21T10:18:16.926061+0000","last_became_active":"2024-09-21T10:18:12.633151+0000","last_became_peered":"2024-09-21T10:18:12.633151+0000","last_unstale":"2024-09-21T10:18:16.926061+0000","last_undegraded":"2024-09-21T10:18:16.926061+0000","last_fullsized":"2024-09-21T10:18:16.926061+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:34:31.105292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":359,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.629835+0000","last_change":"2024-09-21T10:09:23.839563+0000","last_active":"2024-09-21T10:18:12.629835+0000","last_peered":"2024-09-21T10:18:12.629835+0000","last_clean":"2024-09-21T10:18:12.629835+0000","last_became_active":"2024-09-21T10:09:23.839207+0000","last_became_peered":"2024-09-21T10:09:23.839207+0000","last_unstale":"2024-09-21T10:18:12.629835+0000","last_undegraded":"2024-09-21T10:18:12.629835+0000","last_fullsized":"2024-09-21T10:18:12.629835+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:49.226496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:49.226496+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:03.960125+0000","scrub_duration":0.001656865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":452,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.654656+0000","last_change":"2024-09-21T10:14:05.478152+0000","last_active":"2024-09-21T10:18:19.654656+0000","last_peered":"2024-09-21T10:18:19.654656+0000","last_clean":"2024-09-21T10:18:19.654656+0000","last_became_active":"2024-09-21T10:14:05.477547+0000","last_became_peered":"2024-09-21T10:14:05.477547+0000","last_unstale":"2024-09-21T10:18:19.654656+0000","last_undegraded":"2024-09-21T10:18:19.654656+0000","last_fullsized":"2024-09-21T10:18:19.654656+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:34.646046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":264,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.182084+0000","last_change":"2024-09-21T10:14:30.708090+0000","last_active":"2024-09-21T10:18:14.182084+0000","last_peered":"2024-09-21T10:18:14.182084+0000","last_clean":"2024-09-21T10:18:14.182084+0000","last_became_active":"2024-09-21T10:14:30.707266+0000","last_became_peered":"2024-09-21T10:14:30.707266+0000","last_unstale":"2024-09-21T10:18:14.182084+0000","last_undegraded":"2024-09-21T10:18:14.182084+0000","last_fullsized":"2024-09-21T10:18:14.182084+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:38.474537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":512,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.647244+0000","last_change":"2024-09-21T10:10:57.397251+0000","last_active":"2024-09-21T10:18:16.647244+0000","last_peered":"2024-09-21T10:18:16.647244+0000","last_clean":"2024-09-21T10:18:16.647244+0000","last_became_active":"2024-09-21T10:10:57.396271+0000","last_became_peered":"2024-09-21T10:10:57.396271+0000","last_unstale":"2024-09-21T10:18:16.647244+0000","last_undegraded":"2024-09-21T10:18:16.647244+0000","last_fullsized":"2024-09-21T10:18:16.647244+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:17.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":16,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.647081+0000","last_change":"2024-09-21T10:18:13.037459+0000","last_active":"2024-09-21T10:18:16.647081+0000","last_peered":"2024-09-21T10:18:16.647081+0000","last_clean":"2024-09-21T10:18:16.647081+0000","last_became_active":"2024-09-21T10:18:13.037118+0000","last_became_peered":"2024-09-21T10:18:13.037118+0000","last_unstale":"2024-09-21T10:18:16.647081+0000","last_undegraded":"2024-09-21T10:18:16.647081+0000","last_fullsized":"2024-09-21T10:18:16.647081+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:45:04.094658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":75,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.626489+0000","last_change":"2024-09-21T10:14:30.709001+0000","last_active":"2024-09-21T10:18:12.626489+0000","last_peered":"2024-09-21T10:18:12.626489+0000","last_clean":"2024-09-21T10:18:12.626489+0000","last_became_active":"2024-09-21T10:14:30.708636+0000","last_became_peered":"2024-09-21T10:14:30.708636+0000","last_unstale":"2024-09-21T10:18:12.626489+0000","last_undegraded":"2024-09-21T10:18:12.626489+0000","last_fullsized":"2024-09-21T10:18:12.626489+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:59.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:59.099852+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:20:01.949185+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":602,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.210188+0000","last_change":"2024-09-21T10:09:23.841186+0000","last_active":"2024-09-21T10:18:14.210188+0000","last_peered":"2024-09-21T10:18:14.210188+0000","last_clean":"2024-09-21T10:18:14.210188+0000","last_became_active":"2024-09-21T10:09:23.840854+0000","last_became_peered":"2024-09-21T10:09:23.840854+0000","last_unstale":"2024-09-21T10:18:14.210188+0000","last_undegraded":"2024-09-21T10:18:14.210188+0000","last_fullsized":"2024-09-21T10:18:14.210188+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:29.666358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":458,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655009+0000","last_change":"2024-09-21T10:14:30.708385+0000","last_active":"2024-09-21T10:18:19.655009+0000","last_peered":"2024-09-21T10:18:19.655009+0000","last_clean":"2024-09-21T10:18:19.655009+0000","last_became_active":"2024-09-21T10:14:30.708046+0000","last_became_peered":"2024-09-21T10:14:30.708046+0000","last_unstale":"2024-09-21T10:18:19.655009+0000","last_undegraded":"2024-09-21T10:18:19.655009+0000","last_fullsized":"2024-09-21T10:18:19.655009+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:16:45.563095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":599,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.209501+0000","last_change":"2024-09-21T10:09:49.926548+0000","last_active":"2024-09-21T10:18:14.209501+0000","last_peered":"2024-09-21T10:18:14.209501+0000","last_clean":"2024-09-21T10:18:14.209501+0000","last_became_active":"2024-09-21T10:09:49.926255+0000","last_became_peered":"2024-09-21T10:09:49.926255+0000","last_unstale":"2024-09-21T10:18:14.209501+0000","last_undegraded":"2024-09-21T10:18:14.209501+0000","last_fullsized":"2024-09-21T10:18:14.209501+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:49.855878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":16,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.925926+0000","last_change":"2024-09-21T10:18:13.038950+0000","last_active":"2024-09-21T10:18:16.925926+0000","last_peered":"2024-09-21T10:18:16.925926+0000","last_clean":"2024-09-21T10:18:16.925926+0000","last_became_active":"2024-09-21T10:18:13.038641+0000","last_became_peered":"2024-09-21T10:18:13.038641+0000","last_unstale":"2024-09-21T10:18:16.925926+0000","last_undegraded":"2024-09-21T10:18:16.925926+0000","last_fullsized":"2024-09-21T10:18:16.925926+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:46:26.410913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":400,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.209586+0000","last_change":"2024-09-21T10:10:57.390804+0000","last_active":"2024-09-21T10:18:14.209586+0000","last_peered":"2024-09-21T10:18:14.209586+0000","last_clean":"2024-09-21T10:18:14.209586+0000","last_became_active":"2024-09-21T10:10:57.390323+0000","last_became_peered":"2024-09-21T10:10:57.390323+0000","last_unstale":"2024-09-21T10:18:14.209586+0000","last_undegraded":"2024-09-21T10:18:14.209586+0000","last_fullsized":"2024-09-21T10:18:14.209586+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.005477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.005477+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:29:57.702046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":257,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.647643+0000","last_change":"2024-09-21T10:14:30.718942+0000","last_active":"2024-09-21T10:18:16.647643+0000","last_peered":"2024-09-21T10:18:16.647643+0000","last_clean":"2024-09-21T10:18:16.647643+0000","last_became_active":"2024-09-21T10:14:30.709129+0000","last_became_peered":"2024-09-21T10:14:30.709129+0000","last_unstale":"2024-09-21T10:18:16.647643+0000","last_undegraded":"2024-09-21T10:18:16.647643+0000","last_fullsized":"2024-09-21T10:18:16.647643+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:18.171064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":381,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.631028+0000","last_change":"2024-09-21T10:03:38.392767+0000","last_active":"2024-09-21T10:18:12.631028+0000","last_peered":"2024-09-21T10:18:12.631028+0000","last_clean":"2024-09-21T10:18:12.631028+0000","last_became_active":"2024-09-21T10:03:38.392424+0000","last_became_peered":"2024-09-21T10:03:38.392424+0000","last_unstale":"2024-09-21T10:18:12.631028+0000","last_undegraded":"2024-09-21T10:18:12.631028+0000","last_fullsized":"2024-09-21T10:18:12.631028+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:25.358291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":75,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.626182+0000","last_change":"2024-09-21T10:14:30.703352+0000","last_active":"2024-09-21T10:18:12.626182+0000","last_peered":"2024-09-21T10:18:12.626182+0000","last_clean":"2024-09-21T10:18:12.626182+0000","last_became_active":"2024-09-21T10:14:30.703068+0000","last_became_peered":"2024-09-21T10:14:30.703068+0000","last_unstale":"2024-09-21T10:18:12.626182+0000","last_undegraded":"2024-09-21T10:18:12.626182+0000","last_fullsized":"2024-09-21T10:18:12.626182+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:00:26.966633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":75,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.626046+0000","last_change":"2024-09-21T10:14:30.702750+0000","last_active":"2024-09-21T10:18:12.626046+0000","last_peered":"2024-09-21T10:18:12.626046+0000","last_clean":"2024-09-21T10:18:12.626046+0000","last_became_active":"2024-09-21T10:14:30.702443+0000","last_became_peered":"2024-09-21T10:14:30.702443+0000","last_unstale":"2024-09-21T10:18:12.626046+0000","last_undegraded":"2024-09-21T10:18:12.626046+0000","last_fullsized":"2024-09-21T10:18:12.626046+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:38:19.353586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":85,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.625318+0000","last_change":"2024-09-21T10:14:30.716754+0000","last_active":"2024-09-21T10:18:12.625318+0000","last_peered":"2024-09-21T10:18:12.625318+0000","last_clean":"2024-09-21T10:18:12.625318+0000","last_became_active":"2024-09-21T10:14:30.704683+0000","last_became_peered":"2024-09-21T10:14:30.704683+0000","last_unstale":"2024-09-21T10:18:12.625318+0000","last_undegraded":"2024-09-21T10:18:12.625318+0000","last_fullsized":"2024-09-21T10:18:12.625318+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:45.993877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":510,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.654591+0000","last_change":"2024-09-21T10:14:05.477429+0000","last_active":"2024-09-21T10:18:19.654591+0000","last_peered":"2024-09-21T10:18:19.654591+0000","last_clean":"2024-09-21T10:18:19.654591+0000","last_became_active":"2024-09-21T10:14:05.476794+0000","last_became_peered":"2024-09-21T10:14:05.476794+0000","last_unstale":"2024-09-21T10:18:19.654591+0000","last_undegraded":"2024-09-21T10:18:19.654591+0000","last_fullsized":"2024-09-21T10:18:19.654591+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:18:54.963704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":252,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.926389+0000","last_change":"2024-09-21T10:14:30.713873+0000","last_active":"2024-09-21T10:18:16.926389+0000","last_peered":"2024-09-21T10:18:16.926389+0000","last_clean":"2024-09-21T10:18:16.926389+0000","last_became_active":"2024-09-21T10:14:30.703149+0000","last_became_peered":"2024-09-21T10:14:30.703149+0000","last_unstale":"2024-09-21T10:18:16.926389+0000","last_undegraded":"2024-09-21T10:18:16.926389+0000","last_fullsized":"2024-09-21T10:18:16.926389+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:26.453902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'100","reported_seq":401,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.926278+0000","last_change":"2024-09-21T10:18:12.633959+0000","last_active":"2024-09-21T10:18:16.926278+0000","last_peered":"2024-09-21T10:18:16.926278+0000","last_clean":"2024-09-21T10:18:16.926278+0000","last_became_active":"2024-09-21T10:18:12.633591+0000","last_became_peered":"2024-09-21T10:18:12.633591+0000","last_unstale":"2024-09-21T10:18:16.926278+0000","last_undegraded":"2024-09-21T10:18:16.926278+0000","last_fullsized":"2024-09-21T10:18:16.926278+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.014993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.014993+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:16:55.324246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":283,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.182346+0000","last_change":"2024-09-21T10:15:01.483814+0000","last_active":"2024-09-21T10:18:14.182346+0000","last_peered":"2024-09-21T10:18:14.182346+0000","last_clean":"2024-09-21T10:18:14.182346+0000","last_became_active":"2024-09-21T10:15:01.483522+0000","last_became_peered":"2024-09-21T10:15:01.483522+0000","last_unstale":"2024-09-21T10:18:14.182346+0000","last_undegraded":"2024-09-21T10:18:14.182346+0000","last_fullsized":"2024-09-21T10:18:14.182346+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:13.632066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":584,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.209540+0000","last_change":"2024-09-21T10:03:38.425111+0000","last_active":"2024-09-21T10:18:14.209540+0000","last_peered":"2024-09-21T10:18:14.209540+0000","last_clean":"2024-09-21T10:18:14.209540+0000","last_became_active":"2024-09-21T10:03:38.424704+0000","last_became_peered":"2024-09-21T10:03:38.424704+0000","last_unstale":"2024-09-21T10:18:14.209540+0000","last_undegraded":"2024-09-21T10:18:14.209540+0000","last_fullsized":"2024-09-21T10:18:14.209540+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:35.157076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":486,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.646800+0000","last_change":"2024-09-21T10:10:57.397755+0000","last_active":"2024-09-21T10:18:16.646800+0000","last_peered":"2024-09-21T10:18:16.646800+0000","last_clean":"2024-09-21T10:18:16.646800+0000","last_became_active":"2024-09-21T10:10:57.397249+0000","last_became_peered":"2024-09-21T10:10:57.397249+0000","last_unstale":"2024-09-21T10:18:16.646800+0000","last_undegraded":"2024-09-21T10:18:16.646800+0000","last_fullsized":"2024-09-21T10:18:16.646800+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:40.813503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":228,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.654248+0000","last_change":"2024-09-21T10:14:05.471573+0000","last_active":"2024-09-21T10:18:19.654248+0000","last_peered":"2024-09-21T10:18:19.654248+0000","last_clean":"2024-09-21T10:18:19.654248+0000","last_became_active":"2024-09-21T10:14:05.471270+0000","last_became_peered":"2024-09-21T10:14:05.471270+0000","last_unstale":"2024-09-21T10:18:19.654248+0000","last_undegraded":"2024-09-21T10:18:19.654248+0000","last_fullsized":"2024-09-21T10:18:19.654248+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:56:13.468768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":331,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.630959+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T10:18:12.630959+0000","last_peered":"2024-09-21T10:18:12.630959+0000","last_clean":"2024-09-21T10:18:12.630959+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T10:18:12.630959+0000","last_undegraded":"2024-09-21T10:18:12.630959+0000","last_fullsized":"2024-09-21T10:18:12.630959+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":605,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.209349+0000","last_change":"2024-09-21T10:09:49.928053+0000","last_active":"2024-09-21T10:18:14.209349+0000","last_peered":"2024-09-21T10:18:14.209349+0000","last_clean":"2024-09-21T10:18:14.209349+0000","last_became_active":"2024-09-21T10:09:49.927231+0000","last_became_peered":"2024-09-21T10:09:49.927231+0000","last_unstale":"2024-09-21T10:18:14.209349+0000","last_undegraded":"2024-09-21T10:18:14.209349+0000","last_fullsized":"2024-09-21T10:18:14.209349+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:26.987553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":519,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.647806+0000","last_change":"2024-09-21T10:09:23.840717+0000","last_active":"2024-09-21T10:18:16.647806+0000","last_peered":"2024-09-21T10:18:16.647806+0000","last_clean":"2024-09-21T10:18:16.647806+0000","last_became_active":"2024-09-21T10:09:23.840323+0000","last_became_peered":"2024-09-21T10:09:23.840323+0000","last_unstale":"2024-09-21T10:18:16.647806+0000","last_undegraded":"2024-09-21T10:18:16.647806+0000","last_fullsized":"2024-09-21T10:18:16.647806+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:55.769770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":240,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.926055+0000","last_change":"2024-09-21T10:18:12.632498+0000","last_active":"2024-09-21T10:18:16.926055+0000","last_peered":"2024-09-21T10:18:16.926055+0000","last_clean":"2024-09-21T10:18:16.926055+0000","last_became_active":"2024-09-21T10:18:12.631648+0000","last_became_peered":"2024-09-21T10:18:12.631648+0000","last_unstale":"2024-09-21T10:18:16.926055+0000","last_undegraded":"2024-09-21T10:18:16.926055+0000","last_fullsized":"2024-09-21T10:18:16.926055+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:17.592618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":582,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.209688+0000","last_change":"2024-09-21T10:14:30.702101+0000","last_active":"2024-09-21T10:18:14.209688+0000","last_peered":"2024-09-21T10:18:14.209688+0000","last_clean":"2024-09-21T10:18:14.209688+0000","last_became_active":"2024-09-21T10:14:30.701783+0000","last_became_peered":"2024-09-21T10:14:30.701783+0000","last_unstale":"2024-09-21T10:18:14.209688+0000","last_undegraded":"2024-09-21T10:18:14.209688+0000","last_fullsized":"2024-09-21T10:18:14.209688+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:13.087010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":75,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.624999+0000","last_change":"2024-09-21T10:14:30.707503+0000","last_active":"2024-09-21T10:18:12.624999+0000","last_peered":"2024-09-21T10:18:12.624999+0000","last_clean":"2024-09-21T10:18:12.624999+0000","last_became_active":"2024-09-21T10:14:30.706881+0000","last_became_peered":"2024-09-21T10:14:30.706881+0000","last_unstale":"2024-09-21T10:18:12.624999+0000","last_undegraded":"2024-09-21T10:18:12.624999+0000","last_fullsized":"2024-09-21T10:18:12.624999+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:23:07.485461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":624,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.926342+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:18:16.926342+0000","last_peered":"2024-09-21T10:18:16.926342+0000","last_clean":"2024-09-21T10:18:16.926342+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:18:16.926342+0000","last_undegraded":"2024-09-21T10:18:16.926342+0000","last_fullsized":"2024-09-21T10:18:16.926342+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":249,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.646961+0000","last_change":"2024-09-21T10:14:30.702414+0000","last_active":"2024-09-21T10:18:16.646961+0000","last_peered":"2024-09-21T10:18:16.646961+0000","last_clean":"2024-09-21T10:18:16.646961+0000","last_became_active":"2024-09-21T10:14:30.702009+0000","last_became_peered":"2024-09-21T10:14:30.702009+0000","last_unstale":"2024-09-21T10:18:16.646961+0000","last_undegraded":"2024-09-21T10:18:16.646961+0000","last_fullsized":"2024-09-21T10:18:16.646961+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:50:48.396082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":11,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.635479+0000","last_change":"2024-09-21T10:18:12.633151+0000","last_active":"2024-09-21T10:18:12.635479+0000","last_peered":"2024-09-21T10:18:12.635479+0000","last_clean":"2024-09-21T10:18:12.635479+0000","last_became_active":"2024-09-21T10:18:12.632854+0000","last_became_peered":"2024-09-21T10:18:12.632854+0000","last_unstale":"2024-09-21T10:18:12.635479+0000","last_undegraded":"2024-09-21T10:18:12.635479+0000","last_fullsized":"2024-09-21T10:18:12.635479+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:14.602893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":79,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.625590+0000","last_change":"2024-09-21T10:14:30.720787+0000","last_active":"2024-09-21T10:18:12.625590+0000","last_peered":"2024-09-21T10:18:12.625590+0000","last_clean":"2024-09-21T10:18:12.625590+0000","last_became_active":"2024-09-21T10:14:30.706001+0000","last_became_peered":"2024-09-21T10:14:30.706001+0000","last_unstale":"2024-09-21T10:18:12.625590+0000","last_undegraded":"2024-09-21T10:18:12.625590+0000","last_fullsized":"2024-09-21T10:18:12.625590+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:40:11.987108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":11,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.637666+0000","last_change":"2024-09-21T10:18:12.636252+0000","last_active":"2024-09-21T10:18:12.637666+0000","last_peered":"2024-09-21T10:18:12.637666+0000","last_clean":"2024-09-21T10:18:12.637666+0000","last_became_active":"2024-09-21T10:18:12.635945+0000","last_became_peered":"2024-09-21T10:18:12.635945+0000","last_unstale":"2024-09-21T10:18:12.637666+0000","last_undegraded":"2024-09-21T10:18:12.637666+0000","last_fullsized":"2024-09-21T10:18:12.637666+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:59:29.464045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":79,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.626576+0000","last_change":"2024-09-21T10:14:30.722338+0000","last_active":"2024-09-21T10:18:12.626576+0000","last_peered":"2024-09-21T10:18:12.626576+0000","last_clean":"2024-09-21T10:18:12.626576+0000","last_became_active":"2024-09-21T10:14:30.708164+0000","last_became_peered":"2024-09-21T10:14:30.708164+0000","last_unstale":"2024-09-21T10:18:12.626576+0000","last_undegraded":"2024-09-21T10:18:12.626576+0000","last_fullsized":"2024-09-21T10:18:12.626576+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:32:13.825114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":606,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.210420+0000","last_change":"2024-09-21T10:09:23.840467+0000","last_active":"2024-09-21T10:18:14.210420+0000","last_peered":"2024-09-21T10:18:14.210420+0000","last_clean":"2024-09-21T10:18:14.210420+0000","last_became_active":"2024-09-21T10:09:23.840177+0000","last_became_peered":"2024-09-21T10:09:23.840177+0000","last_unstale":"2024-09-21T10:18:14.210420+0000","last_undegraded":"2024-09-21T10:18:14.210420+0000","last_fullsized":"2024-09-21T10:18:14.210420+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:17.887307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":596,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-09-21T10:18:14.210132+0000","last_change":"2024-09-21T10:11:05.451613+0000","last_active":"2024-09-21T10:18:14.210132+0000","last_peered":"2024-09-21T10:18:14.210132+0000","last_clean":"2024-09-21T10:18:14.210132+0000","last_became_active":"2024-09-21T10:11:05.451075+0000","last_became_peered":"2024-09-21T10:11:05.451075+0000","last_unstale":"2024-09-21T10:18:14.210132+0000","last_undegraded":"2024-09-21T10:18:14.210132+0000","last_fullsized":"2024-09-21T10:18:14.210132+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:18.505045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":75,"reported_epoch":368,"state":"active+clean","last_fresh":"2024-09-21T10:18:12.624609+0000","last_change":"2024-09-21T10:14:30.707548+0000","last_active":"2024-09-21T10:18:12.624609+0000","last_peered":"2024-09-21T10:18:12.624609+0000","last_clean":"2024-09-21T10:18:12.624609+0000","last_became_active":"2024-09-21T10:14:30.707232+0000","last_became_peered":"2024-09-21T10:14:30.707232+0000","last_unstale":"2024-09-21T10:18:12.624609+0000","last_undegraded":"2024-09-21T10:18:12.624609+0000","last_fullsized":"2024-09-21T10:18:12.624609+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:11:38.905767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":262,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-09-21T10:18:16.646089+0000","last_change":"2024-09-21T10:10:57.401588+0000","last_active":"2024-09-21T10:18:16.646089+0000","last_peered":"2024-09-21T10:18:16.646089+0000","last_clean":"2024-09-21T10:18:16.646089+0000","last_became_active":"2024-09-21T10:10:57.397724+0000","last_became_peered":"2024-09-21T10:10:57.397724+0000","last_unstale":"2024-09-21T10:18:16.646089+0000","last_undegraded":"2024-09-21T10:18:16.646089+0000","last_fullsized":"2024-09-21T10:18:16.646089+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:47.843630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":11856,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26580,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":670872,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":102,"num_write":20,"num_write_kb":222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":686385,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626911,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":905732,"kb_used_data":5364,"kb_used_omap":1,"kb_used_meta":900350,"kb_avail":92839420,"statfs":{"total":95995035648,"available":95067566080,"internally_reserved":0,"allocated":5492736,"data_stored":3230762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":921958744},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019369,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":900172,"kb_used_data":5048,"kb_used_omap":1,"kb_used_meta":895102,"kb_avail":92844980,"statfs":{"total":95995035648,"available":95073259520,"internally_reserved":0,"allocated":5169152,"data_stored":2903123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916584792},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":245,"seq":1052266987628,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":415364,"kb_used_data":5368,"kb_used_omap":8,"kb_used_meta":409975,"kb_avail":93329788,"statfs":{"total":95995035648,"available":95569702912,"internally_reserved":0,"allocated":5496832,"data_stored":3234987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8940,"internal_metadata":419814676},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":175,"seq":751619276977,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":938180,"kb_used_data":5048,"kb_used_omap":12,"kb_used_meta":933107,"kb_avail":92806972,"statfs":{"total":95995035648,"available":95034339328,"internally_reserved":0,"allocated":5169152,"data_stored":2903119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12845,"internal_metadata":955502035},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823140,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1494780,"kb_used_data":5288,"kb_used_omap":1,"kb_used_meta":1489470,"kb_avail":92250372,"statfs":{"total":95995035648,"available":94464380928,"internally_reserved":0,"allocated":5414912,"data_stored":3160966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1525217624},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879563,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1004852,"kb_used_data":5608,"kb_used_omap":4,"kb_used_meta":999227,"kb_avail":92740300,"statfs":{"total":95995035648,"available":94966067200,"internally_reserved":0,"allocated":5742592,"data_stored":3488727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":1023208892},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566404,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":965068,"kb_used_data":5312,"kb_used_omap":6,"kb_used_meta":959737,"kb_avail":92780084,"statfs":{"total":95995035648,"available":95006806016,"internally_reserved":0,"allocated":5439488,"data_stored":3170097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":982771111},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860749,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1069644,"kb_used_data":5056,"kb_used_omap":5,"kb_used_meta":1064570,"kb_avail":92675508,"statfs":{"total":95995035648,"available":94899720192,"internally_reserved":0,"allocated":5177344,"data_stored":2915194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":1090119919},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7236,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9976,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:18:25.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:18:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:24 smithi129 ceph-mon[88471]: pgmap v1257: 109 pgs: 109 active+clean; 29 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.5 MiB/s wr, 59 op/s; 6.5 KiB/s, 0 objects/s recovering 2024-09-21T10:18:25.241 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:25.276 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:25.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:25.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:25.377 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:25.399 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:25.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:25.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:25.500 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:25.534 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:25.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:25.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:25.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:25.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:25.679 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:25.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:25.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:25.817 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:25.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:25.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:25.919 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:25.954 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:25.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:25.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:26.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:26.057 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 4 16 13814 13798 215.578 158.562 0.00452333 0.00463129 2024-09-21T10:18:26.091 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:26.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:26.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:26.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:26.228 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:26.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:26.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:26.270 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:18:26.270 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:18:26.270 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.270 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:13283 13.9 13:97648176:::benchmark_data_smithi107_278655_object13282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-21T10:18:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:18:25.824408+0000", 2024-09-21T10:18:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0064562150000000004, 2024-09-21T10:18:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0064884959999999998, 2024-09-21T10:18:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:18:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:18:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:18:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 13283 2024-09-21T10:18:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.271 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.824408+0000", 2024-09-21T10:18:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.824408+0000", 2024-09-21T10:18:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.824410+0000", 2024-09-21T10:18:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.403e-06 2024-09-21T10:18:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.272 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:26.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.824433+0000", 2024-09-21T10:18:26.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3210999999999999e-05 2024-09-21T10:18:26.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:26.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.824454+0000", 2024-09-21T10:18:26.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0897999999999999e-05 2024-09-21T10:18:26.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:26.273 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.824462+0000", 2024-09-21T10:18:26.273 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6049999999999998e-06 2024-09-21T10:18:26.273 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.273 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.273 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.824717+0000", 2024-09-21T10:18:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00025429599999999999 2024-09-21T10:18:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:18:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.824850+0000", 2024-09-21T10:18:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013373500000000001 2024-09-21T10:18:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:18:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.824985+0000", 2024-09-21T10:18:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013476499999999999 2024-09-21T10:18:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:18:26.274 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.826511+0000", 2024-09-21T10:18:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0015256880000000001 2024-09-21T10:18:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.830759+0000", 2024-09-21T10:18:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0042486210000000002 2024-09-21T10:18:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:13287 13.9 13:9cdec7db:::benchmark_data_smithi107_278655_object13286:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-21T10:18:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:18:25.829034+0000", 2024-09-21T10:18:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.001829949, 2024-09-21T10:18:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001938834, 2024-09-21T10:18:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:18:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:18:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:18:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 13287 2024-09-21T10:18:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.829034+0000", 2024-09-21T10:18:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.276 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.829034+0000", 2024-09-21T10:18:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.829036+0000", 2024-09-21T10:18:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8920000000000001e-06 2024-09-21T10:18:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.829063+0000", 2024-09-21T10:18:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6407999999999999e-05 2024-09-21T10:18:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.277 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.829077+0000", 2024-09-21T10:18:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4729000000000001e-05 2024-09-21T10:18:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.829086+0000", 2024-09-21T10:18:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4780000000000008e-06 2024-09-21T10:18:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.829320+0000", 2024-09-21T10:18:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023421 2024-09-21T10:18:26.278 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:18:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.829477+0000", 2024-09-21T10:18:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015680199999999999 2024-09-21T10:18:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:18:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.829657+0000", 2024-09-21T10:18:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017971299999999999 2024-09-21T10:18:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:18:26.279 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:25.830709+0000", 2024-09-21T10:18:26.280 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001052124 2024-09-21T10:18:26.280 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:26.280 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:26.280 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:26.280 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:26.280 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-09-21T10:18:26.280 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 2 2024-09-21T10:18:26.280 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:18:26.330 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:26.366 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:26.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:26.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:26.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:26.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:26.502 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:26.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:26.603 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:26.639 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:26.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:26.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:26.709 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:26.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:26.751 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:18:26.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:26.799 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:26.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:26.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:26.935 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:26.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:26.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:27.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:27.057 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 5 15 16647 16632 207.868 177.125 0.00546523 0.00480586 2024-09-21T10:18:27.072 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:27.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:27.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:27.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:26 smithi107 ceph-mon[105022]: pgmap v1258: 109 pgs: 109 active+clean; 112 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 3.9 KiB/s rd, 13 MiB/s wr, 205 op/s; 6.0 KiB/s, 0 objects/s recovering 2024-09-21T10:18:27.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:26 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/288103690' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:18:27.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:27.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:26 smithi107 ceph-mon[102221]: pgmap v1258: 109 pgs: 109 active+clean; 112 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 3.9 KiB/s rd, 13 MiB/s wr, 205 op/s; 6.0 KiB/s, 0 objects/s recovering 2024-09-21T10:18:27.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:26 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/288103690' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:18:27.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:26 smithi129 ceph-mon[88471]: pgmap v1258: 109 pgs: 109 active+clean; 112 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 3.9 KiB/s rd, 13 MiB/s wr, 205 op/s; 6.0 KiB/s, 0 objects/s recovering 2024-09-21T10:18:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:26 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/288103690' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:18:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:27.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:27.212 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:27.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:27.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:27.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:27.346 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:27.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:27.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:27.448 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:27.485 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:27.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:27.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:27.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:18:27.587 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:27.629 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:27.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:27.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:27.731 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:27.766 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:27.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:27.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:27.868 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:27.889 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:27.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:27.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:27.991 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:28.027 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:28.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:28.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:28.057 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 6 16 19383 19367 201.711 170.938 0.00639258 0.00495118 2024-09-21T10:18:28.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:27 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:18:28.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:27 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/288103690' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:18:28.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:27 smithi107 ceph-mon[102221]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T10:18:28.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:27 smithi107 ceph-mon[102221]: 1.e scrub starts 2024-09-21T10:18:28.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:27 smithi107 ceph-mon[102221]: 1.e scrub ok 2024-09-21T10:18:28.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:27 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:18:28.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:27 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/288103690' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:18:28.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:27 smithi107 ceph-mon[105022]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T10:18:28.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:27 smithi107 ceph-mon[105022]: 1.e scrub starts 2024-09-21T10:18:28.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:27 smithi107 ceph-mon[105022]: 1.e scrub ok 2024-09-21T10:18:28.129 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:28.164 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:28.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:28.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:27 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:18:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:27 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/288103690' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:18:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:27 smithi129 ceph-mon[88471]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T10:18:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:27 smithi129 ceph-mon[88471]: 1.e scrub starts 2024-09-21T10:18:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:27 smithi129 ceph-mon[88471]: 1.e scrub ok 2024-09-21T10:18:28.266 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:28.301 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:28.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:28.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:28.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:28.425 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:28.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:28.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:28.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:28.563 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:28.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:28.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:28.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:28.688 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:28.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:28.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:28.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:28.824 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:28.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:28.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:28.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:28.960 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:28.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:28.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:29.057 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 7 16 21891 21875 195.286 156.75 0.00698045 0.00511468 2024-09-21T10:18:29.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:29.096 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:29.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:29.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:29.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:28 smithi107 ceph-mon[105022]: pgmap v1260: 109 pgs: 109 active+clean; 852 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-21T10:18:29.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:28 smithi107 ceph-mon[102221]: pgmap v1260: 109 pgs: 109 active+clean; 852 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-21T10:18:29.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:28 smithi129 ceph-mon[88471]: pgmap v1260: 109 pgs: 109 active+clean; 852 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-21T10:18:29.198 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:29.234 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:29.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:29.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:29.336 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:29.373 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:29.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:29.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:29.475 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:29.513 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:29.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:29.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:29.615 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:29.638 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:29.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:29.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:29.740 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:29.775 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:29.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:29.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:29.877 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:29.898 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:29.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:29.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:29.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:30.020 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:30.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:30.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:30.057 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 8 15 23943 23928 186.912 128.312 0.00527715 0.00534458 2024-09-21T10:18:30.122 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:30.154 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:30.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:30.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:30.256 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:30.291 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:30.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:30.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:30.393 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:30.437 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:30.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:30.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:30.539 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:30.576 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:30.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:30.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:18:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:18:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:18:30.678 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:30.714 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:30.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:30.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:30.816 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:30.858 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:30.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:30.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:30.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:30.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:30.997 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:30.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:31.057 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 9 16 26203 26187 181.831 141.188 0.00235004 0.00546567 2024-09-21T10:18:31.098 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:31.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:30 smithi107 ceph-mon[102221]: pgmap v1261: 109 pgs: 109 active+clean; 980 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-21T10:18:31.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:30 smithi107 ceph-mon[105022]: pgmap v1261: 109 pgs: 109 active+clean; 980 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-21T10:18:31.132 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:31.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:31.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:31.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:30 smithi129 ceph-mon[88471]: pgmap v1261: 109 pgs: 109 active+clean; 980 MiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-21T10:18:31.234 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:31.273 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:31.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:31.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:31.375 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:31.413 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:31.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:31.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:31.515 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:31.538 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:31.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:31.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:31.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:31.661 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:31.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:31.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:31.763 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:31.783 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:31.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:31.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:31.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:31.919 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:31.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:31.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:32.017 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:31 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:18:32.017 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:31 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:32.017 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:31 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:32.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:32.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:32.093 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:32.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:32.094 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 10 16 27607 27591 172.422 87.75 0.00922353 0.00579369 2024-09-21T10:18:32.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:31 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:18:32.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:31 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:32.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:31 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:32.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:31 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:18:32.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:31 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:32.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:31 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:32.110 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:18:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:18:31] "GET /metrics HTTP/1.1" 200 36785 "" "Prometheus/2.51.0" 2024-09-21T10:18:32.195 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:32.217 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:32.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:32.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:32.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:32.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:32.341 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:32.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:32.442 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:32.469 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:32.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:32.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:32.570 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:32.608 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:32.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:32.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:32.709 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:32.741 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:32.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:32.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:32.844 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:32.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:32.865 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:32.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:32 smithi129 ceph-mon[88471]: pgmap v1262: 109 pgs: 109 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-21T10:18:32.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:33.001 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:33.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:33.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:33.058 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 11 16 28848 28832 163.799 77.5625 0.00410029 0.00609995 2024-09-21T10:18:33.104 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:33.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:32 smithi107 ceph-mon[105022]: pgmap v1262: 109 pgs: 109 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-21T10:18:33.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:32 smithi107 ceph-mon[102221]: pgmap v1262: 109 pgs: 109 active+clean; 1.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-21T10:18:33.124 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:33.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:33.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:33.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:33.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:33.264 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:33.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:33.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:33.406 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:33.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:33.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:33.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:33.548 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:33.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:33.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:33.649 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:33.699 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:33.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:33.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:33.801 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:33.842 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:33.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:33.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:33.944 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:33.980 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:33.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:33.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:34.058 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 12 16 30902 30886 160.846 128.375 0.00837626 0.00621007 2024-09-21T10:18:34.081 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:34.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:34.126 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:34.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:34.227 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:34.268 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:34.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:34.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:34.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:34.403 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:34.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:34.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:34.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:34 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:34.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:34 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:34.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:34 smithi129 ceph-mon[88471]: pgmap v1263: 109 pgs: 109 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 165 MiB/s wr, 2.63k op/s 2024-09-21T10:18:34.504 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:34.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:34 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:34.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:34 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:34.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:34 smithi107 ceph-mon[102221]: pgmap v1263: 109 pgs: 109 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 165 MiB/s wr, 2.63k op/s 2024-09-21T10:18:34.530 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:34 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:34.530 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:34 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:34.530 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:34 smithi107 ceph-mon[105022]: pgmap v1263: 109 pgs: 109 active+clean; 1.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 165 MiB/s wr, 2.63k op/s 2024-09-21T10:18:34.541 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:34.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:34.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:34.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:34.670 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:34.699 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:34.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:34.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:34.801 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:34.824 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:34.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:34.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:34.926 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:34.961 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:34.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:34.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:35.058 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 13 16 32935 32919 158.247 127.062 0.00287714 0.00631329 2024-09-21T10:18:35.064 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:35.098 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:35.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:35.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:35.200 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:35.243 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:35.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:35.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:35.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:35.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:35.381 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:35.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:35.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:35.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:18:35.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:35.534 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:35.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:35.578 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:18:35.579 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:18:35.579 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:18:35.579 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:18:35.598 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:35.619 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:18:35.619 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:18:35.636 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:35.654 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:18:35.655 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:18:35.668 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:35.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:35.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:35.689 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:18:35.690 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:18:35.721 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:18:35.721 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:18:35.751 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:18:35.751 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:18:35.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:35.785 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:18:35.785 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:18:35.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:35.819 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:35.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:35.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:35.955 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:35.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:35.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:36.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:36.059 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 14 16 34937 34921 155.876 125.125 0.00571683 0.00640961 2024-09-21T10:18:36.089 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:36.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:36.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:36.191 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:36.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:36.236 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:36.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:36.337 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:36.375 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:36.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:36.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:36.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:36.510 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:36.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:36.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:36.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:36 smithi107 ceph-mon[102221]: pgmap v1264: 109 pgs: 109 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-09-21T10:18:36.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:36 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/199640319' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:18:36.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:36 smithi107 ceph-mon[105022]: pgmap v1264: 109 pgs: 109 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-09-21T10:18:36.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:36 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/199640319' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:18:36.612 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:36.659 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:36.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:36.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:36 smithi129 ceph-mon[88471]: pgmap v1264: 109 pgs: 109 active+clean; 1.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-09-21T10:18:36.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:36 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/199640319' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:18:36.761 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:36.815 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:36.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:36.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:36.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:36.956 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:36.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:36.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:37.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:37.059 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 15 16 36988 36972 154.029 128.188 0.0156193 0.00648721 2024-09-21T10:18:37.088 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:37.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:37.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:37.190 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:37.227 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:37.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:37.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:37.329 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:37.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:37.368 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:37.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:37.469 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:37.511 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:37.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:37.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:37.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:37.649 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:37.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:37.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:37.751 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:37.793 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:37.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:37.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:37.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:37.929 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:37.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:37.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:38.031 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:38.059 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 16 16 38963 38947 152.116 123.438 0.00733218 0.00656817 2024-09-21T10:18:38.071 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:38.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:38.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:38.173 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:38.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:38.232 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:38.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:38.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:38.367 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:38.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:38.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:38.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:38.499 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:38.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:38.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:38.601 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:38.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:38 smithi107 ceph-mon[102221]: pgmap v1265: 109 pgs: 109 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 189 MiB/s wr, 3.03k op/s 2024-09-21T10:18:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:38 smithi107 ceph-mon[105022]: pgmap v1265: 109 pgs: 109 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 189 MiB/s wr, 3.03k op/s 2024-09-21T10:18:38.636 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:38.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:38.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:38.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:38.739 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:18:38.771 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:38.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:38.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:38.876 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:38 smithi129 ceph-mon[88471]: pgmap v1265: 109 pgs: 109 active+clean; 2.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 189 MiB/s wr, 3.03k op/s 2024-09-21T10:18:38.921 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:38.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:38.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:39.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:39.058 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:39.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:39.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:39.062 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 17 16 40951 40935 150.458 124.25 0.00676474 0.00664218 2024-09-21T10:18:39.160 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:39.208 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:39.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:39.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:39.310 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:39.355 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:39.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:39.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:39.457 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:39.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:39.497 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:39.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:39.598 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:39.624 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:39.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:39.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:39.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:39.763 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:39.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:39.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:39.865 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:39.900 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:39.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:39.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:39.930 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:39 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:39.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:39 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:39.931 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:39 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:39.931 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:39 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:39.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:39 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:39.932 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:39 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:40.001 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:40.040 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:40.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:40.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:40.061 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 18 16 42656 42640 148.019 106.562 0.0048087 0.00675024 2024-09-21T10:18:40.113 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:18:40.113 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:18:40.142 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:40.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:39 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:40.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:39 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:40.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:39 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:40.186 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:40.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:40.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:40.288 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:40.342 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:18:40.343 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:40.347 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:18:40.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:40.348 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:40.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:40.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:40.480 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:40.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:40.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:40.491 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:18:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:18:40.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:18:40.582 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:40.623 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:40.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:40.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:40.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:40.757 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:40.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:40.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:40.859 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:40.896 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:40.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:40.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:40.923 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:18:40.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:41.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:41.034 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:41.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:41.061 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 19 16 44246 44230 145.459 99.375 0.00875854 0.00686955 2024-09-21T10:18:41.135 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:40 smithi129 ceph-mon[88471]: pgmap v1266: 109 pgs: 109 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-21T10:18:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:40 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:41.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:41.174 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:41.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:41.222 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:40 smithi107 ceph-mon[105022]: pgmap v1266: 109 pgs: 109 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-21T10:18:41.222 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:40 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:41.222 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:40 smithi107 ceph-mon[102221]: pgmap v1266: 109 pgs: 109 active+clean; 2.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-21T10:18:41.222 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:40 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:41.275 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:41.313 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:41.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:41.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:41.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:41.451 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:41.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:41.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:41.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:41.586 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:41.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:41.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:41.688 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:41.723 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:41.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:41.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:41.824 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:41.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:18:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:18:41] "GET /metrics HTTP/1.1" 200 36782 "" "Prometheus/2.51.0" 2024-09-21T10:18:41.862 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:41.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:41.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:41.964 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:41.998 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:41.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:41.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:42.061 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:18:42.060131+0000 min lat: 0.00115007 max lat: 0.324531 avg lat: 0.0069845 2024-09-21T10:18:42.061 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:18:42.061 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 20 16 45807 45791 143.064 97.5625 0.0149609 0.0069845 2024-09-21T10:18:42.082 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1266,"stamp":"2024-09-21T10:18:39.433280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2379213170,"num_objects":36320,"num_object_clones":0,"num_object_copies":108960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36320,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":36371,"num_write_kb":2322996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":644571,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38453,"ondisk_log_size":38453,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14385900,"kb_used_data":6668908,"kb_used_omap":43,"kb_used_meta":7716820,"kb_avail":735575316,"statfs":{"total":767960285184,"available":753229123584,"internally_reserved":0,"allocated":6828961792,"data_stored":6810710214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44223,"internal_metadata":7902024513},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,4,5,6,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1486001965,"num_objects":22675,"num_object_clones":0,"num_object_copies":68025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22675,"num_whiteouts":0,"num_read":-14,"num_read_kb":-34,"num_write":22668,"num_write_kb":1451126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":-53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003771"},"pg_stats":[{"pgid":"1.11","version":"77'100","reported_seq":261,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.782633+0000","last_change":"2024-09-21T10:10:57.397786+0000","last_active":"2024-09-21T10:18:26.782633+0000","last_peered":"2024-09-21T10:18:26.782633+0000","last_clean":"2024-09-21T10:18:26.782633+0000","last_became_active":"2024-09-21T10:10:57.396742+0000","last_became_peered":"2024-09-21T10:10:57.396742+0000","last_unstale":"2024-09-21T10:18:26.782633+0000","last_undegraded":"2024-09-21T10:18:26.782633+0000","last_fullsized":"2024-09-21T10:18:26.782633+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:53.631923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:53.631923+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:33:35.364595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":600,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.758083+0000","last_change":"2024-09-21T10:03:38.389577+0000","last_active":"2024-09-21T10:18:26.758083+0000","last_peered":"2024-09-21T10:18:26.758083+0000","last_clean":"2024-09-21T10:18:26.758083+0000","last_became_active":"2024-09-21T10:03:38.389292+0000","last_became_peered":"2024-09-21T10:03:38.389292+0000","last_unstale":"2024-09-21T10:18:26.758083+0000","last_undegraded":"2024-09-21T10:18:26.758083+0000","last_fullsized":"2024-09-21T10:18:26.758083+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:32.113472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":257,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655212+0000","last_change":"2024-09-21T10:14:30.717695+0000","last_active":"2024-09-21T10:18:19.655212+0000","last_peered":"2024-09-21T10:18:19.655212+0000","last_clean":"2024-09-21T10:18:19.655212+0000","last_became_active":"2024-09-21T10:14:30.705952+0000","last_became_peered":"2024-09-21T10:14:30.705952+0000","last_unstale":"2024-09-21T10:18:19.655212+0000","last_undegraded":"2024-09-21T10:18:19.655212+0000","last_fullsized":"2024-09-21T10:18:19.655212+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:08:18.863430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'100","reported_seq":270,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.656368+0000","last_change":"2024-09-21T10:11:05.438031+0000","last_active":"2024-09-21T10:18:19.656368+0000","last_peered":"2024-09-21T10:18:19.656368+0000","last_clean":"2024-09-21T10:18:19.656368+0000","last_became_active":"2024-09-21T10:11:05.437637+0000","last_became_peered":"2024-09-21T10:11:05.437637+0000","last_unstale":"2024-09-21T10:18:19.656368+0000","last_undegraded":"2024-09-21T10:18:19.656368+0000","last_fullsized":"2024-09-21T10:18:19.656368+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.798577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.798577+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:42.343217+0000","scrub_duration":0.0023622640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":362,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655812+0000","last_change":"2024-09-21T10:10:57.392782+0000","last_active":"2024-09-21T10:18:19.655812+0000","last_peered":"2024-09-21T10:18:19.655812+0000","last_clean":"2024-09-21T10:18:19.655812+0000","last_became_active":"2024-09-21T10:10:57.392305+0000","last_became_peered":"2024-09-21T10:10:57.392305+0000","last_unstale":"2024-09-21T10:18:19.655812+0000","last_undegraded":"2024-09-21T10:18:19.655812+0000","last_fullsized":"2024-09-21T10:18:19.655812+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:42:01.387622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":520,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.781427+0000","last_change":"2024-09-21T10:09:23.846703+0000","last_active":"2024-09-21T10:18:26.781427+0000","last_peered":"2024-09-21T10:18:26.781427+0000","last_clean":"2024-09-21T10:18:26.781427+0000","last_became_active":"2024-09-21T10:09:23.846415+0000","last_became_peered":"2024-09-21T10:09:23.846415+0000","last_unstale":"2024-09-21T10:18:26.781427+0000","last_undegraded":"2024-09-21T10:18:26.781427+0000","last_fullsized":"2024-09-21T10:18:26.781427+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:23.129285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'100","reported_seq":262,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.757231+0000","last_change":"2024-09-21T10:18:12.637168+0000","last_active":"2024-09-21T10:18:26.757231+0000","last_peered":"2024-09-21T10:18:26.757231+0000","last_clean":"2024-09-21T10:18:26.757231+0000","last_became_active":"2024-09-21T10:18:12.636410+0000","last_became_peered":"2024-09-21T10:18:12.636410+0000","last_unstale":"2024-09-21T10:18:26.757231+0000","last_undegraded":"2024-09-21T10:18:26.757231+0000","last_fullsized":"2024-09-21T10:18:26.757231+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.433227+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:59:07.270049+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":246,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.781805+0000","last_change":"2024-09-21T10:09:49.922489+0000","last_active":"2024-09-21T10:18:26.781805+0000","last_peered":"2024-09-21T10:18:26.781805+0000","last_clean":"2024-09-21T10:18:26.781805+0000","last_became_active":"2024-09-21T10:09:49.921629+0000","last_became_peered":"2024-09-21T10:09:49.921629+0000","last_unstale":"2024-09-21T10:18:26.781805+0000","last_undegraded":"2024-09-21T10:18:26.781805+0000","last_fullsized":"2024-09-21T10:18:26.781805+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:50.611979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":253,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655252+0000","last_change":"2024-09-21T10:14:30.709707+0000","last_active":"2024-09-21T10:18:19.655252+0000","last_peered":"2024-09-21T10:18:19.655252+0000","last_clean":"2024-09-21T10:18:19.655252+0000","last_became_active":"2024-09-21T10:14:30.709353+0000","last_became_peered":"2024-09-21T10:14:30.709353+0000","last_unstale":"2024-09-21T10:18:19.655252+0000","last_undegraded":"2024-09-21T10:18:19.655252+0000","last_fullsized":"2024-09-21T10:18:19.655252+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:12.822947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"77'100","reported_seq":151,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.757333+0000","last_change":"2024-09-21T10:15:02.842044+0000","last_active":"2024-09-21T10:18:26.757333+0000","last_peered":"2024-09-21T10:18:26.757333+0000","last_clean":"2024-09-21T10:18:26.757333+0000","last_became_active":"2024-09-21T10:15:01.478273+0000","last_became_peered":"2024-09-21T10:15:01.478273+0000","last_unstale":"2024-09-21T10:18:26.757333+0000","last_undegraded":"2024-09-21T10:18:26.757333+0000","last_fullsized":"2024-09-21T10:18:26.757333+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:15:02.841968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:15:02.841968+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:38:23.910899+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":600,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.763219+0000","last_change":"2024-09-21T10:03:38.381552+0000","last_active":"2024-09-21T10:18:26.763219+0000","last_peered":"2024-09-21T10:18:26.763219+0000","last_clean":"2024-09-21T10:18:26.763219+0000","last_became_active":"2024-09-21T10:03:38.380898+0000","last_became_peered":"2024-09-21T10:03:38.380898+0000","last_unstale":"2024-09-21T10:18:26.763219+0000","last_undegraded":"2024-09-21T10:18:26.763219+0000","last_fullsized":"2024-09-21T10:18:26.763219+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:53.409606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":430,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.797378+0000","last_change":"2024-09-21T10:10:57.459057+0000","last_active":"2024-09-21T10:18:19.797378+0000","last_peered":"2024-09-21T10:18:19.797378+0000","last_clean":"2024-09-21T10:18:19.797378+0000","last_became_active":"2024-09-21T10:10:57.451776+0000","last_became_peered":"2024-09-21T10:10:57.451776+0000","last_unstale":"2024-09-21T10:18:19.797378+0000","last_undegraded":"2024-09-21T10:18:19.797378+0000","last_fullsized":"2024-09-21T10:18:19.797378+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:00.138317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"372'2249","reported_seq":2262,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:36.842651+0000","last_change":"2024-09-21T10:18:12.636805+0000","last_active":"2024-09-21T10:18:36.842651+0000","last_peered":"2024-09-21T10:18:36.842651+0000","last_clean":"2024-09-21T10:18:36.842651+0000","last_became_active":"2024-09-21T10:18:12.636542+0000","last_became_peered":"2024-09-21T10:18:12.636542+0000","last_unstale":"2024-09-21T10:18:36.842651+0000","last_undegraded":"2024-09-21T10:18:36.842651+0000","last_fullsized":"2024-09-21T10:18:36.842651+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":2249,"log_dups_size":0,"ondisk_log_size":2249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:29.211883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147390464,"num_objects":2249,"num_object_clones":0,"num_object_copies":6747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2249,"num_write_kb":143936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"77'100","reported_seq":265,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655559+0000","last_change":"2024-09-21T10:14:30.708275+0000","last_active":"2024-09-21T10:18:19.655559+0000","last_peered":"2024-09-21T10:18:19.655559+0000","last_clean":"2024-09-21T10:18:19.655559+0000","last_became_active":"2024-09-21T10:14:30.707763+0000","last_became_peered":"2024-09-21T10:14:30.707763+0000","last_unstale":"2024-09-21T10:18:19.655559+0000","last_undegraded":"2024-09-21T10:18:19.655559+0000","last_fullsized":"2024-09-21T10:18:19.655559+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:55.259385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:55.259385+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:41.322447+0000","scrub_duration":0.0015667260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":300,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.656087+0000","last_change":"2024-09-21T10:14:30.720250+0000","last_active":"2024-09-21T10:18:19.656087+0000","last_peered":"2024-09-21T10:18:19.656087+0000","last_clean":"2024-09-21T10:18:19.656087+0000","last_became_active":"2024-09-21T10:14:30.707036+0000","last_became_peered":"2024-09-21T10:14:30.707036+0000","last_unstale":"2024-09-21T10:18:19.656087+0000","last_undegraded":"2024-09-21T10:18:19.656087+0000","last_fullsized":"2024-09-21T10:18:19.656087+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:17.438534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":343,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.656090+0000","last_change":"2024-09-21T10:15:01.476521+0000","last_active":"2024-09-21T10:18:19.656090+0000","last_peered":"2024-09-21T10:18:19.656090+0000","last_clean":"2024-09-21T10:18:19.656090+0000","last_became_active":"2024-09-21T10:15:01.476083+0000","last_became_peered":"2024-09-21T10:15:01.476083+0000","last_unstale":"2024-09-21T10:18:19.656090+0000","last_undegraded":"2024-09-21T10:18:19.656090+0000","last_fullsized":"2024-09-21T10:18:19.656090+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:50.230443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.0","version":"372'2350","reported_seq":2363,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:36.950943+0000","last_change":"2024-09-21T10:18:12.633194+0000","last_active":"2024-09-21T10:18:36.950943+0000","last_peered":"2024-09-21T10:18:36.950943+0000","last_clean":"2024-09-21T10:18:36.950943+0000","last_became_active":"2024-09-21T10:18:12.632915+0000","last_became_peered":"2024-09-21T10:18:12.632915+0000","last_unstale":"2024-09-21T10:18:36.950943+0000","last_undegraded":"2024-09-21T10:18:36.950943+0000","last_fullsized":"2024-09-21T10:18:36.950943+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":2350,"log_dups_size":0,"ondisk_log_size":2350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:13:26.865830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154009600,"num_objects":2350,"num_object_clones":0,"num_object_copies":7050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2350,"num_write_kb":150400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"77'100","reported_seq":407,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.757922+0000","last_change":"2024-09-21T10:09:23.839211+0000","last_active":"2024-09-21T10:18:26.757922+0000","last_peered":"2024-09-21T10:18:26.757922+0000","last_clean":"2024-09-21T10:18:26.757922+0000","last_became_active":"2024-09-21T10:09:23.838779+0000","last_became_peered":"2024-09-21T10:09:23.838779+0000","last_unstale":"2024-09-21T10:18:26.757922+0000","last_undegraded":"2024-09-21T10:18:26.757922+0000","last_fullsized":"2024-09-21T10:18:26.757922+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.981336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.981336+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:03:15.566873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":279,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655716+0000","last_change":"2024-09-21T10:18:12.643154+0000","last_active":"2024-09-21T10:18:19.655716+0000","last_peered":"2024-09-21T10:18:19.655716+0000","last_clean":"2024-09-21T10:18:19.655716+0000","last_became_active":"2024-09-21T10:18:12.636287+0000","last_became_peered":"2024-09-21T10:18:12.636287+0000","last_unstale":"2024-09-21T10:18:19.655716+0000","last_undegraded":"2024-09-21T10:18:19.655716+0000","last_fullsized":"2024-09-21T10:18:19.655716+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:45:31.008309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":78,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.656117+0000","last_change":"2024-09-21T10:14:30.706497+0000","last_active":"2024-09-21T10:18:19.656117+0000","last_peered":"2024-09-21T10:18:19.656117+0000","last_clean":"2024-09-21T10:18:19.656117+0000","last_became_active":"2024-09-21T10:14:30.706202+0000","last_became_peered":"2024-09-21T10:14:30.706202+0000","last_unstale":"2024-09-21T10:18:19.656117+0000","last_undegraded":"2024-09-21T10:18:19.656117+0000","last_fullsized":"2024-09-21T10:18:19.656117+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:42:57.912707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3","version":"372'2443","reported_seq":2463,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:37.555404+0000","last_change":"2024-09-21T10:18:12.641462+0000","last_active":"2024-09-21T10:18:37.555404+0000","last_peered":"2024-09-21T10:18:37.555404+0000","last_clean":"2024-09-21T10:18:37.555404+0000","last_became_active":"2024-09-21T10:18:12.641056+0000","last_became_peered":"2024-09-21T10:18:12.641056+0000","last_unstale":"2024-09-21T10:18:37.555404+0000","last_undegraded":"2024-09-21T10:18:37.555404+0000","last_fullsized":"2024-09-21T10:18:37.555404+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":2443,"log_dups_size":0,"ondisk_log_size":2443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:49:09.395317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160104448,"num_objects":2443,"num_object_clones":0,"num_object_copies":7329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2443,"num_write_kb":156352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"77'100","reported_seq":365,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.798086+0000","last_change":"2024-09-21T10:14:30.709997+0000","last_active":"2024-09-21T10:18:19.798086+0000","last_peered":"2024-09-21T10:18:19.798086+0000","last_clean":"2024-09-21T10:18:19.798086+0000","last_became_active":"2024-09-21T10:14:30.709580+0000","last_became_peered":"2024-09-21T10:14:30.709580+0000","last_unstale":"2024-09-21T10:18:19.798086+0000","last_undegraded":"2024-09-21T10:18:19.798086+0000","last_fullsized":"2024-09-21T10:18:19.798086+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.289859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.289859+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:27:50.110602+0000","scrub_duration":0.001251809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":251,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655372+0000","last_change":"2024-09-21T10:10:57.460732+0000","last_active":"2024-09-21T10:18:19.655372+0000","last_peered":"2024-09-21T10:18:19.655372+0000","last_clean":"2024-09-21T10:18:19.655372+0000","last_became_active":"2024-09-21T10:10:57.454689+0000","last_became_peered":"2024-09-21T10:10:57.454689+0000","last_unstale":"2024-09-21T10:18:19.655372+0000","last_undegraded":"2024-09-21T10:18:19.655372+0000","last_fullsized":"2024-09-21T10:18:19.655372+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:12.035022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":371,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.797281+0000","last_change":"2024-09-21T10:03:38.378384+0000","last_active":"2024-09-21T10:18:19.797281+0000","last_peered":"2024-09-21T10:18:19.797281+0000","last_clean":"2024-09-21T10:18:19.797281+0000","last_became_active":"2024-09-21T10:03:38.378072+0000","last_became_peered":"2024-09-21T10:03:38.378072+0000","last_unstale":"2024-09-21T10:18:19.797281+0000","last_undegraded":"2024-09-21T10:18:19.797281+0000","last_fullsized":"2024-09-21T10:18:19.797281+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:54:37.039622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"372'2314","reported_seq":2334,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:37.399563+0000","last_change":"2024-09-21T10:18:13.037514+0000","last_active":"2024-09-21T10:18:37.399563+0000","last_peered":"2024-09-21T10:18:37.399563+0000","last_clean":"2024-09-21T10:18:37.399563+0000","last_became_active":"2024-09-21T10:18:13.037222+0000","last_became_peered":"2024-09-21T10:18:13.037222+0000","last_unstale":"2024-09-21T10:18:37.399563+0000","last_undegraded":"2024-09-21T10:18:37.399563+0000","last_fullsized":"2024-09-21T10:18:37.399563+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":2314,"log_dups_size":0,"ondisk_log_size":2314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:00:30.961667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151650304,"num_objects":2314,"num_object_clones":0,"num_object_copies":6942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2314,"num_write_kb":148096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"77'100","reported_seq":24,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:27.085707+0000","last_change":"2024-09-21T10:18:27.085659+0000","last_active":"2024-09-21T10:18:27.085707+0000","last_peered":"2024-09-21T10:18:27.085707+0000","last_clean":"2024-09-21T10:18:27.085707+0000","last_became_active":"2024-09-21T10:18:12.633342+0000","last_became_peered":"2024-09-21T10:18:12.633342+0000","last_unstale":"2024-09-21T10:18:27.085707+0000","last_undegraded":"2024-09-21T10:18:27.085707+0000","last_fullsized":"2024-09-21T10:18:27.085707+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:18:27.085525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:18:27.085525+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:06:53.238321+0000","scrub_duration":0.0036485279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":404,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.758216+0000","last_change":"2024-09-21T10:07:57.551556+0000","last_active":"2024-09-21T10:18:26.758216+0000","last_peered":"2024-09-21T10:18:26.758216+0000","last_clean":"2024-09-21T10:18:26.758216+0000","last_became_active":"2024-09-21T10:07:57.551249+0000","last_became_peered":"2024-09-21T10:07:57.551249+0000","last_unstale":"2024-09-21T10:18:26.758216+0000","last_undegraded":"2024-09-21T10:18:26.758216+0000","last_fullsized":"2024-09-21T10:18:26.758216+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:57.557152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":508,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.781719+0000","last_change":"2024-09-21T10:14:30.711105+0000","last_active":"2024-09-21T10:18:26.781719+0000","last_peered":"2024-09-21T10:18:26.781719+0000","last_clean":"2024-09-21T10:18:26.781719+0000","last_became_active":"2024-09-21T10:14:30.710759+0000","last_became_peered":"2024-09-21T10:14:30.710759+0000","last_unstale":"2024-09-21T10:18:26.781719+0000","last_undegraded":"2024-09-21T10:18:26.781719+0000","last_fullsized":"2024-09-21T10:18:26.781719+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:40.690042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"372'2263","reported_seq":2276,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:36.849179+0000","last_change":"2024-09-21T10:18:13.037771+0000","last_active":"2024-09-21T10:18:36.849179+0000","last_peered":"2024-09-21T10:18:36.849179+0000","last_clean":"2024-09-21T10:18:36.849179+0000","last_became_active":"2024-09-21T10:18:13.037425+0000","last_became_peered":"2024-09-21T10:18:13.037425+0000","last_unstale":"2024-09-21T10:18:36.849179+0000","last_undegraded":"2024-09-21T10:18:36.849179+0000","last_fullsized":"2024-09-21T10:18:36.849179+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":2263,"log_dups_size":0,"ondisk_log_size":2263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:34:43.436187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148307968,"num_objects":2263,"num_object_clones":0,"num_object_copies":6789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2263,"num_write_kb":144832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":245,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655348+0000","last_change":"2024-09-21T10:04:53.355975+0000","last_active":"2024-09-21T10:18:19.655348+0000","last_peered":"2024-09-21T10:18:19.655348+0000","last_clean":"2024-09-21T10:18:19.655348+0000","last_became_active":"2024-09-21T10:04:42.547505+0000","last_became_peered":"2024-09-21T10:04:42.547505+0000","last_unstale":"2024-09-21T10:18:19.655348+0000","last_undegraded":"2024-09-21T10:18:19.655348+0000","last_fullsized":"2024-09-21T10:18:19.655348+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.355912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.355912+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:43.456830+0000","scrub_duration":0.0017980030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":611,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.758637+0000","last_change":"2024-09-21T10:14:30.715303+0000","last_active":"2024-09-21T10:18:26.758637+0000","last_peered":"2024-09-21T10:18:26.758637+0000","last_clean":"2024-09-21T10:18:26.758637+0000","last_became_active":"2024-09-21T10:14:30.703664+0000","last_became_peered":"2024-09-21T10:14:30.703664+0000","last_unstale":"2024-09-21T10:18:26.758637+0000","last_undegraded":"2024-09-21T10:18:26.758637+0000","last_fullsized":"2024-09-21T10:18:26.758637+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:28.967906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":78,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655572+0000","last_change":"2024-09-21T10:14:30.705927+0000","last_active":"2024-09-21T10:18:19.655572+0000","last_peered":"2024-09-21T10:18:19.655572+0000","last_clean":"2024-09-21T10:18:19.655572+0000","last_became_active":"2024-09-21T10:14:30.705112+0000","last_became_peered":"2024-09-21T10:14:30.705112+0000","last_unstale":"2024-09-21T10:18:19.655572+0000","last_undegraded":"2024-09-21T10:18:19.655572+0000","last_fullsized":"2024-09-21T10:18:19.655572+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:38:33.962199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.4","version":"372'2285","reported_seq":2298,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:36.838261+0000","last_change":"2024-09-21T10:18:12.639080+0000","last_active":"2024-09-21T10:18:36.838261+0000","last_peered":"2024-09-21T10:18:36.838261+0000","last_clean":"2024-09-21T10:18:36.838261+0000","last_became_active":"2024-09-21T10:18:12.638789+0000","last_became_peered":"2024-09-21T10:18:12.638789+0000","last_unstale":"2024-09-21T10:18:36.838261+0000","last_undegraded":"2024-09-21T10:18:36.838261+0000","last_fullsized":"2024-09-21T10:18:36.838261+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":2285,"log_dups_size":0,"ondisk_log_size":2285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:53.178338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149749760,"num_objects":2285,"num_object_clones":0,"num_object_copies":6855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2285,"num_write_kb":146240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":360,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.763112+0000","last_change":"2024-09-21T10:10:57.403722+0000","last_active":"2024-09-21T10:18:26.763112+0000","last_peered":"2024-09-21T10:18:26.763112+0000","last_clean":"2024-09-21T10:18:26.763112+0000","last_became_active":"2024-09-21T10:10:57.394700+0000","last_became_peered":"2024-09-21T10:10:57.394700+0000","last_unstale":"2024-09-21T10:18:26.763112+0000","last_undegraded":"2024-09-21T10:18:26.763112+0000","last_fullsized":"2024-09-21T10:18:26.763112+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:02.152981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:02.152981+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:52:34.887763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":355,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.797455+0000","last_change":"2024-09-21T10:14:30.707046+0000","last_active":"2024-09-21T10:18:19.797455+0000","last_peered":"2024-09-21T10:18:19.797455+0000","last_clean":"2024-09-21T10:18:19.797455+0000","last_became_active":"2024-09-21T10:14:30.706753+0000","last_became_peered":"2024-09-21T10:14:30.706753+0000","last_unstale":"2024-09-21T10:18:19.797455+0000","last_undegraded":"2024-09-21T10:18:19.797455+0000","last_fullsized":"2024-09-21T10:18:19.797455+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:58.136096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655985+0000","last_change":"2024-09-21T10:18:13.037945+0000","last_active":"2024-09-21T10:18:19.655985+0000","last_peered":"2024-09-21T10:18:19.655985+0000","last_clean":"2024-09-21T10:18:19.655985+0000","last_became_active":"2024-09-21T10:18:13.037550+0000","last_became_peered":"2024-09-21T10:18:13.037550+0000","last_unstale":"2024-09-21T10:18:19.655985+0000","last_undegraded":"2024-09-21T10:18:19.655985+0000","last_fullsized":"2024-09-21T10:18:19.655985+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:24:24.702636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.7","version":"372'2315","reported_seq":2328,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:36.963904+0000","last_change":"2024-09-21T10:18:12.635921+0000","last_active":"2024-09-21T10:18:36.963904+0000","last_peered":"2024-09-21T10:18:36.963904+0000","last_clean":"2024-09-21T10:18:36.963904+0000","last_became_active":"2024-09-21T10:18:12.635624+0000","last_became_peered":"2024-09-21T10:18:12.635624+0000","last_unstale":"2024-09-21T10:18:36.963904+0000","last_undegraded":"2024-09-21T10:18:36.963904+0000","last_fullsized":"2024-09-21T10:18:36.963904+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":2315,"log_dups_size":0,"ondisk_log_size":2315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:08:14.392867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151715840,"num_objects":2315,"num_object_clones":0,"num_object_copies":6945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2315,"num_write_kb":148160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"77'100","reported_seq":253,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.757175+0000","last_change":"2024-09-21T10:14:05.474100+0000","last_active":"2024-09-21T10:18:26.757175+0000","last_peered":"2024-09-21T10:18:26.757175+0000","last_clean":"2024-09-21T10:18:26.757175+0000","last_became_active":"2024-09-21T10:14:05.473336+0000","last_became_peered":"2024-09-21T10:14:05.473336+0000","last_unstale":"2024-09-21T10:18:26.757175+0000","last_undegraded":"2024-09-21T10:18:26.757175+0000","last_fullsized":"2024-09-21T10:18:26.757175+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.540323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.540323+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:13:30.035766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.654915+0000","last_change":"2024-09-21T10:18:12.629311+0000","last_active":"2024-09-21T10:18:19.654915+0000","last_peered":"2024-09-21T10:18:19.654915+0000","last_clean":"2024-09-21T10:18:19.654915+0000","last_became_active":"2024-09-21T10:18:12.629001+0000","last_became_peered":"2024-09-21T10:18:12.629001+0000","last_unstale":"2024-09-21T10:18:19.654915+0000","last_undegraded":"2024-09-21T10:18:19.654915+0000","last_fullsized":"2024-09-21T10:18:19.654915+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:09:17.165751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":78,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.656599+0000","last_change":"2024-09-21T10:14:30.706905+0000","last_active":"2024-09-21T10:18:19.656599+0000","last_peered":"2024-09-21T10:18:19.656599+0000","last_clean":"2024-09-21T10:18:19.656599+0000","last_became_active":"2024-09-21T10:14:30.706614+0000","last_became_peered":"2024-09-21T10:14:30.706614+0000","last_unstale":"2024-09-21T10:18:19.656599+0000","last_undegraded":"2024-09-21T10:18:19.656599+0000","last_fullsized":"2024-09-21T10:18:19.656599+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:26:33.014993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.6","version":"372'2267","reported_seq":2280,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:36.772896+0000","last_change":"2024-09-21T10:18:13.038010+0000","last_active":"2024-09-21T10:18:36.772896+0000","last_peered":"2024-09-21T10:18:36.772896+0000","last_clean":"2024-09-21T10:18:36.772896+0000","last_became_active":"2024-09-21T10:18:13.037700+0000","last_became_peered":"2024-09-21T10:18:13.037700+0000","last_unstale":"2024-09-21T10:18:36.772896+0000","last_undegraded":"2024-09-21T10:18:36.772896+0000","last_fullsized":"2024-09-21T10:18:36.772896+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":2267,"log_dups_size":0,"ondisk_log_size":2267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:14:58.367606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148570112,"num_objects":2267,"num_object_clones":0,"num_object_copies":6801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2267,"num_write_kb":145088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":323,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.757732+0000","last_change":"2024-09-21T10:08:04.645700+0000","last_active":"2024-09-21T10:18:26.757732+0000","last_peered":"2024-09-21T10:18:26.757732+0000","last_clean":"2024-09-21T10:18:26.757732+0000","last_became_active":"2024-09-21T10:07:57.548189+0000","last_became_peered":"2024-09-21T10:07:57.548189+0000","last_unstale":"2024-09-21T10:18:26.757732+0000","last_undegraded":"2024-09-21T10:18:26.757732+0000","last_fullsized":"2024-09-21T10:18:26.757732+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:48.318010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:48.318010+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T16:31:28.173059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":602,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.756993+0000","last_change":"2024-09-21T10:03:38.386604+0000","last_active":"2024-09-21T10:18:26.756993+0000","last_peered":"2024-09-21T10:18:26.756993+0000","last_clean":"2024-09-21T10:18:26.756993+0000","last_became_active":"2024-09-21T10:03:38.386283+0000","last_became_peered":"2024-09-21T10:03:38.386283+0000","last_unstale":"2024-09-21T10:18:26.756993+0000","last_undegraded":"2024-09-21T10:18:26.756993+0000","last_fullsized":"2024-09-21T10:18:26.756993+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:27:05.944880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":461,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.763929+0000","last_change":"2024-09-21T10:14:30.709615+0000","last_active":"2024-09-21T10:18:26.763929+0000","last_peered":"2024-09-21T10:18:26.763929+0000","last_clean":"2024-09-21T10:18:26.763929+0000","last_became_active":"2024-09-21T10:14:30.709184+0000","last_became_peered":"2024-09-21T10:14:30.709184+0000","last_unstale":"2024-09-21T10:18:26.763929+0000","last_undegraded":"2024-09-21T10:18:26.763929+0000","last_fullsized":"2024-09-21T10:18:26.763929+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:55:27.220232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.c","version":"372'2164","reported_seq":2185,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:35.691559+0000","last_change":"2024-09-21T10:18:13.037466+0000","last_active":"2024-09-21T10:18:35.691559+0000","last_peered":"2024-09-21T10:18:35.691559+0000","last_clean":"2024-09-21T10:18:35.691559+0000","last_became_active":"2024-09-21T10:18:13.037151+0000","last_became_peered":"2024-09-21T10:18:13.037151+0000","last_unstale":"2024-09-21T10:18:35.691559+0000","last_undegraded":"2024-09-21T10:18:35.691559+0000","last_fullsized":"2024-09-21T10:18:35.691559+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":2164,"log_dups_size":0,"ondisk_log_size":2164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:35:23.611663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141819904,"num_objects":2164,"num_object_clones":0,"num_object_copies":6492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2164,"num_write_kb":138496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":262,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.656186+0000","last_change":"2024-09-21T10:09:49.924286+0000","last_active":"2024-09-21T10:18:19.656186+0000","last_peered":"2024-09-21T10:18:19.656186+0000","last_clean":"2024-09-21T10:18:19.656186+0000","last_became_active":"2024-09-21T10:09:49.923897+0000","last_became_peered":"2024-09-21T10:09:49.923897+0000","last_unstale":"2024-09-21T10:18:19.656186+0000","last_undegraded":"2024-09-21T10:18:19.656186+0000","last_fullsized":"2024-09-21T10:18:19.656186+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:47.393146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:47.393146+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:44.898083+0000","scrub_duration":0.001601086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":357,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.758147+0000","last_change":"2024-09-21T10:10:57.402946+0000","last_active":"2024-09-21T10:18:26.758147+0000","last_peered":"2024-09-21T10:18:26.758147+0000","last_clean":"2024-09-21T10:18:26.758147+0000","last_became_active":"2024-09-21T10:10:57.395450+0000","last_became_peered":"2024-09-21T10:10:57.395450+0000","last_unstale":"2024-09-21T10:18:26.758147+0000","last_undegraded":"2024-09-21T10:18:26.758147+0000","last_fullsized":"2024-09-21T10:18:26.758147+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:08.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":586,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.760157+0000","last_change":"2024-09-21T10:10:57.390595+0000","last_active":"2024-09-21T10:18:26.760157+0000","last_peered":"2024-09-21T10:18:26.760157+0000","last_clean":"2024-09-21T10:18:26.760157+0000","last_became_active":"2024-09-21T10:10:57.390265+0000","last_became_peered":"2024-09-21T10:10:57.390265+0000","last_unstale":"2024-09-21T10:18:26.760157+0000","last_undegraded":"2024-09-21T10:18:26.760157+0000","last_fullsized":"2024-09-21T10:18:26.760157+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:37.485063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":388,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.797873+0000","last_change":"2024-09-21T10:14:30.711719+0000","last_active":"2024-09-21T10:18:19.797873+0000","last_peered":"2024-09-21T10:18:19.797873+0000","last_clean":"2024-09-21T10:18:19.797873+0000","last_became_active":"2024-09-21T10:14:30.711452+0000","last_became_peered":"2024-09-21T10:14:30.711452+0000","last_unstale":"2024-09-21T10:18:19.797873+0000","last_undegraded":"2024-09-21T10:18:19.797873+0000","last_fullsized":"2024-09-21T10:18:19.797873+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:46.907528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"372'2297","reported_seq":2310,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:36.853875+0000","last_change":"2024-09-21T10:18:12.639005+0000","last_active":"2024-09-21T10:18:36.853875+0000","last_peered":"2024-09-21T10:18:36.853875+0000","last_clean":"2024-09-21T10:18:36.853875+0000","last_became_active":"2024-09-21T10:18:12.638716+0000","last_became_peered":"2024-09-21T10:18:12.638716+0000","last_unstale":"2024-09-21T10:18:36.853875+0000","last_undegraded":"2024-09-21T10:18:36.853875+0000","last_fullsized":"2024-09-21T10:18:36.853875+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":2297,"log_dups_size":0,"ondisk_log_size":2297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:08.923853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150536192,"num_objects":2297,"num_object_clones":0,"num_object_copies":6891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2297,"num_write_kb":147008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":263,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.757381+0000","last_change":"2024-09-21T10:14:30.701644+0000","last_active":"2024-09-21T10:18:26.757381+0000","last_peered":"2024-09-21T10:18:26.757381+0000","last_clean":"2024-09-21T10:18:26.757381+0000","last_became_active":"2024-09-21T10:14:30.701301+0000","last_became_peered":"2024-09-21T10:14:30.701301+0000","last_unstale":"2024-09-21T10:18:26.757381+0000","last_undegraded":"2024-09-21T10:18:26.757381+0000","last_fullsized":"2024-09-21T10:18:26.757381+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.550551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.550551+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:23.637695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":236,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.656276+0000","last_change":"2024-09-21T10:09:23.838944+0000","last_active":"2024-09-21T10:18:19.656276+0000","last_peered":"2024-09-21T10:18:19.656276+0000","last_clean":"2024-09-21T10:18:19.656276+0000","last_became_active":"2024-09-21T10:09:23.837886+0000","last_became_peered":"2024-09-21T10:09:23.837886+0000","last_unstale":"2024-09-21T10:18:19.656276+0000","last_undegraded":"2024-09-21T10:18:19.656276+0000","last_fullsized":"2024-09-21T10:18:19.656276+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:16.928195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":485,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.781027+0000","last_change":"2024-09-21T10:18:12.637103+0000","last_active":"2024-09-21T10:18:26.781027+0000","last_peered":"2024-09-21T10:18:26.781027+0000","last_clean":"2024-09-21T10:18:26.781027+0000","last_became_active":"2024-09-21T10:18:12.636645+0000","last_became_peered":"2024-09-21T10:18:12.636645+0000","last_unstale":"2024-09-21T10:18:26.781027+0000","last_undegraded":"2024-09-21T10:18:26.781027+0000","last_fullsized":"2024-09-21T10:18:26.781027+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:53:50.745484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":247,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.781756+0000","last_change":"2024-09-21T10:09:49.923044+0000","last_active":"2024-09-21T10:18:26.781756+0000","last_peered":"2024-09-21T10:18:26.781756+0000","last_clean":"2024-09-21T10:18:26.781756+0000","last_became_active":"2024-09-21T10:09:49.922711+0000","last_became_peered":"2024-09-21T10:09:49.922711+0000","last_unstale":"2024-09-21T10:18:26.781756+0000","last_undegraded":"2024-09-21T10:18:26.781756+0000","last_fullsized":"2024-09-21T10:18:26.781756+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:14.090084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"372'2254","reported_seq":2267,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:36.961410+0000","last_change":"2024-09-21T10:18:13.038029+0000","last_active":"2024-09-21T10:18:36.961410+0000","last_peered":"2024-09-21T10:18:36.961410+0000","last_clean":"2024-09-21T10:18:36.961410+0000","last_became_active":"2024-09-21T10:18:13.037584+0000","last_became_peered":"2024-09-21T10:18:13.037584+0000","last_unstale":"2024-09-21T10:18:36.961410+0000","last_undegraded":"2024-09-21T10:18:36.961410+0000","last_fullsized":"2024-09-21T10:18:36.961410+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":2254,"log_dups_size":0,"ondisk_log_size":2254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:18.430382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147718144,"num_objects":2254,"num_object_clones":0,"num_object_copies":6762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2254,"num_write_kb":144256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":286,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655624+0000","last_change":"2024-09-21T10:18:12.636851+0000","last_active":"2024-09-21T10:18:19.655624+0000","last_peered":"2024-09-21T10:18:19.655624+0000","last_clean":"2024-09-21T10:18:19.655624+0000","last_became_active":"2024-09-21T10:18:12.636513+0000","last_became_peered":"2024-09-21T10:18:12.636513+0000","last_unstale":"2024-09-21T10:18:19.655624+0000","last_undegraded":"2024-09-21T10:18:19.655624+0000","last_fullsized":"2024-09-21T10:18:19.655624+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:52.361137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:52.361137+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:21.025735+0000","scrub_duration":0.001255358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":640,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.761335+0000","last_change":"2024-09-21T10:09:23.837859+0000","last_active":"2024-09-21T10:18:26.761335+0000","last_peered":"2024-09-21T10:18:26.761335+0000","last_clean":"2024-09-21T10:18:26.761335+0000","last_became_active":"2024-09-21T10:09:23.837488+0000","last_became_peered":"2024-09-21T10:09:23.837488+0000","last_unstale":"2024-09-21T10:18:26.761335+0000","last_undegraded":"2024-09-21T10:18:26.761335+0000","last_fullsized":"2024-09-21T10:18:26.761335+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:25.215501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":120,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655999+0000","last_change":"2024-09-21T10:10:27.806070+0000","last_active":"2024-09-21T10:18:19.655999+0000","last_peered":"2024-09-21T10:18:19.655999+0000","last_clean":"2024-09-21T10:18:19.655999+0000","last_became_active":"2024-09-21T10:10:27.805718+0000","last_became_peered":"2024-09-21T10:10:27.805718+0000","last_unstale":"2024-09-21T10:18:19.655999+0000","last_undegraded":"2024-09-21T10:18:19.655999+0000","last_fullsized":"2024-09-21T10:18:19.655999+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:31.144118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":633,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.769975+0000","last_change":"2024-09-21T10:10:57.395073+0000","last_active":"2024-09-21T10:18:26.769975+0000","last_peered":"2024-09-21T10:18:26.769975+0000","last_clean":"2024-09-21T10:18:26.769975+0000","last_became_active":"2024-09-21T10:10:57.394674+0000","last_became_peered":"2024-09-21T10:10:57.394674+0000","last_unstale":"2024-09-21T10:18:26.769975+0000","last_undegraded":"2024-09-21T10:18:26.769975+0000","last_fullsized":"2024-09-21T10:18:26.769975+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:48:48.912649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.d","version":"372'2294","reported_seq":2307,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:36.821237+0000","last_change":"2024-09-21T10:18:12.632980+0000","last_active":"2024-09-21T10:18:36.821237+0000","last_peered":"2024-09-21T10:18:36.821237+0000","last_clean":"2024-09-21T10:18:36.821237+0000","last_became_active":"2024-09-21T10:18:12.632684+0000","last_became_peered":"2024-09-21T10:18:12.632684+0000","last_unstale":"2024-09-21T10:18:36.821237+0000","last_undegraded":"2024-09-21T10:18:36.821237+0000","last_fullsized":"2024-09-21T10:18:36.821237+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":2294,"log_dups_size":0,"ondisk_log_size":2294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:08.974530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150339584,"num_objects":2294,"num_object_clones":0,"num_object_copies":6882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2294,"num_write_kb":146816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":502,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.782542+0000","last_change":"2024-09-21T10:10:57.398437+0000","last_active":"2024-09-21T10:18:26.782542+0000","last_peered":"2024-09-21T10:18:26.782542+0000","last_clean":"2024-09-21T10:18:26.782542+0000","last_became_active":"2024-09-21T10:10:57.397441+0000","last_became_peered":"2024-09-21T10:10:57.397441+0000","last_unstale":"2024-09-21T10:18:26.782542+0000","last_undegraded":"2024-09-21T10:18:26.782542+0000","last_fullsized":"2024-09-21T10:18:26.782542+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:04.043917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:04.043917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:53:31.813612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":195,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.757094+0000","last_change":"2024-09-21T10:14:05.472790+0000","last_active":"2024-09-21T10:18:26.757094+0000","last_peered":"2024-09-21T10:18:26.757094+0000","last_clean":"2024-09-21T10:18:26.757094+0000","last_became_active":"2024-09-21T10:14:05.472238+0000","last_became_peered":"2024-09-21T10:14:05.472238+0000","last_unstale":"2024-09-21T10:18:26.757094+0000","last_undegraded":"2024-09-21T10:18:26.757094+0000","last_fullsized":"2024-09-21T10:18:26.757094+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:56.561032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":78,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.656002+0000","last_change":"2024-09-21T10:14:30.709652+0000","last_active":"2024-09-21T10:18:19.656002+0000","last_peered":"2024-09-21T10:18:19.656002+0000","last_clean":"2024-09-21T10:18:19.656002+0000","last_became_active":"2024-09-21T10:14:30.705706+0000","last_became_peered":"2024-09-21T10:14:30.705706+0000","last_unstale":"2024-09-21T10:18:19.656002+0000","last_undegraded":"2024-09-21T10:18:19.656002+0000","last_fullsized":"2024-09-21T10:18:19.656002+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:24:25.749923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":466,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.758412+0000","last_change":"2024-09-21T10:14:05.477875+0000","last_active":"2024-09-21T10:18:26.758412+0000","last_peered":"2024-09-21T10:18:26.758412+0000","last_clean":"2024-09-21T10:18:26.758412+0000","last_became_active":"2024-09-21T10:14:05.477192+0000","last_became_peered":"2024-09-21T10:14:05.477192+0000","last_unstale":"2024-09-21T10:18:26.758412+0000","last_undegraded":"2024-09-21T10:18:26.758412+0000","last_fullsized":"2024-09-21T10:18:26.758412+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:19:47.345673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.b","version":"372'2318","reported_seq":2339,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:37.402764+0000","last_change":"2024-09-21T10:18:13.039421+0000","last_active":"2024-09-21T10:18:37.402764+0000","last_peered":"2024-09-21T10:18:37.402764+0000","last_clean":"2024-09-21T10:18:37.402764+0000","last_became_active":"2024-09-21T10:18:13.039085+0000","last_became_peered":"2024-09-21T10:18:13.039085+0000","last_unstale":"2024-09-21T10:18:37.402764+0000","last_undegraded":"2024-09-21T10:18:37.402764+0000","last_fullsized":"2024-09-21T10:18:37.402764+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":2318,"log_dups_size":0,"ondisk_log_size":2318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:36.052065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151912448,"num_objects":2318,"num_object_clones":0,"num_object_copies":6954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2318,"num_write_kb":148352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":345,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.798087+0000","last_change":"2024-09-21T10:04:45.242148+0000","last_active":"2024-09-21T10:18:19.798087+0000","last_peered":"2024-09-21T10:18:19.798087+0000","last_clean":"2024-09-21T10:18:19.798087+0000","last_became_active":"2024-09-21T10:04:42.549844+0000","last_became_peered":"2024-09-21T10:04:42.549844+0000","last_unstale":"2024-09-21T10:18:19.798087+0000","last_undegraded":"2024-09-21T10:18:19.798087+0000","last_fullsized":"2024-09-21T10:18:19.798087+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_deep_scrub":"77'100","last_deep_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.242073+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:03:42.250480+0000","scrub_duration":0.0023466239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":308,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.758515+0000","last_change":"2024-09-21T10:15:01.489326+0000","last_active":"2024-09-21T10:18:26.758515+0000","last_peered":"2024-09-21T10:18:26.758515+0000","last_clean":"2024-09-21T10:18:26.758515+0000","last_became_active":"2024-09-21T10:15:01.489070+0000","last_became_peered":"2024-09-21T10:15:01.489070+0000","last_unstale":"2024-09-21T10:18:26.758515+0000","last_undegraded":"2024-09-21T10:18:26.758515+0000","last_fullsized":"2024-09-21T10:18:26.758515+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:27:41.788029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":522,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.758570+0000","last_change":"2024-09-21T10:14:05.475970+0000","last_active":"2024-09-21T10:18:26.758570+0000","last_peered":"2024-09-21T10:18:26.758570+0000","last_clean":"2024-09-21T10:18:26.758570+0000","last_became_active":"2024-09-21T10:14:05.475331+0000","last_became_peered":"2024-09-21T10:14:05.475331+0000","last_unstale":"2024-09-21T10:18:26.758570+0000","last_undegraded":"2024-09-21T10:18:26.758570+0000","last_fullsized":"2024-09-21T10:18:26.758570+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:17.069736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":458,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.757396+0000","last_change":"2024-09-21T10:18:13.039172+0000","last_active":"2024-09-21T10:18:26.757396+0000","last_peered":"2024-09-21T10:18:26.757396+0000","last_clean":"2024-09-21T10:18:26.757396+0000","last_became_active":"2024-09-21T10:18:13.038756+0000","last_became_peered":"2024-09-21T10:18:13.038756+0000","last_unstale":"2024-09-21T10:18:26.757396+0000","last_undegraded":"2024-09-21T10:18:26.757396+0000","last_fullsized":"2024-09-21T10:18:26.757396+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:06.668926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.a","version":"372'2026","reported_seq":2044,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:34.755102+0000","last_change":"2024-09-21T10:18:12.633418+0000","last_active":"2024-09-21T10:18:34.755102+0000","last_peered":"2024-09-21T10:18:34.755102+0000","last_clean":"2024-09-21T10:18:34.755102+0000","last_became_active":"2024-09-21T10:18:12.633151+0000","last_became_peered":"2024-09-21T10:18:12.633151+0000","last_unstale":"2024-09-21T10:18:34.755102+0000","last_undegraded":"2024-09-21T10:18:34.755102+0000","last_fullsized":"2024-09-21T10:18:34.755102+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":2026,"log_dups_size":0,"ondisk_log_size":2026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:07:26.685329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132775936,"num_objects":2026,"num_object_clones":0,"num_object_copies":6078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2026,"num_write_kb":129664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":362,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.797699+0000","last_change":"2024-09-21T10:09:23.839563+0000","last_active":"2024-09-21T10:18:19.797699+0000","last_peered":"2024-09-21T10:18:19.797699+0000","last_clean":"2024-09-21T10:18:19.797699+0000","last_became_active":"2024-09-21T10:09:23.839207+0000","last_became_peered":"2024-09-21T10:09:23.839207+0000","last_unstale":"2024-09-21T10:18:19.797699+0000","last_undegraded":"2024-09-21T10:18:19.797699+0000","last_fullsized":"2024-09-21T10:18:19.797699+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:49.226496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:49.226496+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:03.960125+0000","scrub_duration":0.001656865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":454,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.758489+0000","last_change":"2024-09-21T10:14:05.478152+0000","last_active":"2024-09-21T10:18:26.758489+0000","last_peered":"2024-09-21T10:18:26.758489+0000","last_clean":"2024-09-21T10:18:26.758489+0000","last_became_active":"2024-09-21T10:14:05.477547+0000","last_became_peered":"2024-09-21T10:14:05.477547+0000","last_unstale":"2024-09-21T10:18:26.758489+0000","last_undegraded":"2024-09-21T10:18:26.758489+0000","last_fullsized":"2024-09-21T10:18:26.758489+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:34.646046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":266,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655473+0000","last_change":"2024-09-21T10:14:30.708090+0000","last_active":"2024-09-21T10:18:19.655473+0000","last_peered":"2024-09-21T10:18:19.655473+0000","last_clean":"2024-09-21T10:18:19.655473+0000","last_became_active":"2024-09-21T10:14:30.707266+0000","last_became_peered":"2024-09-21T10:14:30.707266+0000","last_unstale":"2024-09-21T10:18:19.655473+0000","last_undegraded":"2024-09-21T10:18:19.655473+0000","last_fullsized":"2024-09-21T10:18:19.655473+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:38.474537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":516,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.781922+0000","last_change":"2024-09-21T10:10:57.397251+0000","last_active":"2024-09-21T10:18:26.781922+0000","last_peered":"2024-09-21T10:18:26.781922+0000","last_clean":"2024-09-21T10:18:26.781922+0000","last_became_active":"2024-09-21T10:10:57.396271+0000","last_became_peered":"2024-09-21T10:10:57.396271+0000","last_unstale":"2024-09-21T10:18:26.781922+0000","last_undegraded":"2024-09-21T10:18:26.781922+0000","last_fullsized":"2024-09-21T10:18:26.781922+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:17.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.9","version":"372'2406","reported_seq":2427,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:37.534231+0000","last_change":"2024-09-21T10:18:13.037459+0000","last_active":"2024-09-21T10:18:37.534231+0000","last_peered":"2024-09-21T10:18:37.534231+0000","last_clean":"2024-09-21T10:18:37.534231+0000","last_became_active":"2024-09-21T10:18:13.037118+0000","last_became_peered":"2024-09-21T10:18:13.037118+0000","last_unstale":"2024-09-21T10:18:37.534231+0000","last_undegraded":"2024-09-21T10:18:37.534231+0000","last_fullsized":"2024-09-21T10:18:37.534231+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":2406,"log_dups_size":0,"ondisk_log_size":2406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:52.139485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157679616,"num_objects":2406,"num_object_clones":0,"num_object_copies":7218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2406,"num_write_kb":153984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":78,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.656235+0000","last_change":"2024-09-21T10:14:30.709001+0000","last_active":"2024-09-21T10:18:19.656235+0000","last_peered":"2024-09-21T10:18:19.656235+0000","last_clean":"2024-09-21T10:18:19.656235+0000","last_became_active":"2024-09-21T10:14:30.708636+0000","last_became_peered":"2024-09-21T10:14:30.708636+0000","last_unstale":"2024-09-21T10:18:19.656235+0000","last_undegraded":"2024-09-21T10:18:19.656235+0000","last_fullsized":"2024-09-21T10:18:19.656235+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:59.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:59.099852+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:20:01.949185+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":608,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.763353+0000","last_change":"2024-09-21T10:09:23.841186+0000","last_active":"2024-09-21T10:18:26.763353+0000","last_peered":"2024-09-21T10:18:26.763353+0000","last_clean":"2024-09-21T10:18:26.763353+0000","last_became_active":"2024-09-21T10:09:23.840854+0000","last_became_peered":"2024-09-21T10:09:23.840854+0000","last_unstale":"2024-09-21T10:18:26.763353+0000","last_undegraded":"2024-09-21T10:18:26.763353+0000","last_fullsized":"2024-09-21T10:18:26.763353+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:29.666358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":461,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.763906+0000","last_change":"2024-09-21T10:14:30.708385+0000","last_active":"2024-09-21T10:18:26.763906+0000","last_peered":"2024-09-21T10:18:26.763906+0000","last_clean":"2024-09-21T10:18:26.763906+0000","last_became_active":"2024-09-21T10:14:30.708046+0000","last_became_peered":"2024-09-21T10:14:30.708046+0000","last_unstale":"2024-09-21T10:18:26.763906+0000","last_undegraded":"2024-09-21T10:18:26.763906+0000","last_fullsized":"2024-09-21T10:18:26.763906+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:16:45.563095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":605,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.756778+0000","last_change":"2024-09-21T10:09:49.926548+0000","last_active":"2024-09-21T10:18:26.756778+0000","last_peered":"2024-09-21T10:18:26.756778+0000","last_clean":"2024-09-21T10:18:26.756778+0000","last_became_active":"2024-09-21T10:09:49.926255+0000","last_became_peered":"2024-09-21T10:09:49.926255+0000","last_unstale":"2024-09-21T10:18:26.756778+0000","last_undegraded":"2024-09-21T10:18:26.756778+0000","last_fullsized":"2024-09-21T10:18:26.756778+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:49.855878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.8","version":"372'2049","reported_seq":2070,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:34.747377+0000","last_change":"2024-09-21T10:18:13.038950+0000","last_active":"2024-09-21T10:18:34.747377+0000","last_peered":"2024-09-21T10:18:34.747377+0000","last_clean":"2024-09-21T10:18:34.747377+0000","last_became_active":"2024-09-21T10:18:13.038641+0000","last_became_peered":"2024-09-21T10:18:13.038641+0000","last_unstale":"2024-09-21T10:18:34.747377+0000","last_undegraded":"2024-09-21T10:18:34.747377+0000","last_fullsized":"2024-09-21T10:18:34.747377+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":2049,"log_dups_size":0,"ondisk_log_size":2049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:02.783389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134283264,"num_objects":2049,"num_object_clones":0,"num_object_copies":6147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2049,"num_write_kb":131136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":406,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.757778+0000","last_change":"2024-09-21T10:10:57.390804+0000","last_active":"2024-09-21T10:18:26.757778+0000","last_peered":"2024-09-21T10:18:26.757778+0000","last_clean":"2024-09-21T10:18:26.757778+0000","last_became_active":"2024-09-21T10:10:57.390323+0000","last_became_peered":"2024-09-21T10:10:57.390323+0000","last_unstale":"2024-09-21T10:18:26.757778+0000","last_undegraded":"2024-09-21T10:18:26.757778+0000","last_fullsized":"2024-09-21T10:18:26.757778+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.005477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.005477+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:29:57.702046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":262,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.781933+0000","last_change":"2024-09-21T10:14:30.718942+0000","last_active":"2024-09-21T10:18:26.781933+0000","last_peered":"2024-09-21T10:18:26.781933+0000","last_clean":"2024-09-21T10:18:26.781933+0000","last_became_active":"2024-09-21T10:14:30.709129+0000","last_became_peered":"2024-09-21T10:14:30.709129+0000","last_unstale":"2024-09-21T10:18:26.781933+0000","last_undegraded":"2024-09-21T10:18:26.781933+0000","last_fullsized":"2024-09-21T10:18:26.781933+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:18.171064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":384,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.798183+0000","last_change":"2024-09-21T10:03:38.392767+0000","last_active":"2024-09-21T10:18:19.798183+0000","last_peered":"2024-09-21T10:18:19.798183+0000","last_clean":"2024-09-21T10:18:19.798183+0000","last_became_active":"2024-09-21T10:03:38.392424+0000","last_became_peered":"2024-09-21T10:03:38.392424+0000","last_unstale":"2024-09-21T10:18:19.798183+0000","last_undegraded":"2024-09-21T10:18:19.798183+0000","last_fullsized":"2024-09-21T10:18:19.798183+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:25.358291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":78,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655005+0000","last_change":"2024-09-21T10:14:30.703352+0000","last_active":"2024-09-21T10:18:19.655005+0000","last_peered":"2024-09-21T10:18:19.655005+0000","last_clean":"2024-09-21T10:18:19.655005+0000","last_became_active":"2024-09-21T10:14:30.703068+0000","last_became_peered":"2024-09-21T10:14:30.703068+0000","last_unstale":"2024-09-21T10:18:19.655005+0000","last_undegraded":"2024-09-21T10:18:19.655005+0000","last_fullsized":"2024-09-21T10:18:19.655005+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:00:26.966633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":78,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.656071+0000","last_change":"2024-09-21T10:14:30.702750+0000","last_active":"2024-09-21T10:18:19.656071+0000","last_peered":"2024-09-21T10:18:19.656071+0000","last_clean":"2024-09-21T10:18:19.656071+0000","last_became_active":"2024-09-21T10:14:30.702443+0000","last_became_peered":"2024-09-21T10:14:30.702443+0000","last_unstale":"2024-09-21T10:18:19.656071+0000","last_undegraded":"2024-09-21T10:18:19.656071+0000","last_fullsized":"2024-09-21T10:18:19.656071+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:38:19.353586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":88,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655440+0000","last_change":"2024-09-21T10:14:30.716754+0000","last_active":"2024-09-21T10:18:19.655440+0000","last_peered":"2024-09-21T10:18:19.655440+0000","last_clean":"2024-09-21T10:18:19.655440+0000","last_became_active":"2024-09-21T10:14:30.704683+0000","last_became_peered":"2024-09-21T10:14:30.704683+0000","last_unstale":"2024-09-21T10:18:19.655440+0000","last_undegraded":"2024-09-21T10:18:19.655440+0000","last_fullsized":"2024-09-21T10:18:19.655440+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:45.993877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":512,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.757189+0000","last_change":"2024-09-21T10:14:05.477429+0000","last_active":"2024-09-21T10:18:26.757189+0000","last_peered":"2024-09-21T10:18:26.757189+0000","last_clean":"2024-09-21T10:18:26.757189+0000","last_became_active":"2024-09-21T10:14:05.476794+0000","last_became_peered":"2024-09-21T10:14:05.476794+0000","last_unstale":"2024-09-21T10:18:26.757189+0000","last_undegraded":"2024-09-21T10:18:26.757189+0000","last_fullsized":"2024-09-21T10:18:26.757189+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:18:54.963704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":256,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.762123+0000","last_change":"2024-09-21T10:14:30.713873+0000","last_active":"2024-09-21T10:18:26.762123+0000","last_peered":"2024-09-21T10:18:26.762123+0000","last_clean":"2024-09-21T10:18:26.762123+0000","last_became_active":"2024-09-21T10:14:30.703149+0000","last_became_peered":"2024-09-21T10:14:30.703149+0000","last_unstale":"2024-09-21T10:18:26.762123+0000","last_undegraded":"2024-09-21T10:18:26.762123+0000","last_fullsized":"2024-09-21T10:18:26.762123+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:26.453902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'100","reported_seq":405,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.762603+0000","last_change":"2024-09-21T10:18:12.633959+0000","last_active":"2024-09-21T10:18:26.762603+0000","last_peered":"2024-09-21T10:18:26.762603+0000","last_clean":"2024-09-21T10:18:26.762603+0000","last_became_active":"2024-09-21T10:18:12.633591+0000","last_became_peered":"2024-09-21T10:18:12.633591+0000","last_unstale":"2024-09-21T10:18:26.762603+0000","last_undegraded":"2024-09-21T10:18:26.762603+0000","last_fullsized":"2024-09-21T10:18:26.762603+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.014993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.014993+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:16:55.324246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":285,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655765+0000","last_change":"2024-09-21T10:15:01.483814+0000","last_active":"2024-09-21T10:18:19.655765+0000","last_peered":"2024-09-21T10:18:19.655765+0000","last_clean":"2024-09-21T10:18:19.655765+0000","last_became_active":"2024-09-21T10:15:01.483522+0000","last_became_peered":"2024-09-21T10:15:01.483522+0000","last_unstale":"2024-09-21T10:18:19.655765+0000","last_undegraded":"2024-09-21T10:18:19.655765+0000","last_fullsized":"2024-09-21T10:18:19.655765+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:13.632066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":590,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.757159+0000","last_change":"2024-09-21T10:03:38.425111+0000","last_active":"2024-09-21T10:18:26.757159+0000","last_peered":"2024-09-21T10:18:26.757159+0000","last_clean":"2024-09-21T10:18:26.757159+0000","last_became_active":"2024-09-21T10:03:38.424704+0000","last_became_peered":"2024-09-21T10:03:38.424704+0000","last_unstale":"2024-09-21T10:18:26.757159+0000","last_undegraded":"2024-09-21T10:18:26.757159+0000","last_fullsized":"2024-09-21T10:18:26.757159+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:35.157076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":490,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.782339+0000","last_change":"2024-09-21T10:10:57.397755+0000","last_active":"2024-09-21T10:18:26.782339+0000","last_peered":"2024-09-21T10:18:26.782339+0000","last_clean":"2024-09-21T10:18:26.782339+0000","last_became_active":"2024-09-21T10:10:57.397249+0000","last_became_peered":"2024-09-21T10:10:57.397249+0000","last_unstale":"2024-09-21T10:18:26.782339+0000","last_undegraded":"2024-09-21T10:18:26.782339+0000","last_fullsized":"2024-09-21T10:18:26.782339+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:40.813503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":230,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.762216+0000","last_change":"2024-09-21T10:14:05.471573+0000","last_active":"2024-09-21T10:18:26.762216+0000","last_peered":"2024-09-21T10:18:26.762216+0000","last_clean":"2024-09-21T10:18:26.762216+0000","last_became_active":"2024-09-21T10:14:05.471270+0000","last_became_peered":"2024-09-21T10:14:05.471270+0000","last_unstale":"2024-09-21T10:18:26.762216+0000","last_undegraded":"2024-09-21T10:18:26.762216+0000","last_fullsized":"2024-09-21T10:18:26.762216+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:56:13.468768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":334,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.798139+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T10:18:19.798139+0000","last_peered":"2024-09-21T10:18:19.798139+0000","last_clean":"2024-09-21T10:18:19.798139+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T10:18:19.798139+0000","last_undegraded":"2024-09-21T10:18:19.798139+0000","last_fullsized":"2024-09-21T10:18:19.798139+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":611,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.756883+0000","last_change":"2024-09-21T10:09:49.928053+0000","last_active":"2024-09-21T10:18:26.756883+0000","last_peered":"2024-09-21T10:18:26.756883+0000","last_clean":"2024-09-21T10:18:26.756883+0000","last_became_active":"2024-09-21T10:09:49.927231+0000","last_became_peered":"2024-09-21T10:09:49.927231+0000","last_unstale":"2024-09-21T10:18:26.756883+0000","last_undegraded":"2024-09-21T10:18:26.756883+0000","last_fullsized":"2024-09-21T10:18:26.756883+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:26.987553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":523,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.782031+0000","last_change":"2024-09-21T10:09:23.840717+0000","last_active":"2024-09-21T10:18:26.782031+0000","last_peered":"2024-09-21T10:18:26.782031+0000","last_clean":"2024-09-21T10:18:26.782031+0000","last_became_active":"2024-09-21T10:09:23.840323+0000","last_became_peered":"2024-09-21T10:09:23.840323+0000","last_unstale":"2024-09-21T10:18:26.782031+0000","last_undegraded":"2024-09-21T10:18:26.782031+0000","last_fullsized":"2024-09-21T10:18:26.782031+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:55.769770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":244,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.761602+0000","last_change":"2024-09-21T10:18:12.632498+0000","last_active":"2024-09-21T10:18:26.761602+0000","last_peered":"2024-09-21T10:18:26.761602+0000","last_clean":"2024-09-21T10:18:26.761602+0000","last_became_active":"2024-09-21T10:18:12.631648+0000","last_became_peered":"2024-09-21T10:18:12.631648+0000","last_unstale":"2024-09-21T10:18:26.761602+0000","last_undegraded":"2024-09-21T10:18:26.761602+0000","last_fullsized":"2024-09-21T10:18:26.761602+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:17.592618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":588,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.758366+0000","last_change":"2024-09-21T10:14:30.702101+0000","last_active":"2024-09-21T10:18:26.758366+0000","last_peered":"2024-09-21T10:18:26.758366+0000","last_clean":"2024-09-21T10:18:26.758366+0000","last_became_active":"2024-09-21T10:14:30.701783+0000","last_became_peered":"2024-09-21T10:14:30.701783+0000","last_unstale":"2024-09-21T10:18:26.758366+0000","last_undegraded":"2024-09-21T10:18:26.758366+0000","last_fullsized":"2024-09-21T10:18:26.758366+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:13.087010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":78,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655280+0000","last_change":"2024-09-21T10:14:30.707503+0000","last_active":"2024-09-21T10:18:19.655280+0000","last_peered":"2024-09-21T10:18:19.655280+0000","last_clean":"2024-09-21T10:18:19.655280+0000","last_became_active":"2024-09-21T10:14:30.706881+0000","last_became_peered":"2024-09-21T10:14:30.706881+0000","last_unstale":"2024-09-21T10:18:19.655280+0000","last_undegraded":"2024-09-21T10:18:19.655280+0000","last_fullsized":"2024-09-21T10:18:19.655280+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:23:07.485461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":628,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.762731+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:18:26.762731+0000","last_peered":"2024-09-21T10:18:26.762731+0000","last_clean":"2024-09-21T10:18:26.762731+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:18:26.762731+0000","last_undegraded":"2024-09-21T10:18:26.762731+0000","last_fullsized":"2024-09-21T10:18:26.762731+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":253,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.781633+0000","last_change":"2024-09-21T10:14:30.702414+0000","last_active":"2024-09-21T10:18:26.781633+0000","last_peered":"2024-09-21T10:18:26.781633+0000","last_clean":"2024-09-21T10:18:26.781633+0000","last_became_active":"2024-09-21T10:14:30.702009+0000","last_became_peered":"2024-09-21T10:14:30.702009+0000","last_unstale":"2024-09-21T10:18:26.781633+0000","last_undegraded":"2024-09-21T10:18:26.781633+0000","last_fullsized":"2024-09-21T10:18:26.781633+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:50:48.396082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655618+0000","last_change":"2024-09-21T10:18:12.633151+0000","last_active":"2024-09-21T10:18:19.655618+0000","last_peered":"2024-09-21T10:18:19.655618+0000","last_clean":"2024-09-21T10:18:19.655618+0000","last_became_active":"2024-09-21T10:18:12.632854+0000","last_became_peered":"2024-09-21T10:18:12.632854+0000","last_unstale":"2024-09-21T10:18:19.655618+0000","last_undegraded":"2024-09-21T10:18:19.655618+0000","last_fullsized":"2024-09-21T10:18:19.655618+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:14.602893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":82,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655656+0000","last_change":"2024-09-21T10:14:30.720787+0000","last_active":"2024-09-21T10:18:19.655656+0000","last_peered":"2024-09-21T10:18:19.655656+0000","last_clean":"2024-09-21T10:18:19.655656+0000","last_became_active":"2024-09-21T10:14:30.706001+0000","last_became_peered":"2024-09-21T10:14:30.706001+0000","last_unstale":"2024-09-21T10:18:19.655656+0000","last_undegraded":"2024-09-21T10:18:19.655656+0000","last_fullsized":"2024-09-21T10:18:19.655656+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:40:11.987108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.655114+0000","last_change":"2024-09-21T10:18:12.636252+0000","last_active":"2024-09-21T10:18:19.655114+0000","last_peered":"2024-09-21T10:18:19.655114+0000","last_clean":"2024-09-21T10:18:19.655114+0000","last_became_active":"2024-09-21T10:18:12.635945+0000","last_became_peered":"2024-09-21T10:18:12.635945+0000","last_unstale":"2024-09-21T10:18:19.655114+0000","last_undegraded":"2024-09-21T10:18:19.655114+0000","last_fullsized":"2024-09-21T10:18:19.655114+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:59:29.464045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":82,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.656257+0000","last_change":"2024-09-21T10:14:30.722338+0000","last_active":"2024-09-21T10:18:19.656257+0000","last_peered":"2024-09-21T10:18:19.656257+0000","last_clean":"2024-09-21T10:18:19.656257+0000","last_became_active":"2024-09-21T10:14:30.708164+0000","last_became_peered":"2024-09-21T10:14:30.708164+0000","last_unstale":"2024-09-21T10:18:19.656257+0000","last_undegraded":"2024-09-21T10:18:19.656257+0000","last_fullsized":"2024-09-21T10:18:19.656257+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:32:13.825114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":612,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.757514+0000","last_change":"2024-09-21T10:09:23.840467+0000","last_active":"2024-09-21T10:18:26.757514+0000","last_peered":"2024-09-21T10:18:26.757514+0000","last_clean":"2024-09-21T10:18:26.757514+0000","last_became_active":"2024-09-21T10:09:23.840177+0000","last_became_peered":"2024-09-21T10:09:23.840177+0000","last_unstale":"2024-09-21T10:18:26.757514+0000","last_undegraded":"2024-09-21T10:18:26.757514+0000","last_fullsized":"2024-09-21T10:18:26.757514+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:17.887307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":602,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.758323+0000","last_change":"2024-09-21T10:11:05.451613+0000","last_active":"2024-09-21T10:18:26.758323+0000","last_peered":"2024-09-21T10:18:26.758323+0000","last_clean":"2024-09-21T10:18:26.758323+0000","last_became_active":"2024-09-21T10:11:05.451075+0000","last_became_peered":"2024-09-21T10:11:05.451075+0000","last_unstale":"2024-09-21T10:18:26.758323+0000","last_undegraded":"2024-09-21T10:18:26.758323+0000","last_fullsized":"2024-09-21T10:18:26.758323+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:18.505045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":78,"reported_epoch":371,"state":"active+clean","last_fresh":"2024-09-21T10:18:19.654803+0000","last_change":"2024-09-21T10:14:30.707548+0000","last_active":"2024-09-21T10:18:19.654803+0000","last_peered":"2024-09-21T10:18:19.654803+0000","last_clean":"2024-09-21T10:18:19.654803+0000","last_became_active":"2024-09-21T10:14:30.707232+0000","last_became_peered":"2024-09-21T10:14:30.707232+0000","last_unstale":"2024-09-21T10:18:19.654803+0000","last_undegraded":"2024-09-21T10:18:19.654803+0000","last_fullsized":"2024-09-21T10:18:19.654803+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:11:38.905767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":266,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.782177+0000","last_change":"2024-09-21T10:10:57.401588+0000","last_active":"2024-09-21T10:18:26.782177+0000","last_peered":"2024-09-21T10:18:26.782177+0000","last_clean":"2024-09-21T10:18:26.782177+0000","last_became_active":"2024-09-21T10:10:57.397724+0000","last_became_peered":"2024-09-21T10:10:57.397724+0000","last_unstale":"2024-09-21T10:18:26.782177+0000","last_undegraded":"2024-09-21T10:18:26.782177+0000","last_fullsized":"2024-09-21T10:18:26.782177+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:47.843630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":16,"stat_sum":{"num_bytes":2378563584,"num_objects":36294,"num_object_clones":0,"num_object_copies":108882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36294,"num_write_kb":2322816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7032340480,"data_stored":7032340480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36294,"ondisk_log_size":36294,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":11856,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26580,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":644037,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":632715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626915,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1615980,"kb_used_data":723940,"kb_used_omap":1,"kb_used_meta":892030,"kb_avail":92129172,"statfs":{"total":95995035648,"available":94340272128,"internally_reserved":0,"allocated":741314560,"data_stored":739029114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":913439064},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019373,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2149232,"kb_used_data":1262440,"kb_used_omap":1,"kb_used_meta":886782,"kb_avail":91595920,"statfs":{"total":95995035648,"available":93794222080,"internally_reserved":0,"allocated":1292738560,"data_stored":1290449059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":908065112},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":245,"seq":1052266987632,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":860896,"kb_used_data":455064,"kb_used_omap":8,"kb_used_meta":405815,"kb_avail":92884256,"statfs":{"total":95995035648,"available":95113478144,"internally_reserved":0,"allocated":465985536,"data_stored":463708165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8940,"internal_metadata":415554836},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":175,"seq":751619276981,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1645552,"kb_used_data":708264,"kb_used_omap":12,"kb_used_meta":937267,"kb_avail":92099600,"statfs":{"total":95995035648,"available":94309990400,"internally_reserved":0,"allocated":725262336,"data_stored":722972831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12845,"internal_metadata":959761875},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823144,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2636640,"kb_used_data":1134680,"kb_used_omap":1,"kb_used_meta":1501950,"kb_avail":91108512,"statfs":{"total":95995035648,"available":93295116288,"internally_reserved":0,"allocated":1161912320,"data_stored":1159634902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1537997144},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879567,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1580192,"kb_used_data":579544,"kb_used_omap":4,"kb_used_meta":1000635,"kb_avail":92164960,"statfs":{"total":95995035648,"available":94376919040,"internally_reserved":0,"allocated":593453056,"data_stored":591175719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":1024650684},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566407,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2111592,"kb_used_data":1129632,"kb_used_omap":6,"kb_used_meta":981945,"kb_avail":91633560,"statfs":{"total":95995035648,"available":93832765440,"internally_reserved":0,"allocated":1156743168,"data_stored":1154458251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":1005512103},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860752,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1785816,"kb_used_data":675344,"kb_used_omap":5,"kb_used_meta":1110394,"kb_avail":91959336,"statfs":{"total":95995035648,"available":94166360064,"internally_reserved":0,"allocated":691552256,"data_stored":689282173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":1137043695},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7236,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9976,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":472383488,"data_stored":472383488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":704839680,"data_stored":704839680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1200422912,"data_stored":1200422912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":615251968,"data_stored":615251968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1341259776,"data_stored":1341259776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1192165376,"data_stored":1192165376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":748945408,"data_stored":748945408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":757071872,"data_stored":757071872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:18:42.087 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T10:18:42.087 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:18:42.087 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-21T10:18:42.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:18:42.099 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:42.125 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:42.133 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:42.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:42.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:42.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:41 smithi129 ceph-mon[88471]: from='client.46892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:18:42.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:42.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:42.267 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:42.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:42.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:42.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:41 smithi107 ceph-mon[102221]: from='client.46892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:18:42.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:42.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:41 smithi107 ceph-mon[105022]: from='client.46892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:18:42.323 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:42.369 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:42.409 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:42.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:42.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:42.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:42.537 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:42.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:42.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:42.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:42.670 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:42.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:42.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:42.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:42.803 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:42.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:42.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:42.905 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:42.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:42.942 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:42.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:43.043 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:43.061 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 21 16 47434 47418 141.094 101.688 0.00527664 0.00708216 2024-09-21T10:18:43.072 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:43.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:43.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:43.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:42 smithi107 ceph-mon[102221]: pgmap v1267: 109 pgs: 109 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-09-21T10:18:43.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:42 smithi107 ceph-mon[105022]: pgmap v1267: 109 pgs: 109 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-09-21T10:18:43.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:42 smithi129 ceph-mon[88471]: pgmap v1267: 109 pgs: 109 active+clean; 2.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-09-21T10:18:43.173 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:43.200 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:43.208 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:43.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:43.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:43.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:43.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:43.335 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:43.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:43.436 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:43.473 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:43.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:43.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:43.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:43.596 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:43.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:43.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:43.698 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:43.736 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:43.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:43.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:43.838 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:43.869 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:43.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:43.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:43.970 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:44.006 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:44.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:44.061 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 22 16 49007 48991 139.149 98.3125 0.00805908 0.0071816 2024-09-21T10:18:44.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:44.141 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:44.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:44.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:44.243 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:44.285 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:44.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:44.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:44.387 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:44.424 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:44.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:44.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:44.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:44.553 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:44.561 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:44.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:44.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:44.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:44.711 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:44.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:44.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:44.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:44.838 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:44.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:44.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:44.909 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:18:44.941 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:44.976 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:44.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:44.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:45.062 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 23 16 50418 50402 136.927 88.1875 0.0115622 0.00729799 2024-09-21T10:18:45.078 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:45.116 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:45.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:45.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:45.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:44 smithi129 ceph-mon[88471]: pgmap v1268: 109 pgs: 109 active+clean; 2.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-09-21T10:18:45.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:44 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1566453373' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:18:45.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:45.251 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:45.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:45.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:45.352 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:44 smithi107 ceph-mon[102221]: pgmap v1268: 109 pgs: 109 active+clean; 2.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-09-21T10:18:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:44 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1566453373' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:18:45.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:44 smithi107 ceph-mon[105022]: pgmap v1268: 109 pgs: 109 active+clean; 2.8 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-09-21T10:18:45.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:44 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1566453373' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:18:45.392 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:45.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:45.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:45.494 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:45.539 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:45.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:45.640 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:45.676 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:45.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:45.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:45.778 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:45.815 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:45.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:45.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:45.916 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:45.959 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:45.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:45.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:46.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:46.062 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 24 16 52132 52116 135.685 107.125 0.00499759 0.00736423 2024-09-21T10:18:46.089 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:45 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:18:46.089 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:45 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:18:46.089 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:45 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1566453373' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:18:46.089 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:45 smithi129 ceph-mon[88471]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T10:18:46.090 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:46.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:46.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:46.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:45 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:18:46.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:45 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:18:46.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:45 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1566453373' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:18:46.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:45 smithi107 ceph-mon[102221]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T10:18:46.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:45 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:18:46.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:45 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:18:46.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:45 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1566453373' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:18:46.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:45 smithi107 ceph-mon[105022]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T10:18:46.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:46.224 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:46.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:46.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:46.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:46.363 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:46.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:46.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:46.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:46.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:46.499 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:46.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:46.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:46.634 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:46.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:46.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:46.736 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:46.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:46.773 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:46.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:46.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:46.903 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:46.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:46.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:47.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:47.035 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:47.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:47.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:47.063 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 25 16 53775 53759 134.365 102.688 0.00735469 0.00743683 2024-09-21T10:18:47.136 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:47.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:46 smithi129 ceph-mon[88471]: pgmap v1270: 109 pgs: 109 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-09-21T10:18:47.174 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:47.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:46 smithi107 ceph-mon[102221]: pgmap v1270: 109 pgs: 109 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-09-21T10:18:47.176 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:46 smithi107 ceph-mon[105022]: pgmap v1270: 109 pgs: 109 active+clean; 2.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 115 MiB/s wr, 1.83k op/s 2024-09-21T10:18:47.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:47.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:47.277 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:47.315 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:47.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:47.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:47.488 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:47.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:47.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:47.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:47.623 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:47.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:47.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:47.725 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:47.775 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:47.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:47.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:47.877 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:47.913 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:47.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:47.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:48.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:48.047 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:48.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:48.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:48.054 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:18:48.054 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:18:48.062 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 26 16 55339 55323 132.957 97.75 0.0076522 0.00751661 2024-09-21T10:18:48.091 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:18:48.091 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:18:48.121 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:18:48.121 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:18:48.138 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:18:48.138 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:18:48.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:48.170 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:48.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:48.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:48.272 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:48.293 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:48.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:48.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:48.395 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:48.416 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:48.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:48.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:48.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:18:48.517 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:48.539 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:48.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:48.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:48.641 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:48.660 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:18:48.660 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:18:48.679 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:48.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:48.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:48.693 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:18:48.694 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:18:48.723 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:18:48.724 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:18:48.753 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:18:48.753 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:18:48.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:48.818 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:48.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:48.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:48.919 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:48.955 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:48.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:48.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:49.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:49.063 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 27 16 56151 56135 129.912 50.75 0.00305994 0.00769219 2024-09-21T10:18:49.093 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:49.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:49.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:49.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:48 smithi107 ceph-mon[102221]: pgmap v1271: 109 pgs: 109 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-21T10:18:49.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:48 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2564610351' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:18:49.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:48 smithi107 ceph-mon[105022]: pgmap v1271: 109 pgs: 109 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-21T10:18:49.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:48 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2564610351' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:18:49.195 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:49.232 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:49.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:49.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:49.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:49.341 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:48 smithi129 ceph-mon[88471]: pgmap v1271: 109 pgs: 109 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-21T10:18:49.341 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:48 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2564610351' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:18:49.368 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:49.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:49.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:49.470 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:49.499 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:49.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:49.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:49.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:49.622 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:49.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:49.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:49.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:49.759 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:49.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:49.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:49.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:49.895 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:49.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:49.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:49.997 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:50.032 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:50.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:50.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:50.063 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 28 16 57761 57745 128.866 100.625 0.0138902 0.00775511 2024-09-21T10:18:50.134 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:50.172 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:50.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:50.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:50.274 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:50.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:50.303 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:50.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:50.404 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:50.439 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:50.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:50.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:50.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:50.574 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:50.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:50.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:50 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:50 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:50 smithi129 ceph-mon[88471]: pgmap v1272: 109 pgs: 109 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-21T10:18:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:18:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:18:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:18:50.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:50.708 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:50.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:50.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:50.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:50.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:50.849 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:50.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:50.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:50 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:50.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:50 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:50.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:50 smithi107 ceph-mon[102221]: pgmap v1272: 109 pgs: 109 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-21T10:18:50.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:50 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:50.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:50 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:50.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:50 smithi107 ceph-mon[105022]: pgmap v1272: 109 pgs: 109 active+clean; 3.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 106 MiB/s wr, 1.69k op/s 2024-09-21T10:18:50.950 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:50.988 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:50.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:50.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:51.063 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 29 16 59374 59358 127.899 100.812 0.00492923 0.00781392 2024-09-21T10:18:51.090 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:51.125 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:51.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:51.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:51.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:51.263 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:51.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:51.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:51.364 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:51.405 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:51.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:51.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:51.507 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:51.545 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:51.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:51.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:51.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:51.680 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:51.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:51.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:51.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:51.815 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:51.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:51.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:51.860 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:18:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:18:51] "GET /metrics HTTP/1.1" 200 36782 "" "Prometheus/2.51.0" 2024-09-21T10:18:51.917 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:51.958 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:51.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:51.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:52.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:52.063 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 30 16 60968 60952 126.956 99.625 0.0120456 0.00787216 2024-09-21T10:18:52.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:52.097 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:52.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:52.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:52.230 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:52.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:52.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:52.332 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:52.353 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:52.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:52.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:52.454 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:52.489 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:52.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:52.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:52.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:52.622 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:52.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:52.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:52.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:52 smithi129 ceph-mon[88471]: pgmap v1273: 109 pgs: 109 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-21T10:18:52.724 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:52.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:52.762 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:52.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:52.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:52 smithi107 ceph-mon[102221]: pgmap v1273: 109 pgs: 109 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-21T10:18:52.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:52 smithi107 ceph-mon[105022]: pgmap v1273: 109 pgs: 109 active+clean; 3.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-21T10:18:52.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:52.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:52.897 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:52.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:52.998 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:53.034 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:53.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:53.063 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 31 16 62420 62404 125.788 90.75 0.0106258 0.00794493 2024-09-21T10:18:53.135 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:53.171 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:53.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:53.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:53.272 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:53.307 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:53.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:53.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:53.409 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:53.448 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:53.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:53.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:53.549 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:53.591 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:53.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:53.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:53.693 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:53.696 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:18:53.740 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:53.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:53.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:53.843 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:53.884 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:53.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:53.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:53.986 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:54.023 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:54.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:54.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:54.063 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 32 16 63888 63872 124.724 91.75 0.012782 0.00801245 2024-09-21T10:18:54.127 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:54.160 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:54.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:54.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:54.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:54.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:54.300 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:54.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:18:54.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:54.447 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:54.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:54.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:54.532 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:18:54.532 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-09-21T10:18:54.532 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-09-21T10:18:54.532 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:18:54.532 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.532 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91111 11.7 11:ef4eb245:::benchmark_data_smithi107_218812_object91110:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:18:54.532 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.433783+0000", 2024-09-21T10:18:54.533 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 478.04011536299998, 2024-09-21T10:18:54.533 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2706856480000002, 2024-09-21T10:18:54.533 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:54.533 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:54.533 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:54.533 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:54.533 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:18:54.533 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:18:54.533 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91111 2024-09-21T10:18:54.533 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.533 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:54.533 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.534 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:54.534 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.433783+0000", 2024-09-21T10:18:54.534 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.534 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.534 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.534 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:54.534 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.433783+0000", 2024-09-21T10:18:54.534 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.534 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.534 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.534 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:54.535 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.433784+0000", 2024-09-21T10:18:54.535 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.03e-07 2024-09-21T10:18:54.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.535 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:54.535 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.433806+0000", 2024-09-21T10:18:54.535 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2540000000000001e-05 2024-09-21T10:18:54.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.535 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:54.535 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.433812+0000", 2024-09-21T10:18:54.535 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2110000000000003e-06 2024-09-21T10:18:54.536 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.536 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.536 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:54.536 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.433817+0000", 2024-09-21T10:18:54.536 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6979999999999997e-06 2024-09-21T10:18:54.536 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.536 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.536 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.536 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.433904+0000", 2024-09-21T10:18:54.536 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7126999999999994e-05 2024-09-21T10:18:54.536 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.536 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.536 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.537 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.433914+0000", 2024-09-21T10:18:54.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8409999999999995e-06 2024-09-21T10:18:54.537 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.537 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.537 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.537 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.390487+0000", 2024-09-21T10:18:54.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.95657265800000002 2024-09-21T10:18:54.537 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.537 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.537 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.537 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.390498+0000", 2024-09-21T10:18:54.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1297e-05 2024-09-21T10:18:54.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.538 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.538 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.538 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.688355+0000", 2024-09-21T10:18:54.538 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2978574 2024-09-21T10:18:54.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.538 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.538 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:18:54.538 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.688476+0000", 2024-09-21T10:18:54.538 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000120909 2024-09-21T10:18:54.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.539 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5,7", 2024-09-21T10:18:54.539 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.688621+0000", 2024-09-21T10:18:54.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014499900000000001 2024-09-21T10:18:54.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.539 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.539 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695262+0000", 2024-09-21T10:18:54.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0066403829999999997 2024-09-21T10:18:54.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.539 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695609+0000", 2024-09-21T10:18:54.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034679099999999999 2024-09-21T10:18:54.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:18:54.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695623+0000", 2024-09-21T10:18:54.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4674999999999999e-05 2024-09-21T10:18:54.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.703975+0000", 2024-09-21T10:18:54.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0083513019999999997 2024-09-21T10:18:54.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.541 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:18:54.541 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704313+0000", 2024-09-21T10:18:54.541 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00033821099999999999 2024-09-21T10:18:54.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.541 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:18:54.541 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704469+0000", 2024-09-21T10:18:54.541 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000156203 2024-09-21T10:18:54.541 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.541 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.542 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.542 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91112 11.7 11:e6106ab1:::benchmark_data_smithi107_218812_object91111:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:18:54.542 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.436473+0000", 2024-09-21T10:18:54.542 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 478.037425203, 2024-09-21T10:18:54.542 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2683268590000001, 2024-09-21T10:18:54.542 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:54.542 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:54.542 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:54.542 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:54.542 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:18:54.543 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:18:54.543 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91112 2024-09-21T10:18:54.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.543 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:54.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.543 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:54.543 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.436473+0000", 2024-09-21T10:18:54.543 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.543 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:54.544 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.436473+0000", 2024-09-21T10:18:54.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.544 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.544 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:54.544 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.436475+0000", 2024-09-21T10:18:54.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.823e-06 2024-09-21T10:18:54.544 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.544 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:54.544 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.436498+0000", 2024-09-21T10:18:54.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2611000000000001e-05 2024-09-21T10:18:54.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.545 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:54.545 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.436510+0000", 2024-09-21T10:18:54.545 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2296000000000001e-05 2024-09-21T10:18:54.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.545 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:54.545 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.436517+0000", 2024-09-21T10:18:54.545 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9500000000000004e-06 2024-09-21T10:18:54.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.546 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.546 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.436583+0000", 2024-09-21T10:18:54.546 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5911e-05 2024-09-21T10:18:54.546 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.546 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.546 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.546 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.436596+0000", 2024-09-21T10:18:54.546 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2490000000000001e-05 2024-09-21T10:18:54.546 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.546 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.546 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.546 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.390523+0000", 2024-09-21T10:18:54.546 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.95392772100000001 2024-09-21T10:18:54.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.547 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.390531+0000", 2024-09-21T10:18:54.547 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5349999999999999e-06 2024-09-21T10:18:54.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.547 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.691573+0000", 2024-09-21T10:18:54.547 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3010422479999999 2024-09-21T10:18:54.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:18:54.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.691717+0000", 2024-09-21T10:18:54.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014439300000000001 2024-09-21T10:18:54.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5,7", 2024-09-21T10:18:54.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.692325+0000", 2024-09-21T10:18:54.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000607612 2024-09-21T10:18:54.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:18:54.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695631+0000", 2024-09-21T10:18:54.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0033063060000000002 2024-09-21T10:18:54.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.549 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.696088+0000", 2024-09-21T10:18:54.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000456384 2024-09-21T10:18:54.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.549 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.696458+0000", 2024-09-21T10:18:54.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037030800000000002 2024-09-21T10:18:54.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.549 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704517+0000", 2024-09-21T10:18:54.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0080585199999999996 2024-09-21T10:18:54.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.550 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:18:54.550 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704569+0000", 2024-09-21T10:18:54.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2688000000000001e-05 2024-09-21T10:18:54.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.550 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:18:54.550 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704800+0000", 2024-09-21T10:18:54.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023106299999999999 2024-09-21T10:18:54.550 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.550 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.550 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.551 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:198489 11.7 11:e9174170:::benchmark_data_smithi107_218812_object86002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-21T10:18:54.551 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:02.782897+0000", 2024-09-21T10:18:54.551 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 351.69100153699998, 2024-09-21T10:18:54.551 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0710279199999999, 2024-09-21T10:18:54.551 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:54.551 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:54.551 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:54.551 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:54.551 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:18:54.551 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:18:54.551 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 198489 2024-09-21T10:18:54.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.552 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:54.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.552 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:54.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.782897+0000", 2024-09-21T10:18:54.552 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.552 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:54.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.782897+0000", 2024-09-21T10:18:54.552 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.552 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:54.553 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.782899+0000", 2024-09-21T10:18:54.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8339999999999999e-06 2024-09-21T10:18:54.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.553 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:54.553 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.782905+0000", 2024-09-21T10:18:54.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9769999999999997e-06 2024-09-21T10:18:54.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.553 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:54.553 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.783003+0000", 2024-09-21T10:18:54.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7923000000000004e-05 2024-09-21T10:18:54.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.553 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:54.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.783012+0000", 2024-09-21T10:18:54.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7139999999999999e-06 2024-09-21T10:18:54.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.788021+0000", 2024-09-21T10:18:54.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0050089069999999999 2024-09-21T10:18:54.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.788029+0000", 2024-09-21T10:18:54.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9320000000000002e-06 2024-09-21T10:18:54.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.845768+0000", 2024-09-21T10:18:54.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0577382339999999 2024-09-21T10:18:54.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:18:54.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.845866+0000", 2024-09-21T10:18:54.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8791999999999999e-05 2024-09-21T10:18:54.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:18:54.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.845953+0000", 2024-09-21T10:18:54.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6781999999999997e-05 2024-09-21T10:18:54.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.853675+0000", 2024-09-21T10:18:54.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0077221039999999996 2024-09-21T10:18:54.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:18:54.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.853692+0000", 2024-09-21T10:18:54.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6699000000000001e-05 2024-09-21T10:18:54.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.853871+0000", 2024-09-21T10:18:54.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000178495 2024-09-21T10:18:54.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:18:54.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.853902+0000", 2024-09-21T10:18:54.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1554000000000003e-05 2024-09-21T10:18:54.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:18:54.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.853925+0000", 2024-09-21T10:18:54.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2972999999999998e-05 2024-09-21T10:18:54.557 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.557 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.558 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.558 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:198490 11.7 11:ee345632:::benchmark_data_smithi107_218812_object86003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-21T10:18:54.558 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:02.783038+0000", 2024-09-21T10:18:54.558 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 351.69086042700002, 2024-09-21T10:18:54.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.071039222, 2024-09-21T10:18:54.558 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:54.558 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:54.558 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:54.558 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:54.558 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:18:54.559 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:18:54.559 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 198490 2024-09-21T10:18:54.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.559 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:54.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:54.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.783038+0000", 2024-09-21T10:18:54.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:54.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.783038+0000", 2024-09-21T10:18:54.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:54.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.783039+0000", 2024-09-21T10:18:54.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.13e-07 2024-09-21T10:18:54.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:54.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.783042+0000", 2024-09-21T10:18:54.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0400000000000001e-06 2024-09-21T10:18:54.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:54.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.783050+0000", 2024-09-21T10:18:54.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6410000000000002e-06 2024-09-21T10:18:54.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:54.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.783055+0000", 2024-09-21T10:18:54.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5090000000000001e-06 2024-09-21T10:18:54.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.792028+0000", 2024-09-21T10:18:54.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0089732159999999991 2024-09-21T10:18:54.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.792037+0000", 2024-09-21T10:18:54.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.789e-06 2024-09-21T10:18:54.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.848714+0000", 2024-09-21T10:18:54.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0566770919999999 2024-09-21T10:18:54.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:18:54.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.848862+0000", 2024-09-21T10:18:54.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000147489 2024-09-21T10:18:54.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:18:54.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.848988+0000", 2024-09-21T10:18:54.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000125974 2024-09-21T10:18:54.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.853818+0000", 2024-09-21T10:18:54.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048301840000000004 2024-09-21T10:18:54.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:18:54.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.853937+0000", 2024-09-21T10:18:54.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011904399999999999 2024-09-21T10:18:54.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.854016+0000", 2024-09-21T10:18:54.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8781000000000003e-05 2024-09-21T10:18:54.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:18:54.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.854043+0000", 2024-09-21T10:18:54.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7506000000000001e-05 2024-09-21T10:18:54.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:18:54.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.854077+0000", 2024-09-21T10:18:54.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4344000000000002e-05 2024-09-21T10:18:54.565 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.565 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.565 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.565 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:198551 11.7 11:e9113b5a:::benchmark_data_smithi107_218812_object86064:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-21T10:18:54.566 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:02.847017+0000", 2024-09-21T10:18:54.566 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 351.62688139099998, 2024-09-21T10:18:54.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.010035579, 2024-09-21T10:18:54.566 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:54.566 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:54.566 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:54.566 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:54.566 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:18:54.566 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:18:54.566 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 198551 2024-09-21T10:18:54.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.567 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:54.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:54.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.847017+0000", 2024-09-21T10:18:54.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:54.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.847017+0000", 2024-09-21T10:18:54.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:54.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.847019+0000", 2024-09-21T10:18:54.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5710000000000001e-06 2024-09-21T10:18:54.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:54.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.847023+0000", 2024-09-21T10:18:54.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5560000000000001e-06 2024-09-21T10:18:54.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:54.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.847039+0000", 2024-09-21T10:18:54.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5608999999999999e-05 2024-09-21T10:18:54.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:54.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.847046+0000", 2024-09-21T10:18:54.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9959999999999996e-06 2024-09-21T10:18:54.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.847247+0000", 2024-09-21T10:18:54.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00020094199999999999 2024-09-21T10:18:54.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.847255+0000", 2024-09-21T10:18:54.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6279999999999996e-06 2024-09-21T10:18:54.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.851354+0000", 2024-09-21T10:18:54.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.004098999 2024-09-21T10:18:54.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:18:54.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.851455+0000", 2024-09-21T10:18:54.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101489 2024-09-21T10:18:54.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.571 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:18:54.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.851567+0000", 2024-09-21T10:18:54.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112173 2024-09-21T10:18:54.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.571 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.853969+0000", 2024-09-21T10:18:54.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024016509999999999 2024-09-21T10:18:54.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:18:54.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.853982+0000", 2024-09-21T10:18:54.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.331e-05 2024-09-21T10:18:54.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.856966+0000", 2024-09-21T10:18:54.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0029836670000000002 2024-09-21T10:18:54.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:18:54.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.856997+0000", 2024-09-21T10:18:54.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.15e-05 2024-09-21T10:18:54.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:18:54.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.857053+0000", 2024-09-21T10:18:54.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5488000000000001e-05 2024-09-21T10:18:54.573 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.574 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.574 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.574 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:198557 11.7 11:e69af3d7:::benchmark_data_smithi107_218812_object86070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-21T10:18:54.574 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:02.852910+0000", 2024-09-21T10:18:54.574 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 351.62098873899998, 2024-09-21T10:18:54.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0107109940000001, 2024-09-21T10:18:54.574 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:54.574 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:54.574 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:54.575 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:54.575 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:18:54.575 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:18:54.575 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 198557 2024-09-21T10:18:54.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.575 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:54.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:54.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.852910+0000", 2024-09-21T10:18:54.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:54.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.852910+0000", 2024-09-21T10:18:54.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:54.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.852912+0000", 2024-09-21T10:18:54.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.593e-06 2024-09-21T10:18:54.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:54.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.852917+0000", 2024-09-21T10:18:54.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5210000000000002e-06 2024-09-21T10:18:54.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:54.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.852935+0000", 2024-09-21T10:18:54.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8159999999999999e-05 2024-09-21T10:18:54.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:54.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.852944+0000", 2024-09-21T10:18:54.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6759999999999996e-06 2024-09-21T10:18:54.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.853018+0000", 2024-09-21T10:18:54.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4506999999999997e-05 2024-09-21T10:18:54.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.853029+0000", 2024-09-21T10:18:54.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0232e-05 2024-09-21T10:18:54.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.854128+0000", 2024-09-21T10:18:54.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.001099363 2024-09-21T10:18:54.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:18:54.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.854198+0000", 2024-09-21T10:18:54.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9879e-05 2024-09-21T10:18:54.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:18:54.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.854266+0000", 2024-09-21T10:18:54.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7897000000000004e-05 2024-09-21T10:18:54.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:18:54.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.863355+0000", 2024-09-21T10:18:54.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090896929999999994 2024-09-21T10:18:54.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.863404+0000", 2024-09-21T10:18:54.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8401e-05 2024-09-21T10:18:54.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.863499+0000", 2024-09-21T10:18:54.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5093999999999995e-05 2024-09-21T10:18:54.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:18:54.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.863568+0000", 2024-09-21T10:18:54.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9537999999999997e-05 2024-09-21T10:18:54.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:18:54.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.863621+0000", 2024-09-21T10:18:54.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2439999999999999e-05 2024-09-21T10:18:54.581 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.581 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.581 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.582 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:42515 12.e 12:7fea1a71:::benchmark_data_smithi107_250110_object42514:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-21T10:18:54.582 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:29.657893+0000", 2024-09-21T10:18:54.582 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 264.81600607799999, 2024-09-21T10:18:54.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1570628379999999, 2024-09-21T10:18:54.582 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:54.582 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:54.582 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:54.582 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:54.582 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:18:54.582 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:18:54.582 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 42515 2024-09-21T10:18:54.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.582 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:54.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:54.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.657893+0000", 2024-09-21T10:18:54.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:54.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.657893+0000", 2024-09-21T10:18:54.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:54.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.657894+0000", 2024-09-21T10:18:54.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6200000000000006e-07 2024-09-21T10:18:54.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:54.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.657916+0000", 2024-09-21T10:18:54.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2685999999999999e-05 2024-09-21T10:18:54.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:54.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.657931+0000", 2024-09-21T10:18:54.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4659e-05 2024-09-21T10:18:54.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:54.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.657938+0000", 2024-09-21T10:18:54.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4699999999999996e-06 2024-09-21T10:18:54.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.658022+0000", 2024-09-21T10:18:54.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4006999999999997e-05 2024-09-21T10:18:54.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.658032+0000", 2024-09-21T10:18:54.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9399999999999997e-06 2024-09-21T10:18:54.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698401+0000", 2024-09-21T10:18:54.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0403689709999999 2024-09-21T10:18:54.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698414+0000", 2024-09-21T10:18:54.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2863e-05 2024-09-21T10:18:54.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.801698+0000", 2024-09-21T10:18:54.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.103283749 2024-09-21T10:18:54.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:18:54.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.801861+0000", 2024-09-21T10:18:54.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016259600000000001 2024-09-21T10:18:54.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:18:54.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.802007+0000", 2024-09-21T10:18:54.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014699 2024-09-21T10:18:54.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.805969+0000", 2024-09-21T10:18:54.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0039619770000000002 2024-09-21T10:18:54.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:18:54.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.807396+0000", 2024-09-21T10:18:54.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014265969999999999 2024-09-21T10:18:54.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.812802+0000", 2024-09-21T10:18:54.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054060920000000004 2024-09-21T10:18:54.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:18:54.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.812844+0000", 2024-09-21T10:18:54.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2321999999999999e-05 2024-09-21T10:18:54.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:18:54.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.814955+0000", 2024-09-21T10:18:54.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021109570000000001 2024-09-21T10:18:54.591 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.591 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.591 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.592 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:42540 12.e 12:773d0c07:::benchmark_data_smithi107_250110_object42539:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-21T10:18:54.592 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:29.679940+0000", 2024-09-21T10:18:54.592 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 264.79395856100001, 2024-09-21T10:18:54.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1367923950000001, 2024-09-21T10:18:54.592 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:54.592 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:54.592 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:54.592 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:54.592 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:18:54.592 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:18:54.592 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 42540 2024-09-21T10:18:54.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.593 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:54.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:54.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.679940+0000", 2024-09-21T10:18:54.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:54.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.679940+0000", 2024-09-21T10:18:54.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:54.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.679942+0000", 2024-09-21T10:18:54.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.745e-06 2024-09-21T10:18:54.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:54.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.679969+0000", 2024-09-21T10:18:54.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6673999999999999e-05 2024-09-21T10:18:54.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:54.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.679986+0000", 2024-09-21T10:18:54.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7796e-05 2024-09-21T10:18:54.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:54.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.679994+0000", 2024-09-21T10:18:54.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0080000000000002e-06 2024-09-21T10:18:54.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.680526+0000", 2024-09-21T10:18:54.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00053167399999999999 2024-09-21T10:18:54.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.680537+0000", 2024-09-21T10:18:54.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0528e-05 2024-09-21T10:18:54.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698441+0000", 2024-09-21T10:18:54.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.017904532 2024-09-21T10:18:54.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698449+0000", 2024-09-21T10:18:54.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8199999999999997e-06 2024-09-21T10:18:54.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.804541+0000", 2024-09-21T10:18:54.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.106092007 2024-09-21T10:18:54.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:18:54.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.804639+0000", 2024-09-21T10:18:54.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7731000000000003e-05 2024-09-21T10:18:54.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:18:54.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.804751+0000", 2024-09-21T10:18:54.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011230099999999999 2024-09-21T10:18:54.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:18:54.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.808849+0000", 2024-09-21T10:18:54.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004098272 2024-09-21T10:18:54.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.810325+0000", 2024-09-21T10:18:54.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00147554 2024-09-21T10:18:54.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.815039+0000", 2024-09-21T10:18:54.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004714088 2024-09-21T10:18:54.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:18:54.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.815108+0000", 2024-09-21T10:18:54.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9041999999999994e-05 2024-09-21T10:18:54.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:18:54.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.816733+0000", 2024-09-21T10:18:54.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001624637 2024-09-21T10:18:54.600 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.600 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.600 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.600 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:42557 12.e 12:721663e1:::benchmark_data_smithi107_250110_object42556:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-21T10:18:54.600 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:29.690624+0000", 2024-09-21T10:18:54.601 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 264.78327427599999, 2024-09-21T10:18:54.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1350235880000001, 2024-09-21T10:18:54.601 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:54.601 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:54.601 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:54.601 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:54.601 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:18:54.601 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:18:54.601 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 42557 2024-09-21T10:18:54.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.601 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:54.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:54.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.690624+0000", 2024-09-21T10:18:54.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:54.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.690624+0000", 2024-09-21T10:18:54.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:54.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.690626+0000", 2024-09-21T10:18:54.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.556e-06 2024-09-21T10:18:54.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:54.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.690647+0000", 2024-09-21T10:18:54.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1384999999999999e-05 2024-09-21T10:18:54.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:54.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.690664+0000", 2024-09-21T10:18:54.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6745999999999999e-05 2024-09-21T10:18:54.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:54.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.690671+0000", 2024-09-21T10:18:54.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2350000000000002e-06 2024-09-21T10:18:54.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.690801+0000", 2024-09-21T10:18:54.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012969899999999999 2024-09-21T10:18:54.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.690810+0000", 2024-09-21T10:18:54.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9239999999999996e-06 2024-09-21T10:18:54.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698474+0000", 2024-09-21T10:18:54.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.007663778 2024-09-21T10:18:54.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698481+0000", 2024-09-21T10:18:54.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.25e-06 2024-09-21T10:18:54.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.806017+0000", 2024-09-21T10:18:54.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.107536251 2024-09-21T10:18:54.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:18:54.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.806173+0000", 2024-09-21T10:18:54.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015593900000000001 2024-09-21T10:18:54.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:18:54.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.806333+0000", 2024-09-21T10:18:54.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000160209 2024-09-21T10:18:54.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:18:54.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.812687+0000", 2024-09-21T10:18:54.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0063535359999999999 2024-09-21T10:18:54.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.812747+0000", 2024-09-21T10:18:54.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880999999999998e-05 2024-09-21T10:18:54.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.823402+0000", 2024-09-21T10:18:54.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010655695999999999 2024-09-21T10:18:54.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:18:54.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.823457+0000", 2024-09-21T10:18:54.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4083999999999999e-05 2024-09-21T10:18:54.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:18:54.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.825648+0000", 2024-09-21T10:18:54.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002191419 2024-09-21T10:18:54.609 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.609 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.609 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.609 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:42584 12.e 12:7ce522fc:::benchmark_data_smithi107_250110_object42583:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-21T10:18:54.609 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:29.716131+0000", 2024-09-21T10:18:54.610 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 264.75776719999999, 2024-09-21T10:18:54.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1227035999999999, 2024-09-21T10:18:54.610 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:54.610 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:54.610 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:54.610 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:54.610 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:18:54.610 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:18:54.610 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 42584 2024-09-21T10:18:54.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.610 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:54.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:54.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.716131+0000", 2024-09-21T10:18:54.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:54.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.716131+0000", 2024-09-21T10:18:54.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:54.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.716133+0000", 2024-09-21T10:18:54.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.72e-06 2024-09-21T10:18:54.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:54.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.716172+0000", 2024-09-21T10:18:54.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8525999999999997e-05 2024-09-21T10:18:54.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:54.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.716185+0000", 2024-09-21T10:18:54.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3189e-05 2024-09-21T10:18:54.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:54.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.716192+0000", 2024-09-21T10:18:54.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4850000000000003e-06 2024-09-21T10:18:54.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.716262+0000", 2024-09-21T10:18:54.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0055999999999995e-05 2024-09-21T10:18:54.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.716273+0000", 2024-09-21T10:18:54.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0379e-05 2024-09-21T10:18:54.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698504+0000", 2024-09-21T10:18:54.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.982231101 2024-09-21T10:18:54.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698511+0000", 2024-09-21T10:18:54.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6619999999999999e-06 2024-09-21T10:18:54.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.807514+0000", 2024-09-21T10:18:54.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1090034499999999 2024-09-21T10:18:54.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:18:54.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.807637+0000", 2024-09-21T10:18:54.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012253500000000001 2024-09-21T10:18:54.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:18:54.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.807782+0000", 2024-09-21T10:18:54.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000145351 2024-09-21T10:18:54.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:18:54.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.814974+0000", 2024-09-21T10:18:54.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071919769999999996 2024-09-21T10:18:54.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.816796+0000", 2024-09-21T10:18:54.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00182252 2024-09-21T10:18:54.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.834741+0000", 2024-09-21T10:18:54.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017944882999999998 2024-09-21T10:18:54.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:18:54.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.834979+0000", 2024-09-21T10:18:54.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000237743 2024-09-21T10:18:54.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:18:54.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.838835+0000", 2024-09-21T10:18:54.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038560230000000001 2024-09-21T10:18:54.617 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.617 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.617 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.617 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:42600 12.e 12:702dc90f:::benchmark_data_smithi107_250110_object42599:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-21T10:18:54.617 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:29.723485+0000", 2024-09-21T10:18:54.617 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 264.75041339500001, 2024-09-21T10:18:54.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1177167639999999, 2024-09-21T10:18:54.618 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:54.618 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:54.618 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:54.618 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:54.618 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:18:54.618 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:18:54.618 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 42600 2024-09-21T10:18:54.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.618 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:54.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:54.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.723485+0000", 2024-09-21T10:18:54.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:54.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.723485+0000", 2024-09-21T10:18:54.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:54.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.723486+0000", 2024-09-21T10:18:54.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3399999999999997e-07 2024-09-21T10:18:54.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:54.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.723507+0000", 2024-09-21T10:18:54.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0400000000000001e-05 2024-09-21T10:18:54.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:54.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.723516+0000", 2024-09-21T10:18:54.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7540000000000001e-06 2024-09-21T10:18:54.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:54.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.723523+0000", 2024-09-21T10:18:54.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0990000000000001e-06 2024-09-21T10:18:54.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.723582+0000", 2024-09-21T10:18:54.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8693000000000002e-05 2024-09-21T10:18:54.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.723590+0000", 2024-09-21T10:18:54.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8550000000000002e-06 2024-09-21T10:18:54.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698533+0000", 2024-09-21T10:18:54.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.974942749 2024-09-21T10:18:54.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698540+0000", 2024-09-21T10:18:54.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6079999999999998e-06 2024-09-21T10:18:54.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.808901+0000", 2024-09-21T10:18:54.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1103609670000001 2024-09-21T10:18:54.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:18:54.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.809007+0000", 2024-09-21T10:18:54.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105257 2024-09-21T10:18:54.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:18:54.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.809104+0000", 2024-09-21T10:18:54.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7371e-05 2024-09-21T10:18:54.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:18:54.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.814981+0000", 2024-09-21T10:18:54.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0058765559999999998 2024-09-21T10:18:54.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.816845+0000", 2024-09-21T10:18:54.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0018647920000000001 2024-09-21T10:18:54.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.838903+0000", 2024-09-21T10:18:54.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022057562999999999 2024-09-21T10:18:54.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:18:54.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.838975+0000", 2024-09-21T10:18:54.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2612000000000002e-05 2024-09-21T10:18:54.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:18:54.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.841202+0000", 2024-09-21T10:18:54.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022265539999999999 2024-09-21T10:18:54.625 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.625 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.625 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.625 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:42620 12.e 12:7684604d:::benchmark_data_smithi107_250110_object42619:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-21T10:18:54.625 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:29.823460+0000", 2024-09-21T10:18:54.626 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 264.65043874899999, 2024-09-21T10:18:54.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0200784430000001, 2024-09-21T10:18:54.626 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:54.626 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:54.626 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:54.626 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:54.626 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:18:54.626 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:18:54.626 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 42620 2024-09-21T10:18:54.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.626 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:54.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:54.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.823460+0000", 2024-09-21T10:18:54.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:54.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.823460+0000", 2024-09-21T10:18:54.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:54.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.823461+0000", 2024-09-21T10:18:54.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2300000000000001e-06 2024-09-21T10:18:54.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:54.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.823482+0000", 2024-09-21T10:18:54.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0639e-05 2024-09-21T10:18:54.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:54.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.823494+0000", 2024-09-21T10:18:54.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2225000000000001e-05 2024-09-21T10:18:54.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:54.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.823502+0000", 2024-09-21T10:18:54.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5789999999999998e-06 2024-09-21T10:18:54.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.823581+0000", 2024-09-21T10:18:54.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9048999999999995e-05 2024-09-21T10:18:54.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.823590+0000", 2024-09-21T10:18:54.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.02e-06 2024-09-21T10:18:54.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698565+0000", 2024-09-21T10:18:54.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.87497485200000003 2024-09-21T10:18:54.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698571+0000", 2024-09-21T10:18:54.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6819999999999997e-06 2024-09-21T10:18:54.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.633 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.810360+0000", 2024-09-21T10:18:54.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1117890880000001 2024-09-21T10:18:54.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:18:54.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.810490+0000", 2024-09-21T10:18:54.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000129555 2024-09-21T10:18:54.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:18:54.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.810640+0000", 2024-09-21T10:18:54.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015038900000000001 2024-09-21T10:18:54.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:18:54.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.814987+0000", 2024-09-21T10:18:54.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043463479999999999 2024-09-21T10:18:54.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.816887+0000", 2024-09-21T10:18:54.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019008199999999999 2024-09-21T10:18:54.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.841269+0000", 2024-09-21T10:18:54.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024381554999999999 2024-09-21T10:18:54.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:18:54.638 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.841335+0000", 2024-09-21T10:18:54.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5772000000000007e-05 2024-09-21T10:18:54.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.638 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.638 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:18:54.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.843538+0000", 2024-09-21T10:18:54.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022036400000000002 2024-09-21T10:18:54.639 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.639 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.639 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.640 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.640 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:42631 12.e 12:7324e759:::benchmark_data_smithi107_250110_object42630:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-21T10:18:54.640 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:29.831568+0000", 2024-09-21T10:18:54.640 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 264.64233053999999, 2024-09-21T10:18:54.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0140101700000002, 2024-09-21T10:18:54.640 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:54.640 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:54.641 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:54.641 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:54.641 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:18:54.641 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:18:54.641 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 42631 2024-09-21T10:18:54.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.642 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:54.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:54.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.831568+0000", 2024-09-21T10:18:54.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:54.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.831568+0000", 2024-09-21T10:18:54.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:54.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.831570+0000", 2024-09-21T10:18:54.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8309999999999999e-06 2024-09-21T10:18:54.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.644 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:54.644 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.831591+0000", 2024-09-21T10:18:54.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0667000000000001e-05 2024-09-21T10:18:54.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.644 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:54.644 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.831606+0000", 2024-09-21T10:18:54.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5051999999999999e-05 2024-09-21T10:18:54.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.644 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:54.644 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.831613+0000", 2024-09-21T10:18:54.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6109999999999996e-06 2024-09-21T10:18:54.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.831665+0000", 2024-09-21T10:18:54.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1430000000000001e-05 2024-09-21T10:18:54.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.831672+0000", 2024-09-21T10:18:54.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1749999999999999e-06 2024-09-21T10:18:54.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698593+0000", 2024-09-21T10:18:54.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.86692127299999999 2024-09-21T10:18:54.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.646 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698601+0000", 2024-09-21T10:18:54.646 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7170000000000008e-06 2024-09-21T10:18:54.646 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:54.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.811527+0000", 2024-09-21T10:18:54.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1129258339999999 2024-09-21T10:18:54.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:18:54.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.811670+0000", 2024-09-21T10:18:54.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014346899999999999 2024-09-21T10:18:54.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:18:54.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.811814+0000", 2024-09-21T10:18:54.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000143905 2024-09-21T10:18:54.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:18:54.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.816752+0000", 2024-09-21T10:18:54.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0049376680000000001 2024-09-21T10:18:54.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.817356+0000", 2024-09-21T10:18:54.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00060429899999999998 2024-09-21T10:18:54.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.843622+0000", 2024-09-21T10:18:54.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.026265481 2024-09-21T10:18:54.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.650 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.650 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:18:54.650 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.843679+0000", 2024-09-21T10:18:54.650 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7342000000000001e-05 2024-09-21T10:18:54.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.650 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.650 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:18:54.650 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.845578+0000", 2024-09-21T10:18:54.650 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0018994159999999999 2024-09-21T10:18:54.650 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.650 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.650 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.650 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.651 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.651 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135241 12.10 12.514f79d0 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T10:18:54.651 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:09.574571+0000", 2024-09-21T10:18:54.651 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 164.89932751200001, 2024-09-21T10:18:54.651 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0064763960000001, 2024-09-21T10:18:54.651 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:54.651 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:54.651 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:18:54.651 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:54.651 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:18:54.651 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:18:54.651 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135241 2024-09-21T10:18:54.651 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.651 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:54.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:54.652 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.574571+0000", 2024-09-21T10:18:54.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:54.652 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.574571+0000", 2024-09-21T10:18:54.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.652 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.652 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.652 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:54.652 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.574572+0000", 2024-09-21T10:18:54.652 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4499999999999997e-07 2024-09-21T10:18:54.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:54.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.574574+0000", 2024-09-21T10:18:54.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.967e-06 2024-09-21T10:18:54.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:54.653 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.574580+0000", 2024-09-21T10:18:54.653 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2990000000000002e-06 2024-09-21T10:18:54.653 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.653 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.653 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:54.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.574584+0000", 2024-09-21T10:18:54.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6140000000000002e-06 2024-09-21T10:18:54.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.577644+0000", 2024-09-21T10:18:54.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0030592739999999998 2024-09-21T10:18:54.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.654 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.654 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.654 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.577651+0000", 2024-09-21T10:18:54.654 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7000000000000008e-06 2024-09-21T10:18:54.654 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581037+0000", 2024-09-21T10:18:54.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0033855620000001 2024-09-21T10:18:54.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.655 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.655 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:18:54.655 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581048+0000", 2024-09-21T10:18:54.655 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0535e-05 2024-09-21T10:18:54.655 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.655 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.655 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.655 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.656 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.656 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135271 12.10 12.ae4ac6b0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T10:18:54.656 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:09.599884+0000", 2024-09-21T10:18:54.656 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 164.874014996, 2024-09-21T10:18:54.656 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.98119575800000003, 2024-09-21T10:18:54.656 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:54.656 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:54.656 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:18:54.656 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:54.656 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:18:54.656 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:18:54.656 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135271 2024-09-21T10:18:54.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.657 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:54.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:54.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.599884+0000", 2024-09-21T10:18:54.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.657 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.657 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:54.657 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.599884+0000", 2024-09-21T10:18:54.657 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.657 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.658 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:54.658 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.599884+0000", 2024-09-21T10:18:54.658 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9999999999999998e-07 2024-09-21T10:18:54.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.658 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.658 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:54.658 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.599887+0000", 2024-09-21T10:18:54.658 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5840000000000002e-06 2024-09-21T10:18:54.658 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.659 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:54.659 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.599892+0000", 2024-09-21T10:18:54.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5589999999999996e-06 2024-09-21T10:18:54.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.659 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:54.659 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.599898+0000", 2024-09-21T10:18:54.659 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2490000000000001e-06 2024-09-21T10:18:54.659 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.659 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.660 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.599952+0000", 2024-09-21T10:18:54.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3968999999999998e-05 2024-09-21T10:18:54.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.660 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.599960+0000", 2024-09-21T10:18:54.660 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5210000000000002e-06 2024-09-21T10:18:54.660 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.660 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.660 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.660 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581070+0000", 2024-09-21T10:18:54.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.98110948899999995 2024-09-21T10:18:54.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.661 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.661 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:18:54.661 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581079+0000", 2024-09-21T10:18:54.661 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8870000000000004e-06 2024-09-21T10:18:54.661 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.661 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.661 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.661 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.662 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.662 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135282 12.10 12.125efe10 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T10:18:54.662 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:09.612674+0000", 2024-09-21T10:18:54.662 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 164.861224682, 2024-09-21T10:18:54.662 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.96844876599999996, 2024-09-21T10:18:54.662 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:54.662 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:54.662 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:18:54.662 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:54.662 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:18:54.662 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:18:54.662 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135282 2024-09-21T10:18:54.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.663 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:54.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.663 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:54.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.612674+0000", 2024-09-21T10:18:54.663 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.663 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.663 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.663 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:54.663 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.612674+0000", 2024-09-21T10:18:54.663 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.664 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:54.664 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.612675+0000", 2024-09-21T10:18:54.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.299e-06 2024-09-21T10:18:54.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.664 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:54.664 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.612681+0000", 2024-09-21T10:18:54.664 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2530000000000004e-06 2024-09-21T10:18:54.664 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.664 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:54.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.612696+0000", 2024-09-21T10:18:54.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5328999999999998e-05 2024-09-21T10:18:54.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:54.665 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.612703+0000", 2024-09-21T10:18:54.665 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3900000000000004e-06 2024-09-21T10:18:54.665 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.665 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.665 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.615135+0000", 2024-09-21T10:18:54.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024312639999999998 2024-09-21T10:18:54.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.615143+0000", 2024-09-21T10:18:54.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2910000000000007e-06 2024-09-21T10:18:54.666 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.666 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.666 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.666 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581113+0000", 2024-09-21T10:18:54.666 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.96596997900000003 2024-09-21T10:18:54.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.667 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:18:54.667 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581123+0000", 2024-09-21T10:18:54.667 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.961e-06 2024-09-21T10:18:54.667 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.667 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.667 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.667 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.667 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.667 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135289 12.10 12.79179630 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T10:18:54.667 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:09.619177+0000", 2024-09-21T10:18:54.668 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 164.854721705, 2024-09-21T10:18:54.668 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.96196555699999997, 2024-09-21T10:18:54.668 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:54.668 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:54.668 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:18:54.668 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:54.668 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:18:54.668 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:18:54.668 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135289 2024-09-21T10:18:54.668 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.668 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:54.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:54.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.619177+0000", 2024-09-21T10:18:54.669 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:54.669 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.619177+0000", 2024-09-21T10:18:54.669 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.669 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.669 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.669 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:54.670 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.619178+0000", 2024-09-21T10:18:54.670 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3829999999999999e-06 2024-09-21T10:18:54.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.670 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:54.670 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.619182+0000", 2024-09-21T10:18:54.670 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2710000000000001e-06 2024-09-21T10:18:54.670 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.670 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.670 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:54.670 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.619191+0000", 2024-09-21T10:18:54.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2150000000000007e-06 2024-09-21T10:18:54.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:54.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.619198+0000", 2024-09-21T10:18:54.671 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.669e-06 2024-09-21T10:18:54.671 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.671 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.671 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.671 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.619418+0000", 2024-09-21T10:18:54.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022047200000000001 2024-09-21T10:18:54.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.672 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.619426+0000", 2024-09-21T10:18:54.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3380000000000005e-06 2024-09-21T10:18:54.672 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.672 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.672 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.672 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581137+0000", 2024-09-21T10:18:54.672 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.96171104500000004 2024-09-21T10:18:54.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.673 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:18:54.673 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581143+0000", 2024-09-21T10:18:54.673 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.164e-06 2024-09-21T10:18:54.673 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.673 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.673 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.673 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.673 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.673 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:177097 12.19 12:9f836ee2:::benchmark_data_smithi107_250110_object65687:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e353)", 2024-09-21T10:18:54.673 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:17:06.629165+0000", 2024-09-21T10:18:54.673 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 107.84473401699999, 2024-09-21T10:18:54.674 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0158801159999999, 2024-09-21T10:18:54.674 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:54.674 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:54.674 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:54.674 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:54.674 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:18:54.674 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:18:54.674 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 177097 2024-09-21T10:18:54.674 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.674 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:54.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.675 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:54.675 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.629165+0000", 2024-09-21T10:18:54.675 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.675 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:54.675 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.629165+0000", 2024-09-21T10:18:54.675 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.675 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.675 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:54.675 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.629166+0000", 2024-09-21T10:18:54.675 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1769999999999999e-06 2024-09-21T10:18:54.675 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:54.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.629169+0000", 2024-09-21T10:18:54.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.818e-06 2024-09-21T10:18:54.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:54.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.629177+0000", 2024-09-21T10:18:54.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9549999999999992e-06 2024-09-21T10:18:54.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.676 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.676 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:54.676 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.629183+0000", 2024-09-21T10:18:54.676 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0410000000000003e-06 2024-09-21T10:18:54.676 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.677 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.629693+0000", 2024-09-21T10:18:54.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051066800000000004 2024-09-21T10:18:54.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.677 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.629702+0000", 2024-09-21T10:18:54.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7749999999999998e-06 2024-09-21T10:18:54.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.677 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.677 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.641125+0000", 2024-09-21T10:18:54.677 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0114230399999999 2024-09-21T10:18:54.677 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.677 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.678 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:18:54.678 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.641262+0000", 2024-09-21T10:18:54.678 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013656000000000001 2024-09-21T10:18:54.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.678 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:18:54.678 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.641592+0000", 2024-09-21T10:18:54.678 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00033042800000000002 2024-09-21T10:18:54.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.678 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.678 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.678 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.644895+0000", 2024-09-21T10:18:54.678 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003302532 2024-09-21T10:18:54.678 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.679 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.679 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.679 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:54 smithi107 ceph-mon[102221]: pgmap v1274: 109 pgs: 109 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-21T10:18:54.679 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:54 smithi107 ceph-mon[105022]: pgmap v1274: 109 pgs: 109 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-21T10:18:54.680 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.644943+0000", 2024-09-21T10:18:54.680 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8390000000000003e-05 2024-09-21T10:18:54.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.680 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:18:54.680 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.644957+0000", 2024-09-21T10:18:54.680 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4253999999999999e-05 2024-09-21T10:18:54.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.680 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.680 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:18:54.680 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.645004+0000", 2024-09-21T10:18:54.680 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6802999999999999e-05 2024-09-21T10:18:54.680 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.681 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:18:54.681 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.645045+0000", 2024-09-21T10:18:54.681 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0674999999999999e-05 2024-09-21T10:18:54.681 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.681 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.681 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.681 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.681 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.681 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:177159 12.19 12:9c5c0da3:::benchmark_data_smithi107_250110_object65749:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-21T10:18:54.681 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:17:06.690554+0000", 2024-09-21T10:18:54.681 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 107.783344564, 2024-09-21T10:18:54.681 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.95615797899999999, 2024-09-21T10:18:54.681 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:54.681 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:54.682 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:54.682 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:54.682 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:18:54.682 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:18:54.682 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 177159 2024-09-21T10:18:54.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.682 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:54.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.682 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:54.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.690554+0000", 2024-09-21T10:18:54.682 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.682 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.682 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.682 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:54.682 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.690554+0000", 2024-09-21T10:18:54.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:54.683 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.690556+0000", 2024-09-21T10:18:54.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5340000000000001e-06 2024-09-21T10:18:54.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:54.683 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.690560+0000", 2024-09-21T10:18:54.683 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8829999999999999e-06 2024-09-21T10:18:54.683 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.683 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.683 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:54.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.690569+0000", 2024-09-21T10:18:54.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3060000000000003e-06 2024-09-21T10:18:54.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.684 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:54.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.690575+0000", 2024-09-21T10:18:54.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7459999999999997e-06 2024-09-21T10:18:54.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.684 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.684 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.690633+0000", 2024-09-21T10:18:54.684 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8150999999999999e-05 2024-09-21T10:18:54.684 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.684 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.684 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.685 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.690642+0000", 2024-09-21T10:18:54.685 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3530000000000001e-06 2024-09-21T10:18:54.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.685 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.685 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.643577+0000", 2024-09-21T10:18:54.685 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.95293516700000003 2024-09-21T10:18:54.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.685 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:18:54.685 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.643683+0000", 2024-09-21T10:18:54.685 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105846 2024-09-21T10:18:54.685 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.685 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.686 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:18:54.686 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.643887+0000", 2024-09-21T10:18:54.686 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000204088 2024-09-21T10:18:54.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.686 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.686 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.646574+0000", 2024-09-21T10:18:54.686 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002686842 2024-09-21T10:18:54.686 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.686 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.686 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.686 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.646627+0000", 2024-09-21T10:18:54.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3195000000000001e-05 2024-09-21T10:18:54.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.687 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:18:54.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.646641+0000", 2024-09-21T10:18:54.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3672000000000001e-05 2024-09-21T10:18:54.687 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.687 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.687 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:18:54.687 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.646679+0000", 2024-09-21T10:18:54.687 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8436000000000003e-05 2024-09-21T10:18:54.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.688 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:18:54.688 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.646712+0000", 2024-09-21T10:18:54.688 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2759999999999998e-05 2024-09-21T10:18:54.688 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.688 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.688 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.688 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.688 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.688 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:177165 12.19 12:9fc71d39:::benchmark_data_smithi107_250110_object65755:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-21T10:18:54.688 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:17:06.691575+0000", 2024-09-21T10:18:54.688 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 107.78232362999999, 2024-09-21T10:18:54.689 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.95986069100000004, 2024-09-21T10:18:54.689 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:18:54.689 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:18:54.689 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:18:54.689 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:18:54.689 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:18:54.689 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:18:54.689 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 177165 2024-09-21T10:18:54.689 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.689 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:18:54.689 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.689 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:18:54.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.691575+0000", 2024-09-21T10:18:54.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.690 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:18:54.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.691575+0000", 2024-09-21T10:18:54.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:18:54.690 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.690 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:18:54.690 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.691576+0000", 2024-09-21T10:18:54.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4300000000000003e-07 2024-09-21T10:18:54.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:18:54.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.691579+0000", 2024-09-21T10:18:54.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.022e-06 2024-09-21T10:18:54.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:18:54.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.691586+0000", 2024-09-21T10:18:54.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9859999999999997e-06 2024-09-21T10:18:54.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:18:54.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.691591+0000", 2024-09-21T10:18:54.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9699999999999998e-06 2024-09-21T10:18:54.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.691637+0000", 2024-09-21T10:18:54.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6480999999999998e-05 2024-09-21T10:18:54.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:18:54.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.691643+0000", 2024-09-21T10:18:54.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2260000000000002e-06 2024-09-21T10:18:54.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.693 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:18:54.693 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.645059+0000", 2024-09-21T10:18:54.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.95341608300000003 2024-09-21T10:18:54.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.693 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:18:54.693 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.645126+0000", 2024-09-21T10:18:54.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7008999999999995e-05 2024-09-21T10:18:54.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.694 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:18:54.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.645291+0000", 2024-09-21T10:18:54.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000164387 2024-09-21T10:18:54.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.694 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:18:54.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.651202+0000", 2024-09-21T10:18:54.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0059107969999999998 2024-09-21T10:18:54.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.695 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.695 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.651246+0000", 2024-09-21T10:18:54.695 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4529e-05 2024-09-21T10:18:54.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.695 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:18:54.695 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.651285+0000", 2024-09-21T10:18:54.695 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9368e-05 2024-09-21T10:18:54.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.695 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:18:54.696 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.651361+0000", 2024-09-21T10:18:54.696 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5117999999999995e-05 2024-09-21T10:18:54.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:18:54.696 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:18:54.696 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:18:54.696 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.651436+0000", 2024-09-21T10:18:54.696 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5172000000000005e-05 2024-09-21T10:18:54.696 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.696 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.696 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.696 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:18:54.696 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:18:54.696 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:18:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:54.698 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:54.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:54.799 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:54.833 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:54.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:54.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:54 smithi129 ceph-mon[88471]: pgmap v1274: 109 pgs: 109 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-21T10:18:54.935 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:54.968 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:54.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:54.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:55.064 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 33 16 65307 65291 123.63 88.6875 0.0116272 0.00808343 2024-09-21T10:18:55.070 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:55.111 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:55.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:55.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:55.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:18:55.213 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:55.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:55.247 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:55.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:55.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:55.393 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:55.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:55.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:55.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:55.536 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:55.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:55.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:55.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:55.673 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:55.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:55.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:55.774 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:55.814 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:55.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:55.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:55.918 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:55.956 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:55.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:55.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:56.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:56.064 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 34 16 66665 66649 122.489 84.875 0.0123635 0.008159 2024-09-21T10:18:56.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:56.094 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:56.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:56.195 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:56.230 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:56.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:56.332 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:56.367 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:56.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:56.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:56.469 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:56.504 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:56.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:56.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:56.606 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:56.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:56.646 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:56.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:56.747 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:56.785 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:56.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:56.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:56 smithi107 ceph-mon[102221]: pgmap v1275: 109 pgs: 109 active+clean; 3.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-09-21T10:18:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:56.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:56 smithi107 ceph-mon[105022]: pgmap v1275: 109 pgs: 109 active+clean; 3.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-09-21T10:18:56.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:56.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:56.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:56 smithi129 ceph-mon[88471]: pgmap v1275: 109 pgs: 109 active+clean; 3.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s 2024-09-21T10:18:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:18:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:56.935 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:56.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:56.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:57.037 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:57.064 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 35 16 68096 68080 121.544 89.4375 0.0200856 0.00822243 2024-09-21T10:18:57.074 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:57.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:57.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:57.094 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:18:57.097 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:18:57.176 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:57.210 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:57.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:57.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:57.312 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:57.345 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:57.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:57.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:57.447 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:18:57.481 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:57.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:57.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:18:57.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:18:57.582 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:57.621 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:57.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:57.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:57.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:57.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:57.777 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:57.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:57.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:57 smithi107 ceph-mon[102221]: from='client.46907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:18:57.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:57 smithi107 ceph-mon[105022]: from='client.46907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:18:57.879 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:57.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:57 smithi129 ceph-mon[88471]: from='client.46907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:18:57.921 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:57.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:57.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:58.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:58.060 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:58.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:58.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:58.064 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 36 16 69532 69516 120.661 89.75 0.0101298 0.00828277 2024-09-21T10:18:58.163 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:58.201 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:58.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:58.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:58.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:58.336 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:58.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:58.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:58.438 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:58.468 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:58.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:58.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:58.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:58.606 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:58.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:58.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:58.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:18:58.731 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:58.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:18:58.833 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:18:58 smithi107 ceph-mon[102221]: pgmap v1276: 109 pgs: 109 active+clean; 4.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 107 MiB/s wr, 1.70k op/s 2024-09-21T10:18:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:18:58 smithi107 ceph-mon[105022]: pgmap v1276: 109 pgs: 109 active+clean; 4.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 107 MiB/s wr, 1.70k op/s 2024-09-21T10:18:58.870 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:58.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:58.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:18:58 smithi129 ceph-mon[88471]: pgmap v1276: 109 pgs: 109 active+clean; 4.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 107 MiB/s wr, 1.70k op/s 2024-09-21T10:18:58.972 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:18:59.011 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:59.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:59.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:18:59.064 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 37 16 70893 70877 119.699 85.0625 0.00846907 0.00834931 2024-09-21T10:18:59.113 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:18:59.149 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:59.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:59.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:18:59.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:59.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:59.275 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:59.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:59.376 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:59.411 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:59.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:59.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:59.513 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:18:59.548 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:18:59.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:59.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:18:59.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:18:59.682 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:59.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:59.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:18:59.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:18:59.820 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:59.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:59.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:18:59.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:18:59.957 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:18:59.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:18:59.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:00.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:00.064 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 38 16 72285 72269 118.838 87 0.0098758 0.00841002 2024-09-21T10:19:00.092 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:00.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:00.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:00.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:00.215 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:00.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:00.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:00.317 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:00.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:00.354 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:00.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:00.455 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:00.495 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:00.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:00.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:00.517 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:19:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:19:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:19:00.597 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:00.618 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:00.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:00.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:00.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:00.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:00.742 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:00.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:00.844 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:00.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:00 smithi107 ceph-mon[102221]: pgmap v1277: 109 pgs: 109 active+clean; 4.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-21T10:19:00.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:00 smithi107 ceph-mon[105022]: pgmap v1277: 109 pgs: 109 active+clean; 4.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-21T10:19:00.865 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:00.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:00.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:00.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:00 smithi129 ceph-mon[88471]: pgmap v1277: 109 pgs: 109 active+clean; 4.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-21T10:19:00.966 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:00.988 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:00.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:00.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:01.065 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 39 16 73135 73119 117.153 53.125 0.0125364 0.00853169 2024-09-21T10:19:01.090 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:01.129 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:01.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:01.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:01.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:01.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:01.263 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:01.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:01.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:01.391 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:01.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:01.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:01.493 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:01.528 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:01.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:01.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:01.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:01.664 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:01.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:01.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:01.795 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:01.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:01.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:01.897 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:01.934 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:01.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:01.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:02.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:02.064 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:02.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:02.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:02.065 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:19:02.063748+0000 min lat: 0.00115007 max lat: 0.53677 avg lat: 0.00859318 2024-09-21T10:19:02.065 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:19:02.065 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 40 16 74461 74445 116.296 82.875 0.0146733 0.00859318 2024-09-21T10:19:02.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:19:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:19:01] "GET /metrics HTTP/1.1" 200 36783 "" "Prometheus/2.51.0" 2024-09-21T10:19:02.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:02.198 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:02.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:02.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:02.300 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:02.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:02.321 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:02.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:02.422 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:02.466 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:02.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:02.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:02.567 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:02.607 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:02.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:02.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:02.709 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:02.735 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:02.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:02.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:02.836 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:02 smithi107 ceph-mon[102221]: pgmap v1278: 109 pgs: 109 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 82 MiB/s wr, 1.30k op/s 2024-09-21T10:19:02.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:02 smithi107 ceph-mon[105022]: pgmap v1278: 109 pgs: 109 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 82 MiB/s wr, 1.30k op/s 2024-09-21T10:19:02.872 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:02.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:02.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:02.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:02 smithi129 ceph-mon[88471]: pgmap v1278: 109 pgs: 109 active+clean; 4.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 82 MiB/s wr, 1.30k op/s 2024-09-21T10:19:02.974 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:03.010 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:03.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:03.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:03.065 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 41 16 75652 75636 115.275 74.4375 0.00916743 0.0086704 2024-09-21T10:19:03.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:03.139 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:03.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:03.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:03.241 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:03.279 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:03.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:03.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:03.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:03.415 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:03.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:03.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:03.517 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:03.555 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:03.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:03.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:03.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:03.691 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:03.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:03.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:03.793 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:03.829 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:03.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:03.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:03.930 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:03.963 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:03.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:03.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:03.994 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:04.065 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:04.066 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 42 16 76907 76891 114.397 78.4375 0.00790856 0.00873676 2024-09-21T10:19:04.101 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:04.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:04.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:04.203 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:04.236 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:04.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:04.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:04.338 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:04.374 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:04.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:04.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:04.476 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:04.510 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:04.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:04.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:04.612 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:04.618 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1275,"stamp":"2024-09-21T10:18:55.437942+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4019906930,"num_objects":61355,"num_object_clones":0,"num_object_copies":184065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61355,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":61406,"num_write_kb":3925236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":644571,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63488,"ondisk_log_size":63488,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":19422444,"kb_used_data":11638444,"kb_used_omap":43,"kb_used_meta":7783892,"kb_avail":730538772,"statfs":{"total":767960285184,"available":748071702528,"internally_reserved":0,"allocated":11917766656,"data_stored":11899451614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44223,"internal_metadata":7970706241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,6,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":989659136,"num_objects":15101,"num_object_clones":0,"num_object_copies":45303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15101,"num_write_kb":966464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.526004"},"pg_stats":[{"pgid":"1.11","version":"77'100","reported_seq":263,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.921778+0000","last_change":"2024-09-21T10:10:57.397786+0000","last_active":"2024-09-21T10:18:44.921778+0000","last_peered":"2024-09-21T10:18:44.921778+0000","last_clean":"2024-09-21T10:18:44.921778+0000","last_became_active":"2024-09-21T10:10:57.396742+0000","last_became_peered":"2024-09-21T10:10:57.396742+0000","last_unstale":"2024-09-21T10:18:44.921778+0000","last_undegraded":"2024-09-21T10:18:44.921778+0000","last_fullsized":"2024-09-21T10:18:44.921778+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:53.631923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:53.631923+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:33:35.364595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":602,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.916928+0000","last_change":"2024-09-21T10:03:38.389577+0000","last_active":"2024-09-21T10:18:44.916928+0000","last_peered":"2024-09-21T10:18:44.916928+0000","last_clean":"2024-09-21T10:18:44.916928+0000","last_became_active":"2024-09-21T10:03:38.389292+0000","last_became_peered":"2024-09-21T10:03:38.389292+0000","last_unstale":"2024-09-21T10:18:44.916928+0000","last_undegraded":"2024-09-21T10:18:44.916928+0000","last_fullsized":"2024-09-21T10:18:44.916928+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:32.113472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":258,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.750246+0000","last_change":"2024-09-21T10:14:30.717695+0000","last_active":"2024-09-21T10:18:26.750246+0000","last_peered":"2024-09-21T10:18:26.750246+0000","last_clean":"2024-09-21T10:18:26.750246+0000","last_became_active":"2024-09-21T10:14:30.705952+0000","last_became_peered":"2024-09-21T10:14:30.705952+0000","last_unstale":"2024-09-21T10:18:26.750246+0000","last_undegraded":"2024-09-21T10:18:26.750246+0000","last_fullsized":"2024-09-21T10:18:26.750246+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:08:18.863430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'100","reported_seq":271,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.748159+0000","last_change":"2024-09-21T10:11:05.438031+0000","last_active":"2024-09-21T10:18:26.748159+0000","last_peered":"2024-09-21T10:18:26.748159+0000","last_clean":"2024-09-21T10:18:26.748159+0000","last_became_active":"2024-09-21T10:11:05.437637+0000","last_became_peered":"2024-09-21T10:11:05.437637+0000","last_unstale":"2024-09-21T10:18:26.748159+0000","last_undegraded":"2024-09-21T10:18:26.748159+0000","last_fullsized":"2024-09-21T10:18:26.748159+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.798577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.798577+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:42.343217+0000","scrub_duration":0.0023622640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":363,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.749484+0000","last_change":"2024-09-21T10:10:57.392782+0000","last_active":"2024-09-21T10:18:26.749484+0000","last_peered":"2024-09-21T10:18:26.749484+0000","last_clean":"2024-09-21T10:18:26.749484+0000","last_became_active":"2024-09-21T10:10:57.392305+0000","last_became_peered":"2024-09-21T10:10:57.392305+0000","last_unstale":"2024-09-21T10:18:26.749484+0000","last_undegraded":"2024-09-21T10:18:26.749484+0000","last_fullsized":"2024-09-21T10:18:26.749484+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:42:01.387622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":523,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.921289+0000","last_change":"2024-09-21T10:09:23.846703+0000","last_active":"2024-09-21T10:18:44.921289+0000","last_peered":"2024-09-21T10:18:44.921289+0000","last_clean":"2024-09-21T10:18:44.921289+0000","last_became_active":"2024-09-21T10:09:23.846415+0000","last_became_peered":"2024-09-21T10:09:23.846415+0000","last_unstale":"2024-09-21T10:18:44.921289+0000","last_undegraded":"2024-09-21T10:18:44.921289+0000","last_fullsized":"2024-09-21T10:18:44.921289+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:23.129285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'100","reported_seq":265,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.919182+0000","last_change":"2024-09-21T10:18:12.637168+0000","last_active":"2024-09-21T10:18:44.919182+0000","last_peered":"2024-09-21T10:18:44.919182+0000","last_clean":"2024-09-21T10:18:44.919182+0000","last_became_active":"2024-09-21T10:18:12.636410+0000","last_became_peered":"2024-09-21T10:18:12.636410+0000","last_unstale":"2024-09-21T10:18:44.919182+0000","last_undegraded":"2024-09-21T10:18:44.919182+0000","last_fullsized":"2024-09-21T10:18:44.919182+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.433227+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:59:07.270049+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":248,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.921415+0000","last_change":"2024-09-21T10:09:49.922489+0000","last_active":"2024-09-21T10:18:44.921415+0000","last_peered":"2024-09-21T10:18:44.921415+0000","last_clean":"2024-09-21T10:18:44.921415+0000","last_became_active":"2024-09-21T10:09:49.921629+0000","last_became_peered":"2024-09-21T10:09:49.921629+0000","last_unstale":"2024-09-21T10:18:44.921415+0000","last_undegraded":"2024-09-21T10:18:44.921415+0000","last_fullsized":"2024-09-21T10:18:44.921415+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:50.611979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":254,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.749502+0000","last_change":"2024-09-21T10:14:30.709707+0000","last_active":"2024-09-21T10:18:26.749502+0000","last_peered":"2024-09-21T10:18:26.749502+0000","last_clean":"2024-09-21T10:18:26.749502+0000","last_became_active":"2024-09-21T10:14:30.709353+0000","last_became_peered":"2024-09-21T10:14:30.709353+0000","last_unstale":"2024-09-21T10:18:26.749502+0000","last_undegraded":"2024-09-21T10:18:26.749502+0000","last_fullsized":"2024-09-21T10:18:26.749502+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:12.822947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"77'100","reported_seq":153,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.920603+0000","last_change":"2024-09-21T10:15:02.842044+0000","last_active":"2024-09-21T10:18:44.920603+0000","last_peered":"2024-09-21T10:18:44.920603+0000","last_clean":"2024-09-21T10:18:44.920603+0000","last_became_active":"2024-09-21T10:15:01.478273+0000","last_became_peered":"2024-09-21T10:15:01.478273+0000","last_unstale":"2024-09-21T10:18:44.920603+0000","last_undegraded":"2024-09-21T10:18:44.920603+0000","last_fullsized":"2024-09-21T10:18:44.920603+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:15:02.841968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:15:02.841968+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:38:23.910899+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":602,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.918383+0000","last_change":"2024-09-21T10:03:38.381552+0000","last_active":"2024-09-21T10:18:44.918383+0000","last_peered":"2024-09-21T10:18:44.918383+0000","last_clean":"2024-09-21T10:18:44.918383+0000","last_became_active":"2024-09-21T10:03:38.380898+0000","last_became_peered":"2024-09-21T10:03:38.380898+0000","last_unstale":"2024-09-21T10:18:44.918383+0000","last_undegraded":"2024-09-21T10:18:44.918383+0000","last_fullsized":"2024-09-21T10:18:44.918383+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:53.409606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":431,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.752615+0000","last_change":"2024-09-21T10:10:57.459057+0000","last_active":"2024-09-21T10:18:26.752615+0000","last_peered":"2024-09-21T10:18:26.752615+0000","last_clean":"2024-09-21T10:18:26.752615+0000","last_became_active":"2024-09-21T10:10:57.451776+0000","last_became_peered":"2024-09-21T10:10:57.451776+0000","last_unstale":"2024-09-21T10:18:26.752615+0000","last_undegraded":"2024-09-21T10:18:26.752615+0000","last_fullsized":"2024-09-21T10:18:26.752615+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:00.138317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"373'3772","reported_seq":3786,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:51.832356+0000","last_change":"2024-09-21T10:18:12.636805+0000","last_active":"2024-09-21T10:18:51.832356+0000","last_peered":"2024-09-21T10:18:51.832356+0000","last_clean":"2024-09-21T10:18:51.832356+0000","last_became_active":"2024-09-21T10:18:12.636542+0000","last_became_peered":"2024-09-21T10:18:12.636542+0000","last_unstale":"2024-09-21T10:18:51.832356+0000","last_undegraded":"2024-09-21T10:18:51.832356+0000","last_fullsized":"2024-09-21T10:18:51.832356+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":3772,"log_dups_size":0,"ondisk_log_size":3772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:29.211883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247201792,"num_objects":3772,"num_object_clones":0,"num_object_copies":11316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3772,"num_write_kb":241408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"77'100","reported_seq":266,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.749524+0000","last_change":"2024-09-21T10:14:30.708275+0000","last_active":"2024-09-21T10:18:26.749524+0000","last_peered":"2024-09-21T10:18:26.749524+0000","last_clean":"2024-09-21T10:18:26.749524+0000","last_became_active":"2024-09-21T10:14:30.707763+0000","last_became_peered":"2024-09-21T10:14:30.707763+0000","last_unstale":"2024-09-21T10:18:26.749524+0000","last_undegraded":"2024-09-21T10:18:26.749524+0000","last_fullsized":"2024-09-21T10:18:26.749524+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:55.259385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:55.259385+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:41.322447+0000","scrub_duration":0.0015667260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":301,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.750291+0000","last_change":"2024-09-21T10:14:30.720250+0000","last_active":"2024-09-21T10:18:26.750291+0000","last_peered":"2024-09-21T10:18:26.750291+0000","last_clean":"2024-09-21T10:18:26.750291+0000","last_became_active":"2024-09-21T10:14:30.707036+0000","last_became_peered":"2024-09-21T10:14:30.707036+0000","last_unstale":"2024-09-21T10:18:26.750291+0000","last_undegraded":"2024-09-21T10:18:26.750291+0000","last_fullsized":"2024-09-21T10:18:26.750291+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:17.438534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":344,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.748023+0000","last_change":"2024-09-21T10:15:01.476521+0000","last_active":"2024-09-21T10:18:26.748023+0000","last_peered":"2024-09-21T10:18:26.748023+0000","last_clean":"2024-09-21T10:18:26.748023+0000","last_became_active":"2024-09-21T10:15:01.476083+0000","last_became_peered":"2024-09-21T10:15:01.476083+0000","last_unstale":"2024-09-21T10:18:26.748023+0000","last_undegraded":"2024-09-21T10:18:26.748023+0000","last_fullsized":"2024-09-21T10:18:26.748023+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:50.230443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.0","version":"373'3903","reported_seq":3917,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:51.943260+0000","last_change":"2024-09-21T10:18:12.633194+0000","last_active":"2024-09-21T10:18:51.943260+0000","last_peered":"2024-09-21T10:18:51.943260+0000","last_clean":"2024-09-21T10:18:51.943260+0000","last_became_active":"2024-09-21T10:18:12.632915+0000","last_became_peered":"2024-09-21T10:18:12.632915+0000","last_unstale":"2024-09-21T10:18:51.943260+0000","last_undegraded":"2024-09-21T10:18:51.943260+0000","last_fullsized":"2024-09-21T10:18:51.943260+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":3903,"log_dups_size":0,"ondisk_log_size":3903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:13:26.865830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255787008,"num_objects":3903,"num_object_clones":0,"num_object_copies":11709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3903,"num_write_kb":249792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"77'100","reported_seq":409,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.916787+0000","last_change":"2024-09-21T10:09:23.839211+0000","last_active":"2024-09-21T10:18:44.916787+0000","last_peered":"2024-09-21T10:18:44.916787+0000","last_clean":"2024-09-21T10:18:44.916787+0000","last_became_active":"2024-09-21T10:09:23.838779+0000","last_became_peered":"2024-09-21T10:09:23.838779+0000","last_unstale":"2024-09-21T10:18:44.916787+0000","last_undegraded":"2024-09-21T10:18:44.916787+0000","last_fullsized":"2024-09-21T10:18:44.916787+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.981336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.981336+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:03:15.566873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":280,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.749870+0000","last_change":"2024-09-21T10:18:12.643154+0000","last_active":"2024-09-21T10:18:26.749870+0000","last_peered":"2024-09-21T10:18:26.749870+0000","last_clean":"2024-09-21T10:18:26.749870+0000","last_became_active":"2024-09-21T10:18:12.636287+0000","last_became_peered":"2024-09-21T10:18:12.636287+0000","last_unstale":"2024-09-21T10:18:26.749870+0000","last_undegraded":"2024-09-21T10:18:26.749870+0000","last_fullsized":"2024-09-21T10:18:26.749870+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:45:31.008309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":79,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.753599+0000","last_change":"2024-09-21T10:14:30.706497+0000","last_active":"2024-09-21T10:18:26.753599+0000","last_peered":"2024-09-21T10:18:26.753599+0000","last_clean":"2024-09-21T10:18:26.753599+0000","last_became_active":"2024-09-21T10:14:30.706202+0000","last_became_peered":"2024-09-21T10:14:30.706202+0000","last_unstale":"2024-09-21T10:18:26.753599+0000","last_undegraded":"2024-09-21T10:18:26.753599+0000","last_fullsized":"2024-09-21T10:18:26.753599+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:42:57.912707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3","version":"373'3980","reported_seq":4003,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:52.539055+0000","last_change":"2024-09-21T10:18:12.641462+0000","last_active":"2024-09-21T10:18:52.539055+0000","last_peered":"2024-09-21T10:18:52.539055+0000","last_clean":"2024-09-21T10:18:52.539055+0000","last_became_active":"2024-09-21T10:18:12.641056+0000","last_became_peered":"2024-09-21T10:18:12.641056+0000","last_unstale":"2024-09-21T10:18:52.539055+0000","last_undegraded":"2024-09-21T10:18:52.539055+0000","last_fullsized":"2024-09-21T10:18:52.539055+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":3980,"log_dups_size":0,"ondisk_log_size":3980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:49:09.395317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260833280,"num_objects":3980,"num_object_clones":0,"num_object_copies":11940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3980,"num_write_kb":254720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"77'100","reported_seq":366,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.753297+0000","last_change":"2024-09-21T10:14:30.709997+0000","last_active":"2024-09-21T10:18:26.753297+0000","last_peered":"2024-09-21T10:18:26.753297+0000","last_clean":"2024-09-21T10:18:26.753297+0000","last_became_active":"2024-09-21T10:14:30.709580+0000","last_became_peered":"2024-09-21T10:14:30.709580+0000","last_unstale":"2024-09-21T10:18:26.753297+0000","last_undegraded":"2024-09-21T10:18:26.753297+0000","last_fullsized":"2024-09-21T10:18:26.753297+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.289859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.289859+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:27:50.110602+0000","scrub_duration":0.001251809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":252,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.749318+0000","last_change":"2024-09-21T10:10:57.460732+0000","last_active":"2024-09-21T10:18:26.749318+0000","last_peered":"2024-09-21T10:18:26.749318+0000","last_clean":"2024-09-21T10:18:26.749318+0000","last_became_active":"2024-09-21T10:10:57.454689+0000","last_became_peered":"2024-09-21T10:10:57.454689+0000","last_unstale":"2024-09-21T10:18:26.749318+0000","last_undegraded":"2024-09-21T10:18:26.749318+0000","last_fullsized":"2024-09-21T10:18:26.749318+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:12.035022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":372,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.752452+0000","last_change":"2024-09-21T10:03:38.378384+0000","last_active":"2024-09-21T10:18:26.752452+0000","last_peered":"2024-09-21T10:18:26.752452+0000","last_clean":"2024-09-21T10:18:26.752452+0000","last_became_active":"2024-09-21T10:03:38.378072+0000","last_became_peered":"2024-09-21T10:03:38.378072+0000","last_unstale":"2024-09-21T10:18:26.752452+0000","last_undegraded":"2024-09-21T10:18:26.752452+0000","last_fullsized":"2024-09-21T10:18:26.752452+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:54:37.039622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"373'3793","reported_seq":3816,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:52.401896+0000","last_change":"2024-09-21T10:18:13.037514+0000","last_active":"2024-09-21T10:18:52.401896+0000","last_peered":"2024-09-21T10:18:52.401896+0000","last_clean":"2024-09-21T10:18:52.401896+0000","last_became_active":"2024-09-21T10:18:13.037222+0000","last_became_peered":"2024-09-21T10:18:13.037222+0000","last_unstale":"2024-09-21T10:18:52.401896+0000","last_undegraded":"2024-09-21T10:18:52.401896+0000","last_fullsized":"2024-09-21T10:18:52.401896+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":3793,"log_dups_size":0,"ondisk_log_size":3793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:00:30.961667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248578048,"num_objects":3793,"num_object_clones":0,"num_object_copies":11379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3793,"num_write_kb":242752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"77'100","reported_seq":25,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:27.085743+0000","last_change":"2024-09-21T10:18:27.085659+0000","last_active":"2024-09-21T10:18:27.085743+0000","last_peered":"2024-09-21T10:18:27.085743+0000","last_clean":"2024-09-21T10:18:27.085743+0000","last_became_active":"2024-09-21T10:18:12.633342+0000","last_became_peered":"2024-09-21T10:18:12.633342+0000","last_unstale":"2024-09-21T10:18:27.085743+0000","last_undegraded":"2024-09-21T10:18:27.085743+0000","last_fullsized":"2024-09-21T10:18:27.085743+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:18:27.085525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:18:27.085525+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:06:53.238321+0000","scrub_duration":0.0036485279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":406,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.917090+0000","last_change":"2024-09-21T10:07:57.551556+0000","last_active":"2024-09-21T10:18:44.917090+0000","last_peered":"2024-09-21T10:18:44.917090+0000","last_clean":"2024-09-21T10:18:44.917090+0000","last_became_active":"2024-09-21T10:07:57.551249+0000","last_became_peered":"2024-09-21T10:07:57.551249+0000","last_unstale":"2024-09-21T10:18:44.917090+0000","last_undegraded":"2024-09-21T10:18:44.917090+0000","last_fullsized":"2024-09-21T10:18:44.917090+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:57.557152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":510,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.922251+0000","last_change":"2024-09-21T10:14:30.711105+0000","last_active":"2024-09-21T10:18:44.922251+0000","last_peered":"2024-09-21T10:18:44.922251+0000","last_clean":"2024-09-21T10:18:44.922251+0000","last_became_active":"2024-09-21T10:14:30.710759+0000","last_became_peered":"2024-09-21T10:14:30.710759+0000","last_unstale":"2024-09-21T10:18:44.922251+0000","last_undegraded":"2024-09-21T10:18:44.922251+0000","last_fullsized":"2024-09-21T10:18:44.922251+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:40.690042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"373'3761","reported_seq":3775,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:51.848827+0000","last_change":"2024-09-21T10:18:13.037771+0000","last_active":"2024-09-21T10:18:51.848827+0000","last_peered":"2024-09-21T10:18:51.848827+0000","last_clean":"2024-09-21T10:18:51.848827+0000","last_became_active":"2024-09-21T10:18:13.037425+0000","last_became_peered":"2024-09-21T10:18:13.037425+0000","last_unstale":"2024-09-21T10:18:51.848827+0000","last_undegraded":"2024-09-21T10:18:51.848827+0000","last_fullsized":"2024-09-21T10:18:51.848827+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":3761,"log_dups_size":0,"ondisk_log_size":3761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:34:43.436187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246480896,"num_objects":3761,"num_object_clones":0,"num_object_copies":11283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3761,"num_write_kb":240704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":246,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.749589+0000","last_change":"2024-09-21T10:04:53.355975+0000","last_active":"2024-09-21T10:18:26.749589+0000","last_peered":"2024-09-21T10:18:26.749589+0000","last_clean":"2024-09-21T10:18:26.749589+0000","last_became_active":"2024-09-21T10:04:42.547505+0000","last_became_peered":"2024-09-21T10:04:42.547505+0000","last_unstale":"2024-09-21T10:18:26.749589+0000","last_undegraded":"2024-09-21T10:18:26.749589+0000","last_fullsized":"2024-09-21T10:18:26.749589+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.355912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.355912+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:43.456830+0000","scrub_duration":0.0017980030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":614,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.917132+0000","last_change":"2024-09-21T10:14:30.715303+0000","last_active":"2024-09-21T10:18:44.917132+0000","last_peered":"2024-09-21T10:18:44.917132+0000","last_clean":"2024-09-21T10:18:44.917132+0000","last_became_active":"2024-09-21T10:14:30.703664+0000","last_became_peered":"2024-09-21T10:14:30.703664+0000","last_unstale":"2024-09-21T10:18:44.917132+0000","last_undegraded":"2024-09-21T10:18:44.917132+0000","last_fullsized":"2024-09-21T10:18:44.917132+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:28.967906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":79,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.753287+0000","last_change":"2024-09-21T10:14:30.705927+0000","last_active":"2024-09-21T10:18:26.753287+0000","last_peered":"2024-09-21T10:18:26.753287+0000","last_clean":"2024-09-21T10:18:26.753287+0000","last_became_active":"2024-09-21T10:14:30.705112+0000","last_became_peered":"2024-09-21T10:14:30.705112+0000","last_unstale":"2024-09-21T10:18:26.753287+0000","last_undegraded":"2024-09-21T10:18:26.753287+0000","last_fullsized":"2024-09-21T10:18:26.753287+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:38:33.962199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.4","version":"373'3808","reported_seq":3822,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:51.858545+0000","last_change":"2024-09-21T10:18:12.639080+0000","last_active":"2024-09-21T10:18:51.858545+0000","last_peered":"2024-09-21T10:18:51.858545+0000","last_clean":"2024-09-21T10:18:51.858545+0000","last_became_active":"2024-09-21T10:18:12.638789+0000","last_became_peered":"2024-09-21T10:18:12.638789+0000","last_unstale":"2024-09-21T10:18:51.858545+0000","last_undegraded":"2024-09-21T10:18:51.858545+0000","last_fullsized":"2024-09-21T10:18:51.858545+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":3808,"log_dups_size":0,"ondisk_log_size":3808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:53.178338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249561088,"num_objects":3808,"num_object_clones":0,"num_object_copies":11424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3808,"num_write_kb":243712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":363,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.914935+0000","last_change":"2024-09-21T10:10:57.403722+0000","last_active":"2024-09-21T10:18:44.914935+0000","last_peered":"2024-09-21T10:18:44.914935+0000","last_clean":"2024-09-21T10:18:44.914935+0000","last_became_active":"2024-09-21T10:10:57.394700+0000","last_became_peered":"2024-09-21T10:10:57.394700+0000","last_unstale":"2024-09-21T10:18:44.914935+0000","last_undegraded":"2024-09-21T10:18:44.914935+0000","last_fullsized":"2024-09-21T10:18:44.914935+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:02.152981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:02.152981+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:52:34.887763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":356,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.752588+0000","last_change":"2024-09-21T10:14:30.707046+0000","last_active":"2024-09-21T10:18:26.752588+0000","last_peered":"2024-09-21T10:18:26.752588+0000","last_clean":"2024-09-21T10:18:26.752588+0000","last_became_active":"2024-09-21T10:14:30.706753+0000","last_became_peered":"2024-09-21T10:14:30.706753+0000","last_unstale":"2024-09-21T10:18:26.752588+0000","last_undegraded":"2024-09-21T10:18:26.752588+0000","last_fullsized":"2024-09-21T10:18:26.752588+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:58.136096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":15,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.753153+0000","last_change":"2024-09-21T10:18:13.037945+0000","last_active":"2024-09-21T10:18:26.753153+0000","last_peered":"2024-09-21T10:18:26.753153+0000","last_clean":"2024-09-21T10:18:26.753153+0000","last_became_active":"2024-09-21T10:18:13.037550+0000","last_became_peered":"2024-09-21T10:18:13.037550+0000","last_unstale":"2024-09-21T10:18:26.753153+0000","last_undegraded":"2024-09-21T10:18:26.753153+0000","last_fullsized":"2024-09-21T10:18:26.753153+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:24:24.702636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.7","version":"373'3826","reported_seq":3840,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:51.987257+0000","last_change":"2024-09-21T10:18:12.635921+0000","last_active":"2024-09-21T10:18:51.987257+0000","last_peered":"2024-09-21T10:18:51.987257+0000","last_clean":"2024-09-21T10:18:51.987257+0000","last_became_active":"2024-09-21T10:18:12.635624+0000","last_became_peered":"2024-09-21T10:18:12.635624+0000","last_unstale":"2024-09-21T10:18:51.987257+0000","last_undegraded":"2024-09-21T10:18:51.987257+0000","last_fullsized":"2024-09-21T10:18:51.987257+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":3826,"log_dups_size":0,"ondisk_log_size":3826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:08:14.392867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250740736,"num_objects":3826,"num_object_clones":0,"num_object_copies":11478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3826,"num_write_kb":244864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"77'100","reported_seq":255,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.919057+0000","last_change":"2024-09-21T10:14:05.474100+0000","last_active":"2024-09-21T10:18:44.919057+0000","last_peered":"2024-09-21T10:18:44.919057+0000","last_clean":"2024-09-21T10:18:44.919057+0000","last_became_active":"2024-09-21T10:14:05.473336+0000","last_became_peered":"2024-09-21T10:14:05.473336+0000","last_unstale":"2024-09-21T10:18:44.919057+0000","last_undegraded":"2024-09-21T10:18:44.919057+0000","last_fullsized":"2024-09-21T10:18:44.919057+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.540323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.540323+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:13:30.035766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.752621+0000","last_change":"2024-09-21T10:18:12.629311+0000","last_active":"2024-09-21T10:18:26.752621+0000","last_peered":"2024-09-21T10:18:26.752621+0000","last_clean":"2024-09-21T10:18:26.752621+0000","last_became_active":"2024-09-21T10:18:12.629001+0000","last_became_peered":"2024-09-21T10:18:12.629001+0000","last_unstale":"2024-09-21T10:18:26.752621+0000","last_undegraded":"2024-09-21T10:18:26.752621+0000","last_fullsized":"2024-09-21T10:18:26.752621+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:09:17.165751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":79,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.753891+0000","last_change":"2024-09-21T10:14:30.706905+0000","last_active":"2024-09-21T10:18:26.753891+0000","last_peered":"2024-09-21T10:18:26.753891+0000","last_clean":"2024-09-21T10:18:26.753891+0000","last_became_active":"2024-09-21T10:14:30.706614+0000","last_became_peered":"2024-09-21T10:14:30.706614+0000","last_unstale":"2024-09-21T10:18:26.753891+0000","last_undegraded":"2024-09-21T10:18:26.753891+0000","last_fullsized":"2024-09-21T10:18:26.753891+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:26:33.014993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.6","version":"373'3754","reported_seq":3767,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:51.858587+0000","last_change":"2024-09-21T10:18:13.038010+0000","last_active":"2024-09-21T10:18:51.858587+0000","last_peered":"2024-09-21T10:18:51.858587+0000","last_clean":"2024-09-21T10:18:51.858587+0000","last_became_active":"2024-09-21T10:18:13.037700+0000","last_became_peered":"2024-09-21T10:18:13.037700+0000","last_unstale":"2024-09-21T10:18:51.858587+0000","last_undegraded":"2024-09-21T10:18:51.858587+0000","last_fullsized":"2024-09-21T10:18:51.858587+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":3754,"log_dups_size":0,"ondisk_log_size":3754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:14:58.367606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246022144,"num_objects":3754,"num_object_clones":0,"num_object_copies":11262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3754,"num_write_kb":240256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":326,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.914914+0000","last_change":"2024-09-21T10:08:04.645700+0000","last_active":"2024-09-21T10:18:44.914914+0000","last_peered":"2024-09-21T10:18:44.914914+0000","last_clean":"2024-09-21T10:18:44.914914+0000","last_became_active":"2024-09-21T10:07:57.548189+0000","last_became_peered":"2024-09-21T10:07:57.548189+0000","last_unstale":"2024-09-21T10:18:44.914914+0000","last_undegraded":"2024-09-21T10:18:44.914914+0000","last_fullsized":"2024-09-21T10:18:44.914914+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:48.318010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:48.318010+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T16:31:28.173059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":604,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.916699+0000","last_change":"2024-09-21T10:03:38.386604+0000","last_active":"2024-09-21T10:18:44.916699+0000","last_peered":"2024-09-21T10:18:44.916699+0000","last_clean":"2024-09-21T10:18:44.916699+0000","last_became_active":"2024-09-21T10:03:38.386283+0000","last_became_peered":"2024-09-21T10:03:38.386283+0000","last_unstale":"2024-09-21T10:18:44.916699+0000","last_undegraded":"2024-09-21T10:18:44.916699+0000","last_fullsized":"2024-09-21T10:18:44.916699+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:27:05.944880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":463,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.920099+0000","last_change":"2024-09-21T10:14:30.709615+0000","last_active":"2024-09-21T10:18:44.920099+0000","last_peered":"2024-09-21T10:18:44.920099+0000","last_clean":"2024-09-21T10:18:44.920099+0000","last_became_active":"2024-09-21T10:14:30.709184+0000","last_became_peered":"2024-09-21T10:14:30.709184+0000","last_unstale":"2024-09-21T10:18:44.920099+0000","last_undegraded":"2024-09-21T10:18:44.920099+0000","last_fullsized":"2024-09-21T10:18:44.920099+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:55:27.220232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.c","version":"373'3653","reported_seq":3676,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:50.662588+0000","last_change":"2024-09-21T10:18:13.037466+0000","last_active":"2024-09-21T10:18:50.662588+0000","last_peered":"2024-09-21T10:18:50.662588+0000","last_clean":"2024-09-21T10:18:50.662588+0000","last_became_active":"2024-09-21T10:18:13.037151+0000","last_became_peered":"2024-09-21T10:18:13.037151+0000","last_unstale":"2024-09-21T10:18:50.662588+0000","last_undegraded":"2024-09-21T10:18:50.662588+0000","last_fullsized":"2024-09-21T10:18:50.662588+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":3653,"log_dups_size":0,"ondisk_log_size":3653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:35:23.611663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239403008,"num_objects":3653,"num_object_clones":0,"num_object_copies":10959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3653,"num_write_kb":233792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":263,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.748078+0000","last_change":"2024-09-21T10:09:49.924286+0000","last_active":"2024-09-21T10:18:26.748078+0000","last_peered":"2024-09-21T10:18:26.748078+0000","last_clean":"2024-09-21T10:18:26.748078+0000","last_became_active":"2024-09-21T10:09:49.923897+0000","last_became_peered":"2024-09-21T10:09:49.923897+0000","last_unstale":"2024-09-21T10:18:26.748078+0000","last_undegraded":"2024-09-21T10:18:26.748078+0000","last_fullsized":"2024-09-21T10:18:26.748078+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:47.393146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:47.393146+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:44.898083+0000","scrub_duration":0.001601086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":359,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.913964+0000","last_change":"2024-09-21T10:10:57.402946+0000","last_active":"2024-09-21T10:18:44.913964+0000","last_peered":"2024-09-21T10:18:44.913964+0000","last_clean":"2024-09-21T10:18:44.913964+0000","last_became_active":"2024-09-21T10:10:57.395450+0000","last_became_peered":"2024-09-21T10:10:57.395450+0000","last_unstale":"2024-09-21T10:18:44.913964+0000","last_undegraded":"2024-09-21T10:18:44.913964+0000","last_fullsized":"2024-09-21T10:18:44.913964+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:08.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":588,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.916784+0000","last_change":"2024-09-21T10:10:57.390595+0000","last_active":"2024-09-21T10:18:44.916784+0000","last_peered":"2024-09-21T10:18:44.916784+0000","last_clean":"2024-09-21T10:18:44.916784+0000","last_became_active":"2024-09-21T10:10:57.390265+0000","last_became_peered":"2024-09-21T10:10:57.390265+0000","last_unstale":"2024-09-21T10:18:44.916784+0000","last_undegraded":"2024-09-21T10:18:44.916784+0000","last_fullsized":"2024-09-21T10:18:44.916784+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:37.485063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":389,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.752939+0000","last_change":"2024-09-21T10:14:30.711719+0000","last_active":"2024-09-21T10:18:26.752939+0000","last_peered":"2024-09-21T10:18:26.752939+0000","last_clean":"2024-09-21T10:18:26.752939+0000","last_became_active":"2024-09-21T10:14:30.711452+0000","last_became_peered":"2024-09-21T10:14:30.711452+0000","last_unstale":"2024-09-21T10:18:26.752939+0000","last_undegraded":"2024-09-21T10:18:26.752939+0000","last_fullsized":"2024-09-21T10:18:26.752939+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:46.907528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"373'3794","reported_seq":3808,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:51.860897+0000","last_change":"2024-09-21T10:18:12.639005+0000","last_active":"2024-09-21T10:18:51.860897+0000","last_peered":"2024-09-21T10:18:51.860897+0000","last_clean":"2024-09-21T10:18:51.860897+0000","last_became_active":"2024-09-21T10:18:12.638716+0000","last_became_peered":"2024-09-21T10:18:12.638716+0000","last_unstale":"2024-09-21T10:18:51.860897+0000","last_undegraded":"2024-09-21T10:18:51.860897+0000","last_fullsized":"2024-09-21T10:18:51.860897+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":3794,"log_dups_size":0,"ondisk_log_size":3794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:08.923853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248643584,"num_objects":3794,"num_object_clones":0,"num_object_copies":11382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3794,"num_write_kb":242816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":265,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.919268+0000","last_change":"2024-09-21T10:14:30.701644+0000","last_active":"2024-09-21T10:18:44.919268+0000","last_peered":"2024-09-21T10:18:44.919268+0000","last_clean":"2024-09-21T10:18:44.919268+0000","last_became_active":"2024-09-21T10:14:30.701301+0000","last_became_peered":"2024-09-21T10:14:30.701301+0000","last_unstale":"2024-09-21T10:18:44.919268+0000","last_undegraded":"2024-09-21T10:18:44.919268+0000","last_fullsized":"2024-09-21T10:18:44.919268+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.550551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.550551+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:23.637695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":237,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.748086+0000","last_change":"2024-09-21T10:09:23.838944+0000","last_active":"2024-09-21T10:18:26.748086+0000","last_peered":"2024-09-21T10:18:26.748086+0000","last_clean":"2024-09-21T10:18:26.748086+0000","last_became_active":"2024-09-21T10:09:23.837886+0000","last_became_peered":"2024-09-21T10:09:23.837886+0000","last_unstale":"2024-09-21T10:18:26.748086+0000","last_undegraded":"2024-09-21T10:18:26.748086+0000","last_fullsized":"2024-09-21T10:18:26.748086+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:16.928195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":487,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.921590+0000","last_change":"2024-09-21T10:18:12.637103+0000","last_active":"2024-09-21T10:18:44.921590+0000","last_peered":"2024-09-21T10:18:44.921590+0000","last_clean":"2024-09-21T10:18:44.921590+0000","last_became_active":"2024-09-21T10:18:12.636645+0000","last_became_peered":"2024-09-21T10:18:12.636645+0000","last_unstale":"2024-09-21T10:18:44.921590+0000","last_undegraded":"2024-09-21T10:18:44.921590+0000","last_fullsized":"2024-09-21T10:18:44.921590+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:53:50.745484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":249,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.921298+0000","last_change":"2024-09-21T10:09:49.923044+0000","last_active":"2024-09-21T10:18:44.921298+0000","last_peered":"2024-09-21T10:18:44.921298+0000","last_clean":"2024-09-21T10:18:44.921298+0000","last_became_active":"2024-09-21T10:09:49.922711+0000","last_became_peered":"2024-09-21T10:09:49.922711+0000","last_unstale":"2024-09-21T10:18:44.921298+0000","last_undegraded":"2024-09-21T10:18:44.921298+0000","last_fullsized":"2024-09-21T10:18:44.921298+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:14.090084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"373'3744","reported_seq":3757,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:51.975531+0000","last_change":"2024-09-21T10:18:13.038029+0000","last_active":"2024-09-21T10:18:51.975531+0000","last_peered":"2024-09-21T10:18:51.975531+0000","last_clean":"2024-09-21T10:18:51.975531+0000","last_became_active":"2024-09-21T10:18:13.037584+0000","last_became_peered":"2024-09-21T10:18:13.037584+0000","last_unstale":"2024-09-21T10:18:51.975531+0000","last_undegraded":"2024-09-21T10:18:51.975531+0000","last_fullsized":"2024-09-21T10:18:51.975531+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":3744,"log_dups_size":0,"ondisk_log_size":3744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:18.430382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245366784,"num_objects":3744,"num_object_clones":0,"num_object_copies":11232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3744,"num_write_kb":239616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":287,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.748051+0000","last_change":"2024-09-21T10:18:12.636851+0000","last_active":"2024-09-21T10:18:26.748051+0000","last_peered":"2024-09-21T10:18:26.748051+0000","last_clean":"2024-09-21T10:18:26.748051+0000","last_became_active":"2024-09-21T10:18:12.636513+0000","last_became_peered":"2024-09-21T10:18:12.636513+0000","last_unstale":"2024-09-21T10:18:26.748051+0000","last_undegraded":"2024-09-21T10:18:26.748051+0000","last_fullsized":"2024-09-21T10:18:26.748051+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:52.361137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:52.361137+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:21.025735+0000","scrub_duration":0.001255358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":642,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.914561+0000","last_change":"2024-09-21T10:09:23.837859+0000","last_active":"2024-09-21T10:18:44.914561+0000","last_peered":"2024-09-21T10:18:44.914561+0000","last_clean":"2024-09-21T10:18:44.914561+0000","last_became_active":"2024-09-21T10:09:23.837488+0000","last_became_peered":"2024-09-21T10:09:23.837488+0000","last_unstale":"2024-09-21T10:18:44.914561+0000","last_undegraded":"2024-09-21T10:18:44.914561+0000","last_fullsized":"2024-09-21T10:18:44.914561+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:25.215501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":121,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.747950+0000","last_change":"2024-09-21T10:10:27.806070+0000","last_active":"2024-09-21T10:18:26.747950+0000","last_peered":"2024-09-21T10:18:26.747950+0000","last_clean":"2024-09-21T10:18:26.747950+0000","last_became_active":"2024-09-21T10:10:27.805718+0000","last_became_peered":"2024-09-21T10:10:27.805718+0000","last_unstale":"2024-09-21T10:18:26.747950+0000","last_undegraded":"2024-09-21T10:18:26.747950+0000","last_fullsized":"2024-09-21T10:18:26.747950+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:31.144118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":635,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.914468+0000","last_change":"2024-09-21T10:10:57.395073+0000","last_active":"2024-09-21T10:18:44.914468+0000","last_peered":"2024-09-21T10:18:44.914468+0000","last_clean":"2024-09-21T10:18:44.914468+0000","last_became_active":"2024-09-21T10:10:57.394674+0000","last_became_peered":"2024-09-21T10:10:57.394674+0000","last_unstale":"2024-09-21T10:18:44.914468+0000","last_undegraded":"2024-09-21T10:18:44.914468+0000","last_fullsized":"2024-09-21T10:18:44.914468+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:48:48.912649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.d","version":"373'3770","reported_seq":3784,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:51.856657+0000","last_change":"2024-09-21T10:18:12.632980+0000","last_active":"2024-09-21T10:18:51.856657+0000","last_peered":"2024-09-21T10:18:51.856657+0000","last_clean":"2024-09-21T10:18:51.856657+0000","last_became_active":"2024-09-21T10:18:12.632684+0000","last_became_peered":"2024-09-21T10:18:12.632684+0000","last_unstale":"2024-09-21T10:18:51.856657+0000","last_undegraded":"2024-09-21T10:18:51.856657+0000","last_fullsized":"2024-09-21T10:18:51.856657+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":3770,"log_dups_size":0,"ondisk_log_size":3770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:08.974530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247070720,"num_objects":3770,"num_object_clones":0,"num_object_copies":11310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3770,"num_write_kb":241280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":504,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.921685+0000","last_change":"2024-09-21T10:10:57.398437+0000","last_active":"2024-09-21T10:18:44.921685+0000","last_peered":"2024-09-21T10:18:44.921685+0000","last_clean":"2024-09-21T10:18:44.921685+0000","last_became_active":"2024-09-21T10:10:57.397441+0000","last_became_peered":"2024-09-21T10:10:57.397441+0000","last_unstale":"2024-09-21T10:18:44.921685+0000","last_undegraded":"2024-09-21T10:18:44.921685+0000","last_fullsized":"2024-09-21T10:18:44.921685+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:04.043917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:04.043917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:53:31.813612+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":197,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.920556+0000","last_change":"2024-09-21T10:14:05.472790+0000","last_active":"2024-09-21T10:18:44.920556+0000","last_peered":"2024-09-21T10:18:44.920556+0000","last_clean":"2024-09-21T10:18:44.920556+0000","last_became_active":"2024-09-21T10:14:05.472238+0000","last_became_peered":"2024-09-21T10:14:05.472238+0000","last_unstale":"2024-09-21T10:18:44.920556+0000","last_undegraded":"2024-09-21T10:18:44.920556+0000","last_fullsized":"2024-09-21T10:18:44.920556+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:56.561032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":79,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.753451+0000","last_change":"2024-09-21T10:14:30.709652+0000","last_active":"2024-09-21T10:18:26.753451+0000","last_peered":"2024-09-21T10:18:26.753451+0000","last_clean":"2024-09-21T10:18:26.753451+0000","last_became_active":"2024-09-21T10:14:30.705706+0000","last_became_peered":"2024-09-21T10:14:30.705706+0000","last_unstale":"2024-09-21T10:18:26.753451+0000","last_undegraded":"2024-09-21T10:18:26.753451+0000","last_fullsized":"2024-09-21T10:18:26.753451+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:24:25.749923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":468,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.920712+0000","last_change":"2024-09-21T10:14:05.477875+0000","last_active":"2024-09-21T10:18:44.920712+0000","last_peered":"2024-09-21T10:18:44.920712+0000","last_clean":"2024-09-21T10:18:44.920712+0000","last_became_active":"2024-09-21T10:14:05.477192+0000","last_became_peered":"2024-09-21T10:14:05.477192+0000","last_unstale":"2024-09-21T10:18:44.920712+0000","last_undegraded":"2024-09-21T10:18:44.920712+0000","last_fullsized":"2024-09-21T10:18:44.920712+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:19:47.345673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.b","version":"373'3809","reported_seq":3832,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:52.412970+0000","last_change":"2024-09-21T10:18:13.039421+0000","last_active":"2024-09-21T10:18:52.412970+0000","last_peered":"2024-09-21T10:18:52.412970+0000","last_clean":"2024-09-21T10:18:52.412970+0000","last_became_active":"2024-09-21T10:18:13.039085+0000","last_became_peered":"2024-09-21T10:18:13.039085+0000","last_unstale":"2024-09-21T10:18:52.412970+0000","last_undegraded":"2024-09-21T10:18:52.412970+0000","last_fullsized":"2024-09-21T10:18:52.412970+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":3809,"log_dups_size":0,"ondisk_log_size":3809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:36.052065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249626624,"num_objects":3809,"num_object_clones":0,"num_object_copies":11427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3809,"num_write_kb":243776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":346,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.753188+0000","last_change":"2024-09-21T10:04:45.242148+0000","last_active":"2024-09-21T10:18:26.753188+0000","last_peered":"2024-09-21T10:18:26.753188+0000","last_clean":"2024-09-21T10:18:26.753188+0000","last_became_active":"2024-09-21T10:04:42.549844+0000","last_became_peered":"2024-09-21T10:04:42.549844+0000","last_unstale":"2024-09-21T10:18:26.753188+0000","last_undegraded":"2024-09-21T10:18:26.753188+0000","last_fullsized":"2024-09-21T10:18:26.753188+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_deep_scrub":"77'100","last_deep_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.242073+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:03:42.250480+0000","scrub_duration":0.0023466239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":310,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.918198+0000","last_change":"2024-09-21T10:15:01.489326+0000","last_active":"2024-09-21T10:18:44.918198+0000","last_peered":"2024-09-21T10:18:44.918198+0000","last_clean":"2024-09-21T10:18:44.918198+0000","last_became_active":"2024-09-21T10:15:01.489070+0000","last_became_peered":"2024-09-21T10:15:01.489070+0000","last_unstale":"2024-09-21T10:18:44.918198+0000","last_undegraded":"2024-09-21T10:18:44.918198+0000","last_fullsized":"2024-09-21T10:18:44.918198+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:27:41.788029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":524,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.920981+0000","last_change":"2024-09-21T10:14:05.475970+0000","last_active":"2024-09-21T10:18:44.920981+0000","last_peered":"2024-09-21T10:18:44.920981+0000","last_clean":"2024-09-21T10:18:44.920981+0000","last_became_active":"2024-09-21T10:14:05.475331+0000","last_became_peered":"2024-09-21T10:14:05.475331+0000","last_unstale":"2024-09-21T10:18:44.920981+0000","last_undegraded":"2024-09-21T10:18:44.920981+0000","last_fullsized":"2024-09-21T10:18:44.920981+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:17.069736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":460,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.920804+0000","last_change":"2024-09-21T10:18:13.039172+0000","last_active":"2024-09-21T10:18:44.920804+0000","last_peered":"2024-09-21T10:18:44.920804+0000","last_clean":"2024-09-21T10:18:44.920804+0000","last_became_active":"2024-09-21T10:18:13.038756+0000","last_became_peered":"2024-09-21T10:18:13.038756+0000","last_unstale":"2024-09-21T10:18:44.920804+0000","last_undegraded":"2024-09-21T10:18:44.920804+0000","last_fullsized":"2024-09-21T10:18:44.920804+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:06.668926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.a","version":"373'4044","reported_seq":4067,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:54.771041+0000","last_change":"2024-09-21T10:18:12.633418+0000","last_active":"2024-09-21T10:18:54.771041+0000","last_peered":"2024-09-21T10:18:54.771041+0000","last_clean":"2024-09-21T10:18:54.771041+0000","last_became_active":"2024-09-21T10:18:12.633151+0000","last_became_peered":"2024-09-21T10:18:12.633151+0000","last_unstale":"2024-09-21T10:18:54.771041+0000","last_undegraded":"2024-09-21T10:18:54.771041+0000","last_fullsized":"2024-09-21T10:18:54.771041+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":4044,"log_dups_size":0,"ondisk_log_size":4044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:07:26.685329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265027584,"num_objects":4044,"num_object_clones":0,"num_object_copies":12132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4044,"num_write_kb":258816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":363,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.753025+0000","last_change":"2024-09-21T10:09:23.839563+0000","last_active":"2024-09-21T10:18:26.753025+0000","last_peered":"2024-09-21T10:18:26.753025+0000","last_clean":"2024-09-21T10:18:26.753025+0000","last_became_active":"2024-09-21T10:09:23.839207+0000","last_became_peered":"2024-09-21T10:09:23.839207+0000","last_unstale":"2024-09-21T10:18:26.753025+0000","last_undegraded":"2024-09-21T10:18:26.753025+0000","last_fullsized":"2024-09-21T10:18:26.753025+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:49.226496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:49.226496+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:03.960125+0000","scrub_duration":0.001656865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":456,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.920891+0000","last_change":"2024-09-21T10:14:05.478152+0000","last_active":"2024-09-21T10:18:44.920891+0000","last_peered":"2024-09-21T10:18:44.920891+0000","last_clean":"2024-09-21T10:18:44.920891+0000","last_became_active":"2024-09-21T10:14:05.477547+0000","last_became_peered":"2024-09-21T10:14:05.477547+0000","last_unstale":"2024-09-21T10:18:44.920891+0000","last_undegraded":"2024-09-21T10:18:44.920891+0000","last_fullsized":"2024-09-21T10:18:44.920891+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:34.646046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":267,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.749400+0000","last_change":"2024-09-21T10:14:30.708090+0000","last_active":"2024-09-21T10:18:26.749400+0000","last_peered":"2024-09-21T10:18:26.749400+0000","last_clean":"2024-09-21T10:18:26.749400+0000","last_became_active":"2024-09-21T10:14:30.707266+0000","last_became_peered":"2024-09-21T10:14:30.707266+0000","last_unstale":"2024-09-21T10:18:26.749400+0000","last_undegraded":"2024-09-21T10:18:26.749400+0000","last_fullsized":"2024-09-21T10:18:26.749400+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:38.474537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":518,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.922448+0000","last_change":"2024-09-21T10:10:57.397251+0000","last_active":"2024-09-21T10:18:44.922448+0000","last_peered":"2024-09-21T10:18:44.922448+0000","last_clean":"2024-09-21T10:18:44.922448+0000","last_became_active":"2024-09-21T10:10:57.396271+0000","last_became_peered":"2024-09-21T10:10:57.396271+0000","last_unstale":"2024-09-21T10:18:44.922448+0000","last_undegraded":"2024-09-21T10:18:44.922448+0000","last_fullsized":"2024-09-21T10:18:44.922448+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:17.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.9","version":"373'3838","reported_seq":3861,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:52.551602+0000","last_change":"2024-09-21T10:18:13.037459+0000","last_active":"2024-09-21T10:18:52.551602+0000","last_peered":"2024-09-21T10:18:52.551602+0000","last_clean":"2024-09-21T10:18:52.551602+0000","last_became_active":"2024-09-21T10:18:13.037118+0000","last_became_peered":"2024-09-21T10:18:13.037118+0000","last_unstale":"2024-09-21T10:18:52.551602+0000","last_undegraded":"2024-09-21T10:18:52.551602+0000","last_fullsized":"2024-09-21T10:18:52.551602+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":3838,"log_dups_size":0,"ondisk_log_size":3838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:52.139485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251527168,"num_objects":3838,"num_object_clones":0,"num_object_copies":11514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3838,"num_write_kb":245632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":79,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.753439+0000","last_change":"2024-09-21T10:14:30.709001+0000","last_active":"2024-09-21T10:18:26.753439+0000","last_peered":"2024-09-21T10:18:26.753439+0000","last_clean":"2024-09-21T10:18:26.753439+0000","last_became_active":"2024-09-21T10:14:30.708636+0000","last_became_peered":"2024-09-21T10:14:30.708636+0000","last_unstale":"2024-09-21T10:18:26.753439+0000","last_undegraded":"2024-09-21T10:18:26.753439+0000","last_fullsized":"2024-09-21T10:18:26.753439+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:59.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:59.099852+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:20:01.949185+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":610,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.918513+0000","last_change":"2024-09-21T10:09:23.841186+0000","last_active":"2024-09-21T10:18:44.918513+0000","last_peered":"2024-09-21T10:18:44.918513+0000","last_clean":"2024-09-21T10:18:44.918513+0000","last_became_active":"2024-09-21T10:09:23.840854+0000","last_became_peered":"2024-09-21T10:09:23.840854+0000","last_unstale":"2024-09-21T10:18:44.918513+0000","last_undegraded":"2024-09-21T10:18:44.918513+0000","last_fullsized":"2024-09-21T10:18:44.918513+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:29.666358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":463,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.919639+0000","last_change":"2024-09-21T10:14:30.708385+0000","last_active":"2024-09-21T10:18:44.919639+0000","last_peered":"2024-09-21T10:18:44.919639+0000","last_clean":"2024-09-21T10:18:44.919639+0000","last_became_active":"2024-09-21T10:14:30.708046+0000","last_became_peered":"2024-09-21T10:14:30.708046+0000","last_unstale":"2024-09-21T10:18:44.919639+0000","last_undegraded":"2024-09-21T10:18:44.919639+0000","last_fullsized":"2024-09-21T10:18:44.919639+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:16:45.563095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":607,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.916168+0000","last_change":"2024-09-21T10:09:49.926548+0000","last_active":"2024-09-21T10:18:44.916168+0000","last_peered":"2024-09-21T10:18:44.916168+0000","last_clean":"2024-09-21T10:18:44.916168+0000","last_became_active":"2024-09-21T10:09:49.926255+0000","last_became_peered":"2024-09-21T10:09:49.926255+0000","last_unstale":"2024-09-21T10:18:44.916168+0000","last_undegraded":"2024-09-21T10:18:44.916168+0000","last_fullsized":"2024-09-21T10:18:44.916168+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:49.855878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.8","version":"373'4080","reported_seq":4103,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:54.740938+0000","last_change":"2024-09-21T10:18:13.038950+0000","last_active":"2024-09-21T10:18:54.740938+0000","last_peered":"2024-09-21T10:18:54.740938+0000","last_clean":"2024-09-21T10:18:54.740938+0000","last_became_active":"2024-09-21T10:18:13.038641+0000","last_became_peered":"2024-09-21T10:18:13.038641+0000","last_unstale":"2024-09-21T10:18:54.740938+0000","last_undegraded":"2024-09-21T10:18:54.740938+0000","last_fullsized":"2024-09-21T10:18:54.740938+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":4080,"log_dups_size":0,"ondisk_log_size":4080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:02.783389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267386880,"num_objects":4080,"num_object_clones":0,"num_object_copies":12240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4080,"num_write_kb":261120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":408,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.916695+0000","last_change":"2024-09-21T10:10:57.390804+0000","last_active":"2024-09-21T10:18:44.916695+0000","last_peered":"2024-09-21T10:18:44.916695+0000","last_clean":"2024-09-21T10:18:44.916695+0000","last_became_active":"2024-09-21T10:10:57.390323+0000","last_became_peered":"2024-09-21T10:10:57.390323+0000","last_unstale":"2024-09-21T10:18:44.916695+0000","last_undegraded":"2024-09-21T10:18:44.916695+0000","last_fullsized":"2024-09-21T10:18:44.916695+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.005477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.005477+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:29:57.702046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":264,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.921413+0000","last_change":"2024-09-21T10:14:30.718942+0000","last_active":"2024-09-21T10:18:44.921413+0000","last_peered":"2024-09-21T10:18:44.921413+0000","last_clean":"2024-09-21T10:18:44.921413+0000","last_became_active":"2024-09-21T10:14:30.709129+0000","last_became_peered":"2024-09-21T10:14:30.709129+0000","last_unstale":"2024-09-21T10:18:44.921413+0000","last_undegraded":"2024-09-21T10:18:44.921413+0000","last_fullsized":"2024-09-21T10:18:44.921413+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:18.171064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":385,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.753371+0000","last_change":"2024-09-21T10:03:38.392767+0000","last_active":"2024-09-21T10:18:26.753371+0000","last_peered":"2024-09-21T10:18:26.753371+0000","last_clean":"2024-09-21T10:18:26.753371+0000","last_became_active":"2024-09-21T10:03:38.392424+0000","last_became_peered":"2024-09-21T10:03:38.392424+0000","last_unstale":"2024-09-21T10:18:26.753371+0000","last_undegraded":"2024-09-21T10:18:26.753371+0000","last_fullsized":"2024-09-21T10:18:26.753371+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:25.358291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":79,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.753436+0000","last_change":"2024-09-21T10:14:30.703352+0000","last_active":"2024-09-21T10:18:26.753436+0000","last_peered":"2024-09-21T10:18:26.753436+0000","last_clean":"2024-09-21T10:18:26.753436+0000","last_became_active":"2024-09-21T10:14:30.703068+0000","last_became_peered":"2024-09-21T10:14:30.703068+0000","last_unstale":"2024-09-21T10:18:26.753436+0000","last_undegraded":"2024-09-21T10:18:26.753436+0000","last_fullsized":"2024-09-21T10:18:26.753436+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:00:26.966633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":79,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.753531+0000","last_change":"2024-09-21T10:14:30.702750+0000","last_active":"2024-09-21T10:18:26.753531+0000","last_peered":"2024-09-21T10:18:26.753531+0000","last_clean":"2024-09-21T10:18:26.753531+0000","last_became_active":"2024-09-21T10:14:30.702443+0000","last_became_peered":"2024-09-21T10:14:30.702443+0000","last_unstale":"2024-09-21T10:18:26.753531+0000","last_undegraded":"2024-09-21T10:18:26.753531+0000","last_fullsized":"2024-09-21T10:18:26.753531+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:38:19.353586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":89,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.752808+0000","last_change":"2024-09-21T10:14:30.716754+0000","last_active":"2024-09-21T10:18:26.752808+0000","last_peered":"2024-09-21T10:18:26.752808+0000","last_clean":"2024-09-21T10:18:26.752808+0000","last_became_active":"2024-09-21T10:14:30.704683+0000","last_became_peered":"2024-09-21T10:14:30.704683+0000","last_unstale":"2024-09-21T10:18:26.752808+0000","last_undegraded":"2024-09-21T10:18:26.752808+0000","last_fullsized":"2024-09-21T10:18:26.752808+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:45.993877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":515,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.920497+0000","last_change":"2024-09-21T10:14:05.477429+0000","last_active":"2024-09-21T10:18:44.920497+0000","last_peered":"2024-09-21T10:18:44.920497+0000","last_clean":"2024-09-21T10:18:44.920497+0000","last_became_active":"2024-09-21T10:14:05.476794+0000","last_became_peered":"2024-09-21T10:14:05.476794+0000","last_unstale":"2024-09-21T10:18:44.920497+0000","last_undegraded":"2024-09-21T10:18:44.920497+0000","last_fullsized":"2024-09-21T10:18:44.920497+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:18:54.963704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":258,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.913986+0000","last_change":"2024-09-21T10:14:30.713873+0000","last_active":"2024-09-21T10:18:44.913986+0000","last_peered":"2024-09-21T10:18:44.913986+0000","last_clean":"2024-09-21T10:18:44.913986+0000","last_became_active":"2024-09-21T10:14:30.703149+0000","last_became_peered":"2024-09-21T10:14:30.703149+0000","last_unstale":"2024-09-21T10:18:44.913986+0000","last_undegraded":"2024-09-21T10:18:44.913986+0000","last_fullsized":"2024-09-21T10:18:44.913986+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:26.453902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'100","reported_seq":408,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.918414+0000","last_change":"2024-09-21T10:18:12.633959+0000","last_active":"2024-09-21T10:18:44.918414+0000","last_peered":"2024-09-21T10:18:44.918414+0000","last_clean":"2024-09-21T10:18:44.918414+0000","last_became_active":"2024-09-21T10:18:12.633591+0000","last_became_peered":"2024-09-21T10:18:12.633591+0000","last_unstale":"2024-09-21T10:18:44.918414+0000","last_undegraded":"2024-09-21T10:18:44.918414+0000","last_fullsized":"2024-09-21T10:18:44.918414+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.014993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.014993+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:16:55.324246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":286,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.749945+0000","last_change":"2024-09-21T10:15:01.483814+0000","last_active":"2024-09-21T10:18:26.749945+0000","last_peered":"2024-09-21T10:18:26.749945+0000","last_clean":"2024-09-21T10:18:26.749945+0000","last_became_active":"2024-09-21T10:15:01.483522+0000","last_became_peered":"2024-09-21T10:15:01.483522+0000","last_unstale":"2024-09-21T10:18:26.749945+0000","last_undegraded":"2024-09-21T10:18:26.749945+0000","last_fullsized":"2024-09-21T10:18:26.749945+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:13.632066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":592,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.916235+0000","last_change":"2024-09-21T10:03:38.425111+0000","last_active":"2024-09-21T10:18:44.916235+0000","last_peered":"2024-09-21T10:18:44.916235+0000","last_clean":"2024-09-21T10:18:44.916235+0000","last_became_active":"2024-09-21T10:03:38.424704+0000","last_became_peered":"2024-09-21T10:03:38.424704+0000","last_unstale":"2024-09-21T10:18:44.916235+0000","last_undegraded":"2024-09-21T10:18:44.916235+0000","last_fullsized":"2024-09-21T10:18:44.916235+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:35.157076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":492,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.921800+0000","last_change":"2024-09-21T10:10:57.397755+0000","last_active":"2024-09-21T10:18:44.921800+0000","last_peered":"2024-09-21T10:18:44.921800+0000","last_clean":"2024-09-21T10:18:44.921800+0000","last_became_active":"2024-09-21T10:10:57.397249+0000","last_became_peered":"2024-09-21T10:10:57.397249+0000","last_unstale":"2024-09-21T10:18:44.921800+0000","last_undegraded":"2024-09-21T10:18:44.921800+0000","last_fullsized":"2024-09-21T10:18:44.921800+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:40.813503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":233,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.921431+0000","last_change":"2024-09-21T10:14:05.471573+0000","last_active":"2024-09-21T10:18:44.921431+0000","last_peered":"2024-09-21T10:18:44.921431+0000","last_clean":"2024-09-21T10:18:44.921431+0000","last_became_active":"2024-09-21T10:14:05.471270+0000","last_became_peered":"2024-09-21T10:14:05.471270+0000","last_unstale":"2024-09-21T10:18:44.921431+0000","last_undegraded":"2024-09-21T10:18:44.921431+0000","last_fullsized":"2024-09-21T10:18:44.921431+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:56:13.468768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":335,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.753483+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T10:18:26.753483+0000","last_peered":"2024-09-21T10:18:26.753483+0000","last_clean":"2024-09-21T10:18:26.753483+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T10:18:26.753483+0000","last_undegraded":"2024-09-21T10:18:26.753483+0000","last_fullsized":"2024-09-21T10:18:26.753483+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":613,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.916258+0000","last_change":"2024-09-21T10:09:49.928053+0000","last_active":"2024-09-21T10:18:44.916258+0000","last_peered":"2024-09-21T10:18:44.916258+0000","last_clean":"2024-09-21T10:18:44.916258+0000","last_became_active":"2024-09-21T10:09:49.927231+0000","last_became_peered":"2024-09-21T10:09:49.927231+0000","last_unstale":"2024-09-21T10:18:44.916258+0000","last_undegraded":"2024-09-21T10:18:44.916258+0000","last_fullsized":"2024-09-21T10:18:44.916258+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:26.987553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":525,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.921517+0000","last_change":"2024-09-21T10:09:23.840717+0000","last_active":"2024-09-21T10:18:44.921517+0000","last_peered":"2024-09-21T10:18:44.921517+0000","last_clean":"2024-09-21T10:18:44.921517+0000","last_became_active":"2024-09-21T10:09:23.840323+0000","last_became_peered":"2024-09-21T10:09:23.840323+0000","last_unstale":"2024-09-21T10:18:44.921517+0000","last_undegraded":"2024-09-21T10:18:44.921517+0000","last_fullsized":"2024-09-21T10:18:44.921517+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:55.769770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":246,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.914326+0000","last_change":"2024-09-21T10:18:12.632498+0000","last_active":"2024-09-21T10:18:44.914326+0000","last_peered":"2024-09-21T10:18:44.914326+0000","last_clean":"2024-09-21T10:18:44.914326+0000","last_became_active":"2024-09-21T10:18:12.631648+0000","last_became_peered":"2024-09-21T10:18:12.631648+0000","last_unstale":"2024-09-21T10:18:44.914326+0000","last_undegraded":"2024-09-21T10:18:44.914326+0000","last_fullsized":"2024-09-21T10:18:44.914326+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:17.592618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":591,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.917155+0000","last_change":"2024-09-21T10:14:30.702101+0000","last_active":"2024-09-21T10:18:44.917155+0000","last_peered":"2024-09-21T10:18:44.917155+0000","last_clean":"2024-09-21T10:18:44.917155+0000","last_became_active":"2024-09-21T10:14:30.701783+0000","last_became_peered":"2024-09-21T10:14:30.701783+0000","last_unstale":"2024-09-21T10:18:44.917155+0000","last_undegraded":"2024-09-21T10:18:44.917155+0000","last_fullsized":"2024-09-21T10:18:44.917155+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:13.087010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":79,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.753985+0000","last_change":"2024-09-21T10:14:30.707503+0000","last_active":"2024-09-21T10:18:26.753985+0000","last_peered":"2024-09-21T10:18:26.753985+0000","last_clean":"2024-09-21T10:18:26.753985+0000","last_became_active":"2024-09-21T10:14:30.706881+0000","last_became_peered":"2024-09-21T10:14:30.706881+0000","last_unstale":"2024-09-21T10:18:26.753985+0000","last_undegraded":"2024-09-21T10:18:26.753985+0000","last_fullsized":"2024-09-21T10:18:26.753985+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:23:07.485461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":630,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.918462+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:18:44.918462+0000","last_peered":"2024-09-21T10:18:44.918462+0000","last_clean":"2024-09-21T10:18:44.918462+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:18:44.918462+0000","last_undegraded":"2024-09-21T10:18:44.918462+0000","last_fullsized":"2024-09-21T10:18:44.918462+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":255,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.922154+0000","last_change":"2024-09-21T10:14:30.702414+0000","last_active":"2024-09-21T10:18:44.922154+0000","last_peered":"2024-09-21T10:18:44.922154+0000","last_clean":"2024-09-21T10:18:44.922154+0000","last_became_active":"2024-09-21T10:14:30.702009+0000","last_became_peered":"2024-09-21T10:14:30.702009+0000","last_unstale":"2024-09-21T10:18:44.922154+0000","last_undegraded":"2024-09-21T10:18:44.922154+0000","last_fullsized":"2024-09-21T10:18:44.922154+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:50:48.396082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":15,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.753055+0000","last_change":"2024-09-21T10:18:12.633151+0000","last_active":"2024-09-21T10:18:26.753055+0000","last_peered":"2024-09-21T10:18:26.753055+0000","last_clean":"2024-09-21T10:18:26.753055+0000","last_became_active":"2024-09-21T10:18:12.632854+0000","last_became_peered":"2024-09-21T10:18:12.632854+0000","last_unstale":"2024-09-21T10:18:26.753055+0000","last_undegraded":"2024-09-21T10:18:26.753055+0000","last_fullsized":"2024-09-21T10:18:26.753055+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:14.602893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":83,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.753366+0000","last_change":"2024-09-21T10:14:30.720787+0000","last_active":"2024-09-21T10:18:26.753366+0000","last_peered":"2024-09-21T10:18:26.753366+0000","last_clean":"2024-09-21T10:18:26.753366+0000","last_became_active":"2024-09-21T10:14:30.706001+0000","last_became_peered":"2024-09-21T10:14:30.706001+0000","last_unstale":"2024-09-21T10:18:26.753366+0000","last_undegraded":"2024-09-21T10:18:26.753366+0000","last_fullsized":"2024-09-21T10:18:26.753366+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:40:11.987108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":15,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.753884+0000","last_change":"2024-09-21T10:18:12.636252+0000","last_active":"2024-09-21T10:18:26.753884+0000","last_peered":"2024-09-21T10:18:26.753884+0000","last_clean":"2024-09-21T10:18:26.753884+0000","last_became_active":"2024-09-21T10:18:12.635945+0000","last_became_peered":"2024-09-21T10:18:12.635945+0000","last_unstale":"2024-09-21T10:18:26.753884+0000","last_undegraded":"2024-09-21T10:18:26.753884+0000","last_fullsized":"2024-09-21T10:18:26.753884+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:59:29.464045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":83,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.753518+0000","last_change":"2024-09-21T10:14:30.722338+0000","last_active":"2024-09-21T10:18:26.753518+0000","last_peered":"2024-09-21T10:18:26.753518+0000","last_clean":"2024-09-21T10:18:26.753518+0000","last_became_active":"2024-09-21T10:14:30.708164+0000","last_became_peered":"2024-09-21T10:14:30.708164+0000","last_unstale":"2024-09-21T10:18:26.753518+0000","last_undegraded":"2024-09-21T10:18:26.753518+0000","last_fullsized":"2024-09-21T10:18:26.753518+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:32:13.825114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":614,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.916607+0000","last_change":"2024-09-21T10:09:23.840467+0000","last_active":"2024-09-21T10:18:44.916607+0000","last_peered":"2024-09-21T10:18:44.916607+0000","last_clean":"2024-09-21T10:18:44.916607+0000","last_became_active":"2024-09-21T10:09:23.840177+0000","last_became_peered":"2024-09-21T10:09:23.840177+0000","last_unstale":"2024-09-21T10:18:44.916607+0000","last_undegraded":"2024-09-21T10:18:44.916607+0000","last_fullsized":"2024-09-21T10:18:44.916607+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:17.887307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":604,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.918692+0000","last_change":"2024-09-21T10:11:05.451613+0000","last_active":"2024-09-21T10:18:44.918692+0000","last_peered":"2024-09-21T10:18:44.918692+0000","last_clean":"2024-09-21T10:18:44.918692+0000","last_became_active":"2024-09-21T10:11:05.451075+0000","last_became_peered":"2024-09-21T10:11:05.451075+0000","last_unstale":"2024-09-21T10:18:44.918692+0000","last_undegraded":"2024-09-21T10:18:44.918692+0000","last_fullsized":"2024-09-21T10:18:44.918692+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:18.505045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":79,"reported_epoch":372,"state":"active+clean","last_fresh":"2024-09-21T10:18:26.752504+0000","last_change":"2024-09-21T10:14:30.707548+0000","last_active":"2024-09-21T10:18:26.752504+0000","last_peered":"2024-09-21T10:18:26.752504+0000","last_clean":"2024-09-21T10:18:26.752504+0000","last_became_active":"2024-09-21T10:14:30.707232+0000","last_became_peered":"2024-09-21T10:14:30.707232+0000","last_unstale":"2024-09-21T10:18:26.752504+0000","last_undegraded":"2024-09-21T10:18:26.752504+0000","last_fullsized":"2024-09-21T10:18:26.752504+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:11:38.905767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":268,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-21T10:18:44.921668+0000","last_change":"2024-09-21T10:10:57.401588+0000","last_active":"2024-09-21T10:18:44.921668+0000","last_peered":"2024-09-21T10:18:44.921668+0000","last_clean":"2024-09-21T10:18:44.921668+0000","last_became_active":"2024-09-21T10:10:57.397724+0000","last_became_peered":"2024-09-21T10:10:57.397724+0000","last_unstale":"2024-09-21T10:18:44.921668+0000","last_undegraded":"2024-09-21T10:18:44.921668+0000","last_fullsized":"2024-09-21T10:18:44.921668+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:47.843630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":16,"stat_sum":{"num_bytes":4019257344,"num_objects":61329,"num_object_clones":0,"num_object_copies":183987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61329,"num_write_kb":3925056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12073500672,"data_stored":12073500672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61329,"ondisk_log_size":61329,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":11856,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26580,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":644037,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":632715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626918,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2115580,"kb_used_data":1211060,"kb_used_omap":1,"kb_used_meta":904510,"kb_avail":91629572,"statfs":{"total":95995035648,"available":93828681728,"internally_reserved":0,"allocated":1240125440,"data_stored":1237832061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":926218584},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019376,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3036032,"kb_used_data":2136760,"kb_used_omap":1,"kb_used_meta":899262,"kb_avail":90709120,"statfs":{"total":95995035648,"available":92886138880,"internally_reserved":0,"allocated":2188042240,"data_stored":2185744806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920844632},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":245,"seq":1052266987635,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1151280,"kb_used_data":745448,"kb_used_omap":8,"kb_used_meta":405815,"kb_avail":92593872,"statfs":{"total":95995035648,"available":94816124928,"internally_reserved":0,"allocated":763338752,"data_stored":761053448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8940,"internal_metadata":415554836},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":175,"seq":751619276984,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2126848,"kb_used_data":1193720,"kb_used_omap":12,"kb_used_meta":933107,"kb_avail":91618304,"statfs":{"total":95995035648,"available":93817143296,"internally_reserved":0,"allocated":1222369280,"data_stored":1220071842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12845,"internal_metadata":955502035},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823147,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3400112,"kb_used_data":1906472,"kb_used_omap":1,"kb_used_meta":1493630,"kb_avail":90345040,"statfs":{"total":95995035648,"available":92513320960,"internally_reserved":0,"allocated":1952227328,"data_stored":1949941977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1529477464},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879570,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1995440,"kb_used_data":972136,"kb_used_omap":4,"kb_used_meta":1023291,"kb_avail":91749712,"statfs":{"total":95995035648,"available":93951705088,"internally_reserved":0,"allocated":995467264,"data_stored":993181994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":1047850428},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566411,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3306552,"kb_used_data":2306544,"kb_used_omap":6,"kb_used_meta":999993,"kb_avail":90438600,"statfs":{"total":95995035648,"available":92609126400,"internally_reserved":0,"allocated":2361901056,"data_stored":2359608206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":1023993255},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860755,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2290600,"kb_used_data":1166304,"kb_used_omap":5,"kb_used_meta":1124282,"kb_avail":91454552,"statfs":{"total":95995035648,"available":93649461248,"internally_reserved":0,"allocated":1194295296,"data_stored":1192017280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":1151265007},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7236,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9976,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":762183680,"data_stored":762183680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1207304192,"data_stored":1207304192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2396651520,"data_stored":2396651520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1008205824,"data_stored":1008205824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2229731328,"data_stored":2229731328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1974468608,"data_stored":1974468608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1245708288,"data_stored":1245708288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1249247232,"data_stored":1249247232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:19:04.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pgp_num 21 2024-09-21T10:19:04.646 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:04.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:04.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:04.748 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:04.784 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:04.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:04.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:04.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:04 smithi107 ceph-mon[102221]: pgmap v1279: 109 pgs: 109 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-21T10:19:04.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:04 smithi107 ceph-mon[105022]: pgmap v1279: 109 pgs: 109 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-21T10:19:04.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:04.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:04 smithi129 ceph-mon[88471]: pgmap v1279: 109 pgs: 109 active+clean; 4.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-21T10:19:04.929 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:04.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:04.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:05.031 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:05.065 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 43 16 78203 78187 113.621 81 0.0175022 0.00879635 2024-09-21T10:19:05.070 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:05.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:05.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:05.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:05.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:05.208 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:05.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:05.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:05.342 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:05.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:05.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:05.444 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:05.481 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:05.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:05.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:05.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:05.612 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:05.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:05.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:05.714 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:05.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:05.749 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:05.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:05.850 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:05.885 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:05.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:05.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:05.987 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:06.017 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:06.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:06.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:06.065 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 44 16 79502 79486 112.883 81.1875 0.00712722 0.00885403 2024-09-21T10:19:06.119 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:06.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:06.156 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:06.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:06.257 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:06.296 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:06.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:06.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:06.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:06.438 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:06.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:06.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:06.540 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:06.582 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:06.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:06.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:06.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:06.700 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:06 smithi107 ceph-mon[102221]: pgmap v1280: 109 pgs: 109 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-21T10:19:06.700 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:06 smithi107 ceph-mon[105022]: pgmap v1280: 109 pgs: 109 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-21T10:19:06.731 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:06.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:06.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:06.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:06.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:06.871 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:06.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:06 smithi129 ceph-mon[88471]: pgmap v1280: 109 pgs: 109 active+clean; 4.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-21T10:19:06.972 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:07.008 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:07.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:07.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:07.066 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 45 16 80751 80735 112.109 78.0625 0.00817002 0.00891526 2024-09-21T10:19:07.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:07.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:07.145 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:07.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:07.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:07.300 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:07.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:07.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:07.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:07.431 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:07.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:07.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:07.533 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:07.545 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:19:07.577 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:07.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:07.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:07.678 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:07.711 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:07.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:07.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:07.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:07.850 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:07.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:07.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:07.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:07 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3306929966' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:19:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:07 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3306929966' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:19:07.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:07 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3306929966' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:19:07.952 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:07.983 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:07.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:07.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:08.065 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 46 16 82023 82007 111.4 79.5 0.00552644 0.00897148 2024-09-21T10:19:08.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:08.113 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:08.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:08.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:08.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:08.251 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:08.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:08.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:08.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:08.389 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:08.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:08.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:08.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:08.543 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:08.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:08.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:08.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:08.684 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:08.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:08.785 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:08.829 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:08.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:08.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:08 smithi107 ceph-mon[102221]: pgmap v1281: 109 pgs: 109 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-21T10:19:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:08 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:08 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3306929966' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:19:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:08 smithi107 ceph-mon[102221]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T10:19:08.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:08 smithi107 ceph-mon[105022]: pgmap v1281: 109 pgs: 109 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-21T10:19:08.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:08 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:08.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:08 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3306929966' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:19:08.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:08 smithi107 ceph-mon[105022]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T10:19:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:08 smithi129 ceph-mon[88471]: pgmap v1281: 109 pgs: 109 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-21T10:19:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:08 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:08 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3306929966' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:19:08.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:08 smithi129 ceph-mon[88471]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T10:19:08.931 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:08.974 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:08.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:08.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:09.042 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:09.067 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 47 16 83299 83283 110.726 79.75 0.00644134 0.00902618 2024-09-21T10:19:09.075 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:09.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:09.112 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:09.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:09.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:09.257 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:09.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:09.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:09.347 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:19:09.347 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:19:09.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:09.365 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:19:09.365 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:19:09.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:09.381 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:09.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:09.382 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:19:09.382 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:19:09.404 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:19:09.404 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:19:09.434 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:19:09.434 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:19:09.460 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:19:09.460 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:19:09.482 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:09.484 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:19:09.484 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:19:09.517 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:19:09.517 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:19:09.521 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:09.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:09.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:09.623 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:09.656 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:09.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:09.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:09.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:09.795 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:09.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:09.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:09 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:19:09.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:09 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:19:09.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:09 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/247246049' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:09.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:09 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:19:09.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:09 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:19:09.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:09 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/247246049' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:09.896 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:09 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:19:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:09 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:19:09.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:09 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/247246049' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:09.937 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:09.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:10.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:10.066 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 48 15 84467 84452 109.941 73.0625 0.0115468 0.00909125 2024-09-21T10:19:10.073 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:10.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:10.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:10.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:10.211 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:10.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:10.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:10.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:10.339 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:10.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:10.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:10.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:10.470 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:10.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:10.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:10.572 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:10.586 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:19:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:19:10.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:19:10.609 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:10.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:10.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:10.711 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:10.746 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:10.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:10.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:10.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:10.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:10 smithi107 ceph-mon[102221]: pgmap v1283: 109 pgs: 109 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-21T10:19:10.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:10 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:19:10.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:10 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:19:10.853 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:10 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:19:10.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:10 smithi107 ceph-mon[105022]: pgmap v1283: 109 pgs: 109 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-21T10:19:10.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:10 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:19:10.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:10 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:19:10.853 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:10 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:19:10.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:10.887 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:10.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:10.902 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:10 smithi129 ceph-mon[88471]: pgmap v1283: 109 pgs: 109 active+clean; 4.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-21T10:19:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:10 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:19:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:10 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:19:10.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:10 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:19:10.988 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:11.026 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:11.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:11.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:11.067 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 49 16 85632 85616 109.182 72.75 0.0364982 0.00915293 2024-09-21T10:19:11.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:19:11.128 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:11.160 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:11.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:11.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:11.262 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:11.306 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:11.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:11.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:11.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:11.445 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:11.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:11.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:11.546 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:11.592 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:11.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:11.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:11.694 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:11.733 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:11.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:11.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:11.835 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:11.878 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:11.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:11.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:11.980 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:12.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:12.017 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:12.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:12.067 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 50 16 86715 86699 108.352 67.6875 0.0116964 0.00922468 2024-09-21T10:19:12.089 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:19:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:19:11] "GET /metrics HTTP/1.1" 200 36780 "" "Prometheus/2.51.0" 2024-09-21T10:19:12.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:12.146 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:12.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:12.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:12.248 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:12.284 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:12.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:12.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:19:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:12 smithi107 ceph-mon[102221]: pgmap v1284: 109 pgs: 109 active+clean; 5.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T10:19:12.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:19:12.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:12.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:12 smithi107 ceph-mon[105022]: pgmap v1284: 109 pgs: 109 active+clean; 5.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T10:19:12.385 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:12.426 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:12.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:12.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:12.528 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:12.569 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:12.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:12.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:12.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:19:12.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:12.614 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:12 smithi129 ceph-mon[88471]: pgmap v1284: 109 pgs: 109 active+clean; 5.0 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-21T10:19:12.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:12.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:12.707 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:12.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:12.809 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:12.845 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:12.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:12.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:12.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:19:12.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:12.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:12.992 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:12.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:13.067 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 51 16 87818 87802 107.579 68.9375 0.0157394 0.00929054 2024-09-21T10:19:13.094 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:13.130 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:13.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:13.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:13.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:13.275 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:13.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:13.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:13.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:13.407 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:13.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:13.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:13.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:13.544 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:13.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:13.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:13.581 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:13 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4209990189' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T10:19:13.581 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:13 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T10:19:13.581 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:13 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4209990189' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T10:19:13.581 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:13 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T10:19:13.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:13 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4209990189' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T10:19:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:13 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T10:19:13.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:13.697 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:13.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:13.740 INFO:teuthology.orchestra.run.smithi107.stderr:set pool 1 pgp_num to 21 2024-09-21T10:19:13.799 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:13.836 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:13.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:13.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:13.938 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:13.958 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:13.975 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:13.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:14.067 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 52 15 89006 88991 106.939 74.3125 0.00970195 0.00934696 2024-09-21T10:19:14.076 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:14.110 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:14.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:14.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:14.212 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:14.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:14.250 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:14.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:14.351 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:14.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:14 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-21T10:19:14.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:14 smithi107 ceph-mon[102221]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T10:19:14.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:14 smithi107 ceph-mon[102221]: pgmap v1286: 109 pgs: 109 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T10:19:14.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:14 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-21T10:19:14.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:14 smithi107 ceph-mon[105022]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T10:19:14.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:14 smithi107 ceph-mon[105022]: pgmap v1286: 109 pgs: 109 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T10:19:14.390 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:14.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:14.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:14.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:14.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:14.528 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:14.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:14.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:14.667 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:14.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:14.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:14 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-21T10:19:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:14 smithi129 ceph-mon[88471]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T10:19:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:14 smithi129 ceph-mon[88471]: pgmap v1286: 109 pgs: 109 active+clean; 5.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T10:19:14.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:14.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:14.811 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:14.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:14.912 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:14.951 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:14.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:14.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:15.052 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:15.067 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 53 16 90201 90185 106.329 74.625 0.016061 0.0093989 2024-09-21T10:19:15.091 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:15.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:15.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:15.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:15.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:15.232 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:15.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:15.290 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:19:15.290 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:19:15.333 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:15.369 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:15.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:15.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:15.470 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:15.511 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:15.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:15.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:15.613 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:15.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:15.651 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:15.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:15.752 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:15.791 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:15.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:15.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:15.893 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:15.934 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:15.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:15.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:16.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:16.063 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:16.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:16.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:16.067 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 54 16 91365 91349 105.708 72.75 0.014364 0.00945518 2024-09-21T10:19:16.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:16.196 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:16.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:16.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:16.298 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:16.338 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:16.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:16.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:16.440 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:16.467 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:16.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:16.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:16.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:19:16.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:16.622 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:16.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:16.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:16.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:16.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:16.754 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:16.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:16.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:16 smithi107 ceph-mon[102221]: pgmap v1287: 109 pgs: 109 active+clean; 5.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 59 MiB/s wr, 941 op/s 2024-09-21T10:19:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:16 smithi107 ceph-mon[105022]: pgmap v1287: 109 pgs: 109 active+clean; 5.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 59 MiB/s wr, 941 op/s 2024-09-21T10:19:16.887 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:16.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:16.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:16.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:16 smithi129 ceph-mon[88471]: pgmap v1287: 109 pgs: 109 active+clean; 5.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 59 MiB/s wr, 941 op/s 2024-09-21T10:19:16.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:17.024 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:17.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:17.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:17.067 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 55 16 92536 92520 105.116 73.1875 0.00654721 0.00950816 2024-09-21T10:19:17.126 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:17.159 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:17.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:17.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:17.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:17.293 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:17.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:17.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:17.395 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:17.436 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:17.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:17.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:17.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:17.573 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:17.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:17.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:17.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:17.714 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:17.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:17.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:17.806 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:17.806 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:17.815 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:17.845 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:17.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:17.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:17.983 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:17.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:17.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:18.067 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 56 16 93655 93639 104.488 69.9375 0.00951293 0.00956533 2024-09-21T10:19:18.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:18.120 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:18.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:18.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:18.222 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:18.253 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:18.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:18.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:18.355 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:18.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:18.392 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:18.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:18.493 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:18.518 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:19:18.522 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:18.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:18.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:18.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:18.657 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:18.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:18.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:18.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:18.790 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:18.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:18.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:18.811 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:18 smithi107 ceph-mon[102221]: pgmap v1288: 109 pgs: 109 active+clean; 5.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-21T10:19:18.811 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:18 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4291444896' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:19:18.811 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:18 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:19:18.812 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:18 smithi107 ceph-mon[105022]: pgmap v1288: 109 pgs: 109 active+clean; 5.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-21T10:19:18.812 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:18 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4291444896' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:19:18.812 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:18 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:19:18.891 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:18 smithi129 ceph-mon[88471]: pgmap v1288: 109 pgs: 109 active+clean; 5.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-21T10:19:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:18 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4291444896' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:19:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:18 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:19:18.930 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:18.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:18.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:19.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:19.057 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:19.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:19.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:19.067 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 57 16 94786 94770 103.895 70.6875 0.0159657 0.00961996 2024-09-21T10:19:19.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:19.185 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:19.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:19.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:19.287 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:19.326 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:19.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:19.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:19.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:19.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:19.470 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:19.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:19.552 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:19:19.553 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:19:19.553 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_8 2024-09-21T10:19:19.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:19:19.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:19.643 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:19.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:19.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:19.746 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:19.789 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:19.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:19.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:19 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:19 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:19:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:19 smithi107 ceph-mon[102221]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T10:19:19.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:19 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:19.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:19 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:19:19.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:19 smithi107 ceph-mon[105022]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T10:19:19.891 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:19 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:19 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:19:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:19 smithi129 ceph-mon[88471]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T10:19:19.928 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:19.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:19.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:20.029 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:20.060 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:20.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:20.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:20.067 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 58 16 95855 95839 103.256 66.8125 0.0166712 0.00968034 2024-09-21T10:19:20.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:20.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:20.191 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:20.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:20.292 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:20.316 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:20.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:20.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:20.418 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:20.459 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:20.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:20.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:20.534 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:19:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:19:20.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:19:20.561 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:20.588 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:20.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:20.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:20.690 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:20.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:20.731 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:20.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:20.832 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:20.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:20 smithi107 ceph-mon[102221]: pgmap v1290: 109 pgs: 109 active+clean; 5.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T10:19:20.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:20 smithi107 ceph-mon[105022]: pgmap v1290: 109 pgs: 109 active+clean; 5.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T10:19:20.863 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:20.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:20.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:20 smithi129 ceph-mon[88471]: pgmap v1290: 109 pgs: 109 active+clean; 5.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T10:19:20.966 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:21.003 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:21.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:21.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:21.067 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 59 16 96964 96948 102.68 69.3125 0.0163217 0.00973414 2024-09-21T10:19:21.105 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:21.127 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:21.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:21.229 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:21.263 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:21.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:21.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:21.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:21.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:21.399 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:21.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:21.500 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:21.537 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:21.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:21.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:21.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:21.669 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:21.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:21.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:21.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:21.801 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:21.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:21.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:21.903 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:21.931 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:21.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:21.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:22.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:22.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:22.061 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:22.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:22.067 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:19:22.066515+0000 min lat: 0.00115007 max lat: 0.53677 avg lat: 0.00979939 2024-09-21T10:19:22.067 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:19:22.068 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 60 16 97947 97931 101.993 61.4375 0.00424981 0.00979939 2024-09-21T10:19:22.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:19:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:19:21] "GET /metrics HTTP/1.1" 200 36780 "" "Prometheus/2.51.0" 2024-09-21T10:19:22.162 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:22.199 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:22.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:22.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:22.300 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:22.337 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:22.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:22.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:22.438 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:22.466 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:22.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:22.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:22.568 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:22.599 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:22.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:22.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:22.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:22 smithi107 ceph-mon[102221]: pgmap v1291: 109 pgs: 109 active+clean; 5.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 52 MiB/s wr, 827 op/s 2024-09-21T10:19:22.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:22 smithi107 ceph-mon[105022]: pgmap v1291: 109 pgs: 109 active+clean; 5.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 52 MiB/s wr, 827 op/s 2024-09-21T10:19:22.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:22.731 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:22.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:22.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:22.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:22.866 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:22.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:22.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:22 smithi129 ceph-mon[88471]: pgmap v1291: 109 pgs: 109 active+clean; 5.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 52 MiB/s wr, 827 op/s 2024-09-21T10:19:22.967 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:23.005 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:23.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:23.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:23.068 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 61 16 99062 99046 101.463 69.6875 0.0341616 0.0098512 2024-09-21T10:19:23.108 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:23.150 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:23.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:23.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:23.252 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:23.295 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:23.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:23.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:23.396 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:23.430 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:23.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:23.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:23.532 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:23.565 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:23.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:23.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:23.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:23.706 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:23.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:23.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:23.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:23.846 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:23.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:23.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:23.948 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:23.983 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:23.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:23.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:24.068 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 62 16 100148 100132 100.921 67.875 0.012117 0.0099038 2024-09-21T10:19:24.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:24.086 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:24.155 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:24.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:24.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:24.257 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:24.292 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:24.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:24.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:24.394 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:24.451 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:24.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:24.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:24.553 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:24.586 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:24.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:24.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:24.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:24 smithi107 ceph-mon[102221]: pgmap v1292: 109 pgs: 109 active+clean; 6.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-09-21T10:19:24.674 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:24 smithi107 ceph-mon[105022]: pgmap v1292: 109 pgs: 109 active+clean; 6.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-09-21T10:19:24.688 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:24.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:24.726 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:24.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:24.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:24.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:24.863 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:24.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:24.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:24 smithi129 ceph-mon[88471]: pgmap v1292: 109 pgs: 109 active+clean; 6.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-09-21T10:19:24.964 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:25.003 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:25.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:25.068 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 63 16 101172 101156 100.335 64 0.00716611 0.00996213 2024-09-21T10:19:25.104 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:25.145 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:25.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:25.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:25.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:19:25.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:25.267 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:25.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:25.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:25.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:25.429 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:25.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:25.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:25.531 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:25.566 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:25.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:25.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:25.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:25.698 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:25.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:25.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:25.799 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:25.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:25.834 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:25.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:25.935 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:25.962 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:25.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:25.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:26.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:26.069 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 64 16 102102 102086 99.6746 58.125 0.0139052 0.0100277 2024-09-21T10:19:26.086 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:19:26.086 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:19:26.098 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:26.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:26.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:26.200 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:26.237 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:26.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:26.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:26.339 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:26.371 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:26.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:26.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:26.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:26.505 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:26.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:26.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:26.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:26.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:26.645 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:26.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:26.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:26.786 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:26.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:26.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:26 smithi107 ceph-mon[102221]: pgmap v1293: 109 pgs: 109 active+clean; 6.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T10:19:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:26 smithi107 ceph-mon[105022]: pgmap v1293: 109 pgs: 109 active+clean; 6.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T10:19:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:26.889 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:26.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:26 smithi129 ceph-mon[88471]: pgmap v1293: 109 pgs: 109 active+clean; 6.0 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T10:19:26.921 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:26.926 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:26.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:26.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:27.028 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:27.069 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 65 16 103204 103188 99.2007 68.875 0.0182612 0.0100753 2024-09-21T10:19:27.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:27.073 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:27.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:27.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:27.205 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:27.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:27.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:27.306 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:27.342 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:27.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:27.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:27.444 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:27.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:27.480 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:27.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:27.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:19:27.581 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:27.618 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:27.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:27.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:27.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:27.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:27.768 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:27.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:27.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:27.883 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:19:27.883 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:19:27.909 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:19:27.909 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:19:27.910 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:27.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:27.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:27.946 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:19:27.947 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:19:27.973 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:19:27.974 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:19:27.999 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:19:27.999 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:19:28.011 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:28.033 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:19:28.033 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:19:28.046 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:28.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:28.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:28.066 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:19:28.066 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:19:28.068 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 66 16 104235 104219 98.674 64.4375 0.0155737 0.0101295 2024-09-21T10:19:28.097 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:19:28.097 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:19:28.148 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:28.186 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:28.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:28.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:28.288 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:28.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:28.323 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:28.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:28.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:28.480 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:28.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:28.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:28.581 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:28.621 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:28.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:28.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:28.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:28.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:28.757 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:28.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:28.765 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:28 smithi107 ceph-mon[102221]: pgmap v1294: 109 pgs: 109 active+clean; 6.2 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-21T10:19:28.765 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:28 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3188997167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:28.766 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:28 smithi107 ceph-mon[105022]: pgmap v1294: 109 pgs: 109 active+clean; 6.2 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-21T10:19:28.766 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:28 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3188997167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:28.857 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:28.895 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:28.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:28.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:28.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:28 smithi129 ceph-mon[88471]: pgmap v1294: 109 pgs: 109 active+clean; 6.2 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-21T10:19:28.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:28 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3188997167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:28.997 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:29.045 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:29.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:29.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:29.069 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 67 16 105315 105299 98.2087 67.5 0.0114457 0.0101777 2024-09-21T10:19:29.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:29.178 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:29.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:29.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:29.263 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:29.279 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:29.324 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:29.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:29.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:29.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:29.461 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:29.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:29.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:29.563 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:29.599 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:29.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:29.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:29.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:29.732 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:29.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:29.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:29.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:29.872 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:29.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:29.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:29.974 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:30.010 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:30.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:30.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:30.069 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 68 16 106358 106342 97.723 65.1875 0.00697187 0.0102283 2024-09-21T10:19:30.111 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:30.122 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:30.173 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:30.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:30.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:30.248 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:19:30.252 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:19:30.276 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:30.313 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:30.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:30.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:30.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:30.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:30.457 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:30.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:30.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:30.559 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:19:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:19:30.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:19:30.595 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:30.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:30.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:30.681 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:30 smithi107 ceph-mon[102221]: pgmap v1295: 109 pgs: 109 active+clean; 6.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 61 MiB/s wr, 973 op/s 2024-09-21T10:19:30.682 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:30 smithi107 ceph-mon[105022]: pgmap v1295: 109 pgs: 109 active+clean; 6.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 61 MiB/s wr, 973 op/s 2024-09-21T10:19:30.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:30.725 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:30.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:30.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:30.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:19:30.827 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:30.872 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:30.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:30.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:30.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:30 smithi129 ceph-mon[88471]: pgmap v1295: 109 pgs: 109 active+clean; 6.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 61 MiB/s wr, 973 op/s 2024-09-21T10:19:30.976 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:31.016 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:31.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:31.070 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 69 16 107367 107351 97.2187 63.0625 0.0126413 0.0102814 2024-09-21T10:19:31.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:31.158 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:31.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:31.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:31.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:31.300 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:31.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:31.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:31.402 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:31.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:31.439 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:31.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:31.540 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:31.579 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:31.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:31.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:31.681 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:31.716 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:31.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:31.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:31.767 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:31 smithi107 ceph-mon[102221]: from='client.46940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:19:31.767 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:31 smithi107 ceph-mon[105022]: from='client.46940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:19:31.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:31.854 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:31.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:31.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:31 smithi129 ceph-mon[88471]: from='client.46940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:19:31.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:31.992 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:31.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:31.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:32.070 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 70 16 108418 108402 96.7682 65.6875 0.00540421 0.0103288 2024-09-21T10:19:32.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:19:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:19:31] "GET /metrics HTTP/1.1" 200 36784 "" "Prometheus/2.51.0" 2024-09-21T10:19:32.126 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:32.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:32.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:32.228 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:32.259 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:32.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:32.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:32.360 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:32.393 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1295,"stamp":"2024-09-21T10:19:29.451800+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6739323250,"num_objects":102850,"num_object_clones":0,"num_object_copies":308550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102850,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":102901,"num_write_kb":6580916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":644571,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103883,"ondisk_log_size":103883,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":28049068,"kb_used_data":19607084,"kb_used_omap":43,"kb_used_meta":8441876,"kb_avail":721912148,"statfs":{"total":767960285184,"available":739238039552,"internally_reserved":0,"allocated":20077654016,"data_stored":20059151542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44223,"internal_metadata":8644481857},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":698155008,"num_objects":10653,"num_object_clones":0,"num_object_copies":31959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10653,"num_write_kb":681792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.943679"},"pg_stats":[{"pgid":"1.11","version":"77'100","reported_seq":271,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.528960+0000","last_change":"2024-09-21T10:10:57.397786+0000","last_active":"2024-09-21T10:19:18.528960+0000","last_peered":"2024-09-21T10:19:18.528960+0000","last_clean":"2024-09-21T10:19:18.528960+0000","last_became_active":"2024-09-21T10:10:57.396742+0000","last_became_peered":"2024-09-21T10:10:57.396742+0000","last_unstale":"2024-09-21T10:19:18.528960+0000","last_undegraded":"2024-09-21T10:19:18.528960+0000","last_fullsized":"2024-09-21T10:19:18.528960+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:53.631923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:53.631923+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:53:52.740470+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":608,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.532428+0000","last_change":"2024-09-21T10:03:38.389577+0000","last_active":"2024-09-21T10:19:18.532428+0000","last_peered":"2024-09-21T10:19:18.532428+0000","last_clean":"2024-09-21T10:19:18.532428+0000","last_became_active":"2024-09-21T10:03:38.389292+0000","last_became_peered":"2024-09-21T10:03:38.389292+0000","last_unstale":"2024-09-21T10:19:18.532428+0000","last_undegraded":"2024-09-21T10:19:18.532428+0000","last_fullsized":"2024-09-21T10:19:18.532428+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:32.113472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":261,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.201508+0000","last_change":"2024-09-21T10:14:30.717695+0000","last_active":"2024-09-21T10:19:13.201508+0000","last_peered":"2024-09-21T10:19:13.201508+0000","last_clean":"2024-09-21T10:19:13.201508+0000","last_became_active":"2024-09-21T10:14:30.705952+0000","last_became_peered":"2024-09-21T10:14:30.705952+0000","last_unstale":"2024-09-21T10:19:13.201508+0000","last_undegraded":"2024-09-21T10:19:13.201508+0000","last_fullsized":"2024-09-21T10:19:13.201508+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:08:18.863430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'100","reported_seq":274,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.200679+0000","last_change":"2024-09-21T10:11:05.438031+0000","last_active":"2024-09-21T10:19:13.200679+0000","last_peered":"2024-09-21T10:19:13.200679+0000","last_clean":"2024-09-21T10:19:13.200679+0000","last_became_active":"2024-09-21T10:11:05.437637+0000","last_became_peered":"2024-09-21T10:11:05.437637+0000","last_unstale":"2024-09-21T10:19:13.200679+0000","last_undegraded":"2024-09-21T10:19:13.200679+0000","last_fullsized":"2024-09-21T10:19:13.200679+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.798577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.798577+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:42.343217+0000","scrub_duration":0.0023622640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":366,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.201526+0000","last_change":"2024-09-21T10:10:57.392782+0000","last_active":"2024-09-21T10:19:13.201526+0000","last_peered":"2024-09-21T10:19:13.201526+0000","last_clean":"2024-09-21T10:19:13.201526+0000","last_became_active":"2024-09-21T10:10:57.392305+0000","last_became_peered":"2024-09-21T10:10:57.392305+0000","last_unstale":"2024-09-21T10:19:13.201526+0000","last_undegraded":"2024-09-21T10:19:13.201526+0000","last_fullsized":"2024-09-21T10:19:13.201526+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:42:01.387622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":529,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.531139+0000","last_change":"2024-09-21T10:09:23.846703+0000","last_active":"2024-09-21T10:19:18.531139+0000","last_peered":"2024-09-21T10:19:18.531139+0000","last_clean":"2024-09-21T10:19:18.531139+0000","last_became_active":"2024-09-21T10:09:23.846415+0000","last_became_peered":"2024-09-21T10:09:23.846415+0000","last_unstale":"2024-09-21T10:19:18.531139+0000","last_undegraded":"2024-09-21T10:19:18.531139+0000","last_fullsized":"2024-09-21T10:19:18.531139+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:23.129285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'100","reported_seq":272,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.514744+0000","last_change":"2024-09-21T10:18:12.637168+0000","last_active":"2024-09-21T10:19:18.514744+0000","last_peered":"2024-09-21T10:19:18.514744+0000","last_clean":"2024-09-21T10:19:18.514744+0000","last_became_active":"2024-09-21T10:18:12.636410+0000","last_became_peered":"2024-09-21T10:18:12.636410+0000","last_unstale":"2024-09-21T10:19:18.514744+0000","last_undegraded":"2024-09-21T10:19:18.514744+0000","last_fullsized":"2024-09-21T10:19:18.514744+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.433227+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:40.015470+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":254,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.527565+0000","last_change":"2024-09-21T10:09:49.922489+0000","last_active":"2024-09-21T10:19:18.527565+0000","last_peered":"2024-09-21T10:19:18.527565+0000","last_clean":"2024-09-21T10:19:18.527565+0000","last_became_active":"2024-09-21T10:09:49.921629+0000","last_became_peered":"2024-09-21T10:09:49.921629+0000","last_unstale":"2024-09-21T10:19:18.527565+0000","last_undegraded":"2024-09-21T10:19:18.527565+0000","last_fullsized":"2024-09-21T10:19:18.527565+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:50.611979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":257,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.200623+0000","last_change":"2024-09-21T10:14:30.709707+0000","last_active":"2024-09-21T10:19:13.200623+0000","last_peered":"2024-09-21T10:19:13.200623+0000","last_clean":"2024-09-21T10:19:13.200623+0000","last_became_active":"2024-09-21T10:14:30.709353+0000","last_became_peered":"2024-09-21T10:14:30.709353+0000","last_unstale":"2024-09-21T10:19:13.200623+0000","last_undegraded":"2024-09-21T10:19:13.200623+0000","last_fullsized":"2024-09-21T10:19:13.200623+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:12.822947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"77'100","reported_seq":160,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.514555+0000","last_change":"2024-09-21T10:15:02.842044+0000","last_active":"2024-09-21T10:19:18.514555+0000","last_peered":"2024-09-21T10:19:18.514555+0000","last_clean":"2024-09-21T10:19:18.514555+0000","last_became_active":"2024-09-21T10:15:01.478273+0000","last_became_peered":"2024-09-21T10:15:01.478273+0000","last_unstale":"2024-09-21T10:19:18.514555+0000","last_undegraded":"2024-09-21T10:19:18.514555+0000","last_fullsized":"2024-09-21T10:19:18.514555+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:15:02.841968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:15:02.841968+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:58.075124+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":608,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.532429+0000","last_change":"2024-09-21T10:03:38.381552+0000","last_active":"2024-09-21T10:19:18.532429+0000","last_peered":"2024-09-21T10:19:18.532429+0000","last_clean":"2024-09-21T10:19:18.532429+0000","last_became_active":"2024-09-21T10:03:38.380898+0000","last_became_peered":"2024-09-21T10:03:38.380898+0000","last_unstale":"2024-09-21T10:19:18.532429+0000","last_undegraded":"2024-09-21T10:19:18.532429+0000","last_fullsized":"2024-09-21T10:19:18.532429+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:53.409606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":434,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.205574+0000","last_change":"2024-09-21T10:10:57.459057+0000","last_active":"2024-09-21T10:19:13.205574+0000","last_peered":"2024-09-21T10:19:13.205574+0000","last_clean":"2024-09-21T10:19:13.205574+0000","last_became_active":"2024-09-21T10:10:57.451776+0000","last_became_peered":"2024-09-21T10:10:57.451776+0000","last_unstale":"2024-09-21T10:19:13.205574+0000","last_undegraded":"2024-09-21T10:19:13.205574+0000","last_fullsized":"2024-09-21T10:19:13.205574+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:00.138317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"376'6436","reported_seq":6453,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:26.884400+0000","last_change":"2024-09-21T10:18:12.636805+0000","last_active":"2024-09-21T10:19:26.884400+0000","last_peered":"2024-09-21T10:19:26.884400+0000","last_clean":"2024-09-21T10:19:26.884400+0000","last_became_active":"2024-09-21T10:18:12.636542+0000","last_became_peered":"2024-09-21T10:18:12.636542+0000","last_unstale":"2024-09-21T10:19:26.884400+0000","last_undegraded":"2024-09-21T10:19:26.884400+0000","last_fullsized":"2024-09-21T10:19:26.884400+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6436,"log_dups_size":0,"ondisk_log_size":6436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:15:29.211883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421789696,"num_objects":6436,"num_object_clones":0,"num_object_copies":19308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6436,"num_write_kb":411904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"77'100","reported_seq":269,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.201483+0000","last_change":"2024-09-21T10:14:30.708275+0000","last_active":"2024-09-21T10:19:13.201483+0000","last_peered":"2024-09-21T10:19:13.201483+0000","last_clean":"2024-09-21T10:19:13.201483+0000","last_became_active":"2024-09-21T10:14:30.707763+0000","last_became_peered":"2024-09-21T10:14:30.707763+0000","last_unstale":"2024-09-21T10:19:13.201483+0000","last_undegraded":"2024-09-21T10:19:13.201483+0000","last_fullsized":"2024-09-21T10:19:13.201483+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:55.259385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:55.259385+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:41.322447+0000","scrub_duration":0.0015667260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":304,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.201803+0000","last_change":"2024-09-21T10:14:30.720250+0000","last_active":"2024-09-21T10:19:13.201803+0000","last_peered":"2024-09-21T10:19:13.201803+0000","last_clean":"2024-09-21T10:19:13.201803+0000","last_became_active":"2024-09-21T10:14:30.707036+0000","last_became_peered":"2024-09-21T10:14:30.707036+0000","last_unstale":"2024-09-21T10:19:13.201803+0000","last_undegraded":"2024-09-21T10:19:13.201803+0000","last_fullsized":"2024-09-21T10:19:13.201803+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:17.438534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":347,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.200540+0000","last_change":"2024-09-21T10:15:01.476521+0000","last_active":"2024-09-21T10:19:13.200540+0000","last_peered":"2024-09-21T10:19:13.200540+0000","last_clean":"2024-09-21T10:19:13.200540+0000","last_became_active":"2024-09-21T10:15:01.476083+0000","last_became_peered":"2024-09-21T10:15:01.476083+0000","last_unstale":"2024-09-21T10:19:13.200540+0000","last_undegraded":"2024-09-21T10:19:13.200540+0000","last_fullsized":"2024-09-21T10:19:13.200540+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:34:50.230443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.0","version":"376'6589","reported_seq":6606,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:27.006388+0000","last_change":"2024-09-21T10:18:12.633194+0000","last_active":"2024-09-21T10:19:27.006388+0000","last_peered":"2024-09-21T10:19:27.006388+0000","last_clean":"2024-09-21T10:19:27.006388+0000","last_became_active":"2024-09-21T10:18:12.632915+0000","last_became_peered":"2024-09-21T10:18:12.632915+0000","last_unstale":"2024-09-21T10:19:27.006388+0000","last_undegraded":"2024-09-21T10:19:27.006388+0000","last_fullsized":"2024-09-21T10:19:27.006388+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6589,"log_dups_size":0,"ondisk_log_size":6589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:13:26.865830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431816704,"num_objects":6589,"num_object_clones":0,"num_object_copies":19767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6589,"num_write_kb":421696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"77'100","reported_seq":416,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.532033+0000","last_change":"2024-09-21T10:09:23.839211+0000","last_active":"2024-09-21T10:19:18.532033+0000","last_peered":"2024-09-21T10:19:18.532033+0000","last_clean":"2024-09-21T10:19:18.532033+0000","last_became_active":"2024-09-21T10:09:23.838779+0000","last_became_peered":"2024-09-21T10:09:23.838779+0000","last_unstale":"2024-09-21T10:19:18.532033+0000","last_undegraded":"2024-09-21T10:19:18.532033+0000","last_fullsized":"2024-09-21T10:19:18.532033+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.981336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.981336+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:32:41.233787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":283,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.201746+0000","last_change":"2024-09-21T10:18:12.643154+0000","last_active":"2024-09-21T10:19:13.201746+0000","last_peered":"2024-09-21T10:19:13.201746+0000","last_clean":"2024-09-21T10:19:13.201746+0000","last_became_active":"2024-09-21T10:18:12.636287+0000","last_became_peered":"2024-09-21T10:18:12.636287+0000","last_unstale":"2024-09-21T10:19:13.201746+0000","last_undegraded":"2024-09-21T10:19:13.201746+0000","last_fullsized":"2024-09-21T10:19:13.201746+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:45:31.008309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":82,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.199701+0000","last_change":"2024-09-21T10:14:30.706497+0000","last_active":"2024-09-21T10:19:13.199701+0000","last_peered":"2024-09-21T10:19:13.199701+0000","last_clean":"2024-09-21T10:19:13.199701+0000","last_became_active":"2024-09-21T10:14:30.706202+0000","last_became_peered":"2024-09-21T10:14:30.706202+0000","last_unstale":"2024-09-21T10:19:13.199701+0000","last_undegraded":"2024-09-21T10:19:13.199701+0000","last_fullsized":"2024-09-21T10:19:13.199701+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:42:57.912707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3","version":"376'6682","reported_seq":6711,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:27.545550+0000","last_change":"2024-09-21T10:18:12.641462+0000","last_active":"2024-09-21T10:19:27.545550+0000","last_peered":"2024-09-21T10:19:27.545550+0000","last_clean":"2024-09-21T10:19:27.545550+0000","last_became_active":"2024-09-21T10:18:12.641056+0000","last_became_peered":"2024-09-21T10:18:12.641056+0000","last_unstale":"2024-09-21T10:19:27.545550+0000","last_undegraded":"2024-09-21T10:19:27.545550+0000","last_fullsized":"2024-09-21T10:19:27.545550+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6682,"log_dups_size":0,"ondisk_log_size":6682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:49:09.395317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437911552,"num_objects":6682,"num_object_clones":0,"num_object_copies":20046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6682,"num_write_kb":427648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"77'100","reported_seq":369,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.207651+0000","last_change":"2024-09-21T10:14:30.709997+0000","last_active":"2024-09-21T10:19:13.207651+0000","last_peered":"2024-09-21T10:19:13.207651+0000","last_clean":"2024-09-21T10:19:13.207651+0000","last_became_active":"2024-09-21T10:14:30.709580+0000","last_became_peered":"2024-09-21T10:14:30.709580+0000","last_unstale":"2024-09-21T10:19:13.207651+0000","last_undegraded":"2024-09-21T10:19:13.207651+0000","last_fullsized":"2024-09-21T10:19:13.207651+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.289859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.289859+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:27:50.110602+0000","scrub_duration":0.001251809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":255,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.201329+0000","last_change":"2024-09-21T10:10:57.460732+0000","last_active":"2024-09-21T10:19:13.201329+0000","last_peered":"2024-09-21T10:19:13.201329+0000","last_clean":"2024-09-21T10:19:13.201329+0000","last_became_active":"2024-09-21T10:10:57.454689+0000","last_became_peered":"2024-09-21T10:10:57.454689+0000","last_unstale":"2024-09-21T10:19:13.201329+0000","last_undegraded":"2024-09-21T10:19:13.201329+0000","last_fullsized":"2024-09-21T10:19:13.201329+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:12.035022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":375,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.205650+0000","last_change":"2024-09-21T10:03:38.378384+0000","last_active":"2024-09-21T10:19:13.205650+0000","last_peered":"2024-09-21T10:19:13.205650+0000","last_clean":"2024-09-21T10:19:13.205650+0000","last_became_active":"2024-09-21T10:03:38.378072+0000","last_became_peered":"2024-09-21T10:03:38.378072+0000","last_unstale":"2024-09-21T10:19:13.205650+0000","last_undegraded":"2024-09-21T10:19:13.205650+0000","last_fullsized":"2024-09-21T10:19:13.205650+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:54:37.039622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"376'6406","reported_seq":6435,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:27.420967+0000","last_change":"2024-09-21T10:18:13.037514+0000","last_active":"2024-09-21T10:19:27.420967+0000","last_peered":"2024-09-21T10:19:27.420967+0000","last_clean":"2024-09-21T10:19:27.420967+0000","last_became_active":"2024-09-21T10:18:13.037222+0000","last_became_peered":"2024-09-21T10:18:13.037222+0000","last_unstale":"2024-09-21T10:19:27.420967+0000","last_undegraded":"2024-09-21T10:19:27.420967+0000","last_fullsized":"2024-09-21T10:19:27.420967+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6406,"log_dups_size":0,"ondisk_log_size":6406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:00:30.961667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419823616,"num_objects":6406,"num_object_clones":0,"num_object_copies":19218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6406,"num_write_kb":409984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"77'100","reported_seq":28,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.199975+0000","last_change":"2024-09-21T10:18:27.085659+0000","last_active":"2024-09-21T10:19:13.199975+0000","last_peered":"2024-09-21T10:19:13.199975+0000","last_clean":"2024-09-21T10:19:13.199975+0000","last_became_active":"2024-09-21T10:18:12.633342+0000","last_became_peered":"2024-09-21T10:18:12.633342+0000","last_unstale":"2024-09-21T10:19:13.199975+0000","last_undegraded":"2024-09-21T10:19:13.199975+0000","last_fullsized":"2024-09-21T10:19:13.199975+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:18:27.085525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:18:27.085525+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:06:53.238321+0000","scrub_duration":0.0036485279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":413,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.532291+0000","last_change":"2024-09-21T10:07:57.551556+0000","last_active":"2024-09-21T10:19:18.532291+0000","last_peered":"2024-09-21T10:19:18.532291+0000","last_clean":"2024-09-21T10:19:18.532291+0000","last_became_active":"2024-09-21T10:07:57.551249+0000","last_became_peered":"2024-09-21T10:07:57.551249+0000","last_unstale":"2024-09-21T10:19:18.532291+0000","last_undegraded":"2024-09-21T10:19:18.532291+0000","last_fullsized":"2024-09-21T10:19:18.532291+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:57.557152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":516,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.529609+0000","last_change":"2024-09-21T10:14:30.711105+0000","last_active":"2024-09-21T10:19:18.529609+0000","last_peered":"2024-09-21T10:19:18.529609+0000","last_clean":"2024-09-21T10:19:18.529609+0000","last_became_active":"2024-09-21T10:14:30.710759+0000","last_became_peered":"2024-09-21T10:14:30.710759+0000","last_unstale":"2024-09-21T10:19:18.529609+0000","last_undegraded":"2024-09-21T10:19:18.529609+0000","last_fullsized":"2024-09-21T10:19:18.529609+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:40.690042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"376'6407","reported_seq":6424,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:26.886585+0000","last_change":"2024-09-21T10:18:13.037771+0000","last_active":"2024-09-21T10:19:26.886585+0000","last_peered":"2024-09-21T10:19:26.886585+0000","last_clean":"2024-09-21T10:19:26.886585+0000","last_became_active":"2024-09-21T10:18:13.037425+0000","last_became_peered":"2024-09-21T10:18:13.037425+0000","last_unstale":"2024-09-21T10:19:26.886585+0000","last_undegraded":"2024-09-21T10:19:26.886585+0000","last_fullsized":"2024-09-21T10:19:26.886585+0000","mapping_epoch":367,"log_start":"371'400","ondisk_log_start":"371'400","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6007,"log_dups_size":0,"ondisk_log_size":6007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:34:43.436187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419889152,"num_objects":6407,"num_object_clones":0,"num_object_copies":19221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6407,"num_write_kb":410048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":249,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.200787+0000","last_change":"2024-09-21T10:04:53.355975+0000","last_active":"2024-09-21T10:19:13.200787+0000","last_peered":"2024-09-21T10:19:13.200787+0000","last_clean":"2024-09-21T10:19:13.200787+0000","last_became_active":"2024-09-21T10:04:42.547505+0000","last_became_peered":"2024-09-21T10:04:42.547505+0000","last_unstale":"2024-09-21T10:19:13.200787+0000","last_undegraded":"2024-09-21T10:19:13.200787+0000","last_fullsized":"2024-09-21T10:19:13.200787+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.355912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.355912+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:43.456830+0000","scrub_duration":0.0017980030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":620,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.533719+0000","last_change":"2024-09-21T10:14:30.715303+0000","last_active":"2024-09-21T10:19:18.533719+0000","last_peered":"2024-09-21T10:19:18.533719+0000","last_clean":"2024-09-21T10:19:18.533719+0000","last_became_active":"2024-09-21T10:14:30.703664+0000","last_became_peered":"2024-09-21T10:14:30.703664+0000","last_unstale":"2024-09-21T10:19:18.533719+0000","last_undegraded":"2024-09-21T10:19:18.533719+0000","last_fullsized":"2024-09-21T10:19:18.533719+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:28.967906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":82,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.199921+0000","last_change":"2024-09-21T10:14:30.705927+0000","last_active":"2024-09-21T10:19:13.199921+0000","last_peered":"2024-09-21T10:19:13.199921+0000","last_clean":"2024-09-21T10:19:13.199921+0000","last_became_active":"2024-09-21T10:14:30.705112+0000","last_became_peered":"2024-09-21T10:14:30.705112+0000","last_unstale":"2024-09-21T10:19:13.199921+0000","last_undegraded":"2024-09-21T10:19:13.199921+0000","last_fullsized":"2024-09-21T10:19:13.199921+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:38:33.962199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.4","version":"376'6488","reported_seq":6504,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:26.883941+0000","last_change":"2024-09-21T10:18:12.639080+0000","last_active":"2024-09-21T10:19:26.883941+0000","last_peered":"2024-09-21T10:19:26.883941+0000","last_clean":"2024-09-21T10:19:26.883941+0000","last_became_active":"2024-09-21T10:18:12.638789+0000","last_became_peered":"2024-09-21T10:18:12.638789+0000","last_unstale":"2024-09-21T10:19:26.883941+0000","last_undegraded":"2024-09-21T10:19:26.883941+0000","last_fullsized":"2024-09-21T10:19:26.883941+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6488,"log_dups_size":0,"ondisk_log_size":6488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:53.178338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425197568,"num_objects":6488,"num_object_clones":0,"num_object_copies":19464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6488,"num_write_kb":415232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":370,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.529646+0000","last_change":"2024-09-21T10:10:57.403722+0000","last_active":"2024-09-21T10:19:18.529646+0000","last_peered":"2024-09-21T10:19:18.529646+0000","last_clean":"2024-09-21T10:19:18.529646+0000","last_became_active":"2024-09-21T10:10:57.394700+0000","last_became_peered":"2024-09-21T10:10:57.394700+0000","last_unstale":"2024-09-21T10:19:18.529646+0000","last_undegraded":"2024-09-21T10:19:18.529646+0000","last_fullsized":"2024-09-21T10:19:18.529646+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:02.152981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:02.152981+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:45:33.964266+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":359,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.205736+0000","last_change":"2024-09-21T10:14:30.707046+0000","last_active":"2024-09-21T10:19:13.205736+0000","last_peered":"2024-09-21T10:19:13.205736+0000","last_clean":"2024-09-21T10:19:13.205736+0000","last_became_active":"2024-09-21T10:14:30.706753+0000","last_became_peered":"2024-09-21T10:14:30.706753+0000","last_unstale":"2024-09-21T10:19:13.205736+0000","last_undegraded":"2024-09-21T10:19:13.205736+0000","last_fullsized":"2024-09-21T10:19:13.205736+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:58.136096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.199914+0000","last_change":"2024-09-21T10:18:13.037945+0000","last_active":"2024-09-21T10:19:13.199914+0000","last_peered":"2024-09-21T10:19:13.199914+0000","last_clean":"2024-09-21T10:19:13.199914+0000","last_became_active":"2024-09-21T10:18:13.037550+0000","last_became_peered":"2024-09-21T10:18:13.037550+0000","last_unstale":"2024-09-21T10:19:13.199914+0000","last_undegraded":"2024-09-21T10:19:13.199914+0000","last_fullsized":"2024-09-21T10:19:13.199914+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:24:24.702636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.7","version":"376'6419","reported_seq":6435,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:27.030904+0000","last_change":"2024-09-21T10:18:12.635921+0000","last_active":"2024-09-21T10:19:27.030904+0000","last_peered":"2024-09-21T10:19:27.030904+0000","last_clean":"2024-09-21T10:19:27.030904+0000","last_became_active":"2024-09-21T10:18:12.635624+0000","last_became_peered":"2024-09-21T10:18:12.635624+0000","last_unstale":"2024-09-21T10:19:27.030904+0000","last_undegraded":"2024-09-21T10:19:27.030904+0000","last_fullsized":"2024-09-21T10:19:27.030904+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6419,"log_dups_size":0,"ondisk_log_size":6419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:08:14.392867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420675584,"num_objects":6419,"num_object_clones":0,"num_object_copies":19257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6419,"num_write_kb":410816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"77'100","reported_seq":262,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.514869+0000","last_change":"2024-09-21T10:14:05.474100+0000","last_active":"2024-09-21T10:19:18.514869+0000","last_peered":"2024-09-21T10:19:18.514869+0000","last_clean":"2024-09-21T10:19:18.514869+0000","last_became_active":"2024-09-21T10:14:05.473336+0000","last_became_peered":"2024-09-21T10:14:05.473336+0000","last_unstale":"2024-09-21T10:19:18.514869+0000","last_undegraded":"2024-09-21T10:19:18.514869+0000","last_fullsized":"2024-09-21T10:19:18.514869+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.540323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.540323+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:26:21.305903+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.199501+0000","last_change":"2024-09-21T10:18:12.629311+0000","last_active":"2024-09-21T10:19:13.199501+0000","last_peered":"2024-09-21T10:19:13.199501+0000","last_clean":"2024-09-21T10:19:13.199501+0000","last_became_active":"2024-09-21T10:18:12.629001+0000","last_became_peered":"2024-09-21T10:18:12.629001+0000","last_unstale":"2024-09-21T10:19:13.199501+0000","last_undegraded":"2024-09-21T10:19:13.199501+0000","last_fullsized":"2024-09-21T10:19:13.199501+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:09:17.165751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":82,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.200134+0000","last_change":"2024-09-21T10:14:30.706905+0000","last_active":"2024-09-21T10:19:13.200134+0000","last_peered":"2024-09-21T10:19:13.200134+0000","last_clean":"2024-09-21T10:19:13.200134+0000","last_became_active":"2024-09-21T10:14:30.706614+0000","last_became_peered":"2024-09-21T10:14:30.706614+0000","last_unstale":"2024-09-21T10:19:13.200134+0000","last_undegraded":"2024-09-21T10:19:13.200134+0000","last_fullsized":"2024-09-21T10:19:13.200134+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:26:33.014993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.6","version":"376'6384","reported_seq":6400,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:26.918719+0000","last_change":"2024-09-21T10:18:13.038010+0000","last_active":"2024-09-21T10:19:26.918719+0000","last_peered":"2024-09-21T10:19:26.918719+0000","last_clean":"2024-09-21T10:19:26.918719+0000","last_became_active":"2024-09-21T10:18:13.037700+0000","last_became_peered":"2024-09-21T10:18:13.037700+0000","last_unstale":"2024-09-21T10:19:26.918719+0000","last_undegraded":"2024-09-21T10:19:26.918719+0000","last_fullsized":"2024-09-21T10:19:26.918719+0000","mapping_epoch":367,"log_start":"371'300","ondisk_log_start":"371'300","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6084,"log_dups_size":0,"ondisk_log_size":6084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:14:58.367606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418381824,"num_objects":6384,"num_object_clones":0,"num_object_copies":19152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6384,"num_write_kb":408576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":333,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.530213+0000","last_change":"2024-09-21T10:08:04.645700+0000","last_active":"2024-09-21T10:19:18.530213+0000","last_peered":"2024-09-21T10:19:18.530213+0000","last_clean":"2024-09-21T10:19:18.530213+0000","last_became_active":"2024-09-21T10:07:57.548189+0000","last_became_peered":"2024-09-21T10:07:57.548189+0000","last_unstale":"2024-09-21T10:19:18.530213+0000","last_undegraded":"2024-09-21T10:19:18.530213+0000","last_fullsized":"2024-09-21T10:19:18.530213+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:48.318010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:48.318010+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:20:14.081477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":107339,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":610,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.530977+0000","last_change":"2024-09-21T10:03:38.386604+0000","last_active":"2024-09-21T10:19:18.530977+0000","last_peered":"2024-09-21T10:19:18.530977+0000","last_clean":"2024-09-21T10:19:18.530977+0000","last_became_active":"2024-09-21T10:03:38.386283+0000","last_became_peered":"2024-09-21T10:03:38.386283+0000","last_unstale":"2024-09-21T10:19:18.530977+0000","last_undegraded":"2024-09-21T10:19:18.530977+0000","last_fullsized":"2024-09-21T10:19:18.530977+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:27:05.944880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":469,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.515781+0000","last_change":"2024-09-21T10:14:30.709615+0000","last_active":"2024-09-21T10:19:18.515781+0000","last_peered":"2024-09-21T10:19:18.515781+0000","last_clean":"2024-09-21T10:19:18.515781+0000","last_became_active":"2024-09-21T10:14:30.709184+0000","last_became_peered":"2024-09-21T10:14:30.709184+0000","last_unstale":"2024-09-21T10:19:18.515781+0000","last_undegraded":"2024-09-21T10:19:18.515781+0000","last_fullsized":"2024-09-21T10:19:18.515781+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:55:27.220232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.c","version":"376'6274","reported_seq":6303,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:25.702287+0000","last_change":"2024-09-21T10:18:13.037466+0000","last_active":"2024-09-21T10:19:25.702287+0000","last_peered":"2024-09-21T10:19:25.702287+0000","last_clean":"2024-09-21T10:19:25.702287+0000","last_became_active":"2024-09-21T10:18:13.037151+0000","last_became_peered":"2024-09-21T10:18:13.037151+0000","last_unstale":"2024-09-21T10:19:25.702287+0000","last_undegraded":"2024-09-21T10:19:25.702287+0000","last_fullsized":"2024-09-21T10:19:25.702287+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6274,"log_dups_size":0,"ondisk_log_size":6274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:35:23.611663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411172864,"num_objects":6274,"num_object_clones":0,"num_object_copies":18822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6274,"num_write_kb":401536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":266,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.200591+0000","last_change":"2024-09-21T10:09:49.924286+0000","last_active":"2024-09-21T10:19:13.200591+0000","last_peered":"2024-09-21T10:19:13.200591+0000","last_clean":"2024-09-21T10:19:13.200591+0000","last_became_active":"2024-09-21T10:09:49.923897+0000","last_became_peered":"2024-09-21T10:09:49.923897+0000","last_unstale":"2024-09-21T10:19:13.200591+0000","last_undegraded":"2024-09-21T10:19:13.200591+0000","last_fullsized":"2024-09-21T10:19:13.200591+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:47.393146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:47.393146+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:44.898083+0000","scrub_duration":0.001601086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":365,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.525771+0000","last_change":"2024-09-21T10:10:57.402946+0000","last_active":"2024-09-21T10:19:18.525771+0000","last_peered":"2024-09-21T10:19:18.525771+0000","last_clean":"2024-09-21T10:19:18.525771+0000","last_became_active":"2024-09-21T10:10:57.395450+0000","last_became_peered":"2024-09-21T10:10:57.395450+0000","last_unstale":"2024-09-21T10:19:18.525771+0000","last_undegraded":"2024-09-21T10:19:18.525771+0000","last_fullsized":"2024-09-21T10:19:18.525771+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:08.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":595,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.532540+0000","last_change":"2024-09-21T10:10:57.390595+0000","last_active":"2024-09-21T10:19:18.532540+0000","last_peered":"2024-09-21T10:19:18.532540+0000","last_clean":"2024-09-21T10:19:18.532540+0000","last_became_active":"2024-09-21T10:10:57.390265+0000","last_became_peered":"2024-09-21T10:10:57.390265+0000","last_unstale":"2024-09-21T10:19:18.532540+0000","last_undegraded":"2024-09-21T10:19:18.532540+0000","last_fullsized":"2024-09-21T10:19:18.532540+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:37.485063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":392,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.205842+0000","last_change":"2024-09-21T10:14:30.711719+0000","last_active":"2024-09-21T10:19:13.205842+0000","last_peered":"2024-09-21T10:19:13.205842+0000","last_clean":"2024-09-21T10:19:13.205842+0000","last_became_active":"2024-09-21T10:14:30.711452+0000","last_became_peered":"2024-09-21T10:14:30.711452+0000","last_unstale":"2024-09-21T10:19:13.205842+0000","last_undegraded":"2024-09-21T10:19:13.205842+0000","last_fullsized":"2024-09-21T10:19:13.205842+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:46.907528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"376'6460","reported_seq":6476,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:26.880751+0000","last_change":"2024-09-21T10:18:12.639005+0000","last_active":"2024-09-21T10:19:26.880751+0000","last_peered":"2024-09-21T10:19:26.880751+0000","last_clean":"2024-09-21T10:19:26.880751+0000","last_became_active":"2024-09-21T10:18:12.638716+0000","last_became_peered":"2024-09-21T10:18:12.638716+0000","last_unstale":"2024-09-21T10:19:26.880751+0000","last_undegraded":"2024-09-21T10:19:26.880751+0000","last_fullsized":"2024-09-21T10:19:26.880751+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6460,"log_dups_size":0,"ondisk_log_size":6460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:08.923853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423362560,"num_objects":6460,"num_object_clones":0,"num_object_copies":19380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6460,"num_write_kb":413440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":272,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.515093+0000","last_change":"2024-09-21T10:14:30.701644+0000","last_active":"2024-09-21T10:19:18.515093+0000","last_peered":"2024-09-21T10:19:18.515093+0000","last_clean":"2024-09-21T10:19:18.515093+0000","last_became_active":"2024-09-21T10:14:30.701301+0000","last_became_peered":"2024-09-21T10:14:30.701301+0000","last_unstale":"2024-09-21T10:19:18.515093+0000","last_undegraded":"2024-09-21T10:19:18.515093+0000","last_fullsized":"2024-09-21T10:19:18.515093+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.550551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.550551+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:11:15.291805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":240,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.200685+0000","last_change":"2024-09-21T10:09:23.838944+0000","last_active":"2024-09-21T10:19:13.200685+0000","last_peered":"2024-09-21T10:19:13.200685+0000","last_clean":"2024-09-21T10:19:13.200685+0000","last_became_active":"2024-09-21T10:09:23.837886+0000","last_became_peered":"2024-09-21T10:09:23.837886+0000","last_unstale":"2024-09-21T10:19:13.200685+0000","last_undegraded":"2024-09-21T10:19:13.200685+0000","last_fullsized":"2024-09-21T10:19:13.200685+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:16.928195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":493,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.528749+0000","last_change":"2024-09-21T10:18:12.637103+0000","last_active":"2024-09-21T10:19:18.528749+0000","last_peered":"2024-09-21T10:19:18.528749+0000","last_clean":"2024-09-21T10:19:18.528749+0000","last_became_active":"2024-09-21T10:18:12.636645+0000","last_became_peered":"2024-09-21T10:18:12.636645+0000","last_unstale":"2024-09-21T10:19:18.528749+0000","last_undegraded":"2024-09-21T10:19:18.528749+0000","last_fullsized":"2024-09-21T10:19:18.528749+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:53:50.745484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":255,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.527479+0000","last_change":"2024-09-21T10:09:49.923044+0000","last_active":"2024-09-21T10:19:18.527479+0000","last_peered":"2024-09-21T10:19:18.527479+0000","last_clean":"2024-09-21T10:19:18.527479+0000","last_became_active":"2024-09-21T10:09:49.922711+0000","last_became_peered":"2024-09-21T10:09:49.922711+0000","last_unstale":"2024-09-21T10:19:18.527479+0000","last_undegraded":"2024-09-21T10:19:18.527479+0000","last_fullsized":"2024-09-21T10:19:18.527479+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:14.090084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"376'6300","reported_seq":6317,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:27.001389+0000","last_change":"2024-09-21T10:18:13.038029+0000","last_active":"2024-09-21T10:19:27.001389+0000","last_peered":"2024-09-21T10:19:27.001389+0000","last_clean":"2024-09-21T10:19:27.001389+0000","last_became_active":"2024-09-21T10:18:13.037584+0000","last_became_peered":"2024-09-21T10:18:13.037584+0000","last_unstale":"2024-09-21T10:19:27.001389+0000","last_undegraded":"2024-09-21T10:19:27.001389+0000","last_fullsized":"2024-09-21T10:19:27.001389+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6300,"log_dups_size":0,"ondisk_log_size":6300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:18.430382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412876800,"num_objects":6300,"num_object_clones":0,"num_object_copies":18900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6300,"num_write_kb":403200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":290,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.200946+0000","last_change":"2024-09-21T10:18:12.636851+0000","last_active":"2024-09-21T10:19:13.200946+0000","last_peered":"2024-09-21T10:19:13.200946+0000","last_clean":"2024-09-21T10:19:13.200946+0000","last_became_active":"2024-09-21T10:18:12.636513+0000","last_became_peered":"2024-09-21T10:18:12.636513+0000","last_unstale":"2024-09-21T10:19:13.200946+0000","last_undegraded":"2024-09-21T10:19:13.200946+0000","last_fullsized":"2024-09-21T10:19:13.200946+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:52.361137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:52.361137+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:21.025735+0000","scrub_duration":0.001255358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":648,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.529762+0000","last_change":"2024-09-21T10:09:23.837859+0000","last_active":"2024-09-21T10:19:18.529762+0000","last_peered":"2024-09-21T10:19:18.529762+0000","last_clean":"2024-09-21T10:19:18.529762+0000","last_became_active":"2024-09-21T10:09:23.837488+0000","last_became_peered":"2024-09-21T10:09:23.837488+0000","last_unstale":"2024-09-21T10:19:18.529762+0000","last_undegraded":"2024-09-21T10:19:18.529762+0000","last_fullsized":"2024-09-21T10:19:18.529762+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:25.215501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":124,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.200580+0000","last_change":"2024-09-21T10:10:27.806070+0000","last_active":"2024-09-21T10:19:13.200580+0000","last_peered":"2024-09-21T10:19:13.200580+0000","last_clean":"2024-09-21T10:19:13.200580+0000","last_became_active":"2024-09-21T10:10:27.805718+0000","last_became_peered":"2024-09-21T10:10:27.805718+0000","last_unstale":"2024-09-21T10:19:13.200580+0000","last_undegraded":"2024-09-21T10:19:13.200580+0000","last_fullsized":"2024-09-21T10:19:13.200580+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:31.144118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":641,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.526257+0000","last_change":"2024-09-21T10:10:57.395073+0000","last_active":"2024-09-21T10:19:18.526257+0000","last_peered":"2024-09-21T10:19:18.526257+0000","last_clean":"2024-09-21T10:19:18.526257+0000","last_became_active":"2024-09-21T10:10:57.394674+0000","last_became_peered":"2024-09-21T10:10:57.394674+0000","last_unstale":"2024-09-21T10:19:18.526257+0000","last_undegraded":"2024-09-21T10:19:18.526257+0000","last_fullsized":"2024-09-21T10:19:18.526257+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:48:48.912649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.d","version":"376'6450","reported_seq":6466,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:26.920303+0000","last_change":"2024-09-21T10:18:12.632980+0000","last_active":"2024-09-21T10:19:26.920303+0000","last_peered":"2024-09-21T10:19:26.920303+0000","last_clean":"2024-09-21T10:19:26.920303+0000","last_became_active":"2024-09-21T10:18:12.632684+0000","last_became_peered":"2024-09-21T10:18:12.632684+0000","last_unstale":"2024-09-21T10:19:26.920303+0000","last_undegraded":"2024-09-21T10:19:26.920303+0000","last_fullsized":"2024-09-21T10:19:26.920303+0000","mapping_epoch":367,"log_start":"371'400","ondisk_log_start":"371'400","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6050,"log_dups_size":0,"ondisk_log_size":6050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:08.974530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422707200,"num_objects":6450,"num_object_clones":0,"num_object_copies":19350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6450,"num_write_kb":412800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":511,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.528872+0000","last_change":"2024-09-21T10:10:57.398437+0000","last_active":"2024-09-21T10:19:18.528872+0000","last_peered":"2024-09-21T10:19:18.528872+0000","last_clean":"2024-09-21T10:19:18.528872+0000","last_became_active":"2024-09-21T10:10:57.397441+0000","last_became_peered":"2024-09-21T10:10:57.397441+0000","last_unstale":"2024-09-21T10:19:18.528872+0000","last_undegraded":"2024-09-21T10:19:18.528872+0000","last_fullsized":"2024-09-21T10:19:18.528872+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:04.043917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:04.043917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:59.965868+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":203,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.514225+0000","last_change":"2024-09-21T10:14:05.472790+0000","last_active":"2024-09-21T10:19:18.514225+0000","last_peered":"2024-09-21T10:19:18.514225+0000","last_clean":"2024-09-21T10:19:18.514225+0000","last_became_active":"2024-09-21T10:14:05.472238+0000","last_became_peered":"2024-09-21T10:14:05.472238+0000","last_unstale":"2024-09-21T10:19:18.514225+0000","last_undegraded":"2024-09-21T10:19:18.514225+0000","last_fullsized":"2024-09-21T10:19:18.514225+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:14:56.561032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":82,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.199926+0000","last_change":"2024-09-21T10:14:30.709652+0000","last_active":"2024-09-21T10:19:13.199926+0000","last_peered":"2024-09-21T10:19:13.199926+0000","last_clean":"2024-09-21T10:19:13.199926+0000","last_became_active":"2024-09-21T10:14:30.705706+0000","last_became_peered":"2024-09-21T10:14:30.705706+0000","last_unstale":"2024-09-21T10:19:13.199926+0000","last_undegraded":"2024-09-21T10:19:13.199926+0000","last_fullsized":"2024-09-21T10:19:13.199926+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:24:25.749923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":474,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.519766+0000","last_change":"2024-09-21T10:14:05.477875+0000","last_active":"2024-09-21T10:19:18.519766+0000","last_peered":"2024-09-21T10:19:18.519766+0000","last_clean":"2024-09-21T10:19:18.519766+0000","last_became_active":"2024-09-21T10:14:05.477192+0000","last_became_peered":"2024-09-21T10:14:05.477192+0000","last_unstale":"2024-09-21T10:19:18.519766+0000","last_undegraded":"2024-09-21T10:19:18.519766+0000","last_fullsized":"2024-09-21T10:19:18.519766+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:19:47.345673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.b","version":"376'6472","reported_seq":6501,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:27.370325+0000","last_change":"2024-09-21T10:18:13.039421+0000","last_active":"2024-09-21T10:19:27.370325+0000","last_peered":"2024-09-21T10:19:27.370325+0000","last_clean":"2024-09-21T10:19:27.370325+0000","last_became_active":"2024-09-21T10:18:13.039085+0000","last_became_peered":"2024-09-21T10:18:13.039085+0000","last_unstale":"2024-09-21T10:19:27.370325+0000","last_undegraded":"2024-09-21T10:19:27.370325+0000","last_fullsized":"2024-09-21T10:19:27.370325+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6472,"log_dups_size":0,"ondisk_log_size":6472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:36.052065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424148992,"num_objects":6472,"num_object_clones":0,"num_object_copies":19416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6472,"num_write_kb":414208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":349,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.207559+0000","last_change":"2024-09-21T10:04:45.242148+0000","last_active":"2024-09-21T10:19:13.207559+0000","last_peered":"2024-09-21T10:19:13.207559+0000","last_clean":"2024-09-21T10:19:13.207559+0000","last_became_active":"2024-09-21T10:04:42.549844+0000","last_became_peered":"2024-09-21T10:04:42.549844+0000","last_unstale":"2024-09-21T10:19:13.207559+0000","last_undegraded":"2024-09-21T10:19:13.207559+0000","last_fullsized":"2024-09-21T10:19:13.207559+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_deep_scrub":"77'100","last_deep_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.242073+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:03:42.250480+0000","scrub_duration":0.0023466239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":317,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.532219+0000","last_change":"2024-09-21T10:15:01.489326+0000","last_active":"2024-09-21T10:19:18.532219+0000","last_peered":"2024-09-21T10:19:18.532219+0000","last_clean":"2024-09-21T10:19:18.532219+0000","last_became_active":"2024-09-21T10:15:01.489070+0000","last_became_peered":"2024-09-21T10:15:01.489070+0000","last_unstale":"2024-09-21T10:19:18.532219+0000","last_undegraded":"2024-09-21T10:19:18.532219+0000","last_fullsized":"2024-09-21T10:19:18.532219+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:27:41.788029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":530,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.520917+0000","last_change":"2024-09-21T10:14:05.475970+0000","last_active":"2024-09-21T10:19:18.520917+0000","last_peered":"2024-09-21T10:19:18.520917+0000","last_clean":"2024-09-21T10:19:18.520917+0000","last_became_active":"2024-09-21T10:14:05.475331+0000","last_became_peered":"2024-09-21T10:14:05.475331+0000","last_unstale":"2024-09-21T10:19:18.520917+0000","last_undegraded":"2024-09-21T10:19:18.520917+0000","last_fullsized":"2024-09-21T10:19:18.520917+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:17.069736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":466,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.514637+0000","last_change":"2024-09-21T10:18:13.039172+0000","last_active":"2024-09-21T10:19:18.514637+0000","last_peered":"2024-09-21T10:19:18.514637+0000","last_clean":"2024-09-21T10:19:18.514637+0000","last_became_active":"2024-09-21T10:18:13.038756+0000","last_became_peered":"2024-09-21T10:18:13.038756+0000","last_unstale":"2024-09-21T10:19:18.514637+0000","last_undegraded":"2024-09-21T10:19:18.514637+0000","last_fullsized":"2024-09-21T10:19:18.514637+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:06.668926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.a","version":"376'6281","reported_seq":6309,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:24.824466+0000","last_change":"2024-09-21T10:18:12.633418+0000","last_active":"2024-09-21T10:19:24.824466+0000","last_peered":"2024-09-21T10:19:24.824466+0000","last_clean":"2024-09-21T10:19:24.824466+0000","last_became_active":"2024-09-21T10:18:12.633151+0000","last_became_peered":"2024-09-21T10:18:12.633151+0000","last_unstale":"2024-09-21T10:19:24.824466+0000","last_undegraded":"2024-09-21T10:19:24.824466+0000","last_fullsized":"2024-09-21T10:19:24.824466+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6281,"log_dups_size":0,"ondisk_log_size":6281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:07:26.685329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411631616,"num_objects":6281,"num_object_clones":0,"num_object_copies":18843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6281,"num_write_kb":401984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":366,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.206036+0000","last_change":"2024-09-21T10:09:23.839563+0000","last_active":"2024-09-21T10:19:13.206036+0000","last_peered":"2024-09-21T10:19:13.206036+0000","last_clean":"2024-09-21T10:19:13.206036+0000","last_became_active":"2024-09-21T10:09:23.839207+0000","last_became_peered":"2024-09-21T10:09:23.839207+0000","last_unstale":"2024-09-21T10:19:13.206036+0000","last_undegraded":"2024-09-21T10:19:13.206036+0000","last_fullsized":"2024-09-21T10:19:13.206036+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:49.226496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:49.226496+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:38:03.960125+0000","scrub_duration":0.001656865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":462,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.520517+0000","last_change":"2024-09-21T10:14:05.478152+0000","last_active":"2024-09-21T10:19:18.520517+0000","last_peered":"2024-09-21T10:19:18.520517+0000","last_clean":"2024-09-21T10:19:18.520517+0000","last_became_active":"2024-09-21T10:14:05.477547+0000","last_became_peered":"2024-09-21T10:14:05.477547+0000","last_unstale":"2024-09-21T10:19:18.520517+0000","last_undegraded":"2024-09-21T10:19:18.520517+0000","last_fullsized":"2024-09-21T10:19:18.520517+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:34.646046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":270,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.201396+0000","last_change":"2024-09-21T10:14:30.708090+0000","last_active":"2024-09-21T10:19:13.201396+0000","last_peered":"2024-09-21T10:19:13.201396+0000","last_clean":"2024-09-21T10:19:13.201396+0000","last_became_active":"2024-09-21T10:14:30.707266+0000","last_became_peered":"2024-09-21T10:14:30.707266+0000","last_unstale":"2024-09-21T10:19:13.201396+0000","last_undegraded":"2024-09-21T10:19:13.201396+0000","last_fullsized":"2024-09-21T10:19:13.201396+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:38.474537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":524,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.529937+0000","last_change":"2024-09-21T10:10:57.397251+0000","last_active":"2024-09-21T10:19:18.529937+0000","last_peered":"2024-09-21T10:19:18.529937+0000","last_clean":"2024-09-21T10:19:18.529937+0000","last_became_active":"2024-09-21T10:10:57.396271+0000","last_became_peered":"2024-09-21T10:10:57.396271+0000","last_unstale":"2024-09-21T10:19:18.529937+0000","last_undegraded":"2024-09-21T10:19:18.529937+0000","last_fullsized":"2024-09-21T10:19:18.529937+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:17.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.9","version":"376'6463","reported_seq":6491,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:27.570926+0000","last_change":"2024-09-21T10:18:13.037459+0000","last_active":"2024-09-21T10:19:27.570926+0000","last_peered":"2024-09-21T10:19:27.570926+0000","last_clean":"2024-09-21T10:19:27.570926+0000","last_became_active":"2024-09-21T10:18:13.037118+0000","last_became_peered":"2024-09-21T10:18:13.037118+0000","last_unstale":"2024-09-21T10:19:27.570926+0000","last_undegraded":"2024-09-21T10:19:27.570926+0000","last_fullsized":"2024-09-21T10:19:27.570926+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6463,"log_dups_size":0,"ondisk_log_size":6463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:52.139485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423559168,"num_objects":6463,"num_object_clones":0,"num_object_copies":19389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6463,"num_write_kb":413632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":82,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.200139+0000","last_change":"2024-09-21T10:14:30.709001+0000","last_active":"2024-09-21T10:19:13.200139+0000","last_peered":"2024-09-21T10:19:13.200139+0000","last_clean":"2024-09-21T10:19:13.200139+0000","last_became_active":"2024-09-21T10:14:30.708636+0000","last_became_peered":"2024-09-21T10:14:30.708636+0000","last_unstale":"2024-09-21T10:19:13.200139+0000","last_undegraded":"2024-09-21T10:19:13.200139+0000","last_fullsized":"2024-09-21T10:19:13.200139+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:59.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:59.099852+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:20:01.949185+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":616,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.532585+0000","last_change":"2024-09-21T10:09:23.841186+0000","last_active":"2024-09-21T10:19:18.532585+0000","last_peered":"2024-09-21T10:19:18.532585+0000","last_clean":"2024-09-21T10:19:18.532585+0000","last_became_active":"2024-09-21T10:09:23.840854+0000","last_became_peered":"2024-09-21T10:09:23.840854+0000","last_unstale":"2024-09-21T10:19:18.532585+0000","last_undegraded":"2024-09-21T10:19:18.532585+0000","last_fullsized":"2024-09-21T10:19:18.532585+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:29.666358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":469,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.515563+0000","last_change":"2024-09-21T10:14:30.708385+0000","last_active":"2024-09-21T10:19:18.515563+0000","last_peered":"2024-09-21T10:19:18.515563+0000","last_clean":"2024-09-21T10:19:18.515563+0000","last_became_active":"2024-09-21T10:14:30.708046+0000","last_became_peered":"2024-09-21T10:14:30.708046+0000","last_unstale":"2024-09-21T10:19:18.515563+0000","last_undegraded":"2024-09-21T10:19:18.515563+0000","last_fullsized":"2024-09-21T10:19:18.515563+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:16:45.563095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":613,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.530884+0000","last_change":"2024-09-21T10:09:49.926548+0000","last_active":"2024-09-21T10:19:18.530884+0000","last_peered":"2024-09-21T10:19:18.530884+0000","last_clean":"2024-09-21T10:19:18.530884+0000","last_became_active":"2024-09-21T10:09:49.926255+0000","last_became_peered":"2024-09-21T10:09:49.926255+0000","last_unstale":"2024-09-21T10:19:18.530884+0000","last_undegraded":"2024-09-21T10:19:18.530884+0000","last_fullsized":"2024-09-21T10:19:18.530884+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:49.855878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.8","version":"376'6313","reported_seq":6342,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:24.814609+0000","last_change":"2024-09-21T10:18:13.038950+0000","last_active":"2024-09-21T10:19:24.814609+0000","last_peered":"2024-09-21T10:19:24.814609+0000","last_clean":"2024-09-21T10:19:24.814609+0000","last_became_active":"2024-09-21T10:18:13.038641+0000","last_became_peered":"2024-09-21T10:18:13.038641+0000","last_unstale":"2024-09-21T10:19:24.814609+0000","last_undegraded":"2024-09-21T10:19:24.814609+0000","last_fullsized":"2024-09-21T10:19:24.814609+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6313,"log_dups_size":0,"ondisk_log_size":6313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:44:02.783389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413728768,"num_objects":6313,"num_object_clones":0,"num_object_copies":18939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6313,"num_write_kb":404032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":416,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.531975+0000","last_change":"2024-09-21T10:10:57.390804+0000","last_active":"2024-09-21T10:19:18.531975+0000","last_peered":"2024-09-21T10:19:18.531975+0000","last_clean":"2024-09-21T10:19:18.531975+0000","last_became_active":"2024-09-21T10:10:57.390323+0000","last_became_peered":"2024-09-21T10:10:57.390323+0000","last_unstale":"2024-09-21T10:19:18.531975+0000","last_undegraded":"2024-09-21T10:19:18.531975+0000","last_fullsized":"2024-09-21T10:19:18.531975+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.005477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.005477+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:12.788956+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":270,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.527656+0000","last_change":"2024-09-21T10:14:30.718942+0000","last_active":"2024-09-21T10:19:18.527656+0000","last_peered":"2024-09-21T10:19:18.527656+0000","last_clean":"2024-09-21T10:19:18.527656+0000","last_became_active":"2024-09-21T10:14:30.709129+0000","last_became_peered":"2024-09-21T10:14:30.709129+0000","last_unstale":"2024-09-21T10:19:18.527656+0000","last_undegraded":"2024-09-21T10:19:18.527656+0000","last_fullsized":"2024-09-21T10:19:18.527656+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:18.171064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":388,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.207700+0000","last_change":"2024-09-21T10:03:38.392767+0000","last_active":"2024-09-21T10:19:13.207700+0000","last_peered":"2024-09-21T10:19:13.207700+0000","last_clean":"2024-09-21T10:19:13.207700+0000","last_became_active":"2024-09-21T10:03:38.392424+0000","last_became_peered":"2024-09-21T10:03:38.392424+0000","last_unstale":"2024-09-21T10:19:13.207700+0000","last_undegraded":"2024-09-21T10:19:13.207700+0000","last_fullsized":"2024-09-21T10:19:13.207700+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:04:25.358291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":82,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.199569+0000","last_change":"2024-09-21T10:14:30.703352+0000","last_active":"2024-09-21T10:19:13.199569+0000","last_peered":"2024-09-21T10:19:13.199569+0000","last_clean":"2024-09-21T10:19:13.199569+0000","last_became_active":"2024-09-21T10:14:30.703068+0000","last_became_peered":"2024-09-21T10:14:30.703068+0000","last_unstale":"2024-09-21T10:19:13.199569+0000","last_undegraded":"2024-09-21T10:19:13.199569+0000","last_fullsized":"2024-09-21T10:19:13.199569+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:00:26.966633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":82,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.199998+0000","last_change":"2024-09-21T10:14:30.702750+0000","last_active":"2024-09-21T10:19:13.199998+0000","last_peered":"2024-09-21T10:19:13.199998+0000","last_clean":"2024-09-21T10:19:13.199998+0000","last_became_active":"2024-09-21T10:14:30.702443+0000","last_became_peered":"2024-09-21T10:14:30.702443+0000","last_unstale":"2024-09-21T10:19:13.199998+0000","last_undegraded":"2024-09-21T10:19:13.199998+0000","last_fullsized":"2024-09-21T10:19:13.199998+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:38:19.353586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":92,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.200134+0000","last_change":"2024-09-21T10:14:30.716754+0000","last_active":"2024-09-21T10:19:13.200134+0000","last_peered":"2024-09-21T10:19:13.200134+0000","last_clean":"2024-09-21T10:19:13.200134+0000","last_became_active":"2024-09-21T10:14:30.704683+0000","last_became_peered":"2024-09-21T10:14:30.704683+0000","last_unstale":"2024-09-21T10:19:13.200134+0000","last_undegraded":"2024-09-21T10:19:13.200134+0000","last_fullsized":"2024-09-21T10:19:13.200134+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:45.993877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":521,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.514314+0000","last_change":"2024-09-21T10:14:05.477429+0000","last_active":"2024-09-21T10:19:18.514314+0000","last_peered":"2024-09-21T10:19:18.514314+0000","last_clean":"2024-09-21T10:19:18.514314+0000","last_became_active":"2024-09-21T10:14:05.476794+0000","last_became_peered":"2024-09-21T10:14:05.476794+0000","last_unstale":"2024-09-21T10:19:18.514314+0000","last_undegraded":"2024-09-21T10:19:18.514314+0000","last_fullsized":"2024-09-21T10:19:18.514314+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:18:54.963704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":264,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.526035+0000","last_change":"2024-09-21T10:14:30.713873+0000","last_active":"2024-09-21T10:19:18.526035+0000","last_peered":"2024-09-21T10:19:18.526035+0000","last_clean":"2024-09-21T10:19:18.526035+0000","last_became_active":"2024-09-21T10:14:30.703149+0000","last_became_peered":"2024-09-21T10:14:30.703149+0000","last_unstale":"2024-09-21T10:19:18.526035+0000","last_undegraded":"2024-09-21T10:19:18.526035+0000","last_fullsized":"2024-09-21T10:19:18.526035+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:26.453902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'100","reported_seq":415,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.527114+0000","last_change":"2024-09-21T10:18:12.633959+0000","last_active":"2024-09-21T10:19:18.527114+0000","last_peered":"2024-09-21T10:19:18.527114+0000","last_clean":"2024-09-21T10:19:18.527114+0000","last_became_active":"2024-09-21T10:18:12.633591+0000","last_became_peered":"2024-09-21T10:18:12.633591+0000","last_unstale":"2024-09-21T10:19:18.527114+0000","last_undegraded":"2024-09-21T10:19:18.527114+0000","last_fullsized":"2024-09-21T10:19:18.527114+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.014993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.014993+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:39:07.692739+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":289,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.201782+0000","last_change":"2024-09-21T10:15:01.483814+0000","last_active":"2024-09-21T10:19:13.201782+0000","last_peered":"2024-09-21T10:19:13.201782+0000","last_clean":"2024-09-21T10:19:13.201782+0000","last_became_active":"2024-09-21T10:15:01.483522+0000","last_became_peered":"2024-09-21T10:15:01.483522+0000","last_unstale":"2024-09-21T10:19:13.201782+0000","last_undegraded":"2024-09-21T10:19:13.201782+0000","last_fullsized":"2024-09-21T10:19:13.201782+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:13.632066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":598,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.531139+0000","last_change":"2024-09-21T10:03:38.425111+0000","last_active":"2024-09-21T10:19:18.531139+0000","last_peered":"2024-09-21T10:19:18.531139+0000","last_clean":"2024-09-21T10:19:18.531139+0000","last_became_active":"2024-09-21T10:03:38.424704+0000","last_became_peered":"2024-09-21T10:03:38.424704+0000","last_unstale":"2024-09-21T10:19:18.531139+0000","last_undegraded":"2024-09-21T10:19:18.531139+0000","last_fullsized":"2024-09-21T10:19:18.531139+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:35.157076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":499,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.531386+0000","last_change":"2024-09-21T10:10:57.397755+0000","last_active":"2024-09-21T10:19:18.531386+0000","last_peered":"2024-09-21T10:19:18.531386+0000","last_clean":"2024-09-21T10:19:18.531386+0000","last_became_active":"2024-09-21T10:10:57.397249+0000","last_became_peered":"2024-09-21T10:10:57.397249+0000","last_unstale":"2024-09-21T10:19:18.531386+0000","last_undegraded":"2024-09-21T10:19:18.531386+0000","last_fullsized":"2024-09-21T10:19:18.531386+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:40.813503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":239,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.517740+0000","last_change":"2024-09-21T10:14:05.471573+0000","last_active":"2024-09-21T10:19:18.517740+0000","last_peered":"2024-09-21T10:19:18.517740+0000","last_clean":"2024-09-21T10:19:18.517740+0000","last_became_active":"2024-09-21T10:14:05.471270+0000","last_became_peered":"2024-09-21T10:14:05.471270+0000","last_unstale":"2024-09-21T10:19:18.517740+0000","last_undegraded":"2024-09-21T10:19:18.517740+0000","last_fullsized":"2024-09-21T10:19:18.517740+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:56:13.468768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":338,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.207648+0000","last_change":"2024-09-21T09:50:45.508900+0000","last_active":"2024-09-21T10:19:13.207648+0000","last_peered":"2024-09-21T10:19:13.207648+0000","last_clean":"2024-09-21T10:19:13.207648+0000","last_became_active":"2024-09-21T09:50:45.508561+0000","last_became_peered":"2024-09-21T09:50:45.508561+0000","last_unstale":"2024-09-21T10:19:13.207648+0000","last_undegraded":"2024-09-21T10:19:13.207648+0000","last_fullsized":"2024-09-21T10:19:13.207648+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:31.732313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":619,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.530946+0000","last_change":"2024-09-21T10:09:49.928053+0000","last_active":"2024-09-21T10:19:18.530946+0000","last_peered":"2024-09-21T10:19:18.530946+0000","last_clean":"2024-09-21T10:19:18.530946+0000","last_became_active":"2024-09-21T10:09:49.927231+0000","last_became_peered":"2024-09-21T10:09:49.927231+0000","last_unstale":"2024-09-21T10:19:18.530946+0000","last_undegraded":"2024-09-21T10:19:18.530946+0000","last_fullsized":"2024-09-21T10:19:18.530946+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:26.987553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":532,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.527818+0000","last_change":"2024-09-21T10:09:23.840717+0000","last_active":"2024-09-21T10:19:18.527818+0000","last_peered":"2024-09-21T10:19:18.527818+0000","last_clean":"2024-09-21T10:19:18.527818+0000","last_became_active":"2024-09-21T10:09:23.840323+0000","last_became_peered":"2024-09-21T10:09:23.840323+0000","last_unstale":"2024-09-21T10:19:18.527818+0000","last_undegraded":"2024-09-21T10:19:18.527818+0000","last_fullsized":"2024-09-21T10:19:18.527818+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:55.769770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":252,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.529507+0000","last_change":"2024-09-21T10:18:12.632498+0000","last_active":"2024-09-21T10:19:18.529507+0000","last_peered":"2024-09-21T10:19:18.529507+0000","last_clean":"2024-09-21T10:19:18.529507+0000","last_became_active":"2024-09-21T10:18:12.631648+0000","last_became_peered":"2024-09-21T10:18:12.631648+0000","last_unstale":"2024-09-21T10:19:18.529507+0000","last_undegraded":"2024-09-21T10:19:18.529507+0000","last_fullsized":"2024-09-21T10:19:18.529507+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:17.592618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":597,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.531927+0000","last_change":"2024-09-21T10:14:30.702101+0000","last_active":"2024-09-21T10:19:18.531927+0000","last_peered":"2024-09-21T10:19:18.531927+0000","last_clean":"2024-09-21T10:19:18.531927+0000","last_became_active":"2024-09-21T10:14:30.701783+0000","last_became_peered":"2024-09-21T10:14:30.701783+0000","last_unstale":"2024-09-21T10:19:18.531927+0000","last_undegraded":"2024-09-21T10:19:18.531927+0000","last_fullsized":"2024-09-21T10:19:18.531927+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:13.087010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":82,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.199577+0000","last_change":"2024-09-21T10:14:30.707503+0000","last_active":"2024-09-21T10:19:13.199577+0000","last_peered":"2024-09-21T10:19:13.199577+0000","last_clean":"2024-09-21T10:19:13.199577+0000","last_became_active":"2024-09-21T10:14:30.706881+0000","last_became_peered":"2024-09-21T10:14:30.706881+0000","last_unstale":"2024-09-21T10:19:13.199577+0000","last_undegraded":"2024-09-21T10:19:13.199577+0000","last_fullsized":"2024-09-21T10:19:13.199577+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:23:07.485461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":637,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.527253+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:19:18.527253+0000","last_peered":"2024-09-21T10:19:18.527253+0000","last_clean":"2024-09-21T10:19:18.527253+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:19:18.527253+0000","last_undegraded":"2024-09-21T10:19:18.527253+0000","last_fullsized":"2024-09-21T10:19:18.527253+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":261,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.529268+0000","last_change":"2024-09-21T10:14:30.702414+0000","last_active":"2024-09-21T10:19:18.529268+0000","last_peered":"2024-09-21T10:19:18.529268+0000","last_clean":"2024-09-21T10:19:18.529268+0000","last_became_active":"2024-09-21T10:14:30.702009+0000","last_became_peered":"2024-09-21T10:14:30.702009+0000","last_unstale":"2024-09-21T10:19:18.529268+0000","last_undegraded":"2024-09-21T10:19:18.529268+0000","last_fullsized":"2024-09-21T10:19:18.529268+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:50:48.396082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.199495+0000","last_change":"2024-09-21T10:18:12.633151+0000","last_active":"2024-09-21T10:19:13.199495+0000","last_peered":"2024-09-21T10:19:13.199495+0000","last_clean":"2024-09-21T10:19:13.199495+0000","last_became_active":"2024-09-21T10:18:12.632854+0000","last_became_peered":"2024-09-21T10:18:12.632854+0000","last_unstale":"2024-09-21T10:19:13.199495+0000","last_undegraded":"2024-09-21T10:19:13.199495+0000","last_fullsized":"2024-09-21T10:19:13.199495+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:14.602893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":86,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.200004+0000","last_change":"2024-09-21T10:14:30.720787+0000","last_active":"2024-09-21T10:19:13.200004+0000","last_peered":"2024-09-21T10:19:13.200004+0000","last_clean":"2024-09-21T10:19:13.200004+0000","last_became_active":"2024-09-21T10:14:30.706001+0000","last_became_peered":"2024-09-21T10:14:30.706001+0000","last_unstale":"2024-09-21T10:19:13.200004+0000","last_undegraded":"2024-09-21T10:19:13.200004+0000","last_fullsized":"2024-09-21T10:19:13.200004+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:40:11.987108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.199493+0000","last_change":"2024-09-21T10:18:12.636252+0000","last_active":"2024-09-21T10:19:13.199493+0000","last_peered":"2024-09-21T10:19:13.199493+0000","last_clean":"2024-09-21T10:19:13.199493+0000","last_became_active":"2024-09-21T10:18:12.635945+0000","last_became_peered":"2024-09-21T10:18:12.635945+0000","last_unstale":"2024-09-21T10:19:13.199493+0000","last_undegraded":"2024-09-21T10:19:13.199493+0000","last_fullsized":"2024-09-21T10:19:13.199493+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:59:29.464045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":86,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.200203+0000","last_change":"2024-09-21T10:14:30.722338+0000","last_active":"2024-09-21T10:19:13.200203+0000","last_peered":"2024-09-21T10:19:13.200203+0000","last_clean":"2024-09-21T10:19:13.200203+0000","last_became_active":"2024-09-21T10:14:30.708164+0000","last_became_peered":"2024-09-21T10:14:30.708164+0000","last_unstale":"2024-09-21T10:19:13.200203+0000","last_undegraded":"2024-09-21T10:19:13.200203+0000","last_fullsized":"2024-09-21T10:19:13.200203+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:32:13.825114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":620,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.531486+0000","last_change":"2024-09-21T10:09:23.840467+0000","last_active":"2024-09-21T10:19:18.531486+0000","last_peered":"2024-09-21T10:19:18.531486+0000","last_clean":"2024-09-21T10:19:18.531486+0000","last_became_active":"2024-09-21T10:09:23.840177+0000","last_became_peered":"2024-09-21T10:09:23.840177+0000","last_unstale":"2024-09-21T10:19:18.531486+0000","last_undegraded":"2024-09-21T10:19:18.531486+0000","last_fullsized":"2024-09-21T10:19:18.531486+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:17.887307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":610,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.532510+0000","last_change":"2024-09-21T10:11:05.451613+0000","last_active":"2024-09-21T10:19:18.532510+0000","last_peered":"2024-09-21T10:19:18.532510+0000","last_clean":"2024-09-21T10:19:18.532510+0000","last_became_active":"2024-09-21T10:11:05.451075+0000","last_became_peered":"2024-09-21T10:11:05.451075+0000","last_unstale":"2024-09-21T10:19:18.532510+0000","last_undegraded":"2024-09-21T10:19:18.532510+0000","last_fullsized":"2024-09-21T10:19:18.532510+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:02:18.505045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":82,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-21T10:19:13.199268+0000","last_change":"2024-09-21T10:14:30.707548+0000","last_active":"2024-09-21T10:19:13.199268+0000","last_peered":"2024-09-21T10:19:13.199268+0000","last_clean":"2024-09-21T10:19:13.199268+0000","last_became_active":"2024-09-21T10:14:30.707232+0000","last_became_peered":"2024-09-21T10:14:30.707232+0000","last_unstale":"2024-09-21T10:19:13.199268+0000","last_undegraded":"2024-09-21T10:19:13.199268+0000","last_fullsized":"2024-09-21T10:19:13.199268+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:11:38.905767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":274,"reported_epoch":376,"state":"active+clean","last_fresh":"2024-09-21T10:19:18.527457+0000","last_change":"2024-09-21T10:10:57.401588+0000","last_active":"2024-09-21T10:19:18.527457+0000","last_peered":"2024-09-21T10:19:18.527457+0000","last_clean":"2024-09-21T10:19:18.527457+0000","last_became_active":"2024-09-21T10:10:57.397724+0000","last_became_peered":"2024-09-21T10:10:57.397724+0000","last_unstale":"2024-09-21T10:19:18.527457+0000","last_undegraded":"2024-09-21T10:19:18.527457+0000","last_fullsized":"2024-09-21T10:19:18.527457+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:47.843630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":16,"stat_sum":{"num_bytes":6738673664,"num_objects":102824,"num_object_clones":0,"num_object_copies":308472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102824,"num_write_kb":6580736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20170539008,"data_stored":20170539008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101724,"ondisk_log_size":101724,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":11856,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26580,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":644037,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":68,"num_write":13,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":632715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626925,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3248620,"kb_used_data":2059876,"kb_used_omap":1,"kb_used_meta":1188734,"kb_avail":90496532,"statfs":{"total":95995035648,"available":92668448768,"internally_reserved":0,"allocated":2109313024,"data_stored":2106996216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1217263960},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019383,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4858288,"kb_used_data":3674792,"kb_used_omap":1,"kb_used_meta":1183486,"kb_avail":88886864,"statfs":{"total":95995035648,"available":91020148736,"internally_reserved":0,"allocated":3762987008,"data_stored":3760666145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1211890008},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":245,"seq":1052266987642,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1679456,"kb_used_data":1255576,"kb_used_omap":8,"kb_used_meta":423863,"kb_avail":92065696,"statfs":{"total":95995035648,"available":94275272704,"internally_reserved":0,"allocated":1285709824,"data_stored":1283401091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8940,"internal_metadata":434035988},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":175,"seq":751619276991,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2979120,"kb_used_data":2050152,"kb_used_omap":12,"kb_used_meta":928947,"kb_avail":90766032,"statfs":{"total":95995035648,"available":92944416768,"internally_reserved":0,"allocated":2099355648,"data_stored":2097034781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12845,"internal_metadata":951242195},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823154,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4748128,"kb_used_data":3250328,"kb_used_omap":1,"kb_used_meta":1497790,"kb_avail":88997024,"statfs":{"total":95995035648,"available":91132952576,"internally_reserved":0,"allocated":3328335872,"data_stored":3326027092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1533737304},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879577,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2692832,"kb_used_data":1661208,"kb_used_omap":4,"kb_used_meta":1031611,"kb_avail":91052320,"statfs":{"total":95995035648,"available":93237575680,"internally_reserved":0,"allocated":1701076992,"data_stored":1698768293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":1056370108},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566417,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4665448,"kb_used_data":3620320,"kb_used_omap":6,"kb_used_meta":1045113,"kb_avail":89079704,"statfs":{"total":95995035648,"available":91217616896,"internally_reserved":0,"allocated":3707207680,"data_stored":3704891401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":1070196135},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860762,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3177176,"kb_used_data":2034832,"kb_used_omap":5,"kb_used_meta":1142330,"kb_avail":90567976,"statfs":{"total":95995035648,"available":92741607424,"internally_reserved":0,"allocated":2083667968,"data_stored":2081366523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":1169746159},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7236,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9976,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1283719168,"data_stored":1283719168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2090401792,"data_stored":2090401792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3726376960,"data_stored":3726376960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1702559744,"data_stored":1702559744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3790077952,"data_stored":3790077952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3351117824,"data_stored":3351117824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2113536000,"data_stored":2113536000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2112749568,"data_stored":2112749568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:19:32.397 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_8 pgp_num 16 2024-09-21T10:19:32.401 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:32.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:32.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:32.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:32.531 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:32.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:32.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:32.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:32 smithi107 ceph-mon[102221]: pgmap v1296: 109 pgs: 109 active+clean; 6.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 61 MiB/s wr, 972 op/s 2024-09-21T10:19:32.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:32 smithi107 ceph-mon[105022]: pgmap v1296: 109 pgs: 109 active+clean; 6.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 61 MiB/s wr, 972 op/s 2024-09-21T10:19:32.632 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:32.667 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:32.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:32.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:32.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:32.800 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:32.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:32.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:32.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:32 smithi129 ceph-mon[88471]: pgmap v1296: 109 pgs: 109 active+clean; 6.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 61 MiB/s wr, 972 op/s 2024-09-21T10:19:32.930 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:32.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:32.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:33.032 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:33.060 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:33.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:33.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:33.070 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 71 16 109447 109431 96.311 64.3125 0.0137427 0.0103776 2024-09-21T10:19:33.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:33.195 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:33.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:33.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:33.297 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:33.326 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:33.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:33.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:33.428 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:33.464 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:33.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:33.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:33.566 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:33.602 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:33.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:33.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:33.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:33.732 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:33.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:33.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:33.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:33.865 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:33.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:33.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:33.967 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:34.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:34.008 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:34.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:34.071 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 72 16 110467 110451 95.8587 63.75 0.0222456 0.010427 2024-09-21T10:19:34.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:34.145 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:34.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:34.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:34.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:34.280 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:34.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:34.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:34.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:34.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:34.432 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:34.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:34.533 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:34.558 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:34.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:34.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:34.660 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:34.699 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:34.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:34.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:34.789 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:19:34.789 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-09-21T10:19:34.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-09-21T10:19:34.789 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:19:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.36561.0:50687 11.6 e259/255 11:608cc6b9:::benchmark_data_smithi107_218812_object50686:head v 259'3102, mlcod=259'3099)", 2024-09-21T10:19:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:19.581842+0000", 2024-09-21T10:19:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 554.85081176899996, 2024-09-21T10:19:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.24672677600000001, 2024-09-21T10:19:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:19:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:19:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:19:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:19:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.581842+0000", 2024-09-21T10:19:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:19:34.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.581842+0000", 2024-09-21T10:19:34.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:19:34.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.581843+0000", 2024-09-21T10:19:34.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8000000000000005e-07 2024-09-21T10:19:34.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:19:34.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.581870+0000", 2024-09-21T10:19:34.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6707e-05 2024-09-21T10:19:34.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:19:34.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.581880+0000", 2024-09-21T10:19:34.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0351e-05 2024-09-21T10:19:34.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:19:34.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.581886+0000", 2024-09-21T10:19:34.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2609999999999999e-06 2024-09-21T10:19:34.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:19:34.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.582002+0000", 2024-09-21T10:19:34.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000115463 2024-09-21T10:19:34.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:19:34.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.582019+0000", 2024-09-21T10:19:34.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7544e-05 2024-09-21T10:19:34.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:19:34.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.828492+0000", 2024-09-21T10:19:34.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.24647333599999999 2024-09-21T10:19:34.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:19:34.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.828569+0000", 2024-09-21T10:19:34.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6334000000000005e-05 2024-09-21T10:19:34.794 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.794 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.794 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.794 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.36561.0:50684 11.0 e259/252 11:0fced6bd:::benchmark_data_smithi107_218812_object50683:head v 259'3167, mlcod=259'3164)", 2024-09-21T10:19:34.794 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:19.583753+0000", 2024-09-21T10:19:34.794 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 554.84890081699996, 2024-09-21T10:19:34.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.24399035599999999, 2024-09-21T10:19:34.794 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:19:34.794 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:19:34.795 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:34.795 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:19:34.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:19:34.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.583753+0000", 2024-09-21T10:19:34.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:19:34.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.583753+0000", 2024-09-21T10:19:34.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:19:34.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.583754+0000", 2024-09-21T10:19:34.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2699999999999998e-07 2024-09-21T10:19:34.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:19:34.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.583774+0000", 2024-09-21T10:19:34.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0299999999999999e-05 2024-09-21T10:19:34.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:19:34.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.583784+0000", 2024-09-21T10:19:34.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.025e-05 2024-09-21T10:19:34.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:19:34.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.583790+0000", 2024-09-21T10:19:34.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1199999999999999e-06 2024-09-21T10:19:34.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:19:34.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.583866+0000", 2024-09-21T10:19:34.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5449000000000002e-05 2024-09-21T10:19:34.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:19:34.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.583883+0000", 2024-09-21T10:19:34.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7221e-05 2024-09-21T10:19:34.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:19:34.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.585793+0000", 2024-09-21T10:19:34.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001910265 2024-09-21T10:19:34.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:19:34.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.827743+0000", 2024-09-21T10:19:34.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.24194992400000001 2024-09-21T10:19:34.799 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.799 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.799 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.799 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.36561.0:50688 11.e e259/252 11:7867821a:::benchmark_data_smithi107_218812_object50687:head v 259'3256, mlcod=259'3254)", 2024-09-21T10:19:34.799 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:19.584014+0000", 2024-09-21T10:19:34.799 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 554.84863952600006, 2024-09-21T10:19:34.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.24458989, 2024-09-21T10:19:34.799 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:19:34.799 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:19:34.799 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:34.799 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:19:34.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:19:34.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.584014+0000", 2024-09-21T10:19:34.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:19:34.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.584014+0000", 2024-09-21T10:19:34.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.800 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:19:34.800 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.584015+0000", 2024-09-21T10:19:34.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8000000000000005e-07 2024-09-21T10:19:34.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:19:34.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.584053+0000", 2024-09-21T10:19:34.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8019999999999999e-05 2024-09-21T10:19:34.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:19:34.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.584064+0000", 2024-09-21T10:19:34.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1080999999999999e-05 2024-09-21T10:19:34.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:19:34.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.584076+0000", 2024-09-21T10:19:34.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2330000000000001e-05 2024-09-21T10:19:34.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:19:34.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.584144+0000", 2024-09-21T10:19:34.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7477999999999993e-05 2024-09-21T10:19:34.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:19:34.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.584161+0000", 2024-09-21T10:19:34.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6889e-05 2024-09-21T10:19:34.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:19:34.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.828572+0000", 2024-09-21T10:19:34.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.244411085 2024-09-21T10:19:34.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:19:34.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.828604+0000", 2024-09-21T10:19:34.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2227000000000002e-05 2024-09-21T10:19:34.803 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.803 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.803 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.803 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.36561.0:50689 11.a e259/252 11:5d3fa524:::benchmark_data_smithi107_218812_object50688:head v 259'3141, mlcod=259'3139)", 2024-09-21T10:19:34.803 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:19.585686+0000", 2024-09-21T10:19:34.803 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 554.846967941, 2024-09-21T10:19:34.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.24556945099999999, 2024-09-21T10:19:34.803 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:19:34.803 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:19:34.804 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:34.804 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:19:34.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:19:34.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.585686+0000", 2024-09-21T10:19:34.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:19:34.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.585686+0000", 2024-09-21T10:19:34.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:19:34.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.585687+0000", 2024-09-21T10:19:34.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5300000000000003e-07 2024-09-21T10:19:34.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:19:34.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.585712+0000", 2024-09-21T10:19:34.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5032e-05 2024-09-21T10:19:34.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.805 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:19:34.805 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.585724+0000", 2024-09-21T10:19:34.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1798e-05 2024-09-21T10:19:34.805 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:19:34.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.585731+0000", 2024-09-21T10:19:34.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1520000000000003e-06 2024-09-21T10:19:34.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:19:34.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.585785+0000", 2024-09-21T10:19:34.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4051000000000001e-05 2024-09-21T10:19:34.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:19:34.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.827707+0000", 2024-09-21T10:19:34.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.241922629 2024-09-21T10:19:34.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:19:34.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.831210+0000", 2024-09-21T10:19:34.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035022410000000001 2024-09-21T10:19:34.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:19:34.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.831255+0000", 2024-09-21T10:19:34.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5695000000000002e-05 2024-09-21T10:19:34.807 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.807 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.807 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.808 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.36561.0:50686 11.d e259/252 11:be2ad635:::benchmark_data_smithi107_218812_object50685:head v 259'3239, mlcod=259'3236)", 2024-09-21T10:19:34.808 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:19.585757+0000", 2024-09-21T10:19:34.808 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 554.84689696999999, 2024-09-21T10:19:34.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.25439091600000002, 2024-09-21T10:19:34.808 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:19:34.808 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:19:34.808 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:34.808 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:19:34.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:19:34.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.585757+0000", 2024-09-21T10:19:34.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:19:34.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.585757+0000", 2024-09-21T10:19:34.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:19:34.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.585757+0000", 2024-09-21T10:19:34.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7800000000000002e-07 2024-09-21T10:19:34.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:19:34.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.585779+0000", 2024-09-21T10:19:34.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1821e-05 2024-09-21T10:19:34.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:19:34.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.827667+0000", 2024-09-21T10:19:34.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.24188795199999999 2024-09-21T10:19:34.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:19:34.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.827685+0000", 2024-09-21T10:19:34.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8295e-05 2024-09-21T10:19:34.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:19:34.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.829532+0000", 2024-09-21T10:19:34.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0018470349999999999 2024-09-21T10:19:34.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:19:34.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.829550+0000", 2024-09-21T10:19:34.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7648000000000001e-05 2024-09-21T10:19:34.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:19:34.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.840034+0000", 2024-09-21T10:19:34.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010483622 2024-09-21T10:19:34.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:19:34.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.840148+0000", 2024-09-21T10:19:34.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114065 2024-09-21T10:19:34.812 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.812 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.812 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.812 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.36561.0:50706 11.f e259/252 11:f8bf77a7:::benchmark_data_smithi107_218812_object50705:head v 259'3249, mlcod=259'3247)", 2024-09-21T10:19:34.812 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:19.593307+0000", 2024-09-21T10:19:34.812 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 554.83934684999997, 2024-09-21T10:19:34.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.239927419, 2024-09-21T10:19:34.812 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:19:34.812 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:19:34.812 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:34.812 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:19:34.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:19:34.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.593307+0000", 2024-09-21T10:19:34.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:19:34.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.593307+0000", 2024-09-21T10:19:34.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:19:34.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.593308+0000", 2024-09-21T10:19:34.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4050000000000001e-06 2024-09-21T10:19:34.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:19:34.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.593417+0000", 2024-09-21T10:19:34.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108414 2024-09-21T10:19:34.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:19:34.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.827695+0000", 2024-09-21T10:19:34.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.23427838000000001 2024-09-21T10:19:34.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:19:34.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.827706+0000", 2024-09-21T10:19:34.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0657e-05 2024-09-21T10:19:34.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:19:34.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.827858+0000", 2024-09-21T10:19:34.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015201 2024-09-21T10:19:34.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:19:34.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.827880+0000", 2024-09-21T10:19:34.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2450999999999999e-05 2024-09-21T10:19:34.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:19:34.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.833172+0000", 2024-09-21T10:19:34.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052918849999999996 2024-09-21T10:19:34.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:19:34.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:19.833234+0000", 2024-09-21T10:19:34.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2217000000000004e-05 2024-09-21T10:19:34.816 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.816 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.816 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.816 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:46078 12.6 12:6570b22e:::benchmark_data_smithi107_250110_object46077:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-09-21T10:19:34.816 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:34.307148+0000", 2024-09-21T10:19:34.816 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 300.12550591000002, 2024-09-21T10:19:34.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.34402638499999999, 2024-09-21T10:19:34.816 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:19:34.816 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:19:34.816 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:34.816 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:19:34.816 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:19:34.817 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:19:34.817 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 46078 2024-09-21T10:19:34.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.817 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:19:34.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:19:34.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.307148+0000", 2024-09-21T10:19:34.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:19:34.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.307148+0000", 2024-09-21T10:19:34.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:19:34.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.307149+0000", 2024-09-21T10:19:34.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.471e-06 2024-09-21T10:19:34.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:19:34.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.307171+0000", 2024-09-21T10:19:34.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2093e-05 2024-09-21T10:19:34.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:19:34.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.307181+0000", 2024-09-21T10:19:34.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5370000000000003e-06 2024-09-21T10:19:34.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:19:34.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.307187+0000", 2024-09-21T10:19:34.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1419999999999998e-06 2024-09-21T10:19:34.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:19:34.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.307227+0000", 2024-09-21T10:19:34.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9944000000000002e-05 2024-09-21T10:19:34.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:19:34.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.307359+0000", 2024-09-21T10:19:34.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013215199999999999 2024-09-21T10:19:34.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:19:34.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.307489+0000", 2024-09-21T10:19:34.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000129617 2024-09-21T10:19:34.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:19:34.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.309941+0000", 2024-09-21T10:19:34.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024522160000000001 2024-09-21T10:19:34.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:34.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.310524+0000", 2024-09-21T10:19:34.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00058306899999999997 2024-09-21T10:19:34.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:34.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315276+0000", 2024-09-21T10:19:34.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004752076 2024-09-21T10:19:34.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:19:34.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315322+0000", 2024-09-21T10:19:34.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6217000000000002e-05 2024-09-21T10:19:34.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.822 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:19:34.822 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651174+0000", 2024-09-21T10:19:34.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.33585185099999998 2024-09-21T10:19:34.822 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.822 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.822 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.822 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:46089 12.7 12:ee14e570:::benchmark_data_smithi107_250110_object46088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-09-21T10:19:34.822 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:34.311237+0000", 2024-09-21T10:19:34.822 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 300.12141708399997, 2024-09-21T10:19:34.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.34013115999999999, 2024-09-21T10:19:34.822 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:19:34.822 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:19:34.823 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:34.823 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:19:34.823 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:19:34.823 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:19:34.823 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 46089 2024-09-21T10:19:34.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.823 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:19:34.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:19:34.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.311237+0000", 2024-09-21T10:19:34.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:19:34.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.311237+0000", 2024-09-21T10:19:34.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:19:34.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.311238+0000", 2024-09-21T10:19:34.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6500000000000001e-06 2024-09-21T10:19:34.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:19:34.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.311262+0000", 2024-09-21T10:19:34.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3453000000000001e-05 2024-09-21T10:19:34.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:19:34.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.311274+0000", 2024-09-21T10:19:34.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2310000000000001e-05 2024-09-21T10:19:34.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:19:34.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.311282+0000", 2024-09-21T10:19:34.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5730000000000002e-06 2024-09-21T10:19:34.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:19:34.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.311338+0000", 2024-09-21T10:19:34.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.626e-05 2024-09-21T10:19:34.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:19:34.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.311472+0000", 2024-09-21T10:19:34.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013357700000000001 2024-09-21T10:19:34.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:19:34.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.311596+0000", 2024-09-21T10:19:34.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012442900000000001 2024-09-21T10:19:34.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:19:34.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.313715+0000", 2024-09-21T10:19:34.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021186820000000002 2024-09-21T10:19:34.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:34.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.314073+0000", 2024-09-21T10:19:34.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035868 2024-09-21T10:19:34.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:34.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651265+0000", 2024-09-21T10:19:34.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.33719125300000002 2024-09-21T10:19:34.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:19:34.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651325+0000", 2024-09-21T10:19:34.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0223999999999998e-05 2024-09-21T10:19:34.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:19:34.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651368+0000", 2024-09-21T10:19:34.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3068999999999997e-05 2024-09-21T10:19:34.828 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.828 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.828 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.828 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.36822.0:46083 12.d e305/294 12:b93ee741:::benchmark_data_smithi107_250110_object46082:head v 305'2864, mlcod=305'2860)", 2024-09-21T10:19:34.828 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:34.314801+0000", 2024-09-21T10:19:34.828 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 300.11785301999998, 2024-09-21T10:19:34.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.33645499499999998, 2024-09-21T10:19:34.828 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:19:34.829 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:19:34.829 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:34.829 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:19:34.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:19:34.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.314801+0000", 2024-09-21T10:19:34.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:19:34.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.314801+0000", 2024-09-21T10:19:34.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:19:34.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.314801+0000", 2024-09-21T10:19:34.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9700000000000003e-07 2024-09-21T10:19:34.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:19:34.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.314841+0000", 2024-09-21T10:19:34.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9743999999999998e-05 2024-09-21T10:19:34.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:19:34.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.314848+0000", 2024-09-21T10:19:34.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1210000000000001e-06 2024-09-21T10:19:34.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:19:34.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.314853+0000", 2024-09-21T10:19:34.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7659999999999998e-06 2024-09-21T10:19:34.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:19:34.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315050+0000", 2024-09-21T10:19:34.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019765900000000001 2024-09-21T10:19:34.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:19:34.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315067+0000", 2024-09-21T10:19:34.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6878999999999999e-05 2024-09-21T10:19:34.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:19:34.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651212+0000", 2024-09-21T10:19:34.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.33614440200000001 2024-09-21T10:19:34.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:19:34.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651256+0000", 2024-09-21T10:19:34.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4127e-05 2024-09-21T10:19:34.832 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.832 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.832 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.832 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.36822.0:46078 12.6 e305/298 ondisk, result = 0)", 2024-09-21T10:19:34.832 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:34.315124+0000", 2024-09-21T10:19:34.833 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 300.117529913, 2024-09-21T10:19:34.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.33606297099999999, 2024-09-21T10:19:34.833 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:19:34.833 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:19:34.833 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:19:34.833 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:19:34.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:19:34.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315124+0000", 2024-09-21T10:19:34.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:19:34.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315124+0000", 2024-09-21T10:19:34.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:19:34.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315125+0000", 2024-09-21T10:19:34.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5700000000000002e-07 2024-09-21T10:19:34.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:19:34.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315129+0000", 2024-09-21T10:19:34.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9250000000000003e-06 2024-09-21T10:19:34.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:19:34.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315140+0000", 2024-09-21T10:19:34.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0693000000000001e-05 2024-09-21T10:19:34.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:19:34.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315146+0000", 2024-09-21T10:19:34.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2539999999999998e-06 2024-09-21T10:19:34.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:19:34.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315264+0000", 2024-09-21T10:19:34.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000117442 2024-09-21T10:19:34.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:19:34.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315271+0000", 2024-09-21T10:19:34.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1080000000000004e-06 2024-09-21T10:19:34.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:19:34.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651187+0000", 2024-09-21T10:19:34.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.33591589199999999 2024-09-21T10:19:34.836 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.836 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.836 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.836 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.36822.0:46089 12.7 e305/298 ondisk, result = 0)", 2024-09-21T10:19:34.836 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:34.316234+0000", 2024-09-21T10:19:34.836 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 300.11641998300001, 2024-09-21T10:19:34.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.33514012599999998, 2024-09-21T10:19:34.837 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:19:34.837 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:19:34.837 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:19:34.837 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:19:34.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:19:34.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.316234+0000", 2024-09-21T10:19:34.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:19:34.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.316234+0000", 2024-09-21T10:19:34.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:19:34.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.316234+0000", 2024-09-21T10:19:34.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7599999999999997e-07 2024-09-21T10:19:34.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:19:34.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.316239+0000", 2024-09-21T10:19:34.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7779999999999998e-06 2024-09-21T10:19:34.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:19:34.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651187+0000", 2024-09-21T10:19:34.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.33494754599999998 2024-09-21T10:19:34.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:19:34.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651197+0000", 2024-09-21T10:19:34.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0327e-05 2024-09-21T10:19:34.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:19:34.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651248+0000", 2024-09-21T10:19:34.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1161e-05 2024-09-21T10:19:34.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:19:34.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651257+0000", 2024-09-21T10:19:34.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0040000000000005e-06 2024-09-21T10:19:34.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:19:34.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651374+0000", 2024-09-21T10:19:34.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011663399999999999 2024-09-21T10:19:34.840 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.840 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.840 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.840 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:46096 12.7 12:eb5f27db:::benchmark_data_smithi107_250110_object46095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-09-21T10:19:34.840 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:34.316948+0000", 2024-09-21T10:19:34.840 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 300.115705521, 2024-09-21T10:19:34.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.343015389, 2024-09-21T10:19:34.840 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:19:34.841 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:19:34.841 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:34.841 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:19:34.841 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:19:34.841 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:19:34.841 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 46096 2024-09-21T10:19:34.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.841 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:19:34.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:19:34.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.316948+0000", 2024-09-21T10:19:34.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:19:34.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.316948+0000", 2024-09-21T10:19:34.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:19:34.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.316950+0000", 2024-09-21T10:19:34.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6029999999999999e-06 2024-09-21T10:19:34.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:19:34.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.316971+0000", 2024-09-21T10:19:34.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1094000000000002e-05 2024-09-21T10:19:34.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:19:34.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651187+0000", 2024-09-21T10:19:34.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.33421569200000001 2024-09-21T10:19:34.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:19:34.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651198+0000", 2024-09-21T10:19:34.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1413000000000001e-05 2024-09-21T10:19:34.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:19:34.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651406+0000", 2024-09-21T10:19:34.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00020795500000000001 2024-09-21T10:19:34.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.844 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:34 smithi107 ceph-mon[102221]: pgmap v1297: 109 pgs: 109 active+clean; 6.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:19:34.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.844 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:19:34.844 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651554+0000", 2024-09-21T10:19:34.844 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014769699999999999 2024-09-21T10:19:34.844 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:19:34.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651700+0000", 2024-09-21T10:19:34.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014646799999999999 2024-09-21T10:19:34.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:34.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.657141+0000", 2024-09-21T10:19:34.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054412330000000002 2024-09-21T10:19:34.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:19:34.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.657222+0000", 2024-09-21T10:19:34.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0709000000000003e-05 2024-09-21T10:19:34.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:34.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.659458+0000", 2024-09-21T10:19:34.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022354789999999999 2024-09-21T10:19:34.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:19:34.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.659919+0000", 2024-09-21T10:19:34.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046138700000000001 2024-09-21T10:19:34.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:19:34.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.659964+0000", 2024-09-21T10:19:34.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4659000000000001e-05 2024-09-21T10:19:34.846 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.846 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.847 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.847 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:77581 12.7 12:e0b93a5b:::benchmark_data_smithi107_250110_object77580:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-21T10:19:34.847 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:15:02.372013+0000", 2024-09-21T10:19:34.847 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 272.06064059400001, 2024-09-21T10:19:34.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.25667214399999999, 2024-09-21T10:19:34.847 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:19:34.847 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:19:34.847 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:34.847 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:19:34.847 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:19:34.847 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:19:34.847 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 77581 2024-09-21T10:19:34.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.848 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:19:34.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:19:34.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:02.372013+0000", 2024-09-21T10:19:34.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:19:34.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:02.372013+0000", 2024-09-21T10:19:34.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:19:34.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:02.372014+0000", 2024-09-21T10:19:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3699999999999999e-07 2024-09-21T10:19:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:19:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:02.372096+0000", 2024-09-21T10:19:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2349999999999996e-05 2024-09-21T10:19:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:19:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:02.372121+0000", 2024-09-21T10:19:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.4528e-05 2024-09-21T10:19:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:19:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:02.372128+0000", 2024-09-21T10:19:34.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.832e-06 2024-09-21T10:19:34.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.850 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:19:34.850 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:02.380233+0000", 2024-09-21T10:19:34.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0081047900000000006 2024-09-21T10:19:34.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.850 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:19:34.850 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:02.380540+0000", 2024-09-21T10:19:34.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0003077 2024-09-21T10:19:34.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.850 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:19:34.850 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:02.380932+0000", 2024-09-21T10:19:34.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00039132400000000002 2024-09-21T10:19:34.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:34.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:02.383989+0000", 2024-09-21T10:19:34.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0030577529999999999 2024-09-21T10:19:34.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:19:34.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:02.384002+0000", 2024-09-21T10:19:34.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2228999999999999e-05 2024-09-21T10:19:34.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:34.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:02.628596+0000", 2024-09-21T10:19:34.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.24459440499999999 2024-09-21T10:19:34.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:19:34.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:02.628647+0000", 2024-09-21T10:19:34.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0971999999999999e-05 2024-09-21T10:19:34.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:19:34.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:15:02.628685+0000", 2024-09-21T10:19:34.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8423999999999997e-05 2024-09-21T10:19:34.852 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.852 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.852 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.853 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:26926 13.c 13:3dd20042:::benchmark_data_smithi107_278655_object26925:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:19:34.853 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:18:31.489966+0000", 2024-09-21T10:19:34.853 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 62.942687648000003, 2024-09-21T10:19:34.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.30380030899999999, 2024-09-21T10:19:34.853 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:19:34.853 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:19:34.853 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:34.853 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:19:34.853 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:19:34.853 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:19:34.853 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 26926 2024-09-21T10:19:34.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.854 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:19:34.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:19:34.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.489966+0000", 2024-09-21T10:19:34.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:19:34.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.489966+0000", 2024-09-21T10:19:34.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:19:34.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.489968+0000", 2024-09-21T10:19:34.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7740000000000001e-06 2024-09-21T10:19:34.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:19:34.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.489991+0000", 2024-09-21T10:19:34.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3371999999999999e-05 2024-09-21T10:19:34.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:19:34.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.490004+0000", 2024-09-21T10:19:34.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3148e-05 2024-09-21T10:19:34.855 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:19:34.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.490027+0000", 2024-09-21T10:19:34.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2081000000000001e-05 2024-09-21T10:19:34.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:19:34.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.490119+0000", 2024-09-21T10:19:34.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2196999999999998e-05 2024-09-21T10:19:34.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:19:34.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.490294+0000", 2024-09-21T10:19:34.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017526999999999999 2024-09-21T10:19:34.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:19:34.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.490422+0000", 2024-09-21T10:19:34.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000127797 2024-09-21T10:19:34.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:19:34.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.491892+0000", 2024-09-21T10:19:34.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014706859999999999 2024-09-21T10:19:34.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:34.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.492530+0000", 2024-09-21T10:19:34.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00063751900000000002 2024-09-21T10:19:34.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:34.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793657+0000", 2024-09-21T10:19:34.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.30112738100000003 2024-09-21T10:19:34.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:19:34.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793719+0000", 2024-09-21T10:19:34.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1260999999999995e-05 2024-09-21T10:19:34.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:19:34.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793766+0000", 2024-09-21T10:19:34.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7822999999999999e-05 2024-09-21T10:19:34.858 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.858 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.859 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.859 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:26943 13.c 13:3d5b5678:::benchmark_data_smithi107_278655_object26942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:19:34.859 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:18:31.495178+0000", 2024-09-21T10:19:34.859 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 62.937475751999997, 2024-09-21T10:19:34.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.29875716099999999, 2024-09-21T10:19:34.859 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:19:34.859 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:19:34.859 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:34.859 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:19:34.859 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:19:34.859 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:19:34.859 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 26943 2024-09-21T10:19:34.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.860 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:19:34.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:19:34.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.495178+0000", 2024-09-21T10:19:34.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:19:34.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.495178+0000", 2024-09-21T10:19:34.860 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.860 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:19:34.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.495180+0000", 2024-09-21T10:19:34.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.64e-06 2024-09-21T10:19:34.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:19:34.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.495202+0000", 2024-09-21T10:19:34.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2310000000000002e-05 2024-09-21T10:19:34.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:19:34.861 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.495214+0000", 2024-09-21T10:19:34.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1694e-05 2024-09-21T10:19:34.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:19:34.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.495221+0000", 2024-09-21T10:19:34.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.849e-06 2024-09-21T10:19:34.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:19:34.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.495272+0000", 2024-09-21T10:19:34.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1857999999999997e-05 2024-09-21T10:19:34.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:19:34.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.495405+0000", 2024-09-21T10:19:34.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013294899999999999 2024-09-21T10:19:34.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:19:34.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.495538+0000", 2024-09-21T10:19:34.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013217599999999999 2024-09-21T10:19:34.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:19:34.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.497277+0000", 2024-09-21T10:19:34.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017389840000000001 2024-09-21T10:19:34.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.863 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:34.863 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.497659+0000", 2024-09-21T10:19:34.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038245899999999998 2024-09-21T10:19:34.863 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:34.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793846+0000", 2024-09-21T10:19:34.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.29618721399999998 2024-09-21T10:19:34.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:19:34.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793896+0000", 2024-09-21T10:19:34.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9543000000000003e-05 2024-09-21T10:19:34.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:19:34.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793935+0000", 2024-09-21T10:19:34.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9484999999999999e-05 2024-09-21T10:19:34.864 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.864 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.865 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.865 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.37050.0:26945 13.0 e372/367 13:0d6e0250:::benchmark_data_smithi107_278655_object26944:head v 372'1720, mlcod=372'1716)", 2024-09-21T10:19:34.865 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:18:31.497604+0000", 2024-09-21T10:19:34.865 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 62.935049587999998, 2024-09-21T10:19:34.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.29766990799999998, 2024-09-21T10:19:34.865 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:19:34.865 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:19:34.865 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:34.865 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:19:34.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:19:34.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.497604+0000", 2024-09-21T10:19:34.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:19:34.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.497604+0000", 2024-09-21T10:19:34.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:19:34.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.497605+0000", 2024-09-21T10:19:34.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3300000000000001e-07 2024-09-21T10:19:34.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:19:34.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.497724+0000", 2024-09-21T10:19:34.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011890799999999999 2024-09-21T10:19:34.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:19:34.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.497733+0000", 2024-09-21T10:19:34.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4110000000000002e-06 2024-09-21T10:19:34.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:19:34.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793254+0000", 2024-09-21T10:19:34.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.29552120599999998 2024-09-21T10:19:34.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:19:34.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793349+0000", 2024-09-21T10:19:34.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4561000000000005e-05 2024-09-21T10:19:34.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:19:34.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793376+0000", 2024-09-21T10:19:34.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6478e-05 2024-09-21T10:19:34.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:19:34.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.795235+0000", 2024-09-21T10:19:34.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0018596229999999999 2024-09-21T10:19:34.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:19:34.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.795274+0000", 2024-09-21T10:19:34.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8988e-05 2024-09-21T10:19:34.869 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.869 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.869 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.869 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.37050.0:26946 13.4 e372/367 13:21264d98:::benchmark_data_smithi107_278655_object26945:head v 372'1689, mlcod=372'1683)", 2024-09-21T10:19:34.869 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:18:31.498426+0000", 2024-09-21T10:19:34.869 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 62.93422786, 2024-09-21T10:19:34.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.30159318299999999, 2024-09-21T10:19:34.870 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:19:34.870 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:19:34.870 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:34.870 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:19:34.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:19:34.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.498426+0000", 2024-09-21T10:19:34.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:19:34.871 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:34 smithi107 ceph-mon[105022]: pgmap v1297: 109 pgs: 109 active+clean; 6.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:19:34.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.498426+0000", 2024-09-21T10:19:34.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:19:34.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.498427+0000", 2024-09-21T10:19:34.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9100000000000003e-07 2024-09-21T10:19:34.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:19:34.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.498492+0000", 2024-09-21T10:19:34.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5154999999999995e-05 2024-09-21T10:19:34.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:19:34.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793260+0000", 2024-09-21T10:19:34.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.294767642 2024-09-21T10:19:34.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:19:34.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793276+0000", 2024-09-21T10:19:34.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6750999999999999e-05 2024-09-21T10:19:34.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:19:34.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793802+0000", 2024-09-21T10:19:34.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052541800000000004 2024-09-21T10:19:34.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:19:34.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793818+0000", 2024-09-21T10:19:34.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5909999999999998e-05 2024-09-21T10:19:34.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:19:34.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.799979+0000", 2024-09-21T10:19:34.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0061610099999999998 2024-09-21T10:19:34.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:19:34.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.800019+0000", 2024-09-21T10:19:34.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0506000000000001e-05 2024-09-21T10:19:34.874 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.874 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.874 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.874 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.37050.0:26956 13.d e372/367 13:bd5314ea:::benchmark_data_smithi107_278655_object26955:head v 372'1699, mlcod=372'1697)", 2024-09-21T10:19:34.874 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:18:31.503163+0000", 2024-09-21T10:19:34.874 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 62.929490911000002, 2024-09-21T10:19:34.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.293366558, 2024-09-21T10:19:34.874 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:19:34.874 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:19:34.874 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:34.875 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:19:34.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:19:34.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.503163+0000", 2024-09-21T10:19:34.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:19:34.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.503163+0000", 2024-09-21T10:19:34.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:19:34.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.503164+0000", 2024-09-21T10:19:34.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.184e-06 2024-09-21T10:19:34.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:19:34.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.503213+0000", 2024-09-21T10:19:34.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8591000000000003e-05 2024-09-21T10:19:34.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:19:34.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793270+0000", 2024-09-21T10:19:34.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.29005693500000002 2024-09-21T10:19:34.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:19:34.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793280+0000", 2024-09-21T10:19:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0529e-05 2024-09-21T10:19:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:19:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793650+0000", 2024-09-21T10:19:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037032599999999998 2024-09-21T10:19:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:19:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793670+0000", 2024-09-21T10:19:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9403999999999999e-05 2024-09-21T10:19:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:19:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.796490+0000", 2024-09-21T10:19:34.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028202040000000002 2024-09-21T10:19:34.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:19:34.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.796529+0000", 2024-09-21T10:19:34.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9385000000000003e-05 2024-09-21T10:19:34.878 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.878 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.878 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.878 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:72760 13.c 13:3def1e01:::benchmark_data_smithi107_278655_object72759:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:19:34.878 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:19:00.476307+0000", 2024-09-21T10:19:34.878 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 33.956347100999999, 2024-09-21T10:19:34.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.26900290900000001, 2024-09-21T10:19:34.879 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:19:34.879 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:19:34.879 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:34.879 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:19:34.879 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:19:34.879 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:19:34.879 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 72760 2024-09-21T10:19:34.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.879 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:19:34.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:19:34.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.476307+0000", 2024-09-21T10:19:34.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:19:34.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.476307+0000", 2024-09-21T10:19:34.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:19:34.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.476308+0000", 2024-09-21T10:19:34.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.508e-06 2024-09-21T10:19:34.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:19:34.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.476331+0000", 2024-09-21T10:19:34.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2974000000000001e-05 2024-09-21T10:19:34.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:19:34.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.476343+0000", 2024-09-21T10:19:34.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1721e-05 2024-09-21T10:19:34.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:19:34.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.476350+0000", 2024-09-21T10:19:34.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1999999999999997e-06 2024-09-21T10:19:34.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:19:34.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.476891+0000", 2024-09-21T10:19:34.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00054091100000000004 2024-09-21T10:19:34.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:19:34.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.477025+0000", 2024-09-21T10:19:34.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013419300000000001 2024-09-21T10:19:34.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:19:34.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.477143+0000", 2024-09-21T10:19:34.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000117348 2024-09-21T10:19:34.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:19:34.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.479446+0000", 2024-09-21T10:19:34.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023030680000000001 2024-09-21T10:19:34.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:34.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.481877+0000", 2024-09-21T10:19:34.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002431566 2024-09-21T10:19:34.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:34.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.745214+0000", 2024-09-21T10:19:34.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.26333699100000002 2024-09-21T10:19:34.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:19:34.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.745269+0000", 2024-09-21T10:19:34.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4342000000000002e-05 2024-09-21T10:19:34.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:19:34.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.745310+0000", 2024-09-21T10:19:34.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1087000000000002e-05 2024-09-21T10:19:34.884 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.884 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.884 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.884 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:72769 13.c 13:3404e4d4:::benchmark_data_smithi107_278655_object72768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:19:34.884 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:19:00.482613+0000", 2024-09-21T10:19:34.884 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 33.950040962000003, 2024-09-21T10:19:34.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.265722128, 2024-09-21T10:19:34.885 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:19:34.885 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:19:34.885 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:19:34.885 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:19:34.885 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:19:34.885 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:19:34.885 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 72769 2024-09-21T10:19:34.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.885 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:19:34.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:19:34.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.482613+0000", 2024-09-21T10:19:34.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:19:34.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.482613+0000", 2024-09-21T10:19:34.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:19:34.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:19:34.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.482614+0000", 2024-09-21T10:19:34.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.201e-06 2024-09-21T10:19:34.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:19:34.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.482632+0000", 2024-09-21T10:19:34.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7618999999999999e-05 2024-09-21T10:19:34.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:19:34.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.482641+0000", 2024-09-21T10:19:34.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1989999999999995e-06 2024-09-21T10:19:34.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:19:34.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.482646+0000", 2024-09-21T10:19:34.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9479999999999999e-06 2024-09-21T10:19:34.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:19:34.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.482682+0000", 2024-09-21T10:19:34.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5958999999999999e-05 2024-09-21T10:19:34.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:19:34.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.482783+0000", 2024-09-21T10:19:34.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101352 2024-09-21T10:19:34.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:19:34.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.482872+0000", 2024-09-21T10:19:34.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8972000000000001e-05 2024-09-21T10:19:34.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:19:34.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.484761+0000", 2024-09-21T10:19:34.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0018889390000000001 2024-09-21T10:19:34.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:34.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.486143+0000", 2024-09-21T10:19:34.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001381848 2024-09-21T10:19:34.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:19:34.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.748241+0000", 2024-09-21T10:19:34.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.26209850499999998 2024-09-21T10:19:34.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:19:34.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.748296+0000", 2024-09-21T10:19:34.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4845000000000001e-05 2024-09-21T10:19:34.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:19:34.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:19:34.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:19:34.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:00.748335+0000", 2024-09-21T10:19:34.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8741000000000001e-05 2024-09-21T10:19:34.890 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.890 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.890 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.890 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:19:34.890 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:19:34.890 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:19:34.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:34 smithi129 ceph-mon[88471]: pgmap v1297: 109 pgs: 109 active+clean; 6.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:19:34.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:34.921 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:34.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:35.022 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:35.063 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:35.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:35.071 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 73 16 111468 111452 95.4026 62.5625 0.0165784 0.0104775 2024-09-21T10:19:35.126 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:35.165 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:35.202 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:35.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:35.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:35.304 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:35.343 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:35.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:35.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:35.444 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:35.483 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:35.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:35.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:35.522 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:35.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:19:35.585 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:35.599 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:19:35.628 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:35.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:35.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:35.730 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:35.767 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:35.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:35.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:35.799 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:35 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2329946864' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:19:35.799 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:35 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:19:35.799 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:35 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2329946864' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:19:35.799 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:35 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:19:35.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:35.912 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:35.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:35.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:35.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:35 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2329946864' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:19:35.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:35 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:19:36.014 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:36.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:36.054 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:36.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:36.071 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 74 16 112462 112446 94.9528 62.125 0.0204044 0.0105264 2024-09-21T10:19:36.155 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:36.188 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:36.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:36.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:36.290 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:36.323 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:36.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:36.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:36.424 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:36.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:36.464 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:36.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:36.565 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:36.594 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:36.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:36.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:36.697 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:36.739 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:36.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:36.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:36.841 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:36 smithi107 ceph-mon[102221]: pgmap v1298: 109 pgs: 109 active+clean; 6.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 58 MiB/s wr, 925 op/s 2024-09-21T10:19:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:36 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:36 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:19:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:36 smithi107 ceph-mon[102221]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T10:19:36.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:36 smithi107 ceph-mon[105022]: pgmap v1298: 109 pgs: 109 active+clean; 6.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 58 MiB/s wr, 925 op/s 2024-09-21T10:19:36.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:36 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:36.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:36 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:19:36.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:36 smithi107 ceph-mon[105022]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T10:19:36.874 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:36.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:36.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:36 smithi129 ceph-mon[88471]: pgmap v1298: 109 pgs: 109 active+clean; 6.6 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 58 MiB/s wr, 925 op/s 2024-09-21T10:19:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:36 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:19:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:36 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:19:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:36 smithi129 ceph-mon[88471]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T10:19:36.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:37.006 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:37.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:37.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:37.071 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 75 16 113476 113460 94.5318 63.375 0.00563089 0.010573 2024-09-21T10:19:37.108 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:37.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:37.149 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:37.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:37.250 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:37.287 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:37.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:37.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:37.389 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:37.423 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:37.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:37.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:37.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:37.551 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:37.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:37.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:37.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:37.680 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:37.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:37.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:37.782 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:37.817 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:37.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:37.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:37.919 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:37.959 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:37.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:37.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:38.061 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:38.071 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 76 16 114478 114462 94.1119 62.625 0.0167312 0.010621 2024-09-21T10:19:38.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:38.100 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:38.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:38.177 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:38.201 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:38.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:38.233 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:38.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:38.334 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:38.380 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:38.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:38.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:38.481 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:38.520 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:38.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:38.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:38.622 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:38.662 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:38.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:38.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:38.764 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:38.795 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:38.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:38.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:38 smithi107 ceph-mon[102221]: pgmap v1300: 109 pgs: 109 active+clean; 6.9 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-21T10:19:38.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:38 smithi107 ceph-mon[105022]: pgmap v1300: 109 pgs: 109 active+clean; 6.9 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-21T10:19:38.898 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:38 smithi129 ceph-mon[88471]: pgmap v1300: 109 pgs: 109 active+clean; 6.9 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-21T10:19:38.936 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:38.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:39.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:39.072 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 77 16 115474 115458 93.6973 62.25 0.0296264 0.0106675 2024-09-21T10:19:39.079 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:39.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:39.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:39.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:39.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:39.215 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:39.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:39.254 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:19:39.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:39.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:39.368 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:39.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:39.469 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:39.507 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:39.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:39.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:39.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:39.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:39.645 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:39.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:39.746 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:39.788 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:39.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:39.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:39.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:39.917 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:39.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:39.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:40.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:40.047 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:40.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:40.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:40.072 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 78 16 116441 116425 93.2709 60.4375 0.0132408 0.0107163 2024-09-21T10:19:40.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:40.185 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:40.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:40.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:40.286 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:40.329 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:40.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:40.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:40.431 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:40.470 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:40.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:40.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:40.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:40.599 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:40.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:40.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:40.638 INFO:teuthology.orchestra.run.smithi107.stderr:set pool 13 pgp_num to 16 2024-09-21T10:19:40.639 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:19:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:19:40.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:19:40.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:40.731 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:40.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:40.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:40.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:40 smithi107 ceph-mon[102221]: pgmap v1301: 109 pgs: 109 active+clean; 6.9 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-21T10:19:40.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:40 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1634413934' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:19:40.780 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:40 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:19:40.780 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:40 smithi107 ceph-mon[105022]: pgmap v1301: 109 pgs: 109 active+clean; 6.9 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-21T10:19:40.780 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:40 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1634413934' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:19:40.780 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:40 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:19:40.833 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:40.874 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:40.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:40.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:40 smithi129 ceph-mon[88471]: pgmap v1301: 109 pgs: 109 active+clean; 6.9 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-21T10:19:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:40 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1634413934' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:19:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:40 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num", "val": "16"} : dispatch 2024-09-21T10:19:40.975 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:41.001 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:41.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:41.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:41.072 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 79 16 117440 117424 92.8806 62.4375 0.017257 0.010762 2024-09-21T10:19:41.102 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:41.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:41.144 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:41.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:41.245 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:41.283 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:41.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:41.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:41.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:41.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:41.433 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:41.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:41.458 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:19:41.458 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:19:41.458 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:19:41.458 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:19:41.458 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:19:41.458 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:19:41.487 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:19:41.487 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:19:41.520 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:19:41.520 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:19:41.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:41.547 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:19:41.547 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:19:41.563 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:41.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:41.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:41.582 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:19:41.582 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:19:41.609 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:19:41.609 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:19:41.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:41.697 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:41.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:41.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:41.729 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:41 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T10:19:41.729 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:41 smithi107 ceph-mon[102221]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T10:19:41.729 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:41.729 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:41 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2147721059' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:41.730 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:41 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T10:19:41.730 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:41 smithi107 ceph-mon[105022]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T10:19:41.730 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:41.730 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:41 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2147721059' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:41.799 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:41.840 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:41.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:41.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:41 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_8", "var": "pgp_num", "val": "16"}]': finished 2024-09-21T10:19:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:41 smithi129 ceph-mon[88471]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T10:19:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:41 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2147721059' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:41.942 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:41.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:41.976 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:41.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:42.048 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:19:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:19:41] "GET /metrics HTTP/1.1" 200 36783 "" "Prometheus/2.51.0" 2024-09-21T10:19:42.072 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:19:42.071096+0000 min lat: 0.00115007 max lat: 0.53677 avg lat: 0.0108082 2024-09-21T10:19:42.072 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:19:42.072 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 80 16 118414 118398 92.4805 60.875 0.00544669 0.0108082 2024-09-21T10:19:42.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:42.107 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:42.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:42.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:42.198 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:19:42.198 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:19:42.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:42.253 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:42.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:42.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:42.354 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:42.400 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:42.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:42.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:42.502 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:42.537 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:42.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:42.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:42.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:42.674 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:42.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:42.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:42.776 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:42.809 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:42.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:42.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:42.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:42 smithi129 ceph-mon[88471]: pgmap v1303: 109 pgs: 109 active+clean; 7.0 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 47 MiB/s wr, 751 op/s 2024-09-21T10:19:42.950 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:42.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:42.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:43.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:43.070 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:42 smithi107 ceph-mon[102221]: pgmap v1303: 109 pgs: 109 active+clean; 7.0 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 47 MiB/s wr, 751 op/s 2024-09-21T10:19:43.070 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:42 smithi107 ceph-mon[105022]: pgmap v1303: 109 pgs: 109 active+clean; 7.0 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 47 MiB/s wr, 751 op/s 2024-09-21T10:19:43.072 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 81 16 119408 119392 92.1052 62.125 0.0217201 0.0108528 2024-09-21T10:19:43.093 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:43.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:43.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:43.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:43.228 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:43.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:43.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:43.330 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:43.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:43.373 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:43.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:43.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:19:43.474 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:43.532 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:43.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:43.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:43.633 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:43.671 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:43.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:43.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:43.773 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:43.809 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:43.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:43.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:43.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:43.944 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:43.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:43.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:44.045 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:44.072 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 82 16 120407 120391 91.7433 62.4375 0.0152704 0.0108951 2024-09-21T10:19:44.081 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:44.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:44.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:44.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:44.193 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:44.213 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:44.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:44.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:44.315 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:44.350 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:44.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:44.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:44.451 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:44.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:44.493 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:44.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:44.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:44.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:44.622 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:44.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:44.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:44.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:19:44.752 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:44.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:44.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:44.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:44.893 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:44.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:44.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:44.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:44 smithi129 ceph-mon[88471]: pgmap v1304: 109 pgs: 109 active+clean; 7.2 GiB data, 30 GiB used, 686 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:19:44.995 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:45.032 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:45.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:45.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:45.073 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 83 16 121389 121373 91.3774 61.375 0.0159032 0.0109389 2024-09-21T10:19:45.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:44 smithi107 ceph-mon[105022]: pgmap v1304: 109 pgs: 109 active+clean; 7.2 GiB data, 30 GiB used, 686 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:19:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:44 smithi107 ceph-mon[102221]: pgmap v1304: 109 pgs: 109 active+clean; 7.2 GiB data, 30 GiB used, 686 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-21T10:19:45.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:45.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:45.164 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:45.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:45.265 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:45.295 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:45.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:45.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:45.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:45.430 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:45.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:45.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:45.442 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:45.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:45.554 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:45.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:45.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:45.655 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:45.683 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:45.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:45.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:45.785 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:45.806 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:45.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:45.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:45.908 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:45.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:45.929 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:45.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:46.030 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:46.059 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:46.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:46.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:46.073 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 84 16 122098 122082 90.8171 44.3125 0.0122266 0.0110064 2024-09-21T10:19:46.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:46.195 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:46.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:46.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:46.296 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:46.339 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:46.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:46.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:46.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:46.469 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:46.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:46.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:46.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:46.601 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:46.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:46.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:46.702 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:46.743 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:46.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:46.845 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:46.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:46.888 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:46.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:46 smithi129 ceph-mon[88471]: pgmap v1305: 109 pgs: 109 active+clean; 7.2 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T10:19:46.989 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:47.028 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:47.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:47.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:47.073 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 85 16 123092 123076 90.4796 62.125 0.0069996 0.0110474 2024-09-21T10:19:47.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:46 smithi107 ceph-mon[105022]: pgmap v1305: 109 pgs: 109 active+clean; 7.2 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T10:19:47.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:46 smithi107 ceph-mon[102221]: pgmap v1305: 109 pgs: 109 active+clean; 7.2 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T10:19:47.130 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:47.171 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:47.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:47.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:47.273 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:47.307 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:47.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:47.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:47.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:47.439 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:47.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:47.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:47.540 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:47.574 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:47.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:47.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:47.676 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:47.718 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:47.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:47.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:47.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:47.846 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:47.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:47.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:47.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:47.974 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:47.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:47.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:48.073 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 86 16 124075 124059 90.1418 61.4375 0.0168128 0.0110891 2024-09-21T10:19:48.075 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:48.116 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:48.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:48.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:48.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:48.252 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:48.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:48.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:48.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:48.383 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:48.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:48.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:48.484 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:48.524 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:48.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:48.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:48.625 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:48.660 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:48.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:48.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:48.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:48.798 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:48.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:48.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:48.899 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:48.932 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:48.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:48.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:48.990 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:48 smithi107 ceph-mon[102221]: pgmap v1306: 109 pgs: 109 active+clean; 7.5 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 61 MiB/s wr, 978 op/s 2024-09-21T10:19:48.991 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:48 smithi107 ceph-mon[105022]: pgmap v1306: 109 pgs: 109 active+clean; 7.5 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 61 MiB/s wr, 978 op/s 2024-09-21T10:19:49.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:49.073 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:49.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:49.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:49.074 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 87 16 125036 125020 89.7954 60.0625 0.0197186 0.0111313 2024-09-21T10:19:49.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:48 smithi129 ceph-mon[88471]: pgmap v1306: 109 pgs: 109 active+clean; 7.5 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 61 MiB/s wr, 978 op/s 2024-09-21T10:19:49.175 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:49.177 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:19:49.177 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:19:49.178 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 5, 6, 7, 4] 2024-09-21T10:19:49.178 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd out 6 2024-09-21T10:19:49.211 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:49.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:49.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:49.312 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:49.344 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:49.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:49.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:49.446 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:49.475 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:49.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:49.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:49.577 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:49.610 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:49.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:49.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:49.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:49.750 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:49.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:49.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:49.795 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:49.852 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:49.893 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:49.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:49.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:49.995 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:50.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:50.031 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:50.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:50.074 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 88 16 125974 125958 89.4412 58.625 0.00954475 0.0111756 2024-09-21T10:19:50.133 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:50.172 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:50.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:50.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:50.274 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:50.312 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:50.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:50.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:50.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:50.446 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:50.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:50.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:50.548 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:50.583 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:50.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:50.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:19:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:19:50.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:19:50.684 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:50.686 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:19:50.720 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:50.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:50.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:50.822 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:50.859 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:50.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:50.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:50.961 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:50.999 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:51.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:51.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:51.074 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 89 16 126942 126926 89.116 60.5 0.0202124 0.0112164 2024-09-21T10:19:51.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:50 smithi107 ceph-mon[102221]: pgmap v1307: 109 pgs: 109 active+clean; 7.5 GiB data, 31 GiB used, 685 GiB / 715 GiB avail; 61 MiB/s wr, 969 op/s 2024-09-21T10:19:51.081 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:50 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3899404643' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:19:51.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:50 smithi107 ceph-mon[105022]: pgmap v1307: 109 pgs: 109 active+clean; 7.5 GiB data, 31 GiB used, 685 GiB / 715 GiB avail; 61 MiB/s wr, 969 op/s 2024-09-21T10:19:51.082 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:50 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3899404643' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:19:51.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:51.139 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:51.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:51.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:51.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:50 smithi129 ceph-mon[88471]: pgmap v1307: 109 pgs: 109 active+clean; 7.5 GiB data, 31 GiB used, 685 GiB / 715 GiB avail; 61 MiB/s wr, 969 op/s 2024-09-21T10:19:51.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:50 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3899404643' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:19:51.241 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:51.280 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:51.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:51.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:51.382 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:51.408 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:51.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:51.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:51.509 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:51.544 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:51.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:51.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:51.646 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:51.690 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:51.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:51.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:51.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:51.828 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:51.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:51.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:51.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:51 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:19:51.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:51 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:19:51.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:51 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3899404643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:19:51.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:51 smithi107 ceph-mon[102221]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T10:19:51.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:19:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:19:51] "GET /metrics HTTP/1.1" 200 36783 "" "Prometheus/2.51.0" 2024-09-21T10:19:51.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:51 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:19:51.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:51 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:19:51.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:51 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3899404643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:19:51.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:51 smithi107 ceph-mon[105022]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T10:19:51.929 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:51.968 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:51.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:51.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:52.070 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:52.074 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total time run: 90.0081 2024-09-21T10:19:52.074 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total writes made: 127904 2024-09-21T10:19:52.074 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Write size: 65536 2024-09-21T10:19:52.074 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object size: 65536 2024-09-21T10:19:52.074 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Bandwidth (MB/sec): 88.8143 2024-09-21T10:19:52.074 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Bandwidth: 38.6687 2024-09-21T10:19:52.075 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max bandwidth (MB/sec): 247.125 2024-09-21T10:19:52.075 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min bandwidth (MB/sec): 44.3125 2024-09-21T10:19:52.075 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average IOPS: 1421 2024-09-21T10:19:52.075 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev IOPS: 618.699 2024-09-21T10:19:52.075 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max IOPS: 3954 2024-09-21T10:19:52.075 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min IOPS: 709 2024-09-21T10:19:52.075 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average Latency(s): 0.0112555 2024-09-21T10:19:52.075 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Latency(s): 0.011643 2024-09-21T10:19:52.075 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max latency(s): 0.53677 2024-09-21T10:19:52.075 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min latency(s): 0.00115007 2024-09-21T10:19:52.080 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:19:52.101 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:52.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:52.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:52.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:51 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:19:52.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:51 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:19:52.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:51 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3899404643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:19:52.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:51 smithi129 ceph-mon[88471]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T10:19:52.202 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:52.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:52.233 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:52.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:52.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:52.370 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:52.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:52.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:52.472 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:52.506 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:52.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:52.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:52.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:52.647 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:52.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:52.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:52.749 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:52.778 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:52.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:52.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:52.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:52 smithi107 ceph-mon[102221]: pgmap v1309: 109 pgs: 109 active+clean; 7.6 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 60 MiB/s wr, 962 op/s 2024-09-21T10:19:52.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:52 smithi107 ceph-mon[105022]: pgmap v1309: 109 pgs: 109 active+clean; 7.6 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 60 MiB/s wr, 962 op/s 2024-09-21T10:19:52.879 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:52.917 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:52.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:52.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:53.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:53.050 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:53.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:53.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:53.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:53.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:52 smithi129 ceph-mon[88471]: pgmap v1309: 109 pgs: 109 active+clean; 7.6 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 60 MiB/s wr, 962 op/s 2024-09-21T10:19:53.190 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:53.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:53.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:53.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:53.330 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:53.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:53.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:53.361 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:19:53.361 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:19:53.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:53.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:53.464 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:53.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:53.565 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:53.595 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:53.597 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:53.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:53.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:53.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:53.727 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:53.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:53.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:53.829 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:53.863 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:53.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:53.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:53.964 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:53.999 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:53.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:54.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:54.094 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:19:54.094 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:19:54.101 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:54.128 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:19:54.128 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:19:54.134 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:54.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:54.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:54.161 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:19:54.161 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:19:54.190 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:19:54.190 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:19:54.221 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:19:54.221 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:19:54.236 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:54.251 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:19:54.251 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:19:54.270 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:54.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:54.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:54.277 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:19:54.277 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:19:54.308 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:19:54.308 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:19:54.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:54.411 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:54.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:54.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:54.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:54.549 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:54.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:54.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:54.615 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:19:54.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:54.705 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:54.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:54.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:54.806 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:54.839 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:54.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:54.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:54.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:54.979 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:54.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:54.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:55.080 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:55.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:54 smithi107 ceph-mon[102221]: pgmap v1310: 109 pgs: 109 active+clean; 7.8 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 59 MiB/s wr, 955 op/s 2024-09-21T10:19:55.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:54 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1735958746' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:55.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:54 smithi107 ceph-mon[105022]: pgmap v1310: 109 pgs: 109 active+clean; 7.8 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 59 MiB/s wr, 955 op/s 2024-09-21T10:19:55.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:54 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1735958746' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:55.112 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:55.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:55.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:54 smithi129 ceph-mon[88471]: pgmap v1310: 109 pgs: 109 active+clean; 7.8 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 59 MiB/s wr, 955 op/s 2024-09-21T10:19:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:54 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1735958746' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:19:55.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:19:55.215 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:55.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:55.256 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:55.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:55.357 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:55.383 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:55.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:55.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:55.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:55.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:55.556 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:55.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:55.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:55.692 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:55.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:55.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:55.793 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:55.829 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:55.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:55.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:55.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:55.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:55.969 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:55.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:56.070 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:56.121 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:56.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:56.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:56.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:56.262 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:56.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:56.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:56.364 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:56.399 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:56.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:56.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:56.500 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:56.537 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:56.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:56.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:56.615 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:56.635 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:19:56.638 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:56.664 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:56.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:56.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:56.766 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:56.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:56 smithi107 ceph-mon[102221]: pgmap v1311: 109 pgs: 109 active+clean; 7.8 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 57 MiB/s wr, 957 op/s 2024-09-21T10:19:56.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:19:56.771 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:56.772 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:56 smithi107 ceph-mon[105022]: pgmap v1311: 109 pgs: 109 active+clean; 7.8 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 57 MiB/s wr, 957 op/s 2024-09-21T10:19:56.772 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:19:56.772 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:56.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:56.802 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:56.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:56.902 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:56.934 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:56.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:56.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:57.036 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:57.073 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:57.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:57.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:56 smithi129 ceph-mon[88471]: pgmap v1311: 109 pgs: 109 active+clean; 7.8 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 57 MiB/s wr, 957 op/s 2024-09-21T10:19:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:19:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:57.175 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:57.212 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:57.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:57.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:57.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:57.344 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:57.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:57.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:57.446 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:57.496 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:57.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:57.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:57.598 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:57.636 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:57.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:57.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:57.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:57.774 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:57.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:57.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:57.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:57.911 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:57.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:57.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:57.917 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:57 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a[102198]: 2024-09-21T10:19:57.902+0000 7f3189124640 -1 mon.a@0(leader).osd e379 definitely_dead 0 2024-09-21T10:19:58.013 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:58.051 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:58.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:58.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:19:58.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:58.185 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:58.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:58.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:58.286 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:19:58.318 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:58.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:58.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:19:58.420 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:58.447 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:58.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:58.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:58.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:19:58.576 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:58.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:58.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:19:58.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:19:58.718 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:58.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:58.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:19:58.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:58.841 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:58.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:58.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:58.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:58.964 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:58.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:58.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:19:59.066 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:19:59.088 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:59.150 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:58 smithi107 ceph-mon[102221]: pgmap v1312: 109 pgs: 109 active+clean; 7.6 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 102 B/s rd, 35 MiB/s wr, 966 op/s 2024-09-21T10:19:59.150 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:58 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3431675529' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-21T10:19:59.150 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:58 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-21T10:19:59.150 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:58 smithi107 ceph-mon[102221]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-21T10:19:59.151 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:58 smithi107 ceph-mon[105022]: pgmap v1312: 109 pgs: 109 active+clean; 7.6 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 102 B/s rd, 35 MiB/s wr, 966 op/s 2024-09-21T10:19:59.151 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:58 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3431675529' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-21T10:19:59.151 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:58 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-21T10:19:59.151 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:58 smithi107 ceph-mon[105022]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-21T10:19:59.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:59.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:19:59.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:58 smithi129 ceph-mon[88471]: pgmap v1312: 109 pgs: 109 active+clean; 7.6 GiB data, 31 GiB used, 684 GiB / 715 GiB avail; 102 B/s rd, 35 MiB/s wr, 966 op/s 2024-09-21T10:19:59.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:58 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3431675529' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-21T10:19:59.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:58 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["6"]} : dispatch 2024-09-21T10:19:59.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:58 smithi129 ceph-mon[88471]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-21T10:19:59.187 INFO:teuthology.orchestra.run.smithi107.stderr:marked out osd.6. 2024-09-21T10:19:59.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:59.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:59.276 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:59.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:59.377 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:19:59.402 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:59.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:59.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:19:59.504 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:59.529 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:59.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:59.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:59.631 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:19:59.655 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:19:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:59.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:19:59.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:19:59.778 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:59.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:59.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:19:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:59 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-21T10:19:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:59 smithi107 ceph-mon[102221]: osdmap e380: 8 total, 8 up, 7 in 2024-09-21T10:19:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:19:59 smithi107 ceph-mon[102221]: osdmap e381: 8 total, 8 up, 7 in 2024-09-21T10:19:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:59 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-21T10:19:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:59 smithi107 ceph-mon[105022]: osdmap e380: 8 total, 8 up, 7 in 2024-09-21T10:19:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:19:59 smithi107 ceph-mon[105022]: osdmap e381: 8 total, 8 up, 7 in 2024-09-21T10:19:59.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:19:59.900 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:19:59.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:19:59.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:00.002 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:00.028 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:00.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:00.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:00.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:00.151 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:00.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:00.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:00.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:59 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-21T10:20:00.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:59 smithi129 ceph-mon[88471]: osdmap e380: 8 total, 8 up, 7 in 2024-09-21T10:20:00.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:19:59 smithi129 ceph-mon[88471]: osdmap e381: 8 total, 8 up, 7 in 2024-09-21T10:20:00.254 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:00.280 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:00.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:00.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:00.382 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:00.407 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:00.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:00.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:00.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:00.531 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:00.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:00.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:00.633 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:00.660 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:00.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:00.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:20:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:20:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:20:00.762 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:00.798 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:00.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:00.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:00.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:00.929 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:00.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:00.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:01.031 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:01.059 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:01.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:01.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:01.079 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:00 smithi107 ceph-mon[102221]: pgmap v1314: 109 pgs: 109 active+clean; 7.5 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 116 B/s rd, 32 MiB/s wr, 1.04k op/s 2024-09-21T10:20:01.079 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:00 smithi107 ceph-mon[102221]: overall HEALTH_OK 2024-09-21T10:20:01.079 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:00 smithi107 ceph-mon[102221]: osdmap e382: 8 total, 8 up, 7 in 2024-09-21T10:20:01.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:00 smithi107 ceph-mon[105022]: pgmap v1314: 109 pgs: 109 active+clean; 7.5 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 116 B/s rd, 32 MiB/s wr, 1.04k op/s 2024-09-21T10:20:01.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:00 smithi107 ceph-mon[105022]: overall HEALTH_OK 2024-09-21T10:20:01.081 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:00 smithi107 ceph-mon[105022]: osdmap e382: 8 total, 8 up, 7 in 2024-09-21T10:20:01.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:00 smithi129 ceph-mon[88471]: pgmap v1314: 109 pgs: 109 active+clean; 7.5 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 116 B/s rd, 32 MiB/s wr, 1.04k op/s 2024-09-21T10:20:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:00 smithi129 ceph-mon[88471]: overall HEALTH_OK 2024-09-21T10:20:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:00 smithi129 ceph-mon[88471]: osdmap e382: 8 total, 8 up, 7 in 2024-09-21T10:20:01.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:01.205 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:01.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:01.306 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:01.345 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:01.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:01.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:01.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:01.473 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:01.482 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:20:01.483 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-09-21T10:20:01.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-09-21T10:20:01.483 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:20:01.483 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.483 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135317 12.10 12:0b2cf212:::benchmark_data_smithi107_250110_object23907:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:20:01.483 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581449+0000", 2024-09-21T10:20:01.483 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 230.89355560600001, 2024-09-21T10:20:01.483 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4781515190000001, 2024-09-21T10:20:01.483 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:01.483 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:01.483 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:01.483 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:01.483 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:01.484 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:01.484 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135317 2024-09-21T10:20:01.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.484 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:01.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.484 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:01.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581449+0000", 2024-09-21T10:20:01.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.484 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.484 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:01.484 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581449+0000", 2024-09-21T10:20:01.484 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.484 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:01.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581449+0000", 2024-09-21T10:20:01.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3099999999999998e-07 2024-09-21T10:20:01.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:01.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581451+0000", 2024-09-21T10:20:01.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0360000000000001e-06 2024-09-21T10:20:01.485 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.485 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.485 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:01.485 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581458+0000", 2024-09-21T10:20:01.485 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5169999999999996e-06 2024-09-21T10:20:01.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.486 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:01.486 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581462+0000", 2024-09-21T10:20:01.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6670000000000004e-06 2024-09-21T10:20:01.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.486 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.486 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.486 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581639+0000", 2024-09-21T10:20:01.486 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017724200000000001 2024-09-21T10:20:01.486 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581651+0000", 2024-09-21T10:20:01.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1234e-05 2024-09-21T10:20:01.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.056555+0000", 2024-09-21T10:20:01.487 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.474904277 2024-09-21T10:20:01.487 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.487 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.487 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:01.487 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.056630+0000", 2024-09-21T10:20:01.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5166000000000005e-05 2024-09-21T10:20:01.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.488 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,1,6,7", 2024-09-21T10:20:01.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.056875+0000", 2024-09-21T10:20:01.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024485799999999998 2024-09-21T10:20:01.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.488 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.059202+0000", 2024-09-21T10:20:01.488 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023267159999999999 2024-09-21T10:20:01.488 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.488 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.488 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.488 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.059258+0000", 2024-09-21T10:20:01.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6103999999999997e-05 2024-09-21T10:20:01.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.059322+0000", 2024-09-21T10:20:01.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3839999999999999e-05 2024-09-21T10:20:01.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.059391+0000", 2024-09-21T10:20:01.489 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9194000000000002e-05 2024-09-21T10:20:01.489 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.489 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.489 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:01.489 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.059404+0000", 2024-09-21T10:20:01.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3509999999999999e-05 2024-09-21T10:20:01.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:01.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.059471+0000", 2024-09-21T10:20:01.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6470000000000006e-05 2024-09-21T10:20:01.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.490 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.490 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:01.490 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.059600+0000", 2024-09-21T10:20:01.490 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000129257 2024-09-21T10:20:01.490 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.490 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.490 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.490 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.491 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.491 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135369 12.10 12:0ae61838:::benchmark_data_smithi107_250110_object23959:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:20:01.491 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581471+0000", 2024-09-21T10:20:01.491 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 230.89353307600001, 2024-09-21T10:20:01.491 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.489676792, 2024-09-21T10:20:01.491 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:01.491 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:01.491 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:01.491 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:01.491 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:01.491 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:01.491 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135369 2024-09-21T10:20:01.491 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.491 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:01.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.492 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:01.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581471+0000", 2024-09-21T10:20:01.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.492 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:01.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581471+0000", 2024-09-21T10:20:01.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.492 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.492 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:01.492 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581472+0000", 2024-09-21T10:20:01.492 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1600000000000002e-07 2024-09-21T10:20:01.492 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:01.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581474+0000", 2024-09-21T10:20:01.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1430000000000001e-06 2024-09-21T10:20:01.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:01.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581481+0000", 2024-09-21T10:20:01.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1999999999999997e-06 2024-09-21T10:20:01.493 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.493 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.493 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:01.493 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581486+0000", 2024-09-21T10:20:01.493 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9219999999999996e-06 2024-09-21T10:20:01.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581675+0000", 2024-09-21T10:20:01.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018961 2024-09-21T10:20:01.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581679+0000", 2024-09-21T10:20:01.494 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5250000000000001e-06 2024-09-21T10:20:01.494 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.494 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.494 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.494 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.059634+0000", 2024-09-21T10:20:01.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4779546569999999 2024-09-21T10:20:01.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:01.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.059769+0000", 2024-09-21T10:20:01.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013499599999999999 2024-09-21T10:20:01.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,1,6,7", 2024-09-21T10:20:01.495 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.060043+0000", 2024-09-21T10:20:01.495 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00027409400000000001 2024-09-21T10:20:01.495 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.495 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.495 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.496 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.064993+0000", 2024-09-21T10:20:01.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0049500179999999996 2024-09-21T10:20:01.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.496 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.496 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.065037+0000", 2024-09-21T10:20:01.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4659000000000001e-05 2024-09-21T10:20:01.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.496 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.496 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.065079+0000", 2024-09-21T10:20:01.496 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2101999999999998e-05 2024-09-21T10:20:01.496 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.496 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:01.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.066478+0000", 2024-09-21T10:20:01.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001398329 2024-09-21T10:20:01.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.070931+0000", 2024-09-21T10:20:01.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044529979999999997 2024-09-21T10:20:01.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:01.497 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.071000+0000", 2024-09-21T10:20:01.497 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8955000000000006e-05 2024-09-21T10:20:01.497 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.497 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.497 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:01.498 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.071148+0000", 2024-09-21T10:20:01.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014816800000000001 2024-09-21T10:20:01.498 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.498 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.498 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.498 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.498 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.498 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135398 12.10 12:08ac01ae:::benchmark_data_smithi107_250110_object23988:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:20:01.498 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581512+0000", 2024-09-21T10:20:01.498 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 230.89349232999999, 2024-09-21T10:20:01.498 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4900354410000001, 2024-09-21T10:20:01.498 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:01.498 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:01.498 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:01.499 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:01.499 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:01.499 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:01.499 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135398 2024-09-21T10:20:01.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.499 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:01.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:01.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581512+0000", 2024-09-21T10:20:01.499 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.499 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.499 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.499 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:01.499 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581512+0000", 2024-09-21T10:20:01.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:01.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581512+0000", 2024-09-21T10:20:01.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6899999999999998e-07 2024-09-21T10:20:01.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:01.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581515+0000", 2024-09-21T10:20:01.500 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.244e-06 2024-09-21T10:20:01.500 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.500 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.500 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:01.500 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581521+0000", 2024-09-21T10:20:01.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.268e-06 2024-09-21T10:20:01.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:01.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581526+0000", 2024-09-21T10:20:01.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4870000000000002e-06 2024-09-21T10:20:01.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581926+0000", 2024-09-21T10:20:01.501 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040013100000000002 2024-09-21T10:20:01.501 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.501 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.501 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.501 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581932+0000", 2024-09-21T10:20:01.502 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9529999999999996e-06 2024-09-21T10:20:01.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.502 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.502 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.061470+0000", 2024-09-21T10:20:01.502 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.479537267 2024-09-21T10:20:01.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.502 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:01.502 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.061576+0000", 2024-09-21T10:20:01.502 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010591599999999999 2024-09-21T10:20:01.502 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.502 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.502 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,1,6,7", 2024-09-21T10:20:01.503 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.061783+0000", 2024-09-21T10:20:01.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00020727000000000001 2024-09-21T10:20:01.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.503 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.503 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.065126+0000", 2024-09-21T10:20:01.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003343192 2024-09-21T10:20:01.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.503 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.503 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.066357+0000", 2024-09-21T10:20:01.503 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001230987 2024-09-21T10:20:01.503 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.503 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.503 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.504 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.066392+0000", 2024-09-21T10:20:01.504 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5034999999999999e-05 2024-09-21T10:20:01.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.504 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:01.504 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.066484+0000", 2024-09-21T10:20:01.504 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1453999999999995e-05 2024-09-21T10:20:01.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.504 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.504 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.071353+0000", 2024-09-21T10:20:01.504 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048698889999999996 2024-09-21T10:20:01.504 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.504 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:01.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.071431+0000", 2024-09-21T10:20:01.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7381999999999999e-05 2024-09-21T10:20:01.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.505 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:01.505 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.071547+0000", 2024-09-21T10:20:01.505 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011649700000000001 2024-09-21T10:20:01.505 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.505 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.505 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.505 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.505 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.505 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135454 12.10 12:0c70ad69:::benchmark_data_smithi107_250110_object24044:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:20:01.505 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581549+0000", 2024-09-21T10:20:01.506 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 230.893455343, 2024-09-21T10:20:01.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4902949400000001, 2024-09-21T10:20:01.506 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:01.506 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:01.506 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:01.506 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:01.506 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:01.506 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:01.506 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135454 2024-09-21T10:20:01.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.506 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:01.506 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.506 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:01.506 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581549+0000", 2024-09-21T10:20:01.506 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.506 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.507 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:01.507 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581549+0000", 2024-09-21T10:20:01.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.507 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:01.507 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581549+0000", 2024-09-21T10:20:01.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3999999999999997e-07 2024-09-21T10:20:01.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.507 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.507 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:01.507 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581551+0000", 2024-09-21T10:20:01.507 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.725e-06 2024-09-21T10:20:01.507 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.508 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.508 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:01.508 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581555+0000", 2024-09-21T10:20:01.508 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2459999999999997e-06 2024-09-21T10:20:01.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.508 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.508 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:01.508 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581559+0000", 2024-09-21T10:20:01.508 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1919999999999998e-06 2024-09-21T10:20:01.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.508 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.508 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.508 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581965+0000", 2024-09-21T10:20:01.508 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040525199999999999 2024-09-21T10:20:01.508 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.509 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.509 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.509 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581969+0000", 2024-09-21T10:20:01.509 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.673e-06 2024-09-21T10:20:01.509 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.509 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.509 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.509 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.063322+0000", 2024-09-21T10:20:01.510 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4813525160000001 2024-09-21T10:20:01.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.510 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:01.510 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.063444+0000", 2024-09-21T10:20:01.510 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012229100000000001 2024-09-21T10:20:01.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.510 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,1,6,7", 2024-09-21T10:20:01.510 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.063653+0000", 2024-09-21T10:20:01.510 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00020935100000000001 2024-09-21T10:20:01.510 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.511 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.511 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.065994+0000", 2024-09-21T10:20:01.511 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023409400000000001 2024-09-21T10:20:01.511 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.511 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.511 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.511 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.066461+0000", 2024-09-21T10:20:01.511 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046649400000000002 2024-09-21T10:20:01.511 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.511 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.511 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:01.511 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.066507+0000", 2024-09-21T10:20:01.511 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6591000000000002e-05 2024-09-21T10:20:01.511 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.511 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.512 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.512 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.071312+0000", 2024-09-21T10:20:01.512 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004804656 2024-09-21T10:20:01.512 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.512 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.512 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.512 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.071710+0000", 2024-09-21T10:20:01.512 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00039748399999999998 2024-09-21T10:20:01.512 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.512 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.512 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:01.512 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.071747+0000", 2024-09-21T10:20:01.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7051999999999997e-05 2024-09-21T10:20:01.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:01.513 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.071844+0000", 2024-09-21T10:20:01.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7137000000000002e-05 2024-09-21T10:20:01.513 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.513 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.513 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.513 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135465 12.10 12:0edb3f1f:::benchmark_data_smithi107_250110_object24055:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:20:01.513 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581578+0000", 2024-09-21T10:20:01.513 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 230.893425896, 2024-09-21T10:20:01.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4904590689999999, 2024-09-21T10:20:01.514 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:01.514 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:01.514 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:01.514 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:01.514 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:01.514 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:01.514 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135465 2024-09-21T10:20:01.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.514 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:01.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:01.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581578+0000", 2024-09-21T10:20:01.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:01.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581578+0000", 2024-09-21T10:20:01.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:01.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581579+0000", 2024-09-21T10:20:01.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.34e-07 2024-09-21T10:20:01.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.516 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:01.516 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581581+0000", 2024-09-21T10:20:01.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9520000000000001e-06 2024-09-21T10:20:01.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.516 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:01.516 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581586+0000", 2024-09-21T10:20:01.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7230000000000001e-06 2024-09-21T10:20:01.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.516 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:01.516 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581591+0000", 2024-09-21T10:20:01.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7060000000000003e-06 2024-09-21T10:20:01.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.517 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581992+0000", 2024-09-21T10:20:01.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040077299999999999 2024-09-21T10:20:01.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.517 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581996+0000", 2024-09-21T10:20:01.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8039999999999999e-06 2024-09-21T10:20:01.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.517 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.517 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.066581+0000", 2024-09-21T10:20:01.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.48458494 2024-09-21T10:20:01.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.518 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:01.518 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.066700+0000", 2024-09-21T10:20:01.518 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119287 2024-09-21T10:20:01.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.518 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,1,6,7", 2024-09-21T10:20:01.518 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.066972+0000", 2024-09-21T10:20:01.518 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00027198700000000001 2024-09-21T10:20:01.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.518 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:01.518 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.071162+0000", 2024-09-21T10:20:01.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041900050000000001 2024-09-21T10:20:01.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.519 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.519 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.071205+0000", 2024-09-21T10:20:01.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3307999999999999e-05 2024-09-21T10:20:01.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.519 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.519 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.071263+0000", 2024-09-21T10:20:01.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7538999999999999e-05 2024-09-21T10:20:01.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.519 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.520 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.071884+0000", 2024-09-21T10:20:01.520 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00062166800000000003 2024-09-21T10:20:01.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.520 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.071919+0000", 2024-09-21T10:20:01.520 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4687000000000002e-05 2024-09-21T10:20:01.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:01.520 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.071954+0000", 2024-09-21T10:20:01.520 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4795999999999997e-05 2024-09-21T10:20:01.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.521 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:01.521 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.072037+0000", 2024-09-21T10:20:01.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3460000000000004e-05 2024-09-21T10:20:01.521 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.521 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.521 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.521 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135467 12.10 12:0aa7716b:::benchmark_data_smithi107_250110_object24057:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:20:01.521 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581602+0000", 2024-09-21T10:20:01.521 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 230.89340220299999, 2024-09-21T10:20:01.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.490699421, 2024-09-21T10:20:01.521 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:01.522 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:01.522 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:01.522 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:01.522 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:01.522 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:01.522 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135467 2024-09-21T10:20:01.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.522 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:01.522 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.522 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:01.522 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581602+0000", 2024-09-21T10:20:01.522 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.522 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.523 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:01.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581602+0000", 2024-09-21T10:20:01.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.523 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:01.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581602+0000", 2024-09-21T10:20:01.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:01.524 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:01.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:01.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3099999999999999e-07 2024-09-21T10:20:01.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:01.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581604+0000", 2024-09-21T10:20:01.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5030000000000001e-06 2024-09-21T10:20:01.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.525 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:01.525 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581608+0000", 2024-09-21T10:20:01.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0219999999999998e-06 2024-09-21T10:20:01.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.526 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:01.526 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581611+0000", 2024-09-21T10:20:01.526 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9239999999999999e-06 2024-09-21T10:20:01.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.526 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.526 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582015+0000", 2024-09-21T10:20:01.526 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000404688 2024-09-21T10:20:01.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.526 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.526 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.526 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582019+0000", 2024-09-21T10:20:01.526 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4379999999999999e-06 2024-09-21T10:20:01.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.527 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.527 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.069101+0000", 2024-09-21T10:20:01.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.487082416 2024-09-21T10:20:01.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.527 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:01.527 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.069210+0000", 2024-09-21T10:20:01.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000108189 2024-09-21T10:20:01.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.527 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,1,6,7", 2024-09-21T10:20:01.527 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.069428+0000", 2024-09-21T10:20:01.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000218373 2024-09-21T10:20:01.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.528 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:01.528 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.071218+0000", 2024-09-21T10:20:01.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017902840000000001 2024-09-21T10:20:01.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.528 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.528 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.071616+0000", 2024-09-21T10:20:01.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000397506 2024-09-21T10:20:01.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.528 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.528 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.071677+0000", 2024-09-21T10:20:01.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0850000000000002e-05 2024-09-21T10:20:01.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.072082+0000", 2024-09-21T10:20:01.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040524199999999999 2024-09-21T10:20:01.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.072137+0000", 2024-09-21T10:20:01.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5621999999999997e-05 2024-09-21T10:20:01.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:01.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.072194+0000", 2024-09-21T10:20:01.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6297e-05 2024-09-21T10:20:01.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:01.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.072301+0000", 2024-09-21T10:20:01.530 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000107736 2024-09-21T10:20:01.530 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.530 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.530 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.530 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135469 12.10 12:09240cb7:::benchmark_data_smithi107_250110_object24059:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:20:01.530 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581614+0000", 2024-09-21T10:20:01.530 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 230.89338975300001, 2024-09-21T10:20:01.530 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.508036782, 2024-09-21T10:20:01.531 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:01.531 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:01.531 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:01.531 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:01.531 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:01.531 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:01.531 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135469 2024-09-21T10:20:01.531 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.531 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:01.531 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.531 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:01.531 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581614+0000", 2024-09-21T10:20:01.531 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.532 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.532 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:01.532 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581614+0000", 2024-09-21T10:20:01.532 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.532 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.532 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:01.532 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581615+0000", 2024-09-21T10:20:01.532 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0600000000000001e-07 2024-09-21T10:20:01.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.532 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.532 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:01.532 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581616+0000", 2024-09-21T10:20:01.532 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4500000000000001e-06 2024-09-21T10:20:01.533 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.533 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.533 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:01.533 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581622+0000", 2024-09-21T10:20:01.533 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5289999999999999e-06 2024-09-21T10:20:01.533 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.533 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.533 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:01.533 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581627+0000", 2024-09-21T10:20:01.533 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5459999999999999e-06 2024-09-21T10:20:01.533 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.533 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.533 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.533 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582038+0000", 2024-09-21T10:20:01.533 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041066600000000001 2024-09-21T10:20:01.534 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.534 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.534 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.534 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582041+0000", 2024-09-21T10:20:01.534 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3419999999999999e-06 2024-09-21T10:20:01.534 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.534 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.534 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.534 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.072339+0000", 2024-09-21T10:20:01.534 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.490297354 2024-09-21T10:20:01.534 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.534 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.534 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:01.534 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.072449+0000", 2024-09-21T10:20:01.535 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000110358 2024-09-21T10:20:01.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.535 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,1,6,7", 2024-09-21T10:20:01.535 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.072697+0000", 2024-09-21T10:20:01.535 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024780400000000001 2024-09-21T10:20:01.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.535 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.535 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.079424+0000", 2024-09-21T10:20:01.535 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0067273080000000004 2024-09-21T10:20:01.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.535 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:01.535 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.079437+0000", 2024-09-21T10:20:01.536 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2486e-05 2024-09-21T10:20:01.536 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.536 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.536 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.536 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.079479+0000", 2024-09-21T10:20:01.536 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1922000000000003e-05 2024-09-21T10:20:01.536 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.536 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.536 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.536 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.079514+0000", 2024-09-21T10:20:01.536 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5858000000000001e-05 2024-09-21T10:20:01.536 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.536 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.536 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.536 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.089458+0000", 2024-09-21T10:20:01.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0099432140000000006 2024-09-21T10:20:01.537 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.537 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.537 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:01.537 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.089516+0000", 2024-09-21T10:20:01.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8891000000000002e-05 2024-09-21T10:20:01.537 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.537 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.537 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:01.537 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.089651+0000", 2024-09-21T10:20:01.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013474799999999999 2024-09-21T10:20:01.537 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.537 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.537 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.537 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135559 12.10 12:09651379:::benchmark_data_smithi107_250110_object24149:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:20:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581633+0000", 2024-09-21T10:20:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 230.89337111699999, 2024-09-21T10:20:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5083316020000002, 2024-09-21T10:20:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135559 2024-09-21T10:20:01.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581633+0000", 2024-09-21T10:20:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581633+0000", 2024-09-21T10:20:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:01.539 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581633+0000", 2024-09-21T10:20:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5999999999999999e-07 2024-09-21T10:20:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581639+0000", 2024-09-21T10:20:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.587e-06 2024-09-21T10:20:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581648+0000", 2024-09-21T10:20:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2490000000000007e-06 2024-09-21T10:20:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:01.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581653+0000", 2024-09-21T10:20:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1080000000000001e-06 2024-09-21T10:20:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582064+0000", 2024-09-21T10:20:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041026000000000002 2024-09-21T10:20:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582067+0000", 2024-09-21T10:20:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.32e-06 2024-09-21T10:20:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.541 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.073893+0000", 2024-09-21T10:20:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4918257930000001 2024-09-21T10:20:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.074003+0000", 2024-09-21T10:20:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000109857 2024-09-21T10:20:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,1,6,7", 2024-09-21T10:20:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.074197+0000", 2024-09-21T10:20:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000194594 2024-09-21T10:20:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.079441+0000", 2024-09-21T10:20:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052442749999999996 2024-09-21T10:20:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.079550+0000", 2024-09-21T10:20:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010806099999999999 2024-09-21T10:20:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.079587+0000", 2024-09-21T10:20:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7069e-05 2024-09-21T10:20:01.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.089422+0000", 2024-09-21T10:20:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098354359999999995 2024-09-21T10:20:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.089793+0000", 2024-09-21T10:20:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037076399999999999 2024-09-21T10:20:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.089852+0000", 2024-09-21T10:20:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9669000000000001e-05 2024-09-21T10:20:01.544 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.089965+0000", 2024-09-21T10:20:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0001122 2024-09-21T10:20:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135564 12.10 12:0fa5205d:::benchmark_data_smithi107_250110_object24154:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:20:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581672+0000", 2024-09-21T10:20:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 230.89333228000001, 2024-09-21T10:20:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.508594378, 2024-09-21T10:20:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:01.545 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135564 2024-09-21T10:20:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581672+0000", 2024-09-21T10:20:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.546 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581672+0000", 2024-09-21T10:20:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581672+0000", 2024-09-21T10:20:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.41e-07 2024-09-21T10:20:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581674+0000", 2024-09-21T10:20:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7179999999999999e-06 2024-09-21T10:20:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581678+0000", 2024-09-21T10:20:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.189e-06 2024-09-21T10:20:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581682+0000", 2024-09-21T10:20:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.382e-06 2024-09-21T10:20:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582087+0000", 2024-09-21T10:20:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040544499999999999 2024-09-21T10:20:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582092+0000", 2024-09-21T10:20:01.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5000000000000001e-06 2024-09-21T10:20:01.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.549 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.075502+0000", 2024-09-21T10:20:01.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4934106220000001 2024-09-21T10:20:01.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.549 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:01.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.075616+0000", 2024-09-21T10:20:01.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114258 2024-09-21T10:20:01.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.550 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,1,6,7", 2024-09-21T10:20:01.550 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.076849+0000", 2024-09-21T10:20:01.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0012324829999999999 2024-09-21T10:20:01.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.550 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:01.550 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.079447+0000", 2024-09-21T10:20:01.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025978619999999998 2024-09-21T10:20:01.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.550 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.550 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.089338+0000", 2024-09-21T10:20:01.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098911940000000007 2024-09-21T10:20:01.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.089383+0000", 2024-09-21T10:20:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4777000000000002e-05 2024-09-21T10:20:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.089756+0000", 2024-09-21T10:20:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037313599999999999 2024-09-21T10:20:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.090103+0000", 2024-09-21T10:20:01.551 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034747000000000001 2024-09-21T10:20:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.090156+0000", 2024-09-21T10:20:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2657000000000001e-05 2024-09-21T10:20:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.090266+0000", 2024-09-21T10:20:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000110344 2024-09-21T10:20:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135576 12.10 12:09ddc15e:::benchmark_data_smithi107_250110_object24166:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:20:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581687+0000", 2024-09-21T10:20:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 230.89331675099999, 2024-09-21T10:20:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5088722959999998, 2024-09-21T10:20:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135576 2024-09-21T10:20:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:01.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581687+0000", 2024-09-21T10:20:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581687+0000", 2024-09-21T10:20:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581688+0000", 2024-09-21T10:20:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6100000000000002e-07 2024-09-21T10:20:01.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581689+0000", 2024-09-21T10:20:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5889999999999999e-06 2024-09-21T10:20:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581695+0000", 2024-09-21T10:20:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6779999999999997e-06 2024-09-21T10:20:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581700+0000", 2024-09-21T10:20:01.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8080000000000004e-06 2024-09-21T10:20:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582111+0000", 2024-09-21T10:20:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041119699999999997 2024-09-21T10:20:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582114+0000", 2024-09-21T10:20:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2440000000000002e-06 2024-09-21T10:20:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.077952+0000", 2024-09-21T10:20:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.495837606 2024-09-21T10:20:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.078074+0000", 2024-09-21T10:20:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012204199999999999 2024-09-21T10:20:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,1,6,7", 2024-09-21T10:20:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.078271+0000", 2024-09-21T10:20:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019697200000000001 2024-09-21T10:20:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.082509+0000", 2024-09-21T10:20:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0042379139999999997 2024-09-21T10:20:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.089686+0000", 2024-09-21T10:20:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071776069999999999 2024-09-21T10:20:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.089721+0000", 2024-09-21T10:20:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4625000000000001e-05 2024-09-21T10:20:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.090069+0000", 2024-09-21T10:20:01.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00034759500000000002 2024-09-21T10:20:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.090401+0000", 2024-09-21T10:20:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00033212500000000001 2024-09-21T10:20:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.090453+0000", 2024-09-21T10:20:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2123999999999998e-05 2024-09-21T10:20:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.090560+0000", 2024-09-21T10:20:01.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000106809 2024-09-21T10:20:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135577 12.10 12:0f6a4e81:::benchmark_data_smithi107_250110_object24167:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:20:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581706+0000", 2024-09-21T10:20:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 230.893298469, 2024-09-21T10:20:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.509305576, 2024-09-21T10:20:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:01.560 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135577 2024-09-21T10:20:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581706+0000", 2024-09-21T10:20:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581706+0000", 2024-09-21T10:20:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581706+0000", 2024-09-21T10:20:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0499999999999999e-07 2024-09-21T10:20:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581708+0000", 2024-09-21T10:20:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5820000000000001e-06 2024-09-21T10:20:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581717+0000", 2024-09-21T10:20:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.901e-06 2024-09-21T10:20:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581722+0000", 2024-09-21T10:20:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8510000000000004e-06 2024-09-21T10:20:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582136+0000", 2024-09-21T10:20:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041303599999999998 2024-09-21T10:20:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582139+0000", 2024-09-21T10:20:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7850000000000002e-06 2024-09-21T10:20:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.079614+0000", 2024-09-21T10:20:01.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.497475186 2024-09-21T10:20:01.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.079720+0000", 2024-09-21T10:20:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105258 2024-09-21T10:20:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,1,6,7", 2024-09-21T10:20:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.079927+0000", 2024-09-21T10:20:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000207684 2024-09-21T10:20:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.565 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.090000+0000", 2024-09-21T10:20:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010072384 2024-09-21T10:20:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.090034+0000", 2024-09-21T10:20:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.468e-05 2024-09-21T10:20:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.090368+0000", 2024-09-21T10:20:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00033315999999999999 2024-09-21T10:20:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.090632+0000", 2024-09-21T10:20:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000264646 2024-09-21T10:20:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.090853+0000", 2024-09-21T10:20:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022040699999999999 2024-09-21T10:20:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.090903+0000", 2024-09-21T10:20:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0657e-05 2024-09-21T10:20:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.091011+0000", 2024-09-21T10:20:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000107954 2024-09-21T10:20:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135632 12.10 12:0e1bbd51:::benchmark_data_smithi107_250110_object24222:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e324)", 2024-09-21T10:20:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:10.581728+0000", 2024-09-21T10:20:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 230.89327574199999, 2024-09-21T10:20:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5140253769999998, 2024-09-21T10:20:01.568 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135632 2024-09-21T10:20:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581728+0000", 2024-09-21T10:20:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581728+0000", 2024-09-21T10:20:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581729+0000", 2024-09-21T10:20:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.58e-07 2024-09-21T10:20:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581731+0000", 2024-09-21T10:20:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.689e-06 2024-09-21T10:20:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581736+0000", 2024-09-21T10:20:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8309999999999997e-06 2024-09-21T10:20:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581746+0000", 2024-09-21T10:20:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8179999999999999e-06 2024-09-21T10:20:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582153+0000", 2024-09-21T10:20:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00040682499999999998 2024-09-21T10:20:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.582158+0000", 2024-09-21T10:20:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9350000000000002e-06 2024-09-21T10:20:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.081090+0000", 2024-09-21T10:20:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4989320259999999 2024-09-21T10:20:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.081192+0000", 2024-09-21T10:20:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101622 2024-09-21T10:20:01.572 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,1,6,7", 2024-09-21T10:20:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.081369+0000", 2024-09-21T10:20:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017691399999999999 2024-09-21T10:20:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.090300+0000", 2024-09-21T10:20:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0089316129999999997 2024-09-21T10:20:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.090334+0000", 2024-09-21T10:20:01.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3988000000000001e-05 2024-09-21T10:20:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.090594+0000", 2024-09-21T10:20:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00025982099999999998 2024-09-21T10:20:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.091058+0000", 2024-09-21T10:20:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046370800000000001 2024-09-21T10:20:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.095574+0000", 2024-09-21T10:20:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0045161979999999999 2024-09-21T10:20:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.095623+0000", 2024-09-21T10:20:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9018999999999999e-05 2024-09-21T10:20:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.095754+0000", 2024-09-21T10:20:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013091200000000001 2024-09-21T10:20:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.575 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:134446 13.8 13:1c128096:::benchmark_data_smithi107_278655_object6539:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:20:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:19:58.697685+0000", 2024-09-21T10:20:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.7773193599999999, 2024-09-21T10:20:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0299292210000002, 2024-09-21T10:20:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:20:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:20:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:20:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 134446 2024-09-21T10:20:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.697685+0000", 2024-09-21T10:20:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.697685+0000", 2024-09-21T10:20:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.697686+0000", 2024-09-21T10:20:01.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.606e-06 2024-09-21T10:20:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.697691+0000", 2024-09-21T10:20:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5480000000000003e-06 2024-09-21T10:20:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.697704+0000", 2024-09-21T10:20:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2860999999999999e-05 2024-09-21T10:20:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.697711+0000", 2024-09-21T10:20:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6240000000000002e-06 2024-09-21T10:20:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.697803+0000", 2024-09-21T10:20:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1099000000000003e-05 2024-09-21T10:20:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.697924+0000", 2024-09-21T10:20:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121614 2024-09-21T10:20:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:20:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.698046+0000", 2024-09-21T10:20:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000121632 2024-09-21T10:20:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.706547+0000", 2024-09-21T10:20:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0085012879999999992 2024-09-21T10:20:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.716216+0000", 2024-09-21T10:20:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0096690700000000001 2024-09-21T10:20:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.735223+0000", 2024-09-21T10:20:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019006667000000001 2024-09-21T10:20:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.735233+0000", 2024-09-21T10:20:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0197000000000001e-05 2024-09-21T10:20:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:59.604086+0000", 2024-09-21T10:20:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.868852607 2024-09-21T10:20:01.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:59.604099+0000", 2024-09-21T10:20:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3414000000000001e-05 2024-09-21T10:20:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.727537+0000", 2024-09-21T10:20:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.123438376 2024-09-21T10:20:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:01.582 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.727614+0000", 2024-09-21T10:20:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6618000000000004e-05 2024-09-21T10:20:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:134452 13.8 13:1be35b0b:::benchmark_data_smithi107_278655_object6545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:20:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:19:58.699554+0000", 2024-09-21T10:20:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.7754506920000002, 2024-09-21T10:20:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.028392234, 2024-09-21T10:20:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:20:01.583 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:20:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:20:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 134452 2024-09-21T10:20:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.699554+0000", 2024-09-21T10:20:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.699554+0000", 2024-09-21T10:20:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.699555+0000", 2024-09-21T10:20:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.46e-06 2024-09-21T10:20:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.699560+0000", 2024-09-21T10:20:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6299999999999997e-06 2024-09-21T10:20:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.699571+0000", 2024-09-21T10:20:01.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1752e-05 2024-09-21T10:20:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.699578+0000", 2024-09-21T10:20:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7139999999999996e-06 2024-09-21T10:20:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.701981+0000", 2024-09-21T10:20:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002403292 2024-09-21T10:20:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:01.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.702123+0000", 2024-09-21T10:20:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014150499999999999 2024-09-21T10:20:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:20:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.702293+0000", 2024-09-21T10:20:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017033900000000001 2024-09-21T10:20:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.706552+0000", 2024-09-21T10:20:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0042587989999999997 2024-09-21T10:20:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.716256+0000", 2024-09-21T10:20:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.009704074 2024-09-21T10:20:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.735263+0000", 2024-09-21T10:20:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.019006742 2024-09-21T10:20:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.735270+0000", 2024-09-21T10:20:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.481e-06 2024-09-21T10:20:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:59.604425+0000", 2024-09-21T10:20:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.86915458899999998 2024-09-21T10:20:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:59.604434+0000", 2024-09-21T10:20:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5640000000000002e-06 2024-09-21T10:20:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.727902+0000", 2024-09-21T10:20:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.12346707 2024-09-21T10:20:01.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.727946+0000", 2024-09-21T10:20:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4223e-05 2024-09-21T10:20:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:134457 13.1 13:83460430:::benchmark_data_smithi107_278655_object6550:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:20:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:19:58.722368+0000", 2024-09-21T10:20:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.7526357450000001, 2024-09-21T10:20:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0220835450000001, 2024-09-21T10:20:01.590 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:20:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:20:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:20:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 134457 2024-09-21T10:20:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.722368+0000", 2024-09-21T10:20:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.722368+0000", 2024-09-21T10:20:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.722370+0000", 2024-09-21T10:20:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6670000000000001e-06 2024-09-21T10:20:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.722376+0000", 2024-09-21T10:20:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7599999999999999e-06 2024-09-21T10:20:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.722388+0000", 2024-09-21T10:20:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1802e-05 2024-09-21T10:20:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.722395+0000", 2024-09-21T10:20:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5499999999999997e-06 2024-09-21T10:20:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.728854+0000", 2024-09-21T10:20:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0064584020000000002 2024-09-21T10:20:01.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.728863+0000", 2024-09-21T10:20:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4229999999999994e-06 2024-09-21T10:20:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:59.605626+0000", 2024-09-21T10:20:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.87676325600000005 2024-09-21T10:20:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:59.605635+0000", 2024-09-21T10:20:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0110000000000006e-06 2024-09-21T10:20:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.744360+0000", 2024-09-21T10:20:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.138724979 2024-09-21T10:20:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.744452+0000", 2024-09-21T10:20:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1694999999999995e-05 2024-09-21T10:20:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:134461 13.1 13:80e68256:::benchmark_data_smithi107_278655_object6554:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:20:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:19:58.722414+0000", 2024-09-21T10:20:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.7525903889999999, 2024-09-21T10:20:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0362527830000001, 2024-09-21T10:20:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:20:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:20:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:20:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 134461 2024-09-21T10:20:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.722414+0000", 2024-09-21T10:20:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.722414+0000", 2024-09-21T10:20:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.722414+0000", 2024-09-21T10:20:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2900000000000004e-07 2024-09-21T10:20:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.722417+0000", 2024-09-21T10:20:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.339e-06 2024-09-21T10:20:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.722423+0000", 2024-09-21T10:20:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6409999999999996e-06 2024-09-21T10:20:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.722429+0000", 2024-09-21T10:20:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.609e-06 2024-09-21T10:20:01.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.728882+0000", 2024-09-21T10:20:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0064534229999999998 2024-09-21T10:20:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.728888+0000", 2024-09-21T10:20:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4779999999999999e-06 2024-09-21T10:20:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:59.605659+0000", 2024-09-21T10:20:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.87677101700000004 2024-09-21T10:20:01.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:59.605666+0000", 2024-09-21T10:20:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5070000000000003e-06 2024-09-21T10:20:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.744733+0000", 2024-09-21T10:20:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.139067005 2024-09-21T10:20:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for missing object", 2024-09-21T10:20:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.744839+0000", 2024-09-21T10:20:01.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105535 2024-09-21T10:20:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.758583+0000", 2024-09-21T10:20:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013743751 2024-09-21T10:20:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.758667+0000", 2024-09-21T10:20:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3949000000000005e-05 2024-09-21T10:20:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:134470 13.8 13:185aaec6:::benchmark_data_smithi107_278655_object6563:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:20:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:19:58.742202+0000", 2024-09-21T10:20:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.7328018040000002, 2024-09-21T10:20:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0254232480000001, 2024-09-21T10:20:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:20:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:20:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 134470 2024-09-21T10:20:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.742202+0000", 2024-09-21T10:20:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.742202+0000", 2024-09-21T10:20:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.742205+0000", 2024-09-21T10:20:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1569999999999998e-06 2024-09-21T10:20:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.742365+0000", 2024-09-21T10:20:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000160748 2024-09-21T10:20:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.742384+0000", 2024-09-21T10:20:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8875000000000001e-05 2024-09-21T10:20:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.742394+0000", 2024-09-21T10:20:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8870000000000004e-06 2024-09-21T10:20:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.742481+0000", 2024-09-21T10:20:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6675999999999995e-05 2024-09-21T10:20:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.742491+0000", 2024-09-21T10:20:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0562e-05 2024-09-21T10:20:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:59.604491+0000", 2024-09-21T10:20:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.86199983899999999 2024-09-21T10:20:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:59.604501+0000", 2024-09-21T10:20:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0327e-05 2024-09-21T10:20:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.727988+0000", 2024-09-21T10:20:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1234864760000001 2024-09-21T10:20:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.728086+0000", 2024-09-21T10:20:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8237000000000002e-05 2024-09-21T10:20:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:20:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.728543+0000", 2024-09-21T10:20:01.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00045729000000000001 2024-09-21T10:20:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.739122+0000", 2024-09-21T10:20:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010578384999999999 2024-09-21T10:20:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.741005+0000", 2024-09-21T10:20:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001882828 2024-09-21T10:20:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.746874+0000", 2024-09-21T10:20:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0058694250000000002 2024-09-21T10:20:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.767504+0000", 2024-09-21T10:20:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.020629621000000001 2024-09-21T10:20:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.767541+0000", 2024-09-21T10:20:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6769e-05 2024-09-21T10:20:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:01.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.767626+0000", 2024-09-21T10:20:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5146000000000001e-05 2024-09-21T10:20:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:134471 13.8 13:1a4f231b:::benchmark_data_smithi107_278655_object6564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:20:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:19:58.742456+0000", 2024-09-21T10:20:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.73254792, 2024-09-21T10:20:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0264439090000002, 2024-09-21T10:20:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:01.610 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:20:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:20:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 134471 2024-09-21T10:20:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.742456+0000", 2024-09-21T10:20:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:01.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.742456+0000", 2024-09-21T10:20:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.742457+0000", 2024-09-21T10:20:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3300000000000001e-07 2024-09-21T10:20:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.742461+0000", 2024-09-21T10:20:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8480000000000003e-06 2024-09-21T10:20:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:01.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.742469+0000", 2024-09-21T10:20:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3499999999999997e-06 2024-09-21T10:20:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.742476+0000", 2024-09-21T10:20:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5679999999999996e-06 2024-09-21T10:20:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.742513+0000", 2024-09-21T10:20:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6823999999999998e-05 2024-09-21T10:20:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.742519+0000", 2024-09-21T10:20:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0419999999999999e-06 2024-09-21T10:20:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:59.604678+0000", 2024-09-21T10:20:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.86215917200000003 2024-09-21T10:20:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:59.604686+0000", 2024-09-21T10:20:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3990000000000004e-06 2024-09-21T10:20:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.734683+0000", 2024-09-21T10:20:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1299967799999999 2024-09-21T10:20:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.734816+0000", 2024-09-21T10:20:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000132936 2024-09-21T10:20:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:20:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.734970+0000", 2024-09-21T10:20:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015384700000000001 2024-09-21T10:20:01.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.739394+0000", 2024-09-21T10:20:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0044242129999999998 2024-09-21T10:20:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.741345+0000", 2024-09-21T10:20:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001950533 2024-09-21T10:20:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.746924+0000", 2024-09-21T10:20:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0055790429999999997 2024-09-21T10:20:01.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.768769+0000", 2024-09-21T10:20:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021845442 2024-09-21T10:20:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.768837+0000", 2024-09-21T10:20:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7754000000000002e-05 2024-09-21T10:20:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.768900+0000", 2024-09-21T10:20:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3324999999999994e-05 2024-09-21T10:20:01.617 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:134475 13.4 13:209c41c0:::benchmark_data_smithi107_278655_object6568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:20:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:19:58.750586+0000", 2024-09-21T10:20:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.7244180609999997, 2024-09-21T10:20:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.034400116, 2024-09-21T10:20:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:20:01.618 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:20:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 134475 2024-09-21T10:20:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.750586+0000", 2024-09-21T10:20:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.750586+0000", 2024-09-21T10:20:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.750588+0000", 2024-09-21T10:20:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.57e-06 2024-09-21T10:20:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.750594+0000", 2024-09-21T10:20:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1079999999999998e-06 2024-09-21T10:20:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.750610+0000", 2024-09-21T10:20:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6317e-05 2024-09-21T10:20:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.750616+0000", 2024-09-21T10:20:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7320000000000003e-06 2024-09-21T10:20:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.750668+0000", 2024-09-21T10:20:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1748e-05 2024-09-21T10:20:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.750678+0000", 2024-09-21T10:20:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0475999999999999e-05 2024-09-21T10:20:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:59.604607+0000", 2024-09-21T10:20:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.85392847800000005 2024-09-21T10:20:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:59.604616+0000", 2024-09-21T10:20:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9590000000000001e-06 2024-09-21T10:20:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.763300+0000", 2024-09-21T10:20:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1586846259999999 2024-09-21T10:20:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.766455+0000", 2024-09-21T10:20:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031550810000000001 2024-09-21T10:20:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:20:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.766544+0000", 2024-09-21T10:20:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8308999999999998e-05 2024-09-21T10:20:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.773446+0000", 2024-09-21T10:20:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069022809999999997 2024-09-21T10:20:01.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.773487+0000", 2024-09-21T10:20:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1610999999999999e-05 2024-09-21T10:20:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.781034+0000", 2024-09-21T10:20:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0075465879999999999 2024-09-21T10:20:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.784738+0000", 2024-09-21T10:20:01.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00370404 2024-09-21T10:20:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.784859+0000", 2024-09-21T10:20:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012063800000000001 2024-09-21T10:20:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.784986+0000", 2024-09-21T10:20:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012755399999999999 2024-09-21T10:20:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:134478 13.1 13:8ee8ff5f:::benchmark_data_smithi107_278655_object6571:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:20:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:19:58.758311+0000", 2024-09-21T10:20:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.7166927479999998, 2024-09-21T10:20:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9937532980000001, 2024-09-21T10:20:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:20:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:20:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 134478 2024-09-21T10:20:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:01.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.758311+0000", 2024-09-21T10:20:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.758311+0000", 2024-09-21T10:20:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.758313+0000", 2024-09-21T10:20:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9319999999999999e-06 2024-09-21T10:20:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.758321+0000", 2024-09-21T10:20:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2690000000000002e-06 2024-09-21T10:20:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.758338+0000", 2024-09-21T10:20:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6997999999999999e-05 2024-09-21T10:20:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.758346+0000", 2024-09-21T10:20:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4260000000000003e-06 2024-09-21T10:20:01.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.758421+0000", 2024-09-21T10:20:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4921999999999999e-05 2024-09-21T10:20:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:58.758430+0000", 2024-09-21T10:20:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7660000000000004e-06 2024-09-21T10:20:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:59.605746+0000", 2024-09-21T10:20:01.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.84731647600000004 2024-09-21T10:20:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:19:59.605754+0000", 2024-09-21T10:20:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1349999999999992e-06 2024-09-21T10:20:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.744875+0000", 2024-09-21T10:20:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.139120562 2024-09-21T10:20:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:01.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.745369+0000", 2024-09-21T10:20:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049449800000000001 2024-09-21T10:20:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,6,7", 2024-09-21T10:20:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.745506+0000", 2024-09-21T10:20:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013634199999999999 2024-09-21T10:20:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.751742+0000", 2024-09-21T10:20:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0062365500000000004 2024-09-21T10:20:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.751793+0000", 2024-09-21T10:20:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0244999999999998e-05 2024-09-21T10:20:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.751802+0000", 2024-09-21T10:20:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7049999999999994e-06 2024-09-21T10:20:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.751967+0000", 2024-09-21T10:20:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016458299999999999 2024-09-21T10:20:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.751999+0000", 2024-09-21T10:20:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2388999999999998e-05 2024-09-21T10:20:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:00.752065+0000", 2024-09-21T10:20:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5500000000000006e-05 2024-09-21T10:20:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:01.633 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:01.633 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:20:01.634 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:01.671 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:01.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:01.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:01.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:01.809 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:01.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:01.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:01.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:20:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:20:01] "GET /metrics HTTP/1.1" 200 36782 "" "Prometheus/2.51.0" 2024-09-21T10:20:01.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:01 smithi107 ceph-mon[102221]: 1.a scrub starts 2024-09-21T10:20:01.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:01 smithi107 ceph-mon[102221]: 1.a scrub ok 2024-09-21T10:20:01.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:01 smithi107 ceph-mon[102221]: 13.4 starting backfill to osd.5 from (0'0,0'0] MAX to 379'8433 2024-09-21T10:20:01.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:01 smithi107 ceph-mon[102221]: 13.8 starting backfill to osd.1 from (0'0,0'0] MAX to 379'8479 2024-09-21T10:20:01.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:01 smithi107 ceph-mon[102221]: 13.7 starting backfill to osd.5 from (0'0,0'0] MAX to 379'8413 2024-09-21T10:20:01.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:01 smithi107 ceph-mon[102221]: 13.1 starting backfill to osd.7 from (0'0,0'0] MAX to 379'8424 2024-09-21T10:20:01.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:01 smithi107 ceph-mon[102221]: 13.f starting backfill to osd.5 from (0'0,0'0] MAX to 379'8460 2024-09-21T10:20:01.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:01 smithi107 ceph-mon[105022]: 1.a scrub starts 2024-09-21T10:20:01.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:01 smithi107 ceph-mon[105022]: 1.a scrub ok 2024-09-21T10:20:01.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:01 smithi107 ceph-mon[105022]: 13.4 starting backfill to osd.5 from (0'0,0'0] MAX to 379'8433 2024-09-21T10:20:01.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:01 smithi107 ceph-mon[105022]: 13.8 starting backfill to osd.1 from (0'0,0'0] MAX to 379'8479 2024-09-21T10:20:01.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:01 smithi107 ceph-mon[105022]: 13.7 starting backfill to osd.5 from (0'0,0'0] MAX to 379'8413 2024-09-21T10:20:01.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:01 smithi107 ceph-mon[105022]: 13.1 starting backfill to osd.7 from (0'0,0'0] MAX to 379'8424 2024-09-21T10:20:01.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:01 smithi107 ceph-mon[105022]: 13.f starting backfill to osd.5 from (0'0,0'0] MAX to 379'8460 2024-09-21T10:20:01.911 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:01.947 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:01.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:01.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:02.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:02.087 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:02.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:02.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:01 smithi129 ceph-mon[88471]: 1.a scrub starts 2024-09-21T10:20:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:01 smithi129 ceph-mon[88471]: 1.a scrub ok 2024-09-21T10:20:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:01 smithi129 ceph-mon[88471]: 13.4 starting backfill to osd.5 from (0'0,0'0] MAX to 379'8433 2024-09-21T10:20:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:01 smithi129 ceph-mon[88471]: 13.8 starting backfill to osd.1 from (0'0,0'0] MAX to 379'8479 2024-09-21T10:20:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:01 smithi129 ceph-mon[88471]: 13.7 starting backfill to osd.5 from (0'0,0'0] MAX to 379'8413 2024-09-21T10:20:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:01 smithi129 ceph-mon[88471]: 13.1 starting backfill to osd.7 from (0'0,0'0] MAX to 379'8424 2024-09-21T10:20:02.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:01 smithi129 ceph-mon[88471]: 13.f starting backfill to osd.5 from (0'0,0'0] MAX to 379'8460 2024-09-21T10:20:02.189 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:02.229 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:02.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:02.331 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:02.368 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:02.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:02.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:02.470 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:02.507 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:02.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:02.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:02.609 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:02.649 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:02.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:02.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:02.751 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:02.785 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:02.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:02.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:02.887 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:02.928 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:02.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:02.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:03.030 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:03.056 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:02 smithi107 ceph-mon[102221]: 1.6 scrub starts 2024-09-21T10:20:03.056 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:02 smithi107 ceph-mon[102221]: 1.6 scrub ok 2024-09-21T10:20:03.057 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:02 smithi107 ceph-mon[102221]: pgmap v1317: 109 pgs: 3 remapped+peering, 106 active+clean; 7.5 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 170 B/s rd, 1.1 MiB/s wr, 856 op/s; 3 B/s, 0 objects/s recovering 2024-09-21T10:20:03.057 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:02 smithi107 ceph-mon[105022]: 1.6 scrub starts 2024-09-21T10:20:03.057 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:02 smithi107 ceph-mon[105022]: 1.6 scrub ok 2024-09-21T10:20:03.057 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:02 smithi107 ceph-mon[105022]: pgmap v1317: 109 pgs: 3 remapped+peering, 106 active+clean; 7.5 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 170 B/s rd, 1.1 MiB/s wr, 856 op/s; 3 B/s, 0 objects/s recovering 2024-09-21T10:20:03.067 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:03.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:03.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:03.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:03.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:02 smithi129 ceph-mon[88471]: 1.6 scrub starts 2024-09-21T10:20:03.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:02 smithi129 ceph-mon[88471]: 1.6 scrub ok 2024-09-21T10:20:03.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:02 smithi129 ceph-mon[88471]: pgmap v1317: 109 pgs: 3 remapped+peering, 106 active+clean; 7.5 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 170 B/s rd, 1.1 MiB/s wr, 856 op/s; 3 B/s, 0 objects/s recovering 2024-09-21T10:20:03.200 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:03.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:03.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:03.302 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:03.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:03.339 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:03.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:03.440 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:03.483 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:03.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:03.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:03.585 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:03.620 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:03.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:03.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:03.721 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:03.759 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:20:03.769 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:03.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:03.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:03.870 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:03.895 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:03.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:03.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:03.997 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:04.035 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:04.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:04.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:03 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1558330242' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:20:04.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:03 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:20:04.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:03 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1558330242' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:20:04.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:03 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:20:04.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:04.159 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:04.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:03 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1558330242' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:20:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:03 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:20:04.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:04.291 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:04.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:04.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:04.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:04.416 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:04.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:04.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:04.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:04.553 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:04.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:04.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:04.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:04.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:04.691 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:04.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:04.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:04.821 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:04.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:04.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:04.923 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:04.959 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:04.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:04.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:05.061 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:05.096 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:05.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:05.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:05.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:04 smithi107 ceph-mon[105022]: pgmap v1318: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 3 remapped+peering, 104 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 5.7 KiB/s rd, 12 KiB/s wr, 583 op/s; 14860/361179 objects misplaced (4.114%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:20:05.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:04 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:20:05.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:04 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:20:05.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:04 smithi107 ceph-mon[105022]: osdmap e383: 8 total, 8 up, 7 in 2024-09-21T10:20:05.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:04 smithi107 ceph-mon[102221]: pgmap v1318: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 3 remapped+peering, 104 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 5.7 KiB/s rd, 12 KiB/s wr, 583 op/s; 14860/361179 objects misplaced (4.114%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:20:05.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:04 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:20:05.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:04 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:20:05.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:04 smithi107 ceph-mon[102221]: osdmap e383: 8 total, 8 up, 7 in 2024-09-21T10:20:05.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:04 smithi129 ceph-mon[88471]: pgmap v1318: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 3 remapped+peering, 104 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 5.7 KiB/s rd, 12 KiB/s wr, 583 op/s; 14860/361179 objects misplaced (4.114%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:20:05.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:04 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:20:05.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:04 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:20:05.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:04 smithi129 ceph-mon[88471]: osdmap e383: 8 total, 8 up, 7 in 2024-09-21T10:20:05.198 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:05.237 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:05.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:05.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:05.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:05.369 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:05.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:05.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:05.406 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 4] out_osds: [6] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:20:05.406 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:20:05.406 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-21T10:20:05.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-21T10:20:05.471 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:05.509 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:05.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:05.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:05.611 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:05.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:05.645 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:05.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:05.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:05.790 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:05.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:05.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:05.892 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:05.924 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:05.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:05.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:06.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:06.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:06.056 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:06.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:06.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:06.188 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:06.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:06.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:06.290 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:06.316 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:06.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:06.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:06.418 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:06.455 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:06.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:06.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:06.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:06.596 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:06.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:06.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:06.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:06.718 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:06.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:06.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:06.819 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:06.862 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:06.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:06.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:06.964 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:07.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:07.005 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:07.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:07.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:07.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:06 smithi107 ceph-mon[102221]: pgmap v1320: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 626 op/s; 36327/358476 objects misplaced (10.134%); 9.4 MiB/s, 150 objects/s recovering 2024-09-21T10:20:07.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:06 smithi107 ceph-mon[105022]: pgmap v1320: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 626 op/s; 36327/358476 objects misplaced (10.134%); 9.4 MiB/s, 150 objects/s recovering 2024-09-21T10:20:07.128 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:07.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:07.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:07.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:06 smithi129 ceph-mon[88471]: pgmap v1320: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 626 op/s; 36327/358476 objects misplaced (10.134%); 9.4 MiB/s, 150 objects/s recovering 2024-09-21T10:20:07.230 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:07.264 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:07.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:07.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:07.317 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:20:07.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:07.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:07.404 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:07.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:07.505 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:07.536 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:07.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:07.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:07.638 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:07.681 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:07.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:07.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:07.783 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:07.831 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:07.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:07.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:07.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:07.970 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:07.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:07.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:08.071 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:08.115 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:08.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:08.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:08.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:08.247 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:08.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:08.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:08.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:08.384 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:08.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:08.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:08.485 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:08.524 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:08.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:08.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:08.626 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:08.669 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:08.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:08.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:08.749 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:08.771 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:08.778 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:08.810 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:08.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:08.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:08.889 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:08 smithi107 ceph-mon[102221]: pgmap v1321: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 7.1 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 820 op/s; 35035/350379 objects misplaced (9.999%); 13 MiB/s, 214 objects/s recovering 2024-09-21T10:20:08.889 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:08 smithi107 ceph-mon[105022]: pgmap v1321: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 7.1 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 820 op/s; 35035/350379 objects misplaced (9.999%); 13 MiB/s, 214 objects/s recovering 2024-09-21T10:20:08.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:08.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:08.942 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:08.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:09.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:09.076 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:09.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:09.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:09.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:08 smithi129 ceph-mon[88471]: pgmap v1321: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 7.1 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 0 B/s wr, 820 op/s; 35035/350379 objects misplaced (9.999%); 13 MiB/s, 214 objects/s recovering 2024-09-21T10:20:09.177 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:09.209 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:09.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:09.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:09.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:09.348 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:09.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:09.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:09.449 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:09.485 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:09.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:09.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:09.511 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:20:09.511 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:20:09.539 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:20:09.539 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:20:09.574 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:20:09.575 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:20:09.586 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:09.609 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:20:09.609 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:20:09.619 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:09.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:09.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:09.645 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:20:09.645 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:20:09.690 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:20:09.690 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:20:09.720 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:09.725 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:20:09.725 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:20:09.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:09.756 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:09.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:09.757 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:20:09.757 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:20:09.858 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:09.900 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:09.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:09.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:10.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:10.036 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:10.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:10.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:10.070 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:09 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3061008904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:10.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:20:10.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:09 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3061008904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:10.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:09 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3061008904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:10.138 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:10.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:10.179 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:10.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:10.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:10.321 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:10.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:10.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:10.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:20:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:20:10.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:20:10.423 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:10.458 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:10.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:10.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:10.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:10.589 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:10.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:10.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:10.691 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:10.728 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:10.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:10.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:10.830 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:10.871 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:10 smithi129 ceph-mon[88471]: pgmap v1322: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 713 op/s; 35035/348639 objects misplaced (10.049%); 12 MiB/s, 189 objects/s recovering 2024-09-21T10:20:10.872 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:10 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:20:10.876 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:10.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:10.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:10.978 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:11.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:11.026 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:11.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:11.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:10 smithi107 ceph-mon[102221]: pgmap v1322: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 713 op/s; 35035/348639 objects misplaced (10.049%); 12 MiB/s, 189 objects/s recovering 2024-09-21T10:20:11.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:10 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:20:11.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:10 smithi107 ceph-mon[105022]: pgmap v1322: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 713 op/s; 35035/348639 objects misplaced (10.049%); 12 MiB/s, 189 objects/s recovering 2024-09-21T10:20:11.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:10 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:20:11.127 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:11.171 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:11.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:11.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:11.273 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:11.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:11.309 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:11.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:11.410 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:11.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:11.458 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:11.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:11.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:11.592 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:11.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:11.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:11.693 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:11.741 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:11.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:11.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:11.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:11.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:20:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:20:11] "GET /metrics HTTP/1.1" 200 36830 "" "Prometheus/2.51.0" 2024-09-21T10:20:11.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:11.879 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:11.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:11.980 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:12.013 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:12.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:12.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:12.115 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:12.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:12.150 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:12.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:12.251 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:12.286 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:12.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:12.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:12.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:12.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:12.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:12 smithi129 ceph-mon[88471]: pgmap v1323: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 7.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 757 op/s; 33268/344409 objects misplaced (9.659%); 17 MiB/s, 265 objects/s recovering 2024-09-21T10:20:12.365 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:12.388 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:12.420 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:12.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:12.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:12.522 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:12.562 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:12.562 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:12.562 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:12 smithi107 ceph-mon[105022]: pgmap v1323: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 7.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 757 op/s; 33268/344409 objects misplaced (9.659%); 17 MiB/s, 265 objects/s recovering 2024-09-21T10:20:12.562 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:12.563 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:12.563 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:12.563 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:12 smithi107 ceph-mon[102221]: pgmap v1323: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 7.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 757 op/s; 33268/344409 objects misplaced (9.659%); 17 MiB/s, 265 objects/s recovering 2024-09-21T10:20:12.563 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:12.566 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:12.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:12.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:12.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:12.715 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:12.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:12.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:12.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:20:12.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:12.872 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:12.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:12.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:12.974 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:13.006 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:13.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:13.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:13.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:13.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:13.143 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:13.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:13.244 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:13.274 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:13.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:13.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:13.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:13.405 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:13.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:13.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:13.506 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:13.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:13.552 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:13.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:13.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:13.683 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:13.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:13.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:13.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:13.825 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:13.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:13.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:13.927 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:13.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:13.982 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:13.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:14.083 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:14.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:14.123 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:14.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:14.208 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:20:14.209 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:20:14.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:14.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:14.268 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:14.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:14.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:14.402 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:14.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:14.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:14.461 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-09-21T10:20:14.462 INFO:teuthology.orchestra.run.smithi107.stderr:0 2024-09-21T10:20:14.462 INFO:teuthology.orchestra.run.smithi107.stderr:1 2024-09-21T10:20:14.462 INFO:teuthology.orchestra.run.smithi107.stderr:2 2024-09-21T10:20:14.462 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2024-09-21T10:20:14.462 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2024-09-21T10:20:14.462 INFO:teuthology.orchestra.run.smithi107.stderr:bench [] [] [] [] 2024-09-21T10:20:14.462 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:20:14.462 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2024-09-21T10:20:14.462 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2024-09-21T10:20:14.462 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2024-09-21T10:20:14.462 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-09-21T10:20:14.504 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:14.526 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:14 smithi129 ceph-mon[88471]: pgmap v1324: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 858 op/s; 32366/335391 objects misplaced (9.650%); 18 MiB/s, 287 objects/s recovering 2024-09-21T10:20:14.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:14.536 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:14.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:14.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:14.666 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:14.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:14.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:14.742 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:14 smithi107 ceph-mon[102221]: pgmap v1324: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 858 op/s; 32366/335391 objects misplaced (9.650%); 18 MiB/s, 287 objects/s recovering 2024-09-21T10:20:14.743 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:14 smithi107 ceph-mon[105022]: pgmap v1324: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 858 op/s; 32366/335391 objects misplaced (9.650%); 18 MiB/s, 287 objects/s recovering 2024-09-21T10:20:14.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:14.802 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:14.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:14.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:14.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:14.932 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:14.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:14.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:15.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:15.066 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:15.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:15.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:15.168 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:15.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:20:15.212 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:15.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:15.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:15.314 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:15.363 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:15.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:15.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:15.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:15.494 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:15.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:15.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:15.596 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:15.630 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:15.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:15.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:15.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:15.790 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:15.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:15.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:15.891 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:15.930 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:15.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:15.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:16.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:16.058 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:16.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:16.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:16.159 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:16.202 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:16.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:16.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:16.302 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:16.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:20:16.316 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:20:16.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-21T10:20:16.338 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:16.384 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:16.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:16.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:16.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:16.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:16.522 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:16.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:16.530 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:16 smithi107 ceph-mon[102221]: pgmap v1325: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 825 op/s; 30447/332124 objects misplaced (9.167%); 22 MiB/s, 345 objects/s recovering 2024-09-21T10:20:16.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:16.673 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:16.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:16.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:16.746 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:16.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:16.814 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:16.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:16.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:16.817 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:16 smithi107 ceph-mon[105022]: pgmap v1325: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 825 op/s; 30447/332124 objects misplaced (9.167%); 22 MiB/s, 345 objects/s recovering 2024-09-21T10:20:16.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:16.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:16 smithi129 ceph-mon[88471]: pgmap v1325: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 825 op/s; 30447/332124 objects misplaced (9.167%); 22 MiB/s, 345 objects/s recovering 2024-09-21T10:20:16.959 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:16.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:16.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:17.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:17.095 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:17.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:17.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:17.197 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:17.233 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:17.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:17.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:17.334 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:17.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:17.389 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:17.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:17.489 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:17.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:17.554 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:17.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:17.655 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:17.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:17.694 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:17.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:17.794 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:17.834 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:17.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:17.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:17.936 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:17.978 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:17.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:17.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:18.080 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:18.116 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:18.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:18.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:18.217 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:18.250 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:18.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:18.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:18.352 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:18.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:17 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:18.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:17 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:18.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:17 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:18.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:17 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:18.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:18.384 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:18.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:18.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:17 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:18.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:17 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:18.485 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:18.511 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:18.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:18.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:18.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:18.644 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:18.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:18.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:18.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:18.776 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:18.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:18.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:18.878 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:18.920 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:18.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:18.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:19.022 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:19.063 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:19.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:19.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:19.165 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:19.199 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:19.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:19.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:19.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:19.330 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:19.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:19.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:18 smithi107 ceph-mon[102221]: pgmap v1326: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 914 op/s; 29484/325530 objects misplaced (9.057%); 20 MiB/s, 316 objects/s recovering 2024-09-21T10:20:19.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:18 smithi107 ceph-mon[105022]: pgmap v1326: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 914 op/s; 29484/325530 objects misplaced (9.057%); 20 MiB/s, 316 objects/s recovering 2024-09-21T10:20:19.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:18 smithi129 ceph-mon[88471]: pgmap v1326: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 914 op/s; 29484/325530 objects misplaced (9.057%); 20 MiB/s, 316 objects/s recovering 2024-09-21T10:20:19.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:19.454 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:19.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:19.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:19.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:19.579 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:19.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:19.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:19.681 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:19.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:19.707 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:19.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:19.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:19.839 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:19.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:19.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:19.941 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:19.976 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:19.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:19.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:20.078 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:20.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:20.121 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:20.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:20.222 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:20.257 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:20.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:20.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:20.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:20.391 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:20.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:20.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:20.492 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:20.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:20.537 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:20.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:20.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:20:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:20:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:20:20.672 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:20.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:20.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:20.774 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:20.811 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:20.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:20.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:20.913 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:20.949 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:20.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:20.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:21.050 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:21.083 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:21.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:21.185 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:21.213 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:21.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:21.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:21.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:21.348 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:21.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:21.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:21.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:20 smithi107 ceph-mon[102221]: pgmap v1327: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 778 op/s; 29484/322341 objects misplaced (9.147%); 16 MiB/s, 251 objects/s recovering 2024-09-21T10:20:21.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:20 smithi107 ceph-mon[105022]: pgmap v1327: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 778 op/s; 29484/322341 objects misplaced (9.147%); 16 MiB/s, 251 objects/s recovering 2024-09-21T10:20:21.384 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:20 smithi129 ceph-mon[88471]: pgmap v1327: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 778 op/s; 29484/322341 objects misplaced (9.147%); 16 MiB/s, 251 objects/s recovering 2024-09-21T10:20:21.450 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:21.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:21.495 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:21.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:21.596 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:21.624 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:21.632 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:21.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:21.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:21.733 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:21.768 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:21.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:21.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:21.870 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:21.908 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:21.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:21.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:22.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:22.046 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:22.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:22.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:22.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:20:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:20:21] "GET /metrics HTTP/1.1" 200 36830 "" "Prometheus/2.51.0" 2024-09-21T10:20:22.148 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:22.181 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:22.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:22.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:22.283 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:22.328 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:22.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:22.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:22.431 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:22.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:22.464 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:22.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:22.565 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:22.594 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:22.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:22.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:22.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:22.726 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:22.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:22.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:22.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:22.858 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:22.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:22.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:22.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:22.989 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:22.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:22.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:23.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:23.122 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:23.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:23.220 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:22 smithi107 ceph-mon[102221]: pgmap v1328: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 841 op/s; 27704/318324 objects misplaced (8.703%); 22 MiB/s, 344 objects/s recovering 2024-09-21T10:20:23.220 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:22 smithi107 ceph-mon[105022]: pgmap v1328: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 841 op/s; 27704/318324 objects misplaced (8.703%); 22 MiB/s, 344 objects/s recovering 2024-09-21T10:20:23.224 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:23.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:23.260 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:23.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:23.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:23.393 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:23.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:23.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:23.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:22 smithi129 ceph-mon[88471]: pgmap v1328: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 841 op/s; 27704/318324 objects misplaced (8.703%); 22 MiB/s, 344 objects/s recovering 2024-09-21T10:20:23.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:23.531 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:23.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:23.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:23.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:23.656 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:23.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:23.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:23.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:23.785 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:23.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:23.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:23.886 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:23.922 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:23.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:23.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:24.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:24.055 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:24.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:24.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:24.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:24.188 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:24.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:24.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:24.290 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:24.326 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:24.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:24.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:24.428 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:24.465 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:24.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:24.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:24.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:24.605 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:24.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:24.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:24.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:24.778 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:24.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:24.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:24.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:24.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:24.913 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:24.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:25.013 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:25.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:25.054 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:25.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:25.156 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:25.189 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:25.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:25.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:25.230 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:24 smithi107 ceph-mon[102221]: pgmap v1329: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.3 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 977 op/s; 26551/309210 objects misplaced (8.587%); 20 MiB/s, 319 objects/s recovering 2024-09-21T10:20:25.230 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:24 smithi107 ceph-mon[105022]: pgmap v1329: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.3 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 977 op/s; 26551/309210 objects misplaced (8.587%); 20 MiB/s, 319 objects/s recovering 2024-09-21T10:20:25.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:25.331 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:25.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:25.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:25.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:24 smithi129 ceph-mon[88471]: pgmap v1329: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.3 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 977 op/s; 26551/309210 objects misplaced (8.587%); 20 MiB/s, 319 objects/s recovering 2024-09-21T10:20:25.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:25.469 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:25.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:25.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:25.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:25.620 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:25.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:25.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:25.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:25.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:25.758 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:25.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:25.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:25.892 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:25.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:25.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:25.994 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:26.034 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:26.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:26.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:26.136 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:26.168 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:26.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:26.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:26.269 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:26.305 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:26.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:26.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:26.407 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:26.449 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:26.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:26.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:26.590 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:26.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:26.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:26.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:26.725 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:26.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:26.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:26.827 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:26.860 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:26.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:26.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:26.962 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:27.002 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:27.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:27.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:27.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:27.142 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:27.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:27.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:27.244 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:27.252 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:26 smithi107 ceph-mon[102221]: pgmap v1330: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 819 op/s; 24839/305874 objects misplaced (8.121%); 24 MiB/s, 376 objects/s recovering 2024-09-21T10:20:27.253 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:27.253 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:26 smithi107 ceph-mon[105022]: pgmap v1330: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 819 op/s; 24839/305874 objects misplaced (8.121%); 24 MiB/s, 376 objects/s recovering 2024-09-21T10:20:27.253 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:27.281 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:27.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:27.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:27.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:26 smithi129 ceph-mon[88471]: pgmap v1330: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 819 op/s; 24839/305874 objects misplaced (8.121%); 24 MiB/s, 376 objects/s recovering 2024-09-21T10:20:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:27.420 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:27.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:27.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:27.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:27.552 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:27.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:27.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:27.653 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:27.694 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:27.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:27.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:27.796 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:27.830 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:27.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:27.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:27.932 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:27.974 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:27.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:27.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:28.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:28.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:28.112 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:28.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:28.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:28.252 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:28.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:28.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:28.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:28.389 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:28.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:28.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:28.531 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:28.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:28.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:28.633 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:28.669 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:28.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:28.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:28.771 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:28.808 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:28.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:28.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:28.832 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-09-21T10:20:28.832 INFO:teuthology.orchestra.run.smithi107.stderr:0 2024-09-21T10:20:28.833 INFO:teuthology.orchestra.run.smithi107.stderr:1 2024-09-21T10:20:28.833 INFO:teuthology.orchestra.run.smithi107.stderr:2 2024-09-21T10:20:28.833 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2024-09-21T10:20:28.833 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2024-09-21T10:20:28.833 INFO:teuthology.orchestra.run.smithi107.stderr:bench [] [] [] [] 2024-09-21T10:20:28.833 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:20:28.833 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2024-09-21T10:20:28.833 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2024-09-21T10:20:28.833 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2024-09-21T10:20:28.834 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-09-21T10:20:28.870 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:20:28.870 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:20:28.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:28.942 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:28.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:28.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:29.001 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:20:29.044 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:29.085 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:29.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:29.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:29.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:28 smithi107 ceph-mon[102221]: pgmap v1331: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 925 op/s; 23807/298779 objects misplaced (7.968%); 20 MiB/s, 326 objects/s recovering 2024-09-21T10:20:29.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:28 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/263043619' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:20:29.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:28 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:20:29.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:28 smithi107 ceph-mon[105022]: pgmap v1331: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 925 op/s; 23807/298779 objects misplaced (7.968%); 20 MiB/s, 326 objects/s recovering 2024-09-21T10:20:29.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:28 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/263043619' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:20:29.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:28 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:20:29.186 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:29.222 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:29.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:29.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:29.324 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:29.358 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:29.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:29.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:29.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:28 smithi129 ceph-mon[88471]: pgmap v1331: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 925 op/s; 23807/298779 objects misplaced (7.968%); 20 MiB/s, 326 objects/s recovering 2024-09-21T10:20:29.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:28 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/263043619' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:20:29.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:28 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:20:29.460 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:29.487 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:29.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:29.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:29.589 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:29.620 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:29.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:29.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:29.722 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:29.764 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:29.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:29.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:29.840 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:20:29.840 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:20:29.865 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:29.868 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:20:29.869 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:20:29.903 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:20:29.904 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:20:29.909 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:29.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:29.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:29.931 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:20:29.931 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:20:29.963 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:20:29.964 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:20:29.992 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:20:29.992 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:20:30.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:30.027 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:20:30.027 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:20:30.043 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:30.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:30.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:30.056 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:20:30.056 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:20:30.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:30.173 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:30.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:30.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:30.200 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:29 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:20:30.200 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:29 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:20:30.200 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:29 smithi107 ceph-mon[102221]: osdmap e384: 8 total, 8 up, 7 in 2024-09-21T10:20:30.200 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:29 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1668380897' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:30.200 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:29 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:20:30.200 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:29 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:20:30.200 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:29 smithi107 ceph-mon[105022]: osdmap e384: 8 total, 8 up, 7 in 2024-09-21T10:20:30.200 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:29 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1668380897' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:30.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:30.306 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:30.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:30.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:30.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:30.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:29 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:20:30.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:29 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:20:30.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:29 smithi129 ceph-mon[88471]: osdmap e384: 8 total, 8 up, 7 in 2024-09-21T10:20:30.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:29 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1668380897' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:30.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:20:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:20:30.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:20:30.442 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:30.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:30.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:30.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:20:30.544 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:30.578 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:30.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:30.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:30.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:30.725 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:30.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:30.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:30.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:30.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:30.857 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:30.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:30.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:31.006 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:31.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:31.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:31.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:31.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:31.138 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:31.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:31.239 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:31.279 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:31.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:31.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:31.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:30 smithi107 ceph-mon[102221]: pgmap v1333: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 904 op/s; 23807/295200 objects misplaced (8.065%); 21 MiB/s, 341 objects/s recovering 2024-09-21T10:20:31.311 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:30 smithi107 ceph-mon[105022]: pgmap v1333: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 904 op/s; 23807/295200 objects misplaced (8.065%); 21 MiB/s, 341 objects/s recovering 2024-09-21T10:20:31.381 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:31.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:30 smithi129 ceph-mon[88471]: pgmap v1333: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 904 op/s; 23807/295200 objects misplaced (8.065%); 21 MiB/s, 341 objects/s recovering 2024-09-21T10:20:31.425 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:31.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:31.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:31.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:31.564 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:31.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:31.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:31.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:31.698 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:31.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:31.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:31.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:31.831 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:31.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:31.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:31.933 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:31.972 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:31.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:31.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:32.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:32.105 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:32.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:32.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:20:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:20:31] "GET /metrics HTTP/1.1" 200 36809 "" "Prometheus/2.51.0" 2024-09-21T10:20:32.206 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:32.241 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:32.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:32.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:32.343 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:32.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:32.385 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:32.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:32.424 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:20:32.424 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:20:32.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-21T10:20:32.486 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:32.523 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:32.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:32.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:32.625 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:32.661 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:32.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:32.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:32.762 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:32.798 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:32.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:32.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:32.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:32.937 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:32.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:32.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:32.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:20:33.039 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:33.078 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:33.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:33.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:33.180 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:33.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:33.213 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:33.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:33.280 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:33 smithi107 ceph-mon[102221]: pgmap v1334: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 5.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 918 op/s; 22388/290754 objects misplaced (7.700%); 19 MiB/s, 306 objects/s recovering 2024-09-21T10:20:33.280 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:33 smithi107 ceph-mon[105022]: pgmap v1334: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 5.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 918 op/s; 22388/290754 objects misplaced (7.700%); 19 MiB/s, 306 objects/s recovering 2024-09-21T10:20:33.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:33.354 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:33.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:33.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:33.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:33 smithi129 ceph-mon[88471]: pgmap v1334: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 5.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 918 op/s; 22388/290754 objects misplaced (7.700%); 19 MiB/s, 306 objects/s recovering 2024-09-21T10:20:33.455 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:33.486 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:33.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:33.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:33.588 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:33.620 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:33.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:33.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:33.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:33.754 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:33.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:33.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:33.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:33.859 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:33.905 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:33.945 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:33.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:33.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:34.047 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:34.088 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:34.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:34.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:34.189 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:34.237 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:34.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:34.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:34.338 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:34.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:34.380 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:34.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:34.481 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:34.514 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:34.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:34.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:34.616 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:34.651 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:34.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:34.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:34.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:34.786 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:34.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:34.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:34.887 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:34.913 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:34.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:34.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:35.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:35.036 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:35.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:35.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:35.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:35 smithi107 ceph-mon[105022]: pgmap v1335: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 941 op/s; 21456/280962 objects misplaced (7.637%); 18 MiB/s, 288 objects/s recovering 2024-09-21T10:20:35.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:35 smithi107 ceph-mon[105022]: osdmap e385: 8 total, 8 up, 7 in 2024-09-21T10:20:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:35 smithi107 ceph-mon[102221]: pgmap v1335: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 941 op/s; 21456/280962 objects misplaced (7.637%); 18 MiB/s, 288 objects/s recovering 2024-09-21T10:20:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:35 smithi107 ceph-mon[102221]: osdmap e385: 8 total, 8 up, 7 in 2024-09-21T10:20:35.139 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:35.162 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:35.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:35.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:35.264 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:35.287 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:35.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:35.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:35.322 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:20:35.389 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:35.413 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:35.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:35.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:35.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:35 smithi129 ceph-mon[88471]: pgmap v1335: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 941 op/s; 21456/280962 objects misplaced (7.637%); 18 MiB/s, 288 objects/s recovering 2024-09-21T10:20:35.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:35 smithi129 ceph-mon[88471]: osdmap e385: 8 total, 8 up, 7 in 2024-09-21T10:20:35.514 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:35.538 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:35.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:35.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:35.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:35.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:35.672 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:35.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:35.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:35.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:35.824 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:35.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:35.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:35.957 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:35.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:35.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:36.059 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:36.094 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:36.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:36.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:36.197 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:36.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:36.238 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:36.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:36.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:36.375 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:36.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:36.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:36.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:36.512 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:36.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:36.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:36.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:36.640 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:36.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:36.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:36.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:36.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:36.773 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:36.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:36 smithi107 ceph-mon[102221]: pgmap v1337: 109 pgs: 1 peering, 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 5.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 876 op/s; 19857/277731 objects misplaced (7.150%); 18 MiB/s, 284 objects/s recovering 2024-09-21T10:20:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:36 smithi107 ceph-mon[102221]: osdmap e386: 8 total, 8 up, 7 in 2024-09-21T10:20:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:36 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:36 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:36.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:36 smithi107 ceph-mon[105022]: pgmap v1337: 109 pgs: 1 peering, 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 5.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 876 op/s; 19857/277731 objects misplaced (7.150%); 18 MiB/s, 284 objects/s recovering 2024-09-21T10:20:36.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:36 smithi107 ceph-mon[105022]: osdmap e386: 8 total, 8 up, 7 in 2024-09-21T10:20:36.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:36 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:36.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:36 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:36.874 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:36.902 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:36.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:36.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:36 smithi129 ceph-mon[88471]: pgmap v1337: 109 pgs: 1 peering, 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 5.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 876 op/s; 19857/277731 objects misplaced (7.150%); 18 MiB/s, 284 objects/s recovering 2024-09-21T10:20:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:36 smithi129 ceph-mon[88471]: osdmap e386: 8 total, 8 up, 7 in 2024-09-21T10:20:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:36 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:36 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:37.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:37.036 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:37.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:37.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:37.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:37.165 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:37.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:37.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:37.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:37.300 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:37.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:37.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:37.402 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:37.432 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:37.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:37.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:37.534 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:37.558 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:37.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:37.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:37.659 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:37.680 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:37.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:37.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:37.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:37.811 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:37.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:37.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:37.913 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:37.942 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:37.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:37.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:38.044 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:38.072 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:38.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:38.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:38.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:38.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:38.219 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:38.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:38.322 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:38.361 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:38.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:38.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:38.463 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:38.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:38.500 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:38.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:38.601 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:38.641 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:38.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:38.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:38.743 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:38.781 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:38.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:38.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:38 smithi107 ceph-mon[102221]: pgmap v1339: 109 pgs: 1 peering, 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 974 op/s; 18548/271812 objects misplaced (6.824%); 25 MiB/s, 406 objects/s recovering 2024-09-21T10:20:38.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:38 smithi107 ceph-mon[105022]: pgmap v1339: 109 pgs: 1 peering, 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 974 op/s; 18548/271812 objects misplaced (6.824%); 25 MiB/s, 406 objects/s recovering 2024-09-21T10:20:38.882 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:38.915 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:38.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:38.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:38 smithi129 ceph-mon[88471]: pgmap v1339: 109 pgs: 1 peering, 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 974 op/s; 18548/271812 objects misplaced (6.824%); 25 MiB/s, 406 objects/s recovering 2024-09-21T10:20:39.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:39.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:39.049 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:39.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:39.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:39.176 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:39.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:39.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:39.277 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:39.316 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:39.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:39.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:39.419 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:39.453 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:39.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:39.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:39.555 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:39.596 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:39.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:39.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:39.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:39.729 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:39.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:39.830 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:39.864 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:39.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:39.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:39.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:40.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:40.005 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:40.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:40.057 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:20:40.057 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-09-21T10:20:40.057 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-09-21T10:20:40.057 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:20:40.057 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.057 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91069 11.8 11:1d9ce1cd:::benchmark_data_smithi107_218812_object91068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-21T10:20:40.058 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.369573+0000", 2024-09-21T10:20:40.058 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 583.68261978199996, 2024-09-21T10:20:40.058 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3066178669999999, 2024-09-21T10:20:40.058 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:40.058 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:40.058 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:20:40.058 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:40.058 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:20:40.058 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:20:40.058 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91069 2024-09-21T10:20:40.058 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.058 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:40.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:40.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.369573+0000", 2024-09-21T10:20:40.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:40.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.369573+0000", 2024-09-21T10:20:40.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.059 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.059 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.059 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:40.059 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.369575+0000", 2024-09-21T10:20:40.059 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.612e-06 2024-09-21T10:20:40.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.060 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:40.060 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.369624+0000", 2024-09-21T10:20:40.060 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9036999999999998e-05 2024-09-21T10:20:40.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.060 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:40.060 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.369645+0000", 2024-09-21T10:20:40.060 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0599999999999999e-05 2024-09-21T10:20:40.060 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.060 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.060 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:40.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.369653+0000", 2024-09-21T10:20:40.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1890000000000007e-06 2024-09-21T10:20:40.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.376149+0000", 2024-09-21T10:20:40.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0064962070000000004 2024-09-21T10:20:40.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.061 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.061 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:40.061 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.376301+0000", 2024-09-21T10:20:40.061 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000151757 2024-09-21T10:20:40.061 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:20:40.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.376460+0000", 2024-09-21T10:20:40.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000159586 2024-09-21T10:20:40.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:40.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379481+0000", 2024-09-21T10:20:40.062 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0030210580000000001 2024-09-21T10:20:40.062 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.062 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.062 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.062 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.386415+0000", 2024-09-21T10:20:40.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0069336040000000003 2024-09-21T10:20:40.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.403651+0000", 2024-09-21T10:20:40.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017236089 2024-09-21T10:20:40.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.063 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.403662+0000", 2024-09-21T10:20:40.063 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0421999999999999e-05 2024-09-21T10:20:40.063 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.063 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.063 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384390+0000", 2024-09-21T10:20:40.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.98072825299999999 2024-09-21T10:20:40.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384403+0000", 2024-09-21T10:20:40.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3186e-05 2024-09-21T10:20:40.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.064 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.676117+0000", 2024-09-21T10:20:40.064 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.291714217 2024-09-21T10:20:40.064 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.064 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.064 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:40.065 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.676191+0000", 2024-09-21T10:20:40.065 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4049999999999997e-05 2024-09-21T10:20:40.065 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.065 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.065 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.065 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.065 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.065 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91074 11.a 11:5a4cfa77:::benchmark_data_smithi107_218812_object91073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-09-21T10:20:40.065 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.379421+0000", 2024-09-21T10:20:40.065 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 583.67277266999997, 2024-09-21T10:20:40.065 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3240689140000002, 2024-09-21T10:20:40.065 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:40.065 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:40.066 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:20:40.066 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:40.066 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:20:40.066 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:20:40.066 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91074 2024-09-21T10:20:40.066 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.066 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:40.066 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.066 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:40.066 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379421+0000", 2024-09-21T10:20:40.066 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.066 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.066 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.066 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:40.067 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379421+0000", 2024-09-21T10:20:40.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.067 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.067 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.067 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:40.067 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379422+0000", 2024-09-21T10:20:40.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.513e-06 2024-09-21T10:20:40.067 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.067 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.067 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:40.067 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379443+0000", 2024-09-21T10:20:40.067 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1273000000000001e-05 2024-09-21T10:20:40.067 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.067 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.068 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:40.068 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379455+0000", 2024-09-21T10:20:40.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.203e-05 2024-09-21T10:20:40.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.068 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:40.068 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.379463+0000", 2024-09-21T10:20:40.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3329999999999999e-06 2024-09-21T10:20:40.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.068 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.068 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.386937+0000", 2024-09-21T10:20:40.068 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0074744290000000003 2024-09-21T10:20:40.068 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.068 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.069 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:40.069 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.387075+0000", 2024-09-21T10:20:40.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013819900000000001 2024-09-21T10:20:40.069 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.069 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.069 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:20:40.069 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.387218+0000", 2024-09-21T10:20:40.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000142705 2024-09-21T10:20:40.069 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.069 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.069 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:40.069 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.393122+0000", 2024-09-21T10:20:40.069 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005903747 2024-09-21T10:20:40.069 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.070 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.412062+0000", 2024-09-21T10:20:40.070 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.018940354 2024-09-21T10:20:40.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.070 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.412074+0000", 2024-09-21T10:20:40.070 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2333e-05 2024-09-21T10:20:40.070 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.070 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.070 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.070 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385002+0000", 2024-09-21T10:20:40.070 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.97292794599999999 2024-09-21T10:20:40.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.071 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.071 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385014+0000", 2024-09-21T10:20:40.071 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.182e-05 2024-09-21T10:20:40.071 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.071 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.687683+0000", 2024-09-21T10:20:40.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.302668766 2024-09-21T10:20:40.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.072 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.072 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for degraded object", 2024-09-21T10:20:40.072 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.688842+0000", 2024-09-21T10:20:40.072 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011593440000000001 2024-09-21T10:20:40.072 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.073 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.073 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.703390+0000", 2024-09-21T10:20:40.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.014547396000000001 2024-09-21T10:20:40.073 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.073 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:40.073 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.703489+0000", 2024-09-21T10:20:40.073 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9726e-05 2024-09-21T10:20:40.073 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.073 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.073 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.073 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.073 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.074 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91083 11.8 11:1024c88d:::benchmark_data_smithi107_218812_object91082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:20:40.074 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.401346+0000", 2024-09-21T10:20:40.074 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 583.65084762399999, 2024-09-21T10:20:40.074 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2836915819999999, 2024-09-21T10:20:40.074 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:40.074 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:40.074 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:40.074 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:40.074 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:20:40.074 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:20:40.074 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91083 2024-09-21T10:20:40.074 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.074 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:40.074 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:40.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401346+0000", 2024-09-21T10:20:40.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:40.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401346+0000", 2024-09-21T10:20:40.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.075 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.075 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:40.075 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401347+0000", 2024-09-21T10:20:40.075 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7010000000000001e-06 2024-09-21T10:20:40.075 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:40.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401621+0000", 2024-09-21T10:20:40.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00027367500000000003 2024-09-21T10:20:40.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:40.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401632+0000", 2024-09-21T10:20:40.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0723000000000001e-05 2024-09-21T10:20:40.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.076 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.076 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:40.076 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401640+0000", 2024-09-21T10:20:40.076 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9880000000000004e-06 2024-09-21T10:20:40.076 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.077 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.077 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404683+0000", 2024-09-21T10:20:40.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0030434799999999999 2024-09-21T10:20:40.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.077 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.077 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404691+0000", 2024-09-21T10:20:40.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0099999999999995e-06 2024-09-21T10:20:40.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.077 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.077 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.077 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384431+0000", 2024-09-21T10:20:40.077 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.97973983200000003 2024-09-21T10:20:40.077 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.078 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.078 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384438+0000", 2024-09-21T10:20:40.078 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8979999999999999e-06 2024-09-21T10:20:40.078 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.078 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.078 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.676420+0000", 2024-09-21T10:20:40.078 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.291981727 2024-09-21T10:20:40.078 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.078 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.078 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:40.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.676514+0000", 2024-09-21T10:20:40.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3908999999999998e-05 2024-09-21T10:20:40.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:20:40.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.676614+0000", 2024-09-21T10:20:40.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000100746 2024-09-21T10:20:40.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.079 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.079 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.681917+0000", 2024-09-21T10:20:40.079 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0053031550000000004 2024-09-21T10:20:40.079 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.079 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:40.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.682073+0000", 2024-09-21T10:20:40.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015567900000000001 2024-09-21T10:20:40.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.682113+0000", 2024-09-21T10:20:40.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9536999999999998e-05 2024-09-21T10:20:40.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.080 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.684892+0000", 2024-09-21T10:20:40.080 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027794719999999998 2024-09-21T10:20:40.080 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.080 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.080 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:40.081 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.684940+0000", 2024-09-21T10:20:40.081 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7732000000000003e-05 2024-09-21T10:20:40.081 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.081 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.081 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:40.081 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.685037+0000", 2024-09-21T10:20:40.081 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7318000000000006e-05 2024-09-21T10:20:40.081 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.081 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.081 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.081 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.081 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.081 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91086 11.8 11:14b98868:::benchmark_data_smithi107_218812_object91085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:20:40.081 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.401700+0000", 2024-09-21T10:20:40.081 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 583.65049295799997, 2024-09-21T10:20:40.082 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2935442479999999, 2024-09-21T10:20:40.082 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:40.082 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:40.082 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:40.082 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:40.082 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:20:40.082 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:20:40.082 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91086 2024-09-21T10:20:40.082 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.082 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:40.082 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.082 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:40.082 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401700+0000", 2024-09-21T10:20:40.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:40.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401700+0000", 2024-09-21T10:20:40.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:40.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401701+0000", 2024-09-21T10:20:40.083 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6899999999999997e-07 2024-09-21T10:20:40.083 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.083 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.083 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:40.083 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401935+0000", 2024-09-21T10:20:40.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000233957 2024-09-21T10:20:40.084 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.084 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.084 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:40.084 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401944+0000", 2024-09-21T10:20:40.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.799e-06 2024-09-21T10:20:40.084 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.084 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.084 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:40.084 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.401950+0000", 2024-09-21T10:20:40.084 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6499999999999999e-06 2024-09-21T10:20:40.084 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.084 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.084 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.085 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404712+0000", 2024-09-21T10:20:40.085 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0027615949999999999 2024-09-21T10:20:40.085 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.085 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.085 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.085 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404718+0000", 2024-09-21T10:20:40.085 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9359999999999996e-06 2024-09-21T10:20:40.085 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.085 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.085 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.085 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384462+0000", 2024-09-21T10:20:40.085 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.97974425499999995 2024-09-21T10:20:40.085 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.085 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.086 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.086 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384470+0000", 2024-09-21T10:20:40.086 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5209999999999997e-06 2024-09-21T10:20:40.086 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.086 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.086 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.086 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.679346+0000", 2024-09-21T10:20:40.086 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2948768799999999 2024-09-21T10:20:40.086 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.086 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.086 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:40.086 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.679498+0000", 2024-09-21T10:20:40.086 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015129 2024-09-21T10:20:40.086 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.087 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.087 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:20:40.087 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.679670+0000", 2024-09-21T10:20:40.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017223 2024-09-21T10:20:40.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.087 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.087 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:40.087 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.684756+0000", 2024-09-21T10:20:40.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0050856470000000004 2024-09-21T10:20:40.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.087 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.087 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.087 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.684829+0000", 2024-09-21T10:20:40.087 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3808999999999998e-05 2024-09-21T10:20:40.087 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695008+0000", 2024-09-21T10:20:40.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010178159000000001 2024-09-21T10:20:40.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695080+0000", 2024-09-21T10:20:40.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2880999999999996e-05 2024-09-21T10:20:40.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.088 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.088 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:40.088 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695139+0000", 2024-09-21T10:20:40.088 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8390000000000002e-05 2024-09-21T10:20:40.088 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.089 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:40.089 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695245+0000", 2024-09-21T10:20:40.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010568 2024-09-21T10:20:40.089 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.089 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.089 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.089 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.089 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.089 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91090 11.a 11:532a4ae5:::benchmark_data_smithi107_218812_object91089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:20:40.089 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.404927+0000", 2024-09-21T10:20:40.089 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 583.64726644400002, 2024-09-21T10:20:40.089 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.299055724, 2024-09-21T10:20:40.089 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:40.090 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:40.090 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:40.090 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:40.090 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:20:40.090 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:20:40.090 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91090 2024-09-21T10:20:40.090 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.090 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:40.090 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.090 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:40.090 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404927+0000", 2024-09-21T10:20:40.090 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.090 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.090 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.091 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:40.091 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404927+0000", 2024-09-21T10:20:40.091 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.091 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.091 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.091 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:40.091 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404929+0000", 2024-09-21T10:20:40.091 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9750000000000001e-06 2024-09-21T10:20:40.091 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.091 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.091 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:40.091 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404955+0000", 2024-09-21T10:20:40.091 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5783e-05 2024-09-21T10:20:40.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.092 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:40.092 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404965+0000", 2024-09-21T10:20:40.092 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0400999999999999e-05 2024-09-21T10:20:40.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.092 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:40.092 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.404973+0000", 2024-09-21T10:20:40.092 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4479999999999994e-06 2024-09-21T10:20:40.092 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.092 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.092 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.092 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.413662+0000", 2024-09-21T10:20:40.093 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0086882260000000003 2024-09-21T10:20:40.093 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.093 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.093 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.093 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.413672+0000", 2024-09-21T10:20:40.093 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0008e-05 2024-09-21T10:20:40.093 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.093 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.093 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.093 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385041+0000", 2024-09-21T10:20:40.093 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.97136919799999999 2024-09-21T10:20:40.093 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.093 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.093 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.093 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385049+0000", 2024-09-21T10:20:40.094 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.49e-06 2024-09-21T10:20:40.094 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.094 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.094 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.094 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.689104+0000", 2024-09-21T10:20:40.094 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.304054893 2024-09-21T10:20:40.094 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.094 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.094 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:40.094 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.689228+0000", 2024-09-21T10:20:40.094 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000124058 2024-09-21T10:20:40.094 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.094 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.094 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:20:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.689383+0000", 2024-09-21T10:20:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000154857 2024-09-21T10:20:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.700561+0000", 2024-09-21T10:20:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011177678 2024-09-21T10:20:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.700579+0000", 2024-09-21T10:20:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7887e-05 2024-09-21T10:20:40.095 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.096 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.096 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.096 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.701752+0000", 2024-09-21T10:20:40.096 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0011736050000000001 2024-09-21T10:20:40.096 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.096 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.096 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.096 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.703561+0000", 2024-09-21T10:20:40.096 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001809045 2024-09-21T10:20:40.096 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.096 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.096 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:40.096 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.703884+0000", 2024-09-21T10:20:40.096 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000322715 2024-09-21T10:20:40.096 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.097 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.097 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:40.097 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.703983+0000", 2024-09-21T10:20:40.097 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8456999999999996e-05 2024-09-21T10:20:40.097 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.097 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.097 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.097 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.097 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.097 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91107 11.8 11:16a99f3f:::benchmark_data_smithi107_218812_object91106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:20:40.097 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.432894+0000", 2024-09-21T10:20:40.097 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 583.61929887899998, 2024-09-21T10:20:40.097 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2626001310000001, 2024-09-21T10:20:40.097 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:40.098 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:40.098 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:40.098 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:40.098 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:20:40.098 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:20:40.098 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91107 2024-09-21T10:20:40.098 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.098 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:40.098 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.098 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:40.098 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.432894+0000", 2024-09-21T10:20:40.098 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.098 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.098 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.099 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:40.099 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.432894+0000", 2024-09-21T10:20:40.099 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.099 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.099 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.099 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:40.099 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.432896+0000", 2024-09-21T10:20:40.099 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.302e-06 2024-09-21T10:20:40.099 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.099 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.099 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:40.099 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.432919+0000", 2024-09-21T10:20:40.099 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2988000000000001e-05 2024-09-21T10:20:40.100 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.100 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.100 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:40.100 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.432937+0000", 2024-09-21T10:20:40.100 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8147000000000001e-05 2024-09-21T10:20:40.100 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.100 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.100 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:40.100 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.432944+0000", 2024-09-21T10:20:40.100 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6809999999999998e-06 2024-09-21T10:20:40.100 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.100 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.100 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.101 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.433008+0000", 2024-09-21T10:20:40.101 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3972000000000003e-05 2024-09-21T10:20:40.101 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.101 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.101 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.101 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.433025+0000", 2024-09-21T10:20:40.101 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6109000000000001e-05 2024-09-21T10:20:40.101 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.101 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.101 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.101 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384493+0000", 2024-09-21T10:20:40.101 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.95146804900000004 2024-09-21T10:20:40.101 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.101 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.102 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.102 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384499+0000", 2024-09-21T10:20:40.102 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8040000000000004e-06 2024-09-21T10:20:40.102 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.102 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.102 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.102 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.682149+0000", 2024-09-21T10:20:40.102 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.297649716 2024-09-21T10:20:40.102 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.102 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.102 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:40.102 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.682241+0000", 2024-09-21T10:20:40.102 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1515999999999996e-05 2024-09-21T10:20:40.103 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.103 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.103 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:20:40.103 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.682337+0000", 2024-09-21T10:20:40.103 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6506000000000001e-05 2024-09-21T10:20:40.103 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.103 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.103 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.103 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.685192+0000", 2024-09-21T10:20:40.103 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028547640000000001 2024-09-21T10:20:40.103 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.103 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.103 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:40.103 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.685203+0000", 2024-09-21T10:20:40.103 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1555e-05 2024-09-21T10:20:40.104 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.104 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.104 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.104 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695046+0000", 2024-09-21T10:20:40.104 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0098422700000000002 2024-09-21T10:20:40.104 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.104 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.104 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.104 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695355+0000", 2024-09-21T10:20:40.104 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000309005 2024-09-21T10:20:40.104 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.104 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.105 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:40.105 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695402+0000", 2024-09-21T10:20:40.105 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7172999999999997e-05 2024-09-21T10:20:40.105 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.105 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.105 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:40.105 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695494+0000", 2024-09-21T10:20:40.105 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2573999999999998e-05 2024-09-21T10:20:40.105 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.105 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.105 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.105 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.105 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.105 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91117 11.a 11:54639fb7:::benchmark_data_smithi107_218812_object91116:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:20:40.105 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.443321+0000", 2024-09-21T10:20:40.106 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 583.60887215000002, 2024-09-21T10:20:40.106 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2608621439999999, 2024-09-21T10:20:40.106 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:40.106 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:40.106 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:40.106 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:40.106 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:20:40.106 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:20:40.106 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91117 2024-09-21T10:20:40.106 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.106 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:40.106 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.106 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:40.107 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443321+0000", 2024-09-21T10:20:40.107 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.107 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.107 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.107 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:40.107 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443321+0000", 2024-09-21T10:20:40.107 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.107 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.107 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.107 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:40.107 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443323+0000", 2024-09-21T10:20:40.107 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.604e-06 2024-09-21T10:20:40.107 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.107 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.108 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:40.108 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443345+0000", 2024-09-21T10:20:40.108 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2370000000000001e-05 2024-09-21T10:20:40.108 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.108 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.108 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:40.108 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443358+0000", 2024-09-21T10:20:40.108 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3047999999999999e-05 2024-09-21T10:20:40.108 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.108 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.108 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:40.108 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443366+0000", 2024-09-21T10:20:40.108 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1089999999999998e-06 2024-09-21T10:20:40.108 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.108 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.109 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.109 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443456+0000", 2024-09-21T10:20:40.109 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0204999999999994e-05 2024-09-21T10:20:40.109 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.109 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.109 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.109 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443464+0000", 2024-09-21T10:20:40.109 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.323e-06 2024-09-21T10:20:40.109 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.109 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.109 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.109 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385072+0000", 2024-09-21T10:20:40.109 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.94160824399999998 2024-09-21T10:20:40.109 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.110 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.110 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.110 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385079+0000", 2024-09-21T10:20:40.110 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3880000000000002e-06 2024-09-21T10:20:40.110 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.110 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.110 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.110 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.692299+0000", 2024-09-21T10:20:40.110 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3072192389999999 2024-09-21T10:20:40.110 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.110 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.110 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:40.110 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.692459+0000", 2024-09-21T10:20:40.110 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000160686 2024-09-21T10:20:40.111 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.111 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.111 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:20:40.111 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.692664+0000", 2024-09-21T10:20:40.111 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000204692 2024-09-21T10:20:40.111 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.111 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.111 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:40.111 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.700714+0000", 2024-09-21T10:20:40.111 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.008050092 2024-09-21T10:20:40.111 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.111 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.111 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.111 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.700959+0000", 2024-09-21T10:20:40.112 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024535799999999999 2024-09-21T10:20:40.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.112 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.112 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.112 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.701790+0000", 2024-09-21T10:20:40.112 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00083083300000000001 2024-09-21T10:20:40.112 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.112 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:40.113 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.113 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.113 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704027+0000", 2024-09-21T10:20:40.113 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022363589999999998 2024-09-21T10:20:40.113 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.114 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.114 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:40.114 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704086+0000", 2024-09-21T10:20:40.114 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9502000000000001e-05 2024-09-21T10:20:40.114 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.114 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.114 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:40.114 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704183+0000", 2024-09-21T10:20:40.114 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7091999999999998e-05 2024-09-21T10:20:40.114 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.114 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.114 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.114 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.115 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.115 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91118 11.a 11:56ad5097:::benchmark_data_smithi107_218812_object91117:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:20:40.115 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.443509+0000", 2024-09-21T10:20:40.115 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 583.60868396299998, 2024-09-21T10:20:40.115 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2609485060000001, 2024-09-21T10:20:40.115 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:40.115 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:40.115 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:40.115 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:40.115 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:20:40.115 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:20:40.115 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91118 2024-09-21T10:20:40.115 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.115 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:40.116 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.116 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:40.116 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443509+0000", 2024-09-21T10:20:40.116 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.116 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.116 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:40.116 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443509+0000", 2024-09-21T10:20:40.116 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.116 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.116 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.116 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:40.116 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443510+0000", 2024-09-21T10:20:40.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6799999999999999e-07 2024-09-21T10:20:40.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.117 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:40.117 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443530+0000", 2024-09-21T10:20:40.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9551e-05 2024-09-21T10:20:40.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.117 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:40.117 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443543+0000", 2024-09-21T10:20:40.117 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3575e-05 2024-09-21T10:20:40.117 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.117 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.117 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:40.117 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443550+0000", 2024-09-21T10:20:40.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5969999999999997e-06 2024-09-21T10:20:40.118 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.118 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.118 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.118 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443616+0000", 2024-09-21T10:20:40.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6740000000000001e-05 2024-09-21T10:20:40.118 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.118 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.118 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.118 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.443624+0000", 2024-09-21T10:20:40.118 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8220000000000007e-06 2024-09-21T10:20:40.118 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.118 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.118 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.118 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385103+0000", 2024-09-21T10:20:40.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.94147888199999996 2024-09-21T10:20:40.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.119 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.119 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.385111+0000", 2024-09-21T10:20:40.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.486e-06 2024-09-21T10:20:40.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.119 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.119 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695503+0000", 2024-09-21T10:20:40.119 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3103919070000001 2024-09-21T10:20:40.119 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.119 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.119 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:40.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695648+0000", 2024-09-21T10:20:40.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014547799999999999 2024-09-21T10:20:40.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.120 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:20:40.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.696550+0000", 2024-09-21T10:20:40.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00090179799999999999 2024-09-21T10:20:40.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.120 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:40.120 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.700722+0000", 2024-09-21T10:20:40.120 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041717899999999999 2024-09-21T10:20:40.120 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.120 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.121 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.121 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.701004+0000", 2024-09-21T10:20:40.121 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00028281900000000002 2024-09-21T10:20:40.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.121 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.121 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.701834+0000", 2024-09-21T10:20:40.121 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00082920400000000005 2024-09-21T10:20:40.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.121 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.121 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704324+0000", 2024-09-21T10:20:40.121 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002490632 2024-09-21T10:20:40.121 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.121 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.122 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:40.122 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704377+0000", 2024-09-21T10:20:40.122 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2439999999999999e-05 2024-09-21T10:20:40.122 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.122 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.122 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:40.122 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.704458+0000", 2024-09-21T10:20:40.122 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1017e-05 2024-09-21T10:20:40.122 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.122 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.122 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.122 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.122 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.122 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:91121 11.8 11:16dab647:::benchmark_data_smithi107_218812_object91120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-09-21T10:20:40.123 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:10:56.447395+0000", 2024-09-21T10:20:40.123 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 583.60479777900002, 2024-09-21T10:20:40.123 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.248324582, 2024-09-21T10:20:40.123 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:40.123 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:40.123 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:40.123 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:40.123 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:20:40.123 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:20:40.123 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 91121 2024-09-21T10:20:40.123 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.124 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:40.124 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.124 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:40.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447395+0000", 2024-09-21T10:20:40.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.124 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.124 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.124 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:40.124 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447395+0000", 2024-09-21T10:20:40.124 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.124 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.124 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.124 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:40.125 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447397+0000", 2024-09-21T10:20:40.125 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2330000000000001e-06 2024-09-21T10:20:40.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.125 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.125 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:40.125 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447448+0000", 2024-09-21T10:20:40.125 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1107999999999999e-05 2024-09-21T10:20:40.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.125 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.125 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:40.125 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447459+0000", 2024-09-21T10:20:40.125 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1241e-05 2024-09-21T10:20:40.125 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.125 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.126 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:40.126 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447467+0000", 2024-09-21T10:20:40.126 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8970000000000008e-06 2024-09-21T10:20:40.126 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.126 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.126 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.126 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447518+0000", 2024-09-21T10:20:40.126 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1329999999999998e-05 2024-09-21T10:20:40.126 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.126 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.126 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.126 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:56.447525+0000", 2024-09-21T10:20:40.126 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5379999999999999e-06 2024-09-21T10:20:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384523+0000", 2024-09-21T10:20:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.93699787099999998 2024-09-21T10:20:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:57.384530+0000", 2024-09-21T10:20:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8480000000000003e-06 2024-09-21T10:20:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.127 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.685237+0000", 2024-09-21T10:20:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.300707727 2024-09-21T10:20:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.685376+0000", 2024-09-21T10:20:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000139204 2024-09-21T10:20:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:20:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.685545+0000", 2024-09-21T10:20:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000168853 2024-09-21T10:20:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.128 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695283+0000", 2024-09-21T10:20:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0097372459999999997 2024-09-21T10:20:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695320+0000", 2024-09-21T10:20:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7492e-05 2024-09-21T10:20:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695535+0000", 2024-09-21T10:20:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021545700000000001 2024-09-21T10:20:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.129 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:40.130 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695584+0000", 2024-09-21T10:20:40.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8458000000000003e-05 2024-09-21T10:20:40.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.130 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.130 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:40.130 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695627+0000", 2024-09-21T10:20:40.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3126e-05 2024-09-21T10:20:40.130 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.130 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.130 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:40.130 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:10:58.695720+0000", 2024-09-21T10:20:40.130 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2953000000000003e-05 2024-09-21T10:20:40.130 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.130 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.131 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.131 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.131 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.131 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135668 12.11 12:88ec77bf:::benchmark_data_smithi107_250110_object24258:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:20:40.131 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.091435+0000", 2024-09-21T10:20:40.131 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 267.96075800099999, 2024-09-21T10:20:40.131 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7784578739999999, 2024-09-21T10:20:40.131 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:40.131 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:40.131 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:40.131 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:40.131 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:40.131 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:40.132 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135668 2024-09-21T10:20:40.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.132 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:40.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:40.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.091435+0000", 2024-09-21T10:20:40.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.132 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:40.132 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.091435+0000", 2024-09-21T10:20:40.132 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.132 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.132 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:40.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.091437+0000", 2024-09-21T10:20:40.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.341e-06 2024-09-21T10:20:40.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:40.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.091441+0000", 2024-09-21T10:20:40.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8210000000000001e-06 2024-09-21T10:20:40.133 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.133 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.133 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:40.133 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.091460+0000", 2024-09-21T10:20:40.133 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8746999999999999e-05 2024-09-21T10:20:40.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:40.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.091469+0000", 2024-09-21T10:20:40.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.388e-06 2024-09-21T10:20:40.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.091551+0000", 2024-09-21T10:20:40.134 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2844999999999997e-05 2024-09-21T10:20:40.134 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.134 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.134 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.134 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.091560+0000", 2024-09-21T10:20:40.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0659999999999993e-06 2024-09-21T10:20:40.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.135 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.135 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.135 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.580580+0000", 2024-09-21T10:20:40.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.489019859 2024-09-21T10:20:40.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.135 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.135 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.135 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.580591+0000", 2024-09-21T10:20:40.135 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1009000000000001e-05 2024-09-21T10:20:40.135 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.135 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.135 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.136 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.865360+0000", 2024-09-21T10:20:40.136 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2847688399999999 2024-09-21T10:20:40.136 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.136 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:40.136 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.865459+0000", 2024-09-21T10:20:40.136 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8939999999999998e-05 2024-09-21T10:20:40.136 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.136 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:20:40.136 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.865645+0000", 2024-09-21T10:20:40.136 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018592599999999999 2024-09-21T10:20:40.136 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.136 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.137 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869202+0000", 2024-09-21T10:20:40.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0035568499999999999 2024-09-21T10:20:40.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.137 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869418+0000", 2024-09-21T10:20:40.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021635299999999999 2024-09-21T10:20:40.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:40.137 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869438+0000", 2024-09-21T10:20:40.137 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.942e-05 2024-09-21T10:20:40.137 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.137 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.137 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.138 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869617+0000", 2024-09-21T10:20:40.138 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000179703 2024-09-21T10:20:40.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.138 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.138 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.138 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869674+0000", 2024-09-21T10:20:40.138 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6777000000000002e-05 2024-09-21T10:20:40.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.138 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.138 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:40.138 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869734+0000", 2024-09-21T10:20:40.138 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0359000000000003e-05 2024-09-21T10:20:40.138 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.138 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.138 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:40.139 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869893+0000", 2024-09-21T10:20:40.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015862999999999999 2024-09-21T10:20:40.139 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.139 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.139 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.139 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.139 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.139 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135696 12.11 12:8cd3885c:::benchmark_data_smithi107_250110_object24286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:20:40.139 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.109943+0000", 2024-09-21T10:20:40.139 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 267.94225009899998, 2024-09-21T10:20:40.139 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.761551254, 2024-09-21T10:20:40.139 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:40.140 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:40.140 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:40.140 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:40.140 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:40.140 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:40.140 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135696 2024-09-21T10:20:40.140 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.140 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:40.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.140 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:40.140 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.109943+0000", 2024-09-21T10:20:40.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.140 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.140 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:40.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.109943+0000", 2024-09-21T10:20:40.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:40.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.109944+0000", 2024-09-21T10:20:40.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.89e-07 2024-09-21T10:20:40.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:40.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.109946+0000", 2024-09-21T10:20:40.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3590000000000002e-06 2024-09-21T10:20:40.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:40.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.109952+0000", 2024-09-21T10:20:40.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7830000000000004e-06 2024-09-21T10:20:40.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:40.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.109957+0000", 2024-09-21T10:20:40.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9679999999999997e-06 2024-09-21T10:20:40.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.110899+0000", 2024-09-21T10:20:40.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00094190099999999998 2024-09-21T10:20:40.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.110908+0000", 2024-09-21T10:20:40.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9099999999999994e-06 2024-09-21T10:20:40.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.143 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581020+0000", 2024-09-21T10:20:40.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.470112478 2024-09-21T10:20:40.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.143 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581029+0000", 2024-09-21T10:20:40.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1130000000000007e-06 2024-09-21T10:20:40.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.866702+0000", 2024-09-21T10:20:40.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.285672996 2024-09-21T10:20:40.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:40.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.866775+0000", 2024-09-21T10:20:40.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2391000000000005e-05 2024-09-21T10:20:40.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:20:40.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.866946+0000", 2024-09-21T10:20:40.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017150299999999999 2024-09-21T10:20:40.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869480+0000", 2024-09-21T10:20:40.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002534002 2024-09-21T10:20:40.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:40.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869506+0000", 2024-09-21T10:20:40.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6100000000000001e-05 2024-09-21T10:20:40.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869569+0000", 2024-09-21T10:20:40.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3360000000000003e-05 2024-09-21T10:20:40.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871245+0000", 2024-09-21T10:20:40.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001675775 2024-09-21T10:20:40.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871299+0000", 2024-09-21T10:20:40.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3761999999999998e-05 2024-09-21T10:20:40.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.147 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:40.147 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871360+0000", 2024-09-21T10:20:40.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1044999999999996e-05 2024-09-21T10:20:40.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.147 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:40.147 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871494+0000", 2024-09-21T10:20:40.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013431900000000001 2024-09-21T10:20:40.147 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.147 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.147 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.148 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135783 12.11 12:8ba445ab:::benchmark_data_smithi107_250110_object24373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:20:40.148 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.184409+0000", 2024-09-21T10:20:40.148 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 267.86778403800002, 2024-09-21T10:20:40.148 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.687499938, 2024-09-21T10:20:40.148 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:40.148 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:40.148 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:40.148 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:40.148 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:40.148 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:40.148 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135783 2024-09-21T10:20:40.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.148 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:40.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:40.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.184409+0000", 2024-09-21T10:20:40.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:40.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.184409+0000", 2024-09-21T10:20:40.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:40.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.184411+0000", 2024-09-21T10:20:40.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.455e-06 2024-09-21T10:20:40.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:40.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.184416+0000", 2024-09-21T10:20:40.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3059999999999997e-06 2024-09-21T10:20:40.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:40.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.184427+0000", 2024-09-21T10:20:40.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1483e-05 2024-09-21T10:20:40.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:40.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.184434+0000", 2024-09-21T10:20:40.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0310000000000001e-06 2024-09-21T10:20:40.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.184495+0000", 2024-09-21T10:20:40.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0273999999999999e-05 2024-09-21T10:20:40.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.184505+0000", 2024-09-21T10:20:40.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8349999999999999e-06 2024-09-21T10:20:40.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581054+0000", 2024-09-21T10:20:40.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.39654935299999999 2024-09-21T10:20:40.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581061+0000", 2024-09-21T10:20:40.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5299999999999999e-06 2024-09-21T10:20:40.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.867976+0000", 2024-09-21T10:20:40.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.286914812 2024-09-21T10:20:40.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:40.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.868095+0000", 2024-09-21T10:20:40.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011861799999999999 2024-09-21T10:20:40.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.153 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:20:40.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.868306+0000", 2024-09-21T10:20:40.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021122400000000001 2024-09-21T10:20:40.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.153 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:40.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869515+0000", 2024-09-21T10:20:40.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001208885 2024-09-21T10:20:40.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.153 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871185+0000", 2024-09-21T10:20:40.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0016699849999999999 2024-09-21T10:20:40.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871567+0000", 2024-09-21T10:20:40.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038179499999999999 2024-09-21T10:20:40.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871683+0000", 2024-09-21T10:20:40.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011604600000000001 2024-09-21T10:20:40.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871741+0000", 2024-09-21T10:20:40.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8096000000000001e-05 2024-09-21T10:20:40.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:40.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871791+0000", 2024-09-21T10:20:40.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0105999999999998e-05 2024-09-21T10:20:40.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:40.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871909+0000", 2024-09-21T10:20:40.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011810400000000001 2024-09-21T10:20:40.155 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.155 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.156 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.156 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135785 12.11 12:8b794dd9:::benchmark_data_smithi107_250110_object24375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:20:40.156 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.186985+0000", 2024-09-21T10:20:40.156 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 267.86520809299998, 2024-09-21T10:20:40.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6870419459999999, 2024-09-21T10:20:40.156 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:40.156 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:40.156 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:40.156 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:40.156 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:40.156 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:40.156 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135785 2024-09-21T10:20:40.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.157 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:40.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.157 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:40.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.186985+0000", 2024-09-21T10:20:40.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.157 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:40.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.186985+0000", 2024-09-21T10:20:40.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.157 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:40.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.186987+0000", 2024-09-21T10:20:40.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.508e-06 2024-09-21T10:20:40.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:40.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.186992+0000", 2024-09-21T10:20:40.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7049999999999996e-06 2024-09-21T10:20:40.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:40.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.187005+0000", 2024-09-21T10:20:40.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2259000000000001e-05 2024-09-21T10:20:40.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:40.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.187012+0000", 2024-09-21T10:20:40.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5190000000000003e-06 2024-09-21T10:20:40.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.159 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.188537+0000", 2024-09-21T10:20:40.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0015246579999999999 2024-09-21T10:20:40.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.159 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.188545+0000", 2024-09-21T10:20:40.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6959999999999994e-06 2024-09-21T10:20:40.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.160 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.160 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.160 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581086+0000", 2024-09-21T10:20:40.160 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.39254024199999998 2024-09-21T10:20:40.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.160 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.160 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.160 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581093+0000", 2024-09-21T10:20:40.160 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5850000000000002e-06 2024-09-21T10:20:40.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.160 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.160 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.160 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.869950+0000", 2024-09-21T10:20:40.161 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2888568010000001 2024-09-21T10:20:40.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.161 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:40.161 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.870065+0000", 2024-09-21T10:20:40.161 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114547 2024-09-21T10:20:40.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.161 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:20:40.161 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.870201+0000", 2024-09-21T10:20:40.161 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013678099999999999 2024-09-21T10:20:40.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.161 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:40.162 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871587+0000", 2024-09-21T10:20:40.162 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001385266 2024-09-21T10:20:40.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.162 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.162 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.873496+0000", 2024-09-21T10:20:40.162 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001909315 2024-09-21T10:20:40.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.162 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.162 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.873546+0000", 2024-09-21T10:20:40.162 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9809999999999999e-05 2024-09-21T10:20:40.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.163 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.163 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.873607+0000", 2024-09-21T10:20:40.163 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1660000000000003e-05 2024-09-21T10:20:40.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.163 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.163 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.163 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.873848+0000", 2024-09-21T10:20:40.163 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024019500000000001 2024-09-21T10:20:40.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.163 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.163 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:40.163 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.873905+0000", 2024-09-21T10:20:40.163 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6950000000000002e-05 2024-09-21T10:20:40.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.164 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:40.164 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.874027+0000", 2024-09-21T10:20:40.164 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012244900000000001 2024-09-21T10:20:40.164 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.164 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.164 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.164 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135795 12.11 12:8de107fc:::benchmark_data_smithi107_250110_object24385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:20:40.164 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.194829+0000", 2024-09-21T10:20:40.164 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 267.857364647, 2024-09-21T10:20:40.164 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6837946939999999, 2024-09-21T10:20:40.164 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:40.164 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:40.165 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:40.165 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:40.165 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:40.165 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:40.165 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135795 2024-09-21T10:20:40.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.165 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:40.165 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.165 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:40.165 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.194829+0000", 2024-09-21T10:20:40.165 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.165 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.166 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:40.166 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.194829+0000", 2024-09-21T10:20:40.166 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.166 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:40.166 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.194830+0000", 2024-09-21T10:20:40.166 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.544e-06 2024-09-21T10:20:40.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.166 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:40.166 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.194838+0000", 2024-09-21T10:20:40.166 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8259999999999993e-06 2024-09-21T10:20:40.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.167 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:40.167 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.194849+0000", 2024-09-21T10:20:40.167 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1112e-05 2024-09-21T10:20:40.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.167 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:40.167 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.194856+0000", 2024-09-21T10:20:40.167 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0190000000000001e-06 2024-09-21T10:20:40.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.167 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.168 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.194912+0000", 2024-09-21T10:20:40.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5736999999999998e-05 2024-09-21T10:20:40.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.169 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.194921+0000", 2024-09-21T10:20:40.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6600000000000007e-06 2024-09-21T10:20:40.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.169 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581119+0000", 2024-09-21T10:20:40.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.38619732899999998 2024-09-21T10:20:40.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.170 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.170 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581126+0000", 2024-09-21T10:20:40.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.092e-06 2024-09-21T10:20:40.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.170 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.170 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.871943+0000", 2024-09-21T10:20:40.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2908169979999999 2024-09-21T10:20:40.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.170 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:40.170 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.872055+0000", 2024-09-21T10:20:40.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000111839 2024-09-21T10:20:40.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.171 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.171 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:20:40.171 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.872274+0000", 2024-09-21T10:20:40.171 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021912799999999999 2024-09-21T10:20:40.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.171 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.171 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:40.171 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.876414+0000", 2024-09-21T10:20:40.171 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004140477 2024-09-21T10:20:40.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.171 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.171 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.171 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878212+0000", 2024-09-21T10:20:40.172 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017976890000000001 2024-09-21T10:20:40.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.172 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.172 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878270+0000", 2024-09-21T10:20:40.172 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7739999999999999e-05 2024-09-21T10:20:40.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.172 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.172 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878329+0000", 2024-09-21T10:20:40.172 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9713999999999998e-05 2024-09-21T10:20:40.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.172 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.173 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878413+0000", 2024-09-21T10:20:40.173 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3217e-05 2024-09-21T10:20:40.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.173 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:40.173 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878508+0000", 2024-09-21T10:20:40.173 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5184000000000002e-05 2024-09-21T10:20:40.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.173 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:40.173 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878623+0000", 2024-09-21T10:20:40.173 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000115389 2024-09-21T10:20:40.173 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.173 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.174 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.174 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.174 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.174 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135809 12.11 12:8db42dc0:::benchmark_data_smithi107_250110_object24399:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:20:40.174 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.207496+0000", 2024-09-21T10:20:40.174 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 267.84469759000001, 2024-09-21T10:20:40.174 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6738327019999999, 2024-09-21T10:20:40.174 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:40.174 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:40.174 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:40.174 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:40.174 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:40.174 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:40.175 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135809 2024-09-21T10:20:40.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.175 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:40.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.175 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:40.175 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207496+0000", 2024-09-21T10:20:40.175 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.175 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:40.175 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207496+0000", 2024-09-21T10:20:40.175 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.176 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:40.176 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207497+0000", 2024-09-21T10:20:40.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.263e-06 2024-09-21T10:20:40.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.176 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.176 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:40.176 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207501+0000", 2024-09-21T10:20:40.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0890000000000002e-06 2024-09-21T10:20:40.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.176 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.176 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:40.176 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207510+0000", 2024-09-21T10:20:40.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.5620000000000004e-06 2024-09-21T10:20:40.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.177 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.177 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:40.177 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207516+0000", 2024-09-21T10:20:40.177 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4779999999999996e-06 2024-09-21T10:20:40.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.177 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.177 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.177 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207580+0000", 2024-09-21T10:20:40.177 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4138000000000002e-05 2024-09-21T10:20:40.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.177 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.177 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.177 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207589+0000", 2024-09-21T10:20:40.177 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0510000000000002e-06 2024-09-21T10:20:40.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.178 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.178 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581151+0000", 2024-09-21T10:20:40.178 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.37356210099999998 2024-09-21T10:20:40.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.178 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.178 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581228+0000", 2024-09-21T10:20:40.178 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7077999999999996e-05 2024-09-21T10:20:40.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.178 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.178 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.874074+0000", 2024-09-21T10:20:40.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2928457309999999 2024-09-21T10:20:40.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.179 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:40.179 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.874194+0000", 2024-09-21T10:20:40.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000120315 2024-09-21T10:20:40.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.179 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:20:40.179 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.874495+0000", 2024-09-21T10:20:40.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030059999999999999 2024-09-21T10:20:40.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:40.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878342+0000", 2024-09-21T10:20:40.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038473410000000002 2024-09-21T10:20:40.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878373+0000", 2024-09-21T10:20:40.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0340000000000001e-05 2024-09-21T10:20:40.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878659+0000", 2024-09-21T10:20:40.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00028606999999999998 2024-09-21T10:20:40.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878766+0000", 2024-09-21T10:20:40.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000107095 2024-09-21T10:20:40.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.881101+0000", 2024-09-21T10:20:40.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0023348700000000002 2024-09-21T10:20:40.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:40.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.881202+0000", 2024-09-21T10:20:40.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000100945 2024-09-21T10:20:40.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:40.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.881328+0000", 2024-09-21T10:20:40.182 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000126635 2024-09-21T10:20:40.182 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.182 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.182 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.182 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135810 12.11 12:8861f427:::benchmark_data_smithi107_250110_object24400:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:20:40.182 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.207613+0000", 2024-09-21T10:20:40.182 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 267.84458057099999, 2024-09-21T10:20:40.182 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6877408169999999, 2024-09-21T10:20:40.182 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:40.182 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:40.182 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:40.182 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:40.183 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:40.183 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:40.183 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135810 2024-09-21T10:20:40.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.183 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:40.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.183 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:40.183 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207613+0000", 2024-09-21T10:20:40.183 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.183 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:40.183 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207613+0000", 2024-09-21T10:20:40.183 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.184 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.184 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:40.184 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207613+0000", 2024-09-21T10:20:40.184 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5700000000000002e-07 2024-09-21T10:20:40.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.184 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.184 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:40.184 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207616+0000", 2024-09-21T10:20:40.184 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8650000000000001e-06 2024-09-21T10:20:40.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.184 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.184 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:40.184 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207623+0000", 2024-09-21T10:20:40.184 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.199e-06 2024-09-21T10:20:40.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.185 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.185 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:40.185 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207628+0000", 2024-09-21T10:20:40.185 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1900000000000003e-06 2024-09-21T10:20:40.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.185 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.185 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.185 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207671+0000", 2024-09-21T10:20:40.185 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2518999999999997e-05 2024-09-21T10:20:40.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.185 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.185 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.186 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.207678+0000", 2024-09-21T10:20:40.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4830000000000002e-06 2024-09-21T10:20:40.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.186 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.186 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581308+0000", 2024-09-21T10:20:40.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.37362985799999998 2024-09-21T10:20:40.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.186 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.186 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581318+0000", 2024-09-21T10:20:40.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2380000000000003e-06 2024-09-21T10:20:40.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.876458+0000", 2024-09-21T10:20:40.187 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.295139985 2024-09-21T10:20:40.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:40.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.876571+0000", 2024-09-21T10:20:40.187 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000113198 2024-09-21T10:20:40.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:20:40.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.876799+0000", 2024-09-21T10:20:40.187 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022833099999999999 2024-09-21T10:20:40.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:40.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878384+0000", 2024-09-21T10:20:40.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001584785 2024-09-21T10:20:40.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.880888+0000", 2024-09-21T10:20:40.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025038119999999998 2024-09-21T10:20:40.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.880931+0000", 2024-09-21T10:20:40.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3463e-05 2024-09-21T10:20:40.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.881056+0000", 2024-09-21T10:20:40.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012447000000000001 2024-09-21T10:20:40.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.895044+0000", 2024-09-21T10:20:40.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.013988377999999999 2024-09-21T10:20:40.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:40.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.895100+0000", 2024-09-21T10:20:40.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5686000000000002e-05 2024-09-21T10:20:40.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.190 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:40.190 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.895353+0000", 2024-09-21T10:20:40.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00025379999999999999 2024-09-21T10:20:40.190 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.190 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.190 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.190 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135845 12.11 12:8c5bb777:::benchmark_data_smithi107_250110_object24435:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:20:40.190 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.237799+0000", 2024-09-21T10:20:40.190 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 267.81439402699999, 2024-09-21T10:20:40.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6577566849999998, 2024-09-21T10:20:40.191 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:40.191 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:40.191 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:40.191 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:40.191 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:40.191 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:40.191 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135845 2024-09-21T10:20:40.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.191 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:40.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.191 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:40.191 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.237799+0000", 2024-09-21T10:20:40.192 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.192 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:40.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.237799+0000", 2024-09-21T10:20:40.192 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.192 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:40.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.237801+0000", 2024-09-21T10:20:40.192 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7099999999999999e-06 2024-09-21T10:20:40.192 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.192 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.192 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:40.192 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.237807+0000", 2024-09-21T10:20:40.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7969999999999997e-06 2024-09-21T10:20:40.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.193 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:40.193 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.237829+0000", 2024-09-21T10:20:40.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2353000000000001e-05 2024-09-21T10:20:40.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.193 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:40.193 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.237837+0000", 2024-09-21T10:20:40.193 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5190000000000003e-06 2024-09-21T10:20:40.193 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.193 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.193 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.237953+0000", 2024-09-21T10:20:40.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000115948 2024-09-21T10:20:40.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.237962+0000", 2024-09-21T10:20:40.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9690000000000006e-06 2024-09-21T10:20:40.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.194 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.194 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581347+0000", 2024-09-21T10:20:40.194 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.34338489700000002 2024-09-21T10:20:40.194 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.194 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581355+0000", 2024-09-21T10:20:40.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9780000000000005e-06 2024-09-21T10:20:40.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878792+0000", 2024-09-21T10:20:40.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2974364190000001 2024-09-21T10:20:40.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.195 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:40.195 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.878909+0000", 2024-09-21T10:20:40.195 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011755 2024-09-21T10:20:40.195 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.195 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:20:40.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.879130+0000", 2024-09-21T10:20:40.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022026599999999999 2024-09-21T10:20:40.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.881376+0000", 2024-09-21T10:20:40.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022460739999999998 2024-09-21T10:20:40.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.196 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.196 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.196 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.881415+0000", 2024-09-21T10:20:40.196 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9267999999999997e-05 2024-09-21T10:20:40.196 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.197 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.197 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.881456+0000", 2024-09-21T10:20:40.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1029000000000003e-05 2024-09-21T10:20:40.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.197 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:40.197 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.886954+0000", 2024-09-21T10:20:40.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054981810000000004 2024-09-21T10:20:40.197 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.197 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.197 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.197 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.895400+0000", 2024-09-21T10:20:40.197 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0084454530000000003 2024-09-21T10:20:40.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.198 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.198 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:40.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.895449+0000", 2024-09-21T10:20:40.198 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8937999999999998e-05 2024-09-21T10:20:40.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.198 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.198 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:40.198 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.895556+0000", 2024-09-21T10:20:40.198 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000107336 2024-09-21T10:20:40.198 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.198 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.198 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.198 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.199 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.199 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135861 12.11 12:895efe19:::benchmark_data_smithi107_250110_object24451:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:20:40.199 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.255163+0000", 2024-09-21T10:20:40.199 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 267.79703065000001, 2024-09-21T10:20:40.199 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6406458740000001, 2024-09-21T10:20:40.199 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:40.199 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:40.199 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:40.199 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:40.199 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:40.199 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:40.199 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135861 2024-09-21T10:20:40.199 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.199 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:40.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:40.200 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.255163+0000", 2024-09-21T10:20:40.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:40.200 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.255163+0000", 2024-09-21T10:20:40.200 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.200 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.200 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.200 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:40.201 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.255164+0000", 2024-09-21T10:20:40.201 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8199999999999999e-06 2024-09-21T10:20:40.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.201 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:40.201 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.255171+0000", 2024-09-21T10:20:40.201 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9099999999999999e-06 2024-09-21T10:20:40.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.201 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.201 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:40.201 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.255188+0000", 2024-09-21T10:20:40.201 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6798000000000001e-05 2024-09-21T10:20:40.201 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.202 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:40.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.255197+0000", 2024-09-21T10:20:40.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4020000000000002e-06 2024-09-21T10:20:40.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.202 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.255267+0000", 2024-09-21T10:20:40.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0282000000000002e-05 2024-09-21T10:20:40.202 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.202 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.202 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.202 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.255278+0000", 2024-09-21T10:20:40.202 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0736e-05 2024-09-21T10:20:40.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.203 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.203 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581433+0000", 2024-09-21T10:20:40.203 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.3261558 2024-09-21T10:20:40.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.203 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.203 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581443+0000", 2024-09-21T10:20:40.203 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5300000000000002e-06 2024-09-21T10:20:40.203 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.203 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.203 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.203 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.881496+0000", 2024-09-21T10:20:40.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3000530180000001 2024-09-21T10:20:40.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:40.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.881665+0000", 2024-09-21T10:20:40.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016944099999999999 2024-09-21T10:20:40.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:20:40.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.881969+0000", 2024-09-21T10:20:40.204 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030320500000000002 2024-09-21T10:20:40.204 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.204 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.204 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.204 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.886854+0000", 2024-09-21T10:20:40.205 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004885375 2024-09-21T10:20:40.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.205 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.205 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.886941+0000", 2024-09-21T10:20:40.205 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6817000000000006e-05 2024-09-21T10:20:40.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.205 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:40.205 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.886970+0000", 2024-09-21T10:20:40.205 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8867e-05 2024-09-21T10:20:40.205 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.205 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.205 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.205 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.887019+0000", 2024-09-21T10:20:40.206 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9780000000000001e-05 2024-09-21T10:20:40.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.206 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.206 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.895654+0000", 2024-09-21T10:20:40.206 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0086348669999999992 2024-09-21T10:20:40.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.206 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:40.206 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.895705+0000", 2024-09-21T10:20:40.206 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0757000000000002e-05 2024-09-21T10:20:40.206 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.206 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.206 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:40.207 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.895808+0000", 2024-09-21T10:20:40.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010346899999999999 2024-09-21T10:20:40.207 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.207 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.207 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.207 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.207 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.207 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135947 12.11 12:8e645876:::benchmark_data_smithi107_250110_object24537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:20:40.207 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.353234+0000", 2024-09-21T10:20:40.207 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 267.69895902399998, 2024-09-21T10:20:40.207 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.543056319, 2024-09-21T10:20:40.207 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:40.207 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:40.207 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:40.208 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:40.208 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:40.208 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:40.208 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135947 2024-09-21T10:20:40.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.208 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:40.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.208 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:40.208 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.353234+0000", 2024-09-21T10:20:40.208 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.208 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.208 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.208 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:40.208 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.353234+0000", 2024-09-21T10:20:40.208 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.209 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:40.209 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.353236+0000", 2024-09-21T10:20:40.209 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7099999999999999e-06 2024-09-21T10:20:40.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.209 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:40.209 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.353241+0000", 2024-09-21T10:20:40.209 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8049999999999997e-06 2024-09-21T10:20:40.209 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.209 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.209 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:40.210 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.353258+0000", 2024-09-21T10:20:40.210 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7113e-05 2024-09-21T10:20:40.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.210 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:40.210 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.353266+0000", 2024-09-21T10:20:40.210 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4300000000000006e-06 2024-09-21T10:20:40.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.210 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.210 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.353325+0000", 2024-09-21T10:20:40.210 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9212999999999997e-05 2024-09-21T10:20:40.210 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.210 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.211 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.353332+0000", 2024-09-21T10:20:40.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5799999999999997e-06 2024-09-21T10:20:40.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.211 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581471+0000", 2024-09-21T10:20:40.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.22813892099999999 2024-09-21T10:20:40.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.211 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.211 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.211 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581480+0000", 2024-09-21T10:20:40.211 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1789999999999997e-06 2024-09-21T10:20:40.211 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.883441+0000", 2024-09-21T10:20:40.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.301960647 2024-09-21T10:20:40.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:40.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.883578+0000", 2024-09-21T10:20:40.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013736 2024-09-21T10:20:40.212 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.212 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.212 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:20:40.212 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.883807+0000", 2024-09-21T10:20:40.212 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000228414 2024-09-21T10:20:40.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.213 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.213 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:40.213 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.886974+0000", 2024-09-21T10:20:40.213 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031678969999999998 2024-09-21T10:20:40.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.213 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.213 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.213 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.887061+0000", 2024-09-21T10:20:40.213 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6939000000000002e-05 2024-09-21T10:20:40.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.213 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.213 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.213 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.887102+0000", 2024-09-21T10:20:40.213 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0284999999999998e-05 2024-09-21T10:20:40.213 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.887190+0000", 2024-09-21T10:20:40.214 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8722999999999997e-05 2024-09-21T10:20:40.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.895851+0000", 2024-09-21T10:20:40.214 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0086606219999999998 2024-09-21T10:20:40.214 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.214 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.214 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:40.214 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.896174+0000", 2024-09-21T10:20:40.214 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00032276800000000002 2024-09-21T10:20:40.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.215 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:40.215 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.896291+0000", 2024-09-21T10:20:40.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011671300000000001 2024-09-21T10:20:40.215 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.215 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.215 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.215 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.215 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.215 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135988 12.11 12:8e060443:::benchmark_data_smithi107_250110_object24578:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-21T10:20:40.215 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:12.397279+0000", 2024-09-21T10:20:40.215 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 267.65491448900002, 2024-09-21T10:20:40.215 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4995267540000001, 2024-09-21T10:20:40.216 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:20:40.216 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:20:40.216 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:20:40.216 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:20:40.216 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:20:40.216 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:20:40.216 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135988 2024-09-21T10:20:40.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.216 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:20:40.216 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.216 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:20:40.216 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.397279+0000", 2024-09-21T10:20:40.216 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.216 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.217 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.217 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:20:40.217 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.397279+0000", 2024-09-21T10:20:40.217 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:20:40.217 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.217 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.217 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:20:40.217 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.397280+0000", 2024-09-21T10:20:40.217 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.378e-06 2024-09-21T10:20:40.217 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.217 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.217 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:20:40.217 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.397316+0000", 2024-09-21T10:20:40.217 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5926000000000001e-05 2024-09-21T10:20:40.217 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.217 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.218 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:20:40.218 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.397330+0000", 2024-09-21T10:20:40.218 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.413e-05 2024-09-21T10:20:40.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.218 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.218 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:20:40.218 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.397338+0000", 2024-09-21T10:20:40.218 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9349999999999994e-06 2024-09-21T10:20:40.218 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.218 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.218 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.218 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.397406+0000", 2024-09-21T10:20:40.218 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7875999999999999e-05 2024-09-21T10:20:40.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.219 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.397416+0000", 2024-09-21T10:20:40.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0361e-05 2024-09-21T10:20:40.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.219 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581531+0000", 2024-09-21T10:20:40.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.184114578 2024-09-21T10:20:40.219 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.219 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.219 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:20:40.219 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:12.581542+0000", 2024-09-21T10:20:40.219 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1333e-05 2024-09-21T10:20:40.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:20:40.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.885172+0000", 2024-09-21T10:20:40.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.303630093 2024-09-21T10:20:40.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:20:40.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.885311+0000", 2024-09-21T10:20:40.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013817500000000001 2024-09-21T10:20:40.220 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.220 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.220 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5,6,7", 2024-09-21T10:20:40.220 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.885506+0000", 2024-09-21T10:20:40.220 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019548600000000001 2024-09-21T10:20:40.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.887151+0000", 2024-09-21T10:20:40.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001644572 2024-09-21T10:20:40.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.887231+0000", 2024-09-21T10:20:40.221 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0036000000000004e-05 2024-09-21T10:20:40.221 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.221 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.221 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.221 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.887272+0000", 2024-09-21T10:20:40.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.155e-05 2024-09-21T10:20:40.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.222 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:20:40.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.891257+0000", 2024-09-21T10:20:40.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0039846200000000003 2024-09-21T10:20:40.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.222 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:40.222 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.896475+0000", 2024-09-21T10:20:40.222 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052183869999999997 2024-09-21T10:20:40.222 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.222 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.222 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:20:40.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.896698+0000", 2024-09-21T10:20:40.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00022309700000000001 2024-09-21T10:20:40.223 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:20:40.223 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:20:40.223 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:20:40.223 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:13.896805+0000", 2024-09-21T10:20:40.223 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000107221 2024-09-21T10:20:40.223 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.223 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.223 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.223 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:20:40.223 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:20:40.223 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:20:40.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:40.225 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:40.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:40.326 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:40.370 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:40.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:40.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:40.472 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:40.513 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:40.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:40.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:40.614 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:20:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:20:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:20:40.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:40.649 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:40.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:40.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:40.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:40.783 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:40.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:40.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:40.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:40.911 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:40.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:40.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:40 smithi129 ceph-mon[88471]: pgmap v1340: 109 pgs: 1 peering, 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 5.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 913 op/s; 18548/268824 objects misplaced (6.900%); 19 MiB/s, 311 objects/s recovering 2024-09-21T10:20:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:40 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:20:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:40 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:20:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:40 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:41.013 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:41.042 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:41.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:41.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:40 smithi107 ceph-mon[102221]: pgmap v1340: 109 pgs: 1 peering, 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 5.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 913 op/s; 18548/268824 objects misplaced (6.900%); 19 MiB/s, 311 objects/s recovering 2024-09-21T10:20:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:40 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:20:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:40 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:20:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:40 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:41.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:40 smithi107 ceph-mon[105022]: pgmap v1340: 109 pgs: 1 peering, 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 5.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 913 op/s; 18548/268824 objects misplaced (6.900%); 19 MiB/s, 311 objects/s recovering 2024-09-21T10:20:41.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:40 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:20:41.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:40 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:20:41.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:40 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:20:41.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:41.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:41.175 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:41.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:41.276 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:41.316 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:41.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:41.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:41.418 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:41.450 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:20:41.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:41.458 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:41.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:41.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:41.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:41.588 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:41.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:41.690 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:41.757 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:41.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:41.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:41.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:41.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:41.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:41.882 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:41.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:41.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:41.975 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:20:41.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:42.011 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:42.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:42.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:42.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:20:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:20:41] "GET /metrics HTTP/1.1" 200 36807 "" "Prometheus/2.51.0" 2024-09-21T10:20:42.112 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:42.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:42.154 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:42.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:42.255 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:42.302 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:42.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:42.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:42.404 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:42.438 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:42.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:42.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:42.540 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:42.574 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:42.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:42.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:42.676 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:42.711 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:42.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:42.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:42.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:42.843 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:42.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:42.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:42 smithi129 ceph-mon[88471]: pgmap v1341: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 662 op/s; 17137/265062 objects misplaced (6.465%); 23 MiB/s, 370 objects/s recovering 2024-09-21T10:20:42.944 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:42.987 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:42.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:42.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:43.089 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:43.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:42 smithi107 ceph-mon[102221]: pgmap v1341: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 662 op/s; 17137/265062 objects misplaced (6.465%); 23 MiB/s, 370 objects/s recovering 2024-09-21T10:20:43.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:42 smithi107 ceph-mon[105022]: pgmap v1341: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 662 op/s; 17137/265062 objects misplaced (6.465%); 23 MiB/s, 370 objects/s recovering 2024-09-21T10:20:43.119 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:43.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:43.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:43.221 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:43.245 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:43.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:43.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:43.346 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:43.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:43.391 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:43.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:43.492 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:43.547 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:43.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:43.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:43.650 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:43.698 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:43.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:43.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:43.800 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:43.846 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:43.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:43.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:43.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:43.981 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:43.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:43.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:44.083 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:44.108 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:44.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:44.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:44.143 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:20:44.143 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:20:44.143 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.143 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:172698 13.d 13:bd8018c2:::benchmark_data_smithi107_278655_object44791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-21T10:20:44.143 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:44.129196+0000", 2024-09-21T10:20:44.143 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.010428715999999999, 2024-09-21T10:20:44.143 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.010460845, 2024-09-21T10:20:44.143 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:20:44.143 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:20:44.143 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:20:44.143 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:20:44.143 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:20:44.143 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 172698 2024-09-21T10:20:44.144 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.144 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:20:44.144 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.144 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:20:44.144 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.129196+0000", 2024-09-21T10:20:44.144 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:20:44.144 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.144 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.144 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:20:44.144 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.129196+0000", 2024-09-21T10:20:44.144 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:20:44.144 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.144 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.144 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:20:44.144 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.129194+0000", 2024-09-21T10:20:44.144 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:20:44.145 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.145 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.145 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:20:44.145 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.129202+0000", 2024-09-21T10:20:44.145 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.8369999999999997e-06 2024-09-21T10:20:44.145 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.145 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.145 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:20:44.145 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.129221+0000", 2024-09-21T10:20:44.145 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.9286000000000001e-05 2024-09-21T10:20:44.145 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.145 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.145 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:20:44.145 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.129228+0000", 2024-09-21T10:20:44.145 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5629999999999997e-06 2024-09-21T10:20:44.145 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.146 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.146 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:20:44.146 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.129273+0000", 2024-09-21T10:20:44.146 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4960000000000003e-05 2024-09-21T10:20:44.146 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.146 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.146 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:20:44.146 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.129381+0000", 2024-09-21T10:20:44.146 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000108799 2024-09-21T10:20:44.146 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.146 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.146 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,7", 2024-09-21T10:20:44.146 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.129496+0000", 2024-09-21T10:20:44.146 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000114256 2024-09-21T10:20:44.146 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.147 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.147 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:44.147 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.139301+0000", 2024-09-21T10:20:44.147 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0098050889999999995 2024-09-21T10:20:44.147 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:20:44.147 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:20:44.147 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:20:44.147 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.147 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.147 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:172703 13.6 13:6fc20900:::benchmark_data_smithi107_278655_object44796:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-21T10:20:44.147 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:44.131567+0000", 2024-09-21T10:20:44.147 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.0080574219999999998, 2024-09-21T10:20:44.147 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.008170676, 2024-09-21T10:20:44.147 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:20:44.148 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:20:44.148 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:20:44.148 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:20:44.148 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:20:44.148 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 172703 2024-09-21T10:20:44.148 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.148 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:20:44.148 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.148 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:20:44.148 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.131567+0000", 2024-09-21T10:20:44.148 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:20:44.148 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.148 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.148 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:20:44.148 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.131567+0000", 2024-09-21T10:20:44.148 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:20:44.149 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.149 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.149 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:20:44.149 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.131566+0000", 2024-09-21T10:20:44.149 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:20:44.149 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.149 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.149 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:20:44.149 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.131572+0000", 2024-09-21T10:20:44.149 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9490000000000001e-06 2024-09-21T10:20:44.149 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.149 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.149 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:20:44.150 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.131586+0000", 2024-09-21T10:20:44.150 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4063000000000001e-05 2024-09-21T10:20:44.150 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.150 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.150 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:20:44.150 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.131591+0000", 2024-09-21T10:20:44.150 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6559999999999997e-06 2024-09-21T10:20:44.150 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.150 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.150 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:20:44.150 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.132218+0000", 2024-09-21T10:20:44.150 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00062636699999999996 2024-09-21T10:20:44.151 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.151 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.151 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:20:44.151 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.132325+0000", 2024-09-21T10:20:44.151 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000106791 2024-09-21T10:20:44.151 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.151 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.151 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:20:44.151 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.132429+0000", 2024-09-21T10:20:44.151 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010456299999999999 2024-09-21T10:20:44.151 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.151 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.151 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:20:44.151 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.136324+0000", 2024-09-21T10:20:44.151 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0038945870000000001 2024-09-21T10:20:44.152 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.152 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.152 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:20:44.152 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.136494+0000", 2024-09-21T10:20:44.152 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000170643 2024-09-21T10:20:44.152 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:20:44.152 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:20:44.152 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:20:44.152 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.152 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.152 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.37050.0:172699 13.7 e386/381 13:e1d02597:::benchmark_data_smithi107_278655_object44792:head v 386'10805, mlcod=386'10802)", 2024-09-21T10:20:44.152 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:44.138150+0000", 2024-09-21T10:20:44.152 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.0014744999999999999, 2024-09-21T10:20:44.152 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001662962, 2024-09-21T10:20:44.152 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:20:44.152 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "started", 2024-09-21T10:20:44.153 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:20:44.153 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.153 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:20:44.153 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.138150+0000", 2024-09-21T10:20:44.153 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:20:44.153 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.153 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.153 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:20:44.153 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.138150+0000", 2024-09-21T10:20:44.153 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:20:44.153 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.153 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.153 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:20:44.153 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.138149+0000", 2024-09-21T10:20:44.153 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:20:44.153 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.154 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.154 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:20:44.154 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.138158+0000", 2024-09-21T10:20:44.154 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.9949999999999994e-06 2024-09-21T10:20:44.154 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.154 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.154 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:20:44.154 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.138175+0000", 2024-09-21T10:20:44.154 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.7739000000000001e-05 2024-09-21T10:20:44.154 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.154 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.154 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:20:44.154 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.138182+0000", 2024-09-21T10:20:44.154 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5950000000000004e-06 2024-09-21T10:20:44.154 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.155 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.155 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:20:44.155 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.138262+0000", 2024-09-21T10:20:44.155 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.0296000000000005e-05 2024-09-21T10:20:44.155 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.155 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.155 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:20:44.155 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.138287+0000", 2024-09-21T10:20:44.155 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.4281999999999999e-05 2024-09-21T10:20:44.155 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:20:44.155 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:20:44.155 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:20:44.155 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.155 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.155 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop_reply(client.37050.0:172698 13.d e386/367)", 2024-09-21T10:20:44.156 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:44.138194+0000", 2024-09-21T10:20:44.156 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.001429937, 2024-09-21T10:20:44.156 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0016680709999999999, 2024-09-21T10:20:44.156 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:20:44.156 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "queued for pg", 2024-09-21T10:20:44.156 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:20:44.156 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.156 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:20:44.156 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.138194+0000", 2024-09-21T10:20:44.156 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:20:44.156 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.156 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.156 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:20:44.156 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.138194+0000", 2024-09-21T10:20:44.156 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:20:44.156 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.157 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.157 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:20:44.157 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.138194+0000", 2024-09-21T10:20:44.157 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:20:44.157 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.157 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.157 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:20:44.157 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.138197+0000", 2024-09-21T10:20:44.157 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.6129999999999998e-06 2024-09-21T10:20:44.157 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.157 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.157 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:20:44.157 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.138202+0000", 2024-09-21T10:20:44.157 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4720000000000003e-06 2024-09-21T10:20:44.157 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:20:44.157 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:20:44.158 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:20:44.158 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:44.138206+0000", 2024-09-21T10:20:44.158 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.354e-06 2024-09-21T10:20:44.158 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:20:44.158 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:20:44.158 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:20:44.158 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:20:44.158 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-21T10:20:44.158 INFO:teuthology.orchestra.run.smithi129.stdout: "num_ops": 4 2024-09-21T10:20:44.158 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:20:44.209 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:44.244 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:44.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:44.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:44.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:44.372 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:44.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:44.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:44.474 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:44.515 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:44.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:44.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:44.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:44.649 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:44.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:44.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:44.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:44.785 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:44.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:44.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:44.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:44.890 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:44 smithi129 ceph-mon[88471]: pgmap v1342: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 5.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 939 op/s; 16345/255957 objects misplaced (6.386%); 24 MiB/s, 386 objects/s recovering 2024-09-21T10:20:44.916 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:44.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:44.936 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:20:45.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:45.046 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:45.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:45.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:45.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:44 smithi107 ceph-mon[102221]: pgmap v1342: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 5.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 939 op/s; 16345/255957 objects misplaced (6.386%); 24 MiB/s, 386 objects/s recovering 2024-09-21T10:20:45.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:44 smithi107 ceph-mon[105022]: pgmap v1342: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 5.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 939 op/s; 16345/255957 objects misplaced (6.386%); 24 MiB/s, 386 objects/s recovering 2024-09-21T10:20:45.148 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:45.200 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:45.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:45.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:45.302 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:45.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:45.356 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:45.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:45.406 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:20:45.457 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:45.482 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:45.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:45.583 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:45.619 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:45.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:45.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:45.653 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:45.721 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:45.760 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:45.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:45.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:45.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:45.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:45.889 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:45.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:45.989 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:46.025 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:46.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:46.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:46.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:46.151 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:46.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:46.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:46.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:46.278 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:46.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:46.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:46.379 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:46.402 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:46.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:46.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:46.504 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:46.533 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:46.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:46.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:46.635 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:46.680 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:46.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:46.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:46.782 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:46.817 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:46.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:46.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:46.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:46 smithi107 ceph-mon[102221]: pgmap v1343: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 5.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 846 op/s; 15306/252324 objects misplaced (6.066%); 18 MiB/s, 295 objects/s recovering 2024-09-21T10:20:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:46 smithi107 ceph-mon[105022]: pgmap v1343: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 5.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 846 op/s; 15306/252324 objects misplaced (6.066%); 18 MiB/s, 295 objects/s recovering 2024-09-21T10:20:46.918 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:46.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:46 smithi129 ceph-mon[88471]: pgmap v1343: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 5.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 846 op/s; 15306/252324 objects misplaced (6.066%); 18 MiB/s, 295 objects/s recovering 2024-09-21T10:20:46.940 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:46.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:46.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:47.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:47.063 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:47.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:47.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:47.164 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:47.186 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:47.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:47.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:47.287 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:47.325 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:47.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:47.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:47.427 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:47.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:47.471 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:47.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:47.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:47.601 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:47.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:47.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:47.641 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:20:47.656 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:47 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3976495967' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:20:47.702 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:47.740 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:47.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:47.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:47.841 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:47.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:47 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3976495967' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:20:47.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:47 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3976495967' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:20:47.878 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:47.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:47.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:47.980 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:48.021 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:48.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:48.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:48.106 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:20:48.106 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:20:48.106 INFO:teuthology.orchestra.run.smithi129.stdout: "complaint_time": 30, 2024-09-21T10:20:48.107 INFO:teuthology.orchestra.run.smithi129.stdout: "num_blocked_ops": 0 2024-09-21T10:20:48.107 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:20:48.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:48.150 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:48.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:48.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:48.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:48.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:48.280 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:48.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:48.381 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:48.415 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:48.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:48.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:48.517 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:48.553 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:48.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:48.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:48.637 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-09-21T10:20:48.637 INFO:teuthology.orchestra.run.smithi107.stderr:0 2024-09-21T10:20:48.638 INFO:teuthology.orchestra.run.smithi107.stderr:1 2024-09-21T10:20:48.638 INFO:teuthology.orchestra.run.smithi107.stderr:2 2024-09-21T10:20:48.638 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2024-09-21T10:20:48.638 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2024-09-21T10:20:48.638 INFO:teuthology.orchestra.run.smithi107.stderr:bench [] [] [] [] 2024-09-21T10:20:48.638 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:20:48.638 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2024-09-21T10:20:48.638 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2024-09-21T10:20:48.638 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2024-09-21T10:20:48.638 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-09-21T10:20:48.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:48.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:48.695 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:48.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:48.730 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:48 smithi107 ceph-mon[102221]: pgmap v1344: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 14389/247029 objects misplaced (5.825%); 19 MiB/s, 300 objects/s recovering 2024-09-21T10:20:48.730 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:48 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:20:48.731 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:48 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3976495967' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:20:48.731 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:48 smithi107 ceph-mon[102221]: osdmap e387: 8 total, 8 up, 7 in 2024-09-21T10:20:48.731 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:48 smithi107 ceph-mon[105022]: pgmap v1344: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 14389/247029 objects misplaced (5.825%); 19 MiB/s, 300 objects/s recovering 2024-09-21T10:20:48.731 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:48 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:20:48.731 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:48 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3976495967' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:20:48.731 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:48 smithi107 ceph-mon[105022]: osdmap e387: 8 total, 8 up, 7 in 2024-09-21T10:20:48.795 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:48 smithi129 ceph-mon[88471]: pgmap v1344: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 14389/247029 objects misplaced (5.825%); 19 MiB/s, 300 objects/s recovering 2024-09-21T10:20:48.795 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:48 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:20:48.795 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:48 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3976495967' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:20:48.795 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:48 smithi129 ceph-mon[88471]: osdmap e387: 8 total, 8 up, 7 in 2024-09-21T10:20:48.797 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:48.824 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:48.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:48.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:48.839 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:20:48.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:48.951 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:48.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:48.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:49.052 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:49.096 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:49.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:49.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:49.198 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:49.263 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:49.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:49.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:49.324 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:20:49.365 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:49.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:49.400 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:49.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:49.501 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:49.556 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:49.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:49.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:49.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:49.690 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:49.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:49.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:49.791 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:49.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:49.816 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:49.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:49.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:49.966 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:49.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:49.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:50.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:50.097 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:50.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:50.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:50.199 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:50.233 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:50.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:50.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:50.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:50.369 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:50.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:50.470 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:50.492 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:20:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:20:50.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:20:50.518 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:50.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:50.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:50.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:50.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:50.647 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:50.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:50.684 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:20:50.684 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:20:50.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-21T10:20:50.748 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:50.789 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:50.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:50.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:50.891 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:50.914 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:50 smithi129 ceph-mon[88471]: pgmap v1346: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 4.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 854 op/s; 14389/243177 objects misplaced (5.917%); 16 MiB/s, 258 objects/s recovering 2024-09-21T10:20:50.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:50.925 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:50.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:50.951 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:20:50.951 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:50 smithi107 ceph-mon[102221]: pgmap v1346: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 4.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 854 op/s; 14389/243177 objects misplaced (5.917%); 16 MiB/s, 258 objects/s recovering 2024-09-21T10:20:50.952 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:50 smithi107 ceph-mon[105022]: pgmap v1346: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 4.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 854 op/s; 14389/243177 objects misplaced (5.917%); 16 MiB/s, 258 objects/s recovering 2024-09-21T10:20:50.952 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:20:50.979 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:20:50.980 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:20:51.017 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:20:51.018 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:20:51.026 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:51.048 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:20:51.048 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:20:51.061 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:51.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:51.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:51.081 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:20:51.081 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:20:51.113 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:20:51.113 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:20:51.145 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:20:51.145 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:20:51.148 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:51.163 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:51.181 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:20:51.182 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:20:51.199 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:51.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:51.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:51.331 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:51.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:51.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:51.433 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:51.472 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:51.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:51.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:51.574 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:51.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:51.618 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:51.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:51.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:51.754 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:51.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:51.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:51.853 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:51 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/69936488' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:51.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:51.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:51 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/69936488' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:51.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:51 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/69936488' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:20:51.860 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:20:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:20:51] "GET /metrics HTTP/1.1" 200 36807 "" "Prometheus/2.51.0" 2024-09-21T10:20:51.895 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:51.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:51.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:51.922 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:20:51.922 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:20:51.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:20:51.997 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:52.041 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:52.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:52.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:52.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:52.179 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:52.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:52.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:52.313 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:52.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:52.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:52.414 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:52.452 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:52.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:52.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:52.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:52.583 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:52.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:52.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:52.671 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:20:52.686 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:52.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:52.780 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:52.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:52.881 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:52.911 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:52.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:52.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:52 smithi129 ceph-mon[88471]: pgmap v1347: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 4.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 832 op/s; 13375/240093 objects misplaced (5.571%); 14 MiB/s, 220 objects/s recovering 2024-09-21T10:20:53.012 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:53.087 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:53.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:53.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:53.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:52 smithi107 ceph-mon[102221]: pgmap v1347: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 4.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 832 op/s; 13375/240093 objects misplaced (5.571%); 14 MiB/s, 220 objects/s recovering 2024-09-21T10:20:53.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:52 smithi107 ceph-mon[105022]: pgmap v1347: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 4.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 832 op/s; 13375/240093 objects misplaced (5.571%); 14 MiB/s, 220 objects/s recovering 2024-09-21T10:20:53.146 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:20:53.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:53.214 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:53.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:53.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:53.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:53.359 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:53.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:53.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:53.461 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:53.494 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:53.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:53.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:53.596 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:53.631 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:53.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:53.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:53.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:53.754 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:53.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:53.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:53.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:53.887 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:53.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:53.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:53.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:20:53.989 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:54.029 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:54.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:54.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:54.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:54.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:54.166 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:54.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:54.267 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:54.295 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:54.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:54.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:54.397 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:54.424 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:54.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:54.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:54.440 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:54.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:54.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:54.548 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:54.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:54.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:54.673 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:54.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:54.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:54.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:54.808 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:54.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:54.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:54.910 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:54.948 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:54.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:54.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:55.006 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:54 smithi129 ceph-mon[88471]: pgmap v1348: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 4.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 835 op/s; 12661/230874 objects misplaced (5.484%); 14 MiB/s, 216 objects/s recovering 2024-09-21T10:20:55.006 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:54 smithi129 ceph-mon[88471]: osdmap e388: 8 total, 8 up, 7 in 2024-09-21T10:20:55.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:54 smithi107 ceph-mon[102221]: pgmap v1348: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 4.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 835 op/s; 12661/230874 objects misplaced (5.484%); 14 MiB/s, 216 objects/s recovering 2024-09-21T10:20:55.041 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:54 smithi107 ceph-mon[102221]: osdmap e388: 8 total, 8 up, 7 in 2024-09-21T10:20:55.041 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:54 smithi107 ceph-mon[105022]: pgmap v1348: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 4.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 835 op/s; 12661/230874 objects misplaced (5.484%); 14 MiB/s, 216 objects/s recovering 2024-09-21T10:20:55.041 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:54 smithi107 ceph-mon[105022]: osdmap e388: 8 total, 8 up, 7 in 2024-09-21T10:20:55.048 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:20:55.048 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:20:55.049 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:55.091 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:55.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:55.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:55.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:55.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:55.237 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:55.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:55.338 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:55.373 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:55.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:55.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:55.475 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:55.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:55.511 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:55.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:55.612 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:55.650 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:55.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:55.752 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:55.774 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:55.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:55.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:55.797 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-09-21T10:20:55.797 INFO:teuthology.orchestra.run.smithi107.stderr:0 2024-09-21T10:20:55.797 INFO:teuthology.orchestra.run.smithi107.stderr:1 2024-09-21T10:20:55.798 INFO:teuthology.orchestra.run.smithi107.stderr:2 2024-09-21T10:20:55.798 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2024-09-21T10:20:55.798 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2024-09-21T10:20:55.798 INFO:teuthology.orchestra.run.smithi107.stderr:bench [] [] [] [] 2024-09-21T10:20:55.798 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:20:55.798 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2024-09-21T10:20:55.798 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2024-09-21T10:20:55.798 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2024-09-21T10:20:55.798 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-09-21T10:20:55.877 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:55.880 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:55 smithi107 ceph-mon[105022]: osdmap e389: 8 total, 8 up, 7 in 2024-09-21T10:20:55.880 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:55 smithi107 ceph-mon[102221]: osdmap e389: 8 total, 8 up, 7 in 2024-09-21T10:20:55.919 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:55.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:55.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:56.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:56.052 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:56.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:56.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:56.154 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:56.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:55 smithi129 ceph-mon[88471]: osdmap e389: 8 total, 8 up, 7 in 2024-09-21T10:20:56.188 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:56.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:56.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:56.289 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:56.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:56.325 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:56.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:56.425 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:56.452 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:56.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:56.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:56.553 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:56.586 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:56.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:56.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:56.649 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:20:56.649 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:20:56.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-21T10:20:56.688 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:56.716 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:56.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:56.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:56.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:56.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:56.858 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:56.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:56 smithi107 ceph-mon[102221]: pgmap v1351: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 4.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 785 op/s; 11382/227004 objects misplaced (5.014%); 15 MiB/s, 240 objects/s recovering 2024-09-21T10:20:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:56.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:56 smithi107 ceph-mon[105022]: pgmap v1351: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 4.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 785 op/s; 11382/227004 objects misplaced (5.014%); 15 MiB/s, 240 objects/s recovering 2024-09-21T10:20:56.861 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:56.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:56.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:56.984 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:56.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:57.085 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:57.116 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:57.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:57.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:56 smithi129 ceph-mon[88471]: pgmap v1351: 109 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfilling, 105 active+clean; 4.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 785 op/s; 11382/227004 objects misplaced (5.014%); 15 MiB/s, 240 objects/s recovering 2024-09-21T10:20:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:57.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:20:57.254 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:57.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:57.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:20:57.356 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:57.388 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:57.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:57.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:57.407 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:57.491 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:57.525 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:57.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:57.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:57.627 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:57.661 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:57.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:57.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:57.763 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:57.789 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:57.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:57.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:57.890 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:57.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:57.933 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:57.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:58.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:20:58.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:58.067 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:58.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:20:58.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:58.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:58.213 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:58.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:58.314 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:58.356 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:58.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:58.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:58.458 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:58.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:58.497 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:58.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:58.598 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:20:58.634 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:58.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:58.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:20:58.735 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:20:58.760 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:58 smithi107 ceph-mon[102221]: pgmap v1352: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 4.5 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 888 op/s; 10289/221859 objects misplaced (4.638%); 21 MiB/s, 342 objects/s recovering 2024-09-21T10:20:58.760 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:58 smithi107 ceph-mon[105022]: pgmap v1352: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 4.5 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 888 op/s; 10289/221859 objects misplaced (4.638%); 21 MiB/s, 342 objects/s recovering 2024-09-21T10:20:58.770 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:20:58.774 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:58.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:58.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:20:58.875 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:58.918 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:58.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:58.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:59.020 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:59.056 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:59.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:59.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:59.099 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:20:59.158 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:59.177 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:58 smithi129 ceph-mon[88471]: pgmap v1352: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 4.5 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 888 op/s; 10289/221859 objects misplaced (4.638%); 21 MiB/s, 342 objects/s recovering 2024-09-21T10:20:59.214 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:59.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:59.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:59.215 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:20:59.316 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:59.359 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:59.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:59.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:20:59.460 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:20:59.494 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:59.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:59.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:20:59.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:20:59.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:59.635 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:20:59.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:20:59.699 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:20:59.736 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:20:59.781 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:59.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:59.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:20:59.882 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:20:59.911 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:20:59.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:20:59.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:00.012 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:00.048 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:00.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:00.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:00.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:20:59 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3543381289' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:21:00.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:20:59 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3543381289' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:21:00.149 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:00.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:20:59 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3543381289' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:21:00.195 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:00.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:00.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:00.296 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:00.332 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:00.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:00.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:00.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:00.462 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:00.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:00.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:00.526 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:21:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:21:00.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:21:00.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:00.598 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:00.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:00.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:00.699 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:00.736 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:00.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:00.837 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:00.866 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:00.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:00.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:00.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:00 smithi129 ceph-mon[88471]: pgmap v1353: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 4.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 907 op/s; 10289/218298 objects misplaced (4.713%); 17 MiB/s, 269 objects/s recovering 2024-09-21T10:21:00.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:00 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:21:00.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:00 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:21:00.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:00 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3543381289' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:21:00.895 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:00 smithi129 ceph-mon[88471]: osdmap e390: 8 total, 8 up, 7 in 2024-09-21T10:21:00.968 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:01.003 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:01.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:01.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:01.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:01.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:00 smithi107 ceph-mon[102221]: pgmap v1353: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 4.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 907 op/s; 10289/218298 objects misplaced (4.713%); 17 MiB/s, 269 objects/s recovering 2024-09-21T10:21:01.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:00 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:21:01.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:00 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:21:01.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:00 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3543381289' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:21:01.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:00 smithi107 ceph-mon[102221]: osdmap e390: 8 total, 8 up, 7 in 2024-09-21T10:21:01.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:00 smithi107 ceph-mon[105022]: pgmap v1353: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 4.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 907 op/s; 10289/218298 objects misplaced (4.713%); 17 MiB/s, 269 objects/s recovering 2024-09-21T10:21:01.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:00 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:21:01.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:00 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:21:01.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:00 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3543381289' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:21:01.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:00 smithi107 ceph-mon[105022]: osdmap e390: 8 total, 8 up, 7 in 2024-09-21T10:21:01.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:01.138 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:01.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:01.181 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:21:01.181 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:21:01.239 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:01.276 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:01.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:01.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:01.378 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:01.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:01.421 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:01.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:01.522 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:01.563 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:01.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:01.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:01.665 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:01.703 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:01.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:01.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:01.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:01.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:01.837 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:01.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:01.915 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:21:01.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:01.969 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:01.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:01.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:02.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:02.103 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:02.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:02.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:02.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:21:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:21:01] "GET /metrics HTTP/1.1" 200 36810 "" "Prometheus/2.51.0" 2024-09-21T10:21:02.205 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:02.261 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:02.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:02.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:02.363 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:02.370 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:21:02.391 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:02.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:02.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:02.493 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:02.530 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:02.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:02.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:02.632 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:02.672 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:02.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:02.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:02.774 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:02.805 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:02.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:02.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:02.856 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:02 smithi129 ceph-mon[88471]: pgmap v1355: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 4.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 645 op/s; 9478/214920 objects misplaced (4.410%); 18 MiB/s, 281 objects/s recovering 2024-09-21T10:21:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:02 smithi107 ceph-mon[102221]: pgmap v1355: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 4.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 645 op/s; 9478/214920 objects misplaced (4.410%); 18 MiB/s, 281 objects/s recovering 2024-09-21T10:21:02.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:02 smithi107 ceph-mon[105022]: pgmap v1355: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 4.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 645 op/s; 9478/214920 objects misplaced (4.410%); 18 MiB/s, 281 objects/s recovering 2024-09-21T10:21:02.907 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:02.954 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:02.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:02.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:03.055 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:03.096 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:03.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:03.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:03.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:03.229 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:03.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:03.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:03.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:03.372 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:03.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:03.474 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:03.510 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:03.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:03.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:03.612 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:03.649 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:03.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:03.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:03.751 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:03.793 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:03.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:03.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:03.895 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:03.939 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:03.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:03.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:04.041 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:04.078 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:04.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:04.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:04.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:04.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:04.216 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:04.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:04.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:04.343 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:04.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:04.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:04.386 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:21:04.386 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:21:04.446 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:04.479 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:04.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:04.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:04.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:04.610 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:04.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:04.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:04.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:04.738 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:04.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:04.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:04.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:04.874 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:04.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:04.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:04.943 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:04 smithi107 ceph-mon[102221]: pgmap v1356: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 4.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 927 op/s; 8670/205542 objects misplaced (4.218%); 20 MiB/s, 316 objects/s recovering 2024-09-21T10:21:04.944 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:04 smithi107 ceph-mon[105022]: pgmap v1356: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 4.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 927 op/s; 8670/205542 objects misplaced (4.218%); 20 MiB/s, 316 objects/s recovering 2024-09-21T10:21:04.963 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:21:04.963 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:21:04.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:04.993 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:21:04.994 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:21:05.011 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:05.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:05.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:05.027 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:21:05.027 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:21:05.053 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:21:05.053 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:21:05.053 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:04 smithi129 ceph-mon[88471]: pgmap v1356: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 4.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 927 op/s; 8670/205542 objects misplaced (4.218%); 20 MiB/s, 316 objects/s recovering 2024-09-21T10:21:05.074 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:21:05.093 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:21:05.094 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:21:05.094 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:21:05.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:05.126 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:21:05.126 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:21:05.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:05.145 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:05.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:05.160 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:21:05.160 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:21:05.198 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:21:05.198 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:21:05.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:05.282 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:05.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:05.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:05.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:05.414 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:05.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:05.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:05.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:05.546 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:05.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:05.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:05.550 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:21:05.647 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:05.704 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:05.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:05.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:05.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:05.834 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:05.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:05.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:05.936 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:05.950 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:05 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3541297420' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:21:05.975 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:05.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:05.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:05.996 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:05 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3541297420' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:21:05.996 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:05 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3541297420' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:21:06.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:06.111 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:06.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:06.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:06.213 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:06.236 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:06.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:06.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:06.337 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:06.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:06.374 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:06.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:06.474 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:06.516 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:06.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:06.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:06.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:06.645 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:06.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:06.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:06.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:06.773 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:06.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:06.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:06.875 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:06.901 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:06 smithi129 ceph-mon[88471]: pgmap v1357: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 4.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 865 op/s; 7792/201021 objects misplaced (3.876%); 15 MiB/s, 242 objects/s recovering 2024-09-21T10:21:06.915 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:06.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:06.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:07.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:07.047 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:07.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:07.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:07.067 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:06 smithi107 ceph-mon[102221]: pgmap v1357: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 4.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 865 op/s; 7792/201021 objects misplaced (3.876%); 15 MiB/s, 242 objects/s recovering 2024-09-21T10:21:07.067 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:06 smithi107 ceph-mon[105022]: pgmap v1357: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 4.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 865 op/s; 7792/201021 objects misplaced (3.876%); 15 MiB/s, 242 objects/s recovering 2024-09-21T10:21:07.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:07.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:07.191 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:07.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:07.251 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-09-21T10:21:07.251 INFO:teuthology.orchestra.run.smithi107.stderr:0 2024-09-21T10:21:07.251 INFO:teuthology.orchestra.run.smithi107.stderr:1 2024-09-21T10:21:07.251 INFO:teuthology.orchestra.run.smithi107.stderr:2 2024-09-21T10:21:07.251 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2024-09-21T10:21:07.251 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2024-09-21T10:21:07.251 INFO:teuthology.orchestra.run.smithi107.stderr:bench [] [] [] [] 2024-09-21T10:21:07.251 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:21:07.251 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2024-09-21T10:21:07.251 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2024-09-21T10:21:07.252 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2024-09-21T10:21:07.252 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-09-21T10:21:07.292 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:07.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:07.338 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:07.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:07.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:07.475 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:07.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:07.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:07.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:07.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:07.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:07.619 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:21:07.619 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:21:07.619 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:07.619 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.37050.0:192885 13.1 e390/381 13:8be67043:::benchmark_data_smithi107_278655_object64978:head v 390'12099, mlcod=390'12097)", 2024-09-21T10:21:07.619 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:21:07.609686+0000", 2024-09-21T10:21:07.620 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 0.0073267999999999996, 2024-09-21T10:21:07.620 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0073596460000000001, 2024-09-21T10:21:07.620 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:07.620 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "started", 2024-09-21T10:21:07.620 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:07.620 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:07.620 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:07.620 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:07.609686+0000", 2024-09-21T10:21:07.620 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:07.620 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:07.620 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:07.620 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:07.620 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:07.609686+0000", 2024-09-21T10:21:07.620 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:07.620 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:07.620 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:07.621 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:07.621 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:07.609685+0000", 2024-09-21T10:21:07.621 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:21:07.621 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:07.621 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:07.621 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:07.621 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:07.609689+0000", 2024-09-21T10:21:07.621 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8500000000000004e-06 2024-09-21T10:21:07.621 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:07.621 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:07.621 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:07.621 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:07.609695+0000", 2024-09-21T10:21:07.621 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1099999999999999e-06 2024-09-21T10:21:07.621 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:07.621 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:07.622 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:07.622 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:07.609699+0000", 2024-09-21T10:21:07.622 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7670000000000001e-06 2024-09-21T10:21:07.622 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:07.622 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:07.622 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:07.622 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:07.613246+0000", 2024-09-21T10:21:07.622 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0035466 2024-09-21T10:21:07.622 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:07.622 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:07.622 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:07.622 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:07.613268+0000", 2024-09-21T10:21:07.622 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1929e-05 2024-09-21T10:21:07.622 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:07.623 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:07.623 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:07.623 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:07.623 INFO:teuthology.orchestra.run.smithi129.stdout: ], 2024-09-21T10:21:07.623 INFO:teuthology.orchestra.run.smithi129.stdout: "num_ops": 1 2024-09-21T10:21:07.623 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:21:07.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:07.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:07.740 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:07.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:07.841 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:07.874 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:07.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:07.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:07.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:08.004 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:08.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:08.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:08.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:08.138 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:08.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:08.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:08.239 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:08.282 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:08.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:08.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:08.363 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:21:08.384 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:08.449 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:08.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:08.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:08.550 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:08.615 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:08.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:08.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:08.716 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:08.754 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:08.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:08.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:08.775 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:08.806 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:21:08.806 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:21:08.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-21T10:21:08.820 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:21:08.856 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:08.905 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:08.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:08.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:08.988 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:08 smithi129 ceph-mon[88471]: pgmap v1358: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 889 op/s; 7081/195162 objects misplaced (3.628%); 13 MiB/s, 211 objects/s recovering 2024-09-21T10:21:09.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:09.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:09.041 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:09.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:09.107 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:08 smithi107 ceph-mon[102221]: pgmap v1358: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 889 op/s; 7081/195162 objects misplaced (3.628%); 13 MiB/s, 211 objects/s recovering 2024-09-21T10:21:09.107 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:08 smithi107 ceph-mon[105022]: pgmap v1358: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 889 op/s; 7081/195162 objects misplaced (3.628%); 13 MiB/s, 211 objects/s recovering 2024-09-21T10:21:09.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:09.188 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:09.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:09.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:09.289 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:09.332 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:09.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:09.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:09.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:09.465 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:09.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:09.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:09.567 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:09.603 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:09.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:09.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:09.705 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:09.738 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:09.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:09.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:09.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:09.871 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:09.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:09.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:09.973 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:10.017 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:10.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:10.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:10.119 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:10.125 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:10.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:21:10.150 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:10.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:10.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:10.251 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:10.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:10.290 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:10.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:10.391 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:10.407 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:21:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:21:10.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:21:10.441 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:10.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:10.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:10.543 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:10.578 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:10.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:10.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:10.679 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:10.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:10.726 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:10.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:10.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:10.837 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:21:10.837 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:21:10.837 INFO:teuthology.orchestra.run.smithi129.stdout: "complaint_time": 30, 2024-09-21T10:21:10.837 INFO:teuthology.orchestra.run.smithi129.stdout: "num_blocked_ops": 0 2024-09-21T10:21:10.837 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:21:10.861 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:10.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:10.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:10.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:10.984 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:10.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:10.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:11.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:11.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:10 smithi107 ceph-mon[102221]: pgmap v1359: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 912 op/s; 7081/190902 objects misplaced (3.709%); 13 MiB/s, 211 objects/s recovering 2024-09-21T10:21:11.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:10 smithi107 ceph-mon[105022]: pgmap v1359: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 912 op/s; 7081/190902 objects misplaced (3.709%); 13 MiB/s, 211 objects/s recovering 2024-09-21T10:21:11.123 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:11.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:11.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:11.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:10 smithi129 ceph-mon[88471]: pgmap v1359: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 912 op/s; 7081/190902 objects misplaced (3.709%); 13 MiB/s, 211 objects/s recovering 2024-09-21T10:21:11.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:11.255 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:11.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:11.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:11.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:11.389 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:11.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:11.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:11.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:11.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:11.523 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:11.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:11.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:11.656 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:11.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:11.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:11.757 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:11.801 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:11.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:11.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:11.821 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:21:11.903 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:11.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:11.963 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:11.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:12.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:12.091 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:12.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:12.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:12.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:21:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:21:11] "GET /metrics HTTP/1.1" 200 36808 "" "Prometheus/2.51.0" 2024-09-21T10:21:12.192 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:12.257 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:12.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:12.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:12.314 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:21:12.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:12.385 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:12.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:12.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:12.456 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:21:12.456 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:12.456 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:12 smithi129 ceph-mon[88471]: pgmap v1360: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 873 op/s; 6197/187431 objects misplaced (3.306%); 14 MiB/s, 227 objects/s recovering 2024-09-21T10:21:12.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:12.509 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:12.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:12.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:12.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:21:12.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:12.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:12 smithi107 ceph-mon[105022]: pgmap v1360: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 873 op/s; 6197/187431 objects misplaced (3.306%); 14 MiB/s, 227 objects/s recovering 2024-09-21T10:21:12.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:21:12.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:12.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:12 smithi107 ceph-mon[102221]: pgmap v1360: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 873 op/s; 6197/187431 objects misplaced (3.306%); 14 MiB/s, 227 objects/s recovering 2024-09-21T10:21:12.611 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:12.641 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:12.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:12.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:12.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:12.765 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:12.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:12.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:12.867 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:12.891 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:12.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:12.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:12.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:13.014 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:13.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:13.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:13.115 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:13.138 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:13.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:13.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:13.188 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:21:13.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:13.266 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:13.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:13.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:13.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:13 smithi107 ceph-mon[102221]: osdmap e391: 8 total, 8 up, 7 in 2024-09-21T10:21:13.331 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:13 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/755296587' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:21:13.331 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:13 smithi107 ceph-mon[105022]: osdmap e391: 8 total, 8 up, 7 in 2024-09-21T10:21:13.331 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:13 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/755296587' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:21:13.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:13.402 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:13.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:13.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:13.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:13.515 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:13 smithi129 ceph-mon[88471]: osdmap e391: 8 total, 8 up, 7 in 2024-09-21T10:21:13.515 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:13 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/755296587' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:21:13.529 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:13.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:13.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:13.631 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:13.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:13.683 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:13.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:13.784 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:13.819 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:13.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:13.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:13.921 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:13.971 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:13.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:13.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:14.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:14.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:14.106 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:14.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:14.207 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:14.247 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:14.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:14.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:14.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:14.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:14.379 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:14.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:14.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:14 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:21:14.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:14 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/755296587' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:21:14.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:14 smithi107 ceph-mon[102221]: osdmap e392: 8 total, 8 up, 7 in 2024-09-21T10:21:14.433 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:14 smithi107 ceph-mon[102221]: pgmap v1363: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 964 op/s; 5571/177861 objects misplaced (3.132%); 13 MiB/s, 200 objects/s recovering 2024-09-21T10:21:14.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:14 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:21:14.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:14 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/755296587' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:21:14.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:14 smithi107 ceph-mon[105022]: osdmap e392: 8 total, 8 up, 7 in 2024-09-21T10:21:14.433 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:14 smithi107 ceph-mon[105022]: pgmap v1363: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 964 op/s; 5571/177861 objects misplaced (3.132%); 13 MiB/s, 200 objects/s recovering 2024-09-21T10:21:14.480 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:14.518 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:14.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:14.541 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:14 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:21:14.541 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:14 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/755296587' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:21:14.541 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:14 smithi129 ceph-mon[88471]: osdmap e392: 8 total, 8 up, 7 in 2024-09-21T10:21:14.541 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:14 smithi129 ceph-mon[88471]: pgmap v1363: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 964 op/s; 5571/177861 objects misplaced (3.132%); 13 MiB/s, 200 objects/s recovering 2024-09-21T10:21:14.581 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:21:14.581 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 20, 2024-09-21T10:21:14.581 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 600, 2024-09-21T10:21:14.581 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:21:14.581 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.581 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36822.0:197531 12.0 12:07cca061:::benchmark_data_smithi107_250110_object86121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-09-21T10:21:14.581 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:17:27.560362+0000", 2024-09-21T10:21:14.581 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 227.01496342999999, 2024-09-21T10:21:14.581 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.095372886000000004, 2024-09-21T10:21:14.581 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:14.581 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:14.582 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:14.582 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36822", 2024-09-21T10:21:14.582 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:21:14.582 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 197531 2024-09-21T10:21:14.582 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.582 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:14.582 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.582 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:14.582 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.560362+0000", 2024-09-21T10:21:14.582 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.582 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.582 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.582 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:14.583 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.560362+0000", 2024-09-21T10:21:14.583 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.583 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.583 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:14.583 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.560361+0000", 2024-09-21T10:21:14.583 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:21:14.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.583 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.583 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:14.583 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.560367+0000", 2024-09-21T10:21:14.583 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1349999999999997e-06 2024-09-21T10:21:14.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.583 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.583 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:14.584 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.560379+0000", 2024-09-21T10:21:14.584 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1240000000000001e-05 2024-09-21T10:21:14.584 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.584 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.584 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:14.584 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.560384+0000", 2024-09-21T10:21:14.584 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1830000000000002e-06 2024-09-21T10:21:14.584 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.584 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.584 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:14.584 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.560428+0000", 2024-09-21T10:21:14.584 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3850999999999997e-05 2024-09-21T10:21:14.584 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.584 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.584 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:14.585 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.560547+0000", 2024-09-21T10:21:14.585 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000119314 2024-09-21T10:21:14.585 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.585 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.585 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:21:14.585 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.560990+0000", 2024-09-21T10:21:14.585 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00044328 2024-09-21T10:21:14.585 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.585 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.585 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:14.585 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.563234+0000", 2024-09-21T10:21:14.585 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00224398 2024-09-21T10:21:14.585 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.585 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.585 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.586 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.563970+0000", 2024-09-21T10:21:14.586 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00073621600000000004 2024-09-21T10:21:14.586 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.586 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.586 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.586 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.655673+0000", 2024-09-21T10:21:14.586 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.091702131000000006 2024-09-21T10:21:14.586 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.586 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.586 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:14.586 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.655708+0000", 2024-09-21T10:21:14.586 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5407999999999997e-05 2024-09-21T10:21:14.586 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.586 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.586 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:14.586 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.655735+0000", 2024-09-21T10:21:14.587 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7282000000000001e-05 2024-09-21T10:21:14.587 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.587 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.587 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.587 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.587 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.587 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36822.0:197538 12.17 12:e9e12710:::benchmark_data_smithi107_250110_object86128:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-09-21T10:21:14.587 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:17:27.563584+0000", 2024-09-21T10:21:14.587 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 227.01174184199999, 2024-09-21T10:21:14.587 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.10558954800000001, 2024-09-21T10:21:14.587 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:14.587 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:14.587 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:14.587 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36822", 2024-09-21T10:21:14.587 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:21:14.587 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 197538 2024-09-21T10:21:14.588 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.588 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:14.588 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.588 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:14.588 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.563584+0000", 2024-09-21T10:21:14.588 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.588 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.588 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.588 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:14.588 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.563584+0000", 2024-09-21T10:21:14.588 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.588 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.588 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.588 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:14.589 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.563583+0000", 2024-09-21T10:21:14.589 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:21:14.589 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.589 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.589 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:14.589 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.563589+0000", 2024-09-21T10:21:14.589 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7679999999999997e-06 2024-09-21T10:21:14.589 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.589 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.589 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:14.589 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.563601+0000", 2024-09-21T10:21:14.589 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2298e-05 2024-09-21T10:21:14.589 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.589 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.589 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:14.590 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.563607+0000", 2024-09-21T10:21:14.590 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5790000000000003e-06 2024-09-21T10:21:14.590 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.590 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.590 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:14.590 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.563656+0000", 2024-09-21T10:21:14.590 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9462999999999997e-05 2024-09-21T10:21:14.590 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.590 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.590 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:14.590 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.563780+0000", 2024-09-21T10:21:14.590 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012353599999999999 2024-09-21T10:21:14.590 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.590 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.591 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:21:14.591 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.564202+0000", 2024-09-21T10:21:14.591 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00042223299999999999 2024-09-21T10:21:14.591 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.591 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.591 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:14.591 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.566505+0000", 2024-09-21T10:21:14.591 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0023028139999999998 2024-09-21T10:21:14.591 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.591 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.591 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.591 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.663685+0000", 2024-09-21T10:21:14.591 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.097180330999999995 2024-09-21T10:21:14.591 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.592 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.592 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.592 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.668909+0000", 2024-09-21T10:21:14.592 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0052239870000000002 2024-09-21T10:21:14.592 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.592 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.592 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:14.592 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.669132+0000", 2024-09-21T10:21:14.592 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00022286499999999999 2024-09-21T10:21:14.592 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.592 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.592 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:14.592 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.669173+0000", 2024-09-21T10:21:14.592 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1729e-05 2024-09-21T10:21:14.593 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.593 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.593 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.593 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.593 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.593 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36822.0:199967 12.4 12:241d58a0:::benchmark_data_smithi107_250110_object88557:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-09-21T10:21:14.593 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:17:29.893207+0000", 2024-09-21T10:21:14.593 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 224.68211880699999, 2024-09-21T10:21:14.593 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.098946864999999995, 2024-09-21T10:21:14.593 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:14.593 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:14.593 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:14.593 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36822", 2024-09-21T10:21:14.593 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:21:14.594 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 199967 2024-09-21T10:21:14.594 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.594 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:14.594 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.595 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:14.595 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:29.893207+0000", 2024-09-21T10:21:14.595 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.595 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.595 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.595 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:14.595 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:29.893207+0000", 2024-09-21T10:21:14.595 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.595 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.595 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.595 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:14.595 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:29.893206+0000", 2024-09-21T10:21:14.596 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:21:14.596 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.596 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.596 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:14.596 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:29.893212+0000", 2024-09-21T10:21:14.596 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8080000000000001e-06 2024-09-21T10:21:14.596 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.596 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.596 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:14.596 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:29.893222+0000", 2024-09-21T10:21:14.596 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0352e-05 2024-09-21T10:21:14.596 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.596 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.596 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:14.596 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:29.893227+0000", 2024-09-21T10:21:14.596 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2599999999999996e-06 2024-09-21T10:21:14.597 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.597 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.597 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:14.597 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:29.893280+0000", 2024-09-21T10:21:14.597 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2540000000000002e-05 2024-09-21T10:21:14.597 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.597 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.597 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:14.597 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:29.893380+0000", 2024-09-21T10:21:14.597 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000100388 2024-09-21T10:21:14.597 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.597 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.597 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:21:14.597 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:29.893825+0000", 2024-09-21T10:21:14.597 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00044535799999999997 2024-09-21T10:21:14.597 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.598 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.598 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:14.598 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:29.896509+0000", 2024-09-21T10:21:14.598 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0026839759999999998 2024-09-21T10:21:14.598 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.598 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.598 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.598 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:29.976252+0000", 2024-09-21T10:21:14.598 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.079743015 2024-09-21T10:21:14.598 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.598 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.598 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.598 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:29.992102+0000", 2024-09-21T10:21:14.598 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.015849607000000002 2024-09-21T10:21:14.598 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.599 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.599 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:14.599 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:29.992127+0000", 2024-09-21T10:21:14.599 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.5207000000000001e-05 2024-09-21T10:21:14.599 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.599 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.599 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:14.599 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:29.992154+0000", 2024-09-21T10:21:14.599 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.6573000000000001e-05 2024-09-21T10:21:14.599 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.599 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.599 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.599 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.599 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.600 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:20220 13.4 13:2f2ad2fd:::benchmark_data_smithi107_278655_object20219:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:21:14.600 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:28.343730+0000", 2024-09-21T10:21:14.600 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 166.231595785, 2024-09-21T10:21:14.600 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.097372677000000005, 2024-09-21T10:21:14.600 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:14.600 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:14.600 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:14.600 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:14.600 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:14.600 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 20220 2024-09-21T10:21:14.600 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.600 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:14.601 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.601 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:14.601 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.343730+0000", 2024-09-21T10:21:14.601 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.601 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.601 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.601 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:14.601 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.343730+0000", 2024-09-21T10:21:14.601 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.601 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.601 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.601 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:14.601 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.343729+0000", 2024-09-21T10:21:14.601 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:21:14.602 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.602 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.602 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:14.602 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.343793+0000", 2024-09-21T10:21:14.602 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3887999999999995e-05 2024-09-21T10:21:14.602 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.602 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.602 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:14.602 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.343808+0000", 2024-09-21T10:21:14.602 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4453e-05 2024-09-21T10:21:14.602 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.602 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.602 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:14.602 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.343813+0000", 2024-09-21T10:21:14.603 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4670000000000003e-06 2024-09-21T10:21:14.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.603 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.603 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:14.603 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.343855+0000", 2024-09-21T10:21:14.603 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1989000000000001e-05 2024-09-21T10:21:14.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.603 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.603 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:14.603 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.343968+0000", 2024-09-21T10:21:14.603 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000113146 2024-09-21T10:21:14.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.603 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.603 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:21:14.603 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.344102+0000", 2024-09-21T10:21:14.603 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000133752 2024-09-21T10:21:14.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.604 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.604 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:14.604 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.345157+0000", 2024-09-21T10:21:14.604 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001054599 2024-09-21T10:21:14.604 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.604 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.604 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.604 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.346093+0000", 2024-09-21T10:21:14.604 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00093678399999999997 2024-09-21T10:21:14.604 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.604 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.604 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.604 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.440982+0000", 2024-09-21T10:21:14.604 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.094888388000000004 2024-09-21T10:21:14.604 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.604 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.605 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:14.605 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.441056+0000", 2024-09-21T10:21:14.605 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.4479000000000003e-05 2024-09-21T10:21:14.605 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.605 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.605 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:14.605 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.441103+0000", 2024-09-21T10:21:14.605 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6513e-05 2024-09-21T10:21:14.605 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.605 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.605 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.605 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.605 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.605 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:20224 13.1 13:8735fc97:::benchmark_data_smithi107_278655_object20223:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:21:14.605 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:28.345671+0000", 2024-09-21T10:21:14.605 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 166.229654603, 2024-09-21T10:21:14.606 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.095722594999999994, 2024-09-21T10:21:14.606 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:14.606 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:14.606 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:14.606 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:14.606 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:14.606 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 20224 2024-09-21T10:21:14.606 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.606 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:14.606 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.606 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:14.606 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.345671+0000", 2024-09-21T10:21:14.606 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.607 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.607 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.607 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:14.607 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.345671+0000", 2024-09-21T10:21:14.607 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.607 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.607 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.607 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:14.607 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.345670+0000", 2024-09-21T10:21:14.607 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:21:14.607 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.607 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.607 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:14.608 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.345701+0000", 2024-09-21T10:21:14.608 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1216000000000001e-05 2024-09-21T10:21:14.608 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.608 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.608 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:14.608 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.345712+0000", 2024-09-21T10:21:14.608 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1043000000000001e-05 2024-09-21T10:21:14.608 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.608 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.608 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:14.608 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.345718+0000", 2024-09-21T10:21:14.608 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2530000000000004e-06 2024-09-21T10:21:14.608 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.608 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.608 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:14.609 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.345773+0000", 2024-09-21T10:21:14.609 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4935000000000002e-05 2024-09-21T10:21:14.609 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.609 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.609 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:14.609 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.345953+0000", 2024-09-21T10:21:14.609 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00018029499999999999 2024-09-21T10:21:14.609 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.609 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.609 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,4", 2024-09-21T10:21:14.609 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.346152+0000", 2024-09-21T10:21:14.609 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000199302 2024-09-21T10:21:14.609 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.609 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.609 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:14.610 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.347048+0000", 2024-09-21T10:21:14.610 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00089588600000000003 2024-09-21T10:21:14.610 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.610 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.610 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.610 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.347343+0000", 2024-09-21T10:21:14.610 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00029498000000000003 2024-09-21T10:21:14.610 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.610 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.610 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.610 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.441307+0000", 2024-09-21T10:21:14.610 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.093963537 2024-09-21T10:21:14.610 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.610 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.610 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:14.611 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.441357+0000", 2024-09-21T10:21:14.611 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0148000000000002e-05 2024-09-21T10:21:14.611 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.611 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.611 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:14.611 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:28.441394+0000", 2024-09-21T10:21:14.611 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7039999999999998e-05 2024-09-21T10:21:14.611 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.611 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.611 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.611 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.611 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.611 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:23842 13.4 13:2ae185ed:::benchmark_data_smithi107_278655_object23841:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:21:14.611 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:29.864431+0000", 2024-09-21T10:21:14.611 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 164.71089458099999, 2024-09-21T10:21:14.612 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.168915237, 2024-09-21T10:21:14.612 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:14.612 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:14.612 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:14.612 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:14.612 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:14.612 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 23842 2024-09-21T10:21:14.612 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.612 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:14.612 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.612 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:14.612 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864431+0000", 2024-09-21T10:21:14.612 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.612 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.613 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.613 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:14.613 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864431+0000", 2024-09-21T10:21:14.613 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.613 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.613 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.613 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:14.613 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864430+0000", 2024-09-21T10:21:14.613 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:21:14.613 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.613 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.613 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:14.613 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864487+0000", 2024-09-21T10:21:14.613 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7219000000000002e-05 2024-09-21T10:21:14.613 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.613 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.614 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:14.614 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864499+0000", 2024-09-21T10:21:14.614 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2062000000000001e-05 2024-09-21T10:21:14.614 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.614 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.614 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:14.614 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864505+0000", 2024-09-21T10:21:14.614 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0100000000000001e-06 2024-09-21T10:21:14.614 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.614 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.614 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:14.614 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864555+0000", 2024-09-21T10:21:14.614 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9574999999999999e-05 2024-09-21T10:21:14.614 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.614 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.615 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:14.615 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864687+0000", 2024-09-21T10:21:14.615 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013213 2024-09-21T10:21:14.615 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.615 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.615 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:21:14.615 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864915+0000", 2024-09-21T10:21:14.615 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00022785700000000001 2024-09-21T10:21:14.615 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.615 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.615 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:14.615 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.867465+0000", 2024-09-21T10:21:14.615 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0025501230000000001 2024-09-21T10:21:14.615 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.615 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.616 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.616 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.867521+0000", 2024-09-21T10:21:14.616 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5708000000000003e-05 2024-09-21T10:21:14.616 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.616 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.616 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.616 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:30.033258+0000", 2024-09-21T10:21:14.616 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.165737356 2024-09-21T10:21:14.616 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.616 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.616 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:14.616 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:30.033310+0000", 2024-09-21T10:21:14.616 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1539000000000002e-05 2024-09-21T10:21:14.617 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.617 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.617 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:14.617 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:30.033346+0000", 2024-09-21T10:21:14.617 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6936e-05 2024-09-21T10:21:14.617 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.617 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.617 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.617 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.617 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.617 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:23845 13.4 13:27245713:::benchmark_data_smithi107_278655_object23844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:21:14.617 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:29.866129+0000", 2024-09-21T10:21:14.617 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 164.70919680700001, 2024-09-21T10:21:14.617 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.16733213499999999, 2024-09-21T10:21:14.617 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:14.617 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:14.618 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:14.618 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:14.618 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:14.618 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 23845 2024-09-21T10:21:14.618 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.618 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:14.618 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.618 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:14.618 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866129+0000", 2024-09-21T10:21:14.618 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.618 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.618 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.619 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:14.619 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866129+0000", 2024-09-21T10:21:14.619 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.619 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.619 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.619 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:14.619 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866128+0000", 2024-09-21T10:21:14.619 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:21:14.619 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.619 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.619 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:14.619 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866189+0000", 2024-09-21T10:21:14.619 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1872999999999995e-05 2024-09-21T10:21:14.619 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.619 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.619 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:14.620 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866204+0000", 2024-09-21T10:21:14.620 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4739000000000001e-05 2024-09-21T10:21:14.620 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.620 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.620 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:14.620 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866211+0000", 2024-09-21T10:21:14.620 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2949999999999999e-06 2024-09-21T10:21:14.620 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.620 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.620 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:14.620 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866248+0000", 2024-09-21T10:21:14.620 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7704999999999999e-05 2024-09-21T10:21:14.620 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.620 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.620 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:14.621 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866354+0000", 2024-09-21T10:21:14.621 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000106022 2024-09-21T10:21:14.621 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.621 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.621 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:21:14.621 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866575+0000", 2024-09-21T10:21:14.621 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00022120399999999999 2024-09-21T10:21:14.621 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.621 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.621 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.621 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.867933+0000", 2024-09-21T10:21:14.621 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0013572499999999999 2024-09-21T10:21:14.621 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.621 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.622 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:14.622 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.867987+0000", 2024-09-21T10:21:14.622 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3919999999999999e-05 2024-09-21T10:21:14.622 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.622 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.622 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.622 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:30.033382+0000", 2024-09-21T10:21:14.622 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.165395343 2024-09-21T10:21:14.622 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.622 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.622 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:14.622 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:30.033430+0000", 2024-09-21T10:21:14.622 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7997999999999997e-05 2024-09-21T10:21:14.623 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.623 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.623 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:14.623 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:30.033461+0000", 2024-09-21T10:21:14.623 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1149999999999998e-05 2024-09-21T10:21:14.623 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.623 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.623 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.623 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.623 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.623 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:26191 13.f 13:f6831ef3:::benchmark_data_smithi107_278655_object26190:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:21:14.623 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:31.006136+0000", 2024-09-21T10:21:14.623 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 163.569190015, 2024-09-21T10:21:14.623 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.097281242000000004, 2024-09-21T10:21:14.623 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:14.623 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:14.624 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:14.624 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:14.624 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:14.624 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 26191 2024-09-21T10:21:14.624 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.624 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:14.624 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.624 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:14.624 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.006136+0000", 2024-09-21T10:21:14.624 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.624 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.624 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.624 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:14.625 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.006136+0000", 2024-09-21T10:21:14.625 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.625 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.625 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.625 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:14.625 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.006134+0000", 2024-09-21T10:21:14.625 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:21:14.625 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.625 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.625 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:14.625 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.006167+0000", 2024-09-21T10:21:14.625 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2617000000000003e-05 2024-09-21T10:21:14.625 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.625 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.626 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:14.626 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.006185+0000", 2024-09-21T10:21:14.626 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.8451e-05 2024-09-21T10:21:14.626 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.626 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.626 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:14.626 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.006192+0000", 2024-09-21T10:21:14.626 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6329999999999999e-06 2024-09-21T10:21:14.626 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.626 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.626 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:14.626 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.006243+0000", 2024-09-21T10:21:14.626 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1396000000000001e-05 2024-09-21T10:21:14.626 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.627 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.627 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:14.627 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.006390+0000", 2024-09-21T10:21:14.627 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014700600000000001 2024-09-21T10:21:14.627 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.627 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.627 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:21:14.627 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.006546+0000", 2024-09-21T10:21:14.627 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000155716 2024-09-21T10:21:14.627 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.627 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.627 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.627 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.010209+0000", 2024-09-21T10:21:14.627 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0036623889999999998 2024-09-21T10:21:14.628 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.628 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.628 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:14.628 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.010256+0000", 2024-09-21T10:21:14.628 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7011000000000001e-05 2024-09-21T10:21:14.628 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.628 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.628 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.628 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.103314+0000", 2024-09-21T10:21:14.628 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.093058416000000005 2024-09-21T10:21:14.628 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.628 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.628 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:14.628 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.103367+0000", 2024-09-21T10:21:14.629 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3049e-05 2024-09-21T10:21:14.629 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.629 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.629 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:14.629 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.103417+0000", 2024-09-21T10:21:14.629 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9975999999999997e-05 2024-09-21T10:21:14.629 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.629 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.629 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.629 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.629 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.629 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:26194 13.f 13:f8cce869:::benchmark_data_smithi107_278655_object26193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:21:14.629 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:31.007182+0000", 2024-09-21T10:21:14.629 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 163.568143943, 2024-09-21T10:21:14.630 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.10133468, 2024-09-21T10:21:14.630 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:14.630 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:14.630 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:14.630 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:14.630 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:14.630 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 26194 2024-09-21T10:21:14.630 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.630 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:14.630 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.630 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:14.630 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.007182+0000", 2024-09-21T10:21:14.631 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.631 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.631 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.631 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:14.631 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.007182+0000", 2024-09-21T10:21:14.631 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.631 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.631 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.631 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:14.631 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.007181+0000", 2024-09-21T10:21:14.631 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:21:14.631 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.631 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.631 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:14.631 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.007227+0000", 2024-09-21T10:21:14.632 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6006e-05 2024-09-21T10:21:14.632 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.632 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.632 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:14.632 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.007239+0000", 2024-09-21T10:21:14.632 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2275e-05 2024-09-21T10:21:14.632 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.632 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.632 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:14.632 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.007246+0000", 2024-09-21T10:21:14.632 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7229999999999998e-06 2024-09-21T10:21:14.632 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.632 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.633 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:14.633 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.007904+0000", 2024-09-21T10:21:14.633 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00065871400000000002 2024-09-21T10:21:14.633 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.633 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.633 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:14.633 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.008040+0000", 2024-09-21T10:21:14.633 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000135661 2024-09-21T10:21:14.633 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.633 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.633 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:21:14.633 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.008196+0000", 2024-09-21T10:21:14.633 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015607800000000001 2024-09-21T10:21:14.633 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.633 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.634 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.634 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.010242+0000", 2024-09-21T10:21:14.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:14.635 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0020456300000000001 2024-09-21T10:21:14.635 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.635 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.635 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:14.635 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.010262+0000", 2024-09-21T10:21:14.635 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.0285e-05 2024-09-21T10:21:14.635 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.636 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.636 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.636 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.108390+0000", 2024-09-21T10:21:14.636 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.098128180999999995 2024-09-21T10:21:14.636 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.636 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.636 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:14.636 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.108481+0000", 2024-09-21T10:21:14.636 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.0688999999999999e-05 2024-09-21T10:21:14.636 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.636 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.637 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:14.637 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.108517+0000", 2024-09-21T10:21:14.637 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5747000000000002e-05 2024-09-21T10:21:14.637 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.637 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.637 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.637 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.637 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.637 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:26201 13.4 13:2f8b127e:::benchmark_data_smithi107_278655_object26200:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:21:14.637 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:31.009445+0000", 2024-09-21T10:21:14.637 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 163.565881031, 2024-09-21T10:21:14.637 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.104710918, 2024-09-21T10:21:14.637 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:14.638 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:14.638 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:14.638 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:14.638 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:14.638 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 26201 2024-09-21T10:21:14.638 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.638 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:14.638 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.638 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:14.638 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009445+0000", 2024-09-21T10:21:14.638 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.638 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.638 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.638 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:14.638 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009445+0000", 2024-09-21T10:21:14.639 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.639 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.639 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.639 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:14.639 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009443+0000", 2024-09-21T10:21:14.639 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:21:14.639 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.639 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.639 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:14.639 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009480+0000", 2024-09-21T10:21:14.639 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6369999999999999e-05 2024-09-21T10:21:14.639 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.639 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.639 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:14.639 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009535+0000", 2024-09-21T10:21:14.640 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5179999999999997e-05 2024-09-21T10:21:14.640 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.640 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.640 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:14.640 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009543+0000", 2024-09-21T10:21:14.640 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.2099999999999993e-06 2024-09-21T10:21:14.640 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.640 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.640 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:14.640 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009591+0000", 2024-09-21T10:21:14.640 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7868000000000003e-05 2024-09-21T10:21:14.640 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.640 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.640 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:14.641 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009710+0000", 2024-09-21T10:21:14.641 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000118785 2024-09-21T10:21:14.641 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.641 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.641 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:21:14.641 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009840+0000", 2024-09-21T10:21:14.641 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013075200000000001 2024-09-21T10:21:14.641 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.641 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.641 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.641 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.012789+0000", 2024-09-21T10:21:14.641 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0029491000000000001 2024-09-21T10:21:14.641 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.641 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.641 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:14.642 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.012838+0000", 2024-09-21T10:21:14.642 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9020999999999997e-05 2024-09-21T10:21:14.642 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.642 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.642 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.642 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.114063+0000", 2024-09-21T10:21:14.642 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.101224339 2024-09-21T10:21:14.642 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.642 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.642 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:14.642 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.114118+0000", 2024-09-21T10:21:14.643 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5630000000000001e-05 2024-09-21T10:21:14.643 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.643 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.643 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:14.643 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.114156+0000", 2024-09-21T10:21:14.643 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7219999999999999e-05 2024-09-21T10:21:14.643 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.643 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.643 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.643 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.643 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.643 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:26202 13.4 13:29033db0:::benchmark_data_smithi107_278655_object26201:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:21:14.643 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:31.009680+0000", 2024-09-21T10:21:14.644 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 163.565645354, 2024-09-21T10:21:14.644 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.104620422, 2024-09-21T10:21:14.644 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:14.644 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:14.644 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:14.644 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:14.644 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:14.644 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 26202 2024-09-21T10:21:14.644 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.644 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:14.644 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.644 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:14.644 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009680+0000", 2024-09-21T10:21:14.644 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.645 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.645 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.645 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:14.645 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009680+0000", 2024-09-21T10:21:14.645 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.645 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.645 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.645 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:14.645 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009679+0000", 2024-09-21T10:21:14.645 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:21:14.645 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.645 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.645 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:14.645 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009866+0000", 2024-09-21T10:21:14.645 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00018638699999999999 2024-09-21T10:21:14.645 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.646 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.646 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:14.646 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009880+0000", 2024-09-21T10:21:14.646 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4581e-05 2024-09-21T10:21:14.646 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.646 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.646 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:14.646 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009886+0000", 2024-09-21T10:21:14.646 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.93e-06 2024-09-21T10:21:14.646 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.646 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.646 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:14.646 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.011012+0000", 2024-09-21T10:21:14.647 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001125865 2024-09-21T10:21:14.647 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.647 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.647 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:14.647 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.011152+0000", 2024-09-21T10:21:14.647 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013987400000000001 2024-09-21T10:21:14.647 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.647 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.647 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:21:14.647 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.011291+0000", 2024-09-21T10:21:14.647 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013847100000000001 2024-09-21T10:21:14.647 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.647 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.647 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.647 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.012832+0000", 2024-09-21T10:21:14.647 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001541031 2024-09-21T10:21:14.648 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.648 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.648 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:14.648 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.012862+0000", 2024-09-21T10:21:14.648 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0116000000000001e-05 2024-09-21T10:21:14.648 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.648 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.648 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.648 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.114214+0000", 2024-09-21T10:21:14.648 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.10135232700000001 2024-09-21T10:21:14.648 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.648 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.648 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:14.648 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.114262+0000", 2024-09-21T10:21:14.648 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8152000000000002e-05 2024-09-21T10:21:14.649 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.649 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.649 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:14.649 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.114301+0000", 2024-09-21T10:21:14.649 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8625999999999999e-05 2024-09-21T10:21:14.649 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.649 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.649 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.649 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.649 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.649 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:26946 13.4 13:21264d98:::benchmark_data_smithi107_278655_object26945:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:21:14.649 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:31.498036+0000", 2024-09-21T10:21:14.649 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 163.07728990199999, 2024-09-21T10:21:14.649 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.30224518299999997, 2024-09-21T10:21:14.650 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:14.650 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:14.650 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:14.650 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:14.650 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:14.650 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 26946 2024-09-21T10:21:14.650 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.650 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:14.650 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.650 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:14.650 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498036+0000", 2024-09-21T10:21:14.650 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.650 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.651 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.651 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:14.651 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498036+0000", 2024-09-21T10:21:14.651 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.651 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.651 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.651 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:14.651 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498035+0000", 2024-09-21T10:21:14.651 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:21:14.651 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.651 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.651 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:14.651 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498090+0000", 2024-09-21T10:21:14.651 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5785000000000002e-05 2024-09-21T10:21:14.652 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.652 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.652 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:14.652 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498104+0000", 2024-09-21T10:21:14.652 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3871e-05 2024-09-21T10:21:14.652 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.652 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.652 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:14.652 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498111+0000", 2024-09-21T10:21:14.652 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4389999999999997e-06 2024-09-21T10:21:14.652 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.652 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.652 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:14.652 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498154+0000", 2024-09-21T10:21:14.652 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3238999999999997e-05 2024-09-21T10:21:14.652 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.653 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.653 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:14.653 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498237+0000", 2024-09-21T10:21:14.653 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.2571000000000006e-05 2024-09-21T10:21:14.653 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.653 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.653 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:21:14.653 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498343+0000", 2024-09-21T10:21:14.653 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000106498 2024-09-21T10:21:14.653 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.653 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.653 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:14.653 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.499830+0000", 2024-09-21T10:21:14.653 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0014867529999999999 2024-09-21T10:21:14.654 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.654 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.654 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.654 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.499999+0000", 2024-09-21T10:21:14.654 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00016944099999999999 2024-09-21T10:21:14.654 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.654 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.654 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.654 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.800209+0000", 2024-09-21T10:21:14.654 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.300209275 2024-09-21T10:21:14.654 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.654 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.655 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:14.655 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.800250+0000", 2024-09-21T10:21:14.655 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.189e-05 2024-09-21T10:21:14.655 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.655 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.655 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:14.655 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.800281+0000", 2024-09-21T10:21:14.655 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0618999999999999e-05 2024-09-21T10:21:14.655 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.655 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.655 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.655 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.655 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.655 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:61324 13.1 13:8d728946:::benchmark_data_smithi107_278655_object61323:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:21:14.656 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:52.281968+0000", 2024-09-21T10:21:14.656 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 142.29335803500001, 2024-09-21T10:21:14.656 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.105068279, 2024-09-21T10:21:14.656 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:14.656 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:14.656 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:14.656 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:14.656 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:14.656 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 61324 2024-09-21T10:21:14.656 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.656 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:14.656 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.656 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:14.656 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.281968+0000", 2024-09-21T10:21:14.657 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.657 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.657 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.657 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:14.657 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.281968+0000", 2024-09-21T10:21:14.657 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.657 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.657 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.657 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:14.657 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.281966+0000", 2024-09-21T10:21:14.657 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:21:14.657 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.657 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.657 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:14.658 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282001+0000", 2024-09-21T10:21:14.658 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4066000000000002e-05 2024-09-21T10:21:14.658 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.658 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.658 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:14.658 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282016+0000", 2024-09-21T10:21:14.658 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5172e-05 2024-09-21T10:21:14.658 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.658 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.658 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:14.658 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282021+0000", 2024-09-21T10:21:14.658 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7980000000000002e-06 2024-09-21T10:21:14.658 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.658 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.659 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:14.659 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282066+0000", 2024-09-21T10:21:14.659 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4577999999999999e-05 2024-09-21T10:21:14.659 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.659 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.659 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:14.659 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282222+0000", 2024-09-21T10:21:14.659 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015632499999999999 2024-09-21T10:21:14.659 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.659 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.659 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,4", 2024-09-21T10:21:14.659 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282384+0000", 2024-09-21T10:21:14.659 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00016192999999999999 2024-09-21T10:21:14.659 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.659 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.659 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:14.660 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.285898+0000", 2024-09-21T10:21:14.660 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0035140700000000002 2024-09-21T10:21:14.660 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.660 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.660 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.660 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.287757+0000", 2024-09-21T10:21:14.660 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0018584750000000001 2024-09-21T10:21:14.660 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.660 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.660 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.660 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.386791+0000", 2024-09-21T10:21:14.660 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.099033940000000001 2024-09-21T10:21:14.660 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.660 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.660 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:14.661 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.386935+0000", 2024-09-21T10:21:14.661 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014445300000000001 2024-09-21T10:21:14.661 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.661 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.661 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:14.661 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.387036+0000", 2024-09-21T10:21:14.661 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000100777 2024-09-21T10:21:14.661 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.661 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.661 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.661 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.661 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.661 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:61325 13.f 13:f42c75c5:::benchmark_data_smithi107_278655_object61324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:21:14.661 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:52.282123+0000", 2024-09-21T10:21:14.661 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 142.29320263700001, 2024-09-21T10:21:14.662 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.10875503, 2024-09-21T10:21:14.662 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:14.662 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:14.662 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:14.662 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:14.662 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:14.662 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 61325 2024-09-21T10:21:14.662 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.662 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:14.662 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.662 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:14.662 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282123+0000", 2024-09-21T10:21:14.662 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.662 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.662 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.662 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:14.663 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282123+0000", 2024-09-21T10:21:14.663 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.663 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.663 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.663 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:14.663 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282122+0000", 2024-09-21T10:21:14.663 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:21:14.663 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.663 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.663 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:14.663 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282177+0000", 2024-09-21T10:21:14.663 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5003999999999997e-05 2024-09-21T10:21:14.663 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.663 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.663 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:14.664 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282191+0000", 2024-09-21T10:21:14.664 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3472000000000001e-05 2024-09-21T10:21:14.664 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.664 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.664 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:14.664 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282196+0000", 2024-09-21T10:21:14.664 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5629999999999999e-06 2024-09-21T10:21:14.664 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.664 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.664 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:14.664 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282249+0000", 2024-09-21T10:21:14.664 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3183999999999998e-05 2024-09-21T10:21:14.664 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.664 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.664 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:14.664 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282360+0000", 2024-09-21T10:21:14.665 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000110653 2024-09-21T10:21:14.665 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.665 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.665 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:21:14.665 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282491+0000", 2024-09-21T10:21:14.665 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013050400000000001 2024-09-21T10:21:14.665 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.665 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.665 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:14.665 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.288425+0000", 2024-09-21T10:21:14.665 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0059345939999999996 2024-09-21T10:21:14.665 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.665 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.665 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.665 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.288846+0000", 2024-09-21T10:21:14.666 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00042060099999999999 2024-09-21T10:21:14.666 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.666 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.666 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.666 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.390750+0000", 2024-09-21T10:21:14.666 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.101904019 2024-09-21T10:21:14.666 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.666 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.666 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:14.666 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.390835+0000", 2024-09-21T10:21:14.666 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.4732000000000001e-05 2024-09-21T10:21:14.666 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.666 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.667 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:14.667 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.390878+0000", 2024-09-21T10:21:14.667 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3671000000000003e-05 2024-09-21T10:21:14.667 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.667 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.667 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.667 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.667 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.667 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:61328 13.f 13:f2ce0a28:::benchmark_data_smithi107_278655_object61327:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:21:14.667 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:52.285810+0000", 2024-09-21T10:21:14.667 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 142.28951590700001, 2024-09-21T10:21:14.667 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.10897699299999999, 2024-09-21T10:21:14.667 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:14.667 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:14.668 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:14.668 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:14.668 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:14.668 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 61328 2024-09-21T10:21:14.668 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.668 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:14.668 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.668 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:14.668 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.285810+0000", 2024-09-21T10:21:14.668 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.668 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.668 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.668 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:14.668 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.285810+0000", 2024-09-21T10:21:14.669 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.669 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.669 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.669 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:14.669 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.285808+0000", 2024-09-21T10:21:14.669 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:21:14.669 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.669 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.669 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:14.669 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.285847+0000", 2024-09-21T10:21:14.669 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8485000000000002e-05 2024-09-21T10:21:14.669 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.669 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.669 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:14.669 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.285861+0000", 2024-09-21T10:21:14.670 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4246999999999999e-05 2024-09-21T10:21:14.670 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.670 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.670 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:14.670 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.285868+0000", 2024-09-21T10:21:14.670 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8020000000000003e-06 2024-09-21T10:21:14.670 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.670 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.670 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:14.670 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.285914+0000", 2024-09-21T10:21:14.670 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5728e-05 2024-09-21T10:21:14.670 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.670 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.670 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.286057+0000", 2024-09-21T10:21:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014371800000000001 2024-09-21T10:21:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:21:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.286210+0000", 2024-09-21T10:21:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000152972 2024-09-21T10:21:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.288879+0000", 2024-09-21T10:21:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002669051 2024-09-21T10:21:14.671 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.290367+0000", 2024-09-21T10:21:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001487302 2024-09-21T10:21:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.394700+0000", 2024-09-21T10:21:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.104333327 2024-09-21T10:21:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.394751+0000", 2024-09-21T10:21:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0930999999999997e-05 2024-09-21T10:21:14.672 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.673 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.673 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:14.673 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.394787+0000", 2024-09-21T10:21:14.673 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5923000000000001e-05 2024-09-21T10:21:14.673 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.673 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.673 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.673 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.673 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.673 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:72757 13.f 13:f96442f9:::benchmark_data_smithi107_278655_object72756:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:21:14.673 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.476101+0000", 2024-09-21T10:21:14.673 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 134.099224634, 2024-09-21T10:21:14.673 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.26259288400000003, 2024-09-21T10:21:14.673 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:14.674 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:14.674 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:14.674 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:14.674 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:14.674 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 72757 2024-09-21T10:21:14.674 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.674 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:14.674 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.674 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:14.674 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476101+0000", 2024-09-21T10:21:14.674 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.674 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.674 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.674 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:14.674 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476101+0000", 2024-09-21T10:21:14.676 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.676 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.676 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.676 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:14.676 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476100+0000", 2024-09-21T10:21:14.676 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:21:14.676 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.676 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.676 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:14.676 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476136+0000", 2024-09-21T10:21:14.676 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6081000000000002e-05 2024-09-21T10:21:14.676 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.676 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.676 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:14.677 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476151+0000", 2024-09-21T10:21:14.677 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4958e-05 2024-09-21T10:21:14.677 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.677 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.677 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:14.677 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476157+0000", 2024-09-21T10:21:14.677 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9150000000000001e-06 2024-09-21T10:21:14.677 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.677 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.677 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:14.677 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476242+0000", 2024-09-21T10:21:14.677 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.4896999999999997e-05 2024-09-21T10:21:14.677 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.677 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.677 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:14.678 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476468+0000", 2024-09-21T10:21:14.678 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00022605300000000001 2024-09-21T10:21:14.678 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.678 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.678 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:21:14.678 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476602+0000", 2024-09-21T10:21:14.678 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013458 2024-09-21T10:21:14.678 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.678 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.678 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.678 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481355+0000", 2024-09-21T10:21:14.678 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0047528479999999996 2024-09-21T10:21:14.678 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.678 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.679 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:14.679 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485113+0000", 2024-09-21T10:21:14.679 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003757842 2024-09-21T10:21:14.679 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.679 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.679 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.679 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.738610+0000", 2024-09-21T10:21:14.679 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25349742400000003 2024-09-21T10:21:14.679 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.679 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.679 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:14.679 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.738650+0000", 2024-09-21T10:21:14.679 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0089999999999997e-05 2024-09-21T10:21:14.679 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.679 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.679 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:14.680 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.738694+0000", 2024-09-21T10:21:14.680 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3649000000000002e-05 2024-09-21T10:21:14.680 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.680 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.680 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.680 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.680 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.680 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:72765 13.f 13:f6d6e7fe:::benchmark_data_smithi107_278655_object72764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:21:14.680 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.481220+0000", 2024-09-21T10:21:14.680 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 134.09410602599999, 2024-09-21T10:21:14.680 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.26411038599999997, 2024-09-21T10:21:14.680 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:14.680 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:14.680 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:14.680 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:14.680 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:14.681 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 72765 2024-09-21T10:21:14.681 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.681 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:14.681 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.681 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:14.681 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481220+0000", 2024-09-21T10:21:14.681 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.681 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.681 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.681 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:14.681 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481220+0000", 2024-09-21T10:21:14.681 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.681 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.681 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.681 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:14.681 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481218+0000", 2024-09-21T10:21:14.682 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:21:14.682 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.682 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.682 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:14.682 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481277+0000", 2024-09-21T10:21:14.682 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8371000000000001e-05 2024-09-21T10:21:14.682 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.682 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.682 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:14.682 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481289+0000", 2024-09-21T10:21:14.682 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2197e-05 2024-09-21T10:21:14.682 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.682 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.682 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:14.682 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481295+0000", 2024-09-21T10:21:14.682 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8309999999999997e-06 2024-09-21T10:21:14.683 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.683 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.683 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:14.683 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481444+0000", 2024-09-21T10:21:14.683 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014953500000000001 2024-09-21T10:21:14.683 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.683 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.683 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:14.683 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.482468+0000", 2024-09-21T10:21:14.683 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0010236080000000001 2024-09-21T10:21:14.683 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.683 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.683 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:21:14.683 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.482570+0000", 2024-09-21T10:21:14.683 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000101711 2024-09-21T10:21:14.684 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.684 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.684 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:14.684 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.488060+0000", 2024-09-21T10:21:14.684 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0054900289999999996 2024-09-21T10:21:14.684 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.684 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.684 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.684 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489602+0000", 2024-09-21T10:21:14.684 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0015419990000000001 2024-09-21T10:21:14.684 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.684 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.684 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.684 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745244+0000", 2024-09-21T10:21:14.684 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25564244800000002 2024-09-21T10:21:14.685 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.685 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.685 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:14.685 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745299+0000", 2024-09-21T10:21:14.685 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4877999999999999e-05 2024-09-21T10:21:14.685 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.685 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.685 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:14.685 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745330+0000", 2024-09-21T10:21:14.685 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1177000000000003e-05 2024-09-21T10:21:14.685 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.685 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.685 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.685 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.685 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.685 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:72767 13.f 13:fba5e0d5:::benchmark_data_smithi107_278655_object72766:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:21:14.686 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.481760+0000", 2024-09-21T10:21:14.686 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 134.09356572600001, 2024-09-21T10:21:14.686 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.263929786, 2024-09-21T10:21:14.686 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:14.686 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:14.686 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:14.686 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:14.686 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:14.686 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 72767 2024-09-21T10:21:14.686 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.686 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:14.686 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.686 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:14.686 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481760+0000", 2024-09-21T10:21:14.686 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.686 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.687 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.687 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:14.687 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481760+0000", 2024-09-21T10:21:14.687 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.687 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.687 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.687 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:14.687 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481759+0000", 2024-09-21T10:21:14.687 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:21:14.687 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.687 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.687 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:14.687 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481815+0000", 2024-09-21T10:21:14.687 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6230000000000002e-05 2024-09-21T10:21:14.687 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.687 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.688 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:14.688 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481831+0000", 2024-09-21T10:21:14.688 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5613000000000001e-05 2024-09-21T10:21:14.688 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.688 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.688 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:14.688 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481840+0000", 2024-09-21T10:21:14.688 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.9390000000000003e-06 2024-09-21T10:21:14.688 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.688 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.688 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:14.688 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485140+0000", 2024-09-21T10:21:14.688 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0033002909999999999 2024-09-21T10:21:14.688 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.688 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.689 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:14.689 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485285+0000", 2024-09-21T10:21:14.689 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014537099999999999 2024-09-21T10:21:14.689 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.689 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.689 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:21:14.689 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485433+0000", 2024-09-21T10:21:14.689 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014723399999999999 2024-09-21T10:21:14.689 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.689 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.689 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:14.689 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489134+0000", 2024-09-21T10:21:14.689 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0037009009999999999 2024-09-21T10:21:14.689 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.689 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.689 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.689 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489909+0000", 2024-09-21T10:21:14.690 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00077549400000000005 2024-09-21T10:21:14.690 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.690 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.690 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.690 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745617+0000", 2024-09-21T10:21:14.690 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.255707831 2024-09-21T10:21:14.690 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.690 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.690 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:14.690 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745660+0000", 2024-09-21T10:21:14.690 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2803999999999999e-05 2024-09-21T10:21:14.690 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.690 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.690 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:14.691 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745690+0000", 2024-09-21T10:21:14.691 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0079000000000001e-05 2024-09-21T10:21:14.691 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.691 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.691 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.691 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.691 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.691 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:72770 13.1 13:87104e38:::benchmark_data_smithi107_278655_object72769:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:21:14.691 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.485917+0000", 2024-09-21T10:21:14.691 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 134.08940886900001, 2024-09-21T10:21:14.691 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25941609300000001, 2024-09-21T10:21:14.691 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:14.691 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:14.691 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:14.691 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:14.691 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:14.691 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 72770 2024-09-21T10:21:14.692 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.692 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:14.692 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.692 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:14.692 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485917+0000", 2024-09-21T10:21:14.692 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.692 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.692 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.692 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:14.692 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485917+0000", 2024-09-21T10:21:14.692 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.692 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.692 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.692 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:14.692 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485915+0000", 2024-09-21T10:21:14.692 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:21:14.693 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.693 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.693 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:14.693 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485978+0000", 2024-09-21T10:21:14.693 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3101000000000005e-05 2024-09-21T10:21:14.693 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.693 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.693 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:14.693 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485996+0000", 2024-09-21T10:21:14.693 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.7759000000000001e-05 2024-09-21T10:21:14.693 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.693 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.693 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:14.693 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.486004+0000", 2024-09-21T10:21:14.693 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3239999999999997e-06 2024-09-21T10:21:14.693 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.694 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.694 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:14.694 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.486055+0000", 2024-09-21T10:21:14.694 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1193000000000003e-05 2024-09-21T10:21:14.694 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.694 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.694 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:14.694 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.486192+0000", 2024-09-21T10:21:14.694 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013693200000000001 2024-09-21T10:21:14.694 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.694 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.694 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,4", 2024-09-21T10:21:14.694 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.486328+0000", 2024-09-21T10:21:14.695 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013682700000000001 2024-09-21T10:21:14.695 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.695 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.695 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:14.695 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489305+0000", 2024-09-21T10:21:14.695 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002977001 2024-09-21T10:21:14.695 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.695 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.695 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.695 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489525+0000", 2024-09-21T10:21:14.695 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00021918799999999999 2024-09-21T10:21:14.695 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.695 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.695 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.696 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745228+0000", 2024-09-21T10:21:14.696 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25570367500000002 2024-09-21T10:21:14.696 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.696 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.696 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:14.696 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745295+0000", 2024-09-21T10:21:14.696 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7137000000000005e-05 2024-09-21T10:21:14.696 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.696 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.696 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:14.696 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745333+0000", 2024-09-21T10:21:14.696 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7518999999999998e-05 2024-09-21T10:21:14.696 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.696 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.697 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.697 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.697 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.697 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:72774 13.4 13:248222d3:::benchmark_data_smithi107_278655_object72773:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:21:14.697 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.489389+0000", 2024-09-21T10:21:14.697 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 134.085936506, 2024-09-21T10:21:14.697 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25593706900000002, 2024-09-21T10:21:14.697 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:14.697 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:14.697 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:14.697 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:14.697 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:14.697 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 72774 2024-09-21T10:21:14.697 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.697 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:14.697 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.698 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:14.698 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489389+0000", 2024-09-21T10:21:14.698 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.698 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.698 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.698 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:14.698 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489389+0000", 2024-09-21T10:21:14.698 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:14.698 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.698 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.698 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:14.698 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489388+0000", 2024-09-21T10:21:14.698 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:21:14.699 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.699 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.699 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:14.699 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489476+0000", 2024-09-21T10:21:14.699 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.8084999999999994e-05 2024-09-21T10:21:14.699 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.699 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.699 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:14.699 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489490+0000", 2024-09-21T10:21:14.699 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4139e-05 2024-09-21T10:21:14.699 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.699 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.699 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:14.699 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489498+0000", 2024-09-21T10:21:14.699 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.1899999999999995e-06 2024-09-21T10:21:14.699 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.700 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.700 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:14.700 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489554+0000", 2024-09-21T10:21:14.700 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5649999999999997e-05 2024-09-21T10:21:14.700 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.700 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.700 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:14.700 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489701+0000", 2024-09-21T10:21:14.700 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014734700000000001 2024-09-21T10:21:14.700 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.700 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.700 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:21:14.700 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489837+0000", 2024-09-21T10:21:14.700 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013627299999999999 2024-09-21T10:21:14.700 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.701 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.701 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:14.701 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.492678+0000", 2024-09-21T10:21:14.701 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0028408890000000001 2024-09-21T10:21:14.701 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.701 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.701 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.701 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.492823+0000", 2024-09-21T10:21:14.701 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014503199999999999 2024-09-21T10:21:14.701 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.701 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.701 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:14.701 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745232+0000", 2024-09-21T10:21:14.701 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25240888500000003 2024-09-21T10:21:14.702 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.702 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.702 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:14.702 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745285+0000", 2024-09-21T10:21:14.702 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3041000000000003e-05 2024-09-21T10:21:14.702 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:14.702 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:14.702 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:14.702 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745326+0000", 2024-09-21T10:21:14.702 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1245000000000002e-05 2024-09-21T10:21:14.702 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.702 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.702 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.702 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:14.702 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:14.703 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:21:14.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:14.703 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:14.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:14.735 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:14.789 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-09-21T10:21:14.789 INFO:teuthology.orchestra.run.smithi107.stderr:0 2024-09-21T10:21:14.789 INFO:teuthology.orchestra.run.smithi107.stderr:1 2024-09-21T10:21:14.789 INFO:teuthology.orchestra.run.smithi107.stderr:2 2024-09-21T10:21:14.789 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2024-09-21T10:21:14.789 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2024-09-21T10:21:14.789 INFO:teuthology.orchestra.run.smithi107.stderr:bench [] [] [] [] 2024-09-21T10:21:14.789 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:21:14.789 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2024-09-21T10:21:14.789 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2024-09-21T10:21:14.789 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2024-09-21T10:21:14.789 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-09-21T10:21:14.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:14.843 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:14.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:14.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:14.944 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:14.975 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:14.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:14.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:15.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:15.106 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:15.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:15.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:15.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:15.242 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:15.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:15.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:15.344 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:15.383 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:15.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:15.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:15.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:15.514 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:15.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:15.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:15.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:15.648 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:15.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:15.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:15.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:15.752 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:21:15.778 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:15.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:15.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:15.859 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:21:15.859 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:21:15.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-21T10:21:15.880 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:15.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:15.932 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:15.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:16.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:16.074 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:16.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:16.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:16.175 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:16.185 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:21:16.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:21:16.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:16.226 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:16.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:16.327 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:16.369 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:16.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:16.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:16.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:16.518 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:16.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:16.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:16.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:16.649 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:16.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:16.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:16 smithi129 ceph-mon[88471]: pgmap v1364: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 3.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 874 op/s; 5193/174168 objects misplaced (2.982%); 9.2 MiB/s, 147 objects/s recovering 2024-09-21T10:21:16.750 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:16.792 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:16.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:16.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:16.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:16 smithi107 ceph-mon[102221]: pgmap v1364: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 3.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 874 op/s; 5193/174168 objects misplaced (2.982%); 9.2 MiB/s, 147 objects/s recovering 2024-09-21T10:21:16.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:16 smithi107 ceph-mon[105022]: pgmap v1364: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 3.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 874 op/s; 5193/174168 objects misplaced (2.982%); 9.2 MiB/s, 147 objects/s recovering 2024-09-21T10:21:16.893 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:16.914 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:16.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:16.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:17.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:17.041 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:17.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:17.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:17.143 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:17.172 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:17.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:17.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:17.274 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:17.310 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:17.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:17.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:17.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:17.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:17.440 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:17.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:17.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:17.579 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:17.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:17.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:17.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:17.716 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:17.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:17.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:17.817 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:17.855 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:17.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:17.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:17.957 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:17.987 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:17.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:17.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:18.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:18.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:18.132 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:18.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:18.233 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:18.268 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:18.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:18.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:18.370 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:18.407 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:18.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:18.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:18.469 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:18.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:18.560 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:18.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:18.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:18.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:18 smithi107 ceph-mon[102221]: pgmap v1365: 109 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 107 active+clean; 3.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 920 op/s; 4274/168792 objects misplaced (2.532%); 16 MiB/s, 255 objects/s recovering 2024-09-21T10:21:18.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:18 smithi107 ceph-mon[105022]: pgmap v1365: 109 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 107 active+clean; 3.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 920 op/s; 4274/168792 objects misplaced (2.532%); 16 MiB/s, 255 objects/s recovering 2024-09-21T10:21:18.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:18.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:18.712 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:18.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:18.766 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:18.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:18.838 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:21:18.838 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:21:18.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:18.854 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:18.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:18.873 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:21:18.873 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:21:18.907 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:21:18.907 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:21:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:18 smithi129 ceph-mon[88471]: pgmap v1365: 109 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 107 active+clean; 3.4 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 920 op/s; 4274/168792 objects misplaced (2.532%); 16 MiB/s, 255 objects/s recovering 2024-09-21T10:21:18.939 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:21:18.939 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:21:18.955 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:18.968 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:21:18.968 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:21:19.002 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:19.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:19.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:19.020 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:21:19.020 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:21:19.055 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:21:19.056 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:21:19.092 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:21:19.092 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:21:19.104 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:19.134 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:19.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:19.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:19.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:19.270 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:19.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:19.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:19.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:19.411 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:19.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:19.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:19.513 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:19.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:19.556 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:19.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:19.656 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:19 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/98129820' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:21:19.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:19.687 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:19.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:19.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:19.822 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:19.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:19.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:19 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/98129820' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:21:19.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:19 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/98129820' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:21:19.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:19.952 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:19.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:19.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:20.053 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:20.096 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:20.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:20.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:20.198 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:20.235 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:20.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:20.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:20.337 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:20.378 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:20.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:20.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:20.469 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:21:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:21:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:21:20.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:20.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:20.516 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:20.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:20.617 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:20.655 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:20.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:20.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:20.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:20.785 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:20.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:20.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:20.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:20 smithi107 ceph-mon[102221]: pgmap v1366: 109 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 107 active+clean; 3.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 948 op/s; 4274/164649 objects misplaced (2.596%); 12 MiB/s, 185 objects/s recovering 2024-09-21T10:21:20.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:20 smithi107 ceph-mon[105022]: pgmap v1366: 109 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 107 active+clean; 3.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 948 op/s; 4274/164649 objects misplaced (2.596%); 12 MiB/s, 185 objects/s recovering 2024-09-21T10:21:20.887 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:20 smithi129 ceph-mon[88471]: pgmap v1366: 109 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 107 active+clean; 3.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 948 op/s; 4274/164649 objects misplaced (2.596%); 12 MiB/s, 185 objects/s recovering 2024-09-21T10:21:20.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:20.928 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:20.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:21.029 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:21.071 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:21.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:21.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:21.203 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:21.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:21.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:21.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:21.334 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:21.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:21.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:21.436 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:21.462 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:21.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:21.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:21.564 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:21.586 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:21.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:21.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:21.688 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:21.721 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:21.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:21.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:21.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:21.858 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:21.858 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:21:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:21:21] "GET /metrics HTTP/1.1" 200 36808 "" "Prometheus/2.51.0" 2024-09-21T10:21:21.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:21.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:21.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:21.991 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:21.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:21.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:22.093 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:22.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:22.130 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:22.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:22.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:22.261 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:22.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:22.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:22.362 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:22.401 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:22.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:22.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:22.417 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:21:22.417 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:21:22.503 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:22.541 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:22.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:22.616 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-09-21T10:21:22.617 INFO:teuthology.orchestra.run.smithi107.stderr:0 2024-09-21T10:21:22.617 INFO:teuthology.orchestra.run.smithi107.stderr:1 2024-09-21T10:21:22.617 INFO:teuthology.orchestra.run.smithi107.stderr:2 2024-09-21T10:21:22.617 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2024-09-21T10:21:22.617 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2024-09-21T10:21:22.617 INFO:teuthology.orchestra.run.smithi107.stderr:bench [] [] [] [] 2024-09-21T10:21:22.617 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:21:22.617 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2024-09-21T10:21:22.617 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2024-09-21T10:21:22.617 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2024-09-21T10:21:22.617 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-09-21T10:21:22.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:22.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:22 smithi129 ceph-mon[88471]: pgmap v1367: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 3.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 944 op/s; 3973/161073 objects misplaced (2.467%); 9.9 MiB/s, 159 objects/s recovering 2024-09-21T10:21:22.672 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:22.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:22.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:22.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:22.804 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:22.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:22.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:22.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:22 smithi107 ceph-mon[102221]: pgmap v1367: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 3.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 944 op/s; 3973/161073 objects misplaced (2.467%); 9.9 MiB/s, 159 objects/s recovering 2024-09-21T10:21:22.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:22 smithi107 ceph-mon[105022]: pgmap v1367: 109 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 107 active+clean; 3.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 944 op/s; 3973/161073 objects misplaced (2.467%); 9.9 MiB/s, 159 objects/s recovering 2024-09-21T10:21:22.907 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:22.946 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:22.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:22.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:23.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:23.075 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:23.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:23.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:23.177 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:23.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:23.214 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:23.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:23.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:23.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:23.346 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:23.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:23.379 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:21:23.380 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:21:23.447 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:23.492 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:23.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:23.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:23.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:21:23.593 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:23.604 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:21:23.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:23.632 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:23.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:23.733 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:23.780 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:23.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:23.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:23.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:23.934 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:23.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:23.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:24.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:24.077 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:24.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:24.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:24.120 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:21:24.178 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:24.230 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:24.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:24.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:24.332 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:24.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:24.379 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:24.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:24.462 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:24.481 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:24.520 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:24.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:24.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:24.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:24.666 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:24.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:24.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:24.768 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:24.790 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:24.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:24.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:24.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:24 smithi107 ceph-mon[102221]: pgmap v1368: 109 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 107 active+clean; 3.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 877 op/s; 3418/150753 objects misplaced (2.267%); 9.2 MiB/s, 146 objects/s recovering 2024-09-21T10:21:24.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:24 smithi107 ceph-mon[105022]: pgmap v1368: 109 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 107 active+clean; 3.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 877 op/s; 3418/150753 objects misplaced (2.267%); 9.2 MiB/s, 146 objects/s recovering 2024-09-21T10:21:24.893 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:24.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:24 smithi129 ceph-mon[88471]: pgmap v1368: 109 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 107 active+clean; 3.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 877 op/s; 3418/150753 objects misplaced (2.267%); 9.2 MiB/s, 146 objects/s recovering 2024-09-21T10:21:24.921 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:24.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:24.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:25.023 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:25.044 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:25.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:25.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:25.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:25.167 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:25.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:25.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:25.269 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:25.292 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:25.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:25.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:25.394 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:25.417 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:25.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:25.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:25.518 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:25.539 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:25.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:25.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:25.599 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:21:25.641 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:25.669 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:25.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:25.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:25.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:25.772 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:25 smithi129 ceph-mon[88471]: osdmap e393: 8 total, 8 up, 7 in 2024-09-21T10:21:25.772 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:25 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2436841224' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:21:25.772 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:25 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:21:25.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:25.808 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:25.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:25 smithi107 ceph-mon[102221]: osdmap e393: 8 total, 8 up, 7 in 2024-09-21T10:21:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:25 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2436841224' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:21:25.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:25 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:21:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:25 smithi107 ceph-mon[105022]: osdmap e393: 8 total, 8 up, 7 in 2024-09-21T10:21:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:25 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2436841224' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:21:25.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:25 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:21:25.909 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:25.937 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:25.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:25.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:26.039 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:26.080 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:26.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:26.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:26.100 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:21:26.100 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:21:26.182 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:26.222 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:26.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:26.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:26.324 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:26.359 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:26.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:26.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:26.461 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:26.501 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:26.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:26.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:26.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:26.654 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:26.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:26.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:26.755 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:26.785 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:26 smithi129 ceph-mon[88471]: pgmap v1370: 109 pgs: 2 active+remapped+backfilling, 107 active+clean; 3.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 933 op/s; 2911/146151 objects misplaced (1.992%); 9.7 MiB/s, 154 objects/s recovering 2024-09-21T10:21:26.785 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:26 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:21:26.785 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:26 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:21:26.786 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:26 smithi129 ceph-mon[88471]: osdmap e394: 8 total, 8 up, 7 in 2024-09-21T10:21:26.786 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:21:26.786 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:26.788 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:26.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:26.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:26.828 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:21:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:26 smithi107 ceph-mon[102221]: pgmap v1370: 109 pgs: 2 active+remapped+backfilling, 107 active+clean; 3.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 933 op/s; 2911/146151 objects misplaced (1.992%); 9.7 MiB/s, 154 objects/s recovering 2024-09-21T10:21:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:26 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:21:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:26 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:21:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:26 smithi107 ceph-mon[102221]: osdmap e394: 8 total, 8 up, 7 in 2024-09-21T10:21:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:21:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:26 smithi107 ceph-mon[105022]: pgmap v1370: 109 pgs: 2 active+remapped+backfilling, 107 active+clean; 3.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 933 op/s; 2911/146151 objects misplaced (1.992%); 9.7 MiB/s, 154 objects/s recovering 2024-09-21T10:21:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:26 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:21:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:26 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:21:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:26 smithi107 ceph-mon[105022]: osdmap e394: 8 total, 8 up, 7 in 2024-09-21T10:21:26.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:21:26.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:26.889 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:26.917 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:26.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:26.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:27.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:27.040 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:27.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:27.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:27.141 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:27.178 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:27.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:27.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:27.218 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:21:27.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:27.316 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:27.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:27.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:27.418 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:27.456 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:27.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:27.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:27.558 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:27.583 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:27.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:27.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:27.685 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:27.723 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:27.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:27.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:27.824 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:27.854 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:27.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:27.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:27.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:27.992 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:27.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:27.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:28.094 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:28.128 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:28.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:28.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:28.229 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:28.268 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:28.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:28.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:28.370 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:28.380 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 4] out_osds: [6] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:21:28.380 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:21:28.381 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-21T10:21:28.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:21:28.404 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:28.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:28.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:28.506 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:28.542 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:28.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:28.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:28.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:28.689 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:28.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:28.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:28.716 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:28 smithi107 ceph-mon[102221]: pgmap v1372: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 994 op/s; 2769/140778 objects misplaced (1.967%); 6.4 MiB/s, 101 objects/s recovering 2024-09-21T10:21:28.716 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:28 smithi107 ceph-mon[105022]: pgmap v1372: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 994 op/s; 2769/140778 objects misplaced (1.967%); 6.4 MiB/s, 101 objects/s recovering 2024-09-21T10:21:28.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:28.820 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:28.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:28.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:28.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:28 smithi129 ceph-mon[88471]: pgmap v1372: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 994 op/s; 2769/140778 objects misplaced (1.967%); 6.4 MiB/s, 101 objects/s recovering 2024-09-21T10:21:28.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:28.953 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:28.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:29.055 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:29.086 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:29.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:29.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:29.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:29.220 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:29.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:29.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:29.322 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:29.367 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:29.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:29.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:29.468 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:29.509 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:29.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:29.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:29.578 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:21:29.578 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:21:29.610 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:29.617 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:21:29.617 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:21:29.620 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:29.636 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:29.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:29.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:29.652 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:21:29.652 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:21:29.685 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:21:29.686 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:21:29.732 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:21:29.733 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:21:29.737 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:29.757 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:21:29.757 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:21:29.765 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:29.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:29.792 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:21:29.792 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:21:29.826 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:29 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2541747925' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:21:29.826 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:21:29.826 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:21:29.827 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:29 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2541747925' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:21:29.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:29.905 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:29.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:29.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:29.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:29 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2541747925' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:21:30.007 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:30.051 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:30.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:30.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:30.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:30.187 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:30.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:30.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:30.289 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:30.333 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:30.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:30.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:30.434 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:30.465 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:30.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:30.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:30.470 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:21:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:21:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:21:30.527 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:21:30.527 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 20, 2024-09-21T10:21:30.527 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 600, 2024-09-21T10:21:30.527 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:21:30.527 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.527 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36822.0:46083 12.d 12:b93ee741:::benchmark_data_smithi107_250110_object46082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-09-21T10:21:30.527 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:14:34.310117+0000", 2024-09-21T10:21:30.527 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 416.212106796, 2024-09-21T10:21:30.527 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.34146841300000003, 2024-09-21T10:21:30.528 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:30.528 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:30.528 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:30.528 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36822", 2024-09-21T10:21:30.528 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:21:30.528 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 46083 2024-09-21T10:21:30.528 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.528 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:30.528 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.528 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:30.528 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.310117+0000", 2024-09-21T10:21:30.528 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.528 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.528 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.528 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:30.529 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.310117+0000", 2024-09-21T10:21:30.529 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.529 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.529 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.529 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:30.529 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.310116+0000", 2024-09-21T10:21:30.529 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:21:30.529 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.529 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.529 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:30.529 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.310145+0000", 2024-09-21T10:21:30.529 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.8971000000000001e-05 2024-09-21T10:21:30.529 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.529 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.530 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:30.530 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.310158+0000", 2024-09-21T10:21:30.530 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3473e-05 2024-09-21T10:21:30.530 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.530 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.530 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:30.530 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.310163+0000", 2024-09-21T10:21:30.530 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8189999999999999e-06 2024-09-21T10:21:30.530 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.531 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.531 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.531 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.310578+0000", 2024-09-21T10:21:30.531 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000415303 2024-09-21T10:21:30.531 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.531 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.531 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:30.531 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.310719+0000", 2024-09-21T10:21:30.531 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014021699999999999 2024-09-21T10:21:30.531 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.531 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.531 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:21:30.531 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.310885+0000", 2024-09-21T10:21:30.532 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00016659300000000001 2024-09-21T10:21:30.532 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.532 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.532 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.532 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.315059+0000", 2024-09-21T10:21:30.532 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0041734210000000001 2024-09-21T10:21:30.532 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.532 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.532 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:30.532 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.315072+0000", 2024-09-21T10:21:30.532 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3730000000000001e-05 2024-09-21T10:21:30.532 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.532 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.532 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.533 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.651518+0000", 2024-09-21T10:21:30.533 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.33644594999999999 2024-09-21T10:21:30.533 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.533 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.533 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:30.533 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.651558+0000", 2024-09-21T10:21:30.533 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.9121e-05 2024-09-21T10:21:30.533 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.533 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.533 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:30.533 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.651586+0000", 2024-09-21T10:21:30.533 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.8201e-05 2024-09-21T10:21:30.533 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.533 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.534 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.534 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.534 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.534 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36822.0:46123 12.d 12:b01c0a2a:::benchmark_data_smithi107_250110_object46122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-09-21T10:21:30.534 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:14:34.336717+0000", 2024-09-21T10:21:30.534 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 416.18550704799998, 2024-09-21T10:21:30.534 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.320637226, 2024-09-21T10:21:30.534 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:30.534 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:30.534 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:30.534 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36822", 2024-09-21T10:21:30.534 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:21:30.534 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 46123 2024-09-21T10:21:30.534 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.535 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:30.535 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.535 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:30.535 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.336717+0000", 2024-09-21T10:21:30.535 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.535 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.535 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.535 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:30.535 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.336717+0000", 2024-09-21T10:21:30.535 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.535 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.535 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.535 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:30.536 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.336716+0000", 2024-09-21T10:21:30.536 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:21:30.536 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.536 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.536 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:30.536 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.336759+0000", 2024-09-21T10:21:30.536 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3130999999999998e-05 2024-09-21T10:21:30.536 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.536 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.536 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:30.536 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.336771+0000", 2024-09-21T10:21:30.536 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1698e-05 2024-09-21T10:21:30.536 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.536 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.536 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:30.537 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.336775+0000", 2024-09-21T10:21:30.537 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8570000000000003e-06 2024-09-21T10:21:30.537 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.537 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.537 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.537 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.336812+0000", 2024-09-21T10:21:30.537 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6489999999999998e-05 2024-09-21T10:21:30.537 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.537 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.537 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:30.537 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.336894+0000", 2024-09-21T10:21:30.537 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.1966999999999996e-05 2024-09-21T10:21:30.537 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.537 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.538 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:21:30.538 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.336977+0000", 2024-09-21T10:21:30.538 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.3013e-05 2024-09-21T10:21:30.538 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.538 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.538 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.538 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.339135+0000", 2024-09-21T10:21:30.538 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0021584960000000002 2024-09-21T10:21:30.538 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.538 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.538 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:30.538 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.339329+0000", 2024-09-21T10:21:30.539 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00019314700000000001 2024-09-21T10:21:30.539 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.539 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.539 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.539 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.657300+0000", 2024-09-21T10:21:30.539 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.31797129099999999 2024-09-21T10:21:30.539 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.539 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.539 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:30.539 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.657332+0000", 2024-09-21T10:21:30.539 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2005999999999998e-05 2024-09-21T10:21:30.539 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.539 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.539 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:30.540 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:34.657354+0000", 2024-09-21T10:21:30.540 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.2407000000000001e-05 2024-09-21T10:21:30.540 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.540 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.540 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.540 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.540 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.540 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:26945 13.0 13:0d6e0250:::benchmark_data_smithi107_278655_object26944:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:21:30.540 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:31.497184+0000", 2024-09-21T10:21:30.540 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 179.02503960799999, 2024-09-21T10:21:30.540 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.29837365799999999, 2024-09-21T10:21:30.541 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:30.541 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:30.541 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:30.541 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:30.541 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:30.541 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 26945 2024-09-21T10:21:30.542 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.542 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:30.542 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.542 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:30.542 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.497184+0000", 2024-09-21T10:21:30.542 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.542 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.542 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.542 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:30.542 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.497184+0000", 2024-09-21T10:21:30.542 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.543 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.543 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.543 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:30.543 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.497183+0000", 2024-09-21T10:21:30.543 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:21:30.543 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.543 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.543 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:30.543 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.497221+0000", 2024-09-21T10:21:30.543 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7901000000000002e-05 2024-09-21T10:21:30.543 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.543 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.543 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:30.543 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.497235+0000", 2024-09-21T10:21:30.543 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3508e-05 2024-09-21T10:21:30.544 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.544 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.544 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:30.544 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.497241+0000", 2024-09-21T10:21:30.544 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1040000000000003e-06 2024-09-21T10:21:30.544 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.544 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.544 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.544 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.497290+0000", 2024-09-21T10:21:30.544 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8912999999999997e-05 2024-09-21T10:21:30.544 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.544 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.544 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:30.544 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.497385+0000", 2024-09-21T10:21:30.545 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.5767999999999995e-05 2024-09-21T10:21:30.545 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.545 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.545 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:21:30.545 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.497545+0000", 2024-09-21T10:21:30.545 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015969700000000001 2024-09-21T10:21:30.545 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.545 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.545 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:30.545 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498780+0000", 2024-09-21T10:21:30.545 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0012350200000000001 2024-09-21T10:21:30.545 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.545 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.545 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.546 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.499758+0000", 2024-09-21T10:21:30.546 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00097847899999999998 2024-09-21T10:21:30.546 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.546 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.546 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.546 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.795468+0000", 2024-09-21T10:21:30.546 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.29570952499999997 2024-09-21T10:21:30.546 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.546 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.546 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:30.546 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.795517+0000", 2024-09-21T10:21:30.546 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9332000000000001e-05 2024-09-21T10:21:30.547 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.547 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.547 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:30.547 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.795558+0000", 2024-09-21T10:21:30.547 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0800000000000002e-05 2024-09-21T10:21:30.547 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.547 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.547 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.547 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.547 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.547 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:26977 13.0 13:080daddd:::benchmark_data_smithi107_278655_object26976:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:21:30.547 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:31.513658+0000", 2024-09-21T10:21:30.547 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 179.00856639099999, 2024-09-21T10:21:30.548 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.282703076, 2024-09-21T10:21:30.548 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:30.548 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:30.548 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:30.548 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:30.548 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:30.548 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 26977 2024-09-21T10:21:30.548 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.548 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:30.548 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.548 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:30.548 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.513658+0000", 2024-09-21T10:21:30.548 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.549 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.549 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.549 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:30.549 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.513658+0000", 2024-09-21T10:21:30.549 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.549 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.549 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.549 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:30.549 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.513656+0000", 2024-09-21T10:21:30.549 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:21:30.549 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.549 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.549 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:30.549 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.513692+0000", 2024-09-21T10:21:30.550 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6168999999999999e-05 2024-09-21T10:21:30.550 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.550 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.550 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:30.550 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.513706+0000", 2024-09-21T10:21:30.550 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3378e-05 2024-09-21T10:21:30.550 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.550 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.550 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:30.550 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.513712+0000", 2024-09-21T10:21:30.550 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0719999999999996e-06 2024-09-21T10:21:30.550 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.550 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.550 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.551 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.513762+0000", 2024-09-21T10:21:30.551 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0167999999999998e-05 2024-09-21T10:21:30.551 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.551 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.551 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:30.551 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.513876+0000", 2024-09-21T10:21:30.551 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000113838 2024-09-21T10:21:30.551 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.551 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.551 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:21:30.551 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.514002+0000", 2024-09-21T10:21:30.551 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012649199999999999 2024-09-21T10:21:30.551 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.551 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.552 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:30.552 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.515246+0000", 2024-09-21T10:21:30.552 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001243798 2024-09-21T10:21:30.552 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.552 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.552 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.552 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.515347+0000", 2024-09-21T10:21:30.552 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010134899999999999 2024-09-21T10:21:30.552 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.552 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.552 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.552 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.796275+0000", 2024-09-21T10:21:30.552 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28092745699999999 2024-09-21T10:21:30.553 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.553 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.553 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:30.553 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.796318+0000", 2024-09-21T10:21:30.553 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3501999999999998e-05 2024-09-21T10:21:30.553 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.553 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.553 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:30.553 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.796361+0000", 2024-09-21T10:21:30.553 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2580000000000002e-05 2024-09-21T10:21:30.553 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.553 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.553 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.553 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.553 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.554 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:72764 13.0 13:095f2978:::benchmark_data_smithi107_278655_object72763:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:21:30.554 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.480369+0000", 2024-09-21T10:21:30.554 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 150.04185558, 2024-09-21T10:21:30.554 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.27071356299999999, 2024-09-21T10:21:30.554 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:30.554 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:30.554 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:30.554 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:30.554 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:30.554 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 72764 2024-09-21T10:21:30.554 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.554 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:30.554 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.554 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:30.555 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.480369+0000", 2024-09-21T10:21:30.555 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.555 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.555 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.555 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:30.555 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.480369+0000", 2024-09-21T10:21:30.555 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.555 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.555 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.555 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:30.555 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.480367+0000", 2024-09-21T10:21:30.555 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:21:30.555 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.555 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.555 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:30.556 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.480412+0000", 2024-09-21T10:21:30.556 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4873000000000002e-05 2024-09-21T10:21:30.556 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.556 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.556 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:30.556 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.480426+0000", 2024-09-21T10:21:30.556 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4146e-05 2024-09-21T10:21:30.556 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.556 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.556 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:30.556 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.480432+0000", 2024-09-21T10:21:30.556 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5099999999999998e-06 2024-09-21T10:21:30.556 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.556 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.556 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.557 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.480504+0000", 2024-09-21T10:21:30.557 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2225999999999996e-05 2024-09-21T10:21:30.557 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.557 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.557 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:30.557 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.480618+0000", 2024-09-21T10:21:30.557 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000113651 2024-09-21T10:21:30.557 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.557 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.557 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:21:30.557 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.480747+0000", 2024-09-21T10:21:30.557 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000129059 2024-09-21T10:21:30.557 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.557 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.557 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.558 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.483671+0000", 2024-09-21T10:21:30.558 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0029244420000000002 2024-09-21T10:21:30.558 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.558 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.558 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:30.558 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.484638+0000", 2024-09-21T10:21:30.558 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00096638300000000002 2024-09-21T10:21:30.558 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.558 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.558 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.558 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.750988+0000", 2024-09-21T10:21:30.558 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.26635043600000002 2024-09-21T10:21:30.558 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.558 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.558 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:30.559 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.751039+0000", 2024-09-21T10:21:30.559 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1309e-05 2024-09-21T10:21:30.559 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.559 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.559 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:30.559 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.751082+0000", 2024-09-21T10:21:30.559 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2599999999999999e-05 2024-09-21T10:21:30.559 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.559 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.559 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.559 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.559 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.559 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:134448 13.7 13:e615f15f:::benchmark_data_smithi107_278655_object6541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:21:30.559 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:58.699336+0000", 2024-09-21T10:21:30.560 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 91.822888321999997, 2024-09-21T10:21:30.560 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1137340400000002, 2024-09-21T10:21:30.560 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:30.560 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:21:30.560 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:30.560 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:30.560 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:30.560 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 134448 2024-09-21T10:21:30.560 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.560 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:30.560 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.560 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:30.560 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699336+0000", 2024-09-21T10:21:30.560 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.560 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.561 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.561 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:30.561 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699336+0000", 2024-09-21T10:21:30.561 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.561 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.561 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.561 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:30.561 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699335+0000", 2024-09-21T10:21:30.561 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:21:30.561 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.561 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.561 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:30.561 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699340+0000", 2024-09-21T10:21:30.561 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4450000000000004e-06 2024-09-21T10:21:30.561 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.562 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.562 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:30.562 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699355+0000", 2024-09-21T10:21:30.562 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5224e-05 2024-09-21T10:21:30.562 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.562 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.562 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:30.562 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699361+0000", 2024-09-21T10:21:30.562 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8379999999999998e-06 2024-09-21T10:21:30.562 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.562 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.562 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.562 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.701910+0000", 2024-09-21T10:21:30.562 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0025482109999999999 2024-09-21T10:21:30.562 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.563 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.563 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:30.563 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.702024+0000", 2024-09-21T10:21:30.563 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000114465 2024-09-21T10:21:30.563 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.563 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.563 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:21:30.563 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.702137+0000", 2024-09-21T10:21:30.563 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011343199999999999 2024-09-21T10:21:30.563 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.563 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.563 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:30.563 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.710167+0000", 2024-09-21T10:21:30.564 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0080300509999999999 2024-09-21T10:21:30.564 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.564 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.564 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.564 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.715118+0000", 2024-09-21T10:21:30.564 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0049505210000000003 2024-09-21T10:21:30.564 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.564 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.564 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.564 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.737774+0000", 2024-09-21T10:21:30.564 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.022655709 2024-09-21T10:21:30.564 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.564 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.564 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:21:30.564 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.737782+0000", 2024-09-21T10:21:30.564 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.4109999999999996e-06 2024-09-21T10:21:30.565 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.565 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.565 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.565 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601581+0000", 2024-09-21T10:21:30.565 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.86379862200000002 2024-09-21T10:21:30.565 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.565 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.565 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:21:30.565 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601593+0000", 2024-09-21T10:21:30.565 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2381000000000001e-05 2024-09-21T10:21:30.565 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.565 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.565 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.565 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.813002+0000", 2024-09-21T10:21:30.565 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2114087659999999 2024-09-21T10:21:30.565 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.566 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.566 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:30.566 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.813070+0000", 2024-09-21T10:21:30.566 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7929000000000006e-05 2024-09-21T10:21:30.566 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.566 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.566 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.566 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.566 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.566 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:134449 13.7 13:ec92a00a:::benchmark_data_smithi107_278655_object6542:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:21:30.566 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:58.699371+0000", 2024-09-21T10:21:30.566 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 91.822853318, 2024-09-21T10:21:30.566 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1152291590000001, 2024-09-21T10:21:30.566 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:30.566 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:21:30.567 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:30.567 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:30.567 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:30.567 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 134449 2024-09-21T10:21:30.567 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.567 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:30.567 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.567 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:30.567 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699371+0000", 2024-09-21T10:21:30.567 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.567 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.567 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.567 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:30.567 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699371+0000", 2024-09-21T10:21:30.567 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.567 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.567 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.568 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:30.568 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699370+0000", 2024-09-21T10:21:30.568 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:21:30.568 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.568 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.568 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:30.568 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:30.569 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699373+0000", 2024-09-21T10:21:30.569 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.3379999999999999e-06 2024-09-21T10:21:30.569 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.569 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.569 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:30.569 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699378+0000", 2024-09-21T10:21:30.569 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4750000000000001e-06 2024-09-21T10:21:30.569 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.569 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.570 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:30.570 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699381+0000", 2024-09-21T10:21:30.570 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2200000000000001e-06 2024-09-21T10:21:30.570 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.570 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.570 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.570 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.705719+0000", 2024-09-21T10:21:30.570 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0063375430000000002 2024-09-21T10:21:30.570 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.570 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.570 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:30.570 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.705831+0000", 2024-09-21T10:21:30.570 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000111987 2024-09-21T10:21:30.570 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.570 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.571 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:21:30.571 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.705945+0000", 2024-09-21T10:21:30.571 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011391999999999999 2024-09-21T10:21:30.571 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.571 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.571 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.571 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.715146+0000", 2024-09-21T10:21:30.571 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0092007930000000005 2024-09-21T10:21:30.571 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.571 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.571 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:30.571 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.725273+0000", 2024-09-21T10:21:30.571 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.010127196999999999 2024-09-21T10:21:30.571 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.571 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.572 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.572 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.737806+0000", 2024-09-21T10:21:30.572 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.012532896 2024-09-21T10:21:30.572 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.572 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.572 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:21:30.572 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.737812+0000", 2024-09-21T10:21:30.572 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9540000000000001e-06 2024-09-21T10:21:30.572 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.572 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.572 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.572 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601625+0000", 2024-09-21T10:21:30.572 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.86381372599999995 2024-09-21T10:21:30.572 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.572 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.572 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:21:30.573 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601632+0000", 2024-09-21T10:21:30.573 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6529999999999997e-06 2024-09-21T10:21:30.573 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.573 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.573 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.573 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.814527+0000", 2024-09-21T10:21:30.573 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.21289533 2024-09-21T10:21:30.573 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.573 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.573 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:30.573 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.814600+0000", 2024-09-21T10:21:30.573 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.25e-05 2024-09-21T10:21:30.573 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.573 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.573 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.573 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.574 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.574 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:134451 13.7 13:e3630b82:::benchmark_data_smithi107_278655_object6544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:21:30.574 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:58.699385+0000", 2024-09-21T10:21:30.574 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 91.822839078000001, 2024-09-21T10:21:30.574 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1294433669999999, 2024-09-21T10:21:30.574 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:30.574 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:21:30.574 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:30.574 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:30.574 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:30.574 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 134451 2024-09-21T10:21:30.574 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.574 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:30.574 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.574 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:30.574 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699385+0000", 2024-09-21T10:21:30.575 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.575 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.575 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.575 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:30.575 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699385+0000", 2024-09-21T10:21:30.575 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.575 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.575 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.575 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:30.575 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699385+0000", 2024-09-21T10:21:30.575 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:21:30.575 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.575 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.575 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:30.576 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699387+0000", 2024-09-21T10:21:30.576 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1859999999999999e-06 2024-09-21T10:21:30.576 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.576 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.576 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:30.576 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699392+0000", 2024-09-21T10:21:30.576 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7820000000000001e-06 2024-09-21T10:21:30.576 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.576 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.576 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:30.576 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699395+0000", 2024-09-21T10:21:30.576 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0479999999999999e-06 2024-09-21T10:21:30.576 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.576 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.576 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.576 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.710241+0000", 2024-09-21T10:21:30.576 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.010846369999999999 2024-09-21T10:21:30.577 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.577 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.577 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:30.577 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.710514+0000", 2024-09-21T10:21:30.577 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00027252599999999999 2024-09-21T10:21:30.577 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.577 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.577 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:21:30.577 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.710874+0000", 2024-09-21T10:21:30.577 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00036023799999999998 2024-09-21T10:21:30.577 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.577 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.577 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:30.577 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.725278+0000", 2024-09-21T10:21:30.577 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.014403978 2024-09-21T10:21:30.578 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.578 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.578 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.578 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.737834+0000", 2024-09-21T10:21:30.578 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.012555851999999999 2024-09-21T10:21:30.578 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.578 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.578 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:21:30.578 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.737840+0000", 2024-09-21T10:21:30.578 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.421e-06 2024-09-21T10:21:30.578 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.578 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.578 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.578 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601657+0000", 2024-09-21T10:21:30.579 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.86381711699999997 2024-09-21T10:21:30.579 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.579 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.579 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:21:30.579 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601664+0000", 2024-09-21T10:21:30.579 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7390000000000002e-06 2024-09-21T10:21:30.579 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.579 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.579 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.579 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.826620+0000", 2024-09-21T10:21:30.579 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2249555560000001 2024-09-21T10:21:30.579 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.579 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.579 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:30.579 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.828828+0000", 2024-09-21T10:21:30.580 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0022088670000000002 2024-09-21T10:21:30.580 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.580 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.580 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.580 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.580 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.580 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:134459 13.f 13:f700217e:::benchmark_data_smithi107_278655_object6552:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:21:30.580 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:58.722388+0000", 2024-09-21T10:21:30.580 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 91.799835764999997, 2024-09-21T10:21:30.580 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1032898339999999, 2024-09-21T10:21:30.580 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:30.580 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:21:30.580 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:30.580 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:30.581 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:30.581 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 134459 2024-09-21T10:21:30.581 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.581 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:30.581 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.581 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:30.581 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722388+0000", 2024-09-21T10:21:30.581 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.581 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.581 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.581 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:30.581 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722388+0000", 2024-09-21T10:21:30.581 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.581 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.582 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.582 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:30.582 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722387+0000", 2024-09-21T10:21:30.582 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:21:30.582 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.582 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.582 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:30.582 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722393+0000", 2024-09-21T10:21:30.582 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7649999999999999e-06 2024-09-21T10:21:30.582 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.582 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.582 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:30.582 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722402+0000", 2024-09-21T10:21:30.582 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.2590000000000006e-06 2024-09-21T10:21:30.582 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.583 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.583 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:30.583 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722414+0000", 2024-09-21T10:21:30.583 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1579e-05 2024-09-21T10:21:30.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.583 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.583 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.583 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.738001+0000", 2024-09-21T10:21:30.583 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.015586598 2024-09-21T10:21:30.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.583 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.583 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:21:30.583 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.738008+0000", 2024-09-21T10:21:30.583 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2590000000000003e-06 2024-09-21T10:21:30.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.584 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.584 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.584 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601716+0000", 2024-09-21T10:21:30.584 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.86370839799999999 2024-09-21T10:21:30.584 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.584 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.584 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:21:30.584 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601723+0000", 2024-09-21T10:21:30.584 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8569999999999997e-06 2024-09-21T10:21:30.584 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.584 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.584 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.584 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.825551+0000", 2024-09-21T10:21:30.584 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.223827674 2024-09-21T10:21:30.584 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.584 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.585 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:30.585 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.825678+0000", 2024-09-21T10:21:30.585 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000127451 2024-09-21T10:21:30.585 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.585 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.585 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.585 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.585 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.585 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:134462 13.f 13:f29b039a:::benchmark_data_smithi107_278655_object6555:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:21:30.585 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:58.722420+0000", 2024-09-21T10:21:30.585 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 91.799804382000005, 2024-09-21T10:21:30.585 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1428640240000001, 2024-09-21T10:21:30.585 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:30.585 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:21:30.585 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:30.585 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:30.586 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:30.586 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 134462 2024-09-21T10:21:30.586 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.586 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:30.586 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.586 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:30.586 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722420+0000", 2024-09-21T10:21:30.586 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.586 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.586 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.586 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:30.586 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722420+0000", 2024-09-21T10:21:30.586 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.586 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.587 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.587 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:30.587 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722419+0000", 2024-09-21T10:21:30.587 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:21:30.587 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.587 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.587 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:30.587 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722422+0000", 2024-09-21T10:21:30.587 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0989999999999999e-06 2024-09-21T10:21:30.587 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.587 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.587 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:30.587 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722429+0000", 2024-09-21T10:21:30.587 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4989999999999999e-06 2024-09-21T10:21:30.588 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.588 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.588 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:30.588 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722433+0000", 2024-09-21T10:21:30.588 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.9060000000000004e-06 2024-09-21T10:21:30.588 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.588 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.588 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.588 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.738055+0000", 2024-09-21T10:21:30.588 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.015622761000000001 2024-09-21T10:21:30.588 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.588 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.588 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:21:30.588 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.738061+0000", 2024-09-21T10:21:30.588 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7919999999999998e-06 2024-09-21T10:21:30.588 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601742+0000", 2024-09-21T10:21:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.86368112100000005 2024-09-21T10:21:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:21:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601748+0000", 2024-09-21T10:21:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0530000000000003e-06 2024-09-21T10:21:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.826378+0000", 2024-09-21T10:21:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2246290879999999 2024-09-21T10:21:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.589 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.590 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for missing object", 2024-09-21T10:21:30.590 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.826477+0000", 2024-09-21T10:21:30.590 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.9401999999999995e-05 2024-09-21T10:21:30.590 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.590 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.590 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.590 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.865163+0000", 2024-09-21T10:21:30.590 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.038685936999999997 2024-09-21T10:21:30.590 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.590 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.590 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:30.590 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.865284+0000", 2024-09-21T10:21:30.590 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000120866 2024-09-21T10:21:30.590 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.590 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:134465 13.7 13:e628427d:::benchmark_data_smithi107_278655_object6558:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:21:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:58.723761+0000", 2024-09-21T10:21:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 91.798463460999997, 2024-09-21T10:21:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.124022085, 2024-09-21T10:21:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 134465 2024-09-21T10:21:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:30.591 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.723761+0000", 2024-09-21T10:21:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.723761+0000", 2024-09-21T10:21:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.723760+0000", 2024-09-21T10:21:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:21:30.592 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.723764+0000", 2024-09-21T10:21:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3200000000000001e-06 2024-09-21T10:21:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.723773+0000", 2024-09-21T10:21:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6489999999999997e-06 2024-09-21T10:21:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.723778+0000", 2024-09-21T10:21:30.593 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6239999999999999e-06 2024-09-21T10:21:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.738027+0000", 2024-09-21T10:21:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.01424893 2024-09-21T10:21:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:21:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.738034+0000", 2024-09-21T10:21:30.594 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2500000000000003e-06 2024-09-21T10:21:30.595 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.595 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.595 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.595 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601687+0000", 2024-09-21T10:21:30.595 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.86365340599999996 2024-09-21T10:21:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:21:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601694+0000", 2024-09-21T10:21:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.579e-06 2024-09-21T10:21:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.828858+0000", 2024-09-21T10:21:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.227164178 2024-09-21T10:21:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.828974+0000", 2024-09-21T10:21:30.596 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011628700000000001 2024-09-21T10:21:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 4,5,6", 2024-09-21T10:21:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.829556+0000", 2024-09-21T10:21:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00058166700000000001 2024-09-21T10:21:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.843383+0000", 2024-09-21T10:21:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.013827694 2024-09-21T10:21:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.847499+0000", 2024-09-21T10:21:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0041160839999999999 2024-09-21T10:21:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.597 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.847530+0000", 2024-09-21T10:21:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0508999999999999e-05 2024-09-21T10:21:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.847556+0000", 2024-09-21T10:21:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.5749999999999999e-05 2024-09-21T10:21:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.847660+0000", 2024-09-21T10:21:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000104048 2024-09-21T10:21:30.598 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.599 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.599 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:30.599 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.847783+0000", 2024-09-21T10:21:30.599 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000122922 2024-09-21T10:21:30.599 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.599 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.599 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.599 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.599 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.599 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:134466 13.f 13:fdbed5d6:::benchmark_data_smithi107_278655_object6559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:21:30.599 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:58.735102+0000", 2024-09-21T10:21:30.599 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 91.787122143999994, 2024-09-21T10:21:30.599 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1198004730000002, 2024-09-21T10:21:30.599 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:30.599 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:30.600 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:30.600 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:30.600 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:30.600 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 134466 2024-09-21T10:21:30.600 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.600 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:30.600 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.600 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:30.600 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.735102+0000", 2024-09-21T10:21:30.600 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.600 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.600 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.600 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:30.600 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.735102+0000", 2024-09-21T10:21:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.735101+0000", 2024-09-21T10:21:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:21:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.735106+0000", 2024-09-21T10:21:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7040000000000002e-06 2024-09-21T10:21:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.735115+0000", 2024-09-21T10:21:30.601 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.8260000000000005e-06 2024-09-21T10:21:30.602 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.602 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.602 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:30.602 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.735122+0000", 2024-09-21T10:21:30.602 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.64e-06 2024-09-21T10:21:30.602 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.602 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.602 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.602 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.738080+0000", 2024-09-21T10:21:30.602 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002957889 2024-09-21T10:21:30.602 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.602 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.602 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:21:30.602 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.738086+0000", 2024-09-21T10:21:30.602 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2160000000000003e-06 2024-09-21T10:21:30.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.603 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.603 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.603 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601769+0000", 2024-09-21T10:21:30.603 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.86368243499999997 2024-09-21T10:21:30.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.603 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.603 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:21:30.603 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601775+0000", 2024-09-21T10:21:30.603 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6290000000000004e-06 2024-09-21T10:21:30.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.603 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.603 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.603 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.826712+0000", 2024-09-21T10:21:30.603 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2249370209999999 2024-09-21T10:21:30.604 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.604 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.604 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:30.604 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.826829+0000", 2024-09-21T10:21:30.604 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000116885 2024-09-21T10:21:30.604 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.604 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.604 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,5,6", 2024-09-21T10:21:30.604 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.827459+0000", 2024-09-21T10:21:30.604 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00062943700000000005 2024-09-21T10:21:30.604 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.604 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.604 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.604 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.839442+0000", 2024-09-21T10:21:30.604 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.011983053 2024-09-21T10:21:30.604 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.605 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.605 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.605 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.847816+0000", 2024-09-21T10:21:30.605 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0083745539999999993 2024-09-21T10:21:30.605 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.605 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.605 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.605 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.854585+0000", 2024-09-21T10:21:30.605 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0067689880000000001 2024-09-21T10:21:30.605 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.605 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.605 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:30.605 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.854652+0000", 2024-09-21T10:21:30.605 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6283000000000002e-05 2024-09-21T10:21:30.606 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.606 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.606 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:30.606 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.854688+0000", 2024-09-21T10:21:30.606 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6705000000000002e-05 2024-09-21T10:21:30.606 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.606 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.606 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:30.606 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.854902+0000", 2024-09-21T10:21:30.606 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00021421 2024-09-21T10:21:30.606 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.606 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.606 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.606 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.606 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.607 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:134472 13.7 13:ea565dde:::benchmark_data_smithi107_278655_object6565:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:21:30.607 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:58.743620+0000", 2024-09-21T10:21:30.607 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 91.778604465000001, 2024-09-21T10:21:30.607 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1110101480000001, 2024-09-21T10:21:30.607 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:30.607 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:30.607 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:30.607 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:30.607 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:30.607 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 134472 2024-09-21T10:21:30.607 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.607 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:30.607 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.607 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:30.607 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.743620+0000", 2024-09-21T10:21:30.608 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.608 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.608 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.608 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:30.608 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.743620+0000", 2024-09-21T10:21:30.608 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.608 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.608 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.608 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:30.608 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.743618+0000", 2024-09-21T10:21:30.608 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:21:30.608 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.608 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.608 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:30.609 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.743626+0000", 2024-09-21T10:21:30.609 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.34e-06 2024-09-21T10:21:30.609 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.609 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.609 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:30.609 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.743642+0000", 2024-09-21T10:21:30.609 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.6957e-05 2024-09-21T10:21:30.609 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.609 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.609 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:30.609 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.743650+0000", 2024-09-21T10:21:30.609 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2119999999999997e-06 2024-09-21T10:21:30.609 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.609 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.609 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.610 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.743703+0000", 2024-09-21T10:21:30.610 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2948999999999997e-05 2024-09-21T10:21:30.610 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.610 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.610 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:21:30.610 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.743716+0000", 2024-09-21T10:21:30.610 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.38e-05 2024-09-21T10:21:30.610 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.610 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.610 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.610 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.602148+0000", 2024-09-21T10:21:30.610 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.85843178200000003 2024-09-21T10:21:30.610 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.610 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.611 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:21:30.611 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.602157+0000", 2024-09-21T10:21:30.611 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.5019999999999992e-06 2024-09-21T10:21:30.611 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.611 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.611 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.611 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.834845+0000", 2024-09-21T10:21:30.611 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2326882100000001 2024-09-21T10:21:30.611 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.611 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.611 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:30.611 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.834962+0000", 2024-09-21T10:21:30.611 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000116774 2024-09-21T10:21:30.611 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.611 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.612 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 4,5,6", 2024-09-21T10:21:30.612 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.835089+0000", 2024-09-21T10:21:30.612 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012683799999999999 2024-09-21T10:21:30.612 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.612 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.612 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.612 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.847843+0000", 2024-09-21T10:21:30.612 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.012754860999999999 2024-09-21T10:21:30.612 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.612 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.612 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.612 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.847868+0000", 2024-09-21T10:21:30.612 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.4922e-05 2024-09-21T10:21:30.612 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.612 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.612 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.613 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.847894+0000", 2024-09-21T10:21:30.613 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.5323000000000001e-05 2024-09-21T10:21:30.613 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.613 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.613 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:30.613 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.854458+0000", 2024-09-21T10:21:30.613 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0065641099999999997 2024-09-21T10:21:30.613 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.613 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.613 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:30.613 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.854517+0000", 2024-09-21T10:21:30.613 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9561000000000001e-05 2024-09-21T10:21:30.613 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.613 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.613 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:30.614 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.854630+0000", 2024-09-21T10:21:30.614 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000112475 2024-09-21T10:21:30.614 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.614 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.614 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.614 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.614 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.614 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:174977 13.f 13:f3eb7148:::benchmark_data_smithi107_278655_object47070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-21T10:21:30.614 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:46.819794+0000", 2024-09-21T10:21:30.614 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 43.702430481999997, 2024-09-21T10:21:30.614 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.37579683600000002, 2024-09-21T10:21:30.614 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:30.614 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:30.614 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:30.614 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:30.615 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:30.615 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 174977 2024-09-21T10:21:30.615 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.615 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:30.615 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.615 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:30.615 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.819794+0000", 2024-09-21T10:21:30.615 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.615 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.615 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.615 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:30.615 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.819794+0000", 2024-09-21T10:21:30.615 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.615 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.615 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.616 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:30.616 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.819792+0000", 2024-09-21T10:21:30.616 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:21:30.616 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.616 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.616 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:30.616 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.819799+0000", 2024-09-21T10:21:30.616 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9580000000000001e-06 2024-09-21T10:21:30.616 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.616 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.616 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:30.616 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.819814+0000", 2024-09-21T10:21:30.616 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5211000000000001e-05 2024-09-21T10:21:30.616 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.616 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.616 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:30.617 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.819820+0000", 2024-09-21T10:21:30.617 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0480000000000004e-06 2024-09-21T10:21:30.617 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.617 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.617 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.617 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.819849+0000", 2024-09-21T10:21:30.617 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.8303999999999999e-05 2024-09-21T10:21:30.617 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.617 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.617 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:30.617 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.819947+0000", 2024-09-21T10:21:30.617 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.8627000000000003e-05 2024-09-21T10:21:30.617 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.617 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.617 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,5,6", 2024-09-21T10:21:30.617 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.820029+0000", 2024-09-21T10:21:30.618 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.1605000000000002e-05 2024-09-21T10:21:30.618 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.618 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.618 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.618 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.823532+0000", 2024-09-21T10:21:30.618 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0035031070000000001 2024-09-21T10:21:30.618 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.618 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.618 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.618 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.823560+0000", 2024-09-21T10:21:30.618 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7467e-05 2024-09-21T10:21:30.618 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.618 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.618 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:30.618 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.823611+0000", 2024-09-21T10:21:30.619 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0985000000000001e-05 2024-09-21T10:21:30.619 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.619 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.619 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.619 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:47.195419+0000", 2024-09-21T10:21:30.619 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.37180849799999999 2024-09-21T10:21:30.619 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.619 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.619 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:30.619 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:47.195465+0000", 2024-09-21T10:21:30.619 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6353999999999997e-05 2024-09-21T10:21:30.619 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.619 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.619 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:30.619 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:47.195590+0000", 2024-09-21T10:21:30.619 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000125084 2024-09-21T10:21:30.620 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.620 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.620 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.620 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.620 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.620 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:174979 13.f 13:f447ae81:::benchmark_data_smithi107_278655_object47072:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-21T10:21:30.620 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:46.839661+0000", 2024-09-21T10:21:30.620 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 43.682562601999997, 2024-09-21T10:21:30.620 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.35786738699999998, 2024-09-21T10:21:30.620 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:30.620 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:30.620 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:30.620 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:30.620 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:30.620 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 174979 2024-09-21T10:21:30.621 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.621 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:30.621 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.621 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:30.621 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.839661+0000", 2024-09-21T10:21:30.621 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.621 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.621 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.621 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:30.621 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.839661+0000", 2024-09-21T10:21:30.621 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.621 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.621 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.621 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:30.621 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.839660+0000", 2024-09-21T10:21:30.622 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:21:30.622 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.622 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.622 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:30.622 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.839668+0000", 2024-09-21T10:21:30.622 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.5660000000000001e-06 2024-09-21T10:21:30.622 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.622 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.622 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:30.622 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.839680+0000", 2024-09-21T10:21:30.622 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2255999999999999e-05 2024-09-21T10:21:30.622 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.622 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.622 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:30.622 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.839686+0000", 2024-09-21T10:21:30.623 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6810000000000003e-06 2024-09-21T10:21:30.623 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.623 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.623 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.623 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.839726+0000", 2024-09-21T10:21:30.623 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0571999999999997e-05 2024-09-21T10:21:30.623 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.623 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.623 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:30.623 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.839844+0000", 2024-09-21T10:21:30.623 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000117799 2024-09-21T10:21:30.623 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.623 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.623 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,5,6", 2024-09-21T10:21:30.624 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.839971+0000", 2024-09-21T10:21:30.624 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012727999999999999 2024-09-21T10:21:30.624 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.624 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.624 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.624 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.843754+0000", 2024-09-21T10:21:30.624 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0037827870000000001 2024-09-21T10:21:30.624 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.624 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.624 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:30.624 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.843767+0000", 2024-09-21T10:21:30.624 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3079e-05 2024-09-21T10:21:30.624 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.624 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.624 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.625 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.844117+0000", 2024-09-21T10:21:30.625 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00035028799999999998 2024-09-21T10:21:30.625 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.625 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.625 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.625 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:47.197221+0000", 2024-09-21T10:21:30.625 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.35310363 2024-09-21T10:21:30.625 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.625 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.625 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:30.625 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:47.197349+0000", 2024-09-21T10:21:30.625 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012828699999999999 2024-09-21T10:21:30.625 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.625 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.625 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:30.626 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:47.197529+0000", 2024-09-21T10:21:30.626 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000179648 2024-09-21T10:21:30.626 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.626 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.626 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.626 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.626 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.626 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:191613 13.0 13:0428af3c:::benchmark_data_smithi107_278655_object63706:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-21T10:21:30.626 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:21:06.147468+0000", 2024-09-21T10:21:30.626 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 24.374755645, 2024-09-21T10:21:30.626 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.115002938, 2024-09-21T10:21:30.626 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:30.626 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:30.626 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:30.627 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:30.627 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:30.627 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 191613 2024-09-21T10:21:30.627 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.627 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:30.627 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.627 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:30.627 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:06.147468+0000", 2024-09-21T10:21:30.627 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.627 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.627 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.627 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:30.627 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:06.147468+0000", 2024-09-21T10:21:30.627 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.627 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.628 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.628 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:30.628 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:06.147467+0000", 2024-09-21T10:21:30.628 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:21:30.628 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.628 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.628 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:30.628 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:06.147474+0000", 2024-09-21T10:21:30.628 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.8930000000000005e-06 2024-09-21T10:21:30.628 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.628 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.628 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:30.628 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:06.147487+0000", 2024-09-21T10:21:30.628 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2609000000000001e-05 2024-09-21T10:21:30.628 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.628 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.629 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:30.629 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:06.147492+0000", 2024-09-21T10:21:30.629 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3079999999999998e-06 2024-09-21T10:21:30.629 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.629 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.629 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.629 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:06.147640+0000", 2024-09-21T10:21:30.629 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014735 2024-09-21T10:21:30.629 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.629 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.629 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:30.629 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:06.147829+0000", 2024-09-21T10:21:30.629 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000189457 2024-09-21T10:21:30.629 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.629 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.630 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:21:30.630 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:06.147963+0000", 2024-09-21T10:21:30.630 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013427900000000001 2024-09-21T10:21:30.630 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.630 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.630 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:30.630 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:06.152438+0000", 2024-09-21T10:21:30.630 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0044743099999999996 2024-09-21T10:21:30.630 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.630 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.630 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.630 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:06.153949+0000", 2024-09-21T10:21:30.630 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001511208 2024-09-21T10:21:30.630 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.630 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.630 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.631 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:06.262377+0000", 2024-09-21T10:21:30.631 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.10842750299999999 2024-09-21T10:21:30.631 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.631 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.631 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:30.631 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:06.262435+0000", 2024-09-21T10:21:30.631 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8286999999999999e-05 2024-09-21T10:21:30.631 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.631 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.631 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:30.631 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:06.262471+0000", 2024-09-21T10:21:30.631 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6579999999999999e-05 2024-09-21T10:21:30.631 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.631 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.631 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.631 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.632 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.632 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:196035 13.f 13:fce5c4ac:::benchmark_data_smithi107_278655_object68128:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-09-21T10:21:30.632 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:21:10.937214+0000", 2024-09-21T10:21:30.632 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 19.585010469, 2024-09-21T10:21:30.632 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.076396934999999999, 2024-09-21T10:21:30.632 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:30.632 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:30.632 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:30.632 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:30.632 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:30.632 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 196035 2024-09-21T10:21:30.632 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.632 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:30.632 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.632 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:30.632 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:10.937214+0000", 2024-09-21T10:21:30.633 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.633 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.633 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.633 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:30.633 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:10.937214+0000", 2024-09-21T10:21:30.634 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.634 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.634 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.634 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:30.634 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:10.937213+0000", 2024-09-21T10:21:30.634 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:21:30.634 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.634 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.634 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:30.634 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:10.937219+0000", 2024-09-21T10:21:30.634 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0460000000000002e-06 2024-09-21T10:21:30.634 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.635 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.635 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:30.635 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:10.937231+0000", 2024-09-21T10:21:30.635 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2115e-05 2024-09-21T10:21:30.635 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.635 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.635 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:30.635 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:10.937237+0000", 2024-09-21T10:21:30.635 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8139999999999997e-06 2024-09-21T10:21:30.635 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.635 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.635 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.635 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:10.948456+0000", 2024-09-21T10:21:30.636 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.011219331000000001 2024-09-21T10:21:30.636 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.636 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.636 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:30.636 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:10.948747+0000", 2024-09-21T10:21:30.636 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00029139500000000001 2024-09-21T10:21:30.636 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.636 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.636 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,5,6", 2024-09-21T10:21:30.636 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:10.948981+0000", 2024-09-21T10:21:30.636 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000233985 2024-09-21T10:21:30.636 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.636 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.636 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:30.636 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:10.962750+0000", 2024-09-21T10:21:30.637 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.013768169 2024-09-21T10:21:30.637 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.637 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.637 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.637 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:10.968151+0000", 2024-09-21T10:21:30.637 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0054016419999999999 2024-09-21T10:21:30.637 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.637 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.637 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.637 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:10.969351+0000", 2024-09-21T10:21:30.637 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001199603 2024-09-21T10:21:30.637 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.637 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.637 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.638 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:11.013494+0000", 2024-09-21T10:21:30.638 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.044143001000000001 2024-09-21T10:21:30.638 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.638 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.638 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:30.638 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:11.013529+0000", 2024-09-21T10:21:30.638 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5256000000000003e-05 2024-09-21T10:21:30.638 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.638 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.638 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:30.638 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:11.013611+0000", 2024-09-21T10:21:30.638 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.1549000000000001e-05 2024-09-21T10:21:30.638 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.638 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.639 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.639 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.639 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.639 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:200980 13.0 13:04f7b0c2:::benchmark_data_smithi107_278655_object73073:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-09-21T10:21:30.639 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:21:16.842353+0000", 2024-09-21T10:21:30.639 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 13.679871101, 2024-09-21T10:21:30.639 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.081606321999999995, 2024-09-21T10:21:30.639 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:30.639 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:30.639 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:30.639 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:30.639 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:30.639 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 200980 2024-09-21T10:21:30.639 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.640 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:30.640 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.640 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:30.640 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.842353+0000", 2024-09-21T10:21:30.640 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.640 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.640 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.640 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:30.640 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.842353+0000", 2024-09-21T10:21:30.640 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.640 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.640 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.640 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:30.640 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.842351+0000", 2024-09-21T10:21:30.640 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:21:30.640 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.641 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.641 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:30.641 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.842359+0000", 2024-09-21T10:21:30.641 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.5099999999999998e-06 2024-09-21T10:21:30.641 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.641 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.641 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:30.641 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.842382+0000", 2024-09-21T10:21:30.641 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.2563000000000001e-05 2024-09-21T10:21:30.641 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.641 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.641 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:30.641 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.842389+0000", 2024-09-21T10:21:30.641 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8539999999999999e-06 2024-09-21T10:21:30.642 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.642 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.642 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.642 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.842444+0000", 2024-09-21T10:21:30.642 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4799999999999997e-05 2024-09-21T10:21:30.642 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.642 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.642 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:30.642 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.842562+0000", 2024-09-21T10:21:30.642 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000118687 2024-09-21T10:21:30.642 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.642 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.642 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:21:30.642 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.842676+0000", 2024-09-21T10:21:30.642 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011364199999999999 2024-09-21T10:21:30.643 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.643 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.643 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:30.643 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.846520+0000", 2024-09-21T10:21:30.643 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0038443919999999999 2024-09-21T10:21:30.643 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.643 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.643 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.643 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.851313+0000", 2024-09-21T10:21:30.643 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0047925449999999996 2024-09-21T10:21:30.643 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.643 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.643 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.643 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.923863+0000", 2024-09-21T10:21:30.644 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.072550263000000004 2024-09-21T10:21:30.644 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.644 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.644 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:30.644 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.923913+0000", 2024-09-21T10:21:30.644 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9781999999999999e-05 2024-09-21T10:21:30.644 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.644 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.644 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:30.644 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.923959+0000", 2024-09-21T10:21:30.644 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6301999999999998e-05 2024-09-21T10:21:30.644 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.644 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.644 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.644 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.644 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.645 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "MOSDPGPushReply(13.f 392/381 [PushReplyOp(13:fc64b009:::benchmark_data_smithi107_278655_object98447:head)])", 2024-09-21T10:21:30.645 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:21:16.856330+0000", 2024-09-21T10:21:30.645 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 13.665893755999999, 2024-09-21T10:21:30.645 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.12840679299999999, 2024-09-21T10:21:30.645 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:30.645 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:21:30.645 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:30.645 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.645 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:30.645 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.856330+0000", 2024-09-21T10:21:30.645 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.645 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.645 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.645 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:30.645 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.856330+0000", 2024-09-21T10:21:30.646 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.646 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.646 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.646 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:30.646 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.856329+0000", 2024-09-21T10:21:30.646 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:21:30.646 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.646 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.646 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:30.646 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.856337+0000", 2024-09-21T10:21:30.646 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6549999999999993e-06 2024-09-21T10:21:30.646 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.646 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.646 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:30.646 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.923776+0000", 2024-09-21T10:21:30.646 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.067438910000000005 2024-09-21T10:21:30.647 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.647 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.647 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:30.647 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.923808+0000", 2024-09-21T10:21:30.647 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1593000000000001e-05 2024-09-21T10:21:30.647 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.647 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.647 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.647 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.984522+0000", 2024-09-21T10:21:30.647 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.060714177000000001 2024-09-21T10:21:30.647 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.647 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.647 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:30.647 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.984737+0000", 2024-09-21T10:21:30.647 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00021513200000000001 2024-09-21T10:21:30.648 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.648 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.648 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.648 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.648 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.648 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:200999 13.7 13:e01e798d:::benchmark_data_smithi107_278655_object73092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-09-21T10:21:30.648 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:21:16.858998+0000", 2024-09-21T10:21:30.648 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 13.663225977, 2024-09-21T10:21:30.648 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.081716322999999993, 2024-09-21T10:21:30.648 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:21:30.648 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:21:30.648 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:21:30.648 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:21:30.648 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:21:30.649 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 200999 2024-09-21T10:21:30.649 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.649 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:21:30.649 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.649 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:21:30.649 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.858998+0000", 2024-09-21T10:21:30.649 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.649 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.649 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.649 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:21:30.649 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.858998+0000", 2024-09-21T10:21:30.649 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:21:30.649 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.649 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.649 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:21:30.650 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.858996+0000", 2024-09-21T10:21:30.650 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:21:30.650 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.650 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.650 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:21:30.650 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.859004+0000", 2024-09-21T10:21:30.650 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.6529999999999994e-06 2024-09-21T10:21:30.650 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.650 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.650 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:21:30.650 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.923836+0000", 2024-09-21T10:21:30.650 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.064832293999999999 2024-09-21T10:21:30.650 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.650 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.651 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:21:30.651 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.923848+0000", 2024-09-21T10:21:30.651 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1885000000000001e-05 2024-09-21T10:21:30.651 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.651 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.651 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:21:30.651 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.924413+0000", 2024-09-21T10:21:30.651 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00056536599999999998 2024-09-21T10:21:30.651 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.651 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.651 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:21:30.651 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.924513+0000", 2024-09-21T10:21:30.651 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010008 2024-09-21T10:21:30.651 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.651 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.651 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:21:30.652 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.924615+0000", 2024-09-21T10:21:30.652 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010176899999999999 2024-09-21T10:21:30.652 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.652 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.652 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.652 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.940311+0000", 2024-09-21T10:21:30.652 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.01569574 2024-09-21T10:21:30.652 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.652 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.652 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:21:30.652 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.940403+0000", 2024-09-21T10:21:30.652 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.2268000000000005e-05 2024-09-21T10:21:30.652 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.652 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.652 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:21:30.652 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.940607+0000", 2024-09-21T10:21:30.653 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000204293 2024-09-21T10:21:30.653 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.653 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.653 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:21:30.653 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.940682+0000", 2024-09-21T10:21:30.653 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.4852999999999996e-05 2024-09-21T10:21:30.653 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:21:30.653 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:21:30.653 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:21:30.653 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:16.940714+0000", 2024-09-21T10:21:30.653 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2172999999999998e-05 2024-09-21T10:21:30.653 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.653 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.653 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.654 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:21:30.654 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:21:30.654 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:21:30.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:30.654 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:30.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:30.755 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:30.783 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:30.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:30.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:30.884 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:30.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:30 smithi129 ceph-mon[88471]: pgmap v1373: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 1932/134775 objects misplaced (1.434%); 12 MiB/s, 188 objects/s recovering 2024-09-21T10:21:30.930 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:30.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:30.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:31.031 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:31.072 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:31.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:31.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:31.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:30 smithi107 ceph-mon[102221]: pgmap v1373: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 1932/134775 objects misplaced (1.434%); 12 MiB/s, 188 objects/s recovering 2024-09-21T10:21:31.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:30 smithi107 ceph-mon[105022]: pgmap v1373: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 1.09k op/s; 1932/134775 objects misplaced (1.434%); 12 MiB/s, 188 objects/s recovering 2024-09-21T10:21:31.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:31.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:31.206 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:31.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:31.297 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:21:31.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:31.375 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:31.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:31.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:31.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:31.509 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:31.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:31.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:31.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:21:31.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:31.669 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:31.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:31.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:31.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:31.777 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:21:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:21:31] "GET /metrics HTTP/1.1" 200 36804 "" "Prometheus/2.51.0" 2024-09-21T10:21:31.809 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:31.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:31.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:31.910 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:31.973 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:31.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:31.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:32.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:32.110 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:32.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:32.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:32.212 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:32.245 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:32.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:32.249 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:32.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:32.320 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:32.350 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:32.379 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:32.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:32.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:32.480 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:32.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:32.534 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:32.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:32.626 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:21:32.630 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:21:32.635 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:32.652 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:32 smithi107 ceph-mon[102221]: pgmap v1374: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 665 op/s; 1932/134775 objects misplaced (1.434%); 7.8 MiB/s, 124 objects/s recovering 2024-09-21T10:21:32.652 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:32 smithi107 ceph-mon[105022]: pgmap v1374: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 665 op/s; 1932/134775 objects misplaced (1.434%); 7.8 MiB/s, 124 objects/s recovering 2024-09-21T10:21:32.679 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:32.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:32.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:32.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:21:32.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:32.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:32.820 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:32.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:32 smithi129 ceph-mon[88471]: pgmap v1374: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 665 op/s; 1932/134775 objects misplaced (1.434%); 7.8 MiB/s, 124 objects/s recovering 2024-09-21T10:21:32.920 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:32.952 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:32.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:33.054 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:33.081 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:33.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:33.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:33.183 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:33.217 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:33.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:33.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:33.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:33.355 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:33.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:33.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:33.456 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:33.492 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:33.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:33.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:33.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:33.622 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:33.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:33.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:33.723 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:33.747 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:33.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:33.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:33.848 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:33.870 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:33.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:33.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:33.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:33 smithi129 ceph-mon[88471]: from='client.47072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:21:33.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:34.002 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:34.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:34.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:34.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:34.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:33 smithi107 ceph-mon[102221]: from='client.47072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:21:34.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:33 smithi107 ceph-mon[105022]: from='client.47072 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:21:34.125 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:34.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:34.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:34.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:34.248 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:34.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:34.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:34.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:34.384 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:34.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:34.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:34.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:34.516 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:34.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:34.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:34.618 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:34.654 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:34.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:34.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:34.755 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:34.795 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:34.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:34.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:34.896 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:34 smithi129 ceph-mon[88471]: pgmap v1375: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 952 op/s; 1932/124284 objects misplaced (1.555%); 7.0 MiB/s, 111 objects/s recovering 2024-09-21T10:21:34.920 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:34.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:34.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:35.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:35.053 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:35.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:35.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:35.064 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:34 smithi107 ceph-mon[102221]: pgmap v1375: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 952 op/s; 1932/124284 objects misplaced (1.555%); 7.0 MiB/s, 111 objects/s recovering 2024-09-21T10:21:35.065 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:34 smithi107 ceph-mon[105022]: pgmap v1375: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 952 op/s; 1932/124284 objects misplaced (1.555%); 7.0 MiB/s, 111 objects/s recovering 2024-09-21T10:21:35.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:35.186 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:35.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:35.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:35.288 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:35.322 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:35.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:35.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:35.460 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:35.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:35.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:35.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:35.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:35.596 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:35.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:35.697 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:35.739 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:35.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:35.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:35.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:35.875 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:35.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:35.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:35.977 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:36.014 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:36.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:36.116 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:36.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:36.155 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:36.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:36.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:36.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:36.289 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:36.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:36.390 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:36.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:36.425 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:36.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:36.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:36.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:36.562 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:36.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:36.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:36.700 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:36.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:36.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:36.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:36.830 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:36 smithi107 ceph-mon[102221]: pgmap v1376: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 887 op/s; 1274/119508 objects misplaced (1.066%); 8.1 MiB/s, 129 objects/s recovering 2024-09-21T10:21:36.830 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:36 smithi107 ceph-mon[105022]: pgmap v1376: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 887 op/s; 1274/119508 objects misplaced (1.066%); 8.1 MiB/s, 129 objects/s recovering 2024-09-21T10:21:36.839 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:36.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:36.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:36 smithi129 ceph-mon[88471]: pgmap v1376: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 887 op/s; 1274/119508 objects misplaced (1.066%); 8.1 MiB/s, 129 objects/s recovering 2024-09-21T10:21:36.940 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:36.977 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:36.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:36.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:37.079 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:37.111 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:37.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:37.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:37.162 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1374,"stamp":"2024-09-21T10:21:31.487897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2943058104,"num_objects":44925,"num_object_clones":0,"num_object_copies":134775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1932,"num_objects_unfound":0,"num_objects_dirty":44925,"num_whiteouts":0,"num_read":14,"num_read_kb":35,"num_write":210981,"num_write_kb":8185963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22447,"num_bytes_recovered":1467199044,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109564,"ondisk_log_size":109564,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":16397688,"kb_used_data":8629300,"kb_used_omap":41,"kb_used_meta":7768150,"kb_avail":639818376,"statfs":{"total":671965249536,"available":655174017024,"internally_reserved":0,"allocated":8836403200,"data_stored":8819292586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":42519,"internal_metadata":7954586089},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-349044736,"num_objects":-5326,"num_object_clones":0,"num_object_copies":-15978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1486,"num_objects_unfound":0,"num_objects_dirty":-5326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5326,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":993,"num_bytes_recovered":65077248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002224"},"pg_stats":[{"pgid":"1.11","version":"77'100","reported_seq":307,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.635499+0000","last_change":"2024-09-21T10:10:57.397786+0000","last_active":"2024-09-21T10:21:25.635499+0000","last_peered":"2024-09-21T10:21:25.635499+0000","last_clean":"2024-09-21T10:21:25.635499+0000","last_became_active":"2024-09-21T10:10:57.396742+0000","last_became_peered":"2024-09-21T10:10:57.396742+0000","last_unstale":"2024-09-21T10:21:25.635499+0000","last_undegraded":"2024-09-21T10:21:25.635499+0000","last_fullsized":"2024-09-21T10:21:25.635499+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:53.631923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:53.631923+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:53:52.740470+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":645,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.641727+0000","last_change":"2024-09-21T10:03:38.389577+0000","last_active":"2024-09-21T10:21:25.641727+0000","last_peered":"2024-09-21T10:21:25.641727+0000","last_clean":"2024-09-21T10:21:25.641727+0000","last_became_active":"2024-09-21T10:03:38.389292+0000","last_became_peered":"2024-09-21T10:03:38.389292+0000","last_unstale":"2024-09-21T10:21:25.641727+0000","last_undegraded":"2024-09-21T10:21:25.641727+0000","last_fullsized":"2024-09-21T10:21:25.641727+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:32.113472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":279,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.572930+0000","last_change":"2024-09-21T10:14:30.717695+0000","last_active":"2024-09-21T10:21:24.572930+0000","last_peered":"2024-09-21T10:21:24.572930+0000","last_clean":"2024-09-21T10:21:24.572930+0000","last_became_active":"2024-09-21T10:14:30.705952+0000","last_became_peered":"2024-09-21T10:14:30.705952+0000","last_unstale":"2024-09-21T10:21:24.572930+0000","last_undegraded":"2024-09-21T10:21:24.572930+0000","last_fullsized":"2024-09-21T10:21:24.572930+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:08:18.863430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'100","reported_seq":302,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.572624+0000","last_change":"2024-09-21T10:19:59.617108+0000","last_active":"2024-09-21T10:21:24.572624+0000","last_peered":"2024-09-21T10:21:24.572624+0000","last_clean":"2024-09-21T10:21:24.572624+0000","last_became_active":"2024-09-21T10:19:59.611681+0000","last_became_peered":"2024-09-21T10:19:59.611681+0000","last_unstale":"2024-09-21T10:21:24.572624+0000","last_undegraded":"2024-09-21T10:21:24.572624+0000","last_fullsized":"2024-09-21T10:21:24.572624+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.798577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.798577+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:20:43.585212+0000","scrub_duration":0.0023622640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":384,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.576709+0000","last_change":"2024-09-21T10:10:57.392782+0000","last_active":"2024-09-21T10:21:24.576709+0000","last_peered":"2024-09-21T10:21:24.576709+0000","last_clean":"2024-09-21T10:21:24.576709+0000","last_became_active":"2024-09-21T10:10:57.392305+0000","last_became_peered":"2024-09-21T10:10:57.392305+0000","last_unstale":"2024-09-21T10:21:24.576709+0000","last_undegraded":"2024-09-21T10:21:24.576709+0000","last_fullsized":"2024-09-21T10:21:24.576709+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:42:01.387622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":566,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.635171+0000","last_change":"2024-09-21T10:09:23.846703+0000","last_active":"2024-09-21T10:21:25.635171+0000","last_peered":"2024-09-21T10:21:25.635171+0000","last_clean":"2024-09-21T10:21:25.635171+0000","last_became_active":"2024-09-21T10:09:23.846415+0000","last_became_peered":"2024-09-21T10:09:23.846415+0000","last_unstale":"2024-09-21T10:21:25.635171+0000","last_undegraded":"2024-09-21T10:21:25.635171+0000","last_fullsized":"2024-09-21T10:21:25.635171+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:23.129285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'100","reported_seq":309,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.593340+0000","last_change":"2024-09-21T10:18:12.637168+0000","last_active":"2024-09-21T10:21:25.593340+0000","last_peered":"2024-09-21T10:21:25.593340+0000","last_clean":"2024-09-21T10:21:25.593340+0000","last_became_active":"2024-09-21T10:18:12.636410+0000","last_became_peered":"2024-09-21T10:18:12.636410+0000","last_unstale":"2024-09-21T10:21:25.593340+0000","last_undegraded":"2024-09-21T10:21:25.593340+0000","last_fullsized":"2024-09-21T10:21:25.593340+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.433227+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:40.015470+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":291,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.635378+0000","last_change":"2024-09-21T10:09:49.922489+0000","last_active":"2024-09-21T10:21:25.635378+0000","last_peered":"2024-09-21T10:21:25.635378+0000","last_clean":"2024-09-21T10:21:25.635378+0000","last_became_active":"2024-09-21T10:09:49.921629+0000","last_became_peered":"2024-09-21T10:09:49.921629+0000","last_unstale":"2024-09-21T10:21:25.635378+0000","last_undegraded":"2024-09-21T10:21:25.635378+0000","last_fullsized":"2024-09-21T10:21:25.635378+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:50.611979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":285,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.574427+0000","last_change":"2024-09-21T10:19:59.612376+0000","last_active":"2024-09-21T10:21:24.574427+0000","last_peered":"2024-09-21T10:21:24.574427+0000","last_clean":"2024-09-21T10:21:24.574427+0000","last_became_active":"2024-09-21T10:19:59.612053+0000","last_became_peered":"2024-09-21T10:19:59.612053+0000","last_unstale":"2024-09-21T10:21:24.574427+0000","last_undegraded":"2024-09-21T10:21:24.574427+0000","last_fullsized":"2024-09-21T10:21:24.574427+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:50:45.795114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"77'100","reported_seq":204,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.593410+0000","last_change":"2024-09-21T10:19:59.607712+0000","last_active":"2024-09-21T10:21:25.593410+0000","last_peered":"2024-09-21T10:21:25.593410+0000","last_clean":"2024-09-21T10:21:25.593410+0000","last_became_active":"2024-09-21T10:19:59.607334+0000","last_became_peered":"2024-09-21T10:19:59.607334+0000","last_unstale":"2024-09-21T10:21:25.593410+0000","last_undegraded":"2024-09-21T10:21:25.593410+0000","last_fullsized":"2024-09-21T10:21:25.593410+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:15:02.841968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:15:02.841968+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:35:38.085498+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":645,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.642459+0000","last_change":"2024-09-21T10:03:38.381552+0000","last_active":"2024-09-21T10:21:25.642459+0000","last_peered":"2024-09-21T10:21:25.642459+0000","last_clean":"2024-09-21T10:21:25.642459+0000","last_became_active":"2024-09-21T10:03:38.380898+0000","last_became_peered":"2024-09-21T10:03:38.380898+0000","last_unstale":"2024-09-21T10:21:25.642459+0000","last_undegraded":"2024-09-21T10:21:25.642459+0000","last_fullsized":"2024-09-21T10:21:25.642459+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:53.409606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":325,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.634904+0000","last_change":"2024-09-21T10:19:59.628128+0000","last_active":"2024-09-21T10:21:25.634904+0000","last_peered":"2024-09-21T10:21:25.634904+0000","last_clean":"2024-09-21T10:21:25.634904+0000","last_became_active":"2024-09-21T10:19:59.609198+0000","last_became_peered":"2024-09-21T10:19:59.609198+0000","last_unstale":"2024-09-21T10:21:25.634904+0000","last_undegraded":"2024-09-21T10:21:25.634904+0000","last_fullsized":"2024-09-21T10:21:25.634904+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:40:38.694583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1","version":"394'13234","reported_seq":22934,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:28.646376+0000","last_change":"2024-09-21T10:21:13.217152+0000","last_active":"2024-09-21T10:21:28.646376+0000","last_peered":"2024-09-21T10:21:28.646376+0000","last_clean":"2024-09-21T10:21:28.646376+0000","last_became_active":"2024-09-21T10:21:13.193523+0000","last_became_peered":"2024-09-21T10:21:13.193523+0000","last_unstale":"2024-09-21T10:21:28.646376+0000","last_undegraded":"2024-09-21T10:21:28.646376+0000","last_fullsized":"2024-09-21T10:21:28.646376+0000","mapping_epoch":391,"log_start":"374'5300","ondisk_log_start":"374'5300","created":367,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":7934,"log_dups_size":0,"ondisk_log_size":7934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:09.762782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183238656,"num_objects":2796,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13234,"num_write_kb":512960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4815,"num_bytes_recovered":315490304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.d","version":"77'100","reported_seq":287,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.576644+0000","last_change":"2024-09-21T10:14:30.708275+0000","last_active":"2024-09-21T10:21:24.576644+0000","last_peered":"2024-09-21T10:21:24.576644+0000","last_clean":"2024-09-21T10:21:24.576644+0000","last_became_active":"2024-09-21T10:14:30.707763+0000","last_became_peered":"2024-09-21T10:14:30.707763+0000","last_unstale":"2024-09-21T10:21:24.576644+0000","last_undegraded":"2024-09-21T10:21:24.576644+0000","last_fullsized":"2024-09-21T10:21:24.576644+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:55.259385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:55.259385+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:41.322447+0000","scrub_duration":0.0015667260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":322,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.572868+0000","last_change":"2024-09-21T10:14:30.720250+0000","last_active":"2024-09-21T10:21:24.572868+0000","last_peered":"2024-09-21T10:21:24.572868+0000","last_clean":"2024-09-21T10:21:24.572868+0000","last_became_active":"2024-09-21T10:14:30.707036+0000","last_became_peered":"2024-09-21T10:14:30.707036+0000","last_unstale":"2024-09-21T10:21:24.572868+0000","last_undegraded":"2024-09-21T10:21:24.572868+0000","last_fullsized":"2024-09-21T10:21:24.572868+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:17.438534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":375,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.572524+0000","last_change":"2024-09-21T10:19:59.609793+0000","last_active":"2024-09-21T10:21:24.572524+0000","last_peered":"2024-09-21T10:21:24.572524+0000","last_clean":"2024-09-21T10:21:24.572524+0000","last_became_active":"2024-09-21T10:19:59.608585+0000","last_became_peered":"2024-09-21T10:19:59.608585+0000","last_unstale":"2024-09-21T10:21:24.572524+0000","last_undegraded":"2024-09-21T10:21:24.572524+0000","last_fullsized":"2024-09-21T10:21:24.572524+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:12:29.189441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.0","version":"394'13430","reported_seq":13465,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:27.300095+0000","last_change":"2024-09-21T10:18:12.633194+0000","last_active":"2024-09-21T10:21:27.300095+0000","last_peered":"2024-09-21T10:21:27.300095+0000","last_clean":"2024-09-21T10:21:27.300095+0000","last_became_active":"2024-09-21T10:18:12.632915+0000","last_became_peered":"2024-09-21T10:18:12.632915+0000","last_unstale":"2024-09-21T10:21:27.300095+0000","last_undegraded":"2024-09-21T10:21:27.300095+0000","last_fullsized":"2024-09-21T10:21:27.300095+0000","mapping_epoch":367,"log_start":"376'6733","ondisk_log_start":"376'6733","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6697,"log_dups_size":0,"ondisk_log_size":6697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:13:26.865830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183631872,"num_objects":2802,"num_object_clones":0,"num_object_copies":8406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13430,"num_write_kb":519424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"77'100","reported_seq":464,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.641665+0000","last_change":"2024-09-21T10:19:59.625013+0000","last_active":"2024-09-21T10:21:25.641665+0000","last_peered":"2024-09-21T10:21:25.641665+0000","last_clean":"2024-09-21T10:21:25.641665+0000","last_became_active":"2024-09-21T10:19:59.614270+0000","last_became_peered":"2024-09-21T10:19:59.614270+0000","last_unstale":"2024-09-21T10:21:25.641665+0000","last_undegraded":"2024-09-21T10:21:25.641665+0000","last_fullsized":"2024-09-21T10:21:25.641665+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.981336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.981336+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:57:13.740902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":301,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.574435+0000","last_change":"2024-09-21T10:18:12.643154+0000","last_active":"2024-09-21T10:21:24.574435+0000","last_peered":"2024-09-21T10:21:24.574435+0000","last_clean":"2024-09-21T10:21:24.574435+0000","last_became_active":"2024-09-21T10:18:12.636287+0000","last_became_peered":"2024-09-21T10:18:12.636287+0000","last_unstale":"2024-09-21T10:21:24.574435+0000","last_undegraded":"2024-09-21T10:21:24.574435+0000","last_fullsized":"2024-09-21T10:21:24.574435+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:45:31.008309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":100,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.581478+0000","last_change":"2024-09-21T10:14:30.706497+0000","last_active":"2024-09-21T10:21:24.581478+0000","last_peered":"2024-09-21T10:21:24.581478+0000","last_clean":"2024-09-21T10:21:24.581478+0000","last_became_active":"2024-09-21T10:14:30.706202+0000","last_became_peered":"2024-09-21T10:14:30.706202+0000","last_unstale":"2024-09-21T10:21:24.581478+0000","last_undegraded":"2024-09-21T10:21:24.581478+0000","last_fullsized":"2024-09-21T10:21:24.581478+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:42:57.912707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3","version":"394'13596","reported_seq":13662,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:27.671321+0000","last_change":"2024-09-21T10:18:12.641462+0000","last_active":"2024-09-21T10:21:27.671321+0000","last_peered":"2024-09-21T10:21:27.671321+0000","last_clean":"2024-09-21T10:21:27.671321+0000","last_became_active":"2024-09-21T10:18:12.641056+0000","last_became_peered":"2024-09-21T10:18:12.641056+0000","last_unstale":"2024-09-21T10:21:27.671321+0000","last_undegraded":"2024-09-21T10:21:27.671321+0000","last_fullsized":"2024-09-21T10:21:27.671321+0000","mapping_epoch":367,"log_start":"376'6452","ondisk_log_start":"376'6452","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":7144,"log_dups_size":0,"ondisk_log_size":7144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:45:13.755572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181927936,"num_objects":2776,"num_object_clones":0,"num_object_copies":8328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13596,"num_write_kb":523904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"77'100","reported_seq":314,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.581711+0000","last_change":"2024-09-21T10:19:59.623348+0000","last_active":"2024-09-21T10:21:24.581711+0000","last_peered":"2024-09-21T10:21:24.581711+0000","last_clean":"2024-09-21T10:21:24.581711+0000","last_became_active":"2024-09-21T10:19:59.622450+0000","last_became_peered":"2024-09-21T10:19:59.622450+0000","last_unstale":"2024-09-21T10:21:24.581711+0000","last_undegraded":"2024-09-21T10:21:24.581711+0000","last_fullsized":"2024-09-21T10:21:24.581711+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.289859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.289859+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:55.434274+0000","scrub_duration":0.001251809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":287,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.576500+0000","last_change":"2024-09-21T10:19:59.620525+0000","last_active":"2024-09-21T10:21:24.576500+0000","last_peered":"2024-09-21T10:21:24.576500+0000","last_clean":"2024-09-21T10:21:24.576500+0000","last_became_active":"2024-09-21T10:19:59.605967+0000","last_became_peered":"2024-09-21T10:19:59.605967+0000","last_unstale":"2024-09-21T10:21:24.576500+0000","last_undegraded":"2024-09-21T10:21:24.576500+0000","last_fullsized":"2024-09-21T10:21:24.576500+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:43.669683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":206,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.593088+0000","last_change":"2024-09-21T10:19:59.605578+0000","last_active":"2024-09-21T10:21:25.593088+0000","last_peered":"2024-09-21T10:21:25.593088+0000","last_clean":"2024-09-21T10:21:25.593088+0000","last_became_active":"2024-09-21T10:19:59.604885+0000","last_became_peered":"2024-09-21T10:19:59.604885+0000","last_unstale":"2024-09-21T10:21:25.593088+0000","last_undegraded":"2024-09-21T10:21:25.593088+0000","last_fullsized":"2024-09-21T10:21:25.593088+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:48:11.107703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.2","version":"394'12991","reported_seq":13057,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:27.490283+0000","last_change":"2024-09-21T10:18:13.037514+0000","last_active":"2024-09-21T10:21:27.490283+0000","last_peered":"2024-09-21T10:21:27.490283+0000","last_clean":"2024-09-21T10:21:27.490283+0000","last_became_active":"2024-09-21T10:18:13.037222+0000","last_became_peered":"2024-09-21T10:18:13.037222+0000","last_unstale":"2024-09-21T10:21:27.490283+0000","last_undegraded":"2024-09-21T10:21:27.490283+0000","last_fullsized":"2024-09-21T10:21:27.490283+0000","mapping_epoch":367,"log_start":"376'6922","ondisk_log_start":"376'6922","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6069,"log_dups_size":0,"ondisk_log_size":6069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:57.265482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185794560,"num_objects":2835,"num_object_clones":0,"num_object_copies":8505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12991,"num_write_kb":506432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"77'100","reported_seq":46,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.581345+0000","last_change":"2024-09-21T10:18:27.085659+0000","last_active":"2024-09-21T10:21:24.581345+0000","last_peered":"2024-09-21T10:21:24.581345+0000","last_clean":"2024-09-21T10:21:24.581345+0000","last_became_active":"2024-09-21T10:18:12.633342+0000","last_became_peered":"2024-09-21T10:18:12.633342+0000","last_unstale":"2024-09-21T10:21:24.581345+0000","last_undegraded":"2024-09-21T10:21:24.581345+0000","last_fullsized":"2024-09-21T10:21:24.581345+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:18:27.085525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:18:27.085525+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:06:53.238321+0000","scrub_duration":0.0036485279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":456,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.641804+0000","last_change":"2024-09-21T10:19:59.608293+0000","last_active":"2024-09-21T10:21:25.641804+0000","last_peered":"2024-09-21T10:21:25.641804+0000","last_clean":"2024-09-21T10:21:25.641804+0000","last_became_active":"2024-09-21T10:19:59.607922+0000","last_became_peered":"2024-09-21T10:19:59.607922+0000","last_unstale":"2024-09-21T10:21:25.641804+0000","last_undegraded":"2024-09-21T10:21:25.641804+0000","last_fullsized":"2024-09-21T10:21:25.641804+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:02.971474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":553,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.635274+0000","last_change":"2024-09-21T10:14:30.711105+0000","last_active":"2024-09-21T10:21:25.635274+0000","last_peered":"2024-09-21T10:21:25.635274+0000","last_clean":"2024-09-21T10:21:25.635274+0000","last_became_active":"2024-09-21T10:14:30.710759+0000","last_became_peered":"2024-09-21T10:14:30.710759+0000","last_unstale":"2024-09-21T10:21:25.635274+0000","last_undegraded":"2024-09-21T10:21:25.635274+0000","last_fullsized":"2024-09-21T10:21:25.635274+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:40.690042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"394'13093","reported_seq":13127,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:27.273353+0000","last_change":"2024-09-21T10:18:13.037771+0000","last_active":"2024-09-21T10:21:27.273353+0000","last_peered":"2024-09-21T10:21:27.273353+0000","last_clean":"2024-09-21T10:21:27.273353+0000","last_became_active":"2024-09-21T10:18:13.037425+0000","last_became_peered":"2024-09-21T10:18:13.037425+0000","last_unstale":"2024-09-21T10:21:27.273353+0000","last_undegraded":"2024-09-21T10:21:27.273353+0000","last_fullsized":"2024-09-21T10:21:27.273353+0000","mapping_epoch":367,"log_start":"378'7799","ondisk_log_start":"378'7799","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":5294,"log_dups_size":0,"ondisk_log_size":5294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:34:43.436187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184090624,"num_objects":2809,"num_object_clones":0,"num_object_copies":8427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13093,"num_write_kb":508864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":267,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.574505+0000","last_change":"2024-09-21T10:04:53.355975+0000","last_active":"2024-09-21T10:21:24.574505+0000","last_peered":"2024-09-21T10:21:24.574505+0000","last_clean":"2024-09-21T10:21:24.574505+0000","last_became_active":"2024-09-21T10:04:42.547505+0000","last_became_peered":"2024-09-21T10:04:42.547505+0000","last_unstale":"2024-09-21T10:21:24.574505+0000","last_undegraded":"2024-09-21T10:21:24.574505+0000","last_fullsized":"2024-09-21T10:21:24.574505+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.355912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.355912+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:43.456830+0000","scrub_duration":0.0017980030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":657,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.648987+0000","last_change":"2024-09-21T10:14:30.715303+0000","last_active":"2024-09-21T10:21:25.648987+0000","last_peered":"2024-09-21T10:21:25.648987+0000","last_clean":"2024-09-21T10:21:25.648987+0000","last_became_active":"2024-09-21T10:14:30.703664+0000","last_became_peered":"2024-09-21T10:14:30.703664+0000","last_unstale":"2024-09-21T10:21:25.648987+0000","last_undegraded":"2024-09-21T10:21:25.648987+0000","last_fullsized":"2024-09-21T10:21:25.648987+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:28.967906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":110,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.581201+0000","last_change":"2024-09-21T10:19:59.620895+0000","last_active":"2024-09-21T10:21:24.581201+0000","last_peered":"2024-09-21T10:21:24.581201+0000","last_clean":"2024-09-21T10:21:24.581201+0000","last_became_active":"2024-09-21T10:19:59.620381+0000","last_became_peered":"2024-09-21T10:19:59.620381+0000","last_unstale":"2024-09-21T10:21:24.581201+0000","last_undegraded":"2024-09-21T10:21:24.581201+0000","last_fullsized":"2024-09-21T10:21:24.581201+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:20.740326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.4","version":"394'13343","reported_seq":15110,"reported_epoch":394,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T10:21:28.656381+0000","last_change":"2024-09-21T10:21:24.037523+0000","last_active":"2024-09-21T10:21:28.656381+0000","last_peered":"2024-09-21T10:21:28.656381+0000","last_clean":"2024-09-21T10:19:58.662726+0000","last_became_active":"2024-09-21T10:20:00.753799+0000","last_became_peered":"2024-09-21T10:20:00.753799+0000","last_unstale":"2024-09-21T10:21:28.656381+0000","last_undegraded":"2024-09-21T10:21:28.656381+0000","last_fullsized":"2024-09-21T10:21:28.656381+0000","mapping_epoch":381,"log_start":"374'5400","ondisk_log_start":"374'5400","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179765248,"num_objects":2743,"num_object_clones":0,"num_object_copies":8229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1932,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13343,"num_write_kb":514752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":857,"num_bytes_recovered":56164352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,6],"avail_no_missing":["2","1","6"],"object_location_counts":[{"shards":"1,2,6","objects":2743}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":407,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.596451+0000","last_change":"2024-09-21T10:10:57.403722+0000","last_active":"2024-09-21T10:21:25.596451+0000","last_peered":"2024-09-21T10:21:25.596451+0000","last_clean":"2024-09-21T10:21:25.596451+0000","last_became_active":"2024-09-21T10:10:57.394700+0000","last_became_peered":"2024-09-21T10:10:57.394700+0000","last_unstale":"2024-09-21T10:21:25.596451+0000","last_undegraded":"2024-09-21T10:21:25.596451+0000","last_fullsized":"2024-09-21T10:21:25.596451+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:02.152981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:02.152981+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:45:33.964266+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":304,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.581219+0000","last_change":"2024-09-21T10:19:59.622903+0000","last_active":"2024-09-21T10:21:24.581219+0000","last_peered":"2024-09-21T10:21:24.581219+0000","last_clean":"2024-09-21T10:21:24.581219+0000","last_became_active":"2024-09-21T10:19:59.622207+0000","last_became_peered":"2024-09-21T10:19:59.622207+0000","last_unstale":"2024-09-21T10:21:24.581219+0000","last_undegraded":"2024-09-21T10:21:24.581219+0000","last_fullsized":"2024-09-21T10:21:24.581219+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:17:41.944256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":36,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.581895+0000","last_change":"2024-09-21T10:18:13.037945+0000","last_active":"2024-09-21T10:21:24.581895+0000","last_peered":"2024-09-21T10:21:24.581895+0000","last_clean":"2024-09-21T10:21:24.581895+0000","last_became_active":"2024-09-21T10:18:13.037550+0000","last_became_peered":"2024-09-21T10:18:13.037550+0000","last_unstale":"2024-09-21T10:21:24.581895+0000","last_undegraded":"2024-09-21T10:21:24.581895+0000","last_fullsized":"2024-09-21T10:21:24.581895+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:24:24.702636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.7","version":"394'13153","reported_seq":25397,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:27.295615+0000","last_change":"2024-09-21T10:20:54.683344+0000","last_active":"2024-09-21T10:21:27.295615+0000","last_peered":"2024-09-21T10:21:27.295615+0000","last_clean":"2024-09-21T10:21:27.295615+0000","last_became_active":"2024-09-21T10:20:54.661472+0000","last_became_peered":"2024-09-21T10:20:54.661472+0000","last_unstale":"2024-09-21T10:21:27.295615+0000","last_undegraded":"2024-09-21T10:21:27.295615+0000","last_fullsized":"2024-09-21T10:21:27.295615+0000","mapping_epoch":388,"log_start":"376'6447","ondisk_log_start":"376'6447","created":367,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6706,"log_dups_size":0,"ondisk_log_size":6706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:03:50.455981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186318848,"num_objects":2843,"num_object_clones":0,"num_object_copies":8529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13153,"num_write_kb":511872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6091,"num_bytes_recovered":399179776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"77'100","reported_seq":299,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.593286+0000","last_change":"2024-09-21T10:14:05.474100+0000","last_active":"2024-09-21T10:21:25.593286+0000","last_peered":"2024-09-21T10:21:25.593286+0000","last_clean":"2024-09-21T10:21:25.593286+0000","last_became_active":"2024-09-21T10:14:05.473336+0000","last_became_peered":"2024-09-21T10:14:05.473336+0000","last_unstale":"2024-09-21T10:21:25.593286+0000","last_undegraded":"2024-09-21T10:21:25.593286+0000","last_fullsized":"2024-09-21T10:21:25.593286+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.540323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.540323+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:26:21.305903+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":36,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.580941+0000","last_change":"2024-09-21T10:18:12.629311+0000","last_active":"2024-09-21T10:21:24.580941+0000","last_peered":"2024-09-21T10:21:24.580941+0000","last_clean":"2024-09-21T10:21:24.580941+0000","last_became_active":"2024-09-21T10:18:12.629001+0000","last_became_peered":"2024-09-21T10:18:12.629001+0000","last_unstale":"2024-09-21T10:21:24.580941+0000","last_undegraded":"2024-09-21T10:21:24.580941+0000","last_fullsized":"2024-09-21T10:21:24.580941+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:09:17.165751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":110,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.581634+0000","last_change":"2024-09-21T10:19:59.623205+0000","last_active":"2024-09-21T10:21:24.581634+0000","last_peered":"2024-09-21T10:21:24.581634+0000","last_clean":"2024-09-21T10:21:24.581634+0000","last_became_active":"2024-09-21T10:19:59.620596+0000","last_became_peered":"2024-09-21T10:19:59.620596+0000","last_unstale":"2024-09-21T10:21:24.581634+0000","last_undegraded":"2024-09-21T10:21:24.581634+0000","last_fullsized":"2024-09-21T10:21:24.581634+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:56:58.517591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.6","version":"394'13070","reported_seq":13105,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:27.244746+0000","last_change":"2024-09-21T10:18:13.038010+0000","last_active":"2024-09-21T10:21:27.244746+0000","last_peered":"2024-09-21T10:21:27.244746+0000","last_clean":"2024-09-21T10:21:27.244746+0000","last_became_active":"2024-09-21T10:18:13.037700+0000","last_became_peered":"2024-09-21T10:18:13.037700+0000","last_unstale":"2024-09-21T10:21:27.244746+0000","last_undegraded":"2024-09-21T10:21:27.244746+0000","last_fullsized":"2024-09-21T10:21:27.244746+0000","mapping_epoch":367,"log_start":"378'7793","ondisk_log_start":"378'7793","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":5277,"log_dups_size":0,"ondisk_log_size":5277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:14:58.367606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185729024,"num_objects":2834,"num_object_clones":0,"num_object_copies":8502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13070,"num_write_kb":508928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":385,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.596359+0000","last_change":"2024-09-21T10:20:00.237716+0000","last_active":"2024-09-21T10:21:25.596359+0000","last_peered":"2024-09-21T10:21:25.596359+0000","last_clean":"2024-09-21T10:21:25.596359+0000","last_became_active":"2024-09-21T10:19:59.613432+0000","last_became_peered":"2024-09-21T10:19:59.613432+0000","last_unstale":"2024-09-21T10:21:25.596359+0000","last_undegraded":"2024-09-21T10:21:25.596359+0000","last_fullsized":"2024-09-21T10:21:25.596359+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:20:00.237668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:20:00.237668+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:42:48.704787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":647,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.641124+0000","last_change":"2024-09-21T10:03:38.386604+0000","last_active":"2024-09-21T10:21:25.641124+0000","last_peered":"2024-09-21T10:21:25.641124+0000","last_clean":"2024-09-21T10:21:25.641124+0000","last_became_active":"2024-09-21T10:03:38.386283+0000","last_became_peered":"2024-09-21T10:03:38.386283+0000","last_unstale":"2024-09-21T10:21:25.641124+0000","last_undegraded":"2024-09-21T10:21:25.641124+0000","last_fullsized":"2024-09-21T10:21:25.641124+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:27:05.944880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":506,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.594400+0000","last_change":"2024-09-21T10:14:30.709615+0000","last_active":"2024-09-21T10:21:25.594400+0000","last_peered":"2024-09-21T10:21:25.594400+0000","last_clean":"2024-09-21T10:21:25.594400+0000","last_became_active":"2024-09-21T10:14:30.709184+0000","last_became_peered":"2024-09-21T10:14:30.709184+0000","last_unstale":"2024-09-21T10:21:25.594400+0000","last_undegraded":"2024-09-21T10:21:25.594400+0000","last_fullsized":"2024-09-21T10:21:25.594400+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:55:27.220232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.c","version":"394'13187","reported_seq":13254,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:28.470735+0000","last_change":"2024-09-21T10:18:13.037466+0000","last_active":"2024-09-21T10:21:28.470735+0000","last_peered":"2024-09-21T10:21:28.470735+0000","last_clean":"2024-09-21T10:21:28.470735+0000","last_became_active":"2024-09-21T10:18:13.037151+0000","last_became_peered":"2024-09-21T10:18:13.037151+0000","last_unstale":"2024-09-21T10:21:28.470735+0000","last_undegraded":"2024-09-21T10:21:28.470735+0000","last_fullsized":"2024-09-21T10:21:28.470735+0000","mapping_epoch":367,"log_start":"376'6708","ondisk_log_start":"376'6708","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6479,"log_dups_size":0,"ondisk_log_size":6479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:29:14.215799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182517760,"num_objects":2785,"num_object_clones":0,"num_object_copies":8355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13187,"num_write_kb":511104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":294,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.572534+0000","last_change":"2024-09-21T10:19:59.610627+0000","last_active":"2024-09-21T10:21:24.572534+0000","last_peered":"2024-09-21T10:21:24.572534+0000","last_clean":"2024-09-21T10:21:24.572534+0000","last_became_active":"2024-09-21T10:19:59.610087+0000","last_became_peered":"2024-09-21T10:19:59.610087+0000","last_unstale":"2024-09-21T10:21:24.572534+0000","last_undegraded":"2024-09-21T10:21:24.572534+0000","last_fullsized":"2024-09-21T10:21:24.572534+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:47.393146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:47.393146+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:20:59.235205+0000","scrub_duration":0.001601086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":402,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.596567+0000","last_change":"2024-09-21T10:10:57.402946+0000","last_active":"2024-09-21T10:21:25.596567+0000","last_peered":"2024-09-21T10:21:25.596567+0000","last_clean":"2024-09-21T10:21:25.596567+0000","last_became_active":"2024-09-21T10:10:57.395450+0000","last_became_peered":"2024-09-21T10:10:57.395450+0000","last_unstale":"2024-09-21T10:21:25.596567+0000","last_undegraded":"2024-09-21T10:21:25.596567+0000","last_fullsized":"2024-09-21T10:21:25.596567+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:08.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":639,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.641277+0000","last_change":"2024-09-21T10:19:59.613261+0000","last_active":"2024-09-21T10:21:25.641277+0000","last_peered":"2024-09-21T10:21:25.641277+0000","last_clean":"2024-09-21T10:21:25.641277+0000","last_became_active":"2024-09-21T10:19:59.612948+0000","last_became_peered":"2024-09-21T10:19:59.612948+0000","last_unstale":"2024-09-21T10:21:25.641277+0000","last_undegraded":"2024-09-21T10:21:25.641277+0000","last_fullsized":"2024-09-21T10:21:25.641277+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:54:47.936463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":337,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.581349+0000","last_change":"2024-09-21T10:19:59.623034+0000","last_active":"2024-09-21T10:21:24.581349+0000","last_peered":"2024-09-21T10:21:24.581349+0000","last_clean":"2024-09-21T10:21:24.581349+0000","last_became_active":"2024-09-21T10:19:59.622713+0000","last_became_peered":"2024-09-21T10:19:59.622713+0000","last_unstale":"2024-09-21T10:21:24.581349+0000","last_undegraded":"2024-09-21T10:21:24.581349+0000","last_fullsized":"2024-09-21T10:21:24.581349+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:17:27.386485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.f","version":"394'13165","reported_seq":21084,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:27.291430+0000","last_change":"2024-09-21T10:21:25.623560+0000","last_active":"2024-09-21T10:21:27.291430+0000","last_peered":"2024-09-21T10:21:27.291430+0000","last_clean":"2024-09-21T10:21:27.291430+0000","last_became_active":"2024-09-21T10:21:25.597333+0000","last_became_peered":"2024-09-21T10:21:25.597333+0000","last_unstale":"2024-09-21T10:21:27.291430+0000","last_undegraded":"2024-09-21T10:21:27.291430+0000","last_fullsized":"2024-09-21T10:21:27.291430+0000","mapping_epoch":393,"log_start":"376'6494","ondisk_log_start":"376'6494","created":367,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6671,"log_dups_size":0,"ondisk_log_size":6671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:03:00.838286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187760640,"num_objects":2865,"num_object_clones":0,"num_object_copies":8595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13165,"num_write_kb":512960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3928,"num_bytes_recovered":257359872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":316,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.593499+0000","last_change":"2024-09-21T10:19:59.619449+0000","last_active":"2024-09-21T10:21:25.593499+0000","last_peered":"2024-09-21T10:21:25.593499+0000","last_clean":"2024-09-21T10:21:25.593499+0000","last_became_active":"2024-09-21T10:19:59.618546+0000","last_became_peered":"2024-09-21T10:19:59.618546+0000","last_unstale":"2024-09-21T10:21:25.593499+0000","last_undegraded":"2024-09-21T10:21:25.593499+0000","last_fullsized":"2024-09-21T10:21:25.593499+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.550551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.550551+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:18.554860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":258,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.572607+0000","last_change":"2024-09-21T10:09:23.838944+0000","last_active":"2024-09-21T10:21:24.572607+0000","last_peered":"2024-09-21T10:21:24.572607+0000","last_clean":"2024-09-21T10:21:24.572607+0000","last_became_active":"2024-09-21T10:09:23.837886+0000","last_became_peered":"2024-09-21T10:09:23.837886+0000","last_unstale":"2024-09-21T10:21:24.572607+0000","last_undegraded":"2024-09-21T10:21:24.572607+0000","last_fullsized":"2024-09-21T10:21:24.572607+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:16.928195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":530,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.635249+0000","last_change":"2024-09-21T10:18:12.637103+0000","last_active":"2024-09-21T10:21:25.635249+0000","last_peered":"2024-09-21T10:21:25.635249+0000","last_clean":"2024-09-21T10:21:25.635249+0000","last_became_active":"2024-09-21T10:18:12.636645+0000","last_became_peered":"2024-09-21T10:18:12.636645+0000","last_unstale":"2024-09-21T10:21:25.635249+0000","last_undegraded":"2024-09-21T10:21:25.635249+0000","last_fullsized":"2024-09-21T10:21:25.635249+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:53:50.745484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":292,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.635350+0000","last_change":"2024-09-21T10:09:49.923044+0000","last_active":"2024-09-21T10:21:25.635350+0000","last_peered":"2024-09-21T10:21:25.635350+0000","last_clean":"2024-09-21T10:21:25.635350+0000","last_became_active":"2024-09-21T10:09:49.922711+0000","last_became_peered":"2024-09-21T10:09:49.922711+0000","last_unstale":"2024-09-21T10:21:25.635350+0000","last_undegraded":"2024-09-21T10:21:25.635350+0000","last_fullsized":"2024-09-21T10:21:25.635350+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:14.090084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"394'12836","reported_seq":12871,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:27.273092+0000","last_change":"2024-09-21T10:18:13.038029+0000","last_active":"2024-09-21T10:21:27.273092+0000","last_peered":"2024-09-21T10:21:27.273092+0000","last_clean":"2024-09-21T10:21:27.273092+0000","last_became_active":"2024-09-21T10:18:13.037584+0000","last_became_peered":"2024-09-21T10:18:13.037584+0000","last_unstale":"2024-09-21T10:21:27.273092+0000","last_undegraded":"2024-09-21T10:21:27.273092+0000","last_fullsized":"2024-09-21T10:21:27.273092+0000","mapping_epoch":367,"log_start":"376'6116","ondisk_log_start":"376'6116","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6720,"log_dups_size":0,"ondisk_log_size":6720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:29:18.430382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183238656,"num_objects":2796,"num_object_clones":0,"num_object_copies":8388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12836,"num_write_kb":500224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":308,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.572714+0000","last_change":"2024-09-21T10:18:12.636851+0000","last_active":"2024-09-21T10:21:24.572714+0000","last_peered":"2024-09-21T10:21:24.572714+0000","last_clean":"2024-09-21T10:21:24.572714+0000","last_became_active":"2024-09-21T10:18:12.636513+0000","last_became_peered":"2024-09-21T10:18:12.636513+0000","last_unstale":"2024-09-21T10:21:24.572714+0000","last_undegraded":"2024-09-21T10:21:24.572714+0000","last_fullsized":"2024-09-21T10:21:24.572714+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:52.361137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:52.361137+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:21.025735+0000","scrub_duration":0.001255358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":685,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.596707+0000","last_change":"2024-09-21T10:09:23.837859+0000","last_active":"2024-09-21T10:21:25.596707+0000","last_peered":"2024-09-21T10:21:25.596707+0000","last_clean":"2024-09-21T10:21:25.596707+0000","last_became_active":"2024-09-21T10:09:23.837488+0000","last_became_peered":"2024-09-21T10:09:23.837488+0000","last_unstale":"2024-09-21T10:21:25.596707+0000","last_undegraded":"2024-09-21T10:21:25.596707+0000","last_fullsized":"2024-09-21T10:21:25.596707+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:25.215501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":152,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.572416+0000","last_change":"2024-09-21T10:19:59.610430+0000","last_active":"2024-09-21T10:21:24.572416+0000","last_peered":"2024-09-21T10:21:24.572416+0000","last_clean":"2024-09-21T10:21:24.572416+0000","last_became_active":"2024-09-21T10:19:59.609477+0000","last_became_peered":"2024-09-21T10:19:59.609477+0000","last_unstale":"2024-09-21T10:21:24.572416+0000","last_undegraded":"2024-09-21T10:21:24.572416+0000","last_fullsized":"2024-09-21T10:21:24.572416+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:41:08.094180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":678,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.596651+0000","last_change":"2024-09-21T10:10:57.395073+0000","last_active":"2024-09-21T10:21:25.596651+0000","last_peered":"2024-09-21T10:21:25.596651+0000","last_clean":"2024-09-21T10:21:25.596651+0000","last_became_active":"2024-09-21T10:10:57.394674+0000","last_became_peered":"2024-09-21T10:10:57.394674+0000","last_unstale":"2024-09-21T10:21:25.596651+0000","last_undegraded":"2024-09-21T10:21:25.596651+0000","last_fullsized":"2024-09-21T10:21:25.596651+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:48:48.912649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.d","version":"394'13084","reported_seq":13120,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:27.237752+0000","last_change":"2024-09-21T10:18:12.632980+0000","last_active":"2024-09-21T10:21:27.237752+0000","last_peered":"2024-09-21T10:21:27.237752+0000","last_clean":"2024-09-21T10:21:27.237752+0000","last_became_active":"2024-09-21T10:18:12.632684+0000","last_became_peered":"2024-09-21T10:18:12.632684+0000","last_unstale":"2024-09-21T10:21:27.237752+0000","last_undegraded":"2024-09-21T10:21:27.237752+0000","last_fullsized":"2024-09-21T10:21:27.237752+0000","mapping_epoch":367,"log_start":"378'7742","ondisk_log_start":"378'7742","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":5342,"log_dups_size":0,"ondisk_log_size":5342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:08.974530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186712088,"num_objects":2850,"num_object_clones":0,"num_object_copies":8550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2850,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13084,"num_write_kb":509825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":548,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.635350+0000","last_change":"2024-09-21T10:10:57.398437+0000","last_active":"2024-09-21T10:21:25.635350+0000","last_peered":"2024-09-21T10:21:25.635350+0000","last_clean":"2024-09-21T10:21:25.635350+0000","last_became_active":"2024-09-21T10:10:57.397441+0000","last_became_peered":"2024-09-21T10:10:57.397441+0000","last_unstale":"2024-09-21T10:21:25.635350+0000","last_undegraded":"2024-09-21T10:21:25.635350+0000","last_fullsized":"2024-09-21T10:21:25.635350+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:04.043917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:04.043917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:36:59.965868+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":254,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.593180+0000","last_change":"2024-09-21T10:19:59.628742+0000","last_active":"2024-09-21T10:21:25.593180+0000","last_peered":"2024-09-21T10:21:25.593180+0000","last_clean":"2024-09-21T10:21:25.593180+0000","last_became_active":"2024-09-21T10:19:59.610246+0000","last_became_peered":"2024-09-21T10:19:59.610246+0000","last_unstale":"2024-09-21T10:21:25.593180+0000","last_undegraded":"2024-09-21T10:21:25.593180+0000","last_fullsized":"2024-09-21T10:21:25.593180+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:55:40.109353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":110,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.582222+0000","last_change":"2024-09-21T10:19:59.621675+0000","last_active":"2024-09-21T10:21:24.582222+0000","last_peered":"2024-09-21T10:21:24.582222+0000","last_clean":"2024-09-21T10:21:24.582222+0000","last_became_active":"2024-09-21T10:19:59.621372+0000","last_became_peered":"2024-09-21T10:19:59.621372+0000","last_unstale":"2024-09-21T10:21:24.582222+0000","last_undegraded":"2024-09-21T10:21:24.582222+0000","last_fullsized":"2024-09-21T10:21:24.582222+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:59.460712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":511,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.593918+0000","last_change":"2024-09-21T10:14:05.477875+0000","last_active":"2024-09-21T10:21:25.593918+0000","last_peered":"2024-09-21T10:21:25.593918+0000","last_clean":"2024-09-21T10:21:25.593918+0000","last_became_active":"2024-09-21T10:14:05.477192+0000","last_became_peered":"2024-09-21T10:14:05.477192+0000","last_unstale":"2024-09-21T10:21:25.593918+0000","last_undegraded":"2024-09-21T10:21:25.593918+0000","last_fullsized":"2024-09-21T10:21:25.593918+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:19:47.345673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.b","version":"394'13128","reported_seq":13195,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:27.469387+0000","last_change":"2024-09-21T10:18:13.039421+0000","last_active":"2024-09-21T10:21:27.469387+0000","last_peered":"2024-09-21T10:21:27.469387+0000","last_clean":"2024-09-21T10:21:27.469387+0000","last_became_active":"2024-09-21T10:18:13.039085+0000","last_became_peered":"2024-09-21T10:18:13.039085+0000","last_unstale":"2024-09-21T10:21:27.469387+0000","last_undegraded":"2024-09-21T10:21:27.469387+0000","last_fullsized":"2024-09-21T10:21:27.469387+0000","mapping_epoch":367,"log_start":"377'7048","ondisk_log_start":"377'7048","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":6080,"log_dups_size":0,"ondisk_log_size":6080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:40:39.049595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183762944,"num_objects":2804,"num_object_clones":0,"num_object_copies":8412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13128,"num_write_kb":509824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":184,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.642364+0000","last_change":"2024-09-21T10:20:02.461628+0000","last_active":"2024-09-21T10:21:25.642364+0000","last_peered":"2024-09-21T10:21:25.642364+0000","last_clean":"2024-09-21T10:21:25.642364+0000","last_became_active":"2024-09-21T10:19:59.607313+0000","last_became_peered":"2024-09-21T10:19:59.607313+0000","last_unstale":"2024-09-21T10:21:25.642364+0000","last_undegraded":"2024-09-21T10:21:25.642364+0000","last_fullsized":"2024-09-21T10:21:25.642364+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_deep_scrub":"77'100","last_deep_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.242073+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T15:03:19.072397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":6,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":354,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.642106+0000","last_change":"2024-09-21T10:15:01.489326+0000","last_active":"2024-09-21T10:21:25.642106+0000","last_peered":"2024-09-21T10:21:25.642106+0000","last_clean":"2024-09-21T10:21:25.642106+0000","last_became_active":"2024-09-21T10:15:01.489070+0000","last_became_peered":"2024-09-21T10:15:01.489070+0000","last_unstale":"2024-09-21T10:21:25.642106+0000","last_undegraded":"2024-09-21T10:21:25.642106+0000","last_fullsized":"2024-09-21T10:21:25.642106+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:27:41.788029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":567,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.594018+0000","last_change":"2024-09-21T10:14:05.475970+0000","last_active":"2024-09-21T10:21:25.594018+0000","last_peered":"2024-09-21T10:21:25.594018+0000","last_clean":"2024-09-21T10:21:25.594018+0000","last_became_active":"2024-09-21T10:14:05.475331+0000","last_became_peered":"2024-09-21T10:14:05.475331+0000","last_unstale":"2024-09-21T10:21:25.594018+0000","last_undegraded":"2024-09-21T10:21:25.594018+0000","last_fullsized":"2024-09-21T10:21:25.594018+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:17.069736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":503,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.593498+0000","last_change":"2024-09-21T10:18:13.039172+0000","last_active":"2024-09-21T10:21:25.593498+0000","last_peered":"2024-09-21T10:21:25.593498+0000","last_clean":"2024-09-21T10:21:25.593498+0000","last_became_active":"2024-09-21T10:18:13.038756+0000","last_became_peered":"2024-09-21T10:18:13.038756+0000","last_unstale":"2024-09-21T10:21:25.593498+0000","last_undegraded":"2024-09-21T10:21:25.593498+0000","last_fullsized":"2024-09-21T10:21:25.593498+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:06.668926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.a","version":"394'13144","reported_seq":13211,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:28.595666+0000","last_change":"2024-09-21T10:18:12.633418+0000","last_active":"2024-09-21T10:21:28.595666+0000","last_peered":"2024-09-21T10:21:28.595666+0000","last_clean":"2024-09-21T10:21:28.595666+0000","last_became_active":"2024-09-21T10:18:12.633151+0000","last_became_peered":"2024-09-21T10:18:12.633151+0000","last_unstale":"2024-09-21T10:21:28.595666+0000","last_undegraded":"2024-09-21T10:21:28.595666+0000","last_fullsized":"2024-09-21T10:21:28.595666+0000","mapping_epoch":367,"log_start":"374'5183","ondisk_log_start":"374'5183","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":7961,"log_dups_size":0,"ondisk_log_size":7961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:49.740617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177602560,"num_objects":2710,"num_object_clones":0,"num_object_copies":8130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13144,"num_write_kb":507328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":276,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.635482+0000","last_change":"2024-09-21T10:20:01.397078+0000","last_active":"2024-09-21T10:21:25.635482+0000","last_peered":"2024-09-21T10:21:25.635482+0000","last_clean":"2024-09-21T10:21:25.635482+0000","last_became_active":"2024-09-21T10:19:59.623189+0000","last_became_peered":"2024-09-21T10:19:59.623189+0000","last_unstale":"2024-09-21T10:21:25.635482+0000","last_undegraded":"2024-09-21T10:21:25.635482+0000","last_fullsized":"2024-09-21T10:21:25.635482+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:20:01.397007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:20:01.397007+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:15:08.855378+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":499,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.593939+0000","last_change":"2024-09-21T10:14:05.478152+0000","last_active":"2024-09-21T10:21:25.593939+0000","last_peered":"2024-09-21T10:21:25.593939+0000","last_clean":"2024-09-21T10:21:25.593939+0000","last_became_active":"2024-09-21T10:14:05.477547+0000","last_became_peered":"2024-09-21T10:14:05.477547+0000","last_unstale":"2024-09-21T10:21:25.593939+0000","last_undegraded":"2024-09-21T10:21:25.593939+0000","last_fullsized":"2024-09-21T10:21:25.593939+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:34.646046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":288,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.576573+0000","last_change":"2024-09-21T10:14:30.708090+0000","last_active":"2024-09-21T10:21:24.576573+0000","last_peered":"2024-09-21T10:21:24.576573+0000","last_clean":"2024-09-21T10:21:24.576573+0000","last_became_active":"2024-09-21T10:14:30.707266+0000","last_became_peered":"2024-09-21T10:14:30.707266+0000","last_unstale":"2024-09-21T10:21:24.576573+0000","last_undegraded":"2024-09-21T10:21:24.576573+0000","last_fullsized":"2024-09-21T10:21:24.576573+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:38.474537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":561,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.635681+0000","last_change":"2024-09-21T10:10:57.397251+0000","last_active":"2024-09-21T10:21:25.635681+0000","last_peered":"2024-09-21T10:21:25.635681+0000","last_clean":"2024-09-21T10:21:25.635681+0000","last_became_active":"2024-09-21T10:10:57.396271+0000","last_became_peered":"2024-09-21T10:10:57.396271+0000","last_unstale":"2024-09-21T10:21:25.635681+0000","last_undegraded":"2024-09-21T10:21:25.635681+0000","last_fullsized":"2024-09-21T10:21:25.635681+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:17.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.9","version":"394'13149","reported_seq":13216,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:27.686989+0000","last_change":"2024-09-21T10:18:13.037459+0000","last_active":"2024-09-21T10:21:27.686989+0000","last_peered":"2024-09-21T10:21:27.686989+0000","last_clean":"2024-09-21T10:21:27.686989+0000","last_became_active":"2024-09-21T10:18:13.037118+0000","last_became_peered":"2024-09-21T10:18:13.037118+0000","last_unstale":"2024-09-21T10:21:27.686989+0000","last_undegraded":"2024-09-21T10:21:27.686989+0000","last_fullsized":"2024-09-21T10:21:27.686989+0000","mapping_epoch":367,"log_start":"376'5984","ondisk_log_start":"376'5984","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":7165,"log_dups_size":0,"ondisk_log_size":7165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:20.501459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187367424,"num_objects":2859,"num_object_clones":0,"num_object_copies":8577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13149,"num_write_kb":512256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":110,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.581685+0000","last_change":"2024-09-21T10:19:59.613655+0000","last_active":"2024-09-21T10:21:24.581685+0000","last_peered":"2024-09-21T10:21:24.581685+0000","last_clean":"2024-09-21T10:21:24.581685+0000","last_became_active":"2024-09-21T10:19:59.612527+0000","last_became_peered":"2024-09-21T10:19:59.612527+0000","last_unstale":"2024-09-21T10:21:24.581685+0000","last_undegraded":"2024-09-21T10:21:24.581685+0000","last_fullsized":"2024-09-21T10:21:24.581685+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:59.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:59.099852+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:54:56.118470+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":653,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.642595+0000","last_change":"2024-09-21T10:09:23.841186+0000","last_active":"2024-09-21T10:21:25.642595+0000","last_peered":"2024-09-21T10:21:25.642595+0000","last_clean":"2024-09-21T10:21:25.642595+0000","last_became_active":"2024-09-21T10:09:23.840854+0000","last_became_peered":"2024-09-21T10:09:23.840854+0000","last_unstale":"2024-09-21T10:21:25.642595+0000","last_undegraded":"2024-09-21T10:21:25.642595+0000","last_fullsized":"2024-09-21T10:21:25.642595+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:29.666358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":513,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.594291+0000","last_change":"2024-09-21T10:19:59.619479+0000","last_active":"2024-09-21T10:21:25.594291+0000","last_peered":"2024-09-21T10:21:25.594291+0000","last_clean":"2024-09-21T10:21:25.594291+0000","last_became_active":"2024-09-21T10:19:59.619005+0000","last_became_peered":"2024-09-21T10:19:59.619005+0000","last_unstale":"2024-09-21T10:21:25.594291+0000","last_undegraded":"2024-09-21T10:21:25.594291+0000","last_fullsized":"2024-09-21T10:21:25.594291+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:39.005877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":650,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.640831+0000","last_change":"2024-09-21T10:09:49.926548+0000","last_active":"2024-09-21T10:21:25.640831+0000","last_peered":"2024-09-21T10:21:25.640831+0000","last_clean":"2024-09-21T10:21:25.640831+0000","last_became_active":"2024-09-21T10:09:49.926255+0000","last_became_peered":"2024-09-21T10:09:49.926255+0000","last_unstale":"2024-09-21T10:21:25.640831+0000","last_undegraded":"2024-09-21T10:21:25.640831+0000","last_fullsized":"2024-09-21T10:21:25.640831+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:49.855878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.8","version":"394'13308","reported_seq":26762,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:28.656212+0000","last_change":"2024-09-21T10:20:35.620221+0000","last_active":"2024-09-21T10:21:28.656212+0000","last_peered":"2024-09-21T10:21:28.656212+0000","last_clean":"2024-09-21T10:21:28.656212+0000","last_became_active":"2024-09-21T10:20:35.605898+0000","last_became_peered":"2024-09-21T10:20:35.605898+0000","last_unstale":"2024-09-21T10:21:28.656212+0000","last_undegraded":"2024-09-21T10:21:28.656212+0000","last_fullsized":"2024-09-21T10:21:28.656212+0000","mapping_epoch":385,"log_start":"374'5385","ondisk_log_start":"374'5385","created":367,"last_epoch_clean":386,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:18:11.615158+0000","last_clean_scrub_stamp":"2024-09-21T10:18:11.615158+0000","objects_scrubbed":0,"log_size":7923,"log_dups_size":0,"ondisk_log_size":7923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:45.570538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182976512,"num_objects":2792,"num_object_clones":0,"num_object_copies":8376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13308,"num_write_kb":515200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6685,"num_bytes_recovered":438108160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":453,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.641633+0000","last_change":"2024-09-21T10:10:57.390804+0000","last_active":"2024-09-21T10:21:25.641633+0000","last_peered":"2024-09-21T10:21:25.641633+0000","last_clean":"2024-09-21T10:21:25.641633+0000","last_became_active":"2024-09-21T10:10:57.390323+0000","last_became_peered":"2024-09-21T10:10:57.390323+0000","last_unstale":"2024-09-21T10:21:25.641633+0000","last_undegraded":"2024-09-21T10:21:25.641633+0000","last_fullsized":"2024-09-21T10:21:25.641633+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.005477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.005477+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:12.788956+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":307,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.635448+0000","last_change":"2024-09-21T10:14:30.718942+0000","last_active":"2024-09-21T10:21:25.635448+0000","last_peered":"2024-09-21T10:21:25.635448+0000","last_clean":"2024-09-21T10:21:25.635448+0000","last_became_active":"2024-09-21T10:14:30.709129+0000","last_became_peered":"2024-09-21T10:14:30.709129+0000","last_unstale":"2024-09-21T10:21:25.635448+0000","last_undegraded":"2024-09-21T10:21:25.635448+0000","last_fullsized":"2024-09-21T10:21:25.635448+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:18.171064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":206,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.573794+0000","last_change":"2024-09-21T10:19:59.616263+0000","last_active":"2024-09-21T10:21:24.573794+0000","last_peered":"2024-09-21T10:21:24.573794+0000","last_clean":"2024-09-21T10:21:24.573794+0000","last_became_active":"2024-09-21T10:19:59.612264+0000","last_became_peered":"2024-09-21T10:19:59.612264+0000","last_unstale":"2024-09-21T10:21:24.573794+0000","last_undegraded":"2024-09-21T10:21:24.573794+0000","last_fullsized":"2024-09-21T10:21:24.573794+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:48:19.569645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":110,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.581520+0000","last_change":"2024-09-21T10:19:59.621093+0000","last_active":"2024-09-21T10:21:24.581520+0000","last_peered":"2024-09-21T10:21:24.581520+0000","last_clean":"2024-09-21T10:21:24.581520+0000","last_became_active":"2024-09-21T10:19:59.620767+0000","last_became_peered":"2024-09-21T10:19:59.620767+0000","last_unstale":"2024-09-21T10:21:24.581520+0000","last_undegraded":"2024-09-21T10:21:24.581520+0000","last_fullsized":"2024-09-21T10:21:24.581520+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:23.520678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":110,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.582293+0000","last_change":"2024-09-21T10:19:59.612435+0000","last_active":"2024-09-21T10:21:24.582293+0000","last_peered":"2024-09-21T10:21:24.582293+0000","last_clean":"2024-09-21T10:21:24.582293+0000","last_became_active":"2024-09-21T10:19:59.612099+0000","last_became_peered":"2024-09-21T10:19:59.612099+0000","last_unstale":"2024-09-21T10:21:24.582293+0000","last_undegraded":"2024-09-21T10:21:24.582293+0000","last_fullsized":"2024-09-21T10:21:24.582293+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:33:32.837915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":130,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.580940+0000","last_change":"2024-09-21T10:19:59.631237+0000","last_active":"2024-09-21T10:21:24.580940+0000","last_peered":"2024-09-21T10:21:24.580940+0000","last_clean":"2024-09-21T10:21:24.580940+0000","last_became_active":"2024-09-21T10:19:59.622054+0000","last_became_peered":"2024-09-21T10:19:59.622054+0000","last_unstale":"2024-09-21T10:21:24.580940+0000","last_undegraded":"2024-09-21T10:21:24.580940+0000","last_fullsized":"2024-09-21T10:21:24.580940+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:22:19.271411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14232,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":565,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.593270+0000","last_change":"2024-09-21T10:19:59.609857+0000","last_active":"2024-09-21T10:21:25.593270+0000","last_peered":"2024-09-21T10:21:25.593270+0000","last_clean":"2024-09-21T10:21:25.593270+0000","last_became_active":"2024-09-21T10:19:59.609482+0000","last_became_peered":"2024-09-21T10:19:59.609482+0000","last_unstale":"2024-09-21T10:21:25.593270+0000","last_undegraded":"2024-09-21T10:21:25.593270+0000","last_fullsized":"2024-09-21T10:21:25.593270+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:57:20.562696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":301,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.596798+0000","last_change":"2024-09-21T10:14:30.713873+0000","last_active":"2024-09-21T10:21:25.596798+0000","last_peered":"2024-09-21T10:21:25.596798+0000","last_clean":"2024-09-21T10:21:25.596798+0000","last_became_active":"2024-09-21T10:14:30.703149+0000","last_became_peered":"2024-09-21T10:14:30.703149+0000","last_unstale":"2024-09-21T10:21:25.596798+0000","last_undegraded":"2024-09-21T10:21:25.596798+0000","last_fullsized":"2024-09-21T10:21:25.596798+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:26.453902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'100","reported_seq":452,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.596912+0000","last_change":"2024-09-21T10:18:12.633959+0000","last_active":"2024-09-21T10:21:25.596912+0000","last_peered":"2024-09-21T10:21:25.596912+0000","last_clean":"2024-09-21T10:21:25.596912+0000","last_became_active":"2024-09-21T10:18:12.633591+0000","last_became_peered":"2024-09-21T10:18:12.633591+0000","last_unstale":"2024-09-21T10:21:25.596912+0000","last_undegraded":"2024-09-21T10:21:25.596912+0000","last_fullsized":"2024-09-21T10:21:25.596912+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.014993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.014993+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:39:07.692739+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":307,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.574508+0000","last_change":"2024-09-21T10:15:01.483814+0000","last_active":"2024-09-21T10:21:24.574508+0000","last_peered":"2024-09-21T10:21:24.574508+0000","last_clean":"2024-09-21T10:21:24.574508+0000","last_became_active":"2024-09-21T10:15:01.483522+0000","last_became_peered":"2024-09-21T10:15:01.483522+0000","last_unstale":"2024-09-21T10:21:24.574508+0000","last_undegraded":"2024-09-21T10:21:24.574508+0000","last_fullsized":"2024-09-21T10:21:24.574508+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:13.632066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":642,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.640880+0000","last_change":"2024-09-21T10:19:59.607487+0000","last_active":"2024-09-21T10:21:25.640880+0000","last_peered":"2024-09-21T10:21:25.640880+0000","last_clean":"2024-09-21T10:21:25.640880+0000","last_became_active":"2024-09-21T10:19:59.607170+0000","last_became_peered":"2024-09-21T10:19:59.607170+0000","last_unstale":"2024-09-21T10:21:25.640880+0000","last_undegraded":"2024-09-21T10:21:25.640880+0000","last_fullsized":"2024-09-21T10:21:25.640880+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:54.173188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":536,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.635961+0000","last_change":"2024-09-21T10:10:57.397755+0000","last_active":"2024-09-21T10:21:25.635961+0000","last_peered":"2024-09-21T10:21:25.635961+0000","last_clean":"2024-09-21T10:21:25.635961+0000","last_became_active":"2024-09-21T10:10:57.397249+0000","last_became_peered":"2024-09-21T10:10:57.397249+0000","last_unstale":"2024-09-21T10:21:25.635961+0000","last_undegraded":"2024-09-21T10:21:25.635961+0000","last_fullsized":"2024-09-21T10:21:25.635961+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:40.813503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":283,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.592986+0000","last_change":"2024-09-21T10:19:59.619342+0000","last_active":"2024-09-21T10:21:25.592986+0000","last_peered":"2024-09-21T10:21:25.592986+0000","last_clean":"2024-09-21T10:21:25.592986+0000","last_became_active":"2024-09-21T10:19:59.618878+0000","last_became_peered":"2024-09-21T10:19:59.618878+0000","last_unstale":"2024-09-21T10:21:25.592986+0000","last_undegraded":"2024-09-21T10:21:25.592986+0000","last_fullsized":"2024-09-21T10:21:25.592986+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:09:12.480675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":90,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.594250+0000","last_change":"2024-09-21T10:19:59.605836+0000","last_active":"2024-09-21T10:21:25.594250+0000","last_peered":"2024-09-21T10:21:25.594250+0000","last_clean":"2024-09-21T10:21:25.594250+0000","last_became_active":"2024-09-21T10:19:59.605505+0000","last_became_peered":"2024-09-21T10:19:59.605505+0000","last_unstale":"2024-09-21T10:21:25.594250+0000","last_undegraded":"2024-09-21T10:21:25.594250+0000","last_fullsized":"2024-09-21T10:21:25.594250+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:20.144205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":656,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.640873+0000","last_change":"2024-09-21T10:09:49.928053+0000","last_active":"2024-09-21T10:21:25.640873+0000","last_peered":"2024-09-21T10:21:25.640873+0000","last_clean":"2024-09-21T10:21:25.640873+0000","last_became_active":"2024-09-21T10:09:49.927231+0000","last_became_peered":"2024-09-21T10:09:49.927231+0000","last_unstale":"2024-09-21T10:21:25.640873+0000","last_undegraded":"2024-09-21T10:21:25.640873+0000","last_fullsized":"2024-09-21T10:21:25.640873+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:26.987553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":569,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.635586+0000","last_change":"2024-09-21T10:09:23.840717+0000","last_active":"2024-09-21T10:21:25.635586+0000","last_peered":"2024-09-21T10:21:25.635586+0000","last_clean":"2024-09-21T10:21:25.635586+0000","last_became_active":"2024-09-21T10:09:23.840323+0000","last_became_peered":"2024-09-21T10:09:23.840323+0000","last_unstale":"2024-09-21T10:21:25.635586+0000","last_undegraded":"2024-09-21T10:21:25.635586+0000","last_fullsized":"2024-09-21T10:21:25.635586+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:55.769770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":289,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.596450+0000","last_change":"2024-09-21T10:18:12.632498+0000","last_active":"2024-09-21T10:21:25.596450+0000","last_peered":"2024-09-21T10:21:25.596450+0000","last_clean":"2024-09-21T10:21:25.596450+0000","last_became_active":"2024-09-21T10:18:12.631648+0000","last_became_peered":"2024-09-21T10:18:12.631648+0000","last_unstale":"2024-09-21T10:21:25.596450+0000","last_undegraded":"2024-09-21T10:21:25.596450+0000","last_fullsized":"2024-09-21T10:21:25.596450+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:17.592618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":634,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.641958+0000","last_change":"2024-09-21T10:14:30.702101+0000","last_active":"2024-09-21T10:21:25.641958+0000","last_peered":"2024-09-21T10:21:25.641958+0000","last_clean":"2024-09-21T10:21:25.641958+0000","last_became_active":"2024-09-21T10:14:30.701783+0000","last_became_peered":"2024-09-21T10:14:30.701783+0000","last_unstale":"2024-09-21T10:21:25.641958+0000","last_undegraded":"2024-09-21T10:21:25.641958+0000","last_fullsized":"2024-09-21T10:21:25.641958+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:13.087010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":100,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.581310+0000","last_change":"2024-09-21T10:14:30.707503+0000","last_active":"2024-09-21T10:21:24.581310+0000","last_peered":"2024-09-21T10:21:24.581310+0000","last_clean":"2024-09-21T10:21:24.581310+0000","last_became_active":"2024-09-21T10:14:30.706881+0000","last_became_peered":"2024-09-21T10:14:30.706881+0000","last_unstale":"2024-09-21T10:21:24.581310+0000","last_undegraded":"2024-09-21T10:21:24.581310+0000","last_fullsized":"2024-09-21T10:21:24.581310+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:23:07.485461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":674,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.597052+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:21:25.597052+0000","last_peered":"2024-09-21T10:21:25.597052+0000","last_clean":"2024-09-21T10:21:25.597052+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:21:25.597052+0000","last_undegraded":"2024-09-21T10:21:25.597052+0000","last_fullsized":"2024-09-21T10:21:25.597052+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":298,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.635219+0000","last_change":"2024-09-21T10:14:30.702414+0000","last_active":"2024-09-21T10:21:25.635219+0000","last_peered":"2024-09-21T10:21:25.635219+0000","last_clean":"2024-09-21T10:21:25.635219+0000","last_became_active":"2024-09-21T10:14:30.702009+0000","last_became_peered":"2024-09-21T10:14:30.702009+0000","last_unstale":"2024-09-21T10:21:25.635219+0000","last_undegraded":"2024-09-21T10:21:25.635219+0000","last_fullsized":"2024-09-21T10:21:25.635219+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:50:48.396082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":36,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.581856+0000","last_change":"2024-09-21T10:18:12.633151+0000","last_active":"2024-09-21T10:21:24.581856+0000","last_peered":"2024-09-21T10:21:24.581856+0000","last_clean":"2024-09-21T10:21:24.581856+0000","last_became_active":"2024-09-21T10:18:12.632854+0000","last_became_peered":"2024-09-21T10:18:12.632854+0000","last_unstale":"2024-09-21T10:21:24.581856+0000","last_undegraded":"2024-09-21T10:21:24.581856+0000","last_fullsized":"2024-09-21T10:21:24.581856+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:23:14.602893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":104,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.581285+0000","last_change":"2024-09-21T10:14:30.720787+0000","last_active":"2024-09-21T10:21:24.581285+0000","last_peered":"2024-09-21T10:21:24.581285+0000","last_clean":"2024-09-21T10:21:24.581285+0000","last_became_active":"2024-09-21T10:14:30.706001+0000","last_became_peered":"2024-09-21T10:14:30.706001+0000","last_unstale":"2024-09-21T10:21:24.581285+0000","last_undegraded":"2024-09-21T10:21:24.581285+0000","last_fullsized":"2024-09-21T10:21:24.581285+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:40:11.987108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":36,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.581133+0000","last_change":"2024-09-21T10:18:12.636252+0000","last_active":"2024-09-21T10:21:24.581133+0000","last_peered":"2024-09-21T10:21:24.581133+0000","last_clean":"2024-09-21T10:21:24.581133+0000","last_became_active":"2024-09-21T10:18:12.635945+0000","last_became_peered":"2024-09-21T10:18:12.635945+0000","last_unstale":"2024-09-21T10:21:24.581133+0000","last_undegraded":"2024-09-21T10:21:24.581133+0000","last_fullsized":"2024-09-21T10:21:24.581133+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:59:29.464045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":118,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.581775+0000","last_change":"2024-09-21T10:19:59.633102+0000","last_active":"2024-09-21T10:21:24.581775+0000","last_peered":"2024-09-21T10:21:24.581775+0000","last_clean":"2024-09-21T10:21:24.581775+0000","last_became_active":"2024-09-21T10:19:59.622345+0000","last_became_peered":"2024-09-21T10:19:59.622345+0000","last_unstale":"2024-09-21T10:21:24.581775+0000","last_undegraded":"2024-09-21T10:21:24.581775+0000","last_fullsized":"2024-09-21T10:21:24.581775+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:58:02.596213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":657,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.641493+0000","last_change":"2024-09-21T10:09:23.840467+0000","last_active":"2024-09-21T10:21:25.641493+0000","last_peered":"2024-09-21T10:21:25.641493+0000","last_clean":"2024-09-21T10:21:25.641493+0000","last_became_active":"2024-09-21T10:09:23.840177+0000","last_became_peered":"2024-09-21T10:09:23.840177+0000","last_unstale":"2024-09-21T10:21:25.641493+0000","last_undegraded":"2024-09-21T10:21:25.641493+0000","last_fullsized":"2024-09-21T10:21:25.641493+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:17.887307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":654,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.642647+0000","last_change":"2024-09-21T10:19:59.607627+0000","last_active":"2024-09-21T10:21:25.642647+0000","last_peered":"2024-09-21T10:21:25.642647+0000","last_clean":"2024-09-21T10:21:25.642647+0000","last_became_active":"2024-09-21T10:19:59.607290+0000","last_became_peered":"2024-09-21T10:19:59.607290+0000","last_unstale":"2024-09-21T10:21:25.642647+0000","last_undegraded":"2024-09-21T10:21:25.642647+0000","last_fullsized":"2024-09-21T10:21:25.642647+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:50.196482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":110,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-09-21T10:21:24.580773+0000","last_change":"2024-09-21T10:19:59.621829+0000","last_active":"2024-09-21T10:21:24.580773+0000","last_peered":"2024-09-21T10:21:24.580773+0000","last_clean":"2024-09-21T10:21:24.580773+0000","last_became_active":"2024-09-21T10:19:59.620438+0000","last_became_peered":"2024-09-21T10:19:59.620438+0000","last_unstale":"2024-09-21T10:21:24.580773+0000","last_undegraded":"2024-09-21T10:21:24.580773+0000","last_fullsized":"2024-09-21T10:21:24.580773+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:21:31.831558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":311,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-09-21T10:21:25.635740+0000","last_change":"2024-09-21T10:10:57.401588+0000","last_active":"2024-09-21T10:21:25.635740+0000","last_peered":"2024-09-21T10:21:25.635740+0000","last_clean":"2024-09-21T10:21:25.635740+0000","last_became_active":"2024-09-21T10:10:57.397724+0000","last_became_peered":"2024-09-21T10:10:57.397724+0000","last_unstale":"2024-09-21T10:21:25.635740+0000","last_undegraded":"2024-09-21T10:21:25.635740+0000","last_fullsized":"2024-09-21T10:21:25.635740+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:47.843630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":16,"stat_sum":{"num_bytes":2942435352,"num_objects":44899,"num_object_clones":0,"num_object_copies":134697,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1932,"num_objects_unfound":0,"num_objects_dirty":44899,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":210911,"num_write_kb":8185857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22376,"num_bytes_recovered":1466302464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9804853248,"data_stored":9804841032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107405,"ondisk_log_size":107405,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":16670,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":21404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26580,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":6,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":879891,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2109440,"data_stored":2098784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":0,"seq":180388626949,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019407,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3134920,"kb_used_data":1644672,"kb_used_omap":1,"kb_used_meta":1490238,"kb_avail":90610232,"statfs":{"total":95995035648,"available":92784877568,"internally_reserved":0,"allocated":1684144128,"data_stored":1681691038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1526004056},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":245,"seq":1052266987666,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":985848,"kb_used_data":548080,"kb_used_omap":8,"kb_used_meta":437751,"kb_avail":92759304,"statfs":{"total":95995035648,"available":94985527296,"internally_reserved":0,"allocated":561233920,"data_stored":558792955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8940,"internal_metadata":448257300},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":175,"seq":751619277015,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2041036,"kb_used_data":1107840,"kb_used_omap":12,"kb_used_meta":933107,"kb_avail":91704116,"statfs":{"total":95995035648,"available":93905014784,"internally_reserved":0,"allocated":1134428160,"data_stored":1131975066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12845,"internal_metadata":955502035},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823178,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3397052,"kb_used_data":1899244,"kb_used_omap":1,"kb_used_meta":1497790,"kb_avail":90348100,"statfs":{"total":95995035648,"available":92516454400,"internally_reserved":0,"allocated":1944825856,"data_stored":1942388921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1533737304},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879601,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1782136,"kb_used_data":726896,"kb_used_omap":4,"kb_used_meta":1055227,"kb_avail":91963016,"statfs":{"total":95995035648,"available":94170128384,"internally_reserved":0,"allocated":744341504,"data_stored":741900624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":1080552892},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566442,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2765896,"kb_used_data":1630528,"kb_used_omap":6,"kb_used_meta":1135353,"kb_avail":90979256,"statfs":{"total":95995035648,"available":93162758144,"internally_reserved":0,"allocated":1669660672,"data_stored":1667208694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":1162601895},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860787,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2290800,"kb_used_data":1072040,"kb_used_omap":5,"kb_used_meta":1218682,"kb_avail":91454352,"statfs":{"total":95995035648,"available":93649256448,"internally_reserved":0,"allocated":1097768960,"data_stored":1095335288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":1247930607},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7236,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9976,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":551878656,"data_stored":551878656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1083969536,"data_stored":1083965464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1627127808,"data_stored":1627127808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":736952320,"data_stored":736952320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1668091904,"data_stored":1668087832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1915879424,"data_stored":1915879424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1105264640,"data_stored":1105264640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1115688960,"data_stored":1115684888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:21:37.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pgp_num 21 2024-09-21T10:21:37.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:37.249 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:37.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:37.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:37.351 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:37.381 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:37.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:37.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:37.482 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:37.511 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:37.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:37.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:37.572 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:37.586 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:37.613 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:37.640 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:37.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:37.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:37.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:37.768 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:37.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:37.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:37.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:37.902 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:37.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:37.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:38.004 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:38.041 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:38.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:38.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:38.143 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:38.179 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:38.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:38.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:38.281 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:38.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:38.313 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:38.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:38.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:38.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:38.462 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:38.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:38.563 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:38.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:38.610 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:38.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:38.711 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:38.748 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:38.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:38.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:38.773 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:38 smithi107 ceph-mon[102221]: pgmap v1377: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 920 op/s; 1274/113310 objects misplaced (1.124%); 6.8 MiB/s, 108 objects/s recovering 2024-09-21T10:21:38.774 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:38 smithi107 ceph-mon[105022]: pgmap v1377: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 920 op/s; 1274/113310 objects misplaced (1.124%); 6.8 MiB/s, 108 objects/s recovering 2024-09-21T10:21:38.850 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:38.883 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:38.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:38.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:38.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:38 smithi129 ceph-mon[88471]: pgmap v1377: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.3 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 920 op/s; 1274/113310 objects misplaced (1.124%); 6.8 MiB/s, 108 objects/s recovering 2024-09-21T10:21:38.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:39.034 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:39.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:39.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:39.082 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:21:39.083 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:21:39.083 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-09-21T10:21:39.083 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:21:39.136 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:39.169 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:39.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:39.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:39.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:39.310 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:39.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:39.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:39.412 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:39.447 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:39.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:39.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:39.549 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:39.588 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:39.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:39.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:39.667 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:21:39.690 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:39.725 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:39.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:39.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:39.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:39.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:39 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3797313631' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:21:39.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:39 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3797313631' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:21:39.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:39.860 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:39.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:39.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:39 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3797313631' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:21:39.961 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:39.993 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:39.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:39.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:40.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:40.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:40.132 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:40.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:40.233 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:40.273 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:40.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:40.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:40.375 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:40.415 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:40.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:40.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:40.516 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:40.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:40.556 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:40.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:40.650 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:21:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:21:40.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:21:40.657 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:40.722 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:40.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:40.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:40.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:40.852 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:40.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:40.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:40 smithi129 ceph-mon[88471]: pgmap v1378: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 981 op/s; 524/105432 objects misplaced (0.497%); 9.4 MiB/s, 150 objects/s recovering 2024-09-21T10:21:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:40 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:21:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:40 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3797313631' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:21:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:40 smithi129 ceph-mon[88471]: osdmap e395: 8 total, 8 up, 7 in 2024-09-21T10:21:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:40 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:21:40.954 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:40.998 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:40.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:40.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:41.100 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:41.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:40 smithi107 ceph-mon[105022]: pgmap v1378: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 981 op/s; 524/105432 objects misplaced (0.497%); 9.4 MiB/s, 150 objects/s recovering 2024-09-21T10:21:41.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:40 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:21:41.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:40 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3797313631' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:21:41.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:40 smithi107 ceph-mon[105022]: osdmap e395: 8 total, 8 up, 7 in 2024-09-21T10:21:41.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:40 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:21:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:40 smithi107 ceph-mon[102221]: pgmap v1378: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 981 op/s; 524/105432 objects misplaced (0.497%); 9.4 MiB/s, 150 objects/s recovering 2024-09-21T10:21:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:40 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:21:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:40 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3797313631' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:21:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:40 smithi107 ceph-mon[102221]: osdmap e395: 8 total, 8 up, 7 in 2024-09-21T10:21:41.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:40 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:21:41.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:41.139 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:41.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:41.240 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:41.262 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:41.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:41.364 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:41.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:41.411 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:41.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:41.512 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:41.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:41.552 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:41.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:41.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:41.682 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:41.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:41.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:41.783 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:41.823 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:41.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:41.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:41.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:41.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:41.958 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:41.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:41.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:42.012 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:42.012 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:42.012 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:21:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:21:41] "GET /metrics HTTP/1.1" 200 36804 "" "Prometheus/2.51.0" 2024-09-21T10:21:42.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:21:42.060 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:42.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:42.094 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:42.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:42.195 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:42.237 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:42.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:42.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:42.338 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:42.421 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:42.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:42.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:42.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:42.563 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:42.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:42.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:42.695 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:42.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:42.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:42.752 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:42 smithi107 ceph-mon[102221]: pgmap v1380: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 977 op/s; 524/105432 objects misplaced (0.497%); 7.0 MiB/s, 111 objects/s recovering 2024-09-21T10:21:42.752 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:42 smithi107 ceph-mon[105022]: pgmap v1380: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 977 op/s; 524/105432 objects misplaced (0.497%); 7.0 MiB/s, 111 objects/s recovering 2024-09-21T10:21:42.797 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:42.828 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:42.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:42.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:42.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:42 smithi129 ceph-mon[88471]: pgmap v1380: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 977 op/s; 524/105432 objects misplaced (0.497%); 7.0 MiB/s, 111 objects/s recovering 2024-09-21T10:21:42.930 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:42.970 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:42.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:42.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:43.072 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:43.107 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:43.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:43.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:43.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:43.238 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:43.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:43.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:43.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:43.382 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:43.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:43.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:43.484 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:43.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:43.514 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:43.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:43.615 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:43.661 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:43.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:43.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:43.763 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:43.816 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:43.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:43.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:43.918 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:43.956 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:43.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:43.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:44.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:44.080 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:44.103 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:44.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:44.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:44.138 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:21:44.138 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:21:44.170 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:21:44.170 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:21:44.205 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:44.206 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:21:44.207 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:21:44.238 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:44.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:44.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:44.239 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:21:44.239 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:21:44.274 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:21:44.274 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:21:44.309 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:21:44.309 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:21:44.338 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:21:44.339 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:21:44.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:44.378 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:21:44.378 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:21:44.389 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:44.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:44.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:44.492 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:44.535 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:44.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:44.637 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:44 smithi129 ceph-mon[88471]: pgmap v1381: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.9 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 955 op/s; 524/95598 objects misplaced (0.548%); 7.0 MiB/s, 111 objects/s recovering 2024-09-21T10:21:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:44 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/518928642' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:21:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:44 smithi129 ceph-mon[88471]: osdmap e396: 8 total, 8 up, 7 in 2024-09-21T10:21:44.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:44.670 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:44.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:44.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:44.796 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:44.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:44.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:44.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:44 smithi107 ceph-mon[102221]: pgmap v1381: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.9 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 955 op/s; 524/95598 objects misplaced (0.548%); 7.0 MiB/s, 111 objects/s recovering 2024-09-21T10:21:44.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:44 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/518928642' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:21:44.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:44 smithi107 ceph-mon[102221]: osdmap e396: 8 total, 8 up, 7 in 2024-09-21T10:21:44.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:44 smithi107 ceph-mon[105022]: pgmap v1381: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.9 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 955 op/s; 524/95598 objects misplaced (0.548%); 7.0 MiB/s, 111 objects/s recovering 2024-09-21T10:21:44.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:44 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/518928642' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:21:44.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:44 smithi107 ceph-mon[105022]: osdmap e396: 8 total, 8 up, 7 in 2024-09-21T10:21:44.898 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:44.922 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:44.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:44.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:44.995 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:21:45.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:45.059 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:45.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:45.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:45.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:45.182 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:45.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:45.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:45.284 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:45.305 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:45.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:45.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:45.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:45.430 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:45.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:45.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:45.532 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:45.553 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:45.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:45.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:45.635 INFO:teuthology.orchestra.run.smithi107.stderr:set pool 1 pgp_num to 21 2024-09-21T10:21:45.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:45.680 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:45.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:45.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:45.782 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:45.803 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:45.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:45.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:45.826 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:45 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/420414908' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T10:21:45.827 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:45 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T10:21:45.827 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:45 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-21T10:21:45.827 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:45 smithi107 ceph-mon[105022]: osdmap e397: 8 total, 8 up, 7 in 2024-09-21T10:21:45.827 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:45 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/420414908' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T10:21:45.827 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:45 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T10:21:45.827 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:45 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-21T10:21:45.827 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:45 smithi107 ceph-mon[102221]: osdmap e397: 8 total, 8 up, 7 in 2024-09-21T10:21:45.905 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:45.926 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:45.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:45.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:46.028 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:46.029 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:45 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/420414908' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T10:21:46.029 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:45 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-21T10:21:46.029 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:45 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-21T10:21:46.029 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:45 smithi129 ceph-mon[88471]: osdmap e397: 8 total, 8 up, 7 in 2024-09-21T10:21:46.056 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:46.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:46.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:46.157 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:46.194 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:46.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:46.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:46.296 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:46.341 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:46.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:46.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:46.443 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:46.473 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:46.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:46.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:46.575 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:46.622 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:46.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:46.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:46.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:46.760 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:46.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:46.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:46.862 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:46.905 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:46.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:46.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:47.007 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:47.039 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:47.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:47.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:47.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:46 smithi107 ceph-mon[102221]: pgmap v1383: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 917 op/s; 4/91275 objects misplaced (0.004%); 8.9 MiB/s, 143 objects/s recovering 2024-09-21T10:21:47.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:46 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:21:47.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:46 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:21:47.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:46 smithi107 ceph-mon[105022]: pgmap v1383: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 917 op/s; 4/91275 objects misplaced (0.004%); 8.9 MiB/s, 143 objects/s recovering 2024-09-21T10:21:47.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:46 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:21:47.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:46 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:21:47.141 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:47.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:46 smithi129 ceph-mon[88471]: pgmap v1383: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 917 op/s; 4/91275 objects misplaced (0.004%); 8.9 MiB/s, 143 objects/s recovering 2024-09-21T10:21:47.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:46 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:21:47.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:46 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:21:47.177 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:47.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:47.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:47.279 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:47.316 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:47.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:47.417 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:47.449 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:47.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:47.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:47.551 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:47.585 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:47.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:47.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:47.687 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:47.718 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:47.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:47.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:47.820 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:47.846 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:47.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:47.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:47.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:47.978 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:47.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:47.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:48.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:48.100 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:48.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:48.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:48.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:47 smithi107 ceph-mon[102221]: 1.7 scrub starts 2024-09-21T10:21:48.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:47 smithi107 ceph-mon[102221]: 1.7 scrub ok 2024-09-21T10:21:48.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:47 smithi107 ceph-mon[105022]: 1.7 scrub starts 2024-09-21T10:21:48.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:47 smithi107 ceph-mon[105022]: 1.7 scrub ok 2024-09-21T10:21:48.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:47 smithi129 ceph-mon[88471]: 1.7 scrub starts 2024-09-21T10:21:48.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:47 smithi129 ceph-mon[88471]: 1.7 scrub ok 2024-09-21T10:21:48.202 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:48.237 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:48.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:48.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:48.338 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:48.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:48.376 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:48.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:48.477 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:48.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:21:48.507 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:48.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:48.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:48.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:48.646 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:48.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:48.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:48.748 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:48.779 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:48.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:48.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:48.880 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:48.918 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:48.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:48.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:49.020 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:49.053 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:49.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:49.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:49.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:48 smithi107 ceph-mon[102221]: pgmap v1385: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 817 op/s; 4/86247 objects misplaced (0.005%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T10:21:49.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:48 smithi107 ceph-mon[105022]: pgmap v1385: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 817 op/s; 4/86247 objects misplaced (0.005%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T10:21:49.155 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:49.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:48 smithi129 ceph-mon[88471]: pgmap v1385: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 1.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 817 op/s; 4/86247 objects misplaced (0.005%); 3.9 MiB/s, 62 objects/s recovering 2024-09-21T10:21:49.186 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:49.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:49.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:49.288 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:49.328 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:49.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:49.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:49.430 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:49.465 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:49.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:49.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:49.567 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:49.592 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:49.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:49.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:49.693 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:49.725 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:49.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:49.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:49.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:49.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:49.865 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:49.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:49.966 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:50.005 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:50.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:50.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:50.108 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:50.138 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:50.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:50.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:50.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:50.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:50.275 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:50.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:50.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:50.392 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:50.424 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:50.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:50.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:50.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:50.572 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:50.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:50.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:21:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:21:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:21:50.674 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:50.710 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:50.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:50.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:50.811 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:50.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:50.843 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:50.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:50.944 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:50.977 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:50.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:50.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:51.050 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:50 smithi107 ceph-mon[102221]: pgmap v1386: 109 pgs: 109 active+clean; 1.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.07k op/s; 3.8 MiB/s, 61 objects/s recovering 2024-09-21T10:21:51.050 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:50 smithi107 ceph-mon[105022]: pgmap v1386: 109 pgs: 109 active+clean; 1.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.07k op/s; 3.8 MiB/s, 61 objects/s recovering 2024-09-21T10:21:51.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:51.133 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:51.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:51.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:51.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:50 smithi129 ceph-mon[88471]: pgmap v1386: 109 pgs: 109 active+clean; 1.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.07k op/s; 3.8 MiB/s, 61 objects/s recovering 2024-09-21T10:21:51.235 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:51.271 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:51.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:51.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:51.372 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:51.405 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:51.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:51.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:51.507 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:51.540 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:51.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:51.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:51.642 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:51.670 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:51.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:51.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:51.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:51.796 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:21:51.796 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:21:51.843 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:51.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:51.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:51.944 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:51.976 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:51.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:51.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:52.077 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:52.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:21:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:21:51] "GET /metrics HTTP/1.1" 200 36804 "" "Prometheus/2.51.0" 2024-09-21T10:21:52.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:52.113 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:52.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:52.214 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:52.254 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:52.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:52.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:52.356 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:52.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:52.389 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:52.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:52.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:52.528 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:52.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:52.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:52.630 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:52.662 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:52.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:52.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:52.764 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:52.807 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:52.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:52.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:52.908 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:52.942 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:52.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:52.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:52.965 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:52 smithi107 ceph-mon[102221]: pgmap v1387: 109 pgs: 109 active+clean; 1.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 663 op/s; 3.8 MiB/s, 61 objects/s recovering 2024-09-21T10:21:52.965 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:52 smithi107 ceph-mon[105022]: pgmap v1387: 109 pgs: 109 active+clean; 1.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 663 op/s; 3.8 MiB/s, 61 objects/s recovering 2024-09-21T10:21:53.044 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:53.080 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:53.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:53.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:53.087 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:21:53.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:52 smithi129 ceph-mon[88471]: pgmap v1387: 109 pgs: 109 active+clean; 1.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 663 op/s; 3.8 MiB/s, 61 objects/s recovering 2024-09-21T10:21:53.183 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:53.213 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:53.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:53.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:53.315 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:53.349 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:53.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:53.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:53.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:53.498 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:53.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:53.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:53.600 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:53.629 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:53.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:53.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:53.699 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:21:53.730 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:53.764 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:53.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:53.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:53.799 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:53 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/161223445' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:21:53.799 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:53 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/161223445' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:21:53.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:53.899 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:53.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:53.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:54.001 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:54.032 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:54.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:54.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:54.134 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:54.168 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:54.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:54.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:54.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:53 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/161223445' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:21:54.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:54.298 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:54.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:54.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:54.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:54.433 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:54.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:54.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:54.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:54.563 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:54.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:54.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:54.664 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:54.696 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:54.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:54.798 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:54.830 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:54.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:54.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:54.932 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:54.952 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 4] out_osds: [6] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:21:54.952 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:21:54.952 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:21:54.952 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.000000 2024-09-21T10:21:54.952 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 4 0 2024-09-21T10:21:54.962 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:54.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:54.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:54.986 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:54 smithi107 ceph-mon[102221]: pgmap v1388: 109 pgs: 109 active+clean; 1.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 847 op/s; 19 KiB/s, 0 objects/s recovering 2024-09-21T10:21:54.986 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:54 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:21:54.986 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:54 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:21:54.986 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:54 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/161223445' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:21:54.986 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:54 smithi107 ceph-mon[102221]: osdmap e398: 8 total, 8 up, 7 in 2024-09-21T10:21:54.987 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:54 smithi107 ceph-mon[105022]: pgmap v1388: 109 pgs: 109 active+clean; 1.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 847 op/s; 19 KiB/s, 0 objects/s recovering 2024-09-21T10:21:54.987 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:54 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:21:54.987 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:54 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:21:54.987 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:54 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/161223445' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:21:54.987 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:54 smithi107 ceph-mon[105022]: osdmap e398: 8 total, 8 up, 7 in 2024-09-21T10:21:55.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:55.114 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:55.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:55.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:54 smithi129 ceph-mon[88471]: pgmap v1388: 109 pgs: 109 active+clean; 1.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 847 op/s; 19 KiB/s, 0 objects/s recovering 2024-09-21T10:21:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:54 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:21:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:54 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:21:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:54 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/161223445' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:21:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:54 smithi129 ceph-mon[88471]: osdmap e398: 8 total, 8 up, 7 in 2024-09-21T10:21:55.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:55.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:55.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:55.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:55.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:55.398 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:55.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:55.499 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:55.531 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:55.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:55.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:55.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:55.666 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:55.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:55.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:55.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:21:55.803 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:55.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:55.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:21:55.905 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:55.940 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:55.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:55.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:56.041 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:56.087 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:56.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:56.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:56.189 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:56.228 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:56.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:56.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:56.330 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:56.367 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:56.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:56.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:56.470 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:56.509 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:56.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:56.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:56.611 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:56.636 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:56.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:56.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:56.737 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:56.772 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:56.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:56.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:56 smithi107 ceph-mon[102221]: pgmap v1390: 109 pgs: 109 active+clean; 1.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 932 op/s; 17 KiB/s, 0 objects/s recovering 2024-09-21T10:21:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:21:56.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:56 smithi107 ceph-mon[105022]: pgmap v1390: 109 pgs: 109 active+clean; 1.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 932 op/s; 17 KiB/s, 0 objects/s recovering 2024-09-21T10:21:56.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:21:56.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:56.875 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:56.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:56.913 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:56.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:57.014 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:57.046 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:57.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:57.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:57.074 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:21:57.148 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:56 smithi129 ceph-mon[88471]: pgmap v1390: 109 pgs: 109 active+clean; 1.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 932 op/s; 17 KiB/s, 0 objects/s recovering 2024-09-21T10:21:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:21:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:57.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:57.187 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:57.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:57.288 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:57.324 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:57.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:57.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:57.426 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:57.469 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:57.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:57.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:57.571 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:57.610 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:57.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:57.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:57.711 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:57.742 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:57.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:57.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:57.844 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:57.877 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:57.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:57.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:57.979 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:58.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:58.018 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:58.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:58.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:57 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:21:58.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:57 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:21:58.120 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:21:58.164 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:58.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:58.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:21:58.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:57 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:21:58.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:58.308 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:58.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:58.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:58.410 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:58.446 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:58.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:58.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:58.548 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:21:58.587 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:58.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:58.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:21:58.689 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:58.722 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:58.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:58.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:58.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:58.858 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:58.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:58 smithi107 ceph-mon[102221]: pgmap v1391: 109 pgs: 109 active+clean; 1.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 976 op/s; 17 KiB/s, 0 objects/s recovering 2024-09-21T10:21:58.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:58 smithi107 ceph-mon[105022]: pgmap v1391: 109 pgs: 109 active+clean; 1.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 976 op/s; 17 KiB/s, 0 objects/s recovering 2024-09-21T10:21:58.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:58.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:58.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:21:59.000 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:59.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:59.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:21:59.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:21:59.142 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:59.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:21:59.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:58 smithi129 ceph-mon[88471]: pgmap v1391: 109 pgs: 109 active+clean; 1.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 976 op/s; 17 KiB/s, 0 objects/s recovering 2024-09-21T10:21:59.244 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:59.254 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:21:59.255 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:21:59.255 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:21:59.255 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:21:59.276 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:21:59.276 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:21:59.279 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:59.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:59.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:59.306 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:21:59.307 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:21:59.337 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:21:59.337 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:21:59.367 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:21:59.367 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:21:59.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:59.401 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:21:59.401 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:21:59.417 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:59.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:59.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:21:59.435 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:21:59.436 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:21:59.519 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:21:59.556 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:59.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:21:59.658 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:59.694 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:59.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:59.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:59.796 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:21:59.833 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:21:59.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:59.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:21:59.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:21:59 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1462598116' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:21:59.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:21:59 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1462598116' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:21:59.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:21:59.997 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:21:59.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:21:59.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:00.016 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:00.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:00.143 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:00.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:00.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:00.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:21:59 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1462598116' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:00.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:00.288 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:00.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:00.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:00.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:00.423 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:00.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:00.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:00.525 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:00.557 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:00.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:00.630 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:00.659 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:22:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:22:00.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:22:00.696 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:00.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:00.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:00.798 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:00.828 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:00.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:00.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:00.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:00 smithi107 ceph-mon[105022]: pgmap v1392: 109 pgs: 109 active+clean; 999 MiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-21T10:22:00.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:00 smithi107 ceph-mon[102221]: pgmap v1392: 109 pgs: 109 active+clean; 999 MiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-21T10:22:00.929 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:00.971 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:00.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:00.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:01.072 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:01.105 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:01.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:01.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:00 smithi129 ceph-mon[88471]: pgmap v1392: 109 pgs: 109 active+clean; 999 MiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-21T10:22:01.207 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:01.248 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:01.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:01.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:01.349 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:01.389 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:01.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:01.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:01.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:01.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:01.525 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:01.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:01.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:01.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:01.659 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:01.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:01.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:01.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:01.800 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:01.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:01.901 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:01.938 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:01.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:01.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:01.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:22:02.017 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:22:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:22:01] "GET /metrics HTTP/1.1" 200 36781 "" "Prometheus/2.51.0" 2024-09-21T10:22:02.040 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:02.091 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:02.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:02.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:02.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:02.247 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:02.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:02.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:02.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:02.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:02.402 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:02.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:02.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:02.537 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:02.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:02.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:02.639 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:02.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:22:02.673 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:02.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:02.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:02.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:02.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:02.829 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:02.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:02.930 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:02.957 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:02.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:02.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:03.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:03.103 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:03.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:03.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:03.105 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:03 smithi107 ceph-mon[102221]: pgmap v1393: 109 pgs: 109 active+clean; 999 MiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-21T10:22:03.105 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:03.105 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:03.105 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:22:03.106 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:22:03.106 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:03.106 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:03 smithi107 ceph-mon[105022]: pgmap v1393: 109 pgs: 109 active+clean; 999 MiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-21T10:22:03.106 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:03.106 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:03.106 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:22:03.106 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:22:03.106 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:03.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:03.246 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:03.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:03.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:03.348 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:03.372 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:03.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:03.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:03.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:03 smithi129 ceph-mon[88471]: pgmap v1393: 109 pgs: 109 active+clean; 999 MiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-21T10:22:03.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:03.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:03.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:22:03.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:22:03.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:03.473 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:03.507 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:03.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:03.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:03.609 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:03.645 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:03.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:03.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:03.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:03.780 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:03.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:03.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:03.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:03.913 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:03.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:03.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:04.015 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:04.052 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:04.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:04.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:04.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:04.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:04.182 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:04.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:04.283 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:04.314 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:04.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:04.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:04.416 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:04.456 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:04.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:04.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:04.558 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:04.578 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:04.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:04.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:04.680 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:04.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:04.714 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:04.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:04.815 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:04.849 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:04.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:04.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:04.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:04.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:04.984 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:04.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:05.085 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:05.106 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:05.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:05.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:05.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:05 smithi107 ceph-mon[102221]: pgmap v1394: 109 pgs: 109 active+clean; 772 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-21T10:22:05.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:05 smithi107 ceph-mon[105022]: pgmap v1394: 109 pgs: 109 active+clean; 772 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-21T10:22:05.207 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:05.237 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:05.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:05.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:05.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:05.374 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:05.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:05.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:05.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:05 smithi129 ceph-mon[88471]: pgmap v1394: 109 pgs: 109 active+clean; 772 MiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-21T10:22:05.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:05.526 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:05.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:05.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:05.627 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:05.658 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:05.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:05.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:05.736 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:05.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:05.811 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:05.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:05.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:05.913 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:05.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:05.944 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:05.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:06.045 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:06.078 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:06.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:06.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:06.180 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:06.217 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:06.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:06.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:06.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:06.336 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:06.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:06.364 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:06.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:06.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:06.510 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:06.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:06.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:06.612 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:06.651 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:06.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:06.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:06.752 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:06.782 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:06.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:06.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:06.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:06.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:06.917 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:06.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:07.018 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:07.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:07.052 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:07.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:07.089 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.4 primary-affinity to 0 (802) 2024-09-21T10:22:07.153 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:07.177 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:07.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:07.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:07.205 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:07 smithi107 ceph-mon[102221]: pgmap v1395: 109 pgs: 109 active+clean; 668 MiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-21T10:22:07.205 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:07 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3150826925' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-21T10:22:07.205 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:07 smithi107 ceph-mon[105022]: pgmap v1395: 109 pgs: 109 active+clean; 668 MiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-21T10:22:07.205 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:07 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3150826925' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-21T10:22:07.279 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:07.300 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:07.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:07.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:07.402 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:07.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:07 smithi129 ceph-mon[88471]: pgmap v1395: 109 pgs: 109 active+clean; 668 MiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-21T10:22:07.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:07 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3150826925' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 0.0} : dispatch 2024-09-21T10:22:07.423 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:07.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:07.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:07.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:07.546 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:07.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:07.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:07.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:07.668 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:07.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:07.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:07.770 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:07.792 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:07.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:07.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:07.894 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:07.916 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:07.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:07.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:08.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:08.039 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:08.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:08.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:08.141 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:08.166 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:08.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:08.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:08.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:08.301 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:08.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:08.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:08.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:08 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3150826925' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-21T10:22:08.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:08 smithi107 ceph-mon[102221]: osdmap e399: 8 total, 8 up, 7 in 2024-09-21T10:22:08.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:08 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3150826925' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-21T10:22:08.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:08 smithi107 ceph-mon[105022]: osdmap e399: 8 total, 8 up, 7 in 2024-09-21T10:22:08.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:08.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:08 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3150826925' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-21T10:22:08.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:08 smithi129 ceph-mon[88471]: osdmap e399: 8 total, 8 up, 7 in 2024-09-21T10:22:08.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:08.437 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:08.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:08.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:08.569 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:08.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:08.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:08.671 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:08.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:08.705 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:08.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:08.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:08.844 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:08.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:08.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:08.945 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:08.982 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:08.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:08.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:09.084 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:09.117 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:09.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:09.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:09.219 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:09.250 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:09.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:09.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:09.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:09 smithi107 ceph-mon[102221]: pgmap v1397: 109 pgs: 3 peering, 106 active+clean; 591 MiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 949 op/s 2024-09-21T10:22:09.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:09 smithi107 ceph-mon[102221]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:22:09.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:09 smithi107 ceph-mon[102221]: osdmap e400: 8 total, 8 up, 7 in 2024-09-21T10:22:09.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:09 smithi107 ceph-mon[105022]: pgmap v1397: 109 pgs: 3 peering, 106 active+clean; 591 MiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 949 op/s 2024-09-21T10:22:09.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:09 smithi107 ceph-mon[105022]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:22:09.287 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:09 smithi107 ceph-mon[105022]: osdmap e400: 8 total, 8 up, 7 in 2024-09-21T10:22:09.336 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:22:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-09-21T10:22:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-09-21T10:22:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:22:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:198489 11.7 11:e9174170:::benchmark_data_smithi107_218812_object86002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-21T10:22:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:02.782897+0000", 2024-09-21T10:22:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 546.54655236999997, 2024-09-21T10:22:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0710279199999999, 2024-09-21T10:22:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:09.336 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:22:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:22:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 198489 2024-09-21T10:22:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.782897+0000", 2024-09-21T10:22:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.782897+0000", 2024-09-21T10:22:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.782899+0000", 2024-09-21T10:22:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8339999999999999e-06 2024-09-21T10:22:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.782905+0000", 2024-09-21T10:22:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9769999999999997e-06 2024-09-21T10:22:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.783003+0000", 2024-09-21T10:22:09.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7923000000000004e-05 2024-09-21T10:22:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.783012+0000", 2024-09-21T10:22:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7139999999999999e-06 2024-09-21T10:22:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.788021+0000", 2024-09-21T10:22:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0050089069999999999 2024-09-21T10:22:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.339 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.788029+0000", 2024-09-21T10:22:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9320000000000002e-06 2024-09-21T10:22:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.845768+0000", 2024-09-21T10:22:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0577382339999999 2024-09-21T10:22:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.845866+0000", 2024-09-21T10:22:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8791999999999999e-05 2024-09-21T10:22:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.340 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:22:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.845953+0000", 2024-09-21T10:22:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6781999999999997e-05 2024-09-21T10:22:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.853675+0000", 2024-09-21T10:22:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0077221039999999996 2024-09-21T10:22:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.853692+0000", 2024-09-21T10:22:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6699000000000001e-05 2024-09-21T10:22:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.853871+0000", 2024-09-21T10:22:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000178495 2024-09-21T10:22:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.853902+0000", 2024-09-21T10:22:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1554000000000003e-05 2024-09-21T10:22:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.853925+0000", 2024-09-21T10:22:09.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2972999999999998e-05 2024-09-21T10:22:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:198490 11.7 11:ee345632:::benchmark_data_smithi107_218812_object86003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-21T10:22:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:02.783038+0000", 2024-09-21T10:22:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 546.54641126000001, 2024-09-21T10:22:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.071039222, 2024-09-21T10:22:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:09.343 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:22:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:22:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 198490 2024-09-21T10:22:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.783038+0000", 2024-09-21T10:22:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.783038+0000", 2024-09-21T10:22:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.783039+0000", 2024-09-21T10:22:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.13e-07 2024-09-21T10:22:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.783042+0000", 2024-09-21T10:22:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0400000000000001e-06 2024-09-21T10:22:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.783050+0000", 2024-09-21T10:22:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6410000000000002e-06 2024-09-21T10:22:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.783055+0000", 2024-09-21T10:22:09.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5090000000000001e-06 2024-09-21T10:22:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.792028+0000", 2024-09-21T10:22:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0089732159999999991 2024-09-21T10:22:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.792037+0000", 2024-09-21T10:22:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.789e-06 2024-09-21T10:22:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.848714+0000", 2024-09-21T10:22:09.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0566770919999999 2024-09-21T10:22:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.848862+0000", 2024-09-21T10:22:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000147489 2024-09-21T10:22:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:22:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.848988+0000", 2024-09-21T10:22:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000125974 2024-09-21T10:22:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.853818+0000", 2024-09-21T10:22:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048301840000000004 2024-09-21T10:22:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.853937+0000", 2024-09-21T10:22:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011904399999999999 2024-09-21T10:22:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.854016+0000", 2024-09-21T10:22:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8781000000000003e-05 2024-09-21T10:22:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.854043+0000", 2024-09-21T10:22:09.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7506000000000001e-05 2024-09-21T10:22:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.854077+0000", 2024-09-21T10:22:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4344000000000002e-05 2024-09-21T10:22:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:198551 11.7 11:e9113b5a:::benchmark_data_smithi107_218812_object86064:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-21T10:22:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:02.847017+0000", 2024-09-21T10:22:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 546.48243222400004, 2024-09-21T10:22:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.010035579, 2024-09-21T10:22:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:09.349 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:22:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:22:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 198551 2024-09-21T10:22:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.847017+0000", 2024-09-21T10:22:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:09.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.847017+0000", 2024-09-21T10:22:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.847019+0000", 2024-09-21T10:22:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5710000000000001e-06 2024-09-21T10:22:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.847023+0000", 2024-09-21T10:22:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5560000000000001e-06 2024-09-21T10:22:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.847039+0000", 2024-09-21T10:22:09.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5608999999999999e-05 2024-09-21T10:22:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.847046+0000", 2024-09-21T10:22:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9959999999999996e-06 2024-09-21T10:22:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.847247+0000", 2024-09-21T10:22:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00020094199999999999 2024-09-21T10:22:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.847255+0000", 2024-09-21T10:22:09.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6279999999999996e-06 2024-09-21T10:22:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.851354+0000", 2024-09-21T10:22:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.004098999 2024-09-21T10:22:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.851455+0000", 2024-09-21T10:22:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101489 2024-09-21T10:22:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:22:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.851567+0000", 2024-09-21T10:22:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000112173 2024-09-21T10:22:09.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.853969+0000", 2024-09-21T10:22:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024016509999999999 2024-09-21T10:22:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.853982+0000", 2024-09-21T10:22:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.331e-05 2024-09-21T10:22:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.856966+0000", 2024-09-21T10:22:09.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0029836670000000002 2024-09-21T10:22:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.856997+0000", 2024-09-21T10:22:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.15e-05 2024-09-21T10:22:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.857053+0000", 2024-09-21T10:22:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5488000000000001e-05 2024-09-21T10:22:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.355 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36561.0:198557 11.7 11:e69af3d7:::benchmark_data_smithi107_218812_object86070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-21T10:22:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:13:02.852910+0000", 2024-09-21T10:22:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 546.47653957199998, 2024-09-21T10:22:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0107109940000001, 2024-09-21T10:22:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36561", 2024-09-21T10:22:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/32304772", 2024-09-21T10:22:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 198557 2024-09-21T10:22:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.356 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.852910+0000", 2024-09-21T10:22:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.852910+0000", 2024-09-21T10:22:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.852912+0000", 2024-09-21T10:22:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.593e-06 2024-09-21T10:22:09.357 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.852917+0000", 2024-09-21T10:22:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5210000000000002e-06 2024-09-21T10:22:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.852935+0000", 2024-09-21T10:22:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8159999999999999e-05 2024-09-21T10:22:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.852944+0000", 2024-09-21T10:22:09.358 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6759999999999996e-06 2024-09-21T10:22:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.853018+0000", 2024-09-21T10:22:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4506999999999997e-05 2024-09-21T10:22:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:02.853029+0000", 2024-09-21T10:22:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0232e-05 2024-09-21T10:22:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.854128+0000", 2024-09-21T10:22:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.001099363 2024-09-21T10:22:09.359 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.854198+0000", 2024-09-21T10:22:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9879e-05 2024-09-21T10:22:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:22:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.854266+0000", 2024-09-21T10:22:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7897000000000004e-05 2024-09-21T10:22:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.863355+0000", 2024-09-21T10:22:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0090896929999999994 2024-09-21T10:22:09.360 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.863404+0000", 2024-09-21T10:22:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8401e-05 2024-09-21T10:22:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.863499+0000", 2024-09-21T10:22:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5093999999999995e-05 2024-09-21T10:22:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.361 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:09.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:09.362 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.863568+0000", 2024-09-21T10:22:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9537999999999997e-05 2024-09-21T10:22:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:13:03.863621+0000", 2024-09-21T10:22:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2439999999999999e-05 2024-09-21T10:22:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.363 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:42515 12.e 12:7fea1a71:::benchmark_data_smithi107_250110_object42514:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-21T10:22:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:29.657893+0000", 2024-09-21T10:22:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 459.67155691099998, 2024-09-21T10:22:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1570628379999999, 2024-09-21T10:22:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:22:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:22:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 42515 2024-09-21T10:22:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.364 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.657893+0000", 2024-09-21T10:22:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.657893+0000", 2024-09-21T10:22:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.657894+0000", 2024-09-21T10:22:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6200000000000006e-07 2024-09-21T10:22:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.365 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.657916+0000", 2024-09-21T10:22:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2685999999999999e-05 2024-09-21T10:22:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.657931+0000", 2024-09-21T10:22:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4659e-05 2024-09-21T10:22:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.657938+0000", 2024-09-21T10:22:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4699999999999996e-06 2024-09-21T10:22:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.366 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.658022+0000", 2024-09-21T10:22:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4006999999999997e-05 2024-09-21T10:22:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.658032+0000", 2024-09-21T10:22:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9399999999999997e-06 2024-09-21T10:22:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698401+0000", 2024-09-21T10:22:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0403689709999999 2024-09-21T10:22:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.367 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698414+0000", 2024-09-21T10:22:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2863e-05 2024-09-21T10:22:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.801698+0000", 2024-09-21T10:22:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.103283749 2024-09-21T10:22:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.801861+0000", 2024-09-21T10:22:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016259600000000001 2024-09-21T10:22:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.368 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:22:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.802007+0000", 2024-09-21T10:22:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014699 2024-09-21T10:22:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.805969+0000", 2024-09-21T10:22:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0039619770000000002 2024-09-21T10:22:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.807396+0000", 2024-09-21T10:22:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014265969999999999 2024-09-21T10:22:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.369 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.812802+0000", 2024-09-21T10:22:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054060920000000004 2024-09-21T10:22:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.812844+0000", 2024-09-21T10:22:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2321999999999999e-05 2024-09-21T10:22:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.814955+0000", 2024-09-21T10:22:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021109570000000001 2024-09-21T10:22:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.370 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:42540 12.e 12:773d0c07:::benchmark_data_smithi107_250110_object42539:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-21T10:22:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:29.679940+0000", 2024-09-21T10:22:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 459.64950939400001, 2024-09-21T10:22:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1367923950000001, 2024-09-21T10:22:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:22:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:22:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 42540 2024-09-21T10:22:09.371 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.679940+0000", 2024-09-21T10:22:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.679940+0000", 2024-09-21T10:22:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.679942+0000", 2024-09-21T10:22:09.372 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.745e-06 2024-09-21T10:22:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.679969+0000", 2024-09-21T10:22:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6673999999999999e-05 2024-09-21T10:22:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.679986+0000", 2024-09-21T10:22:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7796e-05 2024-09-21T10:22:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.679994+0000", 2024-09-21T10:22:09.373 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0080000000000002e-06 2024-09-21T10:22:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.680526+0000", 2024-09-21T10:22:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00053167399999999999 2024-09-21T10:22:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.680537+0000", 2024-09-21T10:22:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0528e-05 2024-09-21T10:22:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.374 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698441+0000", 2024-09-21T10:22:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.017904532 2024-09-21T10:22:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698449+0000", 2024-09-21T10:22:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8199999999999997e-06 2024-09-21T10:22:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.804541+0000", 2024-09-21T10:22:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.106092007 2024-09-21T10:22:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:09.375 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.804639+0000", 2024-09-21T10:22:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7731000000000003e-05 2024-09-21T10:22:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:22:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.804751+0000", 2024-09-21T10:22:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011230099999999999 2024-09-21T10:22:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.808849+0000", 2024-09-21T10:22:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004098272 2024-09-21T10:22:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.810325+0000", 2024-09-21T10:22:09.376 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00147554 2024-09-21T10:22:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.815039+0000", 2024-09-21T10:22:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004714088 2024-09-21T10:22:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.815108+0000", 2024-09-21T10:22:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9041999999999994e-05 2024-09-21T10:22:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.816733+0000", 2024-09-21T10:22:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001624637 2024-09-21T10:22:09.377 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:42557 12.e 12:721663e1:::benchmark_data_smithi107_250110_object42556:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-21T10:22:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:29.690624+0000", 2024-09-21T10:22:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 459.63882510899998, 2024-09-21T10:22:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1350235880000001, 2024-09-21T10:22:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:22:09.378 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:22:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 42557 2024-09-21T10:22:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.690624+0000", 2024-09-21T10:22:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.690624+0000", 2024-09-21T10:22:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.379 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.690626+0000", 2024-09-21T10:22:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.556e-06 2024-09-21T10:22:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.690647+0000", 2024-09-21T10:22:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1384999999999999e-05 2024-09-21T10:22:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.690664+0000", 2024-09-21T10:22:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6745999999999999e-05 2024-09-21T10:22:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.380 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.690671+0000", 2024-09-21T10:22:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2350000000000002e-06 2024-09-21T10:22:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.690801+0000", 2024-09-21T10:22:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012969899999999999 2024-09-21T10:22:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.690810+0000", 2024-09-21T10:22:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9239999999999996e-06 2024-09-21T10:22:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.381 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698474+0000", 2024-09-21T10:22:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.007663778 2024-09-21T10:22:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698481+0000", 2024-09-21T10:22:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.25e-06 2024-09-21T10:22:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.806017+0000", 2024-09-21T10:22:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.107536251 2024-09-21T10:22:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.382 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.806173+0000", 2024-09-21T10:22:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015593900000000001 2024-09-21T10:22:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:22:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.806333+0000", 2024-09-21T10:22:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000160209 2024-09-21T10:22:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.812687+0000", 2024-09-21T10:22:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0063535359999999999 2024-09-21T10:22:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.383 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.812747+0000", 2024-09-21T10:22:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9880999999999998e-05 2024-09-21T10:22:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.823402+0000", 2024-09-21T10:22:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010655695999999999 2024-09-21T10:22:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.823457+0000", 2024-09-21T10:22:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4083999999999999e-05 2024-09-21T10:22:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:09.384 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.825648+0000", 2024-09-21T10:22:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002191419 2024-09-21T10:22:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:42584 12.e 12:7ce522fc:::benchmark_data_smithi107_250110_object42583:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-21T10:22:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:29.716131+0000", 2024-09-21T10:22:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 459.61331803299998, 2024-09-21T10:22:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1227035999999999, 2024-09-21T10:22:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:09.385 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:22:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:22:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 42584 2024-09-21T10:22:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.716131+0000", 2024-09-21T10:22:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.716131+0000", 2024-09-21T10:22:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.716133+0000", 2024-09-21T10:22:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.72e-06 2024-09-21T10:22:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.716172+0000", 2024-09-21T10:22:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8525999999999997e-05 2024-09-21T10:22:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.716185+0000", 2024-09-21T10:22:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3189e-05 2024-09-21T10:22:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.387 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.716192+0000", 2024-09-21T10:22:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4850000000000003e-06 2024-09-21T10:22:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.388 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.716262+0000", 2024-09-21T10:22:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0055999999999995e-05 2024-09-21T10:22:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.716273+0000", 2024-09-21T10:22:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0379e-05 2024-09-21T10:22:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698504+0000", 2024-09-21T10:22:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.982231101 2024-09-21T10:22:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698511+0000", 2024-09-21T10:22:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6619999999999999e-06 2024-09-21T10:22:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.807514+0000", 2024-09-21T10:22:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1090034499999999 2024-09-21T10:22:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.807637+0000", 2024-09-21T10:22:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012253500000000001 2024-09-21T10:22:09.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:22:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.807782+0000", 2024-09-21T10:22:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000145351 2024-09-21T10:22:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.814974+0000", 2024-09-21T10:22:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071919769999999996 2024-09-21T10:22:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.816796+0000", 2024-09-21T10:22:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00182252 2024-09-21T10:22:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.834741+0000", 2024-09-21T10:22:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.017944882999999998 2024-09-21T10:22:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.834979+0000", 2024-09-21T10:22:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000237743 2024-09-21T10:22:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.838835+0000", 2024-09-21T10:22:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038560230000000001 2024-09-21T10:22:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:42600 12.e 12:702dc90f:::benchmark_data_smithi107_250110_object42599:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-21T10:22:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:29.723485+0000", 2024-09-21T10:22:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 459.605964228, 2024-09-21T10:22:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1177167639999999, 2024-09-21T10:22:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:09.393 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:22:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:22:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 42600 2024-09-21T10:22:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.723485+0000", 2024-09-21T10:22:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.723485+0000", 2024-09-21T10:22:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.394 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.723486+0000", 2024-09-21T10:22:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3399999999999997e-07 2024-09-21T10:22:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.723507+0000", 2024-09-21T10:22:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0400000000000001e-05 2024-09-21T10:22:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:09.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.723516+0000", 2024-09-21T10:22:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7540000000000001e-06 2024-09-21T10:22:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.723523+0000", 2024-09-21T10:22:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0990000000000001e-06 2024-09-21T10:22:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.723582+0000", 2024-09-21T10:22:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8693000000000002e-05 2024-09-21T10:22:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.723590+0000", 2024-09-21T10:22:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8550000000000002e-06 2024-09-21T10:22:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698533+0000", 2024-09-21T10:22:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.974942749 2024-09-21T10:22:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698540+0000", 2024-09-21T10:22:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6079999999999998e-06 2024-09-21T10:22:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.808901+0000", 2024-09-21T10:22:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1103609670000001 2024-09-21T10:22:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.809007+0000", 2024-09-21T10:22:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105257 2024-09-21T10:22:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:22:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.809104+0000", 2024-09-21T10:22:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7371e-05 2024-09-21T10:22:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.814981+0000", 2024-09-21T10:22:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0058765559999999998 2024-09-21T10:22:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.816845+0000", 2024-09-21T10:22:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0018647920000000001 2024-09-21T10:22:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.838903+0000", 2024-09-21T10:22:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.022057562999999999 2024-09-21T10:22:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.838975+0000", 2024-09-21T10:22:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2612000000000002e-05 2024-09-21T10:22:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.841202+0000", 2024-09-21T10:22:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022265539999999999 2024-09-21T10:22:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:42620 12.e 12:7684604d:::benchmark_data_smithi107_250110_object42619:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-21T10:22:09.400 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:29.823460+0000", 2024-09-21T10:22:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 459.50598958199998, 2024-09-21T10:22:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0200784430000001, 2024-09-21T10:22:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:22:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:22:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 42620 2024-09-21T10:22:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.823460+0000", 2024-09-21T10:22:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.823460+0000", 2024-09-21T10:22:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.823461+0000", 2024-09-21T10:22:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2300000000000001e-06 2024-09-21T10:22:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.823482+0000", 2024-09-21T10:22:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0639e-05 2024-09-21T10:22:09.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.823494+0000", 2024-09-21T10:22:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2225000000000001e-05 2024-09-21T10:22:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.823502+0000", 2024-09-21T10:22:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5789999999999998e-06 2024-09-21T10:22:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.823581+0000", 2024-09-21T10:22:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9048999999999995e-05 2024-09-21T10:22:09.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.823590+0000", 2024-09-21T10:22:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.02e-06 2024-09-21T10:22:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698565+0000", 2024-09-21T10:22:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.87497485200000003 2024-09-21T10:22:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698571+0000", 2024-09-21T10:22:09.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6819999999999997e-06 2024-09-21T10:22:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.810360+0000", 2024-09-21T10:22:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1117890880000001 2024-09-21T10:22:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.810490+0000", 2024-09-21T10:22:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000129555 2024-09-21T10:22:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:22:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.810640+0000", 2024-09-21T10:22:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015038900000000001 2024-09-21T10:22:09.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.814987+0000", 2024-09-21T10:22:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0043463479999999999 2024-09-21T10:22:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.816887+0000", 2024-09-21T10:22:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019008199999999999 2024-09-21T10:22:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.841269+0000", 2024-09-21T10:22:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.024381554999999999 2024-09-21T10:22:09.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.841335+0000", 2024-09-21T10:22:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5772000000000007e-05 2024-09-21T10:22:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.843538+0000", 2024-09-21T10:22:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022036400000000002 2024-09-21T10:22:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:42631 12.e 12:7324e759:::benchmark_data_smithi107_250110_object42630:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-21T10:22:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:29.831568+0000", 2024-09-21T10:22:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 459.49788137299998, 2024-09-21T10:22:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0140101700000002, 2024-09-21T10:22:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:22:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:22:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 42631 2024-09-21T10:22:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.408 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.831568+0000", 2024-09-21T10:22:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.831568+0000", 2024-09-21T10:22:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.831570+0000", 2024-09-21T10:22:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8309999999999999e-06 2024-09-21T10:22:09.409 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.831591+0000", 2024-09-21T10:22:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0667000000000001e-05 2024-09-21T10:22:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.831606+0000", 2024-09-21T10:22:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5051999999999999e-05 2024-09-21T10:22:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.831613+0000", 2024-09-21T10:22:09.410 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6109999999999996e-06 2024-09-21T10:22:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.831665+0000", 2024-09-21T10:22:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1430000000000001e-05 2024-09-21T10:22:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:29.831672+0000", 2024-09-21T10:22:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1749999999999999e-06 2024-09-21T10:22:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.411 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698593+0000", 2024-09-21T10:22:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.86692127299999999 2024-09-21T10:22:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:30.698601+0000", 2024-09-21T10:22:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7170000000000008e-06 2024-09-21T10:22:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.811527+0000", 2024-09-21T10:22:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1129258339999999 2024-09-21T10:22:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:09.412 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.811670+0000", 2024-09-21T10:22:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014346899999999999 2024-09-21T10:22:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:22:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.811814+0000", 2024-09-21T10:22:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000143905 2024-09-21T10:22:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.816752+0000", 2024-09-21T10:22:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0049376680000000001 2024-09-21T10:22:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.413 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.817356+0000", 2024-09-21T10:22:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00060429899999999998 2024-09-21T10:22:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.843622+0000", 2024-09-21T10:22:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.026265481 2024-09-21T10:22:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.843679+0000", 2024-09-21T10:22:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7342000000000001e-05 2024-09-21T10:22:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.414 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:31.845578+0000", 2024-09-21T10:22:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0018994159999999999 2024-09-21T10:22:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:135241 12.10 12.514f79d0 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T10:22:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:16:09.574571+0000", 2024-09-21T10:22:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 359.75487834500001, 2024-09-21T10:22:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0064763960000001, 2024-09-21T10:22:09.415 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:22:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:22:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:22:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 135241 2024-09-21T10:22:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.574571+0000", 2024-09-21T10:22:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.416 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.574571+0000", 2024-09-21T10:22:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.574572+0000", 2024-09-21T10:22:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4499999999999997e-07 2024-09-21T10:22:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.574574+0000", 2024-09-21T10:22:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.967e-06 2024-09-21T10:22:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.417 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.574580+0000", 2024-09-21T10:22:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2990000000000002e-06 2024-09-21T10:22:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.574584+0000", 2024-09-21T10:22:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6140000000000002e-06 2024-09-21T10:22:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.577644+0000", 2024-09-21T10:22:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0030592739999999998 2024-09-21T10:22:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.418 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:09.577651+0000", 2024-09-21T10:22:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7000000000000008e-06 2024-09-21T10:22:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581037+0000", 2024-09-21T10:22:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0033855620000001 2024-09-21T10:22:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:16:10.581048+0000", 2024-09-21T10:22:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0535e-05 2024-09-21T10:22:09.419 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:177097 12.19 12:9f836ee2:::benchmark_data_smithi107_250110_object65687:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e353)", 2024-09-21T10:22:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:17:06.629165+0000", 2024-09-21T10:22:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 302.70028485, 2024-09-21T10:22:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0158801159999999, 2024-09-21T10:22:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:22:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:22:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 177097 2024-09-21T10:22:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.629165+0000", 2024-09-21T10:22:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.629165+0000", 2024-09-21T10:22:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.421 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.629166+0000", 2024-09-21T10:22:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1769999999999999e-06 2024-09-21T10:22:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.629169+0000", 2024-09-21T10:22:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.818e-06 2024-09-21T10:22:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.629177+0000", 2024-09-21T10:22:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9549999999999992e-06 2024-09-21T10:22:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.422 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.629183+0000", 2024-09-21T10:22:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0410000000000003e-06 2024-09-21T10:22:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.629693+0000", 2024-09-21T10:22:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00051066800000000004 2024-09-21T10:22:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:06.629702+0000", 2024-09-21T10:22:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7749999999999998e-06 2024-09-21T10:22:09.423 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.641125+0000", 2024-09-21T10:22:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0114230399999999 2024-09-21T10:22:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.641262+0000", 2024-09-21T10:22:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013656000000000001 2024-09-21T10:22:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:22:09.424 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.641592+0000", 2024-09-21T10:22:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00033042800000000002 2024-09-21T10:22:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.644895+0000", 2024-09-21T10:22:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003302532 2024-09-21T10:22:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.644943+0000", 2024-09-21T10:22:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8390000000000003e-05 2024-09-21T10:22:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:09.425 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.644957+0000", 2024-09-21T10:22:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4253999999999999e-05 2024-09-21T10:22:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.645004+0000", 2024-09-21T10:22:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6802999999999999e-05 2024-09-21T10:22:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:17:07.645045+0000", 2024-09-21T10:22:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0674999999999999e-05 2024-09-21T10:22:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.426 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:249514 13.5 13:ac1bf7a6:::benchmark_data_smithi107_278655_object121607:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e399)", 2024-09-21T10:22:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:22:07.100788+0000", 2024-09-21T10:22:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.2286610570000001, 2024-09-21T10:22:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.061392304, 2024-09-21T10:22:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:22:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:22:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:22:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 249514 2024-09-21T10:22:09.427 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.100788+0000", 2024-09-21T10:22:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.428 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:09.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:09 smithi129 ceph-mon[88471]: pgmap v1397: 109 pgs: 3 peering, 106 active+clean; 591 MiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 949 op/s 2024-09-21T10:22:09.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:09 smithi129 ceph-mon[88471]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:22:09.429 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:09 smithi129 ceph-mon[88471]: osdmap e400: 8 total, 8 up, 7 in 2024-09-21T10:22:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.100788+0000", 2024-09-21T10:22:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.100790+0000", 2024-09-21T10:22:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9609999999999999e-06 2024-09-21T10:22:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.430 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.100797+0000", 2024-09-21T10:22:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2339999999999999e-06 2024-09-21T10:22:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.100810+0000", 2024-09-21T10:22:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3114e-05 2024-09-21T10:22:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.100818+0000", 2024-09-21T10:22:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6799999999999999e-06 2024-09-21T10:22:09.431 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.101374+0000", 2024-09-21T10:22:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055542899999999997 2024-09-21T10:22:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-09-21T10:22:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.101382+0000", 2024-09-21T10:22:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7440000000000007e-06 2024-09-21T10:22:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.104016+0000", 2024-09-21T10:22:09.432 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0026347139999999998 2024-09-21T10:22:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.104029+0000", 2024-09-21T10:22:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2510999999999999e-05 2024-09-21T10:22:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.160290+0000", 2024-09-21T10:22:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0562612600000001 2024-09-21T10:22:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.162181+0000", 2024-09-21T10:22:09.433 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001890657 2024-09-21T10:22:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:249527 13.5 13:a508c654:::benchmark_data_smithi107_278655_object121620:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-09-21T10:22:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:22:07.114805+0000", 2024-09-21T10:22:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.214644163, 2024-09-21T10:22:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0590544689999999, 2024-09-21T10:22:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:09.434 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:22:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:22:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 249527 2024-09-21T10:22:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.114805+0000", 2024-09-21T10:22:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.114805+0000", 2024-09-21T10:22:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.435 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.114807+0000", 2024-09-21T10:22:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.875e-06 2024-09-21T10:22:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.114814+0000", 2024-09-21T10:22:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9959999999999996e-06 2024-09-21T10:22:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.114835+0000", 2024-09-21T10:22:09.436 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1148000000000001e-05 2024-09-21T10:22:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.114856+0000", 2024-09-21T10:22:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0716e-05 2024-09-21T10:22:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.114979+0000", 2024-09-21T10:22:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012298300000000001 2024-09-21T10:22:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.114990+0000", 2024-09-21T10:22:09.437 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0723999999999999e-05 2024-09-21T10:22:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.162747+0000", 2024-09-21T10:22:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.047756873 2024-09-21T10:22:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.163228+0000", 2024-09-21T10:22:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048100999999999998 2024-09-21T10:22:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:22:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.163303+0000", 2024-09-21T10:22:09.438 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5537999999999994e-05 2024-09-21T10:22:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.173548+0000", 2024-09-21T10:22:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010244593999999999 2024-09-21T10:22:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.173739+0000", 2024-09-21T10:22:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019120000000000001 2024-09-21T10:22:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.439 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.173786+0000", 2024-09-21T10:22:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7172999999999997e-05 2024-09-21T10:22:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.173816+0000", 2024-09-21T10:22:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9730000000000002e-05 2024-09-21T10:22:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.173860+0000", 2024-09-21T10:22:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3909000000000003e-05 2024-09-21T10:22:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.440 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:249528 13.5 13:a7e02809:::benchmark_data_smithi107_278655_object121621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-09-21T10:22:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:22:07.115911+0000", 2024-09-21T10:22:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.2135386480000001, 2024-09-21T10:22:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0630208059999999, 2024-09-21T10:22:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:22:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:22:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 249528 2024-09-21T10:22:09.441 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.115911+0000", 2024-09-21T10:22:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.115911+0000", 2024-09-21T10:22:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:09.442 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.115912+0000", 2024-09-21T10:22:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.004e-06 2024-09-21T10:22:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.115917+0000", 2024-09-21T10:22:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8500000000000002e-06 2024-09-21T10:22:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.115933+0000", 2024-09-21T10:22:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5968999999999999e-05 2024-09-21T10:22:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.443 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.115941+0000", 2024-09-21T10:22:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1720000000000007e-06 2024-09-21T10:22:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.116004+0000", 2024-09-21T10:22:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2639999999999997e-05 2024-09-21T10:22:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.116013+0000", 2024-09-21T10:22:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3070000000000008e-06 2024-09-21T10:22:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.444 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.165887+0000", 2024-09-21T10:22:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0498742990000001 2024-09-21T10:22:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.165993+0000", 2024-09-21T10:22:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010578600000000001 2024-09-21T10:22:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:22:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.166097+0000", 2024-09-21T10:22:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103948 2024-09-21T10:22:09.445 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.173870+0000", 2024-09-21T10:22:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0077733660000000003 2024-09-21T10:22:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.178758+0000", 2024-09-21T10:22:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048879170000000003 2024-09-21T10:22:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.178838+0000", 2024-09-21T10:22:09.446 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9728999999999995e-05 2024-09-21T10:22:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.178885+0000", 2024-09-21T10:22:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7515999999999997e-05 2024-09-21T10:22:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.178932+0000", 2024-09-21T10:22:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6303e-05 2024-09-21T10:22:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.447 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.448 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.448 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:249549 13.5 13:a14dcfc3:::benchmark_data_smithi107_278655_object121642:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-09-21T10:22:09.448 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:22:07.142498+0000", 2024-09-21T10:22:09.448 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.1869510289999998, 2024-09-21T10:22:09.448 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0365948819999999, 2024-09-21T10:22:09.448 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:09.448 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:09.448 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:09.448 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:09.448 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:22:09.448 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:22:09.448 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 249549 2024-09-21T10:22:09.448 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.448 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:09.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:09.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142498+0000", 2024-09-21T10:22:09.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:09.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142498+0000", 2024-09-21T10:22:09.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.449 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.449 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:09.449 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142501+0000", 2024-09-21T10:22:09.449 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3870000000000002e-06 2024-09-21T10:22:09.449 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:09.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142507+0000", 2024-09-21T10:22:09.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4200000000000004e-06 2024-09-21T10:22:09.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:09.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142530+0000", 2024-09-21T10:22:09.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2259000000000002e-05 2024-09-21T10:22:09.450 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.450 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.450 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:09.450 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142539+0000", 2024-09-21T10:22:09.450 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9410000000000002e-06 2024-09-21T10:22:09.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.451 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142653+0000", 2024-09-21T10:22:09.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000113919 2024-09-21T10:22:09.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.451 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142667+0000", 2024-09-21T10:22:09.451 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3321e-05 2024-09-21T10:22:09.451 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.451 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.451 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.168611+0000", 2024-09-21T10:22:09.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0259440360000001 2024-09-21T10:22:09.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:09.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.168709+0000", 2024-09-21T10:22:09.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8190999999999996e-05 2024-09-21T10:22:09.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:22:09.452 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.168841+0000", 2024-09-21T10:22:09.452 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131703 2024-09-21T10:22:09.452 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.452 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.452 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:09.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.173877+0000", 2024-09-21T10:22:09.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005035973 2024-09-21T10:22:09.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.453 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.178798+0000", 2024-09-21T10:22:09.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0049212250000000004 2024-09-21T10:22:09.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.453 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.453 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.178976+0000", 2024-09-21T10:22:09.453 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017834 2024-09-21T10:22:09.453 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.453 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.453 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:09.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179055+0000", 2024-09-21T10:22:09.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9271999999999996e-05 2024-09-21T10:22:09.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.454 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:09.454 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179093+0000", 2024-09-21T10:22:09.454 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7895000000000002e-05 2024-09-21T10:22:09.454 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.454 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.454 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.454 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.454 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.454 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:249560 13.5 13:a5a5c164:::benchmark_data_smithi107_278655_object121653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-09-21T10:22:09.454 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:22:07.152416+0000", 2024-09-21T10:22:09.455 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.1770332149999998, 2024-09-21T10:22:09.455 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.026876795, 2024-09-21T10:22:09.455 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:09.455 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:09.455 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:09.455 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:09.455 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:22:09.455 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:22:09.455 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 249560 2024-09-21T10:22:09.455 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.455 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:09.455 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.455 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:09.455 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.152416+0000", 2024-09-21T10:22:09.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:09.456 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.152416+0000", 2024-09-21T10:22:09.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:09.456 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.152418+0000", 2024-09-21T10:22:09.456 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8050000000000001e-06 2024-09-21T10:22:09.456 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.456 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.456 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:09.456 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.152423+0000", 2024-09-21T10:22:09.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1139999999999997e-06 2024-09-21T10:22:09.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:09.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.152439+0000", 2024-09-21T10:22:09.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6259000000000001e-05 2024-09-21T10:22:09.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:09.457 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.152447+0000", 2024-09-21T10:22:09.457 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9230000000000002e-06 2024-09-21T10:22:09.457 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.457 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.457 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.152543+0000", 2024-09-21T10:22:09.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5638999999999997e-05 2024-09-21T10:22:09.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.152552+0000", 2024-09-21T10:22:09.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4199999999999996e-06 2024-09-21T10:22:09.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.458 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.458 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.171137+0000", 2024-09-21T10:22:09.458 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.018584741 2024-09-21T10:22:09.458 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.458 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:09.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.171235+0000", 2024-09-21T10:22:09.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7549999999999999e-05 2024-09-21T10:22:09.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:22:09.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.171302+0000", 2024-09-21T10:22:09.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7150999999999995e-05 2024-09-21T10:22:09.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.459 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.459 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:09.459 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.178720+0000", 2024-09-21T10:22:09.459 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0074180560000000001 2024-09-21T10:22:09.459 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179141+0000", 2024-09-21T10:22:09.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00042093000000000002 2024-09-21T10:22:09.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179216+0000", 2024-09-21T10:22:09.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4732000000000002e-05 2024-09-21T10:22:09.460 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.460 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.460 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:09.460 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179260+0000", 2024-09-21T10:22:09.460 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3996999999999999e-05 2024-09-21T10:22:09.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.461 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:09.461 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179293+0000", 2024-09-21T10:22:09.461 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3478e-05 2024-09-21T10:22:09.461 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.461 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.461 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.461 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.461 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.461 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:249568 13.5 13:a205a8d5:::benchmark_data_smithi107_278655_object121661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-09-21T10:22:09.461 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:22:07.161674+0000", 2024-09-21T10:22:09.461 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.167775513, 2024-09-21T10:22:09.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.017720706, 2024-09-21T10:22:09.462 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:09.462 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:09.462 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:09.462 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:09.462 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:22:09.462 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:22:09.462 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 249568 2024-09-21T10:22:09.462 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.462 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:09.462 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.462 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:09.462 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.161674+0000", 2024-09-21T10:22:09.462 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:09.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.161674+0000", 2024-09-21T10:22:09.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:09.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.161676+0000", 2024-09-21T10:22:09.463 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.39e-06 2024-09-21T10:22:09.463 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.463 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.463 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:09.463 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.161685+0000", 2024-09-21T10:22:09.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0580000000000004e-06 2024-09-21T10:22:09.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:09.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.161706+0000", 2024-09-21T10:22:09.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1036e-05 2024-09-21T10:22:09.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:09.464 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.161716+0000", 2024-09-21T10:22:09.464 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8120000000000002e-06 2024-09-21T10:22:09.464 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.464 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.464 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.161907+0000", 2024-09-21T10:22:09.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019061800000000001 2024-09-21T10:22:09.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.161931+0000", 2024-09-21T10:22:09.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3787000000000001e-05 2024-09-21T10:22:09.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.465 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.465 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.465 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.173906+0000", 2024-09-21T10:22:09.465 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0119751219999999 2024-09-21T10:22:09.465 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:09.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.173983+0000", 2024-09-21T10:22:09.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6921000000000004e-05 2024-09-21T10:22:09.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:22:09.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.174049+0000", 2024-09-21T10:22:09.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5901999999999993e-05 2024-09-21T10:22:09.466 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.466 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.466 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:09.466 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.178768+0000", 2024-09-21T10:22:09.466 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0047195259999999999 2024-09-21T10:22:09.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179184+0000", 2024-09-21T10:22:09.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041583100000000002 2024-09-21T10:22:09.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179325+0000", 2024-09-21T10:22:09.467 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014086099999999999 2024-09-21T10:22:09.467 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.467 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.467 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:09.467 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179365+0000", 2024-09-21T10:22:09.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0265999999999997e-05 2024-09-21T10:22:09.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.468 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:09.468 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179395+0000", 2024-09-21T10:22:09.468 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9575999999999999e-05 2024-09-21T10:22:09.468 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.468 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.468 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.468 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.468 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.468 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:249570 13.5 13:ac098df3:::benchmark_data_smithi107_278655_object121663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-09-21T10:22:09.468 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:22:07.163153+0000", 2024-09-21T10:22:09.468 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 2.1662963560000001, 2024-09-21T10:22:09.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0163782960000001, 2024-09-21T10:22:09.469 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:09.469 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:09.469 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:09.469 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:09.469 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:22:09.469 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:22:09.469 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 249570 2024-09-21T10:22:09.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.469 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:09.469 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.469 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:09.469 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.163153+0000", 2024-09-21T10:22:09.469 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.469 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.470 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:09.470 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.163153+0000", 2024-09-21T10:22:09.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:09.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.470 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:09.470 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.163154+0000", 2024-09-21T10:22:09.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.322e-06 2024-09-21T10:22:09.470 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.470 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.470 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:09.470 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.163160+0000", 2024-09-21T10:22:09.470 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.699e-06 2024-09-21T10:22:09.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.471 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:09.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.163179+0000", 2024-09-21T10:22:09.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9046999999999999e-05 2024-09-21T10:22:09.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.471 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:09.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.163188+0000", 2024-09-21T10:22:09.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2730000000000002e-06 2024-09-21T10:22:09.471 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.471 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.471 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.471 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.163241+0000", 2024-09-21T10:22:09.471 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3659000000000002e-05 2024-09-21T10:22:09.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:09.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.163248+0000", 2024-09-21T10:22:09.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5100000000000004e-06 2024-09-21T10:22:09.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:09.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.176127+0000", 2024-09-21T10:22:09.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0128796929999999 2024-09-21T10:22:09.472 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.472 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.472 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:09.472 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.176211+0000", 2024-09-21T10:22:09.472 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4028000000000003e-05 2024-09-21T10:22:09.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:22:09.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.176322+0000", 2024-09-21T10:22:09.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011076199999999999 2024-09-21T10:22:09.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:09.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179155+0000", 2024-09-21T10:22:09.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002832415 2024-09-21T10:22:09.473 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.473 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.473 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.473 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179427+0000", 2024-09-21T10:22:09.473 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000272286 2024-09-21T10:22:09.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:09.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179462+0000", 2024-09-21T10:22:09.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5499e-05 2024-09-21T10:22:09.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:09.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179503+0000", 2024-09-21T10:22:09.474 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0689999999999998e-05 2024-09-21T10:22:09.474 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:09.474 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:09.474 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:09.474 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179531+0000", 2024-09-21T10:22:09.475 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8413000000000001e-05 2024-09-21T10:22:09.475 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.475 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.475 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.475 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:09.475 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:09.475 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:22:09.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:09.476 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:09.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:09.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:09.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:09.610 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:09.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:09.711 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:09.740 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:09.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:09.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:09.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:09.879 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:09.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:09.980 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:10.016 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:10.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:10.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:10.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:10.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:10.154 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:10.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:10.255 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:10.300 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:10.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:10.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:10.402 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:10.445 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:10.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:10.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:10.547 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:10.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:10.588 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:10.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:10.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:22:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:22:10.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:22:10.689 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:10.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:10.726 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:10.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:10.827 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:10.867 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:10.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:10.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:10.892 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:22:10.969 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:11.003 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:11.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:11.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:11.105 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:11.135 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:11.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:11.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:11.237 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:11.271 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:11.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:11.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:11.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:11.398 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:11.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:11.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:11.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:11 smithi129 ceph-mon[88471]: pgmap v1399: 109 pgs: 11 peering, 98 active+clean; 375 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.25k op/s 2024-09-21T10:22:11.500 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:11.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:11.536 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:11.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:11.560 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:11 smithi107 ceph-mon[102221]: pgmap v1399: 109 pgs: 11 peering, 98 active+clean; 375 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.25k op/s 2024-09-21T10:22:11.561 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:11 smithi107 ceph-mon[105022]: pgmap v1399: 109 pgs: 11 peering, 98 active+clean; 375 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.25k op/s 2024-09-21T10:22:11.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:11.694 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:11.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:11.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:11.796 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:11.826 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:11.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:11.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:11.854 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:22:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:22:11] "GET /metrics HTTP/1.1" 200 36773 "" "Prometheus/2.51.0" 2024-09-21T10:22:11.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:11.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:11.970 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:11.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:12.070 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:12.105 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:12.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:12.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:12.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:12.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:12.248 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:12.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:12.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:12.390 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:12.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:12.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:12.477 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:12.477 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:12.478 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:12.478 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:12.486 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:12.492 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:12.521 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:12.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:12.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:12.559 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:22:12.559 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:22:12.593 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:22:12.594 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:22:12.623 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:12.626 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:22:12.626 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:22:12.652 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:12.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:12.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:12.659 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:22:12.659 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:22:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:12.689 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:22:12.690 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:22:12.721 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:22:12.721 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:22:12.750 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:22:12.751 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:22:12.754 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:12.784 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:22:12.785 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:22:12.788 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:12.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:12.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:12.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:12.924 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:12.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:12.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:13.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:13.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:13.059 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:13.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:13.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:13.194 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:13.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:13.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:13.203 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:22:13.296 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:13.327 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:13.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:13.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:13.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:13 smithi107 ceph-mon[102221]: pgmap v1400: 109 pgs: 11 peering, 98 active+clean; 375 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 793 op/s 2024-09-21T10:22:13.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:13 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4093844821' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:22:13.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:13 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3933736439' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:13.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:13 smithi107 ceph-mon[105022]: pgmap v1400: 109 pgs: 11 peering, 98 active+clean; 375 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 793 op/s 2024-09-21T10:22:13.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:13 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4093844821' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:22:13.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:13 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3933736439' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:13.429 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:13.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:13.470 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:13.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:13.570 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:13.599 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:13.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:13.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:13 smithi129 ceph-mon[88471]: pgmap v1400: 109 pgs: 11 peering, 98 active+clean; 375 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 793 op/s 2024-09-21T10:22:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:13 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4093844821' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:22:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:13 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3933736439' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:13.702 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:13.713 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 4] out_osds: [6] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:22:13.713 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:22:13.713 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-09-21T10:22:13.714 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd in 6 2024-09-21T10:22:13.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:13.734 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:13.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:13.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:13.878 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:13.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:13.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:13.980 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:14.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:14.012 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:14.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:14.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:14.151 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:14.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:14.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:14.159 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Removed 127904 objects 2024-09-21T10:22:14.159 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Clean up completed and total clean up time :142.079 2024-09-21T10:22:14.253 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:14.274 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:14.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:14.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:14.376 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:14.397 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:14.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:14.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:14.499 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:14.514 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:14 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:22:14.514 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:14 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4093844821' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:22:14.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:14 smithi107 ceph-mon[102221]: osdmap e401: 8 total, 8 up, 7 in 2024-09-21T10:22:14.515 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:14 smithi107 ceph-mon[102221]: pgmap v1402: 109 pgs: 109 active+clean; 201 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.16k op/s 2024-09-21T10:22:14.515 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:14 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:22:14.515 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:14 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4093844821' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:22:14.515 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:14 smithi107 ceph-mon[105022]: osdmap e401: 8 total, 8 up, 7 in 2024-09-21T10:22:14.516 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:14 smithi107 ceph-mon[105022]: pgmap v1402: 109 pgs: 109 active+clean; 201 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.16k op/s 2024-09-21T10:22:14.522 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:14.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:14.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:14.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:14.648 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:14.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:14.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:14 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:22:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:14 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4093844821' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:22:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:14 smithi129 ceph-mon[88471]: osdmap e401: 8 total, 8 up, 7 in 2024-09-21T10:22:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:14 smithi129 ceph-mon[88471]: pgmap v1402: 109 pgs: 109 active+clean; 201 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.16k op/s 2024-09-21T10:22:14.749 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:14.772 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:14.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:14.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:14.873 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:14.898 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:14.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:14.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:14.999 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:15.022 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:15.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:15.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:15.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:15.146 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:15.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:15.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:15.248 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:15.271 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:15.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:15.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:15.373 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:15.397 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:15.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:15.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:15.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:15.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:15.523 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:15.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:15.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:15 smithi107 ceph-mon[102221]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 9 pgs peering) 2024-09-21T10:22:15.525 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:15 smithi107 ceph-mon[105022]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 9 pgs peering) 2024-09-21T10:22:15.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:15.645 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:15.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:15.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:15.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:15 smithi129 ceph-mon[88471]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 9 pgs peering) 2024-09-21T10:22:15.746 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:15.772 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:15.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:15.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:15.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:15.896 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:15.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:15.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:15.997 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:16.022 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:16.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:16.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:16.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:16.144 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:16.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:16.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:16.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:16.272 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:16.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:16.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:16.373 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:16.399 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:16.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:16.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:16.494 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:16 smithi107 ceph-mon[102221]: pgmap v1403: 109 pgs: 109 active+clean; 81 MiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-21T10:22:16.494 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:16 smithi107 ceph-mon[105022]: pgmap v1403: 109 pgs: 109 active+clean; 81 MiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-21T10:22:16.500 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:16.504 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:22:16.504 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:22:16.525 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:16.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:16.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:16.627 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:16.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:16.650 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:16.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:16.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:16 smithi129 ceph-mon[88471]: pgmap v1403: 109 pgs: 109 active+clean; 81 MiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-21T10:22:16.750 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:16.776 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:16.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:16.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:16.877 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:16.903 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:16.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:16.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:17.005 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:17.030 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:17.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:17.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:17.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:22:17.132 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:17.157 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:17.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:17.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:17.176 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:17.259 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:17.283 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:17.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:17.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:17.289 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:22:17.385 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:17.411 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:17.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:17.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:17.512 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:17.537 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:17.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:17.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:17.639 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:17.665 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:17.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:17.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:17.767 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:17.792 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:17.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:17.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:17.894 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:17.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:17.917 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:17.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:18.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:18.040 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:18.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:18.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:18.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:18.163 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:18.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:18.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:18.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:22:18.265 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:18.291 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:18.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:18.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:18.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:18.412 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:18.413 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:18.417 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:18.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:18.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:18.519 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:18.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:18.542 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:18.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:18.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:18 smithi107 ceph-mon[102221]: pgmap v1404: 109 pgs: 109 active+clean; 50 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 734 op/s 2024-09-21T10:22:18.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:18 smithi107 ceph-mon[105022]: pgmap v1404: 109 pgs: 109 active+clean; 50 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 734 op/s 2024-09-21T10:22:18.643 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:18.668 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:18.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:18.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:18.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:18.792 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:18.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:18.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:18.893 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:18.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:18.916 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:18.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:18 smithi129 ceph-mon[88471]: pgmap v1404: 109 pgs: 109 active+clean; 50 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 734 op/s 2024-09-21T10:22:19.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:19.038 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:19.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:19.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:19.140 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:19.162 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:19.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:19.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:19.263 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:19.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:19.287 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:19.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:19.388 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:19.409 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:19.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:19.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:19.507 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:19 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a[102198]: 2024-09-21T10:22:19.114+0000 7f3189124640 -1 mon.a@0(leader).osd e401 definitely_dead 0 2024-09-21T10:22:19.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:19.532 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:19.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:19.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:19.571 INFO:teuthology.orchestra.run.smithi107.stderr:marked in osd.6. 2024-09-21T10:22:19.633 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:19.669 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:19.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:19.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:19.771 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:19.794 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:19.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:19.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:19 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3287467808' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-21T10:22:19.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:19 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3287467808' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-21T10:22:19.895 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:19.919 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:19 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3287467808' entity='client.admin' cmd={"prefix": "osd in", "ids": ["6"]} : dispatch 2024-09-21T10:22:19.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:19.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:19.958 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_8 2024-09-21T10:22:19.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_8 unique_pool_8 --yes-i-really-really-mean-it 2024-09-21T10:22:20.020 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:20.043 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:20.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:20.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:20.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:20.166 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:20.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:20.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:20.267 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:20.291 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:20.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:20.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:20.393 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:20.416 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:20.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:20.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:20.517 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:20.542 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:20.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:20.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:20.572 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:22:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:22:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:22:20.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:20.668 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:20.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:20.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:20.749 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:20 smithi107 ceph-mon[102221]: pgmap v1405: 109 pgs: 109 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 598 op/s 2024-09-21T10:22:20.749 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:20 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3287467808' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-21T10:22:20.749 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:20 smithi107 ceph-mon[102221]: osdmap e402: 8 total, 8 up, 8 in 2024-09-21T10:22:20.749 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:20 smithi107 ceph-mon[102221]: osdmap e403: 8 total, 8 up, 8 in 2024-09-21T10:22:20.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:20 smithi107 ceph-mon[105022]: pgmap v1405: 109 pgs: 109 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 598 op/s 2024-09-21T10:22:20.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:20 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3287467808' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-21T10:22:20.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:20 smithi107 ceph-mon[105022]: osdmap e402: 8 total, 8 up, 8 in 2024-09-21T10:22:20.749 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:20 smithi107 ceph-mon[105022]: osdmap e403: 8 total, 8 up, 8 in 2024-09-21T10:22:20.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:20.773 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-09-21T10:22:20.835 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:20.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:20.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:20 smithi129 ceph-mon[88471]: pgmap v1405: 109 pgs: 109 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 598 op/s 2024-09-21T10:22:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:20 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3287467808' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-21T10:22:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:20 smithi129 ceph-mon[88471]: osdmap e402: 8 total, 8 up, 8 in 2024-09-21T10:22:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:20 smithi129 ceph-mon[88471]: osdmap e403: 8 total, 8 up, 8 in 2024-09-21T10:22:20.937 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:20.960 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:20.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:20.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:21.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:21.083 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:21.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:21.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:21.184 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:21.205 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:21.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:21.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:21.307 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:21.328 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:21.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:21.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:21.430 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:21.453 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:21.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:21.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:21.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:21.576 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:21.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:21.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:21.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:21.699 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:21.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:21.768 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:21 smithi107 ceph-mon[102221]: 13.8 starting backfill to osd.6 from (372'2385,384'10374] MIN to 401'16100 2024-09-21T10:22:21.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:21 smithi107 ceph-mon[102221]: 13.f starting backfill to osd.6 from (376'6294,392'13053] MIN to 401'16030 2024-09-21T10:22:21.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:21 smithi107 ceph-mon[102221]: 13.1 starting backfill to osd.6 from (373'4400,390'12361] MIN to 401'16030 2024-09-21T10:22:21.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:21 smithi107 ceph-mon[102221]: osdmap e404: 8 total, 8 up, 8 in 2024-09-21T10:22:21.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:21 smithi107 ceph-mon[102221]: 13.7 starting backfill to osd.6 from (373'4647,387'11321] MIN to 401'15996 2024-09-21T10:22:21.769 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:21 smithi107 ceph-mon[102221]: 13.4 starting backfill to osd.6 from (376'6400,395'14295] MIN to 401'16086 2024-09-21T10:22:21.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:21 smithi107 ceph-mon[105022]: 13.8 starting backfill to osd.6 from (372'2385,384'10374] MIN to 401'16100 2024-09-21T10:22:21.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:21 smithi107 ceph-mon[105022]: 13.f starting backfill to osd.6 from (376'6294,392'13053] MIN to 401'16030 2024-09-21T10:22:21.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:21 smithi107 ceph-mon[105022]: 13.1 starting backfill to osd.6 from (373'4400,390'12361] MIN to 401'16030 2024-09-21T10:22:21.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:21 smithi107 ceph-mon[105022]: osdmap e404: 8 total, 8 up, 8 in 2024-09-21T10:22:21.769 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:21 smithi107 ceph-mon[105022]: 13.7 starting backfill to osd.6 from (373'4647,387'11321] MIN to 401'15996 2024-09-21T10:22:21.770 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:21 smithi107 ceph-mon[105022]: 13.4 starting backfill to osd.6 from (376'6400,395'14295] MIN to 401'16086 2024-09-21T10:22:21.801 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:21.824 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:21.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:21.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:21.826 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:21 smithi129 ceph-mon[88471]: 13.8 starting backfill to osd.6 from (372'2385,384'10374] MIN to 401'16100 2024-09-21T10:22:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:21 smithi129 ceph-mon[88471]: 13.f starting backfill to osd.6 from (376'6294,392'13053] MIN to 401'16030 2024-09-21T10:22:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:21 smithi129 ceph-mon[88471]: 13.1 starting backfill to osd.6 from (373'4400,390'12361] MIN to 401'16030 2024-09-21T10:22:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:21 smithi129 ceph-mon[88471]: osdmap e404: 8 total, 8 up, 8 in 2024-09-21T10:22:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:21 smithi129 ceph-mon[88471]: 13.7 starting backfill to osd.6 from (373'4647,387'11321] MIN to 401'15996 2024-09-21T10:22:21.920 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:21 smithi129 ceph-mon[88471]: 13.4 starting backfill to osd.6 from (376'6400,395'14295] MIN to 401'16086 2024-09-21T10:22:21.926 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:21.947 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:21.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:21.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:22.049 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:22.070 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:22.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:22.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:22.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:22:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:22:21] "GET /metrics HTTP/1.1" 200 36773 "" "Prometheus/2.51.0" 2024-09-21T10:22:22.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:22.195 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:22.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:22.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:22.297 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:22.319 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:22.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:22.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:22.393 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:22.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:22.441 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:22.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:22.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:22.543 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:22.564 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:22.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:22.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:22.666 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:22.691 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:22.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:22.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:22.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:22.814 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:22.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:22.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:22.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:22 smithi107 ceph-mon[102221]: pgmap v1409: 109 pgs: 109 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 213 op/s 2024-09-21T10:22:22.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:22 smithi107 ceph-mon[102221]: osdmap e405: 8 total, 8 up, 8 in 2024-09-21T10:22:22.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:22 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:22.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:22 smithi107 ceph-mon[105022]: pgmap v1409: 109 pgs: 109 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 213 op/s 2024-09-21T10:22:22.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:22 smithi107 ceph-mon[105022]: osdmap e405: 8 total, 8 up, 8 in 2024-09-21T10:22:22.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:22 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:22.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:22 smithi129 ceph-mon[88471]: pgmap v1409: 109 pgs: 109 active+clean; 582 KiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 213 op/s 2024-09-21T10:22:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:22 smithi129 ceph-mon[88471]: osdmap e405: 8 total, 8 up, 8 in 2024-09-21T10:22:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:22 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:22.937 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:22.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:22.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:23.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:23.060 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:23.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:23.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:23.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:23.183 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:23.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:23.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:23.284 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:23.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:23.308 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:23.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:23.409 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:23.430 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:23.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:23.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:23.532 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:23.554 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:23.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:23.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:23.656 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:23.677 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:23.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:23.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:23.778 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:23.800 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:23.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:23.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:23.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:23 smithi107 ceph-mon[102221]: osdmap e406: 8 total, 8 up, 8 in 2024-09-21T10:22:23.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:23 smithi107 ceph-mon[105022]: osdmap e406: 8 total, 8 up, 8 in 2024-09-21T10:22:23.902 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:23.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:23 smithi129 ceph-mon[88471]: osdmap e406: 8 total, 8 up, 8 in 2024-09-21T10:22:23.923 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:23.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:23.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:24.025 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:24.047 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:24.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:24.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:24.149 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:24.170 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:24.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:24.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:24.272 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:24.294 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:24.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:24.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:24.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:24.417 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:24.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:24.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:24.519 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:24.540 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:24.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:24.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:24.641 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:24.665 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:24.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:24.767 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:24.790 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:24.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:24.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:24.892 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:24.913 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:24.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:24.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:24.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:24 smithi129 ceph-mon[88471]: pgmap v1412: 109 pgs: 3 peering, 106 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 82 KiB/s, 2 objects/s recovering 2024-09-21T10:22:24.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:24 smithi129 ceph-mon[88471]: osdmap e407: 8 total, 8 up, 8 in 2024-09-21T10:22:24.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:24 smithi107 ceph-mon[102221]: pgmap v1412: 109 pgs: 3 peering, 106 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 82 KiB/s, 2 objects/s recovering 2024-09-21T10:22:24.971 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:24 smithi107 ceph-mon[102221]: osdmap e407: 8 total, 8 up, 8 in 2024-09-21T10:22:24.972 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:24 smithi107 ceph-mon[105022]: pgmap v1412: 109 pgs: 3 peering, 106 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 82 KiB/s, 2 objects/s recovering 2024-09-21T10:22:24.972 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:24 smithi107 ceph-mon[105022]: osdmap e407: 8 total, 8 up, 8 in 2024-09-21T10:22:25.015 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:25.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:25.042 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:25.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:25.067 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:22:25.067 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:22:25.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:25.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:25.166 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:25.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:25.267 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:25.292 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:25.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:25.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:25.394 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:25.415 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:25.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:25.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:25.517 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:25.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:25.544 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:25.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:25.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:25.667 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:25.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:25.769 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:25.773 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 4, 6] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:22:25.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T10:22:25.795 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:25.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:25.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:25.897 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:25.923 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:25.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:25.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:26.024 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:26.050 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:26.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:26.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:26.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:26.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:26.175 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:26.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:26.276 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:26.302 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:26.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:26.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:26.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:22:26.404 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:26.430 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:26.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:26.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:26.454 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:22:26.454 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:22:26.454 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:22:26.455 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:22:26.455 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:22:26.455 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:22:26.455 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:22:26.455 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:22:26.455 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:22:26.455 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:22:26.455 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:22:26.455 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:22:26.455 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:22:26.455 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:22:26.456 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:22:26.456 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:22:26.532 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:26.557 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:26.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:26.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:26.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:26.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:26.681 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:26.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:26.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:26.803 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:26.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:26.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:26.905 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:26.931 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:26.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:26.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:27.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:27.056 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:27.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:27.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:27.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:27.181 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:27.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:27.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:27.283 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:27.309 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:27.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:27.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:27.411 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:27.437 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:27.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:27.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:27.528 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:27 smithi107 ceph-mon[102221]: pgmap v1414: 109 pgs: 3 peering, 106 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 67 KiB/s, 2 objects/s recovering 2024-09-21T10:22:27.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:27.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:27.529 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:27 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2342753554' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:27.529 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:27 smithi107 ceph-mon[105022]: pgmap v1414: 109 pgs: 3 peering, 106 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 67 KiB/s, 2 objects/s recovering 2024-09-21T10:22:27.529 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:27.529 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:27.529 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:27 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2342753554' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:27.539 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:27.561 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:27.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:27.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:27.662 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:27.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:27 smithi129 ceph-mon[88471]: pgmap v1414: 109 pgs: 3 peering, 106 active+clean; 582 KiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 67 KiB/s, 2 objects/s recovering 2024-09-21T10:22:27.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:27.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:27.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:27 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2342753554' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:27.687 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:27.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:27.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:27.782 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:27.782 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:27.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:27.811 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:27.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:27.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:27.913 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:27.938 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:27.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:27.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:28.040 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:28.065 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:28.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:28.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:28.168 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:28.187 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_8' removed 2024-09-21T10:22:28.196 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:28.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:28.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:28.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:28.320 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:28.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:28.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:28.421 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:28.444 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:28.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:28.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:28.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:28.567 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:28.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:28.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:28.595 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:28 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/40345416' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:22:28.595 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:28 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:22:28.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:28 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/40345416' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:22:28.595 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:28 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:22:28.669 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:28.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:28 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/40345416' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:22:28.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:28 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:22:28.692 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:28.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:28.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:28.794 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:28.817 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:28.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:28.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:28.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:28.941 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:28.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:28.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:29.042 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:29.064 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:29.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:29.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:29.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:29.184 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:22:29.188 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:22:29.190 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:29.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:29.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:29.292 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:29.313 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:29.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:29.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:29.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:29.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:29.438 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:29.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:29.539 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:29.560 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:29.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:29.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:29 smithi107 ceph-mon[102221]: pgmap v1415: 109 pgs: 109 active+clean; 582 KiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 54 KiB/s, 1 objects/s recovering 2024-09-21T10:22:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:29 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:22:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:29 smithi107 ceph-mon[102221]: osdmap e408: 8 total, 8 up, 8 in 2024-09-21T10:22:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:29 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1655164649' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:22:29.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:29 smithi107 ceph-mon[105022]: pgmap v1415: 109 pgs: 109 active+clean; 582 KiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 54 KiB/s, 1 objects/s recovering 2024-09-21T10:22:29.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:29 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:22:29.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:29 smithi107 ceph-mon[105022]: osdmap e408: 8 total, 8 up, 8 in 2024-09-21T10:22:29.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:29 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1655164649' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:22:29.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:29 smithi129 ceph-mon[88471]: pgmap v1415: 109 pgs: 109 active+clean; 582 KiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 54 KiB/s, 1 objects/s recovering 2024-09-21T10:22:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:29 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_8", "pool2": "unique_pool_8", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:22:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:29 smithi129 ceph-mon[88471]: osdmap e408: 8 total, 8 up, 8 in 2024-09-21T10:22:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:29 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1655164649' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:22:29.685 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:29.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:29.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:29.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:29.808 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:29.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:29.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:29.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:29.931 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:29.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:29.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:30.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:30.055 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:30.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:30.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:30.157 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:30.183 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:30.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:30.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:30.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:30.307 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:30.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:30.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:30.348 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:30 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:22:30.348 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:30 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1655164649' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:22:30.348 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:30 smithi107 ceph-mon[102221]: osdmap e409: 8 total, 8 up, 8 in 2024-09-21T10:22:30.348 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:30 smithi107 ceph-mon[102221]: pgmap v1418: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:30.349 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:30 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:22:30.349 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:30 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1655164649' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:22:30.349 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:30 smithi107 ceph-mon[105022]: osdmap e409: 8 total, 8 up, 8 in 2024-09-21T10:22:30.349 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:30 smithi107 ceph-mon[105022]: pgmap v1418: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:30.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:30.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:30.435 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:30.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:30.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:30.557 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:30.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:30.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:30.659 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:30 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:22:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:30 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1655164649' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:22:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:30 smithi129 ceph-mon[88471]: osdmap e409: 8 total, 8 up, 8 in 2024-09-21T10:22:30.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:30 smithi129 ceph-mon[88471]: pgmap v1418: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:22:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:22:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:22:30.685 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:30.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:30.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:30.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:30.809 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:30.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:30.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:30.911 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:30.937 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:30.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:30.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:31.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:31.061 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:31.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:31.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:31.164 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:31.189 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:31.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:31.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:31.291 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:31.317 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:31.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:31.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:31.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:31.441 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:31.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:31.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:31.543 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:31.570 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:31.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:31.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:31.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:31.694 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:31.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:31.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:31.796 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:31.822 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:31.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:31.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:31.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:31.948 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:31.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:31.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:32.049 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:32.075 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:32.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:32.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:22:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:22:31] "GET /metrics HTTP/1.1" 200 36780 "" "Prometheus/2.51.0" 2024-09-21T10:22:32.176 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:32.202 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:32.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:32.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:32.304 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:32.330 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:32.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:32.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:32.432 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:32.457 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:32.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:32.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:32.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:32.586 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:32.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:32.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:32.688 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:32.712 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:32.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:32.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:32.814 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:32.838 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:32.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:32.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:32.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:32 smithi107 ceph-mon[102221]: pgmap v1419: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:32.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:32 smithi107 ceph-mon[105022]: pgmap v1419: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:32 smithi129 ceph-mon[88471]: pgmap v1419: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:32.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:32.963 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:32.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:32.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:33.065 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:33.091 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:33.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:33.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:33.193 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:33.218 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:33.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:33.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:33.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:33.344 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:33.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:33.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:33.446 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:33.471 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:33.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:33.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:33.598 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:33.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:33.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:33.700 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:33.726 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:33.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:33.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:33.804 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:22:33.805 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-09-21T10:22:33.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-09-21T10:22:33.805 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:22:33.805 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.805 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:46078 12.6 12:6570b22e:::benchmark_data_smithi107_250110_object46077:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-09-21T10:22:33.805 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:34.307148+0000", 2024-09-21T10:22:33.805 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 479.48700656400001, 2024-09-21T10:22:33.805 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.34402638499999999, 2024-09-21T10:22:33.805 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:33.805 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:33.806 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:33.806 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:33.806 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:22:33.806 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:22:33.806 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 46078 2024-09-21T10:22:33.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.806 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:33.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:33.806 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.307148+0000", 2024-09-21T10:22:33.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.806 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.806 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.806 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:33.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.307148+0000", 2024-09-21T10:22:33.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:33.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.307149+0000", 2024-09-21T10:22:33.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.471e-06 2024-09-21T10:22:33.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.807 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:33.807 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.307171+0000", 2024-09-21T10:22:33.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2093e-05 2024-09-21T10:22:33.807 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:33.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.307181+0000", 2024-09-21T10:22:33.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5370000000000003e-06 2024-09-21T10:22:33.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:33.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.307187+0000", 2024-09-21T10:22:33.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1419999999999998e-06 2024-09-21T10:22:33.808 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.808 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.307227+0000", 2024-09-21T10:22:33.808 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9944000000000002e-05 2024-09-21T10:22:33.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:33.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.307359+0000", 2024-09-21T10:22:33.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013215199999999999 2024-09-21T10:22:33.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:22:33.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.307489+0000", 2024-09-21T10:22:33.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000129617 2024-09-21T10:22:33.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:33.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.309941+0000", 2024-09-21T10:22:33.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024522160000000001 2024-09-21T10:22:33.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.310524+0000", 2024-09-21T10:22:33.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00058306899999999997 2024-09-21T10:22:33.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315276+0000", 2024-09-21T10:22:33.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004752076 2024-09-21T10:22:33.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:33.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315322+0000", 2024-09-21T10:22:33.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6217000000000002e-05 2024-09-21T10:22:33.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:33.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651174+0000", 2024-09-21T10:22:33.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.33585185099999998 2024-09-21T10:22:33.812 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.812 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.812 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.812 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:46089 12.7 12:ee14e570:::benchmark_data_smithi107_250110_object46088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-09-21T10:22:33.812 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:34.311237+0000", 2024-09-21T10:22:33.812 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 479.48291773800003, 2024-09-21T10:22:33.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.34013115999999999, 2024-09-21T10:22:33.812 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:33.812 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:33.812 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:33.812 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:33.813 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:22:33.813 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:22:33.813 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 46089 2024-09-21T10:22:33.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.813 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:33.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:33.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.311237+0000", 2024-09-21T10:22:33.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:33.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.311237+0000", 2024-09-21T10:22:33.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:33.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.311238+0000", 2024-09-21T10:22:33.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6500000000000001e-06 2024-09-21T10:22:33.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:33.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.311262+0000", 2024-09-21T10:22:33.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3453000000000001e-05 2024-09-21T10:22:33.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:33.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.311274+0000", 2024-09-21T10:22:33.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2310000000000001e-05 2024-09-21T10:22:33.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:33.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.311282+0000", 2024-09-21T10:22:33.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5730000000000002e-06 2024-09-21T10:22:33.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.311338+0000", 2024-09-21T10:22:33.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.626e-05 2024-09-21T10:22:33.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:33.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.311472+0000", 2024-09-21T10:22:33.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013357700000000001 2024-09-21T10:22:33.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:22:33.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.311596+0000", 2024-09-21T10:22:33.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012442900000000001 2024-09-21T10:22:33.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:33.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.313715+0000", 2024-09-21T10:22:33.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021186820000000002 2024-09-21T10:22:33.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.314073+0000", 2024-09-21T10:22:33.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035868 2024-09-21T10:22:33.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651265+0000", 2024-09-21T10:22:33.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.33719125300000002 2024-09-21T10:22:33.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.817 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:33.817 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651325+0000", 2024-09-21T10:22:33.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0223999999999998e-05 2024-09-21T10:22:33.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.817 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.818 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:33.818 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651368+0000", 2024-09-21T10:22:33.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3068999999999997e-05 2024-09-21T10:22:33.818 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.818 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.818 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.818 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.818 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.36822.0:46083 12.d e305/294 12:b93ee741:::benchmark_data_smithi107_250110_object46082:head v 305'2864, mlcod=305'2860)", 2024-09-21T10:22:33.818 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:34.314801+0000", 2024-09-21T10:22:33.818 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 479.47935367399998, 2024-09-21T10:22:33.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.33645499499999998, 2024-09-21T10:22:33.818 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:33.819 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:33.819 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:33.819 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:33.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:33.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.314801+0000", 2024-09-21T10:22:33.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.819 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:33.819 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.314801+0000", 2024-09-21T10:22:33.819 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:33.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.314801+0000", 2024-09-21T10:22:33.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9700000000000003e-07 2024-09-21T10:22:33.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:33.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.314841+0000", 2024-09-21T10:22:33.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9743999999999998e-05 2024-09-21T10:22:33.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:33.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.314848+0000", 2024-09-21T10:22:33.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1210000000000001e-06 2024-09-21T10:22:33.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:33.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.314853+0000", 2024-09-21T10:22:33.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7659999999999998e-06 2024-09-21T10:22:33.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315050+0000", 2024-09-21T10:22:33.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019765900000000001 2024-09-21T10:22:33.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:33.822 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315067+0000", 2024-09-21T10:22:33.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6878999999999999e-05 2024-09-21T10:22:33.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.822 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:33.822 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651212+0000", 2024-09-21T10:22:33.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.33614440200000001 2024-09-21T10:22:33.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.822 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.822 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:33.822 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651256+0000", 2024-09-21T10:22:33.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4127e-05 2024-09-21T10:22:33.822 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.823 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.823 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.823 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.36822.0:46078 12.6 e305/298 ondisk, result = 0)", 2024-09-21T10:22:33.823 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:34.315124+0000", 2024-09-21T10:22:33.823 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 479.479030567, 2024-09-21T10:22:33.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.33606297099999999, 2024-09-21T10:22:33.823 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:33.823 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:33.823 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:22:33.823 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:33.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:33.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315124+0000", 2024-09-21T10:22:33.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:33.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315124+0000", 2024-09-21T10:22:33.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:33.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315125+0000", 2024-09-21T10:22:33.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5700000000000002e-07 2024-09-21T10:22:33.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:33.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315129+0000", 2024-09-21T10:22:33.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9250000000000003e-06 2024-09-21T10:22:33.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:33.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315140+0000", 2024-09-21T10:22:33.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0693000000000001e-05 2024-09-21T10:22:33.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:33.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315146+0000", 2024-09-21T10:22:33.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2539999999999998e-06 2024-09-21T10:22:33.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315264+0000", 2024-09-21T10:22:33.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000117442 2024-09-21T10:22:33.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:33.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.315271+0000", 2024-09-21T10:22:33.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1080000000000004e-06 2024-09-21T10:22:33.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:33.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651187+0000", 2024-09-21T10:22:33.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.33591589199999999 2024-09-21T10:22:33.826 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.827 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.827 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.827 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop_reply(client.36822.0:46089 12.7 e305/298 ondisk, result = 0)", 2024-09-21T10:22:33.827 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:34.316234+0000", 2024-09-21T10:22:33.827 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 479.47792063700001, 2024-09-21T10:22:33.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.33514012599999998, 2024-09-21T10:22:33.827 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:33.827 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:33.827 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:22:33.827 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:33.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:33.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.316234+0000", 2024-09-21T10:22:33.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:33.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.316234+0000", 2024-09-21T10:22:33.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.828 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:33.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.316234+0000", 2024-09-21T10:22:33.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7599999999999997e-07 2024-09-21T10:22:33.828 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.828 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:33.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.316239+0000", 2024-09-21T10:22:33.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7779999999999998e-06 2024-09-21T10:22:33.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:33.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651187+0000", 2024-09-21T10:22:33.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.33494754599999998 2024-09-21T10:22:33.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.829 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:33.829 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651197+0000", 2024-09-21T10:22:33.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0327e-05 2024-09-21T10:22:33.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651248+0000", 2024-09-21T10:22:33.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1161e-05 2024-09-21T10:22:33.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:33.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651257+0000", 2024-09-21T10:22:33.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0040000000000005e-06 2024-09-21T10:22:33.830 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.830 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:33.830 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651374+0000", 2024-09-21T10:22:33.830 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011663399999999999 2024-09-21T10:22:33.831 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.831 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.831 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.831 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.36822.0:46096 12.7 12:eb5f27db:::benchmark_data_smithi107_250110_object46095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-09-21T10:22:33.831 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:14:34.316948+0000", 2024-09-21T10:22:33.831 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 479.47720617499999, 2024-09-21T10:22:33.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.343015389, 2024-09-21T10:22:33.831 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:33.831 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:33.831 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:33.831 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:33.831 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.36822", 2024-09-21T10:22:33.832 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:22:33.832 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 46096 2024-09-21T10:22:33.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.832 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:33.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:33.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.316948+0000", 2024-09-21T10:22:33.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:33.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.316948+0000", 2024-09-21T10:22:33.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:33.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.316950+0000", 2024-09-21T10:22:33.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6029999999999999e-06 2024-09-21T10:22:33.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:33.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.316971+0000", 2024-09-21T10:22:33.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1094000000000002e-05 2024-09-21T10:22:33.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:33.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651187+0000", 2024-09-21T10:22:33.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.33421569200000001 2024-09-21T10:22:33.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:33.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651198+0000", 2024-09-21T10:22:33.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1413000000000001e-05 2024-09-21T10:22:33.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651406+0000", 2024-09-21T10:22:33.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00020795500000000001 2024-09-21T10:22:33.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:33.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651554+0000", 2024-09-21T10:22:33.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014769699999999999 2024-09-21T10:22:33.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:22:33.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.651700+0000", 2024-09-21T10:22:33.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014646799999999999 2024-09-21T10:22:33.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.657141+0000", 2024-09-21T10:22:33.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054412330000000002 2024-09-21T10:22:33.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:33.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.657222+0000", 2024-09-21T10:22:33.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0709000000000003e-05 2024-09-21T10:22:33.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.659458+0000", 2024-09-21T10:22:33.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0022354789999999999 2024-09-21T10:22:33.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:33.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.659919+0000", 2024-09-21T10:22:33.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00046138700000000001 2024-09-21T10:22:33.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:33.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:14:34.659964+0000", 2024-09-21T10:22:33.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4659000000000001e-05 2024-09-21T10:22:33.837 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.837 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.837 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.837 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:26926 13.c 13:3dd20042:::benchmark_data_smithi107_278655_object26925:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:22:33.837 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:18:31.489966+0000", 2024-09-21T10:22:33.837 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 242.304188302, 2024-09-21T10:22:33.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.30380030899999999, 2024-09-21T10:22:33.838 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:33.838 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:33.838 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:33.838 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:33.838 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:22:33.838 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:22:33.838 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 26926 2024-09-21T10:22:33.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.838 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:33.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:33.838 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.489966+0000", 2024-09-21T10:22:33.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:33.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.489966+0000", 2024-09-21T10:22:33.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:33.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.489968+0000", 2024-09-21T10:22:33.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7740000000000001e-06 2024-09-21T10:22:33.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.839 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:33.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.489991+0000", 2024-09-21T10:22:33.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3371999999999999e-05 2024-09-21T10:22:33.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:33.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.490004+0000", 2024-09-21T10:22:33.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3148e-05 2024-09-21T10:22:33.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:33.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.490027+0000", 2024-09-21T10:22:33.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2081000000000001e-05 2024-09-21T10:22:33.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.490119+0000", 2024-09-21T10:22:33.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2196999999999998e-05 2024-09-21T10:22:33.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:33.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.490294+0000", 2024-09-21T10:22:33.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017526999999999999 2024-09-21T10:22:33.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:22:33.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.490422+0000", 2024-09-21T10:22:33.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000127797 2024-09-21T10:22:33.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:33.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.491892+0000", 2024-09-21T10:22:33.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0014706859999999999 2024-09-21T10:22:33.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.492530+0000", 2024-09-21T10:22:33.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00063751900000000002 2024-09-21T10:22:33.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793657+0000", 2024-09-21T10:22:33.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.30112738100000003 2024-09-21T10:22:33.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:33.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793719+0000", 2024-09-21T10:22:33.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1260999999999995e-05 2024-09-21T10:22:33.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:33.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793766+0000", 2024-09-21T10:22:33.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7822999999999999e-05 2024-09-21T10:22:33.843 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.843 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.843 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.844 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.844 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:26943 13.c 13:3d5b5678:::benchmark_data_smithi107_278655_object26942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:22:33.844 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:18:31.495178+0000", 2024-09-21T10:22:33.844 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 242.29897640600001, 2024-09-21T10:22:33.844 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.29875716099999999, 2024-09-21T10:22:33.844 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:33.844 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:33.844 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:33.844 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:33.844 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:22:33.844 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:22:33.845 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 26943 2024-09-21T10:22:33.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.845 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:33.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:33.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.495178+0000", 2024-09-21T10:22:33.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:33.845 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.495178+0000", 2024-09-21T10:22:33.845 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.845 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.845 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.845 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:33.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.495180+0000", 2024-09-21T10:22:33.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.64e-06 2024-09-21T10:22:33.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:33.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.495202+0000", 2024-09-21T10:22:33.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2310000000000002e-05 2024-09-21T10:22:33.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.846 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:33.846 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.495214+0000", 2024-09-21T10:22:33.846 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1694e-05 2024-09-21T10:22:33.846 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.846 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:33.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.495221+0000", 2024-09-21T10:22:33.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.849e-06 2024-09-21T10:22:33.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.495272+0000", 2024-09-21T10:22:33.847 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1857999999999997e-05 2024-09-21T10:22:33.847 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.847 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.847 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:33.847 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.495405+0000", 2024-09-21T10:22:33.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013294899999999999 2024-09-21T10:22:33.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:22:33.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.495538+0000", 2024-09-21T10:22:33.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013217599999999999 2024-09-21T10:22:33.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:33.848 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.497277+0000", 2024-09-21T10:22:33.848 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017389840000000001 2024-09-21T10:22:33.848 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.848 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.848 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.497659+0000", 2024-09-21T10:22:33.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00038245899999999998 2024-09-21T10:22:33.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793846+0000", 2024-09-21T10:22:33.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.29618721399999998 2024-09-21T10:22:33.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.849 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:33.849 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793896+0000", 2024-09-21T10:22:33.849 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9543000000000003e-05 2024-09-21T10:22:33.849 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.849 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.850 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:33.850 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793935+0000", 2024-09-21T10:22:33.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9484999999999999e-05 2024-09-21T10:22:33.850 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.850 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.850 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.850 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.850 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.850 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.37050.0:26945 13.0 e372/367 13:0d6e0250:::benchmark_data_smithi107_278655_object26944:head v 372'1720, mlcod=372'1716)", 2024-09-21T10:22:33.850 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:18:31.497604+0000", 2024-09-21T10:22:33.850 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 242.296550242, 2024-09-21T10:22:33.850 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.29766990799999998, 2024-09-21T10:22:33.850 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:33.850 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:33.851 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:33.851 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:33.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:33.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.497604+0000", 2024-09-21T10:22:33.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:33.851 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.497604+0000", 2024-09-21T10:22:33.851 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.851 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.851 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.851 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:33.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.497605+0000", 2024-09-21T10:22:33.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3300000000000001e-07 2024-09-21T10:22:33.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:33.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.497724+0000", 2024-09-21T10:22:33.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011890799999999999 2024-09-21T10:22:33.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.852 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:33.852 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.497733+0000", 2024-09-21T10:22:33.852 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4110000000000002e-06 2024-09-21T10:22:33.852 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.852 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:33.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793254+0000", 2024-09-21T10:22:33.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.29552120599999998 2024-09-21T10:22:33.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793349+0000", 2024-09-21T10:22:33.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4561000000000005e-05 2024-09-21T10:22:33.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.853 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.853 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:33.853 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793376+0000", 2024-09-21T10:22:33.853 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6478e-05 2024-09-21T10:22:33.853 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:33.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.795235+0000", 2024-09-21T10:22:33.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0018596229999999999 2024-09-21T10:22:33.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.854 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.854 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:33.854 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.795274+0000", 2024-09-21T10:22:33.854 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8988e-05 2024-09-21T10:22:33.854 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.854 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.854 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.854 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.855 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.37050.0:26946 13.4 e372/367 13:21264d98:::benchmark_data_smithi107_278655_object26945:head v 372'1689, mlcod=372'1683)", 2024-09-21T10:22:33.855 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:18:31.498426+0000", 2024-09-21T10:22:33.855 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 242.29572851399999, 2024-09-21T10:22:33.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.30159318299999999, 2024-09-21T10:22:33.855 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:33.855 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:33.855 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:33.855 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:33.855 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.855 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:33.855 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.498426+0000", 2024-09-21T10:22:33.855 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:33.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.498426+0000", 2024-09-21T10:22:33.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:33.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.498427+0000", 2024-09-21T10:22:33.856 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9100000000000003e-07 2024-09-21T10:22:33.856 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.856 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.856 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:33.856 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.498492+0000", 2024-09-21T10:22:33.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5154999999999995e-05 2024-09-21T10:22:33.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:33.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793260+0000", 2024-09-21T10:22:33.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.294767642 2024-09-21T10:22:33.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:33.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793276+0000", 2024-09-21T10:22:33.857 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6750999999999999e-05 2024-09-21T10:22:33.857 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.857 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.857 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.857 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793802+0000", 2024-09-21T10:22:33.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00052541800000000004 2024-09-21T10:22:33.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:33.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793818+0000", 2024-09-21T10:22:33.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5909999999999998e-05 2024-09-21T10:22:33.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.858 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:33.858 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.799979+0000", 2024-09-21T10:22:33.858 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0061610099999999998 2024-09-21T10:22:33.858 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.858 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.859 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:33.859 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.800019+0000", 2024-09-21T10:22:33.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0506000000000001e-05 2024-09-21T10:22:33.859 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.859 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.859 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.859 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.859 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.859 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.37050.0:26956 13.d e372/367 13:bd5314ea:::benchmark_data_smithi107_278655_object26955:head v 372'1699, mlcod=372'1697)", 2024-09-21T10:22:33.859 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:18:31.503163+0000", 2024-09-21T10:22:33.859 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 242.29099156500001, 2024-09-21T10:22:33.859 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.293366558, 2024-09-21T10:22:33.859 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:33.859 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:33.859 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:33.860 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:33.860 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.860 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:33.860 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.503163+0000", 2024-09-21T10:22:33.861 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.861 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.861 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.861 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:33.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.503163+0000", 2024-09-21T10:22:33.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:33.862 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.503164+0000", 2024-09-21T10:22:33.862 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.184e-06 2024-09-21T10:22:33.862 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.862 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.862 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:33.862 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:33.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.503213+0000", 2024-09-21T10:22:33.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8591000000000003e-05 2024-09-21T10:22:33.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:33.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793270+0000", 2024-09-21T10:22:33.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.29005693500000002 2024-09-21T10:22:33.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:33.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793280+0000", 2024-09-21T10:22:33.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0529e-05 2024-09-21T10:22:33.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793650+0000", 2024-09-21T10:22:33.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00037032599999999998 2024-09-21T10:22:33.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:33.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.793670+0000", 2024-09-21T10:22:33.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9403999999999999e-05 2024-09-21T10:22:33.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:33.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.796490+0000", 2024-09-21T10:22:33.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0028202040000000002 2024-09-21T10:22:33.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:33.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:18:31.796529+0000", 2024-09-21T10:22:33.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9385000000000003e-05 2024-09-21T10:22:33.867 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.867 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.867 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.867 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:174961 13.c 13:3f32d38e:::benchmark_data_smithi107_278655_object47054:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-21T10:22:33.867 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:20:46.801401+0000", 2024-09-21T10:22:33.867 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 106.992753437, 2024-09-21T10:22:33.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.40734487200000002, 2024-09-21T10:22:33.867 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:33.867 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:33.867 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:33.867 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:33.867 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:22:33.868 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:22:33.868 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 174961 2024-09-21T10:22:33.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.868 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:33.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:33.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.801401+0000", 2024-09-21T10:22:33.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:33.868 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.801401+0000", 2024-09-21T10:22:33.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:33.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.801402+0000", 2024-09-21T10:22:33.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2729999999999999e-06 2024-09-21T10:22:33.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:33.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.801406+0000", 2024-09-21T10:22:33.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7790000000000002e-06 2024-09-21T10:22:33.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:33.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.801415+0000", 2024-09-21T10:22:33.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2040000000000003e-06 2024-09-21T10:22:33.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:33.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.801421+0000", 2024-09-21T10:22:33.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1659999999999999e-06 2024-09-21T10:22:33.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.801468+0000", 2024-09-21T10:22:33.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6181999999999999e-05 2024-09-21T10:22:33.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:33.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.801578+0000", 2024-09-21T10:22:33.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000110486 2024-09-21T10:22:33.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:22:33.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.801691+0000", 2024-09-21T10:22:33.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011253500000000001 2024-09-21T10:22:33.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:33.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.807645+0000", 2024-09-21T10:22:33.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0059540870000000003 2024-09-21T10:22:33.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.812317+0000", 2024-09-21T10:22:33.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046719819999999999 2024-09-21T10:22:33.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:47.208594+0000", 2024-09-21T10:22:33.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.39627713399999998 2024-09-21T10:22:33.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:33.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:47.208649+0000", 2024-09-21T10:22:33.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4752e-05 2024-09-21T10:22:33.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:33.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:47.208746+0000", 2024-09-21T10:22:33.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7292000000000003e-05 2024-09-21T10:22:33.873 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.873 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.873 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.873 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:174963 13.c 13:3f3333fa:::benchmark_data_smithi107_278655_object47056:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-21T10:22:33.873 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:20:46.804150+0000", 2024-09-21T10:22:33.873 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 106.990004024, 2024-09-21T10:22:33.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.40472338699999999, 2024-09-21T10:22:33.874 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:33.874 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:33.874 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:33.874 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:33.874 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:22:33.874 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:22:33.874 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 174963 2024-09-21T10:22:33.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.874 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:33.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:33.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.804150+0000", 2024-09-21T10:22:33.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:33.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.804150+0000", 2024-09-21T10:22:33.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:33.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.804152+0000", 2024-09-21T10:22:33.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.15e-06 2024-09-21T10:22:33.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:33.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.804156+0000", 2024-09-21T10:22:33.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5029999999999999e-06 2024-09-21T10:22:33.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:33.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.804167+0000", 2024-09-21T10:22:33.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1178e-05 2024-09-21T10:22:33.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:33.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.804173+0000", 2024-09-21T10:22:33.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4970000000000001e-06 2024-09-21T10:22:33.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.805060+0000", 2024-09-21T10:22:33.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000887476 2024-09-21T10:22:33.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:33.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.805144+0000", 2024-09-21T10:22:33.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.42e-05 2024-09-21T10:22:33.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:22:33.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.805212+0000", 2024-09-21T10:22:33.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7451000000000002e-05 2024-09-21T10:22:33.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:33.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.807779+0000", 2024-09-21T10:22:33.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0025666410000000001 2024-09-21T10:22:33.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.812412+0000", 2024-09-21T10:22:33.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046338960000000002 2024-09-21T10:22:33.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:47.208784+0000", 2024-09-21T10:22:33.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.39637116999999999 2024-09-21T10:22:33.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:33.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:47.208838+0000", 2024-09-21T10:22:33.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4425999999999997e-05 2024-09-21T10:22:33.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:33.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:47.208874+0000", 2024-09-21T10:22:33.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5799000000000001e-05 2024-09-21T10:22:33.879 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.879 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.879 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.880 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:174981 13.c 13:340cf87f:::benchmark_data_smithi107_278655_object47074:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-21T10:22:33.880 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:20:46.841048+0000", 2024-09-21T10:22:33.880 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 106.95310614500001, 2024-09-21T10:22:33.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.38017756200000002, 2024-09-21T10:22:33.880 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:33.880 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:33.880 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:33.880 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:33.880 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:22:33.880 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:22:33.880 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 174981 2024-09-21T10:22:33.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.880 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:33.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:33.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.841048+0000", 2024-09-21T10:22:33.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:33.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.841048+0000", 2024-09-21T10:22:33.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:33.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.841050+0000", 2024-09-21T10:22:33.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9039999999999999e-06 2024-09-21T10:22:33.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:33.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.841057+0000", 2024-09-21T10:22:33.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0550000000000002e-06 2024-09-21T10:22:33.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:33.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.841072+0000", 2024-09-21T10:22:33.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4593000000000001e-05 2024-09-21T10:22:33.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:33.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.841079+0000", 2024-09-21T10:22:33.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3429999999999998e-06 2024-09-21T10:22:33.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.841140+0000", 2024-09-21T10:22:33.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1159000000000002e-05 2024-09-21T10:22:33.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:33.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.841219+0000", 2024-09-21T10:22:33.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9128000000000005e-05 2024-09-21T10:22:33.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:22:33.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.841285+0000", 2024-09-21T10:22:33.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5835999999999998e-05 2024-09-21T10:22:33.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:33.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.843980+0000", 2024-09-21T10:22:33.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0026946349999999999 2024-09-21T10:22:33.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:46.844899+0000", 2024-09-21T10:22:33.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00091903600000000003 2024-09-21T10:22:33.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:47.221094+0000", 2024-09-21T10:22:33.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.37619532999999999 2024-09-21T10:22:33.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:33.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:47.221183+0000", 2024-09-21T10:22:33.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8240999999999998e-05 2024-09-21T10:22:33.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.885 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:33.885 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:20:47.221226+0000", 2024-09-21T10:22:33.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3302e-05 2024-09-21T10:22:33.885 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.886 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.886 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.886 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:216922 13.c 13:32e25ae1:::benchmark_data_smithi107_278655_object89015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T10:22:33.886 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:21:33.984967+0000", 2024-09-21T10:22:33.886 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 59.809187170000001, 2024-09-21T10:22:33.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.28621976199999999, 2024-09-21T10:22:33.886 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:33.886 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:33.886 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:33.886 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:33.886 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:22:33.887 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:22:33.887 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 216922 2024-09-21T10:22:33.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.887 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:33.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:33.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:21:33.984967+0000", 2024-09-21T10:22:33.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:33.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:21:33.984967+0000", 2024-09-21T10:22:33.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:33.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:21:33.984969+0000", 2024-09-21T10:22:33.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.513e-06 2024-09-21T10:22:33.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:33.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:21:33.984974+0000", 2024-09-21T10:22:33.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7709999999999997e-06 2024-09-21T10:22:33.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:33.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:21:33.984986+0000", 2024-09-21T10:22:33.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2544999999999999e-05 2024-09-21T10:22:33.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:33.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:21:33.984994+0000", 2024-09-21T10:22:33.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8630000000000008e-06 2024-09-21T10:22:33.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:21:33.985048+0000", 2024-09-21T10:22:33.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4515999999999998e-05 2024-09-21T10:22:33.889 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.889 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.889 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:33.889 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:21:33.985155+0000", 2024-09-21T10:22:33.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000106634 2024-09-21T10:22:33.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:22:33.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:21:33.985254+0000", 2024-09-21T10:22:33.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8517999999999994e-05 2024-09-21T10:22:33.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:33.890 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:21:33.989174+0000", 2024-09-21T10:22:33.890 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0039208009999999998 2024-09-21T10:22:33.890 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.890 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.890 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:21:33.990834+0000", 2024-09-21T10:22:33.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0016596899999999999 2024-09-21T10:22:33.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:21:34.271086+0000", 2024-09-21T10:22:33.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.28025141599999998 2024-09-21T10:22:33.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:33.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:21:34.271140+0000", 2024-09-21T10:22:33.891 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4001e-05 2024-09-21T10:22:33.891 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.891 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.891 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:33.891 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:21:34.271187+0000", 2024-09-21T10:22:33.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7494000000000003e-05 2024-09-21T10:22:33.892 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.892 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.892 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.892 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.892 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.892 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:249512 13.d 13:b3571941:::benchmark_data_smithi107_278655_object121605:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e399)", 2024-09-21T10:22:33.892 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:22:07.102165+0000", 2024-09-21T10:22:33.892 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 26.691989219, 2024-09-21T10:22:33.892 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0471749370000001, 2024-09-21T10:22:33.892 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:33.892 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:33.892 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:22:33.893 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:33.893 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:22:33.893 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:22:33.893 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 249512 2024-09-21T10:22:33.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.893 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:33.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:33.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.102165+0000", 2024-09-21T10:22:33.893 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.893 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.893 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.893 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:33.893 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.102165+0000", 2024-09-21T10:22:33.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:33.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.102167+0000", 2024-09-21T10:22:33.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.626e-06 2024-09-21T10:22:33.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:33.894 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.102170+0000", 2024-09-21T10:22:33.894 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3749999999999999e-06 2024-09-21T10:22:33.894 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.894 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.894 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:33.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.102180+0000", 2024-09-21T10:22:33.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5540000000000003e-06 2024-09-21T10:22:33.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:33.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.102188+0000", 2024-09-21T10:22:33.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4400000000000005e-06 2024-09-21T10:22:33.895 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.895 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.895 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.895 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.103248+0000", 2024-09-21T10:22:33.895 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001059379 2024-09-21T10:22:33.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:33.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.103258+0000", 2024-09-21T10:22:33.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9210000000000004e-06 2024-09-21T10:22:33.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.147247+0000", 2024-09-21T10:22:33.896 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0439890890000001 2024-09-21T10:22:33.896 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.896 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.896 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:33.896 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.149340+0000", 2024-09-21T10:22:33.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020935530000000002 2024-09-21T10:22:33.897 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.897 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.897 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.897 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.897 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.897 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:249516 13.d 13:b7eb6053:::benchmark_data_smithi107_278655_object121609:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e399)", 2024-09-21T10:22:33.897 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:22:07.102198+0000", 2024-09-21T10:22:33.897 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 26.69195676, 2024-09-21T10:22:33.897 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.062395172, 2024-09-21T10:22:33.897 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:33.897 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:33.897 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:33.898 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:33.898 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:22:33.898 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:22:33.898 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 249516 2024-09-21T10:22:33.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.898 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:33.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:33.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.102198+0000", 2024-09-21T10:22:33.898 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.898 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.898 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.898 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:33.898 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.102198+0000", 2024-09-21T10:22:33.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:33.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.102198+0000", 2024-09-21T10:22:33.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9999999999999998e-07 2024-09-21T10:22:33.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.899 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:33.899 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.102200+0000", 2024-09-21T10:22:33.899 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1550000000000001e-06 2024-09-21T10:22:33.899 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.899 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:33.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.102207+0000", 2024-09-21T10:22:33.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.438e-06 2024-09-21T10:22:33.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:33.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.102212+0000", 2024-09-21T10:22:33.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9640000000000002e-06 2024-09-21T10:22:33.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.900 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.900 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.103278+0000", 2024-09-21T10:22:33.900 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010663700000000001 2024-09-21T10:22:33.900 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.900 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:33.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.103284+0000", 2024-09-21T10:22:33.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0990000000000004e-06 2024-09-21T10:22:33.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.150249+0000", 2024-09-21T10:22:33.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.046965071 2024-09-21T10:22:33.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.901 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.901 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:33.901 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.150758+0000", 2024-09-21T10:22:33.901 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000509089 2024-09-21T10:22:33.901 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:22:33.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.150918+0000", 2024-09-21T10:22:33.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015939099999999999 2024-09-21T10:22:33.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.159372+0000", 2024-09-21T10:22:33.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0084543770000000008 2024-09-21T10:22:33.902 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.902 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.902 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.902 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.159410+0000", 2024-09-21T10:22:33.902 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7817000000000001e-05 2024-09-21T10:22:33.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:33.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.164535+0000", 2024-09-21T10:22:33.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005125541 2024-09-21T10:22:33.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:33.903 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.164569+0000", 2024-09-21T10:22:33.903 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3747000000000001e-05 2024-09-21T10:22:33.903 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.903 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.903 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:33.904 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.164593+0000", 2024-09-21T10:22:33.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3612999999999999e-05 2024-09-21T10:22:33.904 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.904 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.904 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.904 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.904 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.904 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:249551 13.d 13:bb072929:::benchmark_data_smithi107_278655_object121644:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-09-21T10:22:33.904 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:22:07.142773+0000", 2024-09-21T10:22:33.904 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 26.651381291, 2024-09-21T10:22:33.904 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0300519050000001, 2024-09-21T10:22:33.904 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:33.904 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:33.904 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:33.905 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:33.905 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:22:33.905 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:22:33.905 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 249551 2024-09-21T10:22:33.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.905 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:33.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:33.905 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142773+0000", 2024-09-21T10:22:33.905 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.905 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.905 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.905 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:33.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142773+0000", 2024-09-21T10:22:33.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:33.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142775+0000", 2024-09-21T10:22:33.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.367e-06 2024-09-21T10:22:33.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.906 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:33.906 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142777+0000", 2024-09-21T10:22:33.906 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8150000000000002e-06 2024-09-21T10:22:33.906 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.906 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:33.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142786+0000", 2024-09-21T10:22:33.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6889999999999993e-06 2024-09-21T10:22:33.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:33.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142792+0000", 2024-09-21T10:22:33.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.682e-06 2024-09-21T10:22:33.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.907 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.907 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.907 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142881+0000", 2024-09-21T10:22:33.907 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9287000000000001e-05 2024-09-21T10:22:33.907 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:33.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142891+0000", 2024-09-21T10:22:33.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8940000000000005e-06 2024-09-21T10:22:33.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.156063+0000", 2024-09-21T10:22:33.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.013171636 2024-09-21T10:22:33.908 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.908 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.908 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:33.908 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.156254+0000", 2024-09-21T10:22:33.908 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000190998 2024-09-21T10:22:33.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.909 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:22:33.909 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.156377+0000", 2024-09-21T10:22:33.909 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012365 2024-09-21T10:22:33.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.909 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.909 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.159441+0000", 2024-09-21T10:22:33.909 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0030635380000000002 2024-09-21T10:22:33.909 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.909 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.909 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:33.910 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.164602+0000", 2024-09-21T10:22:33.910 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0051609009999999999 2024-09-21T10:22:33.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.910 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.910 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.910 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.172704+0000", 2024-09-21T10:22:33.910 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0081026050000000006 2024-09-21T10:22:33.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.910 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.910 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:33.910 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.172763+0000", 2024-09-21T10:22:33.910 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8739000000000001e-05 2024-09-21T10:22:33.910 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.911 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:33.911 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.172825+0000", 2024-09-21T10:22:33.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2104e-05 2024-09-21T10:22:33.911 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.911 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.911 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.911 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.911 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.911 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:249573 13.d 13:bdd38b81:::benchmark_data_smithi107_278655_object121666:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-09-21T10:22:33.911 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:22:07.171894+0000", 2024-09-21T10:22:33.911 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 26.622260454999999, 2024-09-21T10:22:33.911 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.001044625, 2024-09-21T10:22:33.911 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:33.911 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:33.912 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:33.912 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:33.912 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:22:33.912 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:22:33.912 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 249573 2024-09-21T10:22:33.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.912 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:33.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.912 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:33.912 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.171894+0000", 2024-09-21T10:22:33.912 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.912 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.912 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.912 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:33.913 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.171894+0000", 2024-09-21T10:22:33.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.913 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:33.913 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.171896+0000", 2024-09-21T10:22:33.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.748e-06 2024-09-21T10:22:33.913 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.913 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:33.913 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.171901+0000", 2024-09-21T10:22:33.913 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1399999999999999e-06 2024-09-21T10:22:33.913 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.913 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:33.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.171912+0000", 2024-09-21T10:22:33.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1552000000000001e-05 2024-09-21T10:22:33.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:33.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.171918+0000", 2024-09-21T10:22:33.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6799999999999998e-06 2024-09-21T10:22:33.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.914 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.914 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.171978+0000", 2024-09-21T10:22:33.914 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9802000000000001e-05 2024-09-21T10:22:33.914 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.914 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:33.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.171985+0000", 2024-09-21T10:22:33.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8240000000000002e-06 2024-09-21T10:22:33.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.159467+0000", 2024-09-21T10:22:33.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.98748258899999997 2024-09-21T10:22:33.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.915 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:33.915 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.159584+0000", 2024-09-21T10:22:33.915 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011648 2024-09-21T10:22:33.915 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.915 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.916 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:22:33.916 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.159699+0000", 2024-09-21T10:22:33.916 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000114889 2024-09-21T10:22:33.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.916 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.916 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.164489+0000", 2024-09-21T10:22:33.916 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0047898630000000001 2024-09-21T10:22:33.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.916 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.916 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:33.916 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.167612+0000", 2024-09-21T10:22:33.916 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0031233160000000001 2024-09-21T10:22:33.916 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.172873+0000", 2024-09-21T10:22:33.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052615370000000002 2024-09-21T10:22:33.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:33.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.172905+0000", 2024-09-21T10:22:33.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1124000000000002e-05 2024-09-21T10:22:33.917 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.917 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.917 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:33.917 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.172939+0000", 2024-09-21T10:22:33.917 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4081000000000001e-05 2024-09-21T10:22:33.917 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.918 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.918 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.918 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.918 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.918 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:249592 13.d 13:b120de45:::benchmark_data_smithi107_278655_object121685:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-09-21T10:22:33.918 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:22:07.203364+0000", 2024-09-21T10:22:33.918 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 26.590790827999999, 2024-09-21T10:22:33.918 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.96971972799999995, 2024-09-21T10:22:33.918 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:22:33.918 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:22:33.918 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:22:33.918 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:22:33.918 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:22:33.919 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:22:33.919 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 249592 2024-09-21T10:22:33.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.919 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:22:33.919 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.919 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:22:33.919 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.203364+0000", 2024-09-21T10:22:33.919 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.919 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.919 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:22:33.919 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.203364+0000", 2024-09-21T10:22:33.919 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:22:33.919 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.920 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.920 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:22:33.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.203365+0000", 2024-09-21T10:22:33.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2559999999999999e-06 2024-09-21T10:22:33.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.920 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.920 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:22:33.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.203369+0000", 2024-09-21T10:22:33.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7960000000000002e-06 2024-09-21T10:22:33.920 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.920 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.920 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:22:33.920 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.203379+0000", 2024-09-21T10:22:33.920 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9399999999999997e-06 2024-09-21T10:22:33.921 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.921 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:22:33.921 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.203385+0000", 2024-09-21T10:22:33.921 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.832e-06 2024-09-21T10:22:33.921 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.921 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.921 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.203448+0000", 2024-09-21T10:22:33.921 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2508000000000005e-05 2024-09-21T10:22:33.921 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.921 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.921 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:22:33.921 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.203457+0000", 2024-09-21T10:22:33.922 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9879999999999993e-06 2024-09-21T10:22:33.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.922 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:22:33.922 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.162221+0000", 2024-09-21T10:22:33.922 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.95876363899999995 2024-09-21T10:22:33.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.922 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:22:33.922 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.162324+0000", 2024-09-21T10:22:33.922 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103232 2024-09-21T10:22:33.922 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.922 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:22:33.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.162385+0000", 2024-09-21T10:22:33.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1255999999999998e-05 2024-09-21T10:22:33.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:22:33.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.167617+0000", 2024-09-21T10:22:33.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0052322369999999998 2024-09-21T10:22:33.923 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.923 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.923 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.923 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.167652+0000", 2024-09-21T10:22:33.923 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4573000000000003e-05 2024-09-21T10:22:33.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.924 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:22:33.924 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.172982+0000", 2024-09-21T10:22:33.924 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0053300120000000003 2024-09-21T10:22:33.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.924 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:22:33.924 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.173050+0000", 2024-09-21T10:22:33.924 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7716e-05 2024-09-21T10:22:33.924 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:22:33.924 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:22:33.925 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:22:33.925 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.173083+0000", 2024-09-21T10:22:33.925 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3742999999999999e-05 2024-09-21T10:22:33.925 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.925 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.925 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.925 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:22:33.925 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:22:33.925 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:22:33.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:33.927 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:33.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:34.027 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:34.053 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:34.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:34.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:34.155 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:34.177 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:34.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:34.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:34.279 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:34.300 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:34.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:34.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:34.402 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:34.428 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:34.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:34.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:34.498 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:22:34.498 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:22:34.499 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_9 2024-09-21T10:22:34.499 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_9 16 2024-09-21T10:22:34.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:34.565 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:34.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:34.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:34.667 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:34.693 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:34.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:34.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:34.736 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:34 smithi107 ceph-mon[102221]: pgmap v1420: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:34.736 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:34 smithi107 ceph-mon[105022]: pgmap v1420: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:34.795 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:34.820 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:34.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:34.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:34 smithi129 ceph-mon[88471]: pgmap v1420: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:34.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:34.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:34.945 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:34.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:35.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:35.067 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:35.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:35.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:35.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:22:35.169 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:35.195 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:35.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:35.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:35.297 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:35.322 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:35.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:35.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:35.424 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:35.449 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:35.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:35.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:35.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:35.573 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:35.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:35.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:35.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:22:35.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:35.700 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:35.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:35.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:35.801 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:35.827 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:35.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:35.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:35.925 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:35.929 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:35.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:35.955 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:35.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:35.961 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:35.977 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:36.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:36.077 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:36.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:36.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:36.178 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:36.199 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:36.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:36.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:36.301 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:36.327 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:36.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:36.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:36.429 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:36.453 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:36.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:36.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:36.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:36.578 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T10:22:36.578 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:36.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:36.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:36.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:36.702 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:36.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:36.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:36.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:36.825 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:36.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:36.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:36.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:36 smithi107 ceph-mon[105022]: pgmap v1421: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:36.845 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:36 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4015862355' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:22:36.846 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:36 smithi107 ceph-mon[102221]: pgmap v1421: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:36.846 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:36 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4015862355' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:22:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:36 smithi129 ceph-mon[88471]: pgmap v1421: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:36 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4015862355' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:22:36.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:36.949 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:36.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:36.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:37.050 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:37.076 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:37.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:37.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:37.178 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:37.204 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:37.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:37.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:37.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:37.327 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:37.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:37.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:37.428 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:37.454 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:37.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:37.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:37.556 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:37.581 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:37.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:37.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:37.683 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:37.707 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:37.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:37.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:37.809 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:37.833 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:37.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:37.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:37.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:37 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4015862355' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:22:37.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:37 smithi107 ceph-mon[102221]: osdmap e410: 8 total, 8 up, 8 in 2024-09-21T10:22:37.845 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:37 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:37.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:37 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4015862355' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:22:37.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:37 smithi107 ceph-mon[105022]: osdmap e410: 8 total, 8 up, 8 in 2024-09-21T10:22:37.846 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:37 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:37 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4015862355' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:22:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:37 smithi129 ceph-mon[88471]: osdmap e410: 8 total, 8 up, 8 in 2024-09-21T10:22:37.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:37 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:37.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:37.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:37.957 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:37.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:38.057 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:38.082 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:38.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:38.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:38.168 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:38.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:38.205 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:38.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:38.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:38.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:38.328 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:38.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:38.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:38.430 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:38.456 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:38.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:38.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:38.557 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:38.583 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:38.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:38.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:38.684 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:38.709 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:38.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:38.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:38.811 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:38.836 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:38.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:38.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:38.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:38 smithi107 ceph-mon[102221]: pgmap v1423: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:38.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:38 smithi107 ceph-mon[105022]: pgmap v1423: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:38.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:38.959 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:38.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:38.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:39.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:39.081 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:39.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:39.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:39.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:38 smithi129 ceph-mon[88471]: pgmap v1423: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:39.183 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:39.204 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:39.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:39.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:39.306 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:39.327 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:39.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:39.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:39.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:39.450 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:39.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:39.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:39.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:39.573 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:39.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:39.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:39.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T10:22:39.674 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:39.698 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:39.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:39.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:39.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:39.821 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:39.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:39.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:39.923 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:39.947 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:39.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:39.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:40.049 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:40.073 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:40.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:40.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:40.100 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:22:40.100 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:22:40.101 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:22:40.101 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:22:40.107 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:22:40.107 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:22:40.124 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:22:40.124 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:22:40.139 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:22:40.140 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:22:40.156 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:22:40.156 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:22:40.172 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:22:40.172 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:22:40.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:40.187 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:22:40.188 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:22:40.198 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:40.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:40.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:40.300 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:40.324 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:40.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:40.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:40.425 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:40.449 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:40.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:40.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:40.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:40.574 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:40.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:40.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:40.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:22:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:22:40.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:22:40.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:40.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:40.698 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:40.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:40.799 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:40.821 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:40.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:40.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:40 smithi107 ceph-mon[102221]: pgmap v1424: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:40 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2211876788' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:40 smithi107 ceph-mon[105022]: pgmap v1424: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:40 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2211876788' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:40.923 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:40.942 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:40 smithi129 ceph-mon[88471]: pgmap v1424: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:40.942 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:40 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2211876788' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:40.944 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:40.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:40.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:41.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:41.067 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:41.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:41.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:41.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:41.191 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:41.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:41.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:41.293 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:41.314 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:41.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:41.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:41.416 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:41.437 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:41.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:41.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:41.539 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:41.560 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:41.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:41.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:41.661 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:41.682 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:41.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:41.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:41.784 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:41.808 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:41.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:41.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:41.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:41.930 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:41.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:41.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:42.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:42.053 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:42.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:42.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:42.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:42.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:42.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:22:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:22:41] "GET /metrics HTTP/1.1" 200 34358 "" "Prometheus/2.51.0" 2024-09-21T10:22:42.155 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:42.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:42.176 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:42.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:42.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:42.278 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:42.299 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:42.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:42.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:42.400 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:42.424 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:42.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:42.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:42.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:42.549 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:42.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:42.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:42.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:42.673 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:42.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:42.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:42.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:42.796 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:42.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:42.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:42.919 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:42.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:42.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:43.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:22:43.020 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:43.046 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:43.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:43.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:43.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:42 smithi107 ceph-mon[102221]: pgmap v1425: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:43.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:42 smithi107 ceph-mon[105022]: pgmap v1425: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:43.147 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:43.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:42 smithi129 ceph-mon[88471]: pgmap v1425: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:43.173 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:43.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:43.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:43.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:43.299 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:43.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:43.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:43.401 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:43.426 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:43.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:43.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:43.528 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:43.553 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:43.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:43.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:43.655 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:43.680 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:43.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:43.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:43.782 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:43.806 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:43.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:43.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:43.818 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_9' created 2024-09-21T10:22:43.908 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:43.932 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:43.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:43.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:44.033 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:44.056 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:44.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:44.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:44.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:43 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3040741347' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16} : dispatch 2024-09-21T10:22:44.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:43 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16} : dispatch 2024-09-21T10:22:44.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:43 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3040741347' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16} : dispatch 2024-09-21T10:22:44.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:43 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16} : dispatch 2024-09-21T10:22:44.158 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:44.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:43 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3040741347' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16} : dispatch 2024-09-21T10:22:44.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:43 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16} : dispatch 2024-09-21T10:22:44.182 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:44.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:44.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:44.266 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:44.284 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:44.308 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:44.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:44.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:44.410 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:44.432 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:44.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:44.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:44.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:44.555 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:44.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:44.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:44.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:44.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_9 rados --yes-i-really-mean-it || true 2024-09-21T10:22:44.678 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:44.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:44.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:44.801 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:44.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:44.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:44.903 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:44.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:44 smithi107 ceph-mon[102221]: pgmap v1426: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:44.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:44 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16}]': finished 2024-09-21T10:22:44.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:44 smithi107 ceph-mon[102221]: osdmap e411: 8 total, 8 up, 8 in 2024-09-21T10:22:44.919 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:44 smithi107 ceph-mon[105022]: pgmap v1426: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:44.919 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:44 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16}]': finished 2024-09-21T10:22:44.920 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:44 smithi107 ceph-mon[105022]: osdmap e411: 8 total, 8 up, 8 in 2024-09-21T10:22:44.926 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:44.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:44.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:45.028 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:45.053 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:45.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:45.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:45.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:45.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:44 smithi129 ceph-mon[88471]: pgmap v1426: 93 pgs: 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:45.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:44 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_9", "pg_num": 16}]': finished 2024-09-21T10:22:45.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:44 smithi129 ceph-mon[88471]: osdmap e411: 8 total, 8 up, 8 in 2024-09-21T10:22:45.176 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:45.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:45.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:45.278 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:45.300 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:45.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:45.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:45.402 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:45.428 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:45.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:45.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:45.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:45.551 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:45.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:45.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:45.572 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:22:45.572 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:22:45.652 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:45.678 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:45.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:45.780 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:45.805 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:45.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:45.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:45.823 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:22:45.907 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:45.932 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:45.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:45.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:46.033 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:46.056 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:46.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:46.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:46.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:45 smithi107 ceph-mon[102221]: osdmap e412: 8 total, 8 up, 8 in 2024-09-21T10:22:46.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:45 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3354389365' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:22:46.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:45 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:22:46.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:45 smithi107 ceph-mon[105022]: osdmap e412: 8 total, 8 up, 8 in 2024-09-21T10:22:46.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:45 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3354389365' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:22:46.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:45 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:22:46.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:22:46.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:46.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:45 smithi129 ceph-mon[88471]: osdmap e412: 8 total, 8 up, 8 in 2024-09-21T10:22:46.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:45 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3354389365' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:22:46.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:45 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:22:46.212 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:46.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:46.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:46.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:46.334 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:46.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:46.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:46.436 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:46.440 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:46.458 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:46.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:46.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:46.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:46.582 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:46.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:46.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:46.683 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:46.709 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:46.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:46.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:46.811 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:46.837 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:46.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:46.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:46.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:46.960 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:46.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:46.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:47.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:47.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:47.084 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:47.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:47.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:46 smithi107 ceph-mon[102221]: pgmap v1429: 109 pgs: 16 unknown, 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:47.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:46 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:22:47.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:46 smithi107 ceph-mon[102221]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:22:47.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:46 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:22:47.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:46 smithi107 ceph-mon[102221]: osdmap e413: 8 total, 8 up, 8 in 2024-09-21T10:22:47.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:46 smithi107 ceph-mon[105022]: pgmap v1429: 109 pgs: 16 unknown, 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:47.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:46 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:22:47.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:46 smithi107 ceph-mon[105022]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:22:47.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:46 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:22:47.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:46 smithi107 ceph-mon[105022]: osdmap e413: 8 total, 8 up, 8 in 2024-09-21T10:22:47.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:46 smithi129 ceph-mon[88471]: pgmap v1429: 109 pgs: 16 unknown, 93 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:47.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:46 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:22:47.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:46 smithi129 ceph-mon[88471]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:22:47.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:46 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:22:47.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:46 smithi129 ceph-mon[88471]: osdmap e413: 8 total, 8 up, 8 in 2024-09-21T10:22:47.186 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:47.207 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:47.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:47.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:47.309 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:47.335 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:47.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:47.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:47.339 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:47.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:47.459 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:47.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:47.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:47.561 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:47.586 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:47.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:47.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:47.688 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:47.712 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:47.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:47.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:47.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:47.836 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:47.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:47.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:47.938 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:47.964 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:47.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:47.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:48.065 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:48.090 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:48.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:48.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:48.194 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:48.219 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:48.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:48.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:48.321 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:48.346 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:48.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:48.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:48.380 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:48.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:48.469 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:48.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:48.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:48.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:48.592 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:48.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:48.694 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:48.715 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:48.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:48.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:48.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:48.838 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:48.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:48.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:48.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:48.961 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:48.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:48.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:49.063 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:49.089 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:49.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:49.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:49.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:48 smithi107 ceph-mon[102221]: pgmap v1431: 109 pgs: 12 unknown, 97 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:49.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:48 smithi107 ceph-mon[105022]: pgmap v1431: 109 pgs: 12 unknown, 97 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:49.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:48 smithi129 ceph-mon[88471]: pgmap v1431: 109 pgs: 12 unknown, 97 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:49.191 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:49.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:49.218 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:49.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:49.319 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:49.345 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:49.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:49.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:49.447 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:49.471 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:49.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:49.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:49.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:49.598 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:49.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:49.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:49.699 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:49.721 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:49.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:49.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:49.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:49.846 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:49.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:49.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:49.948 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:49.954 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:22:49.974 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:49.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:49.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:50.076 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:50.100 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:50.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:50.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:50.202 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:50.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:50.224 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:50.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:50.325 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:50.350 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:50.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:50.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:50.452 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:50.477 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:50.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:50.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:50.578 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:50.600 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:50.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:50.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:22:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:22:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:22:50.701 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:50.727 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:50.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:50.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:50.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:50.835 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T10:22:50.855 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:50.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:50.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:50.957 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:50.978 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:50.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:50.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:51.080 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:51.106 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:51.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:51.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:51.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:50 smithi107 ceph-mon[102221]: pgmap v1432: 109 pgs: 109 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:51.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:50 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/33755296' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:22:51.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:50 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:22:51.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:50 smithi107 ceph-mon[105022]: pgmap v1432: 109 pgs: 109 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:51.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:50 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/33755296' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:22:51.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:50 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:22:51.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:50 smithi129 ceph-mon[88471]: pgmap v1432: 109 pgs: 109 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:51.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:50 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/33755296' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:22:51.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:50 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:22:51.208 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:51.233 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:51.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:51.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:51.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:51.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:51.358 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:51.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:51.379 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:22:51.380 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:22:51.380 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:22:51.380 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:22:51.396 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:22:51.396 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:22:51.412 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:22:51.412 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:22:51.428 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:22:51.428 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:22:51.443 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:22:51.444 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:22:51.459 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:51.459 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:22:51.460 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:22:51.475 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:22:51.475 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:22:51.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:51.484 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:51.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:51.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:51.606 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:51.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:51.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:51.708 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:51.734 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:51.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:51.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:51.836 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:51.841 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:22:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:22:51] "GET /metrics HTTP/1.1" 200 34358 "" "Prometheus/2.51.0" 2024-09-21T10:22:51.859 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:51.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:51.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:51.961 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:51.983 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:51.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:51.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:52.085 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:52.107 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:52.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:52.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:52.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:51 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:22:52.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:51 smithi107 ceph-mon[102221]: osdmap e414: 8 total, 8 up, 8 in 2024-09-21T10:22:52.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:51 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2410148830' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:52.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:51 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:22:52.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:51 smithi107 ceph-mon[105022]: osdmap e414: 8 total, 8 up, 8 in 2024-09-21T10:22:52.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:51 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2410148830' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:52.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:51 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:22:52.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:51 smithi129 ceph-mon[88471]: osdmap e414: 8 total, 8 up, 8 in 2024-09-21T10:22:52.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:51 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2410148830' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:22:52.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T10:22:52.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:52.265 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:52.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:52.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:52.367 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:52.389 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:52.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:52.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:52.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:52.511 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:52.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:52.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:52.613 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:52.638 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:52.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:52.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:52.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:52.762 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:52.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:52.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:52.864 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:52.889 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:52.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:52.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:52.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:53.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:53.012 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:53.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:53.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:52 smithi107 ceph-mon[102221]: pgmap v1434: 109 pgs: 109 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:53.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:52 smithi107 ceph-mon[105022]: pgmap v1434: 109 pgs: 109 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:53.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:53.134 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:53.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:53.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:53.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:52 smithi129 ceph-mon[88471]: pgmap v1434: 109 pgs: 109 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:53.236 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:53.514 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:53.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:53.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:53.616 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:53.641 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:53.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:53.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:53.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:53.767 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:53.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:53.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:53.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:53.891 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:53.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:53.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:53.993 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:54.015 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:54.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:54.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:54.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:54.140 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:54.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:54.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:54.242 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:54.264 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:54.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:54.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:54.359 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:22:54.359 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:22:54.360 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-09-21T10:22:54.360 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-09-21T10:22:54.360 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:22:54.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:54.388 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:54.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:54.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:54.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:54.510 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:54.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:54.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:54.612 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:54.632 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:54.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:54.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:54.651 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:54.651 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:54.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:54.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:54.758 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:54.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:54.846 INFO:teuthology.orchestra.run.smithi107.stderr:enabled application 'rados' on pool 'unique_pool_9' 2024-09-21T10:22:54.859 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:54.884 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:54.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:54.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:54.918 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:54 smithi107 ceph-mon[102221]: pgmap v1435: 109 pgs: 109 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:54.919 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:54 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3048865219' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:22:54.919 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:54 smithi107 ceph-mon[105022]: pgmap v1435: 109 pgs: 109 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:54.919 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:54 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3048865219' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:22:54.985 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:55.011 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:55.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:55.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:55.113 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:55.137 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:55.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:55.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:54 smithi129 ceph-mon[88471]: pgmap v1435: 109 pgs: 109 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:54 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3048865219' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:22:55.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:55.262 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:55.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:55.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:55.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:55.386 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:55.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:55.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:55.487 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:55.509 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:55.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:55.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:55.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:22:55.611 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:55.637 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:55.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:55.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:55.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:55.761 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:55.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:55.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:55.862 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:55.888 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:55.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:55.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:55.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:56.012 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:56.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:56.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:56.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:55 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3048865219' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:22:56.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:55 smithi107 ceph-mon[102221]: osdmap e415: 8 total, 8 up, 8 in 2024-09-21T10:22:56.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:55 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3048865219' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:22:56.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:55 smithi107 ceph-mon[105022]: osdmap e415: 8 total, 8 up, 8 in 2024-09-21T10:22:56.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:56.135 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:56.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:56.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:56.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:55 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3048865219' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_9", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:22:56.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:55 smithi129 ceph-mon[88471]: osdmap e415: 8 total, 8 up, 8 in 2024-09-21T10:22:56.236 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:56.262 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:56.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:56.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:22:56.364 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:56.390 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:56.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:56.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:56.491 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:56.517 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:56.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:56.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:56.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:56.639 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:56.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:56.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:56.653 DEBUG:teuthology.orchestra.run.smithi107:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_9 bench 90 write' 2024-09-21T10:22:56.683 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:22:56.683 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:22:56.729 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:hints = 1 2024-09-21T10:22:56.729 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:22:56.729 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object prefix: benchmark_data_smithi107_312686 2024-09-21T10:22:56.729 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:22:56.729 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:22:56.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:56.783 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:56.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:56.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:56.885 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:56.912 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:56.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:56.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:57.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:57.054 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:57.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:57.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:57.129 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:56 smithi107 ceph-mon[105022]: pgmap v1437: 109 pgs: 109 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:57.129 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:56 smithi107 ceph-mon[105022]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:22:57.129 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:57.130 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:57.130 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:56 smithi107 ceph-mon[102221]: pgmap v1437: 109 pgs: 109 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:57.130 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:56 smithi107 ceph-mon[102221]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:22:57.130 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:57.131 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:57.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:56 smithi129 ceph-mon[88471]: pgmap v1437: 109 pgs: 109 active+clean; 582 KiB data, 9.0 GiB used, 706 GiB / 715 GiB avail 2024-09-21T10:22:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:56 smithi129 ceph-mon[88471]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:22:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:22:57.196 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:57.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:57.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:57.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:57.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:57.337 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:57.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:57.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:57.481 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:57.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:57.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:57.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:57.624 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:57.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:57.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:57.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:57.729 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 1 16 3426 3410 213.114 213.125 0.00389665 0.0046763 2024-09-21T10:22:57.762 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:57.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:57.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:57.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:57.901 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:57.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:57.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:58.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:58.037 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:58.043 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:58.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:58.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:22:58.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:58.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:58.184 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:58.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:58.285 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:22:58.309 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:58.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:58.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:22:58.420 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:58.471 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:58.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:58.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:58.573 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:58.603 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:58.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:58.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:58.706 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:58.730 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 2 16 6536 6520 203.729 194.375 0.00699475 0.00489729 2024-09-21T10:22:58.745 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:58.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:58.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:58.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:58.890 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:58.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:58.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:58.992 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:59.017 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:59.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:59.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:59.119 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:22:59.159 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:59.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:59.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:22:59.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:22:58 smithi129 ceph-mon[88471]: pgmap v1438: 109 pgs: 109 active+clean; 23 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.9 MiB/s wr, 45 op/s 2024-09-21T10:22:59.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:22:59.297 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:59.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:59.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:22:59.310 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:22:58 smithi107 ceph-mon[102221]: pgmap v1438: 109 pgs: 109 active+clean; 23 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.9 MiB/s wr, 45 op/s 2024-09-21T10:22:59.310 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:22:58 smithi107 ceph-mon[105022]: pgmap v1438: 109 pgs: 109 active+clean; 23 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.9 MiB/s wr, 45 op/s 2024-09-21T10:22:59.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:22:59.435 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:59.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:59.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:22:59.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:22:59.575 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:22:59.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:59.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:22:59.677 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:22:59.702 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:59.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:59.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:22:59.729 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 3 16 9440 9424 196.313 181.5 0.00622018 0.00508337 2024-09-21T10:22:59.804 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:22:59.830 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:59.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:59.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:22:59.932 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:22:59.956 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:22:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:22:59.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:00.058 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:00.086 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:00.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:00.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:00.188 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:00.228 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:00.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:00.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:00.330 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:00.358 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:00.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:00.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:00.460 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:00.501 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:00.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:00.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:00.603 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:00.652 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:00.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:00.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:23:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:23:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:23:00.729 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 4 16 11968 11952 186.729 158 0.00432348 0.00534828 2024-09-21T10:23:00.755 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:00.783 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:00.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:00.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:00.883 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T10:23:00.885 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:00.888 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:23:00.910 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:00.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:00.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:01.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:01.050 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:01.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:01.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:01.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:00 smithi107 ceph-mon[102221]: pgmap v1439: 109 pgs: 109 active+clean; 214 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 25 MiB/s wr, 393 op/s 2024-09-21T10:23:01.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:00 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3646163863' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:01.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:00 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:01.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:00 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1122359725' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:23:01.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:00 smithi107 ceph-mon[105022]: pgmap v1439: 109 pgs: 109 active+clean; 214 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 25 MiB/s wr, 393 op/s 2024-09-21T10:23:01.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:00 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3646163863' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:01.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:00 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:01.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:00 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1122359725' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:23:01.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:00 smithi129 ceph-mon[88471]: pgmap v1439: 109 pgs: 109 active+clean; 214 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 25 MiB/s wr, 393 op/s 2024-09-21T10:23:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:00 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3646163863' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:00 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:00 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1122359725' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:23:01.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:01.190 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:01.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:01.291 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:01.318 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:01.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:01.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:01.420 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:01.456 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:01.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:01.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:01.558 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:01.580 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:01.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:01.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:01.682 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:01.710 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:01.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:01.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:01.730 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 5 16 14467 14451 180.618 156.188 0.00470117 0.0055311 2024-09-21T10:23:01.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:01.849 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:01.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:01.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:01.951 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:01.980 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:01.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:01.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:02.082 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:02.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:02.104 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:02.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:02.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:23:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:23:01] "GET /metrics HTTP/1.1" 200 36705 "" "Prometheus/2.51.0" 2024-09-21T10:23:02.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:01 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:23:02.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:01 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:23:02.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:01 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:23:02.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:01 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1122359725' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:23:02.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:01 smithi107 ceph-mon[102221]: osdmap e416: 8 total, 8 up, 8 in 2024-09-21T10:23:02.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:01 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:23:02.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:01 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:23:02.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:01 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:23:02.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:01 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1122359725' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:23:02.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:01 smithi107 ceph-mon[105022]: osdmap e416: 8 total, 8 up, 8 in 2024-09-21T10:23:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:01 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:23:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:01 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:23:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:01 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:23:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:01 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1122359725' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:23:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:01 smithi129 ceph-mon[88471]: osdmap e416: 8 total, 8 up, 8 in 2024-09-21T10:23:02.204 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:02.231 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:02.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:02.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:02.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:02.369 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:02.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:02.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:02.470 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:02.498 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:02.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:02.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:02.600 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:02.621 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:02.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:02.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:02.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:02.730 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 6 16 16810 16794 174.92 146.438 0.00970129 0.00571072 2024-09-21T10:23:02.758 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:02.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:02.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:02.860 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:02.887 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:02.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:02.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:02.989 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:03.018 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:03.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:03.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:03.120 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:03.145 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:03.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:03.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:03.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:02 smithi129 ceph-mon[88471]: pgmap v1441: 109 pgs: 109 active+clean; 214 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 27 MiB/s wr, 427 op/s 2024-09-21T10:23:03.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:03.287 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:03.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:03.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:03.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:02 smithi107 ceph-mon[105022]: pgmap v1441: 109 pgs: 109 active+clean; 214 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 27 MiB/s wr, 427 op/s 2024-09-21T10:23:03.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:02 smithi107 ceph-mon[102221]: pgmap v1441: 109 pgs: 109 active+clean; 214 MiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 27 MiB/s wr, 427 op/s 2024-09-21T10:23:03.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:03.409 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:03.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:03.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:03.511 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:03.533 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:03.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:03.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:03.635 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:03.665 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:03.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:03.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:03.730 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 7 16 18646 18630 166.323 114.75 0.0046669 0.00600628 2024-09-21T10:23:03.767 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:03.803 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:03.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:03.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:03.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:03.926 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:03.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:03.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:04.027 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:04.054 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:04.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:04.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:04.155 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:04.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:03 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:23:04.176 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:04.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:04.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:04.277 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:04.311 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:04.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:04.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:04.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:03 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:23:04.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:03 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:23:04.413 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:04.439 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:04.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:04.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:04.540 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:04.579 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:04.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:04.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:04.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:04.718 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:04.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:04.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:04.730 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 8 16 20588 20572 160.703 121.375 0.0149139 0.00621557 2024-09-21T10:23:04.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:04.858 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:04.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:04.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:04.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:04.996 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:04.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:04.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:05.098 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:05.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:04 smithi107 ceph-mon[105022]: pgmap v1442: 109 pgs: 109 active+clean; 907 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-21T10:23:05.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:04 smithi107 ceph-mon[102221]: pgmap v1442: 109 pgs: 109 active+clean; 907 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-21T10:23:05.133 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:05.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:05.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:05.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:04 smithi129 ceph-mon[88471]: pgmap v1442: 109 pgs: 109 active+clean; 907 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 104 MiB/s wr, 1.67k op/s 2024-09-21T10:23:05.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:05.271 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:05.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:05.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:05.373 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:05.394 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:05.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:05.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:05.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:05.531 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:05.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:05.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:05.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:05.654 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:05.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:05.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:05.730 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 9 16 22628 22612 157.013 127.5 0.0040738 0.00636357 2024-09-21T10:23:05.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:05.792 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:05.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:05.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:05.894 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:05.917 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:05.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:05.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:06.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:06.056 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:06.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:06.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:06.158 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:06.188 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:06.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:06.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:06.290 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:06.321 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:06.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:06.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:06.424 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:06.450 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:06.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:06.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:06.552 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:06.582 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:06.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:06.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:06.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:06.719 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:06.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:06.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:06.730 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 10 16 24743 24727 154.529 132.188 0.00632903 0.00646612 2024-09-21T10:23:06.822 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:06.860 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:06.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:06.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:06.961 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:06.991 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:06.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:06.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:07.094 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:07.120 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:07.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:07.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:07.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:06 smithi129 ceph-mon[88471]: pgmap v1443: 109 pgs: 109 active+clean; 1.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-21T10:23:07.213 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:06 smithi107 ceph-mon[102221]: pgmap v1443: 109 pgs: 109 active+clean; 1.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-21T10:23:07.213 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:06 smithi107 ceph-mon[105022]: pgmap v1443: 109 pgs: 109 active+clean; 1.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-21T10:23:07.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:07.247 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:07.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:07.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:07.349 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:07.375 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:07.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:07.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:07.477 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:07.502 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:07.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:07.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:07.604 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:07.634 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:07.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:07.730 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 11 16 26706 26690 151.634 122.688 0.00494959 0.00658973 2024-09-21T10:23:07.736 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:07.762 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:07.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:07.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:07.864 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:07.891 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:07.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:07.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:07.992 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:08.018 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:08.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:08.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:08.120 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:08.148 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:08.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:08.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:08.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:08.270 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:08.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:08.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:08.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:08.408 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:08.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:08.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:08.510 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:08.548 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:08.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:08.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:08.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:08.681 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:08.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:08.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:08.730 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 12 16 28357 28341 147.595 103.188 0.00167225 0.00673505 2024-09-21T10:23:08.783 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:08.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:08.820 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:08.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:08.921 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:08.949 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:08.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:08.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:09.051 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:09.077 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:09.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:09.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:09.119 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:08 smithi129 ceph-mon[88471]: pgmap v1444: 109 pgs: 109 active+clean; 1.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-21T10:23:09.178 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:09.206 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:09.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:09.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:09.307 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:09.335 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:09.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:09.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:09.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:08 smithi107 ceph-mon[102221]: pgmap v1444: 109 pgs: 109 active+clean; 1.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-21T10:23:09.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:08 smithi107 ceph-mon[105022]: pgmap v1444: 109 pgs: 109 active+clean; 1.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-21T10:23:09.436 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:09.462 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:09.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:09.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:09.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:09.606 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:09.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:09.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:09.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:09.730 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 13 16 30096 30080 144.601 108.688 0.00521687 0.00690974 2024-09-21T10:23:09.742 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:09.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:09.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:09.844 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:09.874 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:09.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:09.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:09.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:10.012 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:10.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:10.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:10.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:10.149 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:10.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:10.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:10.251 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:10.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:10.280 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:10.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:10.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:10.418 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:10.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:10.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:10.520 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:10.532 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:10 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:10.532 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:10 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:10.532 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:10 smithi129 ceph-mon[88471]: pgmap v1445: 109 pgs: 109 active+clean; 1.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 145 MiB/s wr, 2.33k op/s 2024-09-21T10:23:10.532 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:23:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:23:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:23:10.549 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:10.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:10.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:10.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:10.685 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:10.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:10.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:10.730 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 14 16 31920 31904 142.415 114 0.00630354 0.00701621 2024-09-21T10:23:10.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:10.827 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:10.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:10.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:10.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:10 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:10.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:10 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:10.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:10 smithi107 ceph-mon[102221]: pgmap v1445: 109 pgs: 109 active+clean; 1.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 145 MiB/s wr, 2.33k op/s 2024-09-21T10:23:10.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:10 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:10.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:10 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:10.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:10 smithi107 ceph-mon[105022]: pgmap v1445: 109 pgs: 109 active+clean; 1.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 145 MiB/s wr, 2.33k op/s 2024-09-21T10:23:10.929 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:10.957 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:10.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:10.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:11.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:11.089 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:11.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:11.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:11.191 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:11.225 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:11.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:11.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:11.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:11.349 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:11.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:11.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:11.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:11.486 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:11.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:11.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:11.587 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:11.615 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:11.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:11.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:11.717 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:11.731 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 15 16 33533 33517 139.641 100.812 0.00904586 0.00715463 2024-09-21T10:23:11.746 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:11.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:11.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:11.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:11.868 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:11.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:11.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:11.970 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:11.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:11.999 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:11.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:12.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:12.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:23:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:23:11] "GET /metrics HTTP/1.1" 200 36770 "" "Prometheus/2.51.0" 2024-09-21T10:23:12.138 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:12.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:12.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:12.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:12.277 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:12.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:12.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:12.379 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:12.404 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:12.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:12.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:12.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:12 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:12.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:12.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:12 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:12.453 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:12.453 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:12 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:12.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:12.547 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:12.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:12.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:12.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:12.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:12.688 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:12.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:12.731 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 16 16 35025 35009 136.741 93.25 0.0112746 0.00730677 2024-09-21T10:23:12.789 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:12.815 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:12.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:12.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:12.858 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:12.875 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:23:12.876 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:23:12.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T10:23:12.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:12.918 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:23:12.919 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:23:12.956 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:23:12.956 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:23:12.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:12.977 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:12.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:13.001 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:23:13.001 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:23:13.030 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:23:13.031 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:23:13.062 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:23:13.063 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:23:13.078 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:13.097 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:23:13.097 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:23:13.103 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:13.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:13.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:13.130 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:23:13.130 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:23:13.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:13.257 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:13.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:13.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:13.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:13.398 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:13.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:13.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:13.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:13.508 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:13 smithi107 ceph-mon[105022]: pgmap v1446: 109 pgs: 109 active+clean; 1.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-21T10:23:13.508 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:13 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/622756759' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:13.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:13 smithi107 ceph-mon[102221]: pgmap v1446: 109 pgs: 109 active+clean; 1.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-21T10:23:13.509 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:13 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/622756759' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:13.547 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:13.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:13.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:13.649 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:13 smithi129 ceph-mon[88471]: pgmap v1446: 109 pgs: 109 active+clean; 1.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-21T10:23:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:13 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/622756759' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:13.677 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:13.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:13.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:13.731 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 17 16 36648 36632 134.664 101.438 0.00387878 0.00742054 2024-09-21T10:23:13.778 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:13.805 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:13.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:13.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:13.907 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:13.935 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:13.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:13.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:14.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:14.077 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:14.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:14.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:14.178 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:14.205 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:14.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:14.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:14.306 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:14.334 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:14.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:14.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:14.435 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:14.461 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:14.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:14.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:14.534 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:14 smithi107 ceph-mon[105022]: pgmap v1447: 109 pgs: 109 active+clean; 2.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 156 MiB/s wr, 2.50k op/s 2024-09-21T10:23:14.536 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:14 smithi107 ceph-mon[102221]: pgmap v1447: 109 pgs: 109 active+clean; 2.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 156 MiB/s wr, 2.50k op/s 2024-09-21T10:23:14.563 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:14.603 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:14.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:14.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:14.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:23:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:14 smithi129 ceph-mon[88471]: pgmap v1447: 109 pgs: 109 active+clean; 2.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 156 MiB/s wr, 2.50k op/s 2024-09-21T10:23:14.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:14.731 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 18 16 38470 38454 133.508 113.875 0.00863729 0.00748388 2024-09-21T10:23:14.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:14.761 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:14.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:14.862 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:14.889 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:14.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:14.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:14.991 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:15.019 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:15.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:15.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:15.121 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:15.146 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:15.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:15.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:15.248 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:15.275 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:15.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:15.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:15.377 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:15.403 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:15.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:15.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:15.505 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:15.526 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:15.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:15.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:15.628 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:15.650 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:15.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:15.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:15.731 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 19 16 39710 39694 130.56 77.5 0.00206715 0.00760012 2024-09-21T10:23:15.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:15.773 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:15.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:15.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:15.875 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:15.903 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:15.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:15.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:15.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:23:16.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:16.057 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:16.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:16.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:16.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:16.215 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:16.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:16.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:16.317 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:16.343 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:16.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:16.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:16.445 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:16.496 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:16.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:16.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:16.598 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:16.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:16 smithi107 ceph-mon[102221]: pgmap v1448: 109 pgs: 109 active+clean; 2.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-09-21T10:23:16.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:16 smithi107 ceph-mon[105022]: pgmap v1448: 109 pgs: 109 active+clean; 2.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-09-21T10:23:16.624 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:16.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:16.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:16.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:16.732 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:23:16.730989+0000 min lat: 0.00106792 max lat: 0.22341 avg lat: 0.00777955 2024-09-21T10:23:16.732 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:23:16.732 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 20 16 41123 41107 128.442 88.3125 0.0107012 0.00777955 2024-09-21T10:23:16.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:16.773 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:16.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:16.874 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:16.902 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:16.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:16.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:16.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:16 smithi129 ceph-mon[88471]: pgmap v1448: 109 pgs: 109 active+clean; 2.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-09-21T10:23:17.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:17.048 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:17.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:17.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:17.187 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:17.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:17.289 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:17.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.327 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:17.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:17.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:17.464 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:17.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:17.566 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:17.590 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:17.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:17.649 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:23:17.650 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:23:17.692 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:17.719 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:17.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:17.732 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 21 16 42793 42777 127.295 104.375 0.00767259 0.00785043 2024-09-21T10:23:17.821 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:17.852 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:17.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:17.954 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:17.990 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:17.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:18.092 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:18.112 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:18.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:18.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:18.253 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:18.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:18.355 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:18.391 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:18.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:18.493 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:18.531 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:18.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:18.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:18.671 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:18.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:18.732 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 22 16 44261 44245 125.679 91.75 0.00906091 0.00795179 2024-09-21T10:23:18.773 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:18.801 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:18.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:18.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:18 smithi107 ceph-mon[105022]: pgmap v1449: 109 pgs: 109 active+clean; 2.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-21T10:23:18.860 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:18 smithi107 ceph-mon[102221]: pgmap v1449: 109 pgs: 109 active+clean; 2.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-21T10:23:18.905 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:18.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:18 smithi129 ceph-mon[88471]: pgmap v1449: 109 pgs: 109 active+clean; 2.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-21T10:23:18.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.936 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:18.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:19.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:19.073 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:19.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:19.175 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:19.203 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:19.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:19.305 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:19.334 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:19.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:19.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:19.473 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:19.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:19.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:19.611 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:19.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:19.713 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:19.732 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 23 16 45870 45854 124.587 100.562 0.00741824 0.00802088 2024-09-21T10:23:19.740 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:19.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:19.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:19.878 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:19.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:19.980 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:19.984 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:20.004 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:20.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:20.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:20.146 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:20.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:20.248 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:20.278 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:20.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:20.379 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:20.406 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:20.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:20.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:20.544 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:20.592 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:23:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:23:20.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:23:20.646 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:20.672 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:20.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:20.732 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 24 16 47463 47447 123.544 99.5625 0.00800371 0.00808963 2024-09-21T10:23:20.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:20.812 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:20.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:20.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:20 smithi107 ceph-mon[102221]: pgmap v1450: 109 pgs: 109 active+clean; 2.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-09-21T10:23:20.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:20 smithi107 ceph-mon[105022]: pgmap v1450: 109 pgs: 109 active+clean; 2.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-09-21T10:23:20.914 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:20 smithi129 ceph-mon[88471]: pgmap v1450: 109 pgs: 109 active+clean; 2.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 123 MiB/s wr, 1.96k op/s 2024-09-21T10:23:20.946 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:20.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:21.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:23:21.047 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:21.070 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:21.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:21.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:21.196 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:21.196 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:21.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.217 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:21.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:21.318 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:21.345 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:21.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:21.447 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:21.480 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:21.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:21.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:21.624 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:21.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:21.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:21.732 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 25 16 48897 48881 122.186 89.625 0.0120357 0.00817975 2024-09-21T10:23:21.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.768 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:21.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:21.869 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:21.899 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:21.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:22.000 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:22.037 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:22.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:22.085 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:23:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:23:21] "GET /metrics HTTP/1.1" 200 36770 "" "Prometheus/2.51.0" 2024-09-21T10:23:22.139 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:22.170 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:22.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:22.272 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:22.299 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:22.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:22.400 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:22.425 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:22.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:22.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:22.557 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:22.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:22.658 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:22.686 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:22.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:22.733 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 26 16 50429 50413 121.169 95.75 0.00897579 0.00824793 2024-09-21T10:23:22.787 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:22.813 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:22.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:22 smithi107 ceph-mon[102221]: pgmap v1451: 109 pgs: 109 active+clean; 2.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-21T10:23:22.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:22 smithi107 ceph-mon[105022]: pgmap v1451: 109 pgs: 109 active+clean; 2.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-21T10:23:22.914 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:22.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:22 smithi129 ceph-mon[88471]: pgmap v1451: 109 pgs: 109 active+clean; 2.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s 2024-09-21T10:23:22.944 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:22.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:23.045 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:23.073 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:23.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:23.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:23.211 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:23.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:23.312 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:23.361 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:23.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:23.463 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:23.488 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:23.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:23.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:23.629 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:23.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:23.731 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:23.733 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 27 16 52012 51996 120.345 98.9375 0.0089874 0.00830413 2024-09-21T10:23:23.756 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:23.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:23.858 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:23.884 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:23.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:23.986 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:24.012 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:24.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:24.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:24.155 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:24.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:24.257 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:24.284 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:24.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:24.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:24.424 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:24.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:24.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:24.549 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:24.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:24.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:24.699 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:24.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:24.733 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 28 16 53428 53412 119.208 88.5 0.0105437 0.00838391 2024-09-21T10:23:24.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:24.845 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:24.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:24.846 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:24 smithi107 ceph-mon[102221]: pgmap v1452: 109 pgs: 109 active+clean; 3.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-09-21T10:23:24.847 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:24 smithi107 ceph-mon[105022]: pgmap v1452: 109 pgs: 109 active+clean; 3.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-09-21T10:23:24.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:24 smithi129 ceph-mon[88471]: pgmap v1452: 109 pgs: 109 active+clean; 3.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-09-21T10:23:24.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:24.990 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:24.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:25.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:25.126 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:25.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:25.227 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:25.250 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:25.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:25.352 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:25.379 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:25.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:25.480 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:25.509 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:25.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:25.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:25.645 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:25.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:25.733 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 29 16 54735 54719 117.914 81.6875 0.00227524 0.00847148 2024-09-21T10:23:25.746 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:25.769 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:25.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:25.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:25.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.913 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:25.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:26.015 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:26.041 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:26.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:26.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:26.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.184 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:26.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:26.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:26.325 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:26.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:26.427 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:26.450 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:26.551 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:26.581 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:26.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:26.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:26.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.722 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:26.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:26.733 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 30 16 55900 55884 116.41 72.8125 0.00849807 0.00858393 2024-09-21T10:23:26.823 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:26.844 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:26.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:26 smithi107 ceph-mon[102221]: pgmap v1453: 109 pgs: 109 active+clean; 3.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-21T10:23:26.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:26 smithi107 ceph-mon[105022]: pgmap v1453: 109 pgs: 109 active+clean; 3.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-21T10:23:26.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:26 smithi129 ceph-mon[88471]: pgmap v1453: 109 pgs: 109 active+clean; 3.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-21T10:23:26.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:26.945 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:26.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.970 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:26.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:27.071 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:27.096 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:27.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:27.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:27.231 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:27.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:27.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:27.375 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:27.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:27.440 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:27.476 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:27.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.504 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:27.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:27.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:27.621 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T10:23:27.654 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:27.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:27.733 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 31 16 57015 56999 114.903 69.6875 0.0111646 0.0086978 2024-09-21T10:23:27.755 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:27.782 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:27.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:27.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:27 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/716310199' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:23:27.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:27 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:23:27.920 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:27.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:28.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:28.056 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:28.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:28.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:27 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/716310199' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:23:28.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:27 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:23:28.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:27 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/716310199' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:23:28.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:27 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:23:28.158 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:28.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.179 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:28.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:28.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:28.315 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:28.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:28.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:28.460 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:28.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:28.561 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:28.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.597 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:28.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:28.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:28.720 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:28.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:28.733 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 32 16 58144 58128 113.517 70.5625 0.00332468 0.0087729 2024-09-21T10:23:28.822 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:28.848 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:28.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:28.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:28 smithi129 ceph-mon[88471]: pgmap v1454: 109 pgs: 109 active+clean; 3.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-21T10:23:28.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:28 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:23:28.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:28 smithi129 ceph-mon[88471]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T10:23:28.950 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:28.975 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:28.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:29.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:29.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:28 smithi107 ceph-mon[102221]: pgmap v1454: 109 pgs: 109 active+clean; 3.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-21T10:23:29.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:28 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:23:29.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:28 smithi107 ceph-mon[102221]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T10:23:29.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:28 smithi107 ceph-mon[105022]: pgmap v1454: 109 pgs: 109 active+clean; 3.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-21T10:23:29.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:28 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:23:29.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:28 smithi107 ceph-mon[105022]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T10:23:29.115 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:29.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:29.217 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:29.240 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:29.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:29.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:29.381 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:29.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:29.482 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:29.507 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:29.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:29.608 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:29.634 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:29.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:29.733 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 33 16 59530 59514 112.702 86.625 0.00814266 0.00886779 2024-09-21T10:23:29.736 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:29.783 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:29.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:29.885 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:29.910 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:29.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:30.012 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:30.034 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:30.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:30.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:30.181 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:30.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:30.276 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T10:23:30.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:30.363 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:30.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:30.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:30.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.522 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:30.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:30.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:30.632 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:23:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:23:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:23:30.645 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:23:30.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.682 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:30.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:30.734 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 34 16 60695 60679 111.529 72.8125 0.0102809 0.0089615 2024-09-21T10:23:30.783 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:30.826 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:30.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:30.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:30 smithi107 ceph-mon[102221]: pgmap v1456: 109 pgs: 109 active+clean; 3.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-21T10:23:30.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:30 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/439061744' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:23:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:30 smithi107 ceph-mon[105022]: pgmap v1456: 109 pgs: 109 active+clean; 3.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-21T10:23:30.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:30 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/439061744' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:23:30.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:30 smithi129 ceph-mon[88471]: pgmap v1456: 109 pgs: 109 active+clean; 3.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-21T10:23:30.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:30 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/439061744' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:23:30.928 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:30.959 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:30.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:31.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:31.106 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:31.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:31.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:31.257 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:31.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:31.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:31.398 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:31.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:31.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:31.537 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:31.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:31.639 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:31.672 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:31.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:31.733 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 35 16 62061 62045 110.781 85.375 0.012378 0.00902096 2024-09-21T10:23:31.774 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:31.802 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:31.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:31.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:31 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:31.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:31 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/439061744' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:23:31.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:31 smithi107 ceph-mon[102221]: osdmap e418: 8 total, 8 up, 8 in 2024-09-21T10:23:31.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:23:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:23:31] "GET /metrics HTTP/1.1" 200 36772 "" "Prometheus/2.51.0" 2024-09-21T10:23:31.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:31 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:31.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:31 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/439061744' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:23:31.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:31 smithi107 ceph-mon[105022]: osdmap e418: 8 total, 8 up, 8 in 2024-09-21T10:23:31.904 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:31 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:31 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/439061744' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:23:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:31 smithi129 ceph-mon[88471]: osdmap e418: 8 total, 8 up, 8 in 2024-09-21T10:23:31.931 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:31.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:32.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:32.074 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:32.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:32.175 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:32.199 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:32.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:32.295 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:23:32.295 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:23:32.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:32.333 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:23:32.333 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:23:32.340 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:32.365 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:23:32.366 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:23:32.403 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:23:32.404 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:23:32.434 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:23:32.434 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:23:32.442 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:32.466 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:23:32.466 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:23:32.469 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:32.499 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:23:32.499 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:23:32.530 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:23:32.530 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:23:32.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:32.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.606 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:32.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:32.707 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:32.734 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 36 16 63347 63331 109.937 80.375 0.0120484 0.00909052 2024-09-21T10:23:32.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.735 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:32.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:32.836 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:32.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.885 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:32.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:32 smithi129 ceph-mon[88471]: pgmap v1458: 109 pgs: 109 active+clean; 3.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T10:23:32.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:32 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3282473894' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:32.986 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:33.010 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:33.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:33.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:32 smithi107 ceph-mon[102221]: pgmap v1458: 109 pgs: 109 active+clean; 3.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T10:23:33.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:32 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3282473894' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:33.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:32 smithi107 ceph-mon[105022]: pgmap v1458: 109 pgs: 109 active+clean; 3.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T10:23:33.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:32 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3282473894' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:33.112 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:33.138 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:33.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:33.239 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:33.263 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:33.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:33.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:33.406 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:33.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:33.508 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:33.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.537 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:33.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:33.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:23:33.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:33.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.692 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:33.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:33.733 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 37 16 64573 64557 109.036 76.625 0.00802038 0.00916534 2024-09-21T10:23:33.793 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:33.816 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:33.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:33.918 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:33.939 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:33.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:34.040 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:34.065 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:34.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:34.166 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:34.192 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:34.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:34.294 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:34.320 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:34.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:34.422 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:34.449 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:34.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:34.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:34.590 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:34.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:34.691 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:34.715 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:34.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:34.734 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 38 16 65510 65494 107.708 58.5625 0.00456867 0.00927031 2024-09-21T10:23:34.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:34.849 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:34.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:34.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:34 smithi129 ceph-mon[88471]: pgmap v1459: 109 pgs: 109 active+clean; 3.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-21T10:23:34.951 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:34.959 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:34 smithi107 ceph-mon[102221]: pgmap v1459: 109 pgs: 109 active+clean; 3.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-21T10:23:34.959 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:34 smithi107 ceph-mon[105022]: pgmap v1459: 109 pgs: 109 active+clean; 3.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s 2024-09-21T10:23:34.977 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:34.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:35.042 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:35.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:35.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.131 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:35.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:35.232 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:35.257 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:35.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:35.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:35.403 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:35.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:35.504 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:35.528 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:35.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:35.630 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:35.656 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:35.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:35.734 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 39 16 66711 66695 106.871 75.0625 0.0251885 0.0093514 2024-09-21T10:23:35.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:35.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.803 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:35.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:35.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:35.941 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:35.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:36.043 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:36.071 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:36.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:36.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:36.218 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:36.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:36.314 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:23:36.314 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:23:36.314 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.314 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47186.0:66806 14.9 14:983e55a4:::benchmark_data_smithi107_312686_object66805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-09-21T10:23:36.314 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:23:35.821794+0000", 2024-09-21T10:23:36.314 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0083234810000000006, 2024-09-21T10:23:36.314 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0083557809999999996, 2024-09-21T10:23:36.314 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:23:36.314 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:23:36.314 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:23:36.315 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:23:36.315 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47186", 2024-09-21T10:23:36.315 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:23:36.315 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 66806 2024-09-21T10:23:36.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.315 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:23:36.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:23:36.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.821794+0000", 2024-09-21T10:23:36.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:23:36.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.315 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.315 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:23:36.315 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.821794+0000", 2024-09-21T10:23:36.315 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:23:36.315 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:23:36.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.821795+0000", 2024-09-21T10:23:36.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5600000000000001e-06 2024-09-21T10:23:36.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:23:36.316 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.821827+0000", 2024-09-21T10:23:36.316 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.1887000000000001e-05 2024-09-21T10:23:36.316 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.316 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.316 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:23:36.317 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.821845+0000", 2024-09-21T10:23:36.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8006999999999999e-05 2024-09-21T10:23:36.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.317 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:23:36.317 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.821854+0000", 2024-09-21T10:23:36.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6750000000000008e-06 2024-09-21T10:23:36.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.317 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.317 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:23:36.317 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.821918+0000", 2024-09-21T10:23:36.317 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3825000000000006e-05 2024-09-21T10:23:36.317 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:23:36.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.822056+0000", 2024-09-21T10:23:36.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013829200000000001 2024-09-21T10:23:36.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:23:36.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.822192+0000", 2024-09-21T10:23:36.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013636 2024-09-21T10:23:36.318 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.318 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.318 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:36.318 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.825560+0000", 2024-09-21T10:23:36.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0033681290000000001 2024-09-21T10:23:36.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.319 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:36.319 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.827961+0000", 2024-09-21T10:23:36.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024008340000000001 2024-09-21T10:23:36.319 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:23:36.319 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:23:36.319 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:23:36.319 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.319 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47186.0:66811 14.9 14:93579eaa:::benchmark_data_smithi107_312686_object66810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-09-21T10:23:36.319 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:23:35.827750+0000", 2024-09-21T10:23:36.319 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0023672749999999998, 2024-09-21T10:23:36.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024726769999999999, 2024-09-21T10:23:36.320 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:23:36.320 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:23:36.320 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:23:36.320 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:23:36.320 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47186", 2024-09-21T10:23:36.320 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:23:36.320 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 66811 2024-09-21T10:23:36.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.320 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:23:36.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:23:36.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.827750+0000", 2024-09-21T10:23:36.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:23:36.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:23:36.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.827750+0000", 2024-09-21T10:23:36.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:23:36.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:23:36.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.827752+0000", 2024-09-21T10:23:36.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.968e-06 2024-09-21T10:23:36.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:23:36.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.827780+0000", 2024-09-21T10:23:36.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8433999999999999e-05 2024-09-21T10:23:36.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:23:36.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.827800+0000", 2024-09-21T10:23:36.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0043000000000001e-05 2024-09-21T10:23:36.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:23:36.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.827812+0000", 2024-09-21T10:23:36.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1875999999999999e-05 2024-09-21T10:23:36.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:23:36.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.827990+0000", 2024-09-21T10:23:36.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017814799999999999 2024-09-21T10:23:36.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:23:36.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.828115+0000", 2024-09-21T10:23:36.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000124447 2024-09-21T10:23:36.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:23:36.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.828245+0000", 2024-09-21T10:23:36.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013051800000000001 2024-09-21T10:23:36.323 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:23:36.323 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:23:36.324 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:23:36.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.324 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.47186.0:66809 14.4 e418/411 14:2c848b3b:::benchmark_data_smithi107_312686_object66808:head v 418'4153, mlcod=418'4151)", 2024-09-21T10:23:36.324 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:23:35.828540+0000", 2024-09-21T10:23:36.324 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0015767509999999999, 2024-09-21T10:23:36.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017380309999999999, 2024-09-21T10:23:36.324 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:23:36.324 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:23:36.324 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:23:36.324 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:23:36.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.324 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:23:36.325 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.828540+0000", 2024-09-21T10:23:36.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:23:36.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:23:36.325 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.828540+0000", 2024-09-21T10:23:36.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:23:36.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:23:36.325 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.828541+0000", 2024-09-21T10:23:36.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5600000000000005e-07 2024-09-21T10:23:36.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:23:36.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.828580+0000", 2024-09-21T10:23:36.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9229e-05 2024-09-21T10:23:36.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:23:36.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.828592+0000", 2024-09-21T10:23:36.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1389999999999999e-05 2024-09-21T10:23:36.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:23:36.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.828599+0000", 2024-09-21T10:23:36.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8159999999999996e-06 2024-09-21T10:23:36.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:23:36.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.828656+0000", 2024-09-21T10:23:36.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6919000000000002e-05 2024-09-21T10:23:36.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:23:36.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.828677+0000", 2024-09-21T10:23:36.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0993999999999999e-05 2024-09-21T10:23:36.327 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:23:36.327 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:23:36.327 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:23:36.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.327 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.47186.0:66812 14.f e418/411 14:fe4735ea:::benchmark_data_smithi107_312686_object66811:head v 418'4111, mlcod=418'4109)", 2024-09-21T10:23:36.327 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:23:35.828533+0000", 2024-09-21T10:23:36.328 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0015846090000000001, 2024-09-21T10:23:36.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017902899999999999, 2024-09-21T10:23:36.328 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:23:36.328 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:23:36.328 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:23:36.328 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:23:36.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:23:36.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.828533+0000", 2024-09-21T10:23:36.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:23:36.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:23:36.329 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.828533+0000", 2024-09-21T10:23:36.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:23:36.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:23:36.329 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.828533+0000", 2024-09-21T10:23:36.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9899999999999999e-07 2024-09-21T10:23:36.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:23:36.329 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.828593+0000", 2024-09-21T10:23:36.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9519999999999999e-05 2024-09-21T10:23:36.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.330 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:23:36.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.828603+0000", 2024-09-21T10:23:36.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0352e-05 2024-09-21T10:23:36.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.330 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:23:36.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.828610+0000", 2024-09-21T10:23:36.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4590000000000003e-06 2024-09-21T10:23:36.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.330 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:23:36.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.828678+0000", 2024-09-21T10:23:36.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7817000000000005e-05 2024-09-21T10:23:36.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:23:36.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:23:36.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:23:36.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:35.828696+0000", 2024-09-21T10:23:36.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8442999999999999e-05 2024-09-21T10:23:36.331 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:23:36.331 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:23:36.331 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:23:36.331 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:23:36.331 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-09-21T10:23:36.331 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 4 2024-09-21T10:23:36.331 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:23:36.332 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:36.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.372 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:36.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:36.473 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:36.502 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:36.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:36.604 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:36.634 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:36.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:36.734 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:23:36.732894+0000 min lat: 0.00106792 max lat: 0.22341 avg lat: 0.00941675 2024-09-21T10:23:36.734 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:23:36.734 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 40 16 67952 67936 106.138 77.5625 0.0123312 0.00941675 2024-09-21T10:23:36.736 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:36.760 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:36.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:36.862 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:36.884 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:36.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:36.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:36 smithi129 ceph-mon[88471]: pgmap v1460: 109 pgs: 109 active+clean; 3.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-09-21T10:23:36.986 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:37.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:23:37.012 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:37.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:37.045 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:36 smithi107 ceph-mon[102221]: pgmap v1460: 109 pgs: 109 active+clean; 3.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-09-21T10:23:37.046 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:36 smithi107 ceph-mon[105022]: pgmap v1460: 109 pgs: 109 active+clean; 3.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-09-21T10:23:37.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:37.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:23:37.169 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:37.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:37.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:37.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.307 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:37.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:37.408 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:37.431 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:37.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:37.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:37.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.581 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:37.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:37.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:37.721 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:37.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:37.734 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 41 16 69062 69046 105.241 69.375 0.00712494 0.00949645 2024-09-21T10:23:37.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:37.863 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:37.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:37.965 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:37.995 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:37.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:38.098 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:38.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.141 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:38.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:38.243 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:38.267 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:38.369 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:38.399 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:38.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:38.500 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:38.529 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:38.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:38.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:38.671 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:38.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:38.675 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:38.734 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 42 16 70280 70264 104.546 76.125 0.0100927 0.00955953 2024-09-21T10:23:38.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:38.814 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:38.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:38.828 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:38 smithi107 ceph-mon[102221]: pgmap v1461: 109 pgs: 109 active+clean; 4.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-21T10:23:38.829 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:38 smithi107 ceph-mon[105022]: pgmap v1461: 109 pgs: 109 active+clean; 4.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-21T10:23:38.916 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:38.942 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:38.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:39.044 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:39.081 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:39.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:39.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:38 smithi129 ceph-mon[88471]: pgmap v1461: 109 pgs: 109 active+clean; 4.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-21T10:23:39.183 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:39.210 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:39.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:39.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:39.338 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:39.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:39.440 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:39.498 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:39.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:39.600 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:39.626 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:39.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:39.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:39.735 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 43 16 71370 71354 103.699 68.125 0.0225869 0.0096358 2024-09-21T10:23:39.785 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:39.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:39.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:39.921 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:39.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:40.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:40.069 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:40.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:40.171 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:40.195 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:40.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:40.296 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:40.335 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:40.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:40.437 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:40.460 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:40.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:40.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:40.609 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:40.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:40.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:23:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:23:40.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:23:40.687 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T10:23:40.712 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:40.735 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 44 16 72486 72470 102.927 69.75 0.0226761 0.00971013 2024-09-21T10:23:40.735 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:40.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:40.837 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:40 smithi107 ceph-mon[102221]: pgmap v1462: 109 pgs: 109 active+clean; 4.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-21T10:23:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:40 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1710222642' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:23:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:40 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:23:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:40 smithi107 ceph-mon[105022]: pgmap v1462: 109 pgs: 109 active+clean; 4.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-21T10:23:40.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:40 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1710222642' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:23:40.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:40 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:23:40.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.877 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:40.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:40.978 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:41.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.025 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:41.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:41.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:40 smithi129 ceph-mon[88471]: pgmap v1462: 109 pgs: 109 active+clean; 4.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-21T10:23:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:40 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1710222642' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:23:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:40 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:23:41.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.172 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:41.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:41.273 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:41.298 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:41.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:41.400 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:41.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.429 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:41.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:41.531 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:41.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.578 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:41.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:41.679 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:41.721 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:41.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:41.735 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 45 16 73639 73623 102.241 72.0625 0.0103158 0.00977564 2024-09-21T10:23:41.822 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:41.864 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:41.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:41.954 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:41 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:23:41.954 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:41 smithi107 ceph-mon[102221]: osdmap e419: 8 total, 8 up, 8 in 2024-09-21T10:23:41.954 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:41.954 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:41.954 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:23:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:23:41] "GET /metrics HTTP/1.1" 200 36774 "" "Prometheus/2.51.0" 2024-09-21T10:23:41.954 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:41 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:23:41.955 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:41 smithi107 ceph-mon[105022]: osdmap e419: 8 total, 8 up, 8 in 2024-09-21T10:23:41.955 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:41.955 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:41.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:42.013 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:42.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:42.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:42.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:41 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:23:42.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:41 smithi129 ceph-mon[88471]: osdmap e419: 8 total, 8 up, 8 in 2024-09-21T10:23:42.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:42.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:42.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.175 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:42.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:42.200 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:42.276 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:42.325 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:42.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:42.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:42.483 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:42.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:42.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:42.619 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:42.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:42.720 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:42.735 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 46 16 74580 74564 101.297 58.8125 0.015386 0.00986667 2024-09-21T10:23:42.743 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:42.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:42.845 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:42.873 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:42.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:42.974 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:43.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.041 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:43.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:43.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:43.179 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:43.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:43.187 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:23:43.235 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:43 smithi107 ceph-mon[105022]: pgmap v1464: 109 pgs: 109 active+clean; 4.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-21T10:23:43.235 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:43 smithi107 ceph-mon[102221]: pgmap v1464: 109 pgs: 109 active+clean; 4.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-21T10:23:43.282 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:43.311 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:43.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:43.413 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:43.455 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:43.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:43.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:43.601 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:43.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:43.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:43 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4018036092' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:43.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:43 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:43 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4018036092' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:43.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:43 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:43 smithi129 ceph-mon[88471]: pgmap v1464: 109 pgs: 109 active+clean; 4.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-21T10:23:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:43 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4018036092' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:43 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:43.703 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:43.732 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:43.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:43.735 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 47 16 75711 75695 100.646 70.6875 0.0099661 0.00993127 2024-09-21T10:23:43.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:43.876 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:43.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:43.977 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:44.007 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:44.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:44.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:44.144 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:44.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:44.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:44.270 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:44.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:44.371 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:44.394 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:44.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:44.495 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:44.522 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:44.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:44 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:44 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:23:44.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:44 smithi107 ceph-mon[102221]: osdmap e420: 8 total, 8 up, 8 in 2024-09-21T10:23:44.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:44 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:44.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:44 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:23:44.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:44 smithi107 ceph-mon[105022]: osdmap e420: 8 total, 8 up, 8 in 2024-09-21T10:23:44.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:44.659 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:44.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:44 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:44 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:23:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:44 smithi129 ceph-mon[88471]: osdmap e420: 8 total, 8 up, 8 in 2024-09-21T10:23:44.735 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 48 16 76782 76766 99.9432 66.9375 0.00308163 0.00998415 2024-09-21T10:23:44.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:44.781 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:44.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:44.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:44.927 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:44.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:45.029 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:45.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.058 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:45.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:45.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:45.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.193 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:45.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:45.294 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:45.322 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:45.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:45.423 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:45.454 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:45.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:45.556 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:45.584 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:45.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:45 smithi107 ceph-mon[102221]: pgmap v1466: 109 pgs: 109 active+clean; 4.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-21T10:23:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:45 smithi107 ceph-mon[105022]: pgmap v1466: 109 pgs: 109 active+clean; 4.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-21T10:23:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:45 smithi129 ceph-mon[88471]: pgmap v1466: 109 pgs: 109 active+clean; 4.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-21T10:23:45.686 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:45.711 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:45.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:45.735 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 49 16 77818 77802 99.2249 64.75 0.0163252 0.0100726 2024-09-21T10:23:45.813 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:45.841 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:45.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:45.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:45.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.979 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:45.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:46.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:46.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.123 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:46.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:46.224 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:46.255 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:46.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:46.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:46.390 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:46.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:46.491 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:46.513 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:46.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:46.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:46.649 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:46.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:46.735 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 50 16 78847 78831 98.5266 64.3125 0.00542267 0.0101447 2024-09-21T10:23:46.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:46.793 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:46.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:46.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:46.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.935 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:46.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:47.036 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:47.065 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:47.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:47.167 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:47.195 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:47.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:47.297 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:47.330 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:47.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:47.432 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:47.455 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:47.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:47.557 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:47.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.582 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:47.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:47.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:47 smithi107 ceph-mon[102221]: pgmap v1467: 109 pgs: 109 active+clean; 4.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-21T10:23:47.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:47 smithi107 ceph-mon[105022]: pgmap v1467: 109 pgs: 109 active+clean; 4.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-21T10:23:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:47 smithi129 ceph-mon[88471]: pgmap v1467: 109 pgs: 109 active+clean; 4.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-21T10:23:47.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:47.730 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:47.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:47.735 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 51 16 79918 79902 97.9071 66.9375 0.0253417 0.0102084 2024-09-21T10:23:47.832 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:47.876 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:47.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:47.978 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:48.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.023 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:48.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:48.124 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:48.149 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:48.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:48.251 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:48.273 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:48.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:48.375 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:48.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.415 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:48.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:48.516 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:48.541 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:48.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:48.566 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:48 smithi107 ceph-mon[102221]: pgmap v1468: 109 pgs: 109 active+clean; 4.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 54 MiB/s wr, 862 op/s 2024-09-21T10:23:48.567 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:48 smithi107 ceph-mon[105022]: pgmap v1468: 109 pgs: 109 active+clean; 4.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 54 MiB/s wr, 862 op/s 2024-09-21T10:23:48.612 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T10:23:48.642 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:48.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:48 smithi129 ceph-mon[88471]: pgmap v1468: 109 pgs: 109 active+clean; 4.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 54 MiB/s wr, 862 op/s 2024-09-21T10:23:48.670 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:48.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:48.736 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 52 16 80951 80935 97.2654 64.5625 0.00390946 0.0102749 2024-09-21T10:23:48.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:48.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.824 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:48.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:48.925 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:48.950 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:48.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:49.014 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:49.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:49.097 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:49.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:49.199 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:49.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.226 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:49.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:49.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:49.361 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:49.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:49.463 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:49.500 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:49.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:49.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:49.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:49 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:49.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:49 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:49.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:49 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:23:49.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:49 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:23:49.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:49 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:49.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:49 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:49.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:49 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:49.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:49 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:23:49.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:49 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:23:49.611 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:49 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:49.643 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:49.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:49 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:49 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:49 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:23:49.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:49 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:23:49.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:49 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:23:49.736 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 53 16 81979 81963 96.6425 64.25 0.0201329 0.0103423 2024-09-21T10:23:49.744 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:49.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.767 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:49.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:49.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:49.907 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:49.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:50.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:50.041 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:50.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:50.144 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:50.168 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:50.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:50.270 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:50.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.301 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:50.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:50.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:50.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.465 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:50.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:50.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:50.586 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:50 smithi107 ceph-mon[102221]: pgmap v1469: 109 pgs: 109 active+clean; 4.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 77 MiB/s wr, 1.22k op/s 2024-09-21T10:23:50.586 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:50 smithi107 ceph-mon[105022]: pgmap v1469: 109 pgs: 109 active+clean; 4.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 77 MiB/s wr, 1.22k op/s 2024-09-21T10:23:50.613 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:50.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:50 smithi129 ceph-mon[88471]: pgmap v1469: 109 pgs: 109 active+clean; 4.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 77 MiB/s wr, 1.22k op/s 2024-09-21T10:23:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:23:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:23:50.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:23:50.715 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:50.736 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 54 16 83010 82994 96.046 64.4375 0.011387 0.0104066 2024-09-21T10:23:50.744 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:50.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:50.846 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:50.875 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:50.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:50.900 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:50.976 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:51.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.005 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:51.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:51.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:51.145 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:51.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:51.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:51.277 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:51.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:51.379 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:51.420 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:51.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:51.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:51.578 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:51.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:51.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:51.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:23:51.723 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:51.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:51.736 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 55 16 84053 84037 95.4847 65.1875 0.00952183 0.0104662 2024-09-21T10:23:51.764 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:23:51.764 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:23:51.815 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:23:51.815 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:23:51.826 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:51.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.850 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:51.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:51.854 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:23:51.855 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:23:51.894 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:23:51.894 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:23:51.922 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:23:51.922 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:23:51.922 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:23:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:23:51] "GET /metrics HTTP/1.1" 200 36774 "" "Prometheus/2.51.0" 2024-09-21T10:23:51.949 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:23:51.950 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:23:51.951 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:51.974 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:51.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:51.980 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:23:51.980 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:23:52.018 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:23:52.018 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:23:52.075 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:52.103 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:52.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:52.204 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:52.230 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:52.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:52.331 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:52.357 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:52.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:52.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:52.496 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:52.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:52.598 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:52.625 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:52.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:52.727 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:52.736 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 56 16 85064 85048 94.9079 63.1875 0.00730064 0.0105319 2024-09-21T10:23:52.752 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:52.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:52.854 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:52.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:52 smithi107 ceph-mon[102221]: pgmap v1470: 109 pgs: 109 active+clean; 4.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-21T10:23:52.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:52 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3226436412' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:52.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:52 smithi107 ceph-mon[105022]: pgmap v1470: 109 pgs: 109 active+clean; 4.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-21T10:23:52.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:52 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3226436412' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:52.880 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:52.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:52 smithi129 ceph-mon[88471]: pgmap v1470: 109 pgs: 109 active+clean; 4.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-21T10:23:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:52 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3226436412' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:52.983 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:53.006 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:53.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:53.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:53.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.154 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:53.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:53.255 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:53.280 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:53.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:53.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:53.422 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:53.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:53.524 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:53.548 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:53.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:53.651 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:53.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.685 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:53.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:53.736 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 57 16 86164 86148 94.4489 68.75 0.0175501 0.010583 2024-09-21T10:23:53.786 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:53.814 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:53.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:53.856 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:53.915 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:53.940 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:53.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:54.042 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:54.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.070 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:54.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:54.172 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:54.198 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:54.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:54.300 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:54.324 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:54.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:54.425 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:54.471 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:54.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:54.573 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:54.597 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:54.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:54.699 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:54.771 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 58 16 87088 87072 93.8161 57.75 0.0121183 0.0106538 2024-09-21T10:23:54.798 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:54.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:54 smithi107 ceph-mon[102221]: pgmap v1471: 109 pgs: 109 active+clean; 5.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:23:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:54 smithi107 ceph-mon[105022]: pgmap v1471: 109 pgs: 109 active+clean; 5.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:23:54.900 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:54 smithi129 ceph-mon[88471]: pgmap v1471: 109 pgs: 109 active+clean; 5.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:23:54.923 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:54.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:55.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:55.064 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:55.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:55.165 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:55.193 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:55.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:55.286 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:23:55.295 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:55.318 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:55.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:55.420 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:55.442 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:55.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:55.544 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:55.566 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:55.668 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:55.692 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:55.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:55.736 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 59 16 87981 87965 93.1719 55.8125 0.00775032 0.0107274 2024-09-21T10:23:55.794 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:55.820 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:55.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:55.921 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:55.944 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:55.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:56.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:56.087 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:56.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:56.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:56.235 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:56.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:56.337 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:56.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.377 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:56.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:56.475 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:23:56.476 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:23:56.478 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:56.502 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:56.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:56.603 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:56.628 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:56.729 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:56.736 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:23:56.735675+0000 min lat: 0.00106792 max lat: 0.22341 avg lat: 0.0107977 2024-09-21T10:23:56.737 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:23:56.737 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 60 15 88888 88873 92.5646 56.75 0.0172466 0.0107977 2024-09-21T10:23:56.769 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:56.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:56.803 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:56 smithi107 ceph-mon[102221]: pgmap v1472: 109 pgs: 109 active+clean; 5.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 60 MiB/s wr, 958 op/s 2024-09-21T10:23:56.803 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:56.804 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:56 smithi107 ceph-mon[105022]: pgmap v1472: 109 pgs: 109 active+clean; 5.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 60 MiB/s wr, 958 op/s 2024-09-21T10:23:56.804 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:56.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:56.906 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:56.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:56 smithi129 ceph-mon[88471]: pgmap v1472: 109 pgs: 109 active+clean; 5.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 60 MiB/s wr, 958 op/s 2024-09-21T10:23:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:57.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:57.035 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:57.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:57.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:57.177 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:57.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:57.279 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:57.306 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:57.407 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:57.434 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:57.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:57.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:57.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.576 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:57.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:57.677 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:57.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:23:57.700 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:57.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:57.737 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 61 16 89887 89871 92.0695 62.375 0.0160537 0.0108561 2024-09-21T10:23:57.801 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:57.847 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:57.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:57.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:57 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3645588975' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:23:57.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:57 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:23:57.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:57.989 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:57.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:58.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:58.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:57 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3645588975' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:23:58.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:57 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:23:58.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:57 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3645588975' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:23:58.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:57 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:23:58.127 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:58.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:58.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:58.250 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:58.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:58.352 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:58.376 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:58.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:58.478 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:58.502 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:58.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:58.604 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:58.639 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:58.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:58.737 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 62 16 90808 90792 91.5129 57.5625 0.00776524 0.0109224 2024-09-21T10:23:58.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:58.778 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:58.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:58.879 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:58 smithi129 ceph-mon[88471]: pgmap v1473: 109 pgs: 109 active+clean; 5.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 61 MiB/s wr, 975 op/s 2024-09-21T10:23:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:58 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:23:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:23:58 smithi129 ceph-mon[88471]: osdmap e421: 8 total, 8 up, 8 in 2024-09-21T10:23:58.921 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:58.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:59.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:59.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.053 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:59.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:59.054 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T10:23:59.054 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:58 smithi107 ceph-mon[102221]: pgmap v1473: 109 pgs: 109 active+clean; 5.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 61 MiB/s wr, 975 op/s 2024-09-21T10:23:59.054 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:58 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:23:59.054 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:23:58 smithi107 ceph-mon[102221]: osdmap e421: 8 total, 8 up, 8 in 2024-09-21T10:23:59.054 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:58 smithi107 ceph-mon[105022]: pgmap v1473: 109 pgs: 109 active+clean; 5.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 61 MiB/s wr, 975 op/s 2024-09-21T10:23:59.055 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:58 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:23:59.055 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:23:58 smithi107 ceph-mon[105022]: osdmap e421: 8 total, 8 up, 8 in 2024-09-21T10:23:59.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:59.187 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:23:59.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:59.289 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:59.317 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:59.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:59.419 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:59.442 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:59.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:59.544 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:59.573 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:59.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:59.676 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:59.700 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:59.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:59.738 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 63 16 91713 91697 90.958 56.5625 0.00538623 0.010989 2024-09-21T10:23:59.803 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:59.833 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:59.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:59.935 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:59.961 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:23:59.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:00.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:00.099 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:00.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:00.200 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:00.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.222 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:00.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:00.323 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:00.362 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:00.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:00.463 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:00.497 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:00.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:00.598 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:00.643 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:00.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:00.645 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:24:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:24:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:24:00.737 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 64 16 92717 92701 90.5172 62.75 0.0170632 0.0110416 2024-09-21T10:24:00.745 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:00.770 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:00.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:00.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:00.913 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:00.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:00.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:00 smithi129 ceph-mon[88471]: pgmap v1475: 109 pgs: 109 active+clean; 5.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 994 op/s 2024-09-21T10:24:00.956 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:00 smithi107 ceph-mon[102221]: pgmap v1475: 109 pgs: 109 active+clean; 5.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 994 op/s 2024-09-21T10:24:00.956 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:00 smithi107 ceph-mon[105022]: pgmap v1475: 109 pgs: 109 active+clean; 5.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 994 op/s 2024-09-21T10:24:00.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T10:24:01.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:01.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.107 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:01.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:01.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:01.253 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:01.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:01.355 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:01.381 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:01.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:01.483 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:01.511 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:01.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:01.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:01.630 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:01.702 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:01.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:01.727 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:01.738 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 65 16 93643 93627 90.0141 57.875 0.0103651 0.0111031 2024-09-21T10:24:01.770 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:24:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:24:01] "GET /metrics HTTP/1.1" 200 36775 "" "Prometheus/2.51.0" 2024-09-21T10:24:01.804 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:01.834 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:01.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:01.936 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:01.959 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:01.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:02.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:02.107 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:02.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:02.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:02.248 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:02.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:02.350 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:02.378 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:02.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:02.480 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:02.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.521 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:02.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:02.622 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:02.648 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:02.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:02.738 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 66 16 94635 94619 89.5895 62 0.00778091 0.011157 2024-09-21T10:24:02.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:02.789 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:02.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:02.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:02 smithi129 ceph-mon[88471]: pgmap v1476: 109 pgs: 109 active+clean; 5.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 994 op/s 2024-09-21T10:24:02.931 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:02.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:03.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:03.070 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:02 smithi107 ceph-mon[102221]: pgmap v1476: 109 pgs: 109 active+clean; 5.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 994 op/s 2024-09-21T10:24:03.071 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:02 smithi107 ceph-mon[105022]: pgmap v1476: 109 pgs: 109 active+clean; 5.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 994 op/s 2024-09-21T10:24:03.075 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:03.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:03.178 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:03.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.207 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:03.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:03.308 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:03.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.335 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:03.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:03.436 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:03.495 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:03.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:03.597 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:03.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.634 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:03.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:03.735 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:03.738 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 67 16 95567 95551 89.1217 58.25 0.0130198 0.0112152 2024-09-21T10:24:03.765 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:03.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:03.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:03.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.908 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:03.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:04.009 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:04.047 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:04.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:04.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:04.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.196 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:04.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:04.287 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:04.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:04.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.367 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:04.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:04.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:04.509 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:04.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:04.612 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:04.637 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:04.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:04.687 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:24:04.738 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 68 16 96547 96531 88.7118 61.25 0.00836733 0.0112664 2024-09-21T10:24:04.738 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:04.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.765 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:04.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:04.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:04.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.905 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:04.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:04.985 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:04 smithi107 ceph-mon[102221]: pgmap v1477: 109 pgs: 109 active+clean; 5.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 60 MiB/s wr, 967 op/s 2024-09-21T10:24:04.985 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:04 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1785058859' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:24:04.986 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:04 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:24:04.986 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:04 smithi107 ceph-mon[105022]: pgmap v1477: 109 pgs: 109 active+clean; 5.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 60 MiB/s wr, 967 op/s 2024-09-21T10:24:04.986 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:04 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1785058859' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:24:04.986 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:04 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:24:05.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:05.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.059 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:05.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:05.159 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:05.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:04 smithi129 ceph-mon[88471]: pgmap v1477: 109 pgs: 109 active+clean; 5.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 60 MiB/s wr, 967 op/s 2024-09-21T10:24:05.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:04 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1785058859' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:24:05.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:04 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:24:05.190 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:05.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:05.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:05.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.346 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:05.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:05.447 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:05.472 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:05.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:05.574 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:05.598 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:05.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:05.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:05.738 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 69 16 97418 97402 88.215 54.4375 0.00753012 0.0113302 2024-09-21T10:24:05.742 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:05.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:05.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:05.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:05 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:05.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:05 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:24:05.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:05 smithi107 ceph-mon[102221]: osdmap e422: 8 total, 8 up, 8 in 2024-09-21T10:24:05.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:05 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:05.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:05 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:24:05.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:05 smithi107 ceph-mon[105022]: osdmap e422: 8 total, 8 up, 8 in 2024-09-21T10:24:05.884 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:05.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:05.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:06.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.028 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:06.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:06.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:06.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:05 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:06.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:05 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:24:06.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:05 smithi129 ceph-mon[88471]: osdmap e422: 8 total, 8 up, 8 in 2024-09-21T10:24:06.190 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:06.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:06.222 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:24:06.222 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:24:06.293 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:06.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.331 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:06.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:06.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:06.468 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:06.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:06.569 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:06.597 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:06.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:06.699 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:06.735 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T10:24:06.738 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 70 16 98321 98305 87.761 56.4375 0.0197411 0.0113888 2024-09-21T10:24:06.743 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:06.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:06.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:06.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:06 smithi107 ceph-mon[102221]: pgmap v1479: 109 pgs: 109 active+clean; 5.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:24:06.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:06 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3276304353' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:24:06.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:06 smithi107 ceph-mon[105022]: pgmap v1479: 109 pgs: 109 active+clean; 5.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:24:06.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:06 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3276304353' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:24:06.882 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:06.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:06.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:07.020 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:07.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:07.122 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:07.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.150 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:07.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:07.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:06 smithi129 ceph-mon[88471]: pgmap v1479: 109 pgs: 109 active+clean; 5.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:24:07.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:06 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3276304353' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:24:07.251 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:07.277 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:07.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:07.379 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:07.418 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:07.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:07.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:07.561 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:07.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:07.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:07.700 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:07.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:07.738 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 71 16 99286 99270 87.3743 60.3125 0.00976613 0.0114387 2024-09-21T10:24:07.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:07.845 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:07.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:07 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3276304353' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:24:07.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:07 smithi107 ceph-mon[102221]: osdmap e423: 8 total, 8 up, 8 in 2024-09-21T10:24:07.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:07 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3276304353' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:24:07.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:07 smithi107 ceph-mon[105022]: osdmap e423: 8 total, 8 up, 8 in 2024-09-21T10:24:07.947 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:07.976 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:07.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:08.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:08.122 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:08.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:08.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:07 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3276304353' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:24:08.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:07 smithi129 ceph-mon[88471]: osdmap e423: 8 total, 8 up, 8 in 2024-09-21T10:24:08.224 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:08.248 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:08.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:08.350 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:08.376 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:08.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:08.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:08.518 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:08.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:08.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:08.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.679 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:08.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:08.710 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T10:24:08.739 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 72 16 100257 100241 87.0032 60.6875 0.0101432 0.0114888 2024-09-21T10:24:08.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:08.806 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:08 smithi107 ceph-mon[102221]: pgmap v1481: 109 pgs: 109 active+clean; 5.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 46 MiB/s wr, 737 op/s 2024-09-21T10:24:08.807 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:08 smithi107 ceph-mon[105022]: pgmap v1481: 109 pgs: 109 active+clean; 5.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 46 MiB/s wr, 737 op/s 2024-09-21T10:24:08.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.826 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:08.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:08.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:08.981 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:08.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:09.083 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:09.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.129 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:09.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:09.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:24:09.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:08 smithi129 ceph-mon[88471]: pgmap v1481: 109 pgs: 109 active+clean; 5.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 46 MiB/s wr, 737 op/s 2024-09-21T10:24:09.230 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:09.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.257 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:09.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:09.349 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:24:09.349 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:24:09.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:09.404 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:24:09.404 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:24:09.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.423 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:09.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:09.439 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:24:09.439 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:24:09.483 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:24:09.483 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:24:09.519 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:24:09.519 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:24:09.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:09.553 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:24:09.553 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:24:09.575 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:09.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:09.583 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:24:09.583 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:24:09.613 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:24:09.613 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:24:09.677 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:09.705 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:09.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:09.739 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 73 16 101051 101035 86.4911 49.625 0.00414195 0.0115571 2024-09-21T10:24:09.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:09.842 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:09.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:09.927 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:24:09.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:09.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.985 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:09.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:10.070 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:09 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1736236333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:10.086 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:10.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:09 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1736236333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:10.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:09 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1736236333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:10.114 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:10.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:10.216 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:10.255 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:10.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:10.274 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:24:10.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:10.397 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:10.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:10.400 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:24:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:24:10.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:24:10.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:10.535 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:10.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:10.637 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:10.663 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:10.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:10.739 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 74 16 101983 101967 86.1094 58.25 0.0250779 0.0116075 2024-09-21T10:24:10.752 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:10.764 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:10.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.794 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:10.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:10.837 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:10.895 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:10.918 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:10.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:10.968 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:10 smithi107 ceph-mon[102221]: pgmap v1482: 109 pgs: 109 active+clean; 6.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-21T10:24:10.968 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:10 smithi107 ceph-mon[105022]: pgmap v1482: 109 pgs: 109 active+clean; 6.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-21T10:24:11.020 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:11.061 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:11.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:11.163 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:11.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:10 smithi129 ceph-mon[88471]: pgmap v1482: 109 pgs: 109 active+clean; 6.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-21T10:24:11.189 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:11.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:11.290 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:11.332 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:11.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:11.434 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:11.465 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:11.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:11.567 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:11.595 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:11.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:11.697 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:11.724 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:11.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:11.739 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 75 16 102923 102907 85.7446 58.75 0.0153494 0.0116575 2024-09-21T10:24:11.826 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:11.851 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:11.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:11.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:11.995 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:11.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:12.038 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:12.054 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:12.055 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:12.055 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:24:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:24:11] "GET /metrics HTTP/1.1" 200 36773 "" "Prometheus/2.51.0" 2024-09-21T10:24:12.097 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:12.136 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:12.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:12.237 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:12.265 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:12.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:12.306 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:24:12.306 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:24:12.306 INFO:teuthology.orchestra.run.smithi129.stdout: "num_ops": 0 2024-09-21T10:24:12.306 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:24:12.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:12.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.406 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:12.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:12.507 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:12.536 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:12.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:12.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:12.688 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:12.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:12.735 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:24:12.742 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 76 16 103853 103837 85.3804 58.125 0.0125433 0.0117068 2024-09-21T10:24:12.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:12.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.837 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:12.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:12.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:12 smithi129 ceph-mon[88471]: pgmap v1483: 109 pgs: 109 active+clean; 6.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 43 MiB/s wr, 690 op/s 2024-09-21T10:24:12.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:12.981 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:24:12.986 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:12.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:13.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:13.090 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:12 smithi107 ceph-mon[102221]: pgmap v1483: 109 pgs: 109 active+clean; 6.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 43 MiB/s wr, 690 op/s 2024-09-21T10:24:13.090 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:12 smithi107 ceph-mon[105022]: pgmap v1483: 109 pgs: 109 active+clean; 6.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 43 MiB/s wr, 690 op/s 2024-09-21T10:24:13.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.147 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:13.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:13.248 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:13.296 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:24:13.298 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:13.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:13.400 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:13.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.428 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:13.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:13.529 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:13.557 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:13.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:13.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:13.701 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:13.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:13.740 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 77 16 104723 104707 84.9777 54.375 0.00956141 0.0117627 2024-09-21T10:24:13.803 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:13.844 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:13.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:13.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:13.984 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:13.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:14.086 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:14.118 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:14.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:14.220 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:14.244 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:14.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:14.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:14.407 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:14.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:14.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:14.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.562 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:14.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:14.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:14.706 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:14.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:14.740 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 78 16 105543 105527 84.5452 51.25 0.02701 0.0118223 2024-09-21T10:24:14.792 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T10:24:14.808 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:14.833 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:14.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:14 smithi129 ceph-mon[88471]: pgmap v1484: 109 pgs: 109 active+clean; 6.3 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 59 MiB/s wr, 938 op/s 2024-09-21T10:24:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:14 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3560071484' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:24:14.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:14 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:24:14.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:14.985 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:14.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:15.017 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:14 smithi107 ceph-mon[102221]: pgmap v1484: 109 pgs: 109 active+clean; 6.3 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 59 MiB/s wr, 938 op/s 2024-09-21T10:24:15.017 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:14 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3560071484' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:24:15.017 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:14 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:24:15.017 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:14 smithi107 ceph-mon[105022]: pgmap v1484: 109 pgs: 109 active+clean; 6.3 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 59 MiB/s wr, 938 op/s 2024-09-21T10:24:15.017 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:14 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3560071484' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:24:15.017 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:14 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:24:15.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:15.134 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:15.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:15.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:15.237 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:24:15.238 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:24:15.238 INFO:teuthology.orchestra.run.smithi129.stdout: "complaint_time": 30, 2024-09-21T10:24:15.238 INFO:teuthology.orchestra.run.smithi129.stdout: "num_blocked_ops": 0 2024-09-21T10:24:15.238 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:24:15.258 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:15.286 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:15.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:15.388 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:15.413 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:15.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:15.515 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:15.557 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:15.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:15.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:15.693 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:15.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:15.740 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 79 16 106385 106369 84.1411 52.625 0.0167669 0.0118797 2024-09-21T10:24:15.795 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:15.833 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:15.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:15.836 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:24:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:15 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:24:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:15 smithi129 ceph-mon[88471]: osdmap e424: 8 total, 8 up, 8 in 2024-09-21T10:24:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:15 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1164226727' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:15.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:15 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:15.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:15.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.975 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:15.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:16.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:16.106 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:15 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:24:16.106 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:15 smithi107 ceph-mon[102221]: osdmap e424: 8 total, 8 up, 8 in 2024-09-21T10:24:16.106 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:15 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1164226727' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:16.106 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:15 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:16.107 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:15 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:24:16.107 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:15 smithi107 ceph-mon[105022]: osdmap e424: 8 total, 8 up, 8 in 2024-09-21T10:24:16.107 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:15 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1164226727' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:16.107 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:15 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:16.107 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T10:24:16.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:24:16.129 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:16.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:16.230 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:16.253 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:16.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:16.353 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:24:16.355 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:16.379 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:16.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:16.481 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:16.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.511 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:16.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:16.612 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:16.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.646 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:16.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:16.702 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:24:16.740 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:2024-09-21T10:24:16.739185+0000 min lat: 0.00106792 max lat: 0.22341 avg lat: 0.0119335 2024-09-21T10:24:16.740 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:24:16.740 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 80 16 107251 107235 83.7659 54.125 0.0156692 0.0119335 2024-09-21T10:24:16.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:16.788 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:16.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:16.890 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:16.919 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:16.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:17.020 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:17.064 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:17.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:17.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:16 smithi107 ceph-mon[102221]: pgmap v1486: 109 pgs: 109 active+clean; 6.3 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:24:17.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:16 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:24:17.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:16 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:24:17.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:16 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:24:17.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:16 smithi107 ceph-mon[102221]: osdmap e425: 8 total, 8 up, 8 in 2024-09-21T10:24:17.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:16 smithi107 ceph-mon[105022]: pgmap v1486: 109 pgs: 109 active+clean; 6.3 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:24:17.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:16 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:24:17.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:16 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:24:17.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:16 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:24:17.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:16 smithi107 ceph-mon[105022]: osdmap e425: 8 total, 8 up, 8 in 2024-09-21T10:24:17.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:16 smithi129 ceph-mon[88471]: pgmap v1486: 109 pgs: 109 active+clean; 6.3 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:24:17.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:16 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:24:17.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:16 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:24:17.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:16 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:24:17.143 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:16 smithi129 ceph-mon[88471]: osdmap e425: 8 total, 8 up, 8 in 2024-09-21T10:24:17.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:17.212 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:17.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:17.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:17.352 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:17.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:17.454 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:17.483 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:17.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:17.585 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:17.612 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:17.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:17.714 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:17.740 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 81 16 108165 108149 83.4369 57.125 0.0252615 0.0119801 2024-09-21T10:24:17.743 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:17.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:17.845 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:17.880 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:17.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:17.981 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:18.008 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:18.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:18.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:18.163 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:18.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:18.166 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:18.264 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:18.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.297 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:18.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:18.399 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:18.432 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:18.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:18.533 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:18.572 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:24:18.572 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 20, 2024-09-21T10:24:18.573 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 600, 2024-09-21T10:24:18.573 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:24:18.573 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.573 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36822.0:42510 12.2 12.c9bf7ea2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-21T10:24:18.573 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:14:29.658375+0000", 2024-09-21T10:24:18.573 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 588.910526807, 2024-09-21T10:24:18.573 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0371481389999999, 2024-09-21T10:24:18.573 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:18.573 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:24:18.573 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:18.573 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36822", 2024-09-21T10:24:18.573 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:24:18.574 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 42510 2024-09-21T10:24:18.574 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.574 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:18.574 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.574 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:18.574 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.658375+0000", 2024-09-21T10:24:18.574 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.574 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.574 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.574 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:18.574 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.658375+0000", 2024-09-21T10:24:18.574 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.574 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.575 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.575 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:18.575 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.658373+0000", 2024-09-21T10:24:18.575 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:24:18.575 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.575 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.575 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:18.575 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.658570+0000", 2024-09-21T10:24:18.575 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00019689100000000001 2024-09-21T10:24:18.575 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.575 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.575 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:18.575 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.658593+0000", 2024-09-21T10:24:18.575 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.2469000000000001e-05 2024-09-21T10:24:18.575 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.576 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.576 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:18.576 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.658600+0000", 2024-09-21T10:24:18.576 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3220000000000004e-06 2024-09-21T10:24:18.576 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.576 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.576 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.576 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.658643+0000", 2024-09-21T10:24:18.576 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2926999999999997e-05 2024-09-21T10:24:18.576 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.576 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.576 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:24:18.576 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.658651+0000", 2024-09-21T10:24:18.576 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.3790000000000006e-06 2024-09-21T10:24:18.577 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.577 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.577 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.577 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:30.695514+0000", 2024-09-21T10:24:18.577 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.036862079 2024-09-21T10:24:18.577 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.577 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.577 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:18.577 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:30.695523+0000", 2024-09-21T10:24:18.577 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.6889999999999999e-06 2024-09-21T10:24:18.577 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.577 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.577 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.577 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.578 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.578 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36822.0:42519 12.2 12.cc518ec2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-21T10:24:18.578 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:14:29.663020+0000", 2024-09-21T10:24:18.578 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 588.90588225800002, 2024-09-21T10:24:18.578 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0325486260000001, 2024-09-21T10:24:18.578 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:18.578 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:24:18.578 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:18.578 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36822", 2024-09-21T10:24:18.578 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:24:18.578 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 42519 2024-09-21T10:24:18.578 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.578 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:18.578 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.579 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:18.579 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.663020+0000", 2024-09-21T10:24:18.579 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.579 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.579 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.579 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:18.579 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.663020+0000", 2024-09-21T10:24:18.579 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.579 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.579 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.579 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:18.579 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.663018+0000", 2024-09-21T10:24:18.579 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:24:18.579 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.579 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.580 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:18.580 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.663095+0000", 2024-09-21T10:24:18.580 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.7149000000000003e-05 2024-09-21T10:24:18.580 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.580 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.580 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:18.580 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.663110+0000", 2024-09-21T10:24:18.580 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4878999999999999e-05 2024-09-21T10:24:18.580 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.580 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.580 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:18.580 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.663115+0000", 2024-09-21T10:24:18.580 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8439999999999997e-06 2024-09-21T10:24:18.580 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.580 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.581 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.581 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.663146+0000", 2024-09-21T10:24:18.581 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0564000000000001e-05 2024-09-21T10:24:18.581 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.581 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.581 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:24:18.581 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.663152+0000", 2024-09-21T10:24:18.581 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8950000000000003e-06 2024-09-21T10:24:18.581 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.581 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.581 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.581 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:30.695557+0000", 2024-09-21T10:24:18.581 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.032405319 2024-09-21T10:24:18.581 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.581 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.581 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:18.582 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:30.695568+0000", 2024-09-21T10:24:18.582 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1243999999999999e-05 2024-09-21T10:24:18.582 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.582 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.582 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.582 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.582 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.582 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36822.0:42528 12.2 12.9972fd12 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-21T10:24:18.582 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:14:29.670460+0000", 2024-09-21T10:24:18.582 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 588.89844167900003, 2024-09-21T10:24:18.582 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.025132401, 2024-09-21T10:24:18.582 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:18.582 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:24:18.582 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:18.583 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36822", 2024-09-21T10:24:18.583 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:24:18.583 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 42528 2024-09-21T10:24:18.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.583 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:18.583 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.583 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:18.583 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.670460+0000", 2024-09-21T10:24:18.583 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.583 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.583 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:18.583 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.670460+0000", 2024-09-21T10:24:18.583 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.584 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.584 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:18.584 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.670458+0000", 2024-09-21T10:24:18.584 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:24:18.584 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.584 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.584 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:18.584 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.670502+0000", 2024-09-21T10:24:18.584 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4203999999999999e-05 2024-09-21T10:24:18.584 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.584 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.584 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:18.584 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.670517+0000", 2024-09-21T10:24:18.584 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4908e-05 2024-09-21T10:24:18.585 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.585 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.585 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:18.585 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.670524+0000", 2024-09-21T10:24:18.585 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8059999999999997e-06 2024-09-21T10:24:18.585 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.585 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.585 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.585 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.670584+0000", 2024-09-21T10:24:18.585 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9768000000000001e-05 2024-09-21T10:24:18.585 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.585 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.585 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:24:18.585 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:29.670592+0000", 2024-09-21T10:24:18.586 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.1130000000000001e-06 2024-09-21T10:24:18.586 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.586 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.586 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.586 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:30.695586+0000", 2024-09-21T10:24:18.586 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.02499356 2024-09-21T10:24:18.586 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.586 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.586 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:18.586 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:14:30.695593+0000", 2024-09-21T10:24:18.587 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9909999999999996e-06 2024-09-21T10:24:18.588 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.588 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.588 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.588 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.588 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.588 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36822.0:132541 12.12 12:4df3150c:::benchmark_data_smithi107_250110_object21131:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e321)", 2024-09-21T10:24:18.588 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:16:06.565723+0000", 2024-09-21T10:24:18.588 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 492.00317886200003, 2024-09-21T10:24:18.588 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.99997737900000006, 2024-09-21T10:24:18.588 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:18.588 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:18.588 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:18.588 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36822", 2024-09-21T10:24:18.588 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:24:18.589 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 132541 2024-09-21T10:24:18.589 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.589 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:18.589 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.589 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:18.589 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:16:06.565723+0000", 2024-09-21T10:24:18.589 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.589 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.589 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.589 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:18.589 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:16:06.565723+0000", 2024-09-21T10:24:18.589 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.589 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.589 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.590 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:18.590 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:16:06.565722+0000", 2024-09-21T10:24:18.590 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:24:18.590 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.590 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.590 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:18.590 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:16:06.565728+0000", 2024-09-21T10:24:18.590 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9819999999999996e-06 2024-09-21T10:24:18.590 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.590 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.590 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:18.590 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:16:06.565744+0000", 2024-09-21T10:24:18.590 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.6549e-05 2024-09-21T10:24:18.591 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.591 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.591 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:18.591 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:16:06.565752+0000", 2024-09-21T10:24:18.591 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2629999999999997e-06 2024-09-21T10:24:18.591 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.591 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.591 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.591 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:16:06.565809+0000", 2024-09-21T10:24:18.591 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7642000000000001e-05 2024-09-21T10:24:18.591 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.591 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.591 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:24:18.591 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:16:06.565822+0000", 2024-09-21T10:24:18.592 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2473000000000001e-05 2024-09-21T10:24:18.592 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.592 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.592 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.592 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:16:07.551744+0000", 2024-09-21T10:24:18.592 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.98592196600000004 2024-09-21T10:24:18.592 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.592 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.592 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:18.592 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:16:07.551871+0000", 2024-09-21T10:24:18.592 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000126923 2024-09-21T10:24:18.592 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.592 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.593 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:24:18.593 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:16:07.551988+0000", 2024-09-21T10:24:18.593 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000117052 2024-09-21T10:24:18.593 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.593 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.593 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:18.593 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:16:07.555898+0000", 2024-09-21T10:24:18.593 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0039102440000000002 2024-09-21T10:24:18.593 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.593 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.593 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.593 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:16:07.565614+0000", 2024-09-21T10:24:18.593 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0097165259999999996 2024-09-21T10:24:18.593 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.594 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.594 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.594 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:16:07.565643+0000", 2024-09-21T10:24:18.594 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.8262000000000001e-05 2024-09-21T10:24:18.594 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.594 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.594 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:18.594 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:16:07.565678+0000", 2024-09-21T10:24:18.594 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5376999999999997e-05 2024-09-21T10:24:18.594 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.594 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.594 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:18.594 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:16:07.565700+0000", 2024-09-21T10:24:18.594 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.2408999999999998e-05 2024-09-21T10:24:18.595 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.595 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.595 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.595 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.595 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.595 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:26956 13.d 13:bd5314ea:::benchmark_data_smithi107_278655_object26955:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:24:18.595 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:31.500265+0000", 2024-09-21T10:24:18.595 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 347.06863666100003, 2024-09-21T10:24:18.595 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.29662678999999997, 2024-09-21T10:24:18.595 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:18.595 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:18.595 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:18.596 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:18.596 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:18.596 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 26956 2024-09-21T10:24:18.596 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.596 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:18.596 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.596 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:18.596 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.500265+0000", 2024-09-21T10:24:18.596 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.596 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.596 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.596 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:18.596 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.500265+0000", 2024-09-21T10:24:18.596 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.596 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.597 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.597 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:18.597 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.500264+0000", 2024-09-21T10:24:18.597 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:24:18.597 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.597 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.597 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:18.597 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.500344+0000", 2024-09-21T10:24:18.597 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9652000000000002e-05 2024-09-21T10:24:18.597 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.597 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.597 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:18.597 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.500357+0000", 2024-09-21T10:24:18.597 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3478e-05 2024-09-21T10:24:18.598 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.598 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.598 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:18.598 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.501187+0000", 2024-09-21T10:24:18.598 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00082909100000000001 2024-09-21T10:24:18.598 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.598 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.598 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.598 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.501716+0000", 2024-09-21T10:24:18.598 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00052933500000000003 2024-09-21T10:24:18.598 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.598 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.598 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:18.598 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.502950+0000", 2024-09-21T10:24:18.599 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001233956 2024-09-21T10:24:18.599 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.599 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.599 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,7", 2024-09-21T10:24:18.599 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.503103+0000", 2024-09-21T10:24:18.599 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015349100000000001 2024-09-21T10:24:18.599 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.599 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.599 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.599 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.506935+0000", 2024-09-21T10:24:18.599 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0038317970000000001 2024-09-21T10:24:18.599 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.599 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.599 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:18.599 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.507019+0000", 2024-09-21T10:24:18.600 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.3638000000000001e-05 2024-09-21T10:24:18.600 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.600 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.600 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.600 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.796834+0000", 2024-09-21T10:24:18.600 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28981557800000002 2024-09-21T10:24:18.600 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.600 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.600 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:18.600 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.796868+0000", 2024-09-21T10:24:18.600 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3315000000000002e-05 2024-09-21T10:24:18.600 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.600 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.600 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:18.600 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.796892+0000", 2024-09-21T10:24:18.601 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.4389e-05 2024-09-21T10:24:18.601 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.601 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.601 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.601 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.601 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.601 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:26967 13.d 13:bbe0f6e9:::benchmark_data_smithi107_278655_object26966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:24:18.601 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:31.507873+0000", 2024-09-21T10:24:18.601 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 347.06102897, 2024-09-21T10:24:18.601 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28977464600000002, 2024-09-21T10:24:18.601 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:18.601 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:18.601 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:18.601 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:18.602 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:18.602 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 26967 2024-09-21T10:24:18.602 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.602 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:18.602 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.602 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:18.602 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.507873+0000", 2024-09-21T10:24:18.602 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.602 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.602 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.602 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:18.602 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.507873+0000", 2024-09-21T10:24:18.602 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.603 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.603 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:18.603 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.507871+0000", 2024-09-21T10:24:18.603 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:24:18.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.603 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.603 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:18.603 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.507905+0000", 2024-09-21T10:24:18.603 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3399999999999999e-05 2024-09-21T10:24:18.603 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.603 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.603 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:18.603 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.507920+0000", 2024-09-21T10:24:18.604 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5605999999999999e-05 2024-09-21T10:24:18.604 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.604 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.604 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:18.604 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.507927+0000", 2024-09-21T10:24:18.604 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6560000000000003e-06 2024-09-21T10:24:18.604 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.604 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.604 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.604 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.507974+0000", 2024-09-21T10:24:18.604 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7207999999999999e-05 2024-09-21T10:24:18.604 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.604 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.604 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:18.604 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.508069+0000", 2024-09-21T10:24:18.605 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.5157999999999999e-05 2024-09-21T10:24:18.605 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.605 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.605 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,7", 2024-09-21T10:24:18.605 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.508211+0000", 2024-09-21T10:24:18.605 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000141237 2024-09-21T10:24:18.605 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.605 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.605 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.605 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.512456+0000", 2024-09-21T10:24:18.605 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0042457780000000004 2024-09-21T10:24:18.605 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.605 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.605 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:18.606 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.513595+0000", 2024-09-21T10:24:18.606 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001138661 2024-09-21T10:24:18.606 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.606 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.606 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.606 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.797599+0000", 2024-09-21T10:24:18.606 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28400375300000003 2024-09-21T10:24:18.606 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.606 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.606 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:18.606 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.797629+0000", 2024-09-21T10:24:18.606 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0133000000000001e-05 2024-09-21T10:24:18.606 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.607 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.607 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:18.607 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.797648+0000", 2024-09-21T10:24:18.607 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.8675999999999999e-05 2024-09-21T10:24:18.607 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.607 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.607 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.607 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.607 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.607 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:26969 13.d 13:bd4dc0b4:::benchmark_data_smithi107_278655_object26968:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:24:18.607 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:31.509533+0000", 2024-09-21T10:24:18.607 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 347.05936867200001, 2024-09-21T10:24:18.607 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28817326500000001, 2024-09-21T10:24:18.607 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:18.608 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:18.608 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:18.608 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:18.608 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:18.608 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 26969 2024-09-21T10:24:18.608 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.608 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:18.608 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.608 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:18.608 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.509533+0000", 2024-09-21T10:24:18.608 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.608 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.608 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.608 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:18.608 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.509533+0000", 2024-09-21T10:24:18.609 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.609 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.609 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.609 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:18.609 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.509531+0000", 2024-09-21T10:24:18.609 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:24:18.609 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.609 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.609 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:18.609 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.509577+0000", 2024-09-21T10:24:18.609 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5785999999999998e-05 2024-09-21T10:24:18.609 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.609 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.610 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:18.610 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.509590+0000", 2024-09-21T10:24:18.610 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3097e-05 2024-09-21T10:24:18.610 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.610 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.610 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:18.610 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.509597+0000", 2024-09-21T10:24:18.610 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3210000000000001e-06 2024-09-21T10:24:18.610 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.610 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.610 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.610 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.510617+0000", 2024-09-21T10:24:18.610 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001020892 2024-09-21T10:24:18.610 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.611 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.611 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:18.611 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.510736+0000", 2024-09-21T10:24:18.611 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011809 2024-09-21T10:24:18.611 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.611 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.611 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,7", 2024-09-21T10:24:18.611 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.510886+0000", 2024-09-21T10:24:18.611 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015063199999999999 2024-09-21T10:24:18.611 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.611 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.611 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:18.611 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.513601+0000", 2024-09-21T10:24:18.611 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0027145260000000001 2024-09-21T10:24:18.611 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.612 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.612 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.612 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.514624+0000", 2024-09-21T10:24:18.612 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0010236609999999999 2024-09-21T10:24:18.612 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.612 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.612 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.612 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.797668+0000", 2024-09-21T10:24:18.612 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28304337800000001 2024-09-21T10:24:18.612 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.612 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.612 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:18.612 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.797690+0000", 2024-09-21T10:24:18.612 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1895e-05 2024-09-21T10:24:18.613 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.613 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.613 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:18.613 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.797706+0000", 2024-09-21T10:24:18.613 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.6844999999999999e-05 2024-09-21T10:24:18.613 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.613 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.613 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.613 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.613 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.613 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:26974 13.d 13:b6815439:::benchmark_data_smithi107_278655_object26973:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:24:18.613 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:31.510417+0000", 2024-09-21T10:24:18.613 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 347.05848437700001, 2024-09-21T10:24:18.613 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28939799799999999, 2024-09-21T10:24:18.613 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:18.614 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:18.614 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:18.614 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:18.614 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:18.614 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 26974 2024-09-21T10:24:18.614 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.614 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:18.614 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.614 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:18.614 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.510417+0000", 2024-09-21T10:24:18.614 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.614 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.614 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.615 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:18.615 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.510417+0000", 2024-09-21T10:24:18.615 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.615 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.615 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.615 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:18.615 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.510416+0000", 2024-09-21T10:24:18.615 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:24:18.615 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.615 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.615 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:18.615 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.510546+0000", 2024-09-21T10:24:18.615 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012949899999999999 2024-09-21T10:24:18.615 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.616 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.616 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:18.616 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.510649+0000", 2024-09-21T10:24:18.616 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000103317 2024-09-21T10:24:18.616 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.616 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.616 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:18.616 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.510658+0000", 2024-09-21T10:24:18.616 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.3899999999999993e-06 2024-09-21T10:24:18.616 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.616 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.616 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.616 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.512480+0000", 2024-09-21T10:24:18.616 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0018222170000000001 2024-09-21T10:24:18.617 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.617 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.617 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:18.617 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.512578+0000", 2024-09-21T10:24:18.617 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.8188000000000002e-05 2024-09-21T10:24:18.617 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.617 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.617 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,7", 2024-09-21T10:24:18.617 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.512695+0000", 2024-09-21T10:24:18.617 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000116772 2024-09-21T10:24:18.617 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.617 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.617 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:18.617 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.513907+0000", 2024-09-21T10:24:18.618 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0012121 2024-09-21T10:24:18.618 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.618 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.618 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.618 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.514721+0000", 2024-09-21T10:24:18.618 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00081401499999999996 2024-09-21T10:24:18.618 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.618 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.618 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.618 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.799477+0000", 2024-09-21T10:24:18.618 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28475630099999999 2024-09-21T10:24:18.618 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.618 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.618 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:18.618 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.799523+0000", 2024-09-21T10:24:18.619 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5825000000000003e-05 2024-09-21T10:24:18.619 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.619 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.619 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:18.619 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.799815+0000", 2024-09-21T10:24:18.619 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00029237799999999998 2024-09-21T10:24:18.619 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.619 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.619 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.619 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.619 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.619 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:26981 13.d 13:b8ebef98:::benchmark_data_smithi107_278655_object26980:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:24:18.619 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:31.515243+0000", 2024-09-21T10:24:18.619 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 347.05365888199998, 2024-09-21T10:24:18.620 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28469607299999999, 2024-09-21T10:24:18.620 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:18.620 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:18.620 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:18.620 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:18.620 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:18.620 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 26981 2024-09-21T10:24:18.620 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.620 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:18.620 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.620 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:18.620 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.515243+0000", 2024-09-21T10:24:18.620 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.621 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.621 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.621 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:18.621 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.515243+0000", 2024-09-21T10:24:18.621 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.621 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.621 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.621 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:18.621 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.515242+0000", 2024-09-21T10:24:18.621 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:24:18.621 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.621 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.621 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:18.621 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.515303+0000", 2024-09-21T10:24:18.621 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0236999999999999e-05 2024-09-21T10:24:18.622 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.622 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.622 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:18.622 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.515314+0000", 2024-09-21T10:24:18.622 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1790000000000001e-05 2024-09-21T10:24:18.622 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.622 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.622 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:18.622 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.515320+0000", 2024-09-21T10:24:18.622 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4790000000000004e-06 2024-09-21T10:24:18.622 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.622 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.622 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.622 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.515441+0000", 2024-09-21T10:24:18.623 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000120829 2024-09-21T10:24:18.623 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.623 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.623 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:18.623 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.515525+0000", 2024-09-21T10:24:18.623 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.4447e-05 2024-09-21T10:24:18.623 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.623 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.623 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,7", 2024-09-21T10:24:18.623 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.515600+0000", 2024-09-21T10:24:18.623 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.5296000000000006e-05 2024-09-21T10:24:18.623 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.623 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.623 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:18.624 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.517008+0000", 2024-09-21T10:24:18.624 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0014072729999999999 2024-09-21T10:24:18.624 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.624 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.624 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.624 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.518079+0000", 2024-09-21T10:24:18.624 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0010713579999999999 2024-09-21T10:24:18.624 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.624 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.624 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.624 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.799849+0000", 2024-09-21T10:24:18.624 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28177011800000001 2024-09-21T10:24:18.624 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.624 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.624 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:18.625 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.799907+0000", 2024-09-21T10:24:18.625 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7875999999999999e-05 2024-09-21T10:24:18.625 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.625 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.625 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:18.625 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.799939+0000", 2024-09-21T10:24:18.625 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1968999999999998e-05 2024-09-21T10:24:18.625 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.625 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.625 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.625 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.625 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.625 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:137698 13.6 13:68f18825:::benchmark_data_smithi107_278655_object9791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-21T10:24:18.625 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:04.261146+0000", 2024-09-21T10:24:18.626 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 254.30775591899999, 2024-09-21T10:24:18.626 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.16119175799999999, 2024-09-21T10:24:18.626 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:18.626 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:18.626 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:18.626 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:18.626 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:18.626 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 137698 2024-09-21T10:24:18.626 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.626 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:18.626 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.626 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:18.626 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.261146+0000", 2024-09-21T10:24:18.626 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.626 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.627 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.627 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:18.627 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.261146+0000", 2024-09-21T10:24:18.627 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.627 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.627 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.627 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:18.627 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.261145+0000", 2024-09-21T10:24:18.627 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:24:18.627 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.627 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.627 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:18.627 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.261149+0000", 2024-09-21T10:24:18.627 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1670000000000001e-06 2024-09-21T10:24:18.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.628 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:18.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:18.630 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.630 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.630 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:18.630 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.261169+0000", 2024-09-21T10:24:18.630 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.98e-05 2024-09-21T10:24:18.630 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.630 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.630 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:18.630 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.261175+0000", 2024-09-21T10:24:18.630 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9549999999999997e-06 2024-09-21T10:24:18.630 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.630 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.631 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.631 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.265547+0000", 2024-09-21T10:24:18.631 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0043717720000000003 2024-09-21T10:24:18.631 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.631 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.631 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:18.631 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.265671+0000", 2024-09-21T10:24:18.631 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012366699999999999 2024-09-21T10:24:18.631 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.631 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.631 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:24:18.631 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.265788+0000", 2024-09-21T10:24:18.631 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011778299999999999 2024-09-21T10:24:18.632 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.632 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.632 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:18.632 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.268654+0000", 2024-09-21T10:24:18.632 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0028651890000000002 2024-09-21T10:24:18.632 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.632 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.632 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.632 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.276729+0000", 2024-09-21T10:24:18.632 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0080756720000000008 2024-09-21T10:24:18.632 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.632 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.632 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.632 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.422154+0000", 2024-09-21T10:24:18.633 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.14542430100000001 2024-09-21T10:24:18.633 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.633 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.633 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:18.633 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.422292+0000", 2024-09-21T10:24:18.633 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013814799999999999 2024-09-21T10:24:18.633 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.633 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.633 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:18.633 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.422338+0000", 2024-09-21T10:24:18.633 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6031e-05 2024-09-21T10:24:18.633 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.633 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.633 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.634 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.634 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.634 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:137699 13.6 13:6261f82e:::benchmark_data_smithi107_278655_object9792:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-21T10:24:18.634 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:04.261187+0000", 2024-09-21T10:24:18.634 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 254.30771484799999, 2024-09-21T10:24:18.634 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.16128873299999999, 2024-09-21T10:24:18.634 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:18.634 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:18.634 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:18.634 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:18.634 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:18.634 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 137699 2024-09-21T10:24:18.634 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.635 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:18.635 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.635 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:18.635 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.261187+0000", 2024-09-21T10:24:18.635 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.635 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.635 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.635 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:18.635 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.261187+0000", 2024-09-21T10:24:18.635 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.635 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.635 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.635 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:18.635 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.261186+0000", 2024-09-21T10:24:18.636 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:24:18.636 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.636 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.636 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:18.636 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.261190+0000", 2024-09-21T10:24:18.636 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1379999999999999e-06 2024-09-21T10:24:18.636 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.636 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.636 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:18.636 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.261200+0000", 2024-09-21T10:24:18.636 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.029e-05 2024-09-21T10:24:18.636 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.636 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.636 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:18.637 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.261204+0000", 2024-09-21T10:24:18.637 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5059999999999997e-06 2024-09-21T10:24:18.637 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.637 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.637 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.637 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.268986+0000", 2024-09-21T10:24:18.637 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0077817880000000004 2024-09-21T10:24:18.637 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.637 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.637 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:18.637 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.269112+0000", 2024-09-21T10:24:18.637 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000126062 2024-09-21T10:24:18.637 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.637 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.638 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:24:18.638 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.269255+0000", 2024-09-21T10:24:18.638 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000142528 2024-09-21T10:24:18.638 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.638 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.638 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:18.638 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.272379+0000", 2024-09-21T10:24:18.638 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0031243130000000001 2024-09-21T10:24:18.638 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.638 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.638 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.638 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.276935+0000", 2024-09-21T10:24:18.638 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0045562299999999997 2024-09-21T10:24:18.638 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.639 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.639 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.639 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.422375+0000", 2024-09-21T10:24:18.639 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.14544003799999999 2024-09-21T10:24:18.639 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.639 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.639 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:18.639 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.422443+0000", 2024-09-21T10:24:18.639 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7259000000000001e-05 2024-09-21T10:24:18.639 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.639 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.639 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:18.639 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:04.422476+0000", 2024-09-21T10:24:18.639 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3158999999999999e-05 2024-09-21T10:24:18.640 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.640 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.640 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.640 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.640 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.640 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:174613 13.6 13:644b7fdf:::benchmark_data_smithi107_278655_object46706:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-21T10:24:18.640 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:46.287516+0000", 2024-09-21T10:24:18.640 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 212.28138634600001, 2024-09-21T10:24:18.640 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.18132815699999999, 2024-09-21T10:24:18.640 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:18.640 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:18.640 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:18.640 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:18.640 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:18.641 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 174613 2024-09-21T10:24:18.641 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.641 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:18.641 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.641 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:18.641 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.287516+0000", 2024-09-21T10:24:18.641 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.641 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.641 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.641 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:18.641 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.287516+0000", 2024-09-21T10:24:18.641 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.641 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.641 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.642 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:18.642 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.287515+0000", 2024-09-21T10:24:18.642 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:24:18.642 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.642 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.642 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:18.642 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.287518+0000", 2024-09-21T10:24:18.642 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.9289999999999998e-06 2024-09-21T10:24:18.642 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.642 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.642 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:18.642 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.287524+0000", 2024-09-21T10:24:18.642 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0229999999999998e-06 2024-09-21T10:24:18.642 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.643 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.643 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:18.643 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.287528+0000", 2024-09-21T10:24:18.643 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8720000000000004e-06 2024-09-21T10:24:18.643 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.643 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.643 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.643 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.290860+0000", 2024-09-21T10:24:18.643 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0033325799999999999 2024-09-21T10:24:18.643 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.643 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.643 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:18.643 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.290968+0000", 2024-09-21T10:24:18.644 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000107441 2024-09-21T10:24:18.644 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.644 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.644 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:24:18.644 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.291125+0000", 2024-09-21T10:24:18.644 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015701199999999999 2024-09-21T10:24:18.644 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.644 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.644 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:18.644 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.295216+0000", 2024-09-21T10:24:18.644 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.004090771 2024-09-21T10:24:18.644 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.644 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.644 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.645 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.295247+0000", 2024-09-21T10:24:18.645 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1111000000000001e-05 2024-09-21T10:24:18.645 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.645 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.645 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.645 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468708+0000", 2024-09-21T10:24:18.645 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.17346099800000001 2024-09-21T10:24:18.645 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.645 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.645 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:18.645 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468798+0000", 2024-09-21T10:24:18.645 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.9849e-05 2024-09-21T10:24:18.645 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.646 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.646 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:18.646 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468844+0000", 2024-09-21T10:24:18.646 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6069999999999998e-05 2024-09-21T10:24:18.646 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.646 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.646 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.646 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.646 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.646 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:174621 13.5 13:ab2f47f8:::benchmark_data_smithi107_278655_object46714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-21T10:24:18.646 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:46.303003+0000", 2024-09-21T10:24:18.646 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 212.26589841200001, 2024-09-21T10:24:18.646 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.16895734300000001, 2024-09-21T10:24:18.646 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:18.647 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:18.647 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:18.647 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:18.647 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:18.647 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 174621 2024-09-21T10:24:18.647 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.647 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:18.647 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.647 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:18.647 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303003+0000", 2024-09-21T10:24:18.647 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.647 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.647 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.647 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:18.648 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303003+0000", 2024-09-21T10:24:18.648 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.648 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.648 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.648 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:18.648 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303002+0000", 2024-09-21T10:24:18.648 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:24:18.648 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.648 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.648 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:18.648 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303014+0000", 2024-09-21T10:24:18.648 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2323e-05 2024-09-21T10:24:18.648 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.648 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.649 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:18.649 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303026+0000", 2024-09-21T10:24:18.649 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2465e-05 2024-09-21T10:24:18.649 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.649 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.649 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:18.649 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303032+0000", 2024-09-21T10:24:18.649 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.015e-06 2024-09-21T10:24:18.649 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.649 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.649 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.649 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303117+0000", 2024-09-21T10:24:18.649 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.4678000000000005e-05 2024-09-21T10:24:18.649 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.650 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.650 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:18.650 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303227+0000", 2024-09-21T10:24:18.650 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000109898 2024-09-21T10:24:18.650 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.650 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.650 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:24:18.650 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303334+0000", 2024-09-21T10:24:18.650 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000107386 2024-09-21T10:24:18.650 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.650 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.650 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:18.650 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.306686+0000", 2024-09-21T10:24:18.650 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003351845 2024-09-21T10:24:18.651 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.651 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.651 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.651 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.306839+0000", 2024-09-21T10:24:18.651 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015287699999999999 2024-09-21T10:24:18.651 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.651 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.651 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.651 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.471872+0000", 2024-09-21T10:24:18.651 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.165033294 2024-09-21T10:24:18.652 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.652 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.652 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:18.652 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.471923+0000", 2024-09-21T10:24:18.652 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0623e-05 2024-09-21T10:24:18.652 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.652 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.652 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:18.652 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.471961+0000", 2024-09-21T10:24:18.652 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7778000000000003e-05 2024-09-21T10:24:18.652 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.652 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.652 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.652 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.652 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.653 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:174635 13.6 13:6b286960:::benchmark_data_smithi107_278655_object46728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-21T10:24:18.653 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:46.312268+0000", 2024-09-21T10:24:18.653 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 212.25663357900001, 2024-09-21T10:24:18.653 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.15997352000000001, 2024-09-21T10:24:18.653 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:18.653 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:18.653 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:18.653 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:18.653 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:18.653 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 174635 2024-09-21T10:24:18.653 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.653 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:18.653 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.653 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:18.654 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.312268+0000", 2024-09-21T10:24:18.654 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.654 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.654 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.654 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:18.654 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.312268+0000", 2024-09-21T10:24:18.654 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.654 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.654 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.654 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:18.654 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.312267+0000", 2024-09-21T10:24:18.654 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:24:18.654 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.655 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.655 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:18.655 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.312274+0000", 2024-09-21T10:24:18.655 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9879999999999998e-06 2024-09-21T10:24:18.655 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.655 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.655 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:18.655 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.312285+0000", 2024-09-21T10:24:18.655 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1472e-05 2024-09-21T10:24:18.655 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.655 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.655 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:18.655 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.312291+0000", 2024-09-21T10:24:18.655 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9089999999999997e-06 2024-09-21T10:24:18.656 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.656 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.656 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.656 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.312342+0000", 2024-09-21T10:24:18.656 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0553999999999997e-05 2024-09-21T10:24:18.656 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.656 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.656 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:18.656 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.312462+0000", 2024-09-21T10:24:18.656 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000120567 2024-09-21T10:24:18.656 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.656 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.656 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:24:18.656 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.312576+0000", 2024-09-21T10:24:18.657 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000114175 2024-09-21T10:24:18.657 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.657 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.657 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:18.657 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.315529+0000", 2024-09-21T10:24:18.657 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0029525240000000002 2024-09-21T10:24:18.657 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.657 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.657 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.657 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.316334+0000", 2024-09-21T10:24:18.657 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00080524399999999999 2024-09-21T10:24:18.657 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.657 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.658 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.658 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.472163+0000", 2024-09-21T10:24:18.658 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.155829196 2024-09-21T10:24:18.658 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.658 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.658 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:18.658 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.472207+0000", 2024-09-21T10:24:18.658 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3751000000000002e-05 2024-09-21T10:24:18.658 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.658 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.658 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:18.658 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.472242+0000", 2024-09-21T10:24:18.658 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4703000000000003e-05 2024-09-21T10:24:18.658 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.659 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.659 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.659 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.659 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.659 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:216805 13.5 13:a98e27c6:::benchmark_data_smithi107_278655_object88898:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T10:24:18.659 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:21:33.650817+0000", 2024-09-21T10:24:18.659 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 164.91808449199999, 2024-09-21T10:24:18.659 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.296630371, 2024-09-21T10:24:18.659 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:18.659 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:18.659 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:18.659 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:18.659 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:18.659 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 216805 2024-09-21T10:24:18.659 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.660 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:18.660 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.660 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:18.660 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.650817+0000", 2024-09-21T10:24:18.660 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.660 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.660 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.660 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:18.660 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.650817+0000", 2024-09-21T10:24:18.660 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.660 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.660 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.660 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:18.660 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.650816+0000", 2024-09-21T10:24:18.661 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:24:18.661 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.661 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.661 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:18.661 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.650822+0000", 2024-09-21T10:24:18.661 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5010000000000001e-06 2024-09-21T10:24:18.661 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.661 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.661 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:18.661 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.650835+0000", 2024-09-21T10:24:18.661 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2646999999999999e-05 2024-09-21T10:24:18.661 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.661 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.662 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:18.662 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.650841+0000", 2024-09-21T10:24:18.662 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.925e-06 2024-09-21T10:24:18.662 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.662 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.662 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.662 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.651904+0000", 2024-09-21T10:24:18.662 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001063469 2024-09-21T10:24:18.662 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.662 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.662 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:18.662 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.652011+0000", 2024-09-21T10:24:18.663 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000106272 2024-09-21T10:24:18.663 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.663 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.663 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:24:18.663 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.652130+0000", 2024-09-21T10:24:18.663 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011948299999999999 2024-09-21T10:24:18.663 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.663 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.663 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.663 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.657883+0000", 2024-09-21T10:24:18.663 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0057525110000000001 2024-09-21T10:24:18.663 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.663 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.664 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:18.664 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.657898+0000", 2024-09-21T10:24:18.664 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4976e-05 2024-09-21T10:24:18.664 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.664 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.664 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.664 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.947343+0000", 2024-09-21T10:24:18.664 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28944531200000001 2024-09-21T10:24:18.664 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.664 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.664 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:18.664 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.947399+0000", 2024-09-21T10:24:18.664 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5498999999999998e-05 2024-09-21T10:24:18.664 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.665 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.665 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:18.665 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.947448+0000", 2024-09-21T10:24:18.665 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9206999999999998e-05 2024-09-21T10:24:18.665 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.665 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.665 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.665 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.665 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.665 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:216807 13.5 13:a7a54d27:::benchmark_data_smithi107_278655_object88900:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T10:24:18.665 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:21:33.651449+0000", 2024-09-21T10:24:18.665 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 164.91745295800001, 2024-09-21T10:24:18.665 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.296311936, 2024-09-21T10:24:18.665 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:18.666 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:18.666 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:18.666 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:18.666 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:18.666 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 216807 2024-09-21T10:24:18.666 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.666 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:18.666 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.666 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:18.666 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.651449+0000", 2024-09-21T10:24:18.666 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.666 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.666 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.666 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:18.667 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.651449+0000", 2024-09-21T10:24:18.667 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.667 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.667 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.667 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:18.667 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.651448+0000", 2024-09-21T10:24:18.667 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:24:18.667 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.667 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.667 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:18.667 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.651453+0000", 2024-09-21T10:24:18.667 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8930000000000004e-06 2024-09-21T10:24:18.667 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.667 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.667 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:18.668 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.651466+0000", 2024-09-21T10:24:18.668 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3044000000000001e-05 2024-09-21T10:24:18.668 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.668 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.668 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:18.668 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.651472+0000", 2024-09-21T10:24:18.668 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4180000000000002e-06 2024-09-21T10:24:18.668 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.668 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.668 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.668 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.654852+0000", 2024-09-21T10:24:18.668 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0033795729999999999 2024-09-21T10:24:18.668 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.668 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.669 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:18.669 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.654957+0000", 2024-09-21T10:24:18.669 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010497000000000001 2024-09-21T10:24:18.669 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.669 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.669 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:24:18.669 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.655067+0000", 2024-09-21T10:24:18.669 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010964 2024-09-21T10:24:18.669 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.669 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.669 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:18.669 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.658342+0000", 2024-09-21T10:24:18.669 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0032757960000000001 2024-09-21T10:24:18.669 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.670 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.670 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.670 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.661060+0000", 2024-09-21T10:24:18.670 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0027180030000000001 2024-09-21T10:24:18.670 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.670 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.670 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.670 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.947590+0000", 2024-09-21T10:24:18.670 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28652943600000003 2024-09-21T10:24:18.670 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.670 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.670 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:18.671 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.947709+0000", 2024-09-21T10:24:18.671 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011957500000000001 2024-09-21T10:24:18.671 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.671 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.671 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:18.671 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.947761+0000", 2024-09-21T10:24:18.671 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1412000000000002e-05 2024-09-21T10:24:18.671 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.671 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.671 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.671 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.671 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.671 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:216819 13.5 13:ae6306bd:::benchmark_data_smithi107_278655_object88912:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T10:24:18.671 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:21:33.660954+0000", 2024-09-21T10:24:18.672 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 164.90794786399999, 2024-09-21T10:24:18.672 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28693384399999999, 2024-09-21T10:24:18.672 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:18.672 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:18.672 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:18.672 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:18.672 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:18.672 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 216819 2024-09-21T10:24:18.672 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.672 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:18.672 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.672 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:18.672 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.660954+0000", 2024-09-21T10:24:18.672 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.673 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.673 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.673 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:18.673 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.660954+0000", 2024-09-21T10:24:18.673 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.673 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.673 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.673 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:18.673 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.660953+0000", 2024-09-21T10:24:18.673 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:24:18.673 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.673 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.673 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:18.673 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.660958+0000", 2024-09-21T10:24:18.674 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8559999999999998e-06 2024-09-21T10:24:18.674 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.674 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.674 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:18.674 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.660968+0000", 2024-09-21T10:24:18.674 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.859e-06 2024-09-21T10:24:18.674 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.674 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.674 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:18.674 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.660972+0000", 2024-09-21T10:24:18.674 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4290000000000003e-06 2024-09-21T10:24:18.674 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.674 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.675 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.675 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.661081+0000", 2024-09-21T10:24:18.675 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000108513 2024-09-21T10:24:18.675 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.675 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.675 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:18.675 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.661217+0000", 2024-09-21T10:24:18.675 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013665800000000001 2024-09-21T10:24:18.675 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.675 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.675 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:24:18.675 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.661336+0000", 2024-09-21T10:24:18.675 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011879099999999999 2024-09-21T10:24:18.675 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.676 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.676 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:18.676 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.664332+0000", 2024-09-21T10:24:18.676 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0029958889999999998 2024-09-21T10:24:18.676 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.676 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.676 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.676 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.664513+0000", 2024-09-21T10:24:18.676 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00018089500000000001 2024-09-21T10:24:18.676 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.676 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.676 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.676 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.947804+0000", 2024-09-21T10:24:18.676 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28329080499999998 2024-09-21T10:24:18.676 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.677 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.677 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:18.677 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.947855+0000", 2024-09-21T10:24:18.677 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1479e-05 2024-09-21T10:24:18.677 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.677 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.677 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:18.677 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.947888+0000", 2024-09-21T10:24:18.677 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2709999999999997e-05 2024-09-21T10:24:18.677 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.677 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.677 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.677 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.677 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.677 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:216825 13.5 13:a837b2e0:::benchmark_data_smithi107_278655_object88918:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T10:24:18.678 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:21:33.666107+0000", 2024-09-21T10:24:18.678 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 164.90279526, 2024-09-21T10:24:18.678 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28505344399999999, 2024-09-21T10:24:18.678 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:18.678 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:18.678 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:18.678 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:18.678 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:18.678 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 216825 2024-09-21T10:24:18.678 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.678 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:18.678 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.678 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:18.678 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.666107+0000", 2024-09-21T10:24:18.679 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.679 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.679 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.679 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:18.679 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.666107+0000", 2024-09-21T10:24:18.679 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.679 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.679 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.679 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:18.679 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.666106+0000", 2024-09-21T10:24:18.679 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:24:18.679 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.679 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.679 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:18.680 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.666109+0000", 2024-09-21T10:24:18.680 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4680000000000001e-06 2024-09-21T10:24:18.680 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.680 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.680 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:18.680 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.666120+0000", 2024-09-21T10:24:18.680 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0781999999999999e-05 2024-09-21T10:24:18.680 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.680 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.680 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:18.680 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.666125+0000", 2024-09-21T10:24:18.680 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4000000000000002e-06 2024-09-21T10:24:18.680 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.680 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.681 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.681 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.666164+0000", 2024-09-21T10:24:18.681 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.9527999999999998e-05 2024-09-21T10:24:18.681 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.681 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.681 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:18.681 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.666326+0000", 2024-09-21T10:24:18.681 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00016215000000000001 2024-09-21T10:24:18.681 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.681 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.681 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:24:18.681 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.666430+0000", 2024-09-21T10:24:18.681 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010415499999999999 2024-09-21T10:24:18.681 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.681 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.682 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.682 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.670898+0000", 2024-09-21T10:24:18.682 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0044672870000000003 2024-09-21T10:24:18.682 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.682 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.682 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:18.682 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.671171+0000", 2024-09-21T10:24:18.682 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00027320900000000001 2024-09-21T10:24:18.682 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.682 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.682 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.682 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.951084+0000", 2024-09-21T10:24:18.682 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.27991285199999999 2024-09-21T10:24:18.682 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.682 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.683 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:18.683 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.951125+0000", 2024-09-21T10:24:18.683 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0692000000000003e-05 2024-09-21T10:24:18.683 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.683 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.683 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:18.683 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.951160+0000", 2024-09-21T10:24:18.683 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5525999999999998e-05 2024-09-21T10:24:18.683 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.683 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.683 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.683 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.683 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.683 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:216837 13.5 13:a6449060:::benchmark_data_smithi107_278655_object88930:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T10:24:18.684 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:21:33.681334+0000", 2024-09-21T10:24:18.684 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 164.887567649, 2024-09-21T10:24:18.684 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.26991779199999999, 2024-09-21T10:24:18.684 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:18.684 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:18.684 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:18.684 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:18.684 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:18.684 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 216837 2024-09-21T10:24:18.684 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.684 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:18.684 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.684 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:18.685 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.681334+0000", 2024-09-21T10:24:18.685 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.685 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.685 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.685 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:18.685 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.681334+0000", 2024-09-21T10:24:18.685 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.685 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.685 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.685 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:18.685 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.681333+0000", 2024-09-21T10:24:18.685 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:24:18.685 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.685 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.685 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:18.686 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.681340+0000", 2024-09-21T10:24:18.686 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.0160000000000003e-06 2024-09-21T10:24:18.686 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.686 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.686 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:18.686 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.681354+0000", 2024-09-21T10:24:18.686 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4632e-05 2024-09-21T10:24:18.686 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.686 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.686 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:18.686 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.681360+0000", 2024-09-21T10:24:18.686 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9370000000000001e-06 2024-09-21T10:24:18.686 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.686 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.686 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.686 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.681400+0000", 2024-09-21T10:24:18.687 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.9702e-05 2024-09-21T10:24:18.687 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.687 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.687 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:18.687 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.681496+0000", 2024-09-21T10:24:18.687 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.5669999999999997e-05 2024-09-21T10:24:18.687 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.687 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.687 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:24:18.687 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.681589+0000", 2024-09-21T10:24:18.687 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.3090999999999994e-05 2024-09-21T10:24:18.687 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.687 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.687 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:18.687 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.684403+0000", 2024-09-21T10:24:18.688 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0028146180000000001 2024-09-21T10:24:18.688 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.688 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.688 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.688 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.684810+0000", 2024-09-21T10:24:18.688 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00040696200000000002 2024-09-21T10:24:18.688 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.688 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.688 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.688 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.951189+0000", 2024-09-21T10:24:18.688 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.26637907599999999 2024-09-21T10:24:18.688 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.688 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.688 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:18.688 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.951223+0000", 2024-09-21T10:24:18.688 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3339e-05 2024-09-21T10:24:18.689 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.689 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.689 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:18.689 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:33.951252+0000", 2024-09-21T10:24:18.689 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.9269e-05 2024-09-21T10:24:18.689 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.689 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.689 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.689 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.689 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.689 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:216964 13.d 13:b191c665:::benchmark_data_smithi107_278655_object89057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T10:24:18.689 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:21:34.030684+0000", 2024-09-21T10:24:18.689 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 164.538218097, 2024-09-21T10:24:18.689 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.243808786, 2024-09-21T10:24:18.689 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:18.689 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:18.690 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:18.690 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:18.690 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:18.690 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 216964 2024-09-21T10:24:18.690 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.690 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:18.690 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.690 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:18.690 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.030684+0000", 2024-09-21T10:24:18.690 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.690 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.690 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.690 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:18.690 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.030684+0000", 2024-09-21T10:24:18.691 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:18.691 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.691 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.691 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:18.691 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.030683+0000", 2024-09-21T10:24:18.691 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:24:18.691 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.691 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.691 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:18.691 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.030688+0000", 2024-09-21T10:24:18.691 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7289999999999997e-06 2024-09-21T10:24:18.691 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.691 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.691 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:18.691 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.030698+0000", 2024-09-21T10:24:18.692 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.5899999999999997e-06 2024-09-21T10:24:18.692 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.692 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.692 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:18.692 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.030703+0000", 2024-09-21T10:24:18.692 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0690000000000001e-06 2024-09-21T10:24:18.692 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.692 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.692 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:18.692 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.030970+0000", 2024-09-21T10:24:18.692 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000267259 2024-09-21T10:24:18.692 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.692 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.692 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:18.692 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.031079+0000", 2024-09-21T10:24:18.692 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000108719 2024-09-21T10:24:18.693 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.693 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.693 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,7", 2024-09-21T10:24:18.693 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.031186+0000", 2024-09-21T10:24:18.693 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000107252 2024-09-21T10:24:18.693 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.693 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.693 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.693 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.035398+0000", 2024-09-21T10:24:18.693 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0042116089999999998 2024-09-21T10:24:18.693 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.693 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.693 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:18.693 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.036441+0000", 2024-09-21T10:24:18.693 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00104276 2024-09-21T10:24:18.694 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.694 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.694 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:18.694 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.274387+0000", 2024-09-21T10:24:18.694 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23794637499999999 2024-09-21T10:24:18.694 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.694 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.694 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:18.694 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.274446+0000", 2024-09-21T10:24:18.694 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9058000000000003e-05 2024-09-21T10:24:18.694 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:18.694 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:18.694 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:18.695 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.274493+0000", 2024-09-21T10:24:18.695 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6538000000000001e-05 2024-09-21T10:24:18.695 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.695 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.695 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.695 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:18.695 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:18.695 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:24:18.729 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:18.741 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 82 16 109030 109014 83.0774 54.0625 0.0351377 0.0120314 2024-09-21T10:24:18.754 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:18.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:18.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:18.864 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:18 smithi107 ceph-mon[102221]: pgmap v1488: 109 pgs: 109 active+clean; 6.4 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 44 MiB/s wr, 701 op/s 2024-09-21T10:24:18.864 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:18 smithi107 ceph-mon[105022]: pgmap v1488: 109 pgs: 109 active+clean; 6.4 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 44 MiB/s wr, 701 op/s 2024-09-21T10:24:18.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.905 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:18.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:19.005 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:19.029 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:19.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:19.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:19.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:19.168 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:19.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:19.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:19.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:18 smithi129 ceph-mon[88471]: pgmap v1488: 109 pgs: 109 active+clean; 6.4 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 44 MiB/s wr, 701 op/s 2024-09-21T10:24:19.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:24:19.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:19.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:19.339 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:19.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:19.440 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:19.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:19.478 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:19.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:19.531 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:24:19.531 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:24:19.531 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:24:19.531 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:24:19.531 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:24:19.531 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:24:19.531 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:24:19.531 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:24:19.531 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:24:19.532 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:24:19.532 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:24:19.532 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:24:19.532 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:24:19.532 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:24:19.532 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:24:19.532 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:24:19.579 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:19.611 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:19.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:19.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:19.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:19.744 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 83 16 109846 109830 82.6884 51 0.0105176 0.0120889 2024-09-21T10:24:19.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:19.752 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:19.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:19.853 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:19.861 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:24:19.892 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:19.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:19.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:19.994 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:20.023 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:20.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:20.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:19 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2481538898' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:20.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:19 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2481538898' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:20.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:20.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.167 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:20.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:20.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:19 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2481538898' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:20.204 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:24:20.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:20.303 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:20.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:20.405 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:20.450 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:20.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:20.552 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:20.579 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:20.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:24:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:24:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:24:20.681 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:20.717 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:20.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:20.744 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 84 16 110716 110700 82.3513 54.375 0.0242484 0.0121378 2024-09-21T10:24:20.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:20.854 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:20.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:20.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:20.993 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:20.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:21.008 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:20 smithi129 ceph-mon[88471]: pgmap v1489: 109 pgs: 109 active+clean; 6.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-21T10:24:21.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:21.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:20 smithi107 ceph-mon[102221]: pgmap v1489: 109 pgs: 109 active+clean; 6.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-21T10:24:21.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:20 smithi107 ceph-mon[105022]: pgmap v1489: 109 pgs: 109 active+clean; 6.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-21T10:24:21.134 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:21.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:21.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:21.267 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:21.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:21.368 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:21.394 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:21.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:21.496 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:21.525 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:21.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:21.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:21.648 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:21.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:21.744 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 85 16 111570 111554 82.0104 53.375 0.00368823 0.0121853 2024-09-21T10:24:21.751 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:21.773 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:21.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:21.874 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:21.895 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:21.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:21.997 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:22.037 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:22.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:24:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:24:21] "GET /metrics HTTP/1.1" 200 36773 "" "Prometheus/2.51.0" 2024-09-21T10:24:22.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:22.178 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:22.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:22.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:22.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.331 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:22.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:22.432 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:22.457 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:22.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:22.560 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:22.598 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:22.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:22.699 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:22.733 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:22.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:22.744 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 86 16 112310 112294 81.5946 46.25 0.0121415 0.0122506 2024-09-21T10:24:22.835 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:22.863 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:22.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:22.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:22.986 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:22.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:23.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:23.106 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:22 smithi129 ceph-mon[88471]: pgmap v1490: 109 pgs: 109 active+clean; 6.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 40 MiB/s wr, 646 op/s 2024-09-21T10:24:23.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:22 smithi107 ceph-mon[102221]: pgmap v1490: 109 pgs: 109 active+clean; 6.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 40 MiB/s wr, 646 op/s 2024-09-21T10:24:23.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:22 smithi107 ceph-mon[105022]: pgmap v1490: 109 pgs: 109 active+clean; 6.6 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 40 MiB/s wr, 646 op/s 2024-09-21T10:24:23.125 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:23.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:23.227 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:23.253 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:23.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:23.354 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:23.368 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:24:23.368 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:24:23.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.390 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:23.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:23.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:23.534 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:23.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:23.636 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:23.664 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:23.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:23.744 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 87 16 113162 113146 81.2688 53.25 0.0152107 0.0122999 2024-09-21T10:24:23.766 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:23.798 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:23.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:23.900 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:23.922 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:23.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:24.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:24.060 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:24.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:24.078 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:24:24.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:24.201 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:24.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:24.303 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:24.340 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:24.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:24.426 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:24:24.442 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:24.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.487 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:24.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:24.588 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:24.627 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:24.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:24.729 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:24.744 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 88 16 114061 114045 80.9837 56.1875 0.00992093 0.0123422 2024-09-21T10:24:24.761 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:24.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:24.862 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:24.888 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:24.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:24.990 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:25.016 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:25.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:25.100 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:24 smithi107 ceph-mon[102221]: pgmap v1491: 109 pgs: 109 active+clean; 6.8 GiB data, 30 GiB used, 686 GiB / 715 GiB avail; 55 MiB/s wr, 876 op/s 2024-09-21T10:24:25.100 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:24 smithi107 ceph-mon[105022]: pgmap v1491: 109 pgs: 109 active+clean; 6.8 GiB data, 30 GiB used, 686 GiB / 715 GiB avail; 55 MiB/s wr, 876 op/s 2024-09-21T10:24:25.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:25.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.158 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:25.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:24 smithi129 ceph-mon[88471]: pgmap v1491: 109 pgs: 109 active+clean; 6.8 GiB data, 30 GiB used, 686 GiB / 715 GiB avail; 55 MiB/s wr, 876 op/s 2024-09-21T10:24:25.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:25.308 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:25.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:25.410 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:25.451 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:25.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:25.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:25.590 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:25.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:25.729 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:25.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:25.745 INFO:tasks.radosbench.radosbench.0.smithi107.stdout: 89 16 114947 114931 80.6959 55.375 0.0134272 0.012387 2024-09-21T10:24:25.830 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:25.868 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:25.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:25.970 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:25.999 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:26.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:26.101 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:26.128 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:26.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:26.177 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:26.230 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:26.252 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:26.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:26.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:26.395 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:26.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:26.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:26.548 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:26.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:26.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:26.692 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:26.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:26.745 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total time run: 90.0121 2024-09-21T10:24:26.745 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Total writes made: 115788 2024-09-21T10:24:26.746 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Write size: 65536 2024-09-21T10:24:26.746 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Object size: 65536 2024-09-21T10:24:26.746 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Bandwidth (MB/sec): 80.3975 2024-09-21T10:24:26.746 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Bandwidth: 33.162 2024-09-21T10:24:26.746 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max bandwidth (MB/sec): 213.125 2024-09-21T10:24:26.746 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min bandwidth (MB/sec): 46.25 2024-09-21T10:24:26.746 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average IOPS: 1286 2024-09-21T10:24:26.747 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev IOPS: 530.592 2024-09-21T10:24:26.747 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max IOPS: 3410 2024-09-21T10:24:26.747 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min IOPS: 740 2024-09-21T10:24:26.747 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Average Latency(s): 0.012434 2024-09-21T10:24:26.747 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Stddev Latency(s): 0.0110278 2024-09-21T10:24:26.747 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Max latency(s): 0.258966 2024-09-21T10:24:26.747 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Min latency(s): 0.00106792 2024-09-21T10:24:26.750 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:24:26.794 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:26.820 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:26.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:26.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:26.935 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:24:26.935 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:24:26.967 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:26.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:27.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:27.113 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:27.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:27.215 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:27.239 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:27.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:27.271 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:27 smithi129 ceph-mon[88471]: pgmap v1492: 109 pgs: 109 active+clean; 6.9 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 53 MiB/s wr, 847 op/s 2024-09-21T10:24:27.271 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:24:27.271 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:27.341 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:27.384 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:27.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:27.392 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:24:27.393 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:24:27.486 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:27.510 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:27.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:27.561 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:24:27.599 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:27 smithi107 ceph-mon[102221]: pgmap v1492: 109 pgs: 109 active+clean; 6.9 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 53 MiB/s wr, 847 op/s 2024-09-21T10:24:27.599 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:24:27.599 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:27.600 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:27 smithi107 ceph-mon[105022]: pgmap v1492: 109 pgs: 109 active+clean; 6.9 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 53 MiB/s wr, 847 op/s 2024-09-21T10:24:27.600 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:24:27.600 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:27.604 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:24:27.612 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:27.671 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:27.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:27.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:27.814 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:27.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:27.916 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:27.943 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:24:27.950 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:27.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:28.052 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:28.076 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:28.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:28.178 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:28.214 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:28.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:28.316 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:28.342 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:28.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:28.444 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:28.467 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:28.569 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:28.594 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:28.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:28.696 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:28.719 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:28.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:28.820 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:28.846 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:28.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:28.948 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:28.971 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:28.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:29.073 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:29.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.099 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:29.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:29.200 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:29.241 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:29.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:29.343 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:29.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:29 smithi107 ceph-mon[102221]: from='client.37422 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:24:29.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:29 smithi107 ceph-mon[102221]: pgmap v1493: 109 pgs: 109 active+clean; 6.9 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 87 B/s rd, 50 MiB/s wr, 814 op/s 2024-09-21T10:24:29.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:29 smithi107 ceph-mon[105022]: from='client.37422 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:24:29.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:29 smithi107 ceph-mon[105022]: pgmap v1493: 109 pgs: 109 active+clean; 6.9 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 87 B/s rd, 50 MiB/s wr, 814 op/s 2024-09-21T10:24:29.366 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:29.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:29.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:29 smithi129 ceph-mon[88471]: from='client.37422 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:24:29.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:29 smithi129 ceph-mon[88471]: pgmap v1493: 109 pgs: 109 active+clean; 6.9 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 87 B/s rd, 50 MiB/s wr, 814 op/s 2024-09-21T10:24:29.467 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:29.505 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:29.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:29.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:29.608 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1492,"stamp":"2024-09-21T10:24:25.548195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7370315725,"num_objects":112479,"num_object_clones":0,"num_object_copies":337437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":112479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112517,"num_write_kb":7197024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":1197514,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113912,"ondisk_log_size":113912,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":31267820,"kb_used_data":21555820,"kb_used_omap":43,"kb_used_meta":9711892,"kb_avail":718693396,"statfs":{"total":767960285184,"available":735942037504,"internally_reserved":0,"allocated":22073159680,"data_stored":22052555622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44223,"internal_metadata":9944978241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,4,3,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":555417600,"num_objects":8475,"num_object_clones":0,"num_object_copies":25425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8475,"num_write_kb":542400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002580"},"pg_stats":[{"pgid":"1.11","version":"77'100","reported_seq":372,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.830587+0000","last_change":"2024-09-21T10:10:57.397786+0000","last_active":"2024-09-21T10:24:15.830587+0000","last_peered":"2024-09-21T10:24:15.830587+0000","last_clean":"2024-09-21T10:24:15.830587+0000","last_became_active":"2024-09-21T10:10:57.396742+0000","last_became_peered":"2024-09-21T10:10:57.396742+0000","last_unstale":"2024-09-21T10:24:15.830587+0000","last_undegraded":"2024-09-21T10:24:15.830587+0000","last_fullsized":"2024-09-21T10:24:15.830587+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:53.631923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:53.631923+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:48:41.225820+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":709,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.828567+0000","last_change":"2024-09-21T10:03:38.389577+0000","last_active":"2024-09-21T10:24:15.828567+0000","last_peered":"2024-09-21T10:24:15.828567+0000","last_clean":"2024-09-21T10:24:15.828567+0000","last_became_active":"2024-09-21T10:03:38.389292+0000","last_became_peered":"2024-09-21T10:03:38.389292+0000","last_unstale":"2024-09-21T10:24:15.828567+0000","last_undegraded":"2024-09-21T10:24:15.828567+0000","last_fullsized":"2024-09-21T10:24:15.828567+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:32.113472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":310,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.789296+0000","last_change":"2024-09-21T10:14:30.717695+0000","last_active":"2024-09-21T10:24:14.789296+0000","last_peered":"2024-09-21T10:24:14.789296+0000","last_clean":"2024-09-21T10:24:14.789296+0000","last_became_active":"2024-09-21T10:14:30.705952+0000","last_became_peered":"2024-09-21T10:14:30.705952+0000","last_unstale":"2024-09-21T10:24:14.789296+0000","last_undegraded":"2024-09-21T10:24:14.789296+0000","last_fullsized":"2024-09-21T10:24:14.789296+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:08:18.863430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'100","reported_seq":343,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.789059+0000","last_change":"2024-09-21T10:22:19.656201+0000","last_active":"2024-09-21T10:24:14.789059+0000","last_peered":"2024-09-21T10:24:14.789059+0000","last_clean":"2024-09-21T10:24:14.789059+0000","last_became_active":"2024-09-21T10:22:19.655644+0000","last_became_peered":"2024-09-21T10:22:19.655644+0000","last_unstale":"2024-09-21T10:24:14.789059+0000","last_undegraded":"2024-09-21T10:24:14.789059+0000","last_fullsized":"2024-09-21T10:24:14.789059+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.798577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.798577+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:58:58.525208+0000","scrub_duration":0.0023622640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":415,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.789827+0000","last_change":"2024-09-21T10:10:57.392782+0000","last_active":"2024-09-21T10:24:14.789827+0000","last_peered":"2024-09-21T10:24:14.789827+0000","last_clean":"2024-09-21T10:24:14.789827+0000","last_became_active":"2024-09-21T10:10:57.392305+0000","last_became_peered":"2024-09-21T10:10:57.392305+0000","last_unstale":"2024-09-21T10:24:14.789827+0000","last_undegraded":"2024-09-21T10:24:14.789827+0000","last_fullsized":"2024-09-21T10:24:14.789827+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:42:01.387622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":629,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.836292+0000","last_change":"2024-09-21T10:09:23.846703+0000","last_active":"2024-09-21T10:24:15.836292+0000","last_peered":"2024-09-21T10:24:15.836292+0000","last_clean":"2024-09-21T10:24:15.836292+0000","last_became_active":"2024-09-21T10:09:23.846415+0000","last_became_peered":"2024-09-21T10:09:23.846415+0000","last_unstale":"2024-09-21T10:24:15.836292+0000","last_undegraded":"2024-09-21T10:24:15.836292+0000","last_fullsized":"2024-09-21T10:24:15.836292+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:23.129285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'100","reported_seq":373,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.823608+0000","last_change":"2024-09-21T10:18:12.637168+0000","last_active":"2024-09-21T10:24:15.823608+0000","last_peered":"2024-09-21T10:24:15.823608+0000","last_clean":"2024-09-21T10:24:15.823608+0000","last_became_active":"2024-09-21T10:18:12.636410+0000","last_became_peered":"2024-09-21T10:18:12.636410+0000","last_unstale":"2024-09-21T10:24:15.823608+0000","last_undegraded":"2024-09-21T10:24:15.823608+0000","last_fullsized":"2024-09-21T10:24:15.823608+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.433227+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:34.403451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":355,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.845763+0000","last_change":"2024-09-21T10:09:49.922489+0000","last_active":"2024-09-21T10:24:15.845763+0000","last_peered":"2024-09-21T10:24:15.845763+0000","last_clean":"2024-09-21T10:24:15.845763+0000","last_became_active":"2024-09-21T10:09:49.921629+0000","last_became_peered":"2024-09-21T10:09:49.921629+0000","last_unstale":"2024-09-21T10:24:15.845763+0000","last_undegraded":"2024-09-21T10:24:15.845763+0000","last_fullsized":"2024-09-21T10:24:15.845763+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:50.611979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":326,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.789029+0000","last_change":"2024-09-21T10:22:19.646202+0000","last_active":"2024-09-21T10:24:14.789029+0000","last_peered":"2024-09-21T10:24:14.789029+0000","last_clean":"2024-09-21T10:24:14.789029+0000","last_became_active":"2024-09-21T10:22:19.645690+0000","last_became_peered":"2024-09-21T10:22:19.645690+0000","last_unstale":"2024-09-21T10:24:14.789029+0000","last_undegraded":"2024-09-21T10:24:14.789029+0000","last_fullsized":"2024-09-21T10:24:14.789029+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:44:01.509115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"77'100","reported_seq":276,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.822017+0000","last_change":"2024-09-21T10:22:19.644444+0000","last_active":"2024-09-21T10:24:15.822017+0000","last_peered":"2024-09-21T10:24:15.822017+0000","last_clean":"2024-09-21T10:24:15.822017+0000","last_became_active":"2024-09-21T10:22:19.644051+0000","last_became_peered":"2024-09-21T10:22:19.644051+0000","last_unstale":"2024-09-21T10:24:15.822017+0000","last_undegraded":"2024-09-21T10:24:15.822017+0000","last_fullsized":"2024-09-21T10:24:15.822017+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:15:02.841968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:15:02.841968+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:11.035208+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":708,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.834005+0000","last_change":"2024-09-21T10:03:38.381552+0000","last_active":"2024-09-21T10:24:15.834005+0000","last_peered":"2024-09-21T10:24:15.834005+0000","last_clean":"2024-09-21T10:24:15.834005+0000","last_became_active":"2024-09-21T10:03:38.380898+0000","last_became_peered":"2024-09-21T10:03:38.380898+0000","last_unstale":"2024-09-21T10:24:15.834005+0000","last_undegraded":"2024-09-21T10:24:15.834005+0000","last_fullsized":"2024-09-21T10:24:15.834005+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:53.409606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":42,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.790435+0000","last_change":"2024-09-21T10:22:19.662950+0000","last_active":"2024-09-21T10:24:14.790435+0000","last_peered":"2024-09-21T10:24:14.790435+0000","last_clean":"2024-09-21T10:24:14.790435+0000","last_became_active":"2024-09-21T10:22:19.649590+0000","last_became_peered":"2024-09-21T10:22:19.649590+0000","last_unstale":"2024-09-21T10:24:14.790435+0000","last_undegraded":"2024-09-21T10:24:14.790435+0000","last_fullsized":"2024-09-21T10:24:14.790435+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:38:15.512148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.2","version":"425'6980","reported_seq":7001,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:22.093422+0000","last_change":"2024-09-21T10:22:44.823473+0000","last_active":"2024-09-21T10:24:22.093422+0000","last_peered":"2024-09-21T10:24:22.093422+0000","last_clean":"2024-09-21T10:24:22.093422+0000","last_became_active":"2024-09-21T10:22:44.823115+0000","last_became_peered":"2024-09-21T10:22:44.823115+0000","last_unstale":"2024-09-21T10:24:22.093422+0000","last_undegraded":"2024-09-21T10:24:22.093422+0000","last_fullsized":"2024-09-21T10:24:22.093422+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":6980,"log_dups_size":0,"ondisk_log_size":6980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:55:04.796944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457441280,"num_objects":6980,"num_object_clones":0,"num_object_copies":20940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6980,"num_write_kb":446720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"77'100","reported_seq":318,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.789831+0000","last_change":"2024-09-21T10:14:30.708275+0000","last_active":"2024-09-21T10:24:14.789831+0000","last_peered":"2024-09-21T10:24:14.789831+0000","last_clean":"2024-09-21T10:24:14.789831+0000","last_became_active":"2024-09-21T10:14:30.707763+0000","last_became_peered":"2024-09-21T10:14:30.707763+0000","last_unstale":"2024-09-21T10:24:14.789831+0000","last_undegraded":"2024-09-21T10:24:14.789831+0000","last_fullsized":"2024-09-21T10:24:14.789831+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:55.259385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:55.259385+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:41.322447+0000","scrub_duration":0.0015667260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":353,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.789528+0000","last_change":"2024-09-21T10:14:30.720250+0000","last_active":"2024-09-21T10:24:14.789528+0000","last_peered":"2024-09-21T10:24:14.789528+0000","last_clean":"2024-09-21T10:24:14.789528+0000","last_became_active":"2024-09-21T10:14:30.707036+0000","last_became_peered":"2024-09-21T10:14:30.707036+0000","last_unstale":"2024-09-21T10:24:14.789528+0000","last_undegraded":"2024-09-21T10:24:14.789528+0000","last_fullsized":"2024-09-21T10:24:14.789528+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:17.438534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":416,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.788895+0000","last_change":"2024-09-21T10:22:19.650243+0000","last_active":"2024-09-21T10:24:14.788895+0000","last_peered":"2024-09-21T10:24:14.788895+0000","last_clean":"2024-09-21T10:24:14.788895+0000","last_became_active":"2024-09-21T10:22:19.648182+0000","last_became_peered":"2024-09-21T10:22:19.648182+0000","last_unstale":"2024-09-21T10:24:14.788895+0000","last_undegraded":"2024-09-21T10:24:14.788895+0000","last_fullsized":"2024-09-21T10:24:14.788895+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:59:48.769286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3","version":"425'7008","reported_seq":7046,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:23.033476+0000","last_change":"2024-09-21T10:22:44.823653+0000","last_active":"2024-09-21T10:24:23.033476+0000","last_peered":"2024-09-21T10:24:23.033476+0000","last_clean":"2024-09-21T10:24:23.033476+0000","last_became_active":"2024-09-21T10:22:44.823268+0000","last_became_peered":"2024-09-21T10:22:44.823268+0000","last_unstale":"2024-09-21T10:24:23.033476+0000","last_undegraded":"2024-09-21T10:24:23.033476+0000","last_fullsized":"2024-09-21T10:24:23.033476+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":7008,"log_dups_size":0,"ondisk_log_size":7008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:35:38.400207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459276288,"num_objects":7008,"num_object_clones":0,"num_object_copies":21024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7008,"num_write_kb":448512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"77'100","reported_seq":542,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.828320+0000","last_change":"2024-09-21T10:22:19.667407+0000","last_active":"2024-09-21T10:24:15.828320+0000","last_peered":"2024-09-21T10:24:15.828320+0000","last_clean":"2024-09-21T10:24:15.828320+0000","last_became_active":"2024-09-21T10:22:19.647633+0000","last_became_peered":"2024-09-21T10:22:19.647633+0000","last_unstale":"2024-09-21T10:24:15.828320+0000","last_undegraded":"2024-09-21T10:24:15.828320+0000","last_fullsized":"2024-09-21T10:24:15.828320+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.981336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.981336+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:46.044625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":332,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.789590+0000","last_change":"2024-09-21T10:18:12.643154+0000","last_active":"2024-09-21T10:24:14.789590+0000","last_peered":"2024-09-21T10:24:14.789590+0000","last_clean":"2024-09-21T10:24:14.789590+0000","last_became_active":"2024-09-21T10:18:12.636287+0000","last_became_peered":"2024-09-21T10:18:12.636287+0000","last_unstale":"2024-09-21T10:24:14.789590+0000","last_undegraded":"2024-09-21T10:24:14.789590+0000","last_fullsized":"2024-09-21T10:24:14.789590+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:45:31.008309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":182,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.821039+0000","last_change":"2024-09-21T10:22:08.117935+0000","last_active":"2024-09-21T10:24:15.821039+0000","last_peered":"2024-09-21T10:24:15.821039+0000","last_clean":"2024-09-21T10:24:15.821039+0000","last_became_active":"2024-09-21T10:22:08.117613+0000","last_became_peered":"2024-09-21T10:22:08.117613+0000","last_unstale":"2024-09-21T10:24:15.821039+0000","last_undegraded":"2024-09-21T10:24:15.821039+0000","last_fullsized":"2024-09-21T10:24:15.821039+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:39.820626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.0","version":"425'7155","reported_seq":7193,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:23.862173+0000","last_change":"2024-09-21T10:22:44.830660+0000","last_active":"2024-09-21T10:24:23.862173+0000","last_peered":"2024-09-21T10:24:23.862173+0000","last_clean":"2024-09-21T10:24:23.862173+0000","last_became_active":"2024-09-21T10:22:44.830373+0000","last_became_peered":"2024-09-21T10:22:44.830373+0000","last_unstale":"2024-09-21T10:24:23.862173+0000","last_undegraded":"2024-09-21T10:24:23.862173+0000","last_fullsized":"2024-09-21T10:24:23.862173+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":7155,"log_dups_size":0,"ondisk_log_size":7155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:02.866984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468910080,"num_objects":7155,"num_object_clones":0,"num_object_copies":21465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7155,"num_write_kb":457920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"77'100","reported_seq":32,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.790760+0000","last_change":"2024-09-21T10:22:19.657392+0000","last_active":"2024-09-21T10:24:14.790760+0000","last_peered":"2024-09-21T10:24:14.790760+0000","last_clean":"2024-09-21T10:24:14.790760+0000","last_became_active":"2024-09-21T10:22:19.650821+0000","last_became_peered":"2024-09-21T10:22:19.650821+0000","last_unstale":"2024-09-21T10:24:14.790760+0000","last_undegraded":"2024-09-21T10:24:14.790760+0000","last_fullsized":"2024-09-21T10:24:14.790760+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.289859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.289859+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:54:14.510032+0000","scrub_duration":0.001251809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":332,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.789667+0000","last_change":"2024-09-21T10:22:19.664692+0000","last_active":"2024-09-21T10:24:14.789667+0000","last_peered":"2024-09-21T10:24:14.789667+0000","last_clean":"2024-09-21T10:24:14.789667+0000","last_became_active":"2024-09-21T10:22:19.647696+0000","last_became_peered":"2024-09-21T10:22:19.647696+0000","last_unstale":"2024-09-21T10:24:14.789667+0000","last_undegraded":"2024-09-21T10:24:14.789667+0000","last_fullsized":"2024-09-21T10:24:14.789667+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:57:54.264752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":32,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.790652+0000","last_change":"2024-09-21T10:22:19.652364+0000","last_active":"2024-09-21T10:24:14.790652+0000","last_peered":"2024-09-21T10:24:14.790652+0000","last_clean":"2024-09-21T10:24:14.790652+0000","last_became_active":"2024-09-21T10:22:19.651586+0000","last_became_peered":"2024-09-21T10:22:19.651586+0000","last_unstale":"2024-09-21T10:24:14.790652+0000","last_undegraded":"2024-09-21T10:24:14.790652+0000","last_fullsized":"2024-09-21T10:24:14.790652+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:03.593157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1","version":"425'6976","reported_seq":7015,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:22.648407+0000","last_change":"2024-09-21T10:22:44.821910+0000","last_active":"2024-09-21T10:24:22.648407+0000","last_peered":"2024-09-21T10:24:22.648407+0000","last_clean":"2024-09-21T10:24:22.648407+0000","last_became_active":"2024-09-21T10:22:44.821581+0000","last_became_peered":"2024-09-21T10:22:44.821581+0000","last_unstale":"2024-09-21T10:24:22.648407+0000","last_undegraded":"2024-09-21T10:24:22.648407+0000","last_fullsized":"2024-09-21T10:24:22.648407+0000","mapping_epoch":411,"log_start":"415'300","ondisk_log_start":"415'300","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":6676,"log_dups_size":0,"ondisk_log_size":6676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:05:51.203136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457179136,"num_objects":6976,"num_object_clones":0,"num_object_copies":20928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6976,"num_write_kb":446464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"77'100","reported_seq":254,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.789422+0000","last_change":"2024-09-21T10:22:08.163986+0000","last_active":"2024-09-21T10:24:14.789422+0000","last_peered":"2024-09-21T10:24:14.789422+0000","last_clean":"2024-09-21T10:24:14.789422+0000","last_became_active":"2024-09-21T10:22:08.163669+0000","last_became_peered":"2024-09-21T10:22:08.163669+0000","last_unstale":"2024-09-21T10:24:14.789422+0000","last_undegraded":"2024-09-21T10:24:14.789422+0000","last_fullsized":"2024-09-21T10:24:14.789422+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:18:27.085525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:18:27.085525+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:55:41.477408+0000","scrub_duration":0.021884888000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":527,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.828970+0000","last_change":"2024-09-21T10:22:19.646721+0000","last_active":"2024-09-21T10:24:15.828970+0000","last_peered":"2024-09-21T10:24:15.828970+0000","last_clean":"2024-09-21T10:24:15.828970+0000","last_became_active":"2024-09-21T10:22:19.645541+0000","last_became_peered":"2024-09-21T10:22:19.645541+0000","last_unstale":"2024-09-21T10:24:15.828970+0000","last_undegraded":"2024-09-21T10:24:15.828970+0000","last_fullsized":"2024-09-21T10:24:15.828970+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:59:36.096992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":616,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.831351+0000","last_change":"2024-09-21T10:14:30.711105+0000","last_active":"2024-09-21T10:24:15.831351+0000","last_peered":"2024-09-21T10:24:15.831351+0000","last_clean":"2024-09-21T10:24:15.831351+0000","last_became_active":"2024-09-21T10:14:30.710759+0000","last_became_peered":"2024-09-21T10:14:30.710759+0000","last_unstale":"2024-09-21T10:24:15.831351+0000","last_undegraded":"2024-09-21T10:24:15.831351+0000","last_fullsized":"2024-09-21T10:24:15.831351+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:40.690042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.6","version":"425'6932","reported_seq":6971,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:22.940149+0000","last_change":"2024-09-21T10:22:44.826445+0000","last_active":"2024-09-21T10:24:22.940149+0000","last_peered":"2024-09-21T10:24:22.940149+0000","last_clean":"2024-09-21T10:24:22.940149+0000","last_became_active":"2024-09-21T10:22:44.825882+0000","last_became_peered":"2024-09-21T10:22:44.825882+0000","last_unstale":"2024-09-21T10:24:22.940149+0000","last_undegraded":"2024-09-21T10:24:22.940149+0000","last_fullsized":"2024-09-21T10:24:22.940149+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":6932,"log_dups_size":0,"ondisk_log_size":6932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:26:59.950097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454295552,"num_objects":6932,"num_object_clones":0,"num_object_copies":20796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6932,"num_write_kb":443648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":298,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.789075+0000","last_change":"2024-09-21T10:04:53.355975+0000","last_active":"2024-09-21T10:24:14.789075+0000","last_peered":"2024-09-21T10:24:14.789075+0000","last_clean":"2024-09-21T10:24:14.789075+0000","last_became_active":"2024-09-21T10:04:42.547505+0000","last_became_peered":"2024-09-21T10:04:42.547505+0000","last_unstale":"2024-09-21T10:24:14.789075+0000","last_undegraded":"2024-09-21T10:24:14.789075+0000","last_fullsized":"2024-09-21T10:24:14.789075+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.355912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.355912+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:43.456830+0000","scrub_duration":0.0017980030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":720,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.838626+0000","last_change":"2024-09-21T10:14:30.715303+0000","last_active":"2024-09-21T10:24:15.838626+0000","last_peered":"2024-09-21T10:24:15.838626+0000","last_clean":"2024-09-21T10:24:15.838626+0000","last_became_active":"2024-09-21T10:14:30.703664+0000","last_became_peered":"2024-09-21T10:14:30.703664+0000","last_unstale":"2024-09-21T10:24:15.838626+0000","last_undegraded":"2024-09-21T10:24:15.838626+0000","last_fullsized":"2024-09-21T10:24:15.838626+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:28.967906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":306,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.829088+0000","last_change":"2024-09-21T10:22:19.647110+0000","last_active":"2024-09-21T10:24:15.829088+0000","last_peered":"2024-09-21T10:24:15.829088+0000","last_clean":"2024-09-21T10:24:15.829088+0000","last_became_active":"2024-09-21T10:22:19.645581+0000","last_became_peered":"2024-09-21T10:22:19.645581+0000","last_unstale":"2024-09-21T10:24:15.829088+0000","last_undegraded":"2024-09-21T10:24:15.829088+0000","last_fullsized":"2024-09-21T10:24:15.829088+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.054517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.7","version":"425'7208","reported_seq":7247,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:22.924328+0000","last_change":"2024-09-21T10:22:44.830915+0000","last_active":"2024-09-21T10:24:22.924328+0000","last_peered":"2024-09-21T10:24:22.924328+0000","last_clean":"2024-09-21T10:24:22.924328+0000","last_became_active":"2024-09-21T10:22:44.830464+0000","last_became_peered":"2024-09-21T10:22:44.830464+0000","last_unstale":"2024-09-21T10:24:22.924328+0000","last_undegraded":"2024-09-21T10:24:22.924328+0000","last_fullsized":"2024-09-21T10:24:22.924328+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":7208,"log_dups_size":0,"ondisk_log_size":7208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:58:49.959117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472383488,"num_objects":7208,"num_object_clones":0,"num_object_copies":21624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7208,"num_write_kb":461312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":471,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.833008+0000","last_change":"2024-09-21T10:10:57.403722+0000","last_active":"2024-09-21T10:24:15.833008+0000","last_peered":"2024-09-21T10:24:15.833008+0000","last_clean":"2024-09-21T10:24:15.833008+0000","last_became_active":"2024-09-21T10:10:57.394700+0000","last_became_peered":"2024-09-21T10:10:57.394700+0000","last_unstale":"2024-09-21T10:24:15.833008+0000","last_undegraded":"2024-09-21T10:24:15.833008+0000","last_fullsized":"2024-09-21T10:24:15.833008+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:02.152981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:02.152981+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:25.675519+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":32,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.790574+0000","last_change":"2024-09-21T10:22:19.652182+0000","last_active":"2024-09-21T10:24:14.790574+0000","last_peered":"2024-09-21T10:24:14.790574+0000","last_clean":"2024-09-21T10:24:14.790574+0000","last_became_active":"2024-09-21T10:22:19.651703+0000","last_became_peered":"2024-09-21T10:22:19.651703+0000","last_unstale":"2024-09-21T10:24:14.790574+0000","last_undegraded":"2024-09-21T10:24:14.790574+0000","last_fullsized":"2024-09-21T10:24:14.790574+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:08.752662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":460,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.835735+0000","last_change":"2024-09-21T10:22:08.118607+0000","last_active":"2024-09-21T10:24:15.835735+0000","last_peered":"2024-09-21T10:24:15.835735+0000","last_clean":"2024-09-21T10:24:15.835735+0000","last_became_active":"2024-09-21T10:22:08.117950+0000","last_became_peered":"2024-09-21T10:22:08.117950+0000","last_unstale":"2024-09-21T10:24:15.835735+0000","last_undegraded":"2024-09-21T10:24:15.835735+0000","last_fullsized":"2024-09-21T10:24:15.835735+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:29:17.689281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"425'6989","reported_seq":7011,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:22.048076+0000","last_change":"2024-09-21T10:22:44.823139+0000","last_active":"2024-09-21T10:24:22.048076+0000","last_peered":"2024-09-21T10:24:22.048076+0000","last_clean":"2024-09-21T10:24:22.048076+0000","last_became_active":"2024-09-21T10:22:44.822759+0000","last_became_peered":"2024-09-21T10:22:44.822759+0000","last_unstale":"2024-09-21T10:24:22.048076+0000","last_undegraded":"2024-09-21T10:24:22.048076+0000","last_fullsized":"2024-09-21T10:24:22.048076+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":6989,"log_dups_size":0,"ondisk_log_size":6989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:16.929399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458031104,"num_objects":6989,"num_object_clones":0,"num_object_copies":20967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6989,"num_write_kb":447296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"77'100","reported_seq":363,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.823500+0000","last_change":"2024-09-21T10:14:05.474100+0000","last_active":"2024-09-21T10:24:15.823500+0000","last_peered":"2024-09-21T10:24:15.823500+0000","last_clean":"2024-09-21T10:24:15.823500+0000","last_became_active":"2024-09-21T10:14:05.473336+0000","last_became_peered":"2024-09-21T10:14:05.473336+0000","last_unstale":"2024-09-21T10:24:15.823500+0000","last_undegraded":"2024-09-21T10:24:15.823500+0000","last_fullsized":"2024-09-21T10:24:15.823500+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.540323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.540323+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:06:10.963352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":552,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.833177+0000","last_change":"2024-09-21T10:22:08.118141+0000","last_active":"2024-09-21T10:24:15.833177+0000","last_peered":"2024-09-21T10:24:15.833177+0000","last_clean":"2024-09-21T10:24:15.833177+0000","last_became_active":"2024-09-21T10:22:08.117868+0000","last_became_peered":"2024-09-21T10:22:08.117868+0000","last_unstale":"2024-09-21T10:24:15.833177+0000","last_undegraded":"2024-09-21T10:24:15.833177+0000","last_fullsized":"2024-09-21T10:24:15.833177+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:01.774438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.790699+0000","last_change":"2024-09-21T10:22:19.651485+0000","last_active":"2024-09-21T10:24:14.790699+0000","last_peered":"2024-09-21T10:24:14.790699+0000","last_clean":"2024-09-21T10:24:14.790699+0000","last_became_active":"2024-09-21T10:22:19.651154+0000","last_became_peered":"2024-09-21T10:22:19.651154+0000","last_unstale":"2024-09-21T10:24:14.790699+0000","last_undegraded":"2024-09-21T10:24:14.790699+0000","last_fullsized":"2024-09-21T10:24:14.790699+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:31.496619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"425'6940","reported_seq":6979,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:22.920186+0000","last_change":"2024-09-21T10:22:44.822395+0000","last_active":"2024-09-21T10:24:22.920186+0000","last_peered":"2024-09-21T10:24:22.920186+0000","last_clean":"2024-09-21T10:24:22.920186+0000","last_became_active":"2024-09-21T10:22:44.822097+0000","last_became_peered":"2024-09-21T10:22:44.822097+0000","last_unstale":"2024-09-21T10:24:22.920186+0000","last_undegraded":"2024-09-21T10:24:22.920186+0000","last_fullsized":"2024-09-21T10:24:22.920186+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":6940,"log_dups_size":0,"ondisk_log_size":6940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:58:55.402300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454819840,"num_objects":6940,"num_object_clones":0,"num_object_copies":20820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6940,"num_write_kb":444160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":456,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.833147+0000","last_change":"2024-09-21T10:22:19.646459+0000","last_active":"2024-09-21T10:24:15.833147+0000","last_peered":"2024-09-21T10:24:15.833147+0000","last_clean":"2024-09-21T10:24:15.833147+0000","last_became_active":"2024-09-21T10:22:19.646127+0000","last_became_peered":"2024-09-21T10:22:19.646127+0000","last_unstale":"2024-09-21T10:24:15.833147+0000","last_undegraded":"2024-09-21T10:24:15.833147+0000","last_fullsized":"2024-09-21T10:24:15.833147+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:20:00.237668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:20:00.237668+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:27.159346+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":711,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.828102+0000","last_change":"2024-09-21T10:03:38.386604+0000","last_active":"2024-09-21T10:24:15.828102+0000","last_peered":"2024-09-21T10:24:15.828102+0000","last_clean":"2024-09-21T10:24:15.828102+0000","last_became_active":"2024-09-21T10:03:38.386283+0000","last_became_peered":"2024-09-21T10:03:38.386283+0000","last_unstale":"2024-09-21T10:24:15.828102+0000","last_undegraded":"2024-09-21T10:24:15.828102+0000","last_fullsized":"2024-09-21T10:24:15.828102+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:27:05.944880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":569,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.834115+0000","last_change":"2024-09-21T10:14:30.709615+0000","last_active":"2024-09-21T10:24:15.834115+0000","last_peered":"2024-09-21T10:24:15.834115+0000","last_clean":"2024-09-21T10:24:15.834115+0000","last_became_active":"2024-09-21T10:14:30.709184+0000","last_became_peered":"2024-09-21T10:14:30.709184+0000","last_unstale":"2024-09-21T10:24:15.834115+0000","last_undegraded":"2024-09-21T10:24:15.834115+0000","last_fullsized":"2024-09-21T10:24:15.834115+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:55:27.220232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"425'6860","reported_seq":6882,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:22.359702+0000","last_change":"2024-09-21T10:22:44.824085+0000","last_active":"2024-09-21T10:24:22.359702+0000","last_peered":"2024-09-21T10:24:22.359702+0000","last_clean":"2024-09-21T10:24:22.359702+0000","last_became_active":"2024-09-21T10:22:44.823764+0000","last_became_peered":"2024-09-21T10:22:44.823764+0000","last_unstale":"2024-09-21T10:24:22.359702+0000","last_undegraded":"2024-09-21T10:24:22.359702+0000","last_fullsized":"2024-09-21T10:24:22.359702+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":6860,"log_dups_size":0,"ondisk_log_size":6860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:03.463670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449576960,"num_objects":6860,"num_object_clones":0,"num_object_copies":20580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6860,"num_write_kb":439040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":335,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.788985+0000","last_change":"2024-09-21T10:22:19.655993+0000","last_active":"2024-09-21T10:24:14.788985+0000","last_peered":"2024-09-21T10:24:14.788985+0000","last_clean":"2024-09-21T10:24:14.788985+0000","last_became_active":"2024-09-21T10:22:19.655178+0000","last_became_peered":"2024-09-21T10:22:19.655178+0000","last_unstale":"2024-09-21T10:24:14.788985+0000","last_undegraded":"2024-09-21T10:24:14.788985+0000","last_fullsized":"2024-09-21T10:24:14.788985+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:47.393146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:47.393146+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:48.668712+0000","scrub_duration":0.001601086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":465,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.835112+0000","last_change":"2024-09-21T10:10:57.402946+0000","last_active":"2024-09-21T10:24:15.835112+0000","last_peered":"2024-09-21T10:24:15.835112+0000","last_clean":"2024-09-21T10:24:15.835112+0000","last_became_active":"2024-09-21T10:10:57.395450+0000","last_became_peered":"2024-09-21T10:10:57.395450+0000","last_unstale":"2024-09-21T10:24:15.835112+0000","last_undegraded":"2024-09-21T10:24:15.835112+0000","last_fullsized":"2024-09-21T10:24:15.835112+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:08.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":709,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.829191+0000","last_change":"2024-09-21T10:22:19.647384+0000","last_active":"2024-09-21T10:24:15.829191+0000","last_peered":"2024-09-21T10:24:15.829191+0000","last_clean":"2024-09-21T10:24:15.829191+0000","last_became_active":"2024-09-21T10:22:19.645407+0000","last_became_peered":"2024-09-21T10:22:19.645407+0000","last_unstale":"2024-09-21T10:24:15.829191+0000","last_undegraded":"2024-09-21T10:24:15.829191+0000","last_fullsized":"2024-09-21T10:24:15.829191+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:04:57.385251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":32,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.790577+0000","last_change":"2024-09-21T10:22:19.651173+0000","last_active":"2024-09-21T10:24:14.790577+0000","last_peered":"2024-09-21T10:24:14.790577+0000","last_clean":"2024-09-21T10:24:14.790577+0000","last_became_active":"2024-09-21T10:22:19.649673+0000","last_became_peered":"2024-09-21T10:22:19.649673+0000","last_unstale":"2024-09-21T10:24:14.790577+0000","last_undegraded":"2024-09-21T10:24:14.790577+0000","last_fullsized":"2024-09-21T10:24:14.790577+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:28:37.605825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.c","version":"425'7109","reported_seq":7148,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:22.927099+0000","last_change":"2024-09-21T10:22:44.825224+0000","last_active":"2024-09-21T10:24:22.927099+0000","last_peered":"2024-09-21T10:24:22.927099+0000","last_clean":"2024-09-21T10:24:22.927099+0000","last_became_active":"2024-09-21T10:22:44.824917+0000","last_became_peered":"2024-09-21T10:22:44.824917+0000","last_unstale":"2024-09-21T10:24:22.927099+0000","last_undegraded":"2024-09-21T10:24:22.927099+0000","last_fullsized":"2024-09-21T10:24:22.927099+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":7109,"log_dups_size":0,"ondisk_log_size":7109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:02.699042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465895424,"num_objects":7109,"num_object_clones":0,"num_object_copies":21327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7109,"num_write_kb":454976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":387,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.823755+0000","last_change":"2024-09-21T10:22:19.651858+0000","last_active":"2024-09-21T10:24:15.823755+0000","last_peered":"2024-09-21T10:24:15.823755+0000","last_clean":"2024-09-21T10:24:15.823755+0000","last_became_active":"2024-09-21T10:22:19.651510+0000","last_became_peered":"2024-09-21T10:22:19.651510+0000","last_unstale":"2024-09-21T10:24:15.823755+0000","last_undegraded":"2024-09-21T10:24:15.823755+0000","last_fullsized":"2024-09-21T10:24:15.823755+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.550551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.550551+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:01:42.501915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":289,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.788979+0000","last_change":"2024-09-21T10:09:23.838944+0000","last_active":"2024-09-21T10:24:14.788979+0000","last_peered":"2024-09-21T10:24:14.788979+0000","last_clean":"2024-09-21T10:24:14.788979+0000","last_became_active":"2024-09-21T10:09:23.837886+0000","last_became_peered":"2024-09-21T10:09:23.837886+0000","last_unstale":"2024-09-21T10:24:14.788979+0000","last_undegraded":"2024-09-21T10:24:14.788979+0000","last_fullsized":"2024-09-21T10:24:14.788979+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:16.928195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":594,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.830399+0000","last_change":"2024-09-21T10:18:12.637103+0000","last_active":"2024-09-21T10:24:15.830399+0000","last_peered":"2024-09-21T10:24:15.830399+0000","last_clean":"2024-09-21T10:24:15.830399+0000","last_became_active":"2024-09-21T10:18:12.636645+0000","last_became_peered":"2024-09-21T10:18:12.636645+0000","last_unstale":"2024-09-21T10:24:15.830399+0000","last_undegraded":"2024-09-21T10:24:15.830399+0000","last_fullsized":"2024-09-21T10:24:15.830399+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:53:50.745484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":355,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.845656+0000","last_change":"2024-09-21T10:09:49.923044+0000","last_active":"2024-09-21T10:24:15.845656+0000","last_peered":"2024-09-21T10:24:15.845656+0000","last_clean":"2024-09-21T10:24:15.845656+0000","last_became_active":"2024-09-21T10:09:49.922711+0000","last_became_peered":"2024-09-21T10:09:49.922711+0000","last_unstale":"2024-09-21T10:24:15.845656+0000","last_undegraded":"2024-09-21T10:24:15.845656+0000","last_fullsized":"2024-09-21T10:24:15.845656+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:14.090084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.d","version":"425'6972","reported_seq":6994,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:22.069726+0000","last_change":"2024-09-21T10:22:44.830934+0000","last_active":"2024-09-21T10:24:22.069726+0000","last_peered":"2024-09-21T10:24:22.069726+0000","last_clean":"2024-09-21T10:24:22.069726+0000","last_became_active":"2024-09-21T10:22:44.830615+0000","last_became_peered":"2024-09-21T10:22:44.830615+0000","last_unstale":"2024-09-21T10:24:22.069726+0000","last_undegraded":"2024-09-21T10:24:22.069726+0000","last_fullsized":"2024-09-21T10:24:22.069726+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":6972,"log_dups_size":0,"ondisk_log_size":6972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:27.432511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456916992,"num_objects":6972,"num_object_clones":0,"num_object_copies":20916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6972,"num_write_kb":446208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":339,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.789183+0000","last_change":"2024-09-21T10:18:12.636851+0000","last_active":"2024-09-21T10:24:14.789183+0000","last_peered":"2024-09-21T10:24:14.789183+0000","last_clean":"2024-09-21T10:24:14.789183+0000","last_became_active":"2024-09-21T10:18:12.636513+0000","last_became_peered":"2024-09-21T10:18:12.636513+0000","last_unstale":"2024-09-21T10:24:14.789183+0000","last_undegraded":"2024-09-21T10:24:14.789183+0000","last_fullsized":"2024-09-21T10:24:14.789183+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:52.361137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:52.361137+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:21.025735+0000","scrub_duration":0.001255358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":748,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.833265+0000","last_change":"2024-09-21T10:09:23.837859+0000","last_active":"2024-09-21T10:24:15.833265+0000","last_peered":"2024-09-21T10:24:15.833265+0000","last_clean":"2024-09-21T10:24:15.833265+0000","last_became_active":"2024-09-21T10:09:23.837488+0000","last_became_peered":"2024-09-21T10:09:23.837488+0000","last_unstale":"2024-09-21T10:24:15.833265+0000","last_undegraded":"2024-09-21T10:24:15.833265+0000","last_fullsized":"2024-09-21T10:24:15.833265+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:25.215501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":193,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.788900+0000","last_change":"2024-09-21T10:22:19.647946+0000","last_active":"2024-09-21T10:24:14.788900+0000","last_peered":"2024-09-21T10:24:14.788900+0000","last_clean":"2024-09-21T10:24:14.788900+0000","last_became_active":"2024-09-21T10:22:19.647057+0000","last_became_peered":"2024-09-21T10:22:19.647057+0000","last_unstale":"2024-09-21T10:24:14.788900+0000","last_undegraded":"2024-09-21T10:24:14.788900+0000","last_fullsized":"2024-09-21T10:24:14.788900+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:06.839922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":742,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.836303+0000","last_change":"2024-09-21T10:10:57.395073+0000","last_active":"2024-09-21T10:24:15.836303+0000","last_peered":"2024-09-21T10:24:15.836303+0000","last_clean":"2024-09-21T10:24:15.836303+0000","last_became_active":"2024-09-21T10:10:57.394674+0000","last_became_peered":"2024-09-21T10:10:57.394674+0000","last_unstale":"2024-09-21T10:24:15.836303+0000","last_undegraded":"2024-09-21T10:24:15.836303+0000","last_fullsized":"2024-09-21T10:24:15.836303+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:48:48.912649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.e","version":"425'7164","reported_seq":7203,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:23.835982+0000","last_change":"2024-09-21T10:22:44.821540+0000","last_active":"2024-09-21T10:24:23.835982+0000","last_peered":"2024-09-21T10:24:23.835982+0000","last_clean":"2024-09-21T10:24:23.835982+0000","last_became_active":"2024-09-21T10:22:44.821263+0000","last_became_peered":"2024-09-21T10:22:44.821263+0000","last_unstale":"2024-09-21T10:24:23.835982+0000","last_undegraded":"2024-09-21T10:24:23.835982+0000","last_fullsized":"2024-09-21T10:24:23.835982+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":7164,"log_dups_size":0,"ondisk_log_size":7164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:05:50.494438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":469499904,"num_objects":7164,"num_object_clones":0,"num_object_copies":21492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7164,"num_write_kb":458496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":613,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.830496+0000","last_change":"2024-09-21T10:10:57.398437+0000","last_active":"2024-09-21T10:24:15.830496+0000","last_peered":"2024-09-21T10:24:15.830496+0000","last_clean":"2024-09-21T10:24:15.830496+0000","last_became_active":"2024-09-21T10:10:57.397441+0000","last_became_peered":"2024-09-21T10:10:57.397441+0000","last_unstale":"2024-09-21T10:24:15.830496+0000","last_undegraded":"2024-09-21T10:24:15.830496+0000","last_fullsized":"2024-09-21T10:24:15.830496+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:04.043917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:04.043917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:17.580110+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":331,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.821765+0000","last_change":"2024-09-21T10:22:19.665067+0000","last_active":"2024-09-21T10:24:15.821765+0000","last_peered":"2024-09-21T10:24:15.821765+0000","last_clean":"2024-09-21T10:24:15.821765+0000","last_became_active":"2024-09-21T10:22:19.651363+0000","last_became_peered":"2024-09-21T10:22:19.651363+0000","last_unstale":"2024-09-21T10:24:15.821765+0000","last_undegraded":"2024-09-21T10:24:15.821765+0000","last_fullsized":"2024-09-21T10:24:15.821765+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:29.173718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":240,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.831485+0000","last_change":"2024-09-21T10:22:19.646459+0000","last_active":"2024-09-21T10:24:15.831485+0000","last_peered":"2024-09-21T10:24:15.831485+0000","last_clean":"2024-09-21T10:24:15.831485+0000","last_became_active":"2024-09-21T10:22:19.645956+0000","last_became_peered":"2024-09-21T10:22:19.645956+0000","last_unstale":"2024-09-21T10:24:15.831485+0000","last_undegraded":"2024-09-21T10:24:15.831485+0000","last_fullsized":"2024-09-21T10:24:15.831485+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:11:56.923595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":574,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.821173+0000","last_change":"2024-09-21T10:14:05.477875+0000","last_active":"2024-09-21T10:24:15.821173+0000","last_peered":"2024-09-21T10:24:15.821173+0000","last_clean":"2024-09-21T10:24:15.821173+0000","last_became_active":"2024-09-21T10:14:05.477192+0000","last_became_peered":"2024-09-21T10:14:05.477192+0000","last_unstale":"2024-09-21T10:24:15.821173+0000","last_undegraded":"2024-09-21T10:24:15.821173+0000","last_fullsized":"2024-09-21T10:24:15.821173+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:19:47.345673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.8","version":"425'6988","reported_seq":7027,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:23.027317+0000","last_change":"2024-09-21T10:22:44.826294+0000","last_active":"2024-09-21T10:24:23.027317+0000","last_peered":"2024-09-21T10:24:23.027317+0000","last_clean":"2024-09-21T10:24:23.027317+0000","last_became_active":"2024-09-21T10:22:44.825917+0000","last_became_peered":"2024-09-21T10:22:44.825917+0000","last_unstale":"2024-09-21T10:24:23.027317+0000","last_undegraded":"2024-09-21T10:24:23.027317+0000","last_fullsized":"2024-09-21T10:24:23.027317+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":6988,"log_dups_size":0,"ondisk_log_size":6988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:06:27.760082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457965568,"num_objects":6988,"num_object_clones":0,"num_object_copies":20964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6988,"num_write_kb":447232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":32,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.790657+0000","last_change":"2024-09-21T10:22:19.649129+0000","last_active":"2024-09-21T10:24:14.790657+0000","last_peered":"2024-09-21T10:24:14.790657+0000","last_clean":"2024-09-21T10:24:14.790657+0000","last_became_active":"2024-09-21T10:22:19.648779+0000","last_became_peered":"2024-09-21T10:22:19.648779+0000","last_unstale":"2024-09-21T10:24:14.790657+0000","last_undegraded":"2024-09-21T10:24:14.790657+0000","last_fullsized":"2024-09-21T10:24:14.790657+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:21:47.185890+0000","last_deep_scrub":"77'100","last_deep_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_clean_scrub_stamp":"2024-09-21T10:21:47.185890+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:44:01.608891+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":417,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.828718+0000","last_change":"2024-09-21T10:15:01.489326+0000","last_active":"2024-09-21T10:24:15.828718+0000","last_peered":"2024-09-21T10:24:15.828718+0000","last_clean":"2024-09-21T10:24:15.828718+0000","last_became_active":"2024-09-21T10:15:01.489070+0000","last_became_peered":"2024-09-21T10:15:01.489070+0000","last_unstale":"2024-09-21T10:24:15.828718+0000","last_undegraded":"2024-09-21T10:24:15.828718+0000","last_fullsized":"2024-09-21T10:24:15.828718+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:27:41.788029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":630,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.821492+0000","last_change":"2024-09-21T10:14:05.475970+0000","last_active":"2024-09-21T10:24:15.821492+0000","last_peered":"2024-09-21T10:24:15.821492+0000","last_clean":"2024-09-21T10:24:15.821492+0000","last_became_active":"2024-09-21T10:14:05.475331+0000","last_became_peered":"2024-09-21T10:14:05.475331+0000","last_unstale":"2024-09-21T10:24:15.821492+0000","last_undegraded":"2024-09-21T10:24:15.821492+0000","last_fullsized":"2024-09-21T10:24:15.821492+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:17.069736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":566,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.822059+0000","last_change":"2024-09-21T10:18:13.039172+0000","last_active":"2024-09-21T10:24:15.822059+0000","last_peered":"2024-09-21T10:24:15.822059+0000","last_clean":"2024-09-21T10:24:15.822059+0000","last_became_active":"2024-09-21T10:18:13.038756+0000","last_became_peered":"2024-09-21T10:18:13.038756+0000","last_unstale":"2024-09-21T10:24:15.822059+0000","last_undegraded":"2024-09-21T10:24:15.822059+0000","last_fullsized":"2024-09-21T10:24:15.822059+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:06.668926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.9","version":"425'7062","reported_seq":7101,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:22.638752+0000","last_change":"2024-09-21T10:22:44.830660+0000","last_active":"2024-09-21T10:24:22.638752+0000","last_peered":"2024-09-21T10:24:22.638752+0000","last_clean":"2024-09-21T10:24:22.638752+0000","last_became_active":"2024-09-21T10:22:44.830357+0000","last_became_peered":"2024-09-21T10:22:44.830357+0000","last_unstale":"2024-09-21T10:24:22.638752+0000","last_undegraded":"2024-09-21T10:24:22.638752+0000","last_fullsized":"2024-09-21T10:24:22.638752+0000","mapping_epoch":411,"log_start":"415'300","ondisk_log_start":"415'300","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":6762,"log_dups_size":0,"ondisk_log_size":6762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:55.495054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462815232,"num_objects":7062,"num_object_clones":0,"num_object_copies":21186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7062,"num_write_kb":451968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":32,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.790771+0000","last_change":"2024-09-21T10:22:19.645389+0000","last_active":"2024-09-21T10:24:14.790771+0000","last_peered":"2024-09-21T10:24:14.790771+0000","last_clean":"2024-09-21T10:24:14.790771+0000","last_became_active":"2024-09-21T10:22:19.644671+0000","last_became_peered":"2024-09-21T10:22:19.644671+0000","last_unstale":"2024-09-21T10:24:14.790771+0000","last_undegraded":"2024-09-21T10:24:14.790771+0000","last_fullsized":"2024-09-21T10:24:14.790771+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:20:01.397007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:20:01.397007+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:45.545962+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":562,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.821378+0000","last_change":"2024-09-21T10:14:05.478152+0000","last_active":"2024-09-21T10:24:15.821378+0000","last_peered":"2024-09-21T10:24:15.821378+0000","last_clean":"2024-09-21T10:24:15.821378+0000","last_became_active":"2024-09-21T10:14:05.477547+0000","last_became_peered":"2024-09-21T10:14:05.477547+0000","last_unstale":"2024-09-21T10:24:15.821378+0000","last_undegraded":"2024-09-21T10:24:15.821378+0000","last_fullsized":"2024-09-21T10:24:15.821378+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:34.646046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":319,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.789741+0000","last_change":"2024-09-21T10:14:30.708090+0000","last_active":"2024-09-21T10:24:14.789741+0000","last_peered":"2024-09-21T10:24:14.789741+0000","last_clean":"2024-09-21T10:24:14.789741+0000","last_became_active":"2024-09-21T10:14:30.707266+0000","last_became_peered":"2024-09-21T10:14:30.707266+0000","last_unstale":"2024-09-21T10:24:14.789741+0000","last_undegraded":"2024-09-21T10:24:14.789741+0000","last_fullsized":"2024-09-21T10:24:14.789741+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:38.474537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":625,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.831603+0000","last_change":"2024-09-21T10:10:57.397251+0000","last_active":"2024-09-21T10:24:15.831603+0000","last_peered":"2024-09-21T10:24:15.831603+0000","last_clean":"2024-09-21T10:24:15.831603+0000","last_became_active":"2024-09-21T10:10:57.396271+0000","last_became_peered":"2024-09-21T10:10:57.396271+0000","last_unstale":"2024-09-21T10:24:15.831603+0000","last_undegraded":"2024-09-21T10:24:15.831603+0000","last_fullsized":"2024-09-21T10:24:15.831603+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:17.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"425'7116","reported_seq":7154,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:23.856471+0000","last_change":"2024-09-21T10:22:44.830746+0000","last_active":"2024-09-21T10:24:23.856471+0000","last_peered":"2024-09-21T10:24:23.856471+0000","last_clean":"2024-09-21T10:24:23.856471+0000","last_became_active":"2024-09-21T10:22:44.830482+0000","last_became_peered":"2024-09-21T10:22:44.830482+0000","last_unstale":"2024-09-21T10:24:23.856471+0000","last_undegraded":"2024-09-21T10:24:23.856471+0000","last_fullsized":"2024-09-21T10:24:23.856471+0000","mapping_epoch":411,"log_start":"415'100","ondisk_log_start":"415'100","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":7016,"log_dups_size":0,"ondisk_log_size":7016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:43.196504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466354176,"num_objects":7116,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7116,"num_write_kb":455424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":248,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.835227+0000","last_change":"2024-09-21T10:22:19.646493+0000","last_active":"2024-09-21T10:24:15.835227+0000","last_peered":"2024-09-21T10:24:15.835227+0000","last_clean":"2024-09-21T10:24:15.835227+0000","last_became_active":"2024-09-21T10:22:19.646152+0000","last_became_peered":"2024-09-21T10:22:19.646152+0000","last_unstale":"2024-09-21T10:24:15.835227+0000","last_undegraded":"2024-09-21T10:24:15.835227+0000","last_fullsized":"2024-09-21T10:24:15.835227+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:59.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:59.099852+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:17:29.824712+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":717,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.834745+0000","last_change":"2024-09-21T10:09:23.841186+0000","last_active":"2024-09-21T10:24:15.834745+0000","last_peered":"2024-09-21T10:24:15.834745+0000","last_clean":"2024-09-21T10:24:15.834745+0000","last_became_active":"2024-09-21T10:09:23.840854+0000","last_became_peered":"2024-09-21T10:09:23.840854+0000","last_unstale":"2024-09-21T10:24:15.834745+0000","last_undegraded":"2024-09-21T10:24:15.834745+0000","last_fullsized":"2024-09-21T10:24:15.834745+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:29.666358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":583,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.821448+0000","last_change":"2024-09-21T10:22:19.644169+0000","last_active":"2024-09-21T10:24:15.821448+0000","last_peered":"2024-09-21T10:24:15.821448+0000","last_clean":"2024-09-21T10:24:15.821448+0000","last_became_active":"2024-09-21T10:22:19.643917+0000","last_became_peered":"2024-09-21T10:22:19.643917+0000","last_unstale":"2024-09-21T10:24:15.821448+0000","last_undegraded":"2024-09-21T10:24:15.821448+0000","last_fullsized":"2024-09-21T10:24:15.821448+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:28.976858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":714,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.827887+0000","last_change":"2024-09-21T10:09:49.926548+0000","last_active":"2024-09-21T10:24:15.827887+0000","last_peered":"2024-09-21T10:24:15.827887+0000","last_clean":"2024-09-21T10:24:15.827887+0000","last_became_active":"2024-09-21T10:09:49.926255+0000","last_became_peered":"2024-09-21T10:09:49.926255+0000","last_unstale":"2024-09-21T10:24:15.827887+0000","last_undegraded":"2024-09-21T10:24:15.827887+0000","last_fullsized":"2024-09-21T10:24:15.827887+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:49.855878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.b","version":"425'6994","reported_seq":7032,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:23.033718+0000","last_change":"2024-09-21T10:22:44.825973+0000","last_active":"2024-09-21T10:24:23.033718+0000","last_peered":"2024-09-21T10:24:23.033718+0000","last_clean":"2024-09-21T10:24:23.033718+0000","last_became_active":"2024-09-21T10:22:44.825697+0000","last_became_peered":"2024-09-21T10:22:44.825697+0000","last_unstale":"2024-09-21T10:24:23.033718+0000","last_undegraded":"2024-09-21T10:24:23.033718+0000","last_fullsized":"2024-09-21T10:24:23.033718+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":6994,"log_dups_size":0,"ondisk_log_size":6994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:00:32.419547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458358784,"num_objects":6994,"num_object_clones":0,"num_object_copies":20982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6994,"num_write_kb":447616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":517,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.828173+0000","last_change":"2024-09-21T10:10:57.390804+0000","last_active":"2024-09-21T10:24:15.828173+0000","last_peered":"2024-09-21T10:24:15.828173+0000","last_clean":"2024-09-21T10:24:15.828173+0000","last_became_active":"2024-09-21T10:10:57.390323+0000","last_became_peered":"2024-09-21T10:10:57.390323+0000","last_unstale":"2024-09-21T10:24:15.828173+0000","last_undegraded":"2024-09-21T10:24:15.828173+0000","last_fullsized":"2024-09-21T10:24:15.828173+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.005477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.005477+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:40.776053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":370,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.845806+0000","last_change":"2024-09-21T10:14:30.718942+0000","last_active":"2024-09-21T10:24:15.845806+0000","last_peered":"2024-09-21T10:24:15.845806+0000","last_clean":"2024-09-21T10:24:15.845806+0000","last_became_active":"2024-09-21T10:14:30.709129+0000","last_became_peered":"2024-09-21T10:14:30.709129+0000","last_unstale":"2024-09-21T10:24:15.845806+0000","last_undegraded":"2024-09-21T10:24:15.845806+0000","last_fullsized":"2024-09-21T10:24:15.845806+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:18.171064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":32,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.790560+0000","last_change":"2024-09-21T10:22:19.646015+0000","last_active":"2024-09-21T10:24:14.790560+0000","last_peered":"2024-09-21T10:24:14.790560+0000","last_clean":"2024-09-21T10:24:14.790560+0000","last_became_active":"2024-09-21T10:22:19.645608+0000","last_became_peered":"2024-09-21T10:22:19.645608+0000","last_unstale":"2024-09-21T10:24:14.790560+0000","last_undegraded":"2024-09-21T10:24:14.790560+0000","last_fullsized":"2024-09-21T10:24:14.790560+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:14.920346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":32,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.790327+0000","last_change":"2024-09-21T10:22:19.649811+0000","last_active":"2024-09-21T10:24:14.790327+0000","last_peered":"2024-09-21T10:24:14.790327+0000","last_clean":"2024-09-21T10:24:14.790327+0000","last_became_active":"2024-09-21T10:22:19.649314+0000","last_became_peered":"2024-09-21T10:22:19.649314+0000","last_unstale":"2024-09-21T10:24:14.790327+0000","last_undegraded":"2024-09-21T10:24:14.790327+0000","last_fullsized":"2024-09-21T10:24:14.790327+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:36.322837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.790485+0000","last_change":"2024-09-21T10:22:19.651289+0000","last_active":"2024-09-21T10:24:14.790485+0000","last_peered":"2024-09-21T10:24:14.790485+0000","last_clean":"2024-09-21T10:24:14.790485+0000","last_became_active":"2024-09-21T10:22:19.650947+0000","last_became_peered":"2024-09-21T10:22:19.650947+0000","last_unstale":"2024-09-21T10:24:14.790485+0000","last_undegraded":"2024-09-21T10:24:14.790485+0000","last_fullsized":"2024-09-21T10:24:14.790485+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:54:44.845308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":289,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.789536+0000","last_change":"2024-09-21T10:22:19.662381+0000","last_active":"2024-09-21T10:24:14.789536+0000","last_peered":"2024-09-21T10:24:14.789536+0000","last_clean":"2024-09-21T10:24:14.789536+0000","last_became_active":"2024-09-21T10:22:19.646849+0000","last_became_peered":"2024-09-21T10:22:19.646849+0000","last_unstale":"2024-09-21T10:24:14.789536+0000","last_undegraded":"2024-09-21T10:24:14.789536+0000","last_fullsized":"2024-09-21T10:24:14.789536+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:38:55.399112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14232,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":635,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.821859+0000","last_change":"2024-09-21T10:22:19.644482+0000","last_active":"2024-09-21T10:24:15.821859+0000","last_peered":"2024-09-21T10:24:15.821859+0000","last_clean":"2024-09-21T10:24:15.821859+0000","last_became_active":"2024-09-21T10:22:19.644080+0000","last_became_peered":"2024-09-21T10:22:19.644080+0000","last_unstale":"2024-09-21T10:24:15.821859+0000","last_undegraded":"2024-09-21T10:24:15.821859+0000","last_fullsized":"2024-09-21T10:24:15.821859+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:42:37.848739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":364,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.835639+0000","last_change":"2024-09-21T10:14:30.713873+0000","last_active":"2024-09-21T10:24:15.835639+0000","last_peered":"2024-09-21T10:24:15.835639+0000","last_clean":"2024-09-21T10:24:15.835639+0000","last_became_active":"2024-09-21T10:14:30.703149+0000","last_became_peered":"2024-09-21T10:14:30.703149+0000","last_unstale":"2024-09-21T10:24:15.835639+0000","last_undegraded":"2024-09-21T10:24:15.835639+0000","last_fullsized":"2024-09-21T10:24:15.835639+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:26.453902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'100","reported_seq":516,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.835292+0000","last_change":"2024-09-21T10:18:12.633959+0000","last_active":"2024-09-21T10:24:15.835292+0000","last_peered":"2024-09-21T10:24:15.835292+0000","last_clean":"2024-09-21T10:24:15.835292+0000","last_became_active":"2024-09-21T10:18:12.633591+0000","last_became_peered":"2024-09-21T10:18:12.633591+0000","last_unstale":"2024-09-21T10:24:15.835292+0000","last_undegraded":"2024-09-21T10:24:15.835292+0000","last_fullsized":"2024-09-21T10:24:15.835292+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.014993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.014993+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:29:16.343472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":338,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.789656+0000","last_change":"2024-09-21T10:15:01.483814+0000","last_active":"2024-09-21T10:24:14.789656+0000","last_peered":"2024-09-21T10:24:14.789656+0000","last_clean":"2024-09-21T10:24:14.789656+0000","last_became_active":"2024-09-21T10:15:01.483522+0000","last_became_peered":"2024-09-21T10:15:01.483522+0000","last_unstale":"2024-09-21T10:24:14.789656+0000","last_undegraded":"2024-09-21T10:24:14.789656+0000","last_fullsized":"2024-09-21T10:24:14.789656+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:13.632066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":713,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.828265+0000","last_change":"2024-09-21T10:22:19.645758+0000","last_active":"2024-09-21T10:24:15.828265+0000","last_peered":"2024-09-21T10:24:15.828265+0000","last_clean":"2024-09-21T10:24:15.828265+0000","last_became_active":"2024-09-21T10:22:19.645357+0000","last_became_peered":"2024-09-21T10:22:19.645357+0000","last_unstale":"2024-09-21T10:24:15.828265+0000","last_undegraded":"2024-09-21T10:24:15.828265+0000","last_fullsized":"2024-09-21T10:24:15.828265+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:12:14.337839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":599,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.844688+0000","last_change":"2024-09-21T10:10:57.397755+0000","last_active":"2024-09-21T10:24:15.844688+0000","last_peered":"2024-09-21T10:24:15.844688+0000","last_clean":"2024-09-21T10:24:15.844688+0000","last_became_active":"2024-09-21T10:10:57.397249+0000","last_became_peered":"2024-09-21T10:10:57.397249+0000","last_unstale":"2024-09-21T10:24:15.844688+0000","last_undegraded":"2024-09-21T10:24:15.844688+0000","last_fullsized":"2024-09-21T10:24:15.844688+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:40.813503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":353,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.820716+0000","last_change":"2024-09-21T10:22:19.651921+0000","last_active":"2024-09-21T10:24:15.820716+0000","last_peered":"2024-09-21T10:24:15.820716+0000","last_clean":"2024-09-21T10:24:15.820716+0000","last_became_active":"2024-09-21T10:22:19.651582+0000","last_became_peered":"2024-09-21T10:22:19.651582+0000","last_unstale":"2024-09-21T10:24:15.820716+0000","last_undegraded":"2024-09-21T10:24:15.820716+0000","last_fullsized":"2024-09-21T10:24:15.820716+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:00:14.778412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":32,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.790810+0000","last_change":"2024-09-21T10:22:19.649280+0000","last_active":"2024-09-21T10:24:14.790810+0000","last_peered":"2024-09-21T10:24:14.790810+0000","last_clean":"2024-09-21T10:24:14.790810+0000","last_became_active":"2024-09-21T10:22:19.648950+0000","last_became_peered":"2024-09-21T10:22:19.648950+0000","last_unstale":"2024-09-21T10:24:14.790810+0000","last_undegraded":"2024-09-21T10:24:14.790810+0000","last_fullsized":"2024-09-21T10:24:14.790810+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:06:32.778203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":719,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.827980+0000","last_change":"2024-09-21T10:09:49.928053+0000","last_active":"2024-09-21T10:24:15.827980+0000","last_peered":"2024-09-21T10:24:15.827980+0000","last_clean":"2024-09-21T10:24:15.827980+0000","last_became_active":"2024-09-21T10:09:49.927231+0000","last_became_peered":"2024-09-21T10:09:49.927231+0000","last_unstale":"2024-09-21T10:24:15.827980+0000","last_undegraded":"2024-09-21T10:24:15.827980+0000","last_fullsized":"2024-09-21T10:24:15.827980+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:26.987553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":632,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.845951+0000","last_change":"2024-09-21T10:09:23.840717+0000","last_active":"2024-09-21T10:24:15.845951+0000","last_peered":"2024-09-21T10:24:15.845951+0000","last_clean":"2024-09-21T10:24:15.845951+0000","last_became_active":"2024-09-21T10:09:23.840323+0000","last_became_peered":"2024-09-21T10:09:23.840323+0000","last_unstale":"2024-09-21T10:24:15.845951+0000","last_undegraded":"2024-09-21T10:24:15.845951+0000","last_fullsized":"2024-09-21T10:24:15.845951+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:55.769770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":352,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.833334+0000","last_change":"2024-09-21T10:18:12.632498+0000","last_active":"2024-09-21T10:24:15.833334+0000","last_peered":"2024-09-21T10:24:15.833334+0000","last_clean":"2024-09-21T10:24:15.833334+0000","last_became_active":"2024-09-21T10:18:12.631648+0000","last_became_peered":"2024-09-21T10:18:12.631648+0000","last_unstale":"2024-09-21T10:24:15.833334+0000","last_undegraded":"2024-09-21T10:24:15.833334+0000","last_fullsized":"2024-09-21T10:24:15.833334+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:17.592618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":697,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.828568+0000","last_change":"2024-09-21T10:14:30.702101+0000","last_active":"2024-09-21T10:24:15.828568+0000","last_peered":"2024-09-21T10:24:15.828568+0000","last_clean":"2024-09-21T10:24:15.828568+0000","last_became_active":"2024-09-21T10:14:30.701783+0000","last_became_peered":"2024-09-21T10:14:30.701783+0000","last_unstale":"2024-09-21T10:24:15.828568+0000","last_undegraded":"2024-09-21T10:24:15.828568+0000","last_fullsized":"2024-09-21T10:24:15.828568+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:13.087010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":81,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.822181+0000","last_change":"2024-09-21T10:22:08.118073+0000","last_active":"2024-09-21T10:24:15.822181+0000","last_peered":"2024-09-21T10:24:15.822181+0000","last_clean":"2024-09-21T10:24:15.822181+0000","last_became_active":"2024-09-21T10:22:08.117679+0000","last_became_peered":"2024-09-21T10:22:08.117679+0000","last_unstale":"2024-09-21T10:24:15.822181+0000","last_undegraded":"2024-09-21T10:24:15.822181+0000","last_fullsized":"2024-09-21T10:24:15.822181+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:53.070579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":737,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.835715+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:24:15.835715+0000","last_peered":"2024-09-21T10:24:15.835715+0000","last_clean":"2024-09-21T10:24:15.835715+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:24:15.835715+0000","last_undegraded":"2024-09-21T10:24:15.835715+0000","last_fullsized":"2024-09-21T10:24:15.835715+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":361,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.831269+0000","last_change":"2024-09-21T10:14:30.702414+0000","last_active":"2024-09-21T10:24:15.831269+0000","last_peered":"2024-09-21T10:24:15.831269+0000","last_clean":"2024-09-21T10:24:15.831269+0000","last_became_active":"2024-09-21T10:14:30.702009+0000","last_became_peered":"2024-09-21T10:14:30.702009+0000","last_unstale":"2024-09-21T10:24:15.831269+0000","last_undegraded":"2024-09-21T10:24:15.831269+0000","last_fullsized":"2024-09-21T10:24:15.831269+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:50:48.396082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":487,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.838224+0000","last_change":"2024-09-21T10:22:08.122333+0000","last_active":"2024-09-21T10:24:15.838224+0000","last_peered":"2024-09-21T10:24:15.838224+0000","last_clean":"2024-09-21T10:24:15.838224+0000","last_became_active":"2024-09-21T10:22:08.121898+0000","last_became_peered":"2024-09-21T10:22:08.121898+0000","last_unstale":"2024-09-21T10:24:15.838224+0000","last_undegraded":"2024-09-21T10:24:15.838224+0000","last_fullsized":"2024-09-21T10:24:15.838224+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:36:32.263089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":92,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.821628+0000","last_change":"2024-09-21T10:22:08.119919+0000","last_active":"2024-09-21T10:24:15.821628+0000","last_peered":"2024-09-21T10:24:15.821628+0000","last_clean":"2024-09-21T10:24:15.821628+0000","last_became_active":"2024-09-21T10:22:08.119598+0000","last_became_peered":"2024-09-21T10:22:08.119598+0000","last_unstale":"2024-09-21T10:24:15.821628+0000","last_undegraded":"2024-09-21T10:24:15.821628+0000","last_fullsized":"2024-09-21T10:24:15.821628+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:14:10.751524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":578,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.832412+0000","last_change":"2024-09-21T10:22:08.117372+0000","last_active":"2024-09-21T10:24:15.832412+0000","last_peered":"2024-09-21T10:24:15.832412+0000","last_clean":"2024-09-21T10:24:15.832412+0000","last_became_active":"2024-09-21T10:22:08.116632+0000","last_became_peered":"2024-09-21T10:22:08.116632+0000","last_unstale":"2024-09-21T10:24:15.832412+0000","last_undegraded":"2024-09-21T10:24:15.832412+0000","last_fullsized":"2024-09-21T10:24:15.832412+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:38.062747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":36,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.790778+0000","last_change":"2024-09-21T10:22:19.666163+0000","last_active":"2024-09-21T10:24:14.790778+0000","last_peered":"2024-09-21T10:24:14.790778+0000","last_clean":"2024-09-21T10:24:14.790778+0000","last_became_active":"2024-09-21T10:22:19.651579+0000","last_became_peered":"2024-09-21T10:22:19.651579+0000","last_unstale":"2024-09-21T10:24:14.790778+0000","last_undegraded":"2024-09-21T10:24:14.790778+0000","last_fullsized":"2024-09-21T10:24:14.790778+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:42.750730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":720,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.828724+0000","last_change":"2024-09-21T10:09:23.840467+0000","last_active":"2024-09-21T10:24:15.828724+0000","last_peered":"2024-09-21T10:24:15.828724+0000","last_clean":"2024-09-21T10:24:15.828724+0000","last_became_active":"2024-09-21T10:09:23.840177+0000","last_became_peered":"2024-09-21T10:09:23.840177+0000","last_unstale":"2024-09-21T10:24:15.828724+0000","last_undegraded":"2024-09-21T10:24:15.828724+0000","last_fullsized":"2024-09-21T10:24:15.828724+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:17.887307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":725,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.828813+0000","last_change":"2024-09-21T10:22:19.648325+0000","last_active":"2024-09-21T10:24:15.828813+0000","last_peered":"2024-09-21T10:24:15.828813+0000","last_clean":"2024-09-21T10:24:15.828813+0000","last_became_active":"2024-09-21T10:22:19.648008+0000","last_became_peered":"2024-09-21T10:22:19.648008+0000","last_unstale":"2024-09-21T10:24:15.828813+0000","last_undegraded":"2024-09-21T10:24:15.828813+0000","last_fullsized":"2024-09-21T10:24:15.828813+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:25.881077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":32,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-09-21T10:24:14.790372+0000","last_change":"2024-09-21T10:22:19.652069+0000","last_active":"2024-09-21T10:24:14.790372+0000","last_peered":"2024-09-21T10:24:14.790372+0000","last_clean":"2024-09-21T10:24:14.790372+0000","last_became_active":"2024-09-21T10:22:19.651081+0000","last_became_peered":"2024-09-21T10:22:19.651081+0000","last_unstale":"2024-09-21T10:24:14.790372+0000","last_undegraded":"2024-09-21T10:24:14.790372+0000","last_fullsized":"2024-09-21T10:24:14.790372+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:22:05.604610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":374,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.844541+0000","last_change":"2024-09-21T10:10:57.401588+0000","last_active":"2024-09-21T10:24:15.844541+0000","last_peered":"2024-09-21T10:24:15.844541+0000","last_clean":"2024-09-21T10:24:15.844541+0000","last_became_active":"2024-09-21T10:10:57.397724+0000","last_became_peered":"2024-09-21T10:10:57.397724+0000","last_unstale":"2024-09-21T10:24:15.844541+0000","last_undegraded":"2024-09-21T10:24:15.844541+0000","last_fullsized":"2024-09-21T10:24:15.844541+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:47.843630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":16740,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26580,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":14,"num_pg":16,"stat_sum":{"num_bytes":7369719808,"num_objects":112453,"num_object_clones":0,"num_object_copies":337359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":112453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112453,"num_write_kb":7196992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22112894976,"data_stored":22112894976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111753,"ondisk_log_size":111753,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626984,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3275716,"kb_used_data":1788412,"kb_used_omap":1,"kb_used_meta":1487294,"kb_avail":90469436,"statfs":{"total":95995035648,"available":92640702464,"internally_reserved":0,"allocated":1831333888,"data_stored":1828754382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1522989400},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019442,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4192136,"kb_used_data":2693568,"kb_used_omap":1,"kb_used_meta":1498558,"kb_avail":89553016,"statfs":{"total":95995035648,"available":91702288384,"internally_reserved":0,"allocated":2758213632,"data_stored":2755630071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1534523736},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":245,"seq":1052266987701,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4162808,"kb_used_data":3588080,"kb_used_omap":8,"kb_used_meta":574711,"kb_avail":89582344,"statfs":{"total":95995035648,"available":91732320256,"internally_reserved":0,"allocated":3674193920,"data_stored":3671622489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8940,"internal_metadata":588504340},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,3,3,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":175,"seq":751619277050,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2262856,"kb_used_data":1333888,"kb_used_omap":12,"kb_used_meta":928947,"kb_avail":91482296,"statfs":{"total":95995035648,"available":93677871104,"internally_reserved":0,"allocated":1365901312,"data_stored":1363317747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12845,"internal_metadata":951242195},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823213,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3750520,"kb_used_data":2256880,"kb_used_omap":1,"kb_used_meta":1493630,"kb_avail":89994632,"statfs":{"total":95995035648,"available":92154503168,"internally_reserved":0,"allocated":2311045120,"data_stored":2308473642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1529477464},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879636,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4307000,"kb_used_data":3135152,"kb_used_omap":4,"kb_used_meta":1171835,"kb_avail":89438152,"statfs":{"total":95995035648,"available":91584667648,"internally_reserved":0,"allocated":3210395648,"data_stored":3207824251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":1199959484},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566477,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4399104,"kb_used_data":3165176,"kb_used_omap":6,"kb_used_meta":1233913,"kb_avail":89346048,"statfs":{"total":95995035648,"available":91490353152,"internally_reserved":0,"allocated":3241140224,"data_stored":3238561247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":1263527335},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860822,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4917680,"kb_used_data":3594664,"kb_used_omap":5,"kb_used_meta":1323002,"kb_avail":88827472,"statfs":{"total":95995035648,"available":90959331328,"internally_reserved":0,"allocated":3680935936,"data_stored":3678371793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":1354754287},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7236,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9976,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3688562688,"data_stored":3688562688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3687972864,"data_stored":3687972864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3248750592,"data_stored":3248750592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3210018816,"data_stored":3210018816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2769158144,"data_stored":2769158144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2312896512,"data_stored":2312896512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1835139072,"data_stored":1835139072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1360396288,"data_stored":1360396288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:24:29.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:24:29.636 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:29.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:29.737 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:29.757 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:24:29.757 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:24:29.764 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:29.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:29.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:29.904 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:29.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:30.006 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:30.036 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:30.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:30.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:30.179 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:30.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:30.265 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:30.281 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:30.305 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:30.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:30.336 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:30.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:30.419 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:24:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:24:30.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:24:30.463 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:30.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:30.565 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:30.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.601 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:30.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:30.611 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:24:30.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:30.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.751 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:30.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:30.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:30.902 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:30.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:30.971 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:24:31.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:31.041 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:31.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:31.143 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:31.171 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:31.216 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:24:31.274 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:31.302 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:31.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:31.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:31 smithi129 ceph-mon[88471]: pgmap v1494: 109 pgs: 109 active+clean; 7.0 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 85 B/s rd, 56 MiB/s wr, 977 op/s 2024-09-21T10:24:31.395 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:31 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/103946104' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:24:31.404 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:31.434 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:31.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:31.536 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:31.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.563 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:31.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:31.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:31 smithi107 ceph-mon[102221]: pgmap v1494: 109 pgs: 109 active+clean; 7.0 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 85 B/s rd, 56 MiB/s wr, 977 op/s 2024-09-21T10:24:31.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:31 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/103946104' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:24:31.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:31 smithi107 ceph-mon[105022]: pgmap v1494: 109 pgs: 109 active+clean; 7.0 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 85 B/s rd, 56 MiB/s wr, 977 op/s 2024-09-21T10:24:31.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:31 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/103946104' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:24:31.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:31.702 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:31.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:31.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:31.838 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:31.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:31.941 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:31.969 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:31.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:32.070 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:32.100 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:32.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:24:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:24:31] "GET /metrics HTTP/1.1" 200 36774 "" "Prometheus/2.51.0" 2024-09-21T10:24:32.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:32.241 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:32.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:32.343 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:32.367 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:32.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:32.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:32 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:32.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:32 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/103946104' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:24:32.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:32 smithi129 ceph-mon[88471]: osdmap e426: 8 total, 8 up, 8 in 2024-09-21T10:24:32.469 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:32.494 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:32.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:32 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:32.579 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:32 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/103946104' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:24:32.579 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:32 smithi107 ceph-mon[102221]: osdmap e426: 8 total, 8 up, 8 in 2024-09-21T10:24:32.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:32 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:32.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:32 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/103946104' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:24:32.579 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:32 smithi107 ceph-mon[105022]: osdmap e426: 8 total, 8 up, 8 in 2024-09-21T10:24:32.596 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:32.621 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:32.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:32.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:32.770 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:32.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:32.803 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:24:32.803 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:24:32.872 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:32.901 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:32.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:33.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:33.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.048 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:33.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:33.149 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:33.177 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:33.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:33.279 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:33.302 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:33.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:33 smithi107 ceph-mon[102221]: pgmap v1496: 109 pgs: 109 active+clean; 7.0 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 102 B/s rd, 47 MiB/s wr, 848 op/s 2024-09-21T10:24:33.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:33 smithi107 ceph-mon[105022]: pgmap v1496: 109 pgs: 109 active+clean; 7.0 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 102 B/s rd, 47 MiB/s wr, 848 op/s 2024-09-21T10:24:33.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:33.440 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:33.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:33.495 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:33 smithi129 ceph-mon[88471]: pgmap v1496: 109 pgs: 109 active+clean; 7.0 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 102 B/s rd, 47 MiB/s wr, 848 op/s 2024-09-21T10:24:33.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:24:33.541 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:33.611 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:33.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:33.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:33.749 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:33.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:33.851 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:33.892 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:24:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.893 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:33.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:33.994 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:34.022 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:34.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:34.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:34.164 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:34.265 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:34.290 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:34.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:34.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:34.425 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:34.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:34.526 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:34.553 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:34.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:34.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:34.699 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:34.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:34.801 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:34.837 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:34.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:34.939 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:34.967 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:34.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:35.069 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:35.091 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:35.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:35.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:35.241 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:35.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:35.273 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:35 smithi107 ceph-mon[102221]: pgmap v1497: 109 pgs: 109 active+clean; 6.8 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 102 B/s rd, 26 MiB/s wr, 857 op/s 2024-09-21T10:24:35.273 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:35 smithi107 ceph-mon[105022]: pgmap v1497: 109 pgs: 109 active+clean; 6.8 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 102 B/s rd, 26 MiB/s wr, 857 op/s 2024-09-21T10:24:35.344 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:35.371 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:35.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:35.414 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:35 smithi129 ceph-mon[88471]: pgmap v1497: 109 pgs: 109 active+clean; 6.8 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 102 B/s rd, 26 MiB/s wr, 857 op/s 2024-09-21T10:24:35.473 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:35.498 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:35.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:35.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:24:35.552 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:24:35.553 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:24:35.601 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:35.649 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:35.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:35.670 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:24:35.670 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:24:35.712 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:24:35.712 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:24:35.745 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:24:35.745 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:24:35.753 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:35.782 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:35.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:35.795 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:24:35.795 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:24:35.826 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:24:35.827 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:24:35.862 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:24:35.862 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:24:35.871 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:24:35.871 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:24:35.884 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:35.895 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:24:35.895 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:24:35.910 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:35.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:35.926 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:24:35.926 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:24:36.012 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:36.037 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:36.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:36.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:36.177 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:36.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:36.279 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:36.324 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:36.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:36.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:36.469 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:36.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:36.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:36.607 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:36.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:36.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:36 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3779388060' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:36.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:36 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3779388060' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:36.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:36 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3779388060' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:36.709 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:36.740 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:36.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:36.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:36.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.878 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:36.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:36.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:37.007 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:37.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:37.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:37.144 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:37.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:37.187 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:24:37.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:37.290 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:37.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:37.392 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:37.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:37 smithi129 ceph-mon[88471]: from='client.37431 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:24:37.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:37 smithi129 ceph-mon[88471]: pgmap v1498: 109 pgs: 109 active+clean; 6.7 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 102 B/s rd, 21 MiB/s wr, 858 op/s 2024-09-21T10:24:37.430 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:37.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:37.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:37.533 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:24:37.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.572 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:37.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:37.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:37 smithi107 ceph-mon[102221]: from='client.37431 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:24:37.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:37 smithi107 ceph-mon[102221]: pgmap v1498: 109 pgs: 109 active+clean; 6.7 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 102 B/s rd, 21 MiB/s wr, 858 op/s 2024-09-21T10:24:37.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:37 smithi107 ceph-mon[105022]: from='client.37431 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:24:37.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:37 smithi107 ceph-mon[105022]: pgmap v1498: 109 pgs: 109 active+clean; 6.7 GiB data, 30 GiB used, 685 GiB / 715 GiB avail; 102 B/s rd, 21 MiB/s wr, 858 op/s 2024-09-21T10:24:37.673 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:37.697 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:37.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:37.799 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:37.834 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:37.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:37.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:37.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.972 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:37.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:38.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:38.117 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:38.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:38.218 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:38.243 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:38.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:38.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:38.384 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:38.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:38.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:38 smithi129 ceph-mon[88471]: pgmap v1499: 109 pgs: 109 active+clean; 6.7 GiB data, 30 GiB used, 686 GiB / 715 GiB avail; 15 MiB/s wr, 856 op/s 2024-09-21T10:24:38.485 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:38.510 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:38.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:38.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:38 smithi107 ceph-mon[102221]: pgmap v1499: 109 pgs: 109 active+clean; 6.7 GiB data, 30 GiB used, 686 GiB / 715 GiB avail; 15 MiB/s wr, 856 op/s 2024-09-21T10:24:38.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:38 smithi107 ceph-mon[105022]: pgmap v1499: 109 pgs: 109 active+clean; 6.7 GiB data, 30 GiB used, 686 GiB / 715 GiB avail; 15 MiB/s wr, 856 op/s 2024-09-21T10:24:38.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:38.655 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:38.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:38.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:38.790 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:38.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:38.847 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1497,"stamp":"2024-09-21T10:24:33.550810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7297439717,"num_objects":111368,"num_object_clones":0,"num_object_copies":334104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111368,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120300,"num_write_kb":7410465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":1197514,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119995,"ondisk_log_size":119995,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":31662916,"kb_used_data":21627508,"kb_used_omap":43,"kb_used_meta":10034964,"kb_avail":718298300,"statfs":{"total":767960285184,"available":735537459200,"internally_reserved":0,"allocated":22146568192,"data_stored":22125932443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44223,"internal_metadata":10275803969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20774888,"num_objects":-316,"num_object_clones":0,"num_object_copies":-948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-316,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8578,"num_write_kb":264321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003280"},"pg_stats":[{"pgid":"1.11","version":"77'100","reported_seq":374,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.223913+0000","last_change":"2024-09-21T10:10:57.397786+0000","last_active":"2024-09-21T10:24:31.223913+0000","last_peered":"2024-09-21T10:24:31.223913+0000","last_clean":"2024-09-21T10:24:31.223913+0000","last_became_active":"2024-09-21T10:10:57.396742+0000","last_became_peered":"2024-09-21T10:10:57.396742+0000","last_unstale":"2024-09-21T10:24:31.223913+0000","last_undegraded":"2024-09-21T10:24:31.223913+0000","last_fullsized":"2024-09-21T10:24:31.223913+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:53.631923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:53.631923+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:48:41.225820+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":711,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.230773+0000","last_change":"2024-09-21T10:03:38.389577+0000","last_active":"2024-09-21T10:24:31.230773+0000","last_peered":"2024-09-21T10:24:31.230773+0000","last_clean":"2024-09-21T10:24:31.230773+0000","last_became_active":"2024-09-21T10:03:38.389292+0000","last_became_peered":"2024-09-21T10:03:38.389292+0000","last_unstale":"2024-09-21T10:24:31.230773+0000","last_undegraded":"2024-09-21T10:24:31.230773+0000","last_fullsized":"2024-09-21T10:24:31.230773+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:32.113472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":311,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.817254+0000","last_change":"2024-09-21T10:14:30.717695+0000","last_active":"2024-09-21T10:24:15.817254+0000","last_peered":"2024-09-21T10:24:15.817254+0000","last_clean":"2024-09-21T10:24:15.817254+0000","last_became_active":"2024-09-21T10:14:30.705952+0000","last_became_peered":"2024-09-21T10:14:30.705952+0000","last_unstale":"2024-09-21T10:24:15.817254+0000","last_undegraded":"2024-09-21T10:24:15.817254+0000","last_fullsized":"2024-09-21T10:24:15.817254+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:08:18.863430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'100","reported_seq":344,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.817221+0000","last_change":"2024-09-21T10:22:19.656201+0000","last_active":"2024-09-21T10:24:15.817221+0000","last_peered":"2024-09-21T10:24:15.817221+0000","last_clean":"2024-09-21T10:24:15.817221+0000","last_became_active":"2024-09-21T10:22:19.655644+0000","last_became_peered":"2024-09-21T10:22:19.655644+0000","last_unstale":"2024-09-21T10:24:15.817221+0000","last_undegraded":"2024-09-21T10:24:15.817221+0000","last_fullsized":"2024-09-21T10:24:15.817221+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.798577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.798577+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:58:58.525208+0000","scrub_duration":0.0023622640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":416,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.828573+0000","last_change":"2024-09-21T10:10:57.392782+0000","last_active":"2024-09-21T10:24:15.828573+0000","last_peered":"2024-09-21T10:24:15.828573+0000","last_clean":"2024-09-21T10:24:15.828573+0000","last_became_active":"2024-09-21T10:10:57.392305+0000","last_became_peered":"2024-09-21T10:10:57.392305+0000","last_unstale":"2024-09-21T10:24:15.828573+0000","last_undegraded":"2024-09-21T10:24:15.828573+0000","last_fullsized":"2024-09-21T10:24:15.828573+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:42:01.387622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":631,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.225250+0000","last_change":"2024-09-21T10:09:23.846703+0000","last_active":"2024-09-21T10:24:31.225250+0000","last_peered":"2024-09-21T10:24:31.225250+0000","last_clean":"2024-09-21T10:24:31.225250+0000","last_became_active":"2024-09-21T10:09:23.846415+0000","last_became_peered":"2024-09-21T10:09:23.846415+0000","last_unstale":"2024-09-21T10:24:31.225250+0000","last_undegraded":"2024-09-21T10:24:31.225250+0000","last_fullsized":"2024-09-21T10:24:31.225250+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:23.129285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'100","reported_seq":373,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.823608+0000","last_change":"2024-09-21T10:18:12.637168+0000","last_active":"2024-09-21T10:24:15.823608+0000","last_peered":"2024-09-21T10:24:15.823608+0000","last_clean":"2024-09-21T10:24:15.823608+0000","last_became_active":"2024-09-21T10:18:12.636410+0000","last_became_peered":"2024-09-21T10:18:12.636410+0000","last_unstale":"2024-09-21T10:24:15.823608+0000","last_undegraded":"2024-09-21T10:24:15.823608+0000","last_fullsized":"2024-09-21T10:24:15.823608+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.433227+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:34.403451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":357,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.224590+0000","last_change":"2024-09-21T10:09:49.922489+0000","last_active":"2024-09-21T10:24:31.224590+0000","last_peered":"2024-09-21T10:24:31.224590+0000","last_clean":"2024-09-21T10:24:31.224590+0000","last_became_active":"2024-09-21T10:09:49.921629+0000","last_became_peered":"2024-09-21T10:09:49.921629+0000","last_unstale":"2024-09-21T10:24:31.224590+0000","last_undegraded":"2024-09-21T10:24:31.224590+0000","last_fullsized":"2024-09-21T10:24:31.224590+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:50.611979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":327,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.817019+0000","last_change":"2024-09-21T10:22:19.646202+0000","last_active":"2024-09-21T10:24:15.817019+0000","last_peered":"2024-09-21T10:24:15.817019+0000","last_clean":"2024-09-21T10:24:15.817019+0000","last_became_active":"2024-09-21T10:22:19.645690+0000","last_became_peered":"2024-09-21T10:22:19.645690+0000","last_unstale":"2024-09-21T10:24:15.817019+0000","last_undegraded":"2024-09-21T10:24:15.817019+0000","last_fullsized":"2024-09-21T10:24:15.817019+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:44:01.509115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"77'100","reported_seq":276,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.822017+0000","last_change":"2024-09-21T10:22:19.644444+0000","last_active":"2024-09-21T10:24:15.822017+0000","last_peered":"2024-09-21T10:24:15.822017+0000","last_clean":"2024-09-21T10:24:15.822017+0000","last_became_active":"2024-09-21T10:22:19.644051+0000","last_became_peered":"2024-09-21T10:22:19.644051+0000","last_unstale":"2024-09-21T10:24:15.822017+0000","last_undegraded":"2024-09-21T10:24:15.822017+0000","last_fullsized":"2024-09-21T10:24:15.822017+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:15:02.841968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:15:02.841968+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:11.035208+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":710,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.226620+0000","last_change":"2024-09-21T10:03:38.381552+0000","last_active":"2024-09-21T10:24:31.226620+0000","last_peered":"2024-09-21T10:24:31.226620+0000","last_clean":"2024-09-21T10:24:31.226620+0000","last_became_active":"2024-09-21T10:03:38.380898+0000","last_became_peered":"2024-09-21T10:03:38.380898+0000","last_unstale":"2024-09-21T10:24:31.226620+0000","last_undegraded":"2024-09-21T10:24:31.226620+0000","last_fullsized":"2024-09-21T10:24:31.226620+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:21:53.409606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":43,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.829257+0000","last_change":"2024-09-21T10:22:19.662950+0000","last_active":"2024-09-21T10:24:15.829257+0000","last_peered":"2024-09-21T10:24:15.829257+0000","last_clean":"2024-09-21T10:24:15.829257+0000","last_became_active":"2024-09-21T10:22:19.649590+0000","last_became_peered":"2024-09-21T10:22:19.649590+0000","last_unstale":"2024-09-21T10:24:15.829257+0000","last_undegraded":"2024-09-21T10:24:15.829257+0000","last_fullsized":"2024-09-21T10:24:15.829257+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:38:15.512148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.2","version":"426'7506","reported_seq":7529,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:32.098630+0000","last_change":"2024-09-21T10:22:44.823473+0000","last_active":"2024-09-21T10:24:32.098630+0000","last_peered":"2024-09-21T10:24:32.098630+0000","last_clean":"2024-09-21T10:24:32.098630+0000","last_became_active":"2024-09-21T10:22:44.823115+0000","last_became_peered":"2024-09-21T10:22:44.823115+0000","last_unstale":"2024-09-21T10:24:32.098630+0000","last_undegraded":"2024-09-21T10:24:32.098630+0000","last_fullsized":"2024-09-21T10:24:32.098630+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":7506,"log_dups_size":0,"ondisk_log_size":7506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:55:04.796944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456130560,"num_objects":6960,"num_object_clones":0,"num_object_copies":20880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7506,"num_write_kb":462912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"77'100","reported_seq":319,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.827609+0000","last_change":"2024-09-21T10:14:30.708275+0000","last_active":"2024-09-21T10:24:15.827609+0000","last_peered":"2024-09-21T10:24:15.827609+0000","last_clean":"2024-09-21T10:24:15.827609+0000","last_became_active":"2024-09-21T10:14:30.707763+0000","last_became_peered":"2024-09-21T10:14:30.707763+0000","last_unstale":"2024-09-21T10:24:15.827609+0000","last_undegraded":"2024-09-21T10:24:15.827609+0000","last_fullsized":"2024-09-21T10:24:15.827609+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:55.259385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:55.259385+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:41.322447+0000","scrub_duration":0.0015667260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":354,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.817613+0000","last_change":"2024-09-21T10:14:30.720250+0000","last_active":"2024-09-21T10:24:15.817613+0000","last_peered":"2024-09-21T10:24:15.817613+0000","last_clean":"2024-09-21T10:24:15.817613+0000","last_became_active":"2024-09-21T10:14:30.707036+0000","last_became_peered":"2024-09-21T10:14:30.707036+0000","last_unstale":"2024-09-21T10:24:15.817613+0000","last_undegraded":"2024-09-21T10:24:15.817613+0000","last_fullsized":"2024-09-21T10:24:15.817613+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:17.438534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":417,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.816906+0000","last_change":"2024-09-21T10:22:19.650243+0000","last_active":"2024-09-21T10:24:15.816906+0000","last_peered":"2024-09-21T10:24:15.816906+0000","last_clean":"2024-09-21T10:24:15.816906+0000","last_became_active":"2024-09-21T10:22:19.648182+0000","last_became_peered":"2024-09-21T10:22:19.648182+0000","last_unstale":"2024-09-21T10:24:15.816906+0000","last_undegraded":"2024-09-21T10:24:15.816906+0000","last_fullsized":"2024-09-21T10:24:15.816906+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:59:48.769286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.3","version":"426'7539","reported_seq":7580,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:32.992077+0000","last_change":"2024-09-21T10:22:44.823653+0000","last_active":"2024-09-21T10:24:32.992077+0000","last_peered":"2024-09-21T10:24:32.992077+0000","last_clean":"2024-09-21T10:24:32.992077+0000","last_became_active":"2024-09-21T10:22:44.823268+0000","last_became_peered":"2024-09-21T10:22:44.823268+0000","last_unstale":"2024-09-21T10:24:32.992077+0000","last_undegraded":"2024-09-21T10:24:32.992077+0000","last_fullsized":"2024-09-21T10:24:32.992077+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":7539,"log_dups_size":0,"ondisk_log_size":7539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:35:38.400207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451477504,"num_objects":6889,"num_object_clones":0,"num_object_copies":20667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7539,"num_write_kb":461696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"77'100","reported_seq":544,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.231122+0000","last_change":"2024-09-21T10:22:19.667407+0000","last_active":"2024-09-21T10:24:31.231122+0000","last_peered":"2024-09-21T10:24:31.231122+0000","last_clean":"2024-09-21T10:24:31.231122+0000","last_became_active":"2024-09-21T10:22:19.647633+0000","last_became_peered":"2024-09-21T10:22:19.647633+0000","last_unstale":"2024-09-21T10:24:31.231122+0000","last_undegraded":"2024-09-21T10:24:31.231122+0000","last_fullsized":"2024-09-21T10:24:31.231122+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:45.981336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:45.981336+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:46.044625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":333,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.827415+0000","last_change":"2024-09-21T10:18:12.643154+0000","last_active":"2024-09-21T10:24:15.827415+0000","last_peered":"2024-09-21T10:24:15.827415+0000","last_clean":"2024-09-21T10:24:15.827415+0000","last_became_active":"2024-09-21T10:18:12.636287+0000","last_became_peered":"2024-09-21T10:18:12.636287+0000","last_unstale":"2024-09-21T10:24:15.827415+0000","last_undegraded":"2024-09-21T10:24:15.827415+0000","last_fullsized":"2024-09-21T10:24:15.827415+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:45:31.008309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":182,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.821039+0000","last_change":"2024-09-21T10:22:08.117935+0000","last_active":"2024-09-21T10:24:15.821039+0000","last_peered":"2024-09-21T10:24:15.821039+0000","last_clean":"2024-09-21T10:24:15.821039+0000","last_became_active":"2024-09-21T10:22:08.117613+0000","last_became_peered":"2024-09-21T10:22:08.117613+0000","last_unstale":"2024-09-21T10:24:15.821039+0000","last_undegraded":"2024-09-21T10:24:15.821039+0000","last_fullsized":"2024-09-21T10:24:15.821039+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:39.820626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.0","version":"425'7420","reported_seq":7458,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:28.876182+0000","last_change":"2024-09-21T10:22:44.830660+0000","last_active":"2024-09-21T10:24:28.876182+0000","last_peered":"2024-09-21T10:24:28.876182+0000","last_clean":"2024-09-21T10:24:28.876182+0000","last_became_active":"2024-09-21T10:22:44.830373+0000","last_became_peered":"2024-09-21T10:22:44.830373+0000","last_unstale":"2024-09-21T10:24:28.876182+0000","last_undegraded":"2024-09-21T10:24:28.876182+0000","last_fullsized":"2024-09-21T10:24:28.876182+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":7420,"log_dups_size":0,"ondisk_log_size":7420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:02.866984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472907776,"num_objects":7216,"num_object_clones":0,"num_object_copies":21648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7420,"num_write_kb":468352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.f","version":"77'100","reported_seq":33,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.829719+0000","last_change":"2024-09-21T10:22:19.657392+0000","last_active":"2024-09-21T10:24:15.829719+0000","last_peered":"2024-09-21T10:24:15.829719+0000","last_clean":"2024-09-21T10:24:15.829719+0000","last_became_active":"2024-09-21T10:22:19.650821+0000","last_became_peered":"2024-09-21T10:22:19.650821+0000","last_unstale":"2024-09-21T10:24:15.829719+0000","last_undegraded":"2024-09-21T10:24:15.829719+0000","last_fullsized":"2024-09-21T10:24:15.829719+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.289859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.289859+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:54:14.510032+0000","scrub_duration":0.001251809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":333,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.827425+0000","last_change":"2024-09-21T10:22:19.664692+0000","last_active":"2024-09-21T10:24:15.827425+0000","last_peered":"2024-09-21T10:24:15.827425+0000","last_clean":"2024-09-21T10:24:15.827425+0000","last_became_active":"2024-09-21T10:22:19.647696+0000","last_became_peered":"2024-09-21T10:22:19.647696+0000","last_unstale":"2024-09-21T10:24:15.827425+0000","last_undegraded":"2024-09-21T10:24:15.827425+0000","last_fullsized":"2024-09-21T10:24:15.827425+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:57:54.264752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":33,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.829871+0000","last_change":"2024-09-21T10:22:19.652364+0000","last_active":"2024-09-21T10:24:15.829871+0000","last_peered":"2024-09-21T10:24:15.829871+0000","last_clean":"2024-09-21T10:24:15.829871+0000","last_became_active":"2024-09-21T10:22:19.651586+0000","last_became_peered":"2024-09-21T10:22:19.651586+0000","last_unstale":"2024-09-21T10:24:15.829871+0000","last_undegraded":"2024-09-21T10:24:15.829871+0000","last_fullsized":"2024-09-21T10:24:15.829871+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:03.593157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1","version":"426'7519","reported_seq":7560,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:32.678358+0000","last_change":"2024-09-21T10:22:44.821910+0000","last_active":"2024-09-21T10:24:32.678358+0000","last_peered":"2024-09-21T10:24:32.678358+0000","last_clean":"2024-09-21T10:24:32.678358+0000","last_became_active":"2024-09-21T10:22:44.821581+0000","last_became_peered":"2024-09-21T10:22:44.821581+0000","last_unstale":"2024-09-21T10:24:32.678358+0000","last_undegraded":"2024-09-21T10:24:32.678358+0000","last_fullsized":"2024-09-21T10:24:32.678358+0000","mapping_epoch":411,"log_start":"416'800","ondisk_log_start":"416'800","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":6719,"log_dups_size":0,"ondisk_log_size":6719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:05:51.203136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449380352,"num_objects":6857,"num_object_clones":0,"num_object_copies":20571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7519,"num_write_kb":460032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"77'100","reported_seq":255,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.817590+0000","last_change":"2024-09-21T10:22:08.163986+0000","last_active":"2024-09-21T10:24:15.817590+0000","last_peered":"2024-09-21T10:24:15.817590+0000","last_clean":"2024-09-21T10:24:15.817590+0000","last_became_active":"2024-09-21T10:22:08.163669+0000","last_became_peered":"2024-09-21T10:22:08.163669+0000","last_unstale":"2024-09-21T10:24:15.817590+0000","last_undegraded":"2024-09-21T10:24:15.817590+0000","last_fullsized":"2024-09-21T10:24:15.817590+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:18:27.085525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:18:27.085525+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:55:41.477408+0000","scrub_duration":0.021884888000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":529,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.230949+0000","last_change":"2024-09-21T10:22:19.646721+0000","last_active":"2024-09-21T10:24:31.230949+0000","last_peered":"2024-09-21T10:24:31.230949+0000","last_clean":"2024-09-21T10:24:31.230949+0000","last_became_active":"2024-09-21T10:22:19.645541+0000","last_became_peered":"2024-09-21T10:22:19.645541+0000","last_unstale":"2024-09-21T10:24:31.230949+0000","last_undegraded":"2024-09-21T10:24:31.230949+0000","last_fullsized":"2024-09-21T10:24:31.230949+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:59:36.096992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":618,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.230998+0000","last_change":"2024-09-21T10:14:30.711105+0000","last_active":"2024-09-21T10:24:31.230998+0000","last_peered":"2024-09-21T10:24:31.230998+0000","last_clean":"2024-09-21T10:24:31.230998+0000","last_became_active":"2024-09-21T10:14:30.710759+0000","last_became_peered":"2024-09-21T10:14:30.710759+0000","last_unstale":"2024-09-21T10:24:31.230998+0000","last_undegraded":"2024-09-21T10:24:31.230998+0000","last_fullsized":"2024-09-21T10:24:31.230998+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:40.690042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.6","version":"426'7491","reported_seq":7531,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:33.067977+0000","last_change":"2024-09-21T10:22:44.826445+0000","last_active":"2024-09-21T10:24:33.067977+0000","last_peered":"2024-09-21T10:24:33.067977+0000","last_clean":"2024-09-21T10:24:33.067977+0000","last_became_active":"2024-09-21T10:22:44.825882+0000","last_became_peered":"2024-09-21T10:22:44.825882+0000","last_unstale":"2024-09-21T10:24:33.067977+0000","last_undegraded":"2024-09-21T10:24:33.067977+0000","last_fullsized":"2024-09-21T10:24:33.067977+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":7491,"log_dups_size":0,"ondisk_log_size":7491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:26:59.950097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442957824,"num_objects":6759,"num_object_clones":0,"num_object_copies":20277,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7491,"num_write_kb":456000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":299,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.817231+0000","last_change":"2024-09-21T10:04:53.355975+0000","last_active":"2024-09-21T10:24:15.817231+0000","last_peered":"2024-09-21T10:24:15.817231+0000","last_clean":"2024-09-21T10:24:15.817231+0000","last_became_active":"2024-09-21T10:04:42.547505+0000","last_became_peered":"2024-09-21T10:04:42.547505+0000","last_unstale":"2024-09-21T10:24:15.817231+0000","last_undegraded":"2024-09-21T10:24:15.817231+0000","last_fullsized":"2024-09-21T10:24:15.817231+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.355912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.355912+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:43.456830+0000","scrub_duration":0.0017980030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":722,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.227044+0000","last_change":"2024-09-21T10:14:30.715303+0000","last_active":"2024-09-21T10:24:31.227044+0000","last_peered":"2024-09-21T10:24:31.227044+0000","last_clean":"2024-09-21T10:24:31.227044+0000","last_became_active":"2024-09-21T10:14:30.703664+0000","last_became_peered":"2024-09-21T10:14:30.703664+0000","last_unstale":"2024-09-21T10:24:31.227044+0000","last_undegraded":"2024-09-21T10:24:31.227044+0000","last_fullsized":"2024-09-21T10:24:31.227044+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:28.967906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":308,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.231043+0000","last_change":"2024-09-21T10:22:19.647110+0000","last_active":"2024-09-21T10:24:31.231043+0000","last_peered":"2024-09-21T10:24:31.231043+0000","last_clean":"2024-09-21T10:24:31.231043+0000","last_became_active":"2024-09-21T10:22:19.645581+0000","last_became_peered":"2024-09-21T10:22:19.645581+0000","last_unstale":"2024-09-21T10:24:31.231043+0000","last_undegraded":"2024-09-21T10:24:31.231043+0000","last_fullsized":"2024-09-21T10:24:31.231043+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:08.054517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.7","version":"426'7745","reported_seq":7786,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:33.057019+0000","last_change":"2024-09-21T10:22:44.830915+0000","last_active":"2024-09-21T10:24:33.057019+0000","last_peered":"2024-09-21T10:24:33.057019+0000","last_clean":"2024-09-21T10:24:33.057019+0000","last_became_active":"2024-09-21T10:22:44.830464+0000","last_became_peered":"2024-09-21T10:22:44.830464+0000","last_unstale":"2024-09-21T10:24:33.057019+0000","last_undegraded":"2024-09-21T10:24:33.057019+0000","last_fullsized":"2024-09-21T10:24:33.057019+0000","mapping_epoch":411,"log_start":"415'200","ondisk_log_start":"415'200","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":7545,"log_dups_size":0,"ondisk_log_size":7545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:58:49.959117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465108992,"num_objects":7097,"num_object_clones":0,"num_object_copies":21291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7745,"num_write_kb":474944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":471,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.833008+0000","last_change":"2024-09-21T10:10:57.403722+0000","last_active":"2024-09-21T10:24:15.833008+0000","last_peered":"2024-09-21T10:24:15.833008+0000","last_clean":"2024-09-21T10:24:15.833008+0000","last_became_active":"2024-09-21T10:10:57.394700+0000","last_became_peered":"2024-09-21T10:10:57.394700+0000","last_unstale":"2024-09-21T10:24:15.833008+0000","last_undegraded":"2024-09-21T10:24:15.833008+0000","last_fullsized":"2024-09-21T10:24:15.833008+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:02.152981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:02.152981+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:25.675519+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.829768+0000","last_change":"2024-09-21T10:22:19.652182+0000","last_active":"2024-09-21T10:24:15.829768+0000","last_peered":"2024-09-21T10:24:15.829768+0000","last_clean":"2024-09-21T10:24:15.829768+0000","last_became_active":"2024-09-21T10:22:19.651703+0000","last_became_peered":"2024-09-21T10:22:19.651703+0000","last_unstale":"2024-09-21T10:24:15.829768+0000","last_undegraded":"2024-09-21T10:24:15.829768+0000","last_fullsized":"2024-09-21T10:24:15.829768+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:08.752662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":460,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.835735+0000","last_change":"2024-09-21T10:22:08.118607+0000","last_active":"2024-09-21T10:24:15.835735+0000","last_peered":"2024-09-21T10:24:15.835735+0000","last_clean":"2024-09-21T10:24:15.835735+0000","last_became_active":"2024-09-21T10:22:08.117950+0000","last_became_peered":"2024-09-21T10:22:08.117950+0000","last_unstale":"2024-09-21T10:24:15.835735+0000","last_undegraded":"2024-09-21T10:24:15.835735+0000","last_fullsized":"2024-09-21T10:24:15.835735+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:29:17.689281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"426'7540","reported_seq":7563,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:32.103074+0000","last_change":"2024-09-21T10:22:44.823139+0000","last_active":"2024-09-21T10:24:32.103074+0000","last_peered":"2024-09-21T10:24:32.103074+0000","last_clean":"2024-09-21T10:24:32.103074+0000","last_became_active":"2024-09-21T10:22:44.822759+0000","last_became_peered":"2024-09-21T10:22:44.822759+0000","last_unstale":"2024-09-21T10:24:32.103074+0000","last_undegraded":"2024-09-21T10:24:32.103074+0000","last_fullsized":"2024-09-21T10:24:32.103074+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":7540,"log_dups_size":0,"ondisk_log_size":7540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:16.929399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453771264,"num_objects":6924,"num_object_clones":0,"num_object_copies":20772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7540,"num_write_kb":462848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"77'100","reported_seq":363,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.823500+0000","last_change":"2024-09-21T10:14:05.474100+0000","last_active":"2024-09-21T10:24:15.823500+0000","last_peered":"2024-09-21T10:24:15.823500+0000","last_clean":"2024-09-21T10:24:15.823500+0000","last_became_active":"2024-09-21T10:14:05.473336+0000","last_became_peered":"2024-09-21T10:14:05.473336+0000","last_unstale":"2024-09-21T10:24:15.823500+0000","last_undegraded":"2024-09-21T10:24:15.823500+0000","last_fullsized":"2024-09-21T10:24:15.823500+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.540323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.540323+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:06:10.963352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":552,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.833177+0000","last_change":"2024-09-21T10:22:08.118141+0000","last_active":"2024-09-21T10:24:15.833177+0000","last_peered":"2024-09-21T10:24:15.833177+0000","last_clean":"2024-09-21T10:24:15.833177+0000","last_became_active":"2024-09-21T10:22:08.117868+0000","last_became_peered":"2024-09-21T10:22:08.117868+0000","last_unstale":"2024-09-21T10:24:15.833177+0000","last_undegraded":"2024-09-21T10:24:15.833177+0000","last_fullsized":"2024-09-21T10:24:15.833177+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:01.774438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":33,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.829559+0000","last_change":"2024-09-21T10:22:19.651485+0000","last_active":"2024-09-21T10:24:15.829559+0000","last_peered":"2024-09-21T10:24:15.829559+0000","last_clean":"2024-09-21T10:24:15.829559+0000","last_became_active":"2024-09-21T10:22:19.651154+0000","last_became_peered":"2024-09-21T10:22:19.651154+0000","last_unstale":"2024-09-21T10:24:15.829559+0000","last_undegraded":"2024-09-21T10:24:15.829559+0000","last_fullsized":"2024-09-21T10:24:15.829559+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:31.496619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.5","version":"426'7437","reported_seq":7478,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:33.021877+0000","last_change":"2024-09-21T10:22:44.822395+0000","last_active":"2024-09-21T10:24:33.021877+0000","last_peered":"2024-09-21T10:24:33.021877+0000","last_clean":"2024-09-21T10:24:33.021877+0000","last_became_active":"2024-09-21T10:22:44.822097+0000","last_became_peered":"2024-09-21T10:22:44.822097+0000","last_unstale":"2024-09-21T10:24:33.021877+0000","last_undegraded":"2024-09-21T10:24:33.021877+0000","last_fullsized":"2024-09-21T10:24:33.021877+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":7437,"log_dups_size":0,"ondisk_log_size":7437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:58:55.402300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446627840,"num_objects":6815,"num_object_clones":0,"num_object_copies":20445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7437,"num_write_kb":456064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":456,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.833147+0000","last_change":"2024-09-21T10:22:19.646459+0000","last_active":"2024-09-21T10:24:15.833147+0000","last_peered":"2024-09-21T10:24:15.833147+0000","last_clean":"2024-09-21T10:24:15.833147+0000","last_became_active":"2024-09-21T10:22:19.646127+0000","last_became_peered":"2024-09-21T10:22:19.646127+0000","last_unstale":"2024-09-21T10:24:15.833147+0000","last_undegraded":"2024-09-21T10:24:15.833147+0000","last_fullsized":"2024-09-21T10:24:15.833147+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:20:00.237668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:20:00.237668+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:27.159346+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":713,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.231325+0000","last_change":"2024-09-21T10:03:38.386604+0000","last_active":"2024-09-21T10:24:31.231325+0000","last_peered":"2024-09-21T10:24:31.231325+0000","last_clean":"2024-09-21T10:24:31.231325+0000","last_became_active":"2024-09-21T10:03:38.386283+0000","last_became_peered":"2024-09-21T10:03:38.386283+0000","last_unstale":"2024-09-21T10:24:31.231325+0000","last_undegraded":"2024-09-21T10:24:31.231325+0000","last_fullsized":"2024-09-21T10:24:31.231325+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:27:05.944880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":569,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.834115+0000","last_change":"2024-09-21T10:14:30.709615+0000","last_active":"2024-09-21T10:24:15.834115+0000","last_peered":"2024-09-21T10:24:15.834115+0000","last_clean":"2024-09-21T10:24:15.834115+0000","last_became_active":"2024-09-21T10:14:30.709184+0000","last_became_peered":"2024-09-21T10:14:30.709184+0000","last_unstale":"2024-09-21T10:24:15.834115+0000","last_undegraded":"2024-09-21T10:24:15.834115+0000","last_fullsized":"2024-09-21T10:24:15.834115+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:55:27.220232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.f","version":"426'7381","reported_seq":7404,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:32.315717+0000","last_change":"2024-09-21T10:22:44.824085+0000","last_active":"2024-09-21T10:24:32.315717+0000","last_peered":"2024-09-21T10:24:32.315717+0000","last_clean":"2024-09-21T10:24:32.315717+0000","last_became_active":"2024-09-21T10:22:44.823764+0000","last_became_peered":"2024-09-21T10:22:44.823764+0000","last_unstale":"2024-09-21T10:24:32.315717+0000","last_undegraded":"2024-09-21T10:24:32.315717+0000","last_fullsized":"2024-09-21T10:24:32.315717+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":7381,"log_dups_size":0,"ondisk_log_size":7381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:02:03.463670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444792832,"num_objects":6787,"num_object_clones":0,"num_object_copies":20361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7381,"num_write_kb":453376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":336,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.816996+0000","last_change":"2024-09-21T10:22:19.655993+0000","last_active":"2024-09-21T10:24:15.816996+0000","last_peered":"2024-09-21T10:24:15.816996+0000","last_clean":"2024-09-21T10:24:15.816996+0000","last_became_active":"2024-09-21T10:22:19.655178+0000","last_became_peered":"2024-09-21T10:22:19.655178+0000","last_unstale":"2024-09-21T10:24:15.816996+0000","last_undegraded":"2024-09-21T10:24:15.816996+0000","last_fullsized":"2024-09-21T10:24:15.816996+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:47.393146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:47.393146+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:48.668712+0000","scrub_duration":0.001601086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":465,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.835112+0000","last_change":"2024-09-21T10:10:57.402946+0000","last_active":"2024-09-21T10:24:15.835112+0000","last_peered":"2024-09-21T10:24:15.835112+0000","last_clean":"2024-09-21T10:24:15.835112+0000","last_became_active":"2024-09-21T10:10:57.395450+0000","last_became_peered":"2024-09-21T10:10:57.395450+0000","last_unstale":"2024-09-21T10:24:15.835112+0000","last_undegraded":"2024-09-21T10:24:15.835112+0000","last_fullsized":"2024-09-21T10:24:15.835112+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:08.680434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":711,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.226539+0000","last_change":"2024-09-21T10:22:19.647384+0000","last_active":"2024-09-21T10:24:31.226539+0000","last_peered":"2024-09-21T10:24:31.226539+0000","last_clean":"2024-09-21T10:24:31.226539+0000","last_became_active":"2024-09-21T10:22:19.645407+0000","last_became_peered":"2024-09-21T10:22:19.645407+0000","last_unstale":"2024-09-21T10:24:31.226539+0000","last_undegraded":"2024-09-21T10:24:31.226539+0000","last_fullsized":"2024-09-21T10:24:31.226539+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:04:57.385251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":33,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.830226+0000","last_change":"2024-09-21T10:22:19.651173+0000","last_active":"2024-09-21T10:24:15.830226+0000","last_peered":"2024-09-21T10:24:15.830226+0000","last_clean":"2024-09-21T10:24:15.830226+0000","last_became_active":"2024-09-21T10:22:19.649673+0000","last_became_peered":"2024-09-21T10:22:19.649673+0000","last_unstale":"2024-09-21T10:24:15.830226+0000","last_undegraded":"2024-09-21T10:24:15.830226+0000","last_fullsized":"2024-09-21T10:24:15.830226+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:28:37.605825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.c","version":"426'7677","reported_seq":7718,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:33.054261+0000","last_change":"2024-09-21T10:22:44.825224+0000","last_active":"2024-09-21T10:24:33.054261+0000","last_peered":"2024-09-21T10:24:33.054261+0000","last_clean":"2024-09-21T10:24:33.054261+0000","last_became_active":"2024-09-21T10:22:44.824917+0000","last_became_peered":"2024-09-21T10:22:44.824917+0000","last_unstale":"2024-09-21T10:24:33.054261+0000","last_undegraded":"2024-09-21T10:24:33.054261+0000","last_fullsized":"2024-09-21T10:24:33.054261+0000","mapping_epoch":411,"log_start":"415'100","ondisk_log_start":"415'100","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":7577,"log_dups_size":0,"ondisk_log_size":7577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:02.699042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457375744,"num_objects":6979,"num_object_clones":0,"num_object_copies":20937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7677,"num_write_kb":468992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":387,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.823755+0000","last_change":"2024-09-21T10:22:19.651858+0000","last_active":"2024-09-21T10:24:15.823755+0000","last_peered":"2024-09-21T10:24:15.823755+0000","last_clean":"2024-09-21T10:24:15.823755+0000","last_became_active":"2024-09-21T10:22:19.651510+0000","last_became_peered":"2024-09-21T10:22:19.651510+0000","last_unstale":"2024-09-21T10:24:15.823755+0000","last_undegraded":"2024-09-21T10:24:15.823755+0000","last_fullsized":"2024-09-21T10:24:15.823755+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.550551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.550551+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:01:42.501915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":290,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.817075+0000","last_change":"2024-09-21T10:09:23.838944+0000","last_active":"2024-09-21T10:24:15.817075+0000","last_peered":"2024-09-21T10:24:15.817075+0000","last_clean":"2024-09-21T10:24:15.817075+0000","last_became_active":"2024-09-21T10:09:23.837886+0000","last_became_peered":"2024-09-21T10:09:23.837886+0000","last_unstale":"2024-09-21T10:24:15.817075+0000","last_undegraded":"2024-09-21T10:24:15.817075+0000","last_fullsized":"2024-09-21T10:24:15.817075+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:16.928195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":596,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.224092+0000","last_change":"2024-09-21T10:18:12.637103+0000","last_active":"2024-09-21T10:24:31.224092+0000","last_peered":"2024-09-21T10:24:31.224092+0000","last_clean":"2024-09-21T10:24:31.224092+0000","last_became_active":"2024-09-21T10:18:12.636645+0000","last_became_peered":"2024-09-21T10:18:12.636645+0000","last_unstale":"2024-09-21T10:24:31.224092+0000","last_undegraded":"2024-09-21T10:24:31.224092+0000","last_fullsized":"2024-09-21T10:24:31.224092+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:53:50.745484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":357,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.224534+0000","last_change":"2024-09-21T10:09:49.923044+0000","last_active":"2024-09-21T10:24:31.224534+0000","last_peered":"2024-09-21T10:24:31.224534+0000","last_clean":"2024-09-21T10:24:31.224534+0000","last_became_active":"2024-09-21T10:09:49.922711+0000","last_became_peered":"2024-09-21T10:09:49.922711+0000","last_unstale":"2024-09-21T10:24:31.224534+0000","last_undegraded":"2024-09-21T10:24:31.224534+0000","last_fullsized":"2024-09-21T10:24:31.224534+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:14.090084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.d","version":"426'7498","reported_seq":7522,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:32.047967+0000","last_change":"2024-09-21T10:22:44.830934+0000","last_active":"2024-09-21T10:24:32.047967+0000","last_peered":"2024-09-21T10:24:32.047967+0000","last_clean":"2024-09-21T10:24:32.047967+0000","last_became_active":"2024-09-21T10:22:44.830615+0000","last_became_peered":"2024-09-21T10:22:44.830615+0000","last_unstale":"2024-09-21T10:24:32.047967+0000","last_undegraded":"2024-09-21T10:24:32.047967+0000","last_fullsized":"2024-09-21T10:24:32.047967+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":7498,"log_dups_size":0,"ondisk_log_size":7498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:27.432511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457244696,"num_objects":6978,"num_object_clones":0,"num_object_copies":20934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6978,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7498,"num_write_kb":463169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":340,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.817129+0000","last_change":"2024-09-21T10:18:12.636851+0000","last_active":"2024-09-21T10:24:15.817129+0000","last_peered":"2024-09-21T10:24:15.817129+0000","last_clean":"2024-09-21T10:24:15.817129+0000","last_became_active":"2024-09-21T10:18:12.636513+0000","last_became_peered":"2024-09-21T10:18:12.636513+0000","last_unstale":"2024-09-21T10:24:15.817129+0000","last_undegraded":"2024-09-21T10:24:15.817129+0000","last_fullsized":"2024-09-21T10:24:15.817129+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:52.361137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:52.361137+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:21.025735+0000","scrub_duration":0.001255358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":748,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.833265+0000","last_change":"2024-09-21T10:09:23.837859+0000","last_active":"2024-09-21T10:24:15.833265+0000","last_peered":"2024-09-21T10:24:15.833265+0000","last_clean":"2024-09-21T10:24:15.833265+0000","last_became_active":"2024-09-21T10:09:23.837488+0000","last_became_peered":"2024-09-21T10:09:23.837488+0000","last_unstale":"2024-09-21T10:24:15.833265+0000","last_undegraded":"2024-09-21T10:24:15.833265+0000","last_fullsized":"2024-09-21T10:24:15.833265+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:21:25.215501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":194,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.816913+0000","last_change":"2024-09-21T10:22:19.647946+0000","last_active":"2024-09-21T10:24:15.816913+0000","last_peered":"2024-09-21T10:24:15.816913+0000","last_clean":"2024-09-21T10:24:15.816913+0000","last_became_active":"2024-09-21T10:22:19.647057+0000","last_became_peered":"2024-09-21T10:22:19.647057+0000","last_unstale":"2024-09-21T10:24:15.816913+0000","last_undegraded":"2024-09-21T10:24:15.816913+0000","last_fullsized":"2024-09-21T10:24:15.816913+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:06.839922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":742,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.836303+0000","last_change":"2024-09-21T10:10:57.395073+0000","last_active":"2024-09-21T10:24:15.836303+0000","last_peered":"2024-09-21T10:24:15.836303+0000","last_clean":"2024-09-21T10:24:15.836303+0000","last_became_active":"2024-09-21T10:10:57.394674+0000","last_became_peered":"2024-09-21T10:10:57.394674+0000","last_unstale":"2024-09-21T10:24:15.836303+0000","last_undegraded":"2024-09-21T10:24:15.836303+0000","last_fullsized":"2024-09-21T10:24:15.836303+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:48:48.912649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.e","version":"425'7446","reported_seq":7484,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:28.902624+0000","last_change":"2024-09-21T10:22:44.821540+0000","last_active":"2024-09-21T10:24:28.902624+0000","last_peered":"2024-09-21T10:24:28.902624+0000","last_clean":"2024-09-21T10:24:28.902624+0000","last_became_active":"2024-09-21T10:22:44.821263+0000","last_became_peered":"2024-09-21T10:22:44.821263+0000","last_unstale":"2024-09-21T10:24:28.902624+0000","last_undegraded":"2024-09-21T10:24:28.902624+0000","last_fullsized":"2024-09-21T10:24:28.902624+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":7446,"log_dups_size":0,"ondisk_log_size":7446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:05:50.494438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473432064,"num_objects":7224,"num_object_clones":0,"num_object_copies":21672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7446,"num_write_kb":469440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":615,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.223794+0000","last_change":"2024-09-21T10:10:57.398437+0000","last_active":"2024-09-21T10:24:31.223794+0000","last_peered":"2024-09-21T10:24:31.223794+0000","last_clean":"2024-09-21T10:24:31.223794+0000","last_became_active":"2024-09-21T10:10:57.397441+0000","last_became_peered":"2024-09-21T10:10:57.397441+0000","last_unstale":"2024-09-21T10:24:31.223794+0000","last_undegraded":"2024-09-21T10:24:31.223794+0000","last_fullsized":"2024-09-21T10:24:31.223794+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:04.043917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:04.043917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:40:17.580110+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":332,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.821801+0000","last_change":"2024-09-21T10:22:19.665067+0000","last_active":"2024-09-21T10:24:15.821801+0000","last_peered":"2024-09-21T10:24:15.821801+0000","last_clean":"2024-09-21T10:24:15.821801+0000","last_became_active":"2024-09-21T10:22:19.651363+0000","last_became_peered":"2024-09-21T10:22:19.651363+0000","last_unstale":"2024-09-21T10:24:15.821801+0000","last_undegraded":"2024-09-21T10:24:15.821801+0000","last_fullsized":"2024-09-21T10:24:15.821801+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:29.173718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":242,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.231166+0000","last_change":"2024-09-21T10:22:19.646459+0000","last_active":"2024-09-21T10:24:31.231166+0000","last_peered":"2024-09-21T10:24:31.231166+0000","last_clean":"2024-09-21T10:24:31.231166+0000","last_became_active":"2024-09-21T10:22:19.645956+0000","last_became_peered":"2024-09-21T10:22:19.645956+0000","last_unstale":"2024-09-21T10:24:31.231166+0000","last_undegraded":"2024-09-21T10:24:31.231166+0000","last_fullsized":"2024-09-21T10:24:31.231166+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:11:56.923595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":574,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.821173+0000","last_change":"2024-09-21T10:14:05.477875+0000","last_active":"2024-09-21T10:24:15.821173+0000","last_peered":"2024-09-21T10:24:15.821173+0000","last_clean":"2024-09-21T10:24:15.821173+0000","last_became_active":"2024-09-21T10:14:05.477192+0000","last_became_peered":"2024-09-21T10:14:05.477192+0000","last_unstale":"2024-09-21T10:24:15.821173+0000","last_undegraded":"2024-09-21T10:24:15.821173+0000","last_fullsized":"2024-09-21T10:24:15.821173+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:19:47.345673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.8","version":"426'7544","reported_seq":7585,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:33.038171+0000","last_change":"2024-09-21T10:22:44.826294+0000","last_active":"2024-09-21T10:24:33.038171+0000","last_peered":"2024-09-21T10:24:33.038171+0000","last_clean":"2024-09-21T10:24:33.038171+0000","last_became_active":"2024-09-21T10:22:44.825917+0000","last_became_peered":"2024-09-21T10:22:44.825917+0000","last_unstale":"2024-09-21T10:24:33.038171+0000","last_undegraded":"2024-09-21T10:24:33.038171+0000","last_fullsized":"2024-09-21T10:24:33.038171+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":7544,"log_dups_size":0,"ondisk_log_size":7544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:06:27.760082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449970176,"num_objects":6866,"num_object_clones":0,"num_object_copies":20598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7544,"num_write_kb":461120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":33,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.829706+0000","last_change":"2024-09-21T10:22:19.649129+0000","last_active":"2024-09-21T10:24:15.829706+0000","last_peered":"2024-09-21T10:24:15.829706+0000","last_clean":"2024-09-21T10:24:15.829706+0000","last_became_active":"2024-09-21T10:22:19.648779+0000","last_became_peered":"2024-09-21T10:22:19.648779+0000","last_unstale":"2024-09-21T10:24:15.829706+0000","last_undegraded":"2024-09-21T10:24:15.829706+0000","last_fullsized":"2024-09-21T10:24:15.829706+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:21:47.185890+0000","last_deep_scrub":"77'100","last_deep_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_clean_scrub_stamp":"2024-09-21T10:21:47.185890+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:44:01.608891+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":419,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.226478+0000","last_change":"2024-09-21T10:15:01.489326+0000","last_active":"2024-09-21T10:24:31.226478+0000","last_peered":"2024-09-21T10:24:31.226478+0000","last_clean":"2024-09-21T10:24:31.226478+0000","last_became_active":"2024-09-21T10:15:01.489070+0000","last_became_peered":"2024-09-21T10:15:01.489070+0000","last_unstale":"2024-09-21T10:24:31.226478+0000","last_undegraded":"2024-09-21T10:24:31.226478+0000","last_fullsized":"2024-09-21T10:24:31.226478+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:27:41.788029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":630,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.821492+0000","last_change":"2024-09-21T10:14:05.475970+0000","last_active":"2024-09-21T10:24:15.821492+0000","last_peered":"2024-09-21T10:24:15.821492+0000","last_clean":"2024-09-21T10:24:15.821492+0000","last_became_active":"2024-09-21T10:14:05.475331+0000","last_became_peered":"2024-09-21T10:14:05.475331+0000","last_unstale":"2024-09-21T10:24:15.821492+0000","last_undegraded":"2024-09-21T10:24:15.821492+0000","last_fullsized":"2024-09-21T10:24:15.821492+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:17.069736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":567,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.822089+0000","last_change":"2024-09-21T10:18:13.039172+0000","last_active":"2024-09-21T10:24:15.822089+0000","last_peered":"2024-09-21T10:24:15.822089+0000","last_clean":"2024-09-21T10:24:15.822089+0000","last_became_active":"2024-09-21T10:18:13.038756+0000","last_became_peered":"2024-09-21T10:18:13.038756+0000","last_unstale":"2024-09-21T10:24:15.822089+0000","last_undegraded":"2024-09-21T10:24:15.822089+0000","last_fullsized":"2024-09-21T10:24:15.822089+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:06.668926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.9","version":"426'7582","reported_seq":7623,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:32.633775+0000","last_change":"2024-09-21T10:22:44.830660+0000","last_active":"2024-09-21T10:24:32.633775+0000","last_peered":"2024-09-21T10:24:32.633775+0000","last_clean":"2024-09-21T10:24:32.633775+0000","last_became_active":"2024-09-21T10:22:44.830357+0000","last_became_peered":"2024-09-21T10:22:44.830357+0000","last_unstale":"2024-09-21T10:24:32.633775+0000","last_undegraded":"2024-09-21T10:24:32.633775+0000","last_fullsized":"2024-09-21T10:24:32.633775+0000","mapping_epoch":411,"log_start":"416'900","ondisk_log_start":"416'900","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":6682,"log_dups_size":0,"ondisk_log_size":6682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:55.495054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457048064,"num_objects":6974,"num_object_clones":0,"num_object_copies":20922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7582,"num_write_kb":465792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":33,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.829378+0000","last_change":"2024-09-21T10:22:19.645389+0000","last_active":"2024-09-21T10:24:15.829378+0000","last_peered":"2024-09-21T10:24:15.829378+0000","last_clean":"2024-09-21T10:24:15.829378+0000","last_became_active":"2024-09-21T10:22:19.644671+0000","last_became_peered":"2024-09-21T10:22:19.644671+0000","last_unstale":"2024-09-21T10:24:15.829378+0000","last_undegraded":"2024-09-21T10:24:15.829378+0000","last_fullsized":"2024-09-21T10:24:15.829378+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:20:01.397007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:20:01.397007+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:45.545962+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":562,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.821378+0000","last_change":"2024-09-21T10:14:05.478152+0000","last_active":"2024-09-21T10:24:15.821378+0000","last_peered":"2024-09-21T10:24:15.821378+0000","last_clean":"2024-09-21T10:24:15.821378+0000","last_became_active":"2024-09-21T10:14:05.477547+0000","last_became_peered":"2024-09-21T10:14:05.477547+0000","last_unstale":"2024-09-21T10:24:15.821378+0000","last_undegraded":"2024-09-21T10:24:15.821378+0000","last_fullsized":"2024-09-21T10:24:15.821378+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:34.646046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":320,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.827526+0000","last_change":"2024-09-21T10:14:30.708090+0000","last_active":"2024-09-21T10:24:15.827526+0000","last_peered":"2024-09-21T10:24:15.827526+0000","last_clean":"2024-09-21T10:24:15.827526+0000","last_became_active":"2024-09-21T10:14:30.707266+0000","last_became_peered":"2024-09-21T10:14:30.707266+0000","last_unstale":"2024-09-21T10:24:15.827526+0000","last_undegraded":"2024-09-21T10:24:15.827526+0000","last_fullsized":"2024-09-21T10:24:15.827526+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:38.474537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":627,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.231606+0000","last_change":"2024-09-21T10:10:57.397251+0000","last_active":"2024-09-21T10:24:31.231606+0000","last_peered":"2024-09-21T10:24:31.231606+0000","last_clean":"2024-09-21T10:24:31.231606+0000","last_became_active":"2024-09-21T10:10:57.396271+0000","last_became_peered":"2024-09-21T10:10:57.396271+0000","last_unstale":"2024-09-21T10:24:31.231606+0000","last_undegraded":"2024-09-21T10:24:31.231606+0000","last_fullsized":"2024-09-21T10:24:31.231606+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:17.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.a","version":"425'7383","reported_seq":7421,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:28.896137+0000","last_change":"2024-09-21T10:22:44.830746+0000","last_active":"2024-09-21T10:24:28.896137+0000","last_peered":"2024-09-21T10:24:28.896137+0000","last_clean":"2024-09-21T10:24:28.896137+0000","last_became_active":"2024-09-21T10:22:44.830482+0000","last_became_peered":"2024-09-21T10:22:44.830482+0000","last_unstale":"2024-09-21T10:24:28.896137+0000","last_undegraded":"2024-09-21T10:24:28.896137+0000","last_fullsized":"2024-09-21T10:24:28.896137+0000","mapping_epoch":411,"log_start":"415'400","ondisk_log_start":"415'400","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":6983,"log_dups_size":0,"ondisk_log_size":6983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:43.196504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":469827584,"num_objects":7169,"num_object_clones":0,"num_object_copies":21507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7383,"num_write_kb":465664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":248,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.835227+0000","last_change":"2024-09-21T10:22:19.646493+0000","last_active":"2024-09-21T10:24:15.835227+0000","last_peered":"2024-09-21T10:24:15.835227+0000","last_clean":"2024-09-21T10:24:15.835227+0000","last_became_active":"2024-09-21T10:22:19.646152+0000","last_became_peered":"2024-09-21T10:22:19.646152+0000","last_unstale":"2024-09-21T10:24:15.835227+0000","last_undegraded":"2024-09-21T10:24:15.835227+0000","last_fullsized":"2024-09-21T10:24:15.835227+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:59.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:59.099852+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:17:29.824712+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":719,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.226754+0000","last_change":"2024-09-21T10:09:23.841186+0000","last_active":"2024-09-21T10:24:31.226754+0000","last_peered":"2024-09-21T10:24:31.226754+0000","last_clean":"2024-09-21T10:24:31.226754+0000","last_became_active":"2024-09-21T10:09:23.840854+0000","last_became_peered":"2024-09-21T10:09:23.840854+0000","last_unstale":"2024-09-21T10:24:31.226754+0000","last_undegraded":"2024-09-21T10:24:31.226754+0000","last_fullsized":"2024-09-21T10:24:31.226754+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:29.666358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":583,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.821448+0000","last_change":"2024-09-21T10:22:19.644169+0000","last_active":"2024-09-21T10:24:15.821448+0000","last_peered":"2024-09-21T10:24:15.821448+0000","last_clean":"2024-09-21T10:24:15.821448+0000","last_became_active":"2024-09-21T10:22:19.643917+0000","last_became_peered":"2024-09-21T10:22:19.643917+0000","last_unstale":"2024-09-21T10:24:15.821448+0000","last_undegraded":"2024-09-21T10:24:15.821448+0000","last_fullsized":"2024-09-21T10:24:15.821448+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:28.976858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":716,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.231169+0000","last_change":"2024-09-21T10:09:49.926548+0000","last_active":"2024-09-21T10:24:31.231169+0000","last_peered":"2024-09-21T10:24:31.231169+0000","last_clean":"2024-09-21T10:24:31.231169+0000","last_became_active":"2024-09-21T10:09:49.926255+0000","last_became_peered":"2024-09-21T10:09:49.926255+0000","last_unstale":"2024-09-21T10:24:31.231169+0000","last_undegraded":"2024-09-21T10:24:31.231169+0000","last_fullsized":"2024-09-21T10:24:31.231169+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:49.855878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.b","version":"426'7528","reported_seq":7569,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:33.068352+0000","last_change":"2024-09-21T10:22:44.825973+0000","last_active":"2024-09-21T10:24:33.068352+0000","last_peered":"2024-09-21T10:24:33.068352+0000","last_clean":"2024-09-21T10:24:33.068352+0000","last_became_active":"2024-09-21T10:22:44.825697+0000","last_became_peered":"2024-09-21T10:22:44.825697+0000","last_unstale":"2024-09-21T10:24:33.068352+0000","last_undegraded":"2024-09-21T10:24:33.068352+0000","last_fullsized":"2024-09-21T10:24:33.068352+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:22:43.801215+0000","last_clean_scrub_stamp":"2024-09-21T10:22:43.801215+0000","objects_scrubbed":0,"log_size":7528,"log_dups_size":0,"ondisk_log_size":7528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:00:32.419547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448790528,"num_objects":6848,"num_object_clones":0,"num_object_copies":20544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7528,"num_write_kb":460032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":519,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.230415+0000","last_change":"2024-09-21T10:10:57.390804+0000","last_active":"2024-09-21T10:24:31.230415+0000","last_peered":"2024-09-21T10:24:31.230415+0000","last_clean":"2024-09-21T10:24:31.230415+0000","last_became_active":"2024-09-21T10:10:57.390323+0000","last_became_peered":"2024-09-21T10:10:57.390323+0000","last_unstale":"2024-09-21T10:24:31.230415+0000","last_undegraded":"2024-09-21T10:24:31.230415+0000","last_fullsized":"2024-09-21T10:24:31.230415+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.005477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.005477+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:40.776053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":372,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.224619+0000","last_change":"2024-09-21T10:14:30.718942+0000","last_active":"2024-09-21T10:24:31.224619+0000","last_peered":"2024-09-21T10:24:31.224619+0000","last_clean":"2024-09-21T10:24:31.224619+0000","last_became_active":"2024-09-21T10:14:30.709129+0000","last_became_peered":"2024-09-21T10:14:30.709129+0000","last_unstale":"2024-09-21T10:24:31.224619+0000","last_undegraded":"2024-09-21T10:24:31.224619+0000","last_fullsized":"2024-09-21T10:24:31.224619+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:18.171064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":33,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.829646+0000","last_change":"2024-09-21T10:22:19.646015+0000","last_active":"2024-09-21T10:24:15.829646+0000","last_peered":"2024-09-21T10:24:15.829646+0000","last_clean":"2024-09-21T10:24:15.829646+0000","last_became_active":"2024-09-21T10:22:19.645608+0000","last_became_peered":"2024-09-21T10:22:19.645608+0000","last_unstale":"2024-09-21T10:24:15.829646+0000","last_undegraded":"2024-09-21T10:24:15.829646+0000","last_fullsized":"2024-09-21T10:24:15.829646+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:14.920346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":33,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.829033+0000","last_change":"2024-09-21T10:22:19.649811+0000","last_active":"2024-09-21T10:24:15.829033+0000","last_peered":"2024-09-21T10:24:15.829033+0000","last_clean":"2024-09-21T10:24:15.829033+0000","last_became_active":"2024-09-21T10:22:19.649314+0000","last_became_peered":"2024-09-21T10:22:19.649314+0000","last_unstale":"2024-09-21T10:24:15.829033+0000","last_undegraded":"2024-09-21T10:24:15.829033+0000","last_fullsized":"2024-09-21T10:24:15.829033+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:36.322837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":33,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.829495+0000","last_change":"2024-09-21T10:22:19.651289+0000","last_active":"2024-09-21T10:24:15.829495+0000","last_peered":"2024-09-21T10:24:15.829495+0000","last_clean":"2024-09-21T10:24:15.829495+0000","last_became_active":"2024-09-21T10:22:19.650947+0000","last_became_peered":"2024-09-21T10:22:19.650947+0000","last_unstale":"2024-09-21T10:24:15.829495+0000","last_undegraded":"2024-09-21T10:24:15.829495+0000","last_fullsized":"2024-09-21T10:24:15.829495+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:54:44.845308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":290,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.817564+0000","last_change":"2024-09-21T10:22:19.662381+0000","last_active":"2024-09-21T10:24:15.817564+0000","last_peered":"2024-09-21T10:24:15.817564+0000","last_clean":"2024-09-21T10:24:15.817564+0000","last_became_active":"2024-09-21T10:22:19.646849+0000","last_became_peered":"2024-09-21T10:22:19.646849+0000","last_unstale":"2024-09-21T10:24:15.817564+0000","last_undegraded":"2024-09-21T10:24:15.817564+0000","last_fullsized":"2024-09-21T10:24:15.817564+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:38:55.399112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14232,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":635,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.821859+0000","last_change":"2024-09-21T10:22:19.644482+0000","last_active":"2024-09-21T10:24:15.821859+0000","last_peered":"2024-09-21T10:24:15.821859+0000","last_clean":"2024-09-21T10:24:15.821859+0000","last_became_active":"2024-09-21T10:22:19.644080+0000","last_became_peered":"2024-09-21T10:22:19.644080+0000","last_unstale":"2024-09-21T10:24:15.821859+0000","last_undegraded":"2024-09-21T10:24:15.821859+0000","last_fullsized":"2024-09-21T10:24:15.821859+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:42:37.848739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":364,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.835639+0000","last_change":"2024-09-21T10:14:30.713873+0000","last_active":"2024-09-21T10:24:15.835639+0000","last_peered":"2024-09-21T10:24:15.835639+0000","last_clean":"2024-09-21T10:24:15.835639+0000","last_became_active":"2024-09-21T10:14:30.703149+0000","last_became_peered":"2024-09-21T10:14:30.703149+0000","last_unstale":"2024-09-21T10:24:15.835639+0000","last_undegraded":"2024-09-21T10:24:15.835639+0000","last_fullsized":"2024-09-21T10:24:15.835639+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:43:26.453902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'100","reported_seq":516,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.835292+0000","last_change":"2024-09-21T10:18:12.633959+0000","last_active":"2024-09-21T10:24:15.835292+0000","last_peered":"2024-09-21T10:24:15.835292+0000","last_clean":"2024-09-21T10:24:15.835292+0000","last_became_active":"2024-09-21T10:18:12.633591+0000","last_became_peered":"2024-09-21T10:18:12.633591+0000","last_unstale":"2024-09-21T10:24:15.835292+0000","last_undegraded":"2024-09-21T10:24:15.835292+0000","last_fullsized":"2024-09-21T10:24:15.835292+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.014993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.014993+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:29:16.343472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":339,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.827500+0000","last_change":"2024-09-21T10:15:01.483814+0000","last_active":"2024-09-21T10:24:15.827500+0000","last_peered":"2024-09-21T10:24:15.827500+0000","last_clean":"2024-09-21T10:24:15.827500+0000","last_became_active":"2024-09-21T10:15:01.483522+0000","last_became_peered":"2024-09-21T10:15:01.483522+0000","last_unstale":"2024-09-21T10:24:15.827500+0000","last_undegraded":"2024-09-21T10:24:15.827500+0000","last_fullsized":"2024-09-21T10:24:15.827500+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:13.632066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":715,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.226548+0000","last_change":"2024-09-21T10:22:19.645758+0000","last_active":"2024-09-21T10:24:31.226548+0000","last_peered":"2024-09-21T10:24:31.226548+0000","last_clean":"2024-09-21T10:24:31.226548+0000","last_became_active":"2024-09-21T10:22:19.645357+0000","last_became_peered":"2024-09-21T10:22:19.645357+0000","last_unstale":"2024-09-21T10:24:31.226548+0000","last_undegraded":"2024-09-21T10:24:31.226548+0000","last_fullsized":"2024-09-21T10:24:31.226548+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:12:14.337839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":601,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.224920+0000","last_change":"2024-09-21T10:10:57.397755+0000","last_active":"2024-09-21T10:24:31.224920+0000","last_peered":"2024-09-21T10:24:31.224920+0000","last_clean":"2024-09-21T10:24:31.224920+0000","last_became_active":"2024-09-21T10:10:57.397249+0000","last_became_peered":"2024-09-21T10:10:57.397249+0000","last_unstale":"2024-09-21T10:24:31.224920+0000","last_undegraded":"2024-09-21T10:24:31.224920+0000","last_fullsized":"2024-09-21T10:24:31.224920+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:44:40.813503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":353,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.820716+0000","last_change":"2024-09-21T10:22:19.651921+0000","last_active":"2024-09-21T10:24:15.820716+0000","last_peered":"2024-09-21T10:24:15.820716+0000","last_clean":"2024-09-21T10:24:15.820716+0000","last_became_active":"2024-09-21T10:22:19.651582+0000","last_became_peered":"2024-09-21T10:22:19.651582+0000","last_unstale":"2024-09-21T10:24:15.820716+0000","last_undegraded":"2024-09-21T10:24:15.820716+0000","last_fullsized":"2024-09-21T10:24:15.820716+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:00:14.778412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":33,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.829766+0000","last_change":"2024-09-21T10:22:19.649280+0000","last_active":"2024-09-21T10:24:15.829766+0000","last_peered":"2024-09-21T10:24:15.829766+0000","last_clean":"2024-09-21T10:24:15.829766+0000","last_became_active":"2024-09-21T10:22:19.648950+0000","last_became_peered":"2024-09-21T10:22:19.648950+0000","last_unstale":"2024-09-21T10:24:15.829766+0000","last_undegraded":"2024-09-21T10:24:15.829766+0000","last_fullsized":"2024-09-21T10:24:15.829766+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:06:32.778203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":721,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.231247+0000","last_change":"2024-09-21T10:09:49.928053+0000","last_active":"2024-09-21T10:24:31.231247+0000","last_peered":"2024-09-21T10:24:31.231247+0000","last_clean":"2024-09-21T10:24:31.231247+0000","last_became_active":"2024-09-21T10:09:49.927231+0000","last_became_peered":"2024-09-21T10:09:49.927231+0000","last_unstale":"2024-09-21T10:24:31.231247+0000","last_undegraded":"2024-09-21T10:24:31.231247+0000","last_fullsized":"2024-09-21T10:24:31.231247+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:26.987553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":634,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.224758+0000","last_change":"2024-09-21T10:09:23.840717+0000","last_active":"2024-09-21T10:24:31.224758+0000","last_peered":"2024-09-21T10:24:31.224758+0000","last_clean":"2024-09-21T10:24:31.224758+0000","last_became_active":"2024-09-21T10:09:23.840323+0000","last_became_peered":"2024-09-21T10:09:23.840323+0000","last_unstale":"2024-09-21T10:24:31.224758+0000","last_undegraded":"2024-09-21T10:24:31.224758+0000","last_fullsized":"2024-09-21T10:24:31.224758+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:55.769770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":352,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.833334+0000","last_change":"2024-09-21T10:18:12.632498+0000","last_active":"2024-09-21T10:24:15.833334+0000","last_peered":"2024-09-21T10:24:15.833334+0000","last_clean":"2024-09-21T10:24:15.833334+0000","last_became_active":"2024-09-21T10:18:12.631648+0000","last_became_peered":"2024-09-21T10:18:12.631648+0000","last_unstale":"2024-09-21T10:24:15.833334+0000","last_undegraded":"2024-09-21T10:24:15.833334+0000","last_fullsized":"2024-09-21T10:24:15.833334+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:17.592618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":699,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.226370+0000","last_change":"2024-09-21T10:14:30.702101+0000","last_active":"2024-09-21T10:24:31.226370+0000","last_peered":"2024-09-21T10:24:31.226370+0000","last_clean":"2024-09-21T10:24:31.226370+0000","last_became_active":"2024-09-21T10:14:30.701783+0000","last_became_peered":"2024-09-21T10:14:30.701783+0000","last_unstale":"2024-09-21T10:24:31.226370+0000","last_undegraded":"2024-09-21T10:24:31.226370+0000","last_fullsized":"2024-09-21T10:24:31.226370+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:13.087010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":81,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.822181+0000","last_change":"2024-09-21T10:22:08.118073+0000","last_active":"2024-09-21T10:24:15.822181+0000","last_peered":"2024-09-21T10:24:15.822181+0000","last_clean":"2024-09-21T10:24:15.822181+0000","last_became_active":"2024-09-21T10:22:08.117679+0000","last_became_peered":"2024-09-21T10:22:08.117679+0000","last_unstale":"2024-09-21T10:24:15.822181+0000","last_undegraded":"2024-09-21T10:24:15.822181+0000","last_fullsized":"2024-09-21T10:24:15.822181+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:32:53.070579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":737,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.835715+0000","last_change":"2024-09-21T09:56:12.589160+0000","last_active":"2024-09-21T10:24:15.835715+0000","last_peered":"2024-09-21T10:24:15.835715+0000","last_clean":"2024-09-21T10:24:15.835715+0000","last_became_active":"2024-09-21T09:56:12.588787+0000","last_became_peered":"2024-09-21T09:56:12.588787+0000","last_unstale":"2024-09-21T10:24:15.835715+0000","last_undegraded":"2024-09-21T10:24:15.835715+0000","last_fullsized":"2024-09-21T10:24:15.835715+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:42.013479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":363,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.230834+0000","last_change":"2024-09-21T10:14:30.702414+0000","last_active":"2024-09-21T10:24:31.230834+0000","last_peered":"2024-09-21T10:24:31.230834+0000","last_clean":"2024-09-21T10:24:31.230834+0000","last_became_active":"2024-09-21T10:14:30.702009+0000","last_became_peered":"2024-09-21T10:14:30.702009+0000","last_unstale":"2024-09-21T10:24:31.230834+0000","last_undegraded":"2024-09-21T10:24:31.230834+0000","last_fullsized":"2024-09-21T10:24:31.230834+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:50:48.396082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":489,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.226966+0000","last_change":"2024-09-21T10:22:08.122333+0000","last_active":"2024-09-21T10:24:31.226966+0000","last_peered":"2024-09-21T10:24:31.226966+0000","last_clean":"2024-09-21T10:24:31.226966+0000","last_became_active":"2024-09-21T10:22:08.121898+0000","last_became_peered":"2024-09-21T10:22:08.121898+0000","last_unstale":"2024-09-21T10:24:31.226966+0000","last_undegraded":"2024-09-21T10:24:31.226966+0000","last_fullsized":"2024-09-21T10:24:31.226966+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:36:32.263089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":92,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.821628+0000","last_change":"2024-09-21T10:22:08.119919+0000","last_active":"2024-09-21T10:24:15.821628+0000","last_peered":"2024-09-21T10:24:15.821628+0000","last_clean":"2024-09-21T10:24:15.821628+0000","last_became_active":"2024-09-21T10:22:08.119598+0000","last_became_peered":"2024-09-21T10:22:08.119598+0000","last_unstale":"2024-09-21T10:24:15.821628+0000","last_undegraded":"2024-09-21T10:24:15.821628+0000","last_fullsized":"2024-09-21T10:24:15.821628+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:14:10.751524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":580,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.226760+0000","last_change":"2024-09-21T10:22:08.117372+0000","last_active":"2024-09-21T10:24:31.226760+0000","last_peered":"2024-09-21T10:24:31.226760+0000","last_clean":"2024-09-21T10:24:31.226760+0000","last_became_active":"2024-09-21T10:22:08.116632+0000","last_became_peered":"2024-09-21T10:22:08.116632+0000","last_unstale":"2024-09-21T10:24:31.226760+0000","last_undegraded":"2024-09-21T10:24:31.226760+0000","last_fullsized":"2024-09-21T10:24:31.226760+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:38.062747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":37,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.830365+0000","last_change":"2024-09-21T10:22:19.666163+0000","last_active":"2024-09-21T10:24:15.830365+0000","last_peered":"2024-09-21T10:24:15.830365+0000","last_clean":"2024-09-21T10:24:15.830365+0000","last_became_active":"2024-09-21T10:22:19.651579+0000","last_became_peered":"2024-09-21T10:22:19.651579+0000","last_unstale":"2024-09-21T10:24:15.830365+0000","last_undegraded":"2024-09-21T10:24:15.830365+0000","last_fullsized":"2024-09-21T10:24:15.830365+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:42.750730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":722,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.227074+0000","last_change":"2024-09-21T10:09:23.840467+0000","last_active":"2024-09-21T10:24:31.227074+0000","last_peered":"2024-09-21T10:24:31.227074+0000","last_clean":"2024-09-21T10:24:31.227074+0000","last_became_active":"2024-09-21T10:09:23.840177+0000","last_became_peered":"2024-09-21T10:09:23.840177+0000","last_unstale":"2024-09-21T10:24:31.227074+0000","last_undegraded":"2024-09-21T10:24:31.227074+0000","last_fullsized":"2024-09-21T10:24:31.227074+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:17.887307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":727,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.226777+0000","last_change":"2024-09-21T10:22:19.648325+0000","last_active":"2024-09-21T10:24:31.226777+0000","last_peered":"2024-09-21T10:24:31.226777+0000","last_clean":"2024-09-21T10:24:31.226777+0000","last_became_active":"2024-09-21T10:22:19.648008+0000","last_became_peered":"2024-09-21T10:22:19.648008+0000","last_unstale":"2024-09-21T10:24:31.226777+0000","last_undegraded":"2024-09-21T10:24:31.226777+0000","last_fullsized":"2024-09-21T10:24:31.226777+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:25.881077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":33,"reported_epoch":425,"state":"active+clean","last_fresh":"2024-09-21T10:24:15.829127+0000","last_change":"2024-09-21T10:22:19.652069+0000","last_active":"2024-09-21T10:24:15.829127+0000","last_peered":"2024-09-21T10:24:15.829127+0000","last_clean":"2024-09-21T10:24:15.829127+0000","last_became_active":"2024-09-21T10:22:19.651081+0000","last_became_peered":"2024-09-21T10:22:19.651081+0000","last_unstale":"2024-09-21T10:24:15.829127+0000","last_undegraded":"2024-09-21T10:24:15.829127+0000","last_fullsized":"2024-09-21T10:24:15.829127+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:22:05.604610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":377,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-09-21T10:24:31.224761+0000","last_change":"2024-09-21T10:10:57.401588+0000","last_active":"2024-09-21T10:24:31.224761+0000","last_peered":"2024-09-21T10:24:31.224761+0000","last_clean":"2024-09-21T10:24:31.224761+0000","last_became_active":"2024-09-21T10:10:57.397724+0000","last_became_peered":"2024-09-21T10:10:57.397724+0000","last_unstale":"2024-09-21T10:24:31.224761+0000","last_undegraded":"2024-09-21T10:24:31.224761+0000","last_fullsized":"2024-09-21T10:24:31.224761+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:47.843630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":16740,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26580,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":14,"num_pg":16,"stat_sum":{"num_bytes":7296843800,"num_objects":111342,"num_object_clones":0,"num_object_copies":334026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111342,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120236,"num_write_kb":7410433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21992189952,"data_stored":21992177736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117836,"ondisk_log_size":117836,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388626986,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3303240,"kb_used_data":1807616,"kb_used_omap":1,"kb_used_meta":1495614,"kb_avail":90441912,"statfs":{"total":95995035648,"available":92612517888,"internally_reserved":0,"allocated":1850998784,"data_stored":1848415206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1531509080},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019444,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4205980,"kb_used_data":2711504,"kb_used_omap":1,"kb_used_meta":1494398,"kb_avail":89539172,"statfs":{"total":95995035648,"available":91688112128,"internally_reserved":0,"allocated":2776580096,"data_stored":2773988568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1530263896},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":245,"seq":1052266987703,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4171460,"kb_used_data":3565948,"kb_used_omap":8,"kb_used_meta":605239,"kb_avail":89573692,"statfs":{"total":95995035648,"available":91723460608,"internally_reserved":0,"allocated":3651530752,"data_stored":3648955426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8940,"internal_metadata":619765012},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":175,"seq":751619277052,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2258648,"kb_used_data":1321356,"kb_used_omap":12,"kb_used_meta":937267,"kb_avail":91486504,"statfs":{"total":95995035648,"available":93682180096,"internally_reserved":0,"allocated":1353068544,"data_stored":1350481084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12845,"internal_metadata":959761875},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823215,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4043468,"kb_used_data":2265600,"kb_used_omap":1,"kb_used_meta":1777854,"kb_avail":89701684,"statfs":{"total":95995035648,"available":91854524416,"internally_reserved":0,"allocated":2319974400,"data_stored":2317394955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1820522840},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879638,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4277704,"kb_used_data":3105852,"kb_used_omap":4,"kb_used_meta":1171835,"kb_avail":89467448,"statfs":{"total":95995035648,"available":91614666752,"internally_reserved":0,"allocated":3180392448,"data_stored":3177817156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":1199959484},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566478,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4434880,"kb_used_data":3209272,"kb_used_omap":6,"kb_used_meta":1225593,"kb_avail":89310272,"statfs":{"total":95995035648,"available":91453718528,"internally_reserved":0,"allocated":3286294528,"data_stored":3283715551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":1255007655},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860823,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4967536,"kb_used_data":3640360,"kb_used_omap":5,"kb_used_meta":1327162,"kb_avail":88777616,"statfs":{"total":95995035648,"available":90908278784,"internally_reserved":0,"allocated":3727728640,"data_stored":3725164497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":1359014127},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7236,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9976,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3619880960,"data_stored":3619880960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3707895808,"data_stored":3707895808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3268214784,"data_stored":3268214784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3165519872,"data_stored":3165519872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2753302528,"data_stored":2753298456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2301104128,"data_stored":2301100056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1829441536,"data_stored":1829437464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1346830336,"data_stored":1346830336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:24:38.852 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T10:24:38.852 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:24:38.852 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-21T10:24:38.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-21T10:24:38.892 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:38.940 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:38.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:39.042 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:39.073 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:39.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:39.174 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:39.209 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:39.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:39.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:39.354 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:39.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:39.456 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:39.464 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:24:39.464 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 20, 2024-09-21T10:24:39.464 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 600, 2024-09-21T10:24:39.464 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:24:39.464 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.464 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.36822.0:197538 12.17 12:e9e12710:::benchmark_data_smithi107_250110_object86128:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-09-21T10:24:39.465 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:17:27.563584+0000", 2024-09-21T10:24:39.465 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 431.89656962800001, 2024-09-21T10:24:39.465 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.10558954800000001, 2024-09-21T10:24:39.465 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:39.465 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:39.465 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:39.465 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.36822", 2024-09-21T10:24:39.465 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/3844099828", 2024-09-21T10:24:39.465 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 197538 2024-09-21T10:24:39.465 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.465 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:39.465 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.465 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:39.465 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.563584+0000", 2024-09-21T10:24:39.466 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.466 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.466 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.466 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:39.466 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.563584+0000", 2024-09-21T10:24:39.466 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.466 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.466 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.466 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:39.466 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.563583+0000", 2024-09-21T10:24:39.466 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:24:39.466 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.467 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.467 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:39.467 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.563589+0000", 2024-09-21T10:24:39.467 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7679999999999997e-06 2024-09-21T10:24:39.467 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.467 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.467 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:39.467 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.563601+0000", 2024-09-21T10:24:39.467 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2298e-05 2024-09-21T10:24:39.467 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.467 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.467 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:39.467 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.563607+0000", 2024-09-21T10:24:39.467 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5790000000000003e-06 2024-09-21T10:24:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.563656+0000", 2024-09-21T10:24:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9462999999999997e-05 2024-09-21T10:24:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.563780+0000", 2024-09-21T10:24:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012353599999999999 2024-09-21T10:24:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:24:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.564202+0000", 2024-09-21T10:24:39.468 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00042223299999999999 2024-09-21T10:24:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.566505+0000", 2024-09-21T10:24:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0023028139999999998 2024-09-21T10:24:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.663685+0000", 2024-09-21T10:24:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.097180330999999995 2024-09-21T10:24:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.668909+0000", 2024-09-21T10:24:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0052239870000000002 2024-09-21T10:24:39.469 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.470 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.470 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:39.470 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.669132+0000", 2024-09-21T10:24:39.470 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00022286499999999999 2024-09-21T10:24:39.470 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.470 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.470 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:39.470 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:17:27.669173+0000", 2024-09-21T10:24:39.470 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1729e-05 2024-09-21T10:24:39.470 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.470 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.470 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.471 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.471 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.471 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:23842 13.4 13:2ae185ed:::benchmark_data_smithi107_278655_object23841:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:24:39.471 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:29.864431+0000", 2024-09-21T10:24:39.471 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 369.59572236700001, 2024-09-21T10:24:39.471 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.168915237, 2024-09-21T10:24:39.471 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:39.471 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:39.471 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:39.471 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:39.471 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:39.471 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 23842 2024-09-21T10:24:39.471 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.471 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:39.471 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.472 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:39.472 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864431+0000", 2024-09-21T10:24:39.472 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.472 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.472 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.472 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:39.472 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864431+0000", 2024-09-21T10:24:39.472 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.472 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.472 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.472 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:39.472 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864430+0000", 2024-09-21T10:24:39.472 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:24:39.472 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.472 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.473 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:39.473 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864487+0000", 2024-09-21T10:24:39.473 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7219000000000002e-05 2024-09-21T10:24:39.473 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.473 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.473 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:39.473 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864499+0000", 2024-09-21T10:24:39.473 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2062000000000001e-05 2024-09-21T10:24:39.473 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.473 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.473 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:39.473 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864505+0000", 2024-09-21T10:24:39.473 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0100000000000001e-06 2024-09-21T10:24:39.473 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.473 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.474 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:39.474 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864555+0000", 2024-09-21T10:24:39.474 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9574999999999999e-05 2024-09-21T10:24:39.474 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.474 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.474 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:39.474 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864687+0000", 2024-09-21T10:24:39.474 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013213 2024-09-21T10:24:39.474 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.474 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.474 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:24:39.474 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864915+0000", 2024-09-21T10:24:39.474 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00022785700000000001 2024-09-21T10:24:39.475 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.475 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.475 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:39.475 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.867465+0000", 2024-09-21T10:24:39.475 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0025501230000000001 2024-09-21T10:24:39.475 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.475 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.475 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.475 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.867521+0000", 2024-09-21T10:24:39.475 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5708000000000003e-05 2024-09-21T10:24:39.475 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.475 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.475 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.475 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:30.033258+0000", 2024-09-21T10:24:39.476 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.165737356 2024-09-21T10:24:39.476 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.476 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.476 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:39.476 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:30.033310+0000", 2024-09-21T10:24:39.476 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1539000000000002e-05 2024-09-21T10:24:39.476 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.476 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.476 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:39.476 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:30.033346+0000", 2024-09-21T10:24:39.476 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6936e-05 2024-09-21T10:24:39.476 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.476 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.476 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.476 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.477 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.477 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:23845 13.4 13:27245713:::benchmark_data_smithi107_278655_object23844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:24:39.477 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:29.866129+0000", 2024-09-21T10:24:39.477 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 369.59402459299997, 2024-09-21T10:24:39.477 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.16733213499999999, 2024-09-21T10:24:39.477 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:39.477 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:39.477 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:39.477 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:39.477 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:39.478 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 23845 2024-09-21T10:24:39.478 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.478 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:39.478 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.478 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:39.478 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866129+0000", 2024-09-21T10:24:39.478 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.478 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.478 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.479 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:39.479 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866129+0000", 2024-09-21T10:24:39.479 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.479 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.479 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.479 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:39.479 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866128+0000", 2024-09-21T10:24:39.479 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:24:39.479 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.479 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.479 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:39.479 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866189+0000", 2024-09-21T10:24:39.479 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1872999999999995e-05 2024-09-21T10:24:39.479 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.480 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.480 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:39.480 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866204+0000", 2024-09-21T10:24:39.480 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4739000000000001e-05 2024-09-21T10:24:39.480 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.480 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.480 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:39.480 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866211+0000", 2024-09-21T10:24:39.480 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2949999999999999e-06 2024-09-21T10:24:39.480 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.480 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.480 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:39.480 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866248+0000", 2024-09-21T10:24:39.480 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7704999999999999e-05 2024-09-21T10:24:39.480 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.481 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.481 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:39.481 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866354+0000", 2024-09-21T10:24:39.481 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000106022 2024-09-21T10:24:39.481 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.481 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.481 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:24:39.481 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866575+0000", 2024-09-21T10:24:39.481 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00022120399999999999 2024-09-21T10:24:39.481 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.481 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.481 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.481 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.867933+0000", 2024-09-21T10:24:39.481 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0013572499999999999 2024-09-21T10:24:39.481 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.481 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.482 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:39.482 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.867987+0000", 2024-09-21T10:24:39.482 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3919999999999999e-05 2024-09-21T10:24:39.482 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.482 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.482 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.482 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:30.033382+0000", 2024-09-21T10:24:39.482 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.165395343 2024-09-21T10:24:39.482 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.482 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.482 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:39.482 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:30.033430+0000", 2024-09-21T10:24:39.482 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7997999999999997e-05 2024-09-21T10:24:39.482 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.482 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.482 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:39.483 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:30.033461+0000", 2024-09-21T10:24:39.483 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1149999999999998e-05 2024-09-21T10:24:39.483 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.483 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.483 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.483 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.483 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.483 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:26201 13.4 13:2f8b127e:::benchmark_data_smithi107_278655_object26200:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:24:39.483 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:31.009445+0000", 2024-09-21T10:24:39.483 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 368.45070881700002, 2024-09-21T10:24:39.483 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.104710918, 2024-09-21T10:24:39.483 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:39.483 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:39.483 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:39.483 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:39.484 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:39.484 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 26201 2024-09-21T10:24:39.484 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.484 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:39.484 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.484 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:39.484 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009445+0000", 2024-09-21T10:24:39.484 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.484 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.484 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.484 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:39.484 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009445+0000", 2024-09-21T10:24:39.484 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.484 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.484 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.484 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:39.485 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009443+0000", 2024-09-21T10:24:39.485 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:24:39.485 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.485 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.485 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:39.485 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009480+0000", 2024-09-21T10:24:39.485 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6369999999999999e-05 2024-09-21T10:24:39.485 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.485 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.485 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:39.485 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009535+0000", 2024-09-21T10:24:39.485 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5179999999999997e-05 2024-09-21T10:24:39.485 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.485 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.485 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:39.486 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009543+0000", 2024-09-21T10:24:39.486 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.2099999999999993e-06 2024-09-21T10:24:39.486 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.486 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.486 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:39.486 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009591+0000", 2024-09-21T10:24:39.486 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7868000000000003e-05 2024-09-21T10:24:39.486 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.486 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.486 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:39.486 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009710+0000", 2024-09-21T10:24:39.486 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000118785 2024-09-21T10:24:39.486 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.486 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.486 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:24:39.487 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.009840+0000", 2024-09-21T10:24:39.487 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013075200000000001 2024-09-21T10:24:39.487 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.487 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.487 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.487 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.012789+0000", 2024-09-21T10:24:39.487 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0029491000000000001 2024-09-21T10:24:39.487 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.487 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.487 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:39.487 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.012838+0000", 2024-09-21T10:24:39.487 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9020999999999997e-05 2024-09-21T10:24:39.487 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.487 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.487 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.488 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.114063+0000", 2024-09-21T10:24:39.488 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.101224339 2024-09-21T10:24:39.488 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.488 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.488 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:39.488 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.114118+0000", 2024-09-21T10:24:39.488 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5630000000000001e-05 2024-09-21T10:24:39.488 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.488 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.488 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:39.488 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.114156+0000", 2024-09-21T10:24:39.488 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7219999999999999e-05 2024-09-21T10:24:39.488 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.488 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.488 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.489 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.489 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.489 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:26946 13.4 13:21264d98:::benchmark_data_smithi107_278655_object26945:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:24:39.489 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:31.498036+0000", 2024-09-21T10:24:39.489 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 367.96211768799998, 2024-09-21T10:24:39.489 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.30224518299999997, 2024-09-21T10:24:39.489 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:39.489 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:39.489 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:39.489 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:39.489 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:39.489 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 26946 2024-09-21T10:24:39.489 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.490 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:39.490 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.490 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:39.490 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498036+0000", 2024-09-21T10:24:39.490 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.490 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.490 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.490 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:39.490 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498036+0000", 2024-09-21T10:24:39.490 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.490 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.490 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.490 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:39.490 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498035+0000", 2024-09-21T10:24:39.491 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:24:39.491 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.491 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.491 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:39.491 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498090+0000", 2024-09-21T10:24:39.491 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5785000000000002e-05 2024-09-21T10:24:39.491 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.491 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.491 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:39.491 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498104+0000", 2024-09-21T10:24:39.491 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3871e-05 2024-09-21T10:24:39.491 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.491 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.491 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:39.491 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498111+0000", 2024-09-21T10:24:39.492 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4389999999999997e-06 2024-09-21T10:24:39.492 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.492 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.492 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:39.492 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498154+0000", 2024-09-21T10:24:39.492 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3238999999999997e-05 2024-09-21T10:24:39.492 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.492 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.492 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:39.492 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498237+0000", 2024-09-21T10:24:39.492 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.2571000000000006e-05 2024-09-21T10:24:39.492 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.492 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.492 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:24:39.492 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498343+0000", 2024-09-21T10:24:39.493 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000106498 2024-09-21T10:24:39.493 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.493 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.493 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:39.493 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.499830+0000", 2024-09-21T10:24:39.493 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0014867529999999999 2024-09-21T10:24:39.493 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.493 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.493 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.493 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.499999+0000", 2024-09-21T10:24:39.493 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00016944099999999999 2024-09-21T10:24:39.493 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.493 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.493 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.494 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.800209+0000", 2024-09-21T10:24:39.494 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.300209275 2024-09-21T10:24:39.494 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.494 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.494 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:39.494 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.800250+0000", 2024-09-21T10:24:39.494 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.189e-05 2024-09-21T10:24:39.494 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.494 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.494 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:39.494 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.800281+0000", 2024-09-21T10:24:39.494 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0618999999999999e-05 2024-09-21T10:24:39.494 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.494 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.495 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.495 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.495 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.495 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:61324 13.1 13:8d728946:::benchmark_data_smithi107_278655_object61323:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:24:39.495 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:52.281968+0000", 2024-09-21T10:24:39.495 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 347.178185821, 2024-09-21T10:24:39.495 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.105068279, 2024-09-21T10:24:39.495 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:39.495 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:39.495 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:39.495 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:39.495 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:39.495 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 61324 2024-09-21T10:24:39.495 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.495 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:39.496 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.496 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:39.496 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.281968+0000", 2024-09-21T10:24:39.496 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.496 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.496 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.496 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:39.496 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.281968+0000", 2024-09-21T10:24:39.496 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.496 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.496 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.496 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:39.496 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.281966+0000", 2024-09-21T10:24:39.496 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:24:39.496 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282001+0000", 2024-09-21T10:24:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4066000000000002e-05 2024-09-21T10:24:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282016+0000", 2024-09-21T10:24:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5172e-05 2024-09-21T10:24:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282021+0000", 2024-09-21T10:24:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7980000000000002e-06 2024-09-21T10:24:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.497 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.498 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:39.498 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282066+0000", 2024-09-21T10:24:39.498 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4577999999999999e-05 2024-09-21T10:24:39.498 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.498 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.498 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:39.498 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282222+0000", 2024-09-21T10:24:39.498 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015632499999999999 2024-09-21T10:24:39.498 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.498 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.498 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,4", 2024-09-21T10:24:39.498 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282384+0000", 2024-09-21T10:24:39.498 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00016192999999999999 2024-09-21T10:24:39.498 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.499 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.499 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:39.499 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.285898+0000", 2024-09-21T10:24:39.499 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0035140700000000002 2024-09-21T10:24:39.499 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.499 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.499 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.499 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.287757+0000", 2024-09-21T10:24:39.499 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0018584750000000001 2024-09-21T10:24:39.499 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.499 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.499 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.499 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.386791+0000", 2024-09-21T10:24:39.499 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.099033940000000001 2024-09-21T10:24:39.499 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.500 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.500 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:39.500 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.386935+0000", 2024-09-21T10:24:39.500 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014445300000000001 2024-09-21T10:24:39.500 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.500 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.500 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:39.500 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.387036+0000", 2024-09-21T10:24:39.500 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000100777 2024-09-21T10:24:39.500 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.500 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.500 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.500 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.500 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.500 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:61325 13.f 13:f42c75c5:::benchmark_data_smithi107_278655_object61324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:24:39.501 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:52.282123+0000", 2024-09-21T10:24:39.501 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 347.178030423, 2024-09-21T10:24:39.501 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.10875503, 2024-09-21T10:24:39.501 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:39.501 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:39.501 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:39.501 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:39.501 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:39.501 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 61325 2024-09-21T10:24:39.501 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.501 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:39.501 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.501 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:39.501 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282123+0000", 2024-09-21T10:24:39.501 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.502 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.502 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.502 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:39.502 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282123+0000", 2024-09-21T10:24:39.502 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.502 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.502 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.502 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:39.502 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282122+0000", 2024-09-21T10:24:39.502 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:24:39.502 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.502 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.502 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:39.503 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282177+0000", 2024-09-21T10:24:39.503 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5003999999999997e-05 2024-09-21T10:24:39.503 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.503 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.503 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:39.503 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282191+0000", 2024-09-21T10:24:39.503 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3472000000000001e-05 2024-09-21T10:24:39.503 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.503 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.503 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:39.503 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282196+0000", 2024-09-21T10:24:39.503 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5629999999999999e-06 2024-09-21T10:24:39.503 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.503 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.504 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:39.504 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282249+0000", 2024-09-21T10:24:39.504 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3183999999999998e-05 2024-09-21T10:24:39.504 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.504 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.504 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:39.504 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282360+0000", 2024-09-21T10:24:39.504 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000110653 2024-09-21T10:24:39.504 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.504 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.504 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:24:39.504 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.282491+0000", 2024-09-21T10:24:39.504 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013050400000000001 2024-09-21T10:24:39.504 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.504 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.505 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:39.505 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.288425+0000", 2024-09-21T10:24:39.505 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0059345939999999996 2024-09-21T10:24:39.505 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.505 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.505 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.505 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.288846+0000", 2024-09-21T10:24:39.505 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00042060099999999999 2024-09-21T10:24:39.505 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.505 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.505 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.505 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.390750+0000", 2024-09-21T10:24:39.505 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.101904019 2024-09-21T10:24:39.505 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.505 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.505 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:39.506 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.390835+0000", 2024-09-21T10:24:39.506 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.4732000000000001e-05 2024-09-21T10:24:39.506 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.506 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.506 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:39.506 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.390878+0000", 2024-09-21T10:24:39.506 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3671000000000003e-05 2024-09-21T10:24:39.506 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.506 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.506 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.506 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.506 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.506 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:61328 13.f 13:f2ce0a28:::benchmark_data_smithi107_278655_object61327:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:24:39.506 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:52.285810+0000", 2024-09-21T10:24:39.507 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 347.17434369300003, 2024-09-21T10:24:39.507 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.10897699299999999, 2024-09-21T10:24:39.507 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:39.507 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:39.507 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:39.507 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:39.507 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:39.507 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 61328 2024-09-21T10:24:39.507 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.507 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:39.507 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.507 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:39.507 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.285810+0000", 2024-09-21T10:24:39.507 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.507 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.508 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.508 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:39.508 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.285810+0000", 2024-09-21T10:24:39.508 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.508 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.508 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.508 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:39.508 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.285808+0000", 2024-09-21T10:24:39.508 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:24:39.508 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.508 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.508 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:39.508 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.285847+0000", 2024-09-21T10:24:39.508 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8485000000000002e-05 2024-09-21T10:24:39.508 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.509 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.509 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:39.509 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.285861+0000", 2024-09-21T10:24:39.509 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4246999999999999e-05 2024-09-21T10:24:39.509 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.509 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.509 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:39.509 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.285868+0000", 2024-09-21T10:24:39.509 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8020000000000003e-06 2024-09-21T10:24:39.509 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.509 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.509 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:39.509 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.285914+0000", 2024-09-21T10:24:39.510 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5728e-05 2024-09-21T10:24:39.510 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.510 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.510 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:39.510 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.286057+0000", 2024-09-21T10:24:39.510 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014371800000000001 2024-09-21T10:24:39.510 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.510 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.510 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:24:39.510 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.286210+0000", 2024-09-21T10:24:39.510 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000152972 2024-09-21T10:24:39.510 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.510 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.510 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.511 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.288879+0000", 2024-09-21T10:24:39.511 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002669051 2024-09-21T10:24:39.511 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.511 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.511 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:39.511 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.290367+0000", 2024-09-21T10:24:39.511 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001487302 2024-09-21T10:24:39.511 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.511 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.511 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.511 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.394700+0000", 2024-09-21T10:24:39.511 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.104333327 2024-09-21T10:24:39.511 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.511 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.511 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:39.512 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.394751+0000", 2024-09-21T10:24:39.512 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0930999999999997e-05 2024-09-21T10:24:39.512 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.512 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.512 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:39.512 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:52.394787+0000", 2024-09-21T10:24:39.512 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5923000000000001e-05 2024-09-21T10:24:39.512 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.512 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.512 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.512 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.512 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.512 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:72757 13.f 13:f96442f9:::benchmark_data_smithi107_278655_object72756:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:24:39.512 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.476101+0000", 2024-09-21T10:24:39.512 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 338.98405242000001, 2024-09-21T10:24:39.513 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.26259288400000003, 2024-09-21T10:24:39.513 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:39.513 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:39.513 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:39.513 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:39.513 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:39.513 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 72757 2024-09-21T10:24:39.513 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.513 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:39.513 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.513 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:39.513 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476101+0000", 2024-09-21T10:24:39.513 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.513 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.514 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.514 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:39.514 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476101+0000", 2024-09-21T10:24:39.514 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.514 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.514 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.514 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:39.514 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476100+0000", 2024-09-21T10:24:39.514 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:24:39.514 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.514 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.514 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:39.514 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476136+0000", 2024-09-21T10:24:39.514 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6081000000000002e-05 2024-09-21T10:24:39.515 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.515 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.515 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:39.515 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476151+0000", 2024-09-21T10:24:39.515 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4958e-05 2024-09-21T10:24:39.515 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.515 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.515 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:39.515 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476157+0000", 2024-09-21T10:24:39.515 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9150000000000001e-06 2024-09-21T10:24:39.515 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.515 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.515 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:39.515 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476242+0000", 2024-09-21T10:24:39.515 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.4896999999999997e-05 2024-09-21T10:24:39.516 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.516 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.516 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:39.516 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476468+0000", 2024-09-21T10:24:39.516 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00022605300000000001 2024-09-21T10:24:39.516 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.516 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.516 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:24:39.516 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476602+0000", 2024-09-21T10:24:39.516 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013458 2024-09-21T10:24:39.516 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.516 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.518 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.518 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481355+0000", 2024-09-21T10:24:39.518 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0047528479999999996 2024-09-21T10:24:39.518 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.518 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.518 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:39.519 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485113+0000", 2024-09-21T10:24:39.519 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003757842 2024-09-21T10:24:39.519 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.519 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.519 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.519 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.738610+0000", 2024-09-21T10:24:39.519 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25349742400000003 2024-09-21T10:24:39.519 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.519 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.519 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:39.519 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.738650+0000", 2024-09-21T10:24:39.519 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0089999999999997e-05 2024-09-21T10:24:39.519 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.519 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.520 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:39.520 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.738694+0000", 2024-09-21T10:24:39.520 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3649000000000002e-05 2024-09-21T10:24:39.520 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.520 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.520 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.520 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.520 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.520 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:72765 13.f 13:f6d6e7fe:::benchmark_data_smithi107_278655_object72764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:24:39.520 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.481220+0000", 2024-09-21T10:24:39.520 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 338.97893381199998, 2024-09-21T10:24:39.520 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.26411038599999997, 2024-09-21T10:24:39.520 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:39.520 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:39.520 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:39.521 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:39.521 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:39.521 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 72765 2024-09-21T10:24:39.521 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.521 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:39.521 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.521 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:39.521 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481220+0000", 2024-09-21T10:24:39.521 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.521 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.521 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.521 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:39.521 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481220+0000", 2024-09-21T10:24:39.521 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.522 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.522 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.522 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:39.522 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481218+0000", 2024-09-21T10:24:39.522 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:24:39.522 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.522 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.522 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:39.522 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481277+0000", 2024-09-21T10:24:39.522 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8371000000000001e-05 2024-09-21T10:24:39.522 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.522 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.522 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:39.522 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481289+0000", 2024-09-21T10:24:39.522 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2197e-05 2024-09-21T10:24:39.523 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.523 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.523 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:39.523 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481295+0000", 2024-09-21T10:24:39.523 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8309999999999997e-06 2024-09-21T10:24:39.523 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.523 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.523 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:39.523 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481444+0000", 2024-09-21T10:24:39.523 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014953500000000001 2024-09-21T10:24:39.523 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.523 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.523 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:39.523 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.482468+0000", 2024-09-21T10:24:39.523 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0010236080000000001 2024-09-21T10:24:39.524 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.524 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.524 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:24:39.524 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.482570+0000", 2024-09-21T10:24:39.524 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000101711 2024-09-21T10:24:39.524 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.524 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.524 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:39.524 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.488060+0000", 2024-09-21T10:24:39.524 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0054900289999999996 2024-09-21T10:24:39.524 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.524 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.524 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.524 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489602+0000", 2024-09-21T10:24:39.525 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0015419990000000001 2024-09-21T10:24:39.525 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.525 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.525 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.525 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745244+0000", 2024-09-21T10:24:39.525 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25564244800000002 2024-09-21T10:24:39.525 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.525 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.525 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:39.525 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745299+0000", 2024-09-21T10:24:39.525 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4877999999999999e-05 2024-09-21T10:24:39.525 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.525 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.525 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:39.525 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745330+0000", 2024-09-21T10:24:39.526 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1177000000000003e-05 2024-09-21T10:24:39.526 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.526 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.526 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.526 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.526 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.526 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:72767 13.f 13:fba5e0d5:::benchmark_data_smithi107_278655_object72766:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:24:39.526 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.481760+0000", 2024-09-21T10:24:39.526 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 338.97839351200003, 2024-09-21T10:24:39.526 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.263929786, 2024-09-21T10:24:39.526 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:39.526 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:39.526 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:39.526 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:39.527 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:39.527 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 72767 2024-09-21T10:24:39.527 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.527 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:39.527 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.527 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:39.527 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481760+0000", 2024-09-21T10:24:39.527 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.527 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.527 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.527 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:39.527 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481760+0000", 2024-09-21T10:24:39.527 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.527 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.527 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.528 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:39.528 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481759+0000", 2024-09-21T10:24:39.528 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:24:39.528 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.528 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.528 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:39.528 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481815+0000", 2024-09-21T10:24:39.528 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6230000000000002e-05 2024-09-21T10:24:39.528 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.528 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.528 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:39.528 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481831+0000", 2024-09-21T10:24:39.528 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5613000000000001e-05 2024-09-21T10:24:39.528 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.528 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.528 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:39.529 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481840+0000", 2024-09-21T10:24:39.529 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.9390000000000003e-06 2024-09-21T10:24:39.529 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.529 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.529 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:39.529 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485140+0000", 2024-09-21T10:24:39.529 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0033002909999999999 2024-09-21T10:24:39.529 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.529 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.529 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:39.529 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485285+0000", 2024-09-21T10:24:39.529 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014537099999999999 2024-09-21T10:24:39.529 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.529 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.530 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:24:39.530 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485433+0000", 2024-09-21T10:24:39.530 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014723399999999999 2024-09-21T10:24:39.530 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.530 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.530 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:39.530 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489134+0000", 2024-09-21T10:24:39.530 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0037009009999999999 2024-09-21T10:24:39.530 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.530 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.530 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.530 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489909+0000", 2024-09-21T10:24:39.530 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00077549400000000005 2024-09-21T10:24:39.530 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.531 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.531 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.531 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745617+0000", 2024-09-21T10:24:39.531 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.255707831 2024-09-21T10:24:39.531 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.531 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.531 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:39.531 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745660+0000", 2024-09-21T10:24:39.531 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2803999999999999e-05 2024-09-21T10:24:39.531 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.531 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.531 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:39.531 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745690+0000", 2024-09-21T10:24:39.531 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0079000000000001e-05 2024-09-21T10:24:39.532 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.532 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.532 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.532 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.532 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.532 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:72770 13.1 13:87104e38:::benchmark_data_smithi107_278655_object72769:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:24:39.532 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.485917+0000", 2024-09-21T10:24:39.532 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 338.97423665500003, 2024-09-21T10:24:39.532 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25941609300000001, 2024-09-21T10:24:39.532 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:39.532 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:39.532 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:39.532 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:39.532 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:39.533 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 72770 2024-09-21T10:24:39.533 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.533 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:39.533 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.533 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:39.533 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485917+0000", 2024-09-21T10:24:39.533 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.533 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.533 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.533 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:39.533 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485917+0000", 2024-09-21T10:24:39.533 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.533 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.534 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.534 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:39.534 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485915+0000", 2024-09-21T10:24:39.534 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:24:39.534 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.534 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.534 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:39.534 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485978+0000", 2024-09-21T10:24:39.534 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3101000000000005e-05 2024-09-21T10:24:39.534 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.534 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.534 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:39.534 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485996+0000", 2024-09-21T10:24:39.534 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.7759000000000001e-05 2024-09-21T10:24:39.535 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.535 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.535 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:39.535 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.486004+0000", 2024-09-21T10:24:39.535 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3239999999999997e-06 2024-09-21T10:24:39.535 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.535 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.535 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:39.535 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.486055+0000", 2024-09-21T10:24:39.535 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1193000000000003e-05 2024-09-21T10:24:39.535 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.535 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.535 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:39.535 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.486192+0000", 2024-09-21T10:24:39.536 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013693200000000001 2024-09-21T10:24:39.536 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.536 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.536 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,4", 2024-09-21T10:24:39.536 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.486328+0000", 2024-09-21T10:24:39.536 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013682700000000001 2024-09-21T10:24:39.536 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.536 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.536 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:39.536 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489305+0000", 2024-09-21T10:24:39.536 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002977001 2024-09-21T10:24:39.536 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.536 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.536 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.536 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489525+0000", 2024-09-21T10:24:39.537 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00021918799999999999 2024-09-21T10:24:39.537 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.537 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.537 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.537 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745228+0000", 2024-09-21T10:24:39.537 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25570367500000002 2024-09-21T10:24:39.537 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.537 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.537 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:39.537 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745295+0000", 2024-09-21T10:24:39.537 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7137000000000005e-05 2024-09-21T10:24:39.537 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.537 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.537 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:39.538 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745333+0000", 2024-09-21T10:24:39.538 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7518999999999998e-05 2024-09-21T10:24:39.538 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.538 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.538 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.538 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.538 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.538 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:72774 13.4 13:248222d3:::benchmark_data_smithi107_278655_object72773:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:24:39.538 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.489389+0000", 2024-09-21T10:24:39.538 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 338.97076429200001, 2024-09-21T10:24:39.538 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25593706900000002, 2024-09-21T10:24:39.538 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:39.538 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:39.538 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:39.539 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:24:39.539 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:24:39.539 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 72774 2024-09-21T10:24:39.539 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.539 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:39.539 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.539 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:39.539 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489389+0000", 2024-09-21T10:24:39.539 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.539 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.539 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.539 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:39.539 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489389+0000", 2024-09-21T10:24:39.539 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.540 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.540 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.540 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:39.540 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489388+0000", 2024-09-21T10:24:39.540 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:24:39.540 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.540 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.540 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:39.540 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489476+0000", 2024-09-21T10:24:39.540 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.8084999999999994e-05 2024-09-21T10:24:39.540 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.540 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.540 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:39.540 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489490+0000", 2024-09-21T10:24:39.541 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4139e-05 2024-09-21T10:24:39.541 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.541 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.541 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:39.541 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489498+0000", 2024-09-21T10:24:39.541 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.1899999999999995e-06 2024-09-21T10:24:39.541 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.541 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.541 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:39.541 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489554+0000", 2024-09-21T10:24:39.541 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5649999999999997e-05 2024-09-21T10:24:39.541 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.541 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.541 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:39.541 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489701+0000", 2024-09-21T10:24:39.542 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014734700000000001 2024-09-21T10:24:39.542 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.542 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.542 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:24:39.542 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489837+0000", 2024-09-21T10:24:39.542 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013627299999999999 2024-09-21T10:24:39.542 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.542 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.542 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:39.542 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.492678+0000", 2024-09-21T10:24:39.542 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0028408890000000001 2024-09-21T10:24:39.542 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.542 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.542 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.543 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.492823+0000", 2024-09-21T10:24:39.543 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014503199999999999 2024-09-21T10:24:39.543 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.543 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.543 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.543 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745232+0000", 2024-09-21T10:24:39.543 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25240888500000003 2024-09-21T10:24:39.543 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.543 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.543 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:39.543 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745285+0000", 2024-09-21T10:24:39.543 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3041000000000003e-05 2024-09-21T10:24:39.543 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.543 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.544 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:39.544 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745326+0000", 2024-09-21T10:24:39.544 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1245000000000002e-05 2024-09-21T10:24:39.544 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.544 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.544 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.544 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.544 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.544 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:18227 14.2 14:4ecdcd6d:::benchmark_data_smithi107_312686_object18226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-21T10:24:39.544 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:23:03.357929+0000", 2024-09-21T10:24:39.544 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 96.102224191000005, 2024-09-21T10:24:39.544 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21685865500000001, 2024-09-21T10:24:39.544 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:39.544 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:39.545 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:39.545 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:24:39.545 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:24:39.545 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 18227 2024-09-21T10:24:39.545 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.545 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:39.545 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.545 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:39.545 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.357929+0000", 2024-09-21T10:24:39.545 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.545 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.545 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.545 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:39.545 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.357929+0000", 2024-09-21T10:24:39.545 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.546 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.546 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.546 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:39.546 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.357928+0000", 2024-09-21T10:24:39.546 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:24:39.546 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.546 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.546 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:39.546 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.357966+0000", 2024-09-21T10:24:39.546 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7462999999999997e-05 2024-09-21T10:24:39.546 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.546 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.547 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:39.547 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.357980+0000", 2024-09-21T10:24:39.547 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4613999999999999e-05 2024-09-21T10:24:39.547 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.547 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.547 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:39.547 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.357986+0000", 2024-09-21T10:24:39.547 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7479999999999999e-06 2024-09-21T10:24:39.547 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.547 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.547 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:39.547 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.358093+0000", 2024-09-21T10:24:39.547 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000107163 2024-09-21T10:24:39.547 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.548 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.548 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:39.548 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.358206+0000", 2024-09-21T10:24:39.548 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000113193 2024-09-21T10:24:39.548 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.548 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.548 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:24:39.548 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.358336+0000", 2024-09-21T10:24:39.548 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012923200000000001 2024-09-21T10:24:39.548 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.548 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.548 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:39.548 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.359740+0000", 2024-09-21T10:24:39.548 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0014040840000000001 2024-09-21T10:24:39.549 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.549 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.549 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.549 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.360595+0000", 2024-09-21T10:24:39.549 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00085584699999999997 2024-09-21T10:24:39.549 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.549 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.549 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.549 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.574681+0000", 2024-09-21T10:24:39.549 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.214085942 2024-09-21T10:24:39.549 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.549 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.549 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:39.549 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.574741+0000", 2024-09-21T10:24:39.549 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.999e-05 2024-09-21T10:24:39.550 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.550 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.550 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:39.550 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.574788+0000", 2024-09-21T10:24:39.550 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6610999999999998e-05 2024-09-21T10:24:39.550 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.550 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.550 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.550 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.550 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.550 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:28305 14.2 14:436767ef:::benchmark_data_smithi107_312686_object28304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-21T10:24:39.550 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:23:08.650026+0000", 2024-09-21T10:24:39.550 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 90.810128035000005, 2024-09-21T10:24:39.551 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.114775101, 2024-09-21T10:24:39.551 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:39.551 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:39.551 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:39.551 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:24:39.551 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:24:39.551 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 28305 2024-09-21T10:24:39.551 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.551 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:39.551 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.551 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:39.551 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.650026+0000", 2024-09-21T10:24:39.551 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.551 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.552 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.552 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:39.552 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.650026+0000", 2024-09-21T10:24:39.552 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.552 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.552 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.552 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:39.552 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.650024+0000", 2024-09-21T10:24:39.552 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:24:39.552 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.552 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.552 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:39.552 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.650103+0000", 2024-09-21T10:24:39.552 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9327999999999997e-05 2024-09-21T10:24:39.552 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.553 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.553 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:39.553 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.650113+0000", 2024-09-21T10:24:39.553 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.004e-05 2024-09-21T10:24:39.553 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.553 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.553 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:39.553 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.650118+0000", 2024-09-21T10:24:39.553 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2920000000000003e-06 2024-09-21T10:24:39.553 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.553 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.553 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:39.553 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.650170+0000", 2024-09-21T10:24:39.553 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1579000000000002e-05 2024-09-21T10:24:39.554 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.554 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.554 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:39.554 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.650302+0000", 2024-09-21T10:24:39.554 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013227200000000001 2024-09-21T10:24:39.554 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.554 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.554 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:24:39.554 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.650449+0000", 2024-09-21T10:24:39.554 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000147031 2024-09-21T10:24:39.554 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.554 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.554 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:39.554 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.651586+0000", 2024-09-21T10:24:39.555 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001136749 2024-09-21T10:24:39.555 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.555 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.555 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.555 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.654653+0000", 2024-09-21T10:24:39.555 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00306676 2024-09-21T10:24:39.555 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.555 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.555 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.555 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.764721+0000", 2024-09-21T10:24:39.555 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.11006858799999999 2024-09-21T10:24:39.555 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.555 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.555 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:39.555 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.764764+0000", 2024-09-21T10:24:39.556 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2369e-05 2024-09-21T10:24:39.556 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.556 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.556 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:39.556 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.764801+0000", 2024-09-21T10:24:39.556 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6826999999999998e-05 2024-09-21T10:24:39.556 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.556 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.556 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.556 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.556 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.556 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:28315 14.2 14:43724169:::benchmark_data_smithi107_312686_object28314:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-21T10:24:39.556 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:23:08.655735+0000", 2024-09-21T10:24:39.556 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 90.804418691999999, 2024-09-21T10:24:39.557 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.109156055, 2024-09-21T10:24:39.557 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:39.557 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:39.557 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:39.557 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:24:39.557 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:24:39.557 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 28315 2024-09-21T10:24:39.557 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.557 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:39.557 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.557 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:39.557 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.655735+0000", 2024-09-21T10:24:39.557 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.557 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.558 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.558 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:39.558 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.655735+0000", 2024-09-21T10:24:39.558 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.558 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.558 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.558 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:39.558 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.655734+0000", 2024-09-21T10:24:39.558 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:24:39.558 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.558 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.558 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:39.558 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.655788+0000", 2024-09-21T10:24:39.558 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4453999999999997e-05 2024-09-21T10:24:39.559 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.559 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.559 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:39.559 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.655799+0000", 2024-09-21T10:24:39.559 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0803e-05 2024-09-21T10:24:39.559 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.559 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.559 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:39.559 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.655804+0000", 2024-09-21T10:24:39.559 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5120000000000002e-06 2024-09-21T10:24:39.559 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.559 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.559 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:39.559 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.655843+0000", 2024-09-21T10:24:39.559 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8903999999999999e-05 2024-09-21T10:24:39.560 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.560 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.560 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:39.560 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.655969+0000", 2024-09-21T10:24:39.560 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012635200000000001 2024-09-21T10:24:39.560 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.560 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.560 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:24:39.560 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.656101+0000", 2024-09-21T10:24:39.560 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013252899999999999 2024-09-21T10:24:39.560 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.560 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.560 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.560 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.764667+0000", 2024-09-21T10:24:39.561 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.10856518599999999 2024-09-21T10:24:39.561 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.561 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.561 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:39.561 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.764672+0000", 2024-09-21T10:24:39.561 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7740000000000001e-06 2024-09-21T10:24:39.561 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.561 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.561 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.561 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.764832+0000", 2024-09-21T10:24:39.561 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015917199999999999 2024-09-21T10:24:39.561 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.561 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.561 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:39.561 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.764864+0000", 2024-09-21T10:24:39.562 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2381e-05 2024-09-21T10:24:39.562 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.562 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.562 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:39.562 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.764891+0000", 2024-09-21T10:24:39.562 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7025e-05 2024-09-21T10:24:39.562 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.562 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.562 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.562 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.562 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.562 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:28320 14.2 14:44134065:::benchmark_data_smithi107_312686_object28319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-21T10:24:39.562 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:23:08.655811+0000", 2024-09-21T10:24:39.562 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 90.804342192999997, 2024-09-21T10:24:39.563 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.11174126500000001, 2024-09-21T10:24:39.563 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:39.563 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:39.563 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:39.563 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:24:39.563 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:24:39.563 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 28320 2024-09-21T10:24:39.563 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.563 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:39.563 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.563 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:39.563 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.655811+0000", 2024-09-21T10:24:39.563 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.564 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.564 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.564 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:39.564 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.655811+0000", 2024-09-21T10:24:39.564 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.564 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.564 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.564 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:39.564 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.655811+0000", 2024-09-21T10:24:39.564 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:24:39.564 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.564 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.564 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:39.564 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.655860+0000", 2024-09-21T10:24:39.564 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8943999999999998e-05 2024-09-21T10:24:39.565 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.565 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.565 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:39.565 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.655867+0000", 2024-09-21T10:24:39.565 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.221e-06 2024-09-21T10:24:39.565 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.565 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.565 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:39.565 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.655870+0000", 2024-09-21T10:24:39.565 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.915e-06 2024-09-21T10:24:39.565 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.565 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.565 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:39.565 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.657021+0000", 2024-09-21T10:24:39.566 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0011509409999999999 2024-09-21T10:24:39.566 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.566 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.566 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:39.566 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.657110+0000", 2024-09-21T10:24:39.566 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.9431999999999993e-05 2024-09-21T10:24:39.566 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.566 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.566 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:24:39.566 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.657197+0000", 2024-09-21T10:24:39.566 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6895000000000001e-05 2024-09-21T10:24:39.566 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.566 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.566 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.567 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.764698+0000", 2024-09-21T10:24:39.567 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.107500413 2024-09-21T10:24:39.567 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.567 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.567 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.567 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.764921+0000", 2024-09-21T10:24:39.567 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00022341400000000001 2024-09-21T10:24:39.567 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.567 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.567 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:39.567 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.767461+0000", 2024-09-21T10:24:39.567 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002540156 2024-09-21T10:24:39.567 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.567 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.568 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:39.568 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.767505+0000", 2024-09-21T10:24:39.568 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3649999999999997e-05 2024-09-21T10:24:39.568 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.568 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.568 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:39.568 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.767553+0000", 2024-09-21T10:24:39.568 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7670999999999998e-05 2024-09-21T10:24:39.568 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.568 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.568 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.568 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.568 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.568 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:28321 14.4 14:2c49af65:::benchmark_data_smithi107_312686_object28320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-21T10:24:39.568 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:23:08.657485+0000", 2024-09-21T10:24:39.569 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 90.802668885000003, 2024-09-21T10:24:39.569 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.112163366, 2024-09-21T10:24:39.569 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:39.569 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:39.569 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:39.569 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:24:39.569 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:24:39.569 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 28321 2024-09-21T10:24:39.569 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.569 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:39.569 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.569 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:39.569 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.657485+0000", 2024-09-21T10:24:39.569 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.569 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.570 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.570 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:39.570 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.657485+0000", 2024-09-21T10:24:39.570 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.570 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.570 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.570 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:39.570 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.657484+0000", 2024-09-21T10:24:39.570 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:24:39.570 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.570 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.570 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:39.570 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.657639+0000", 2024-09-21T10:24:39.570 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015507300000000001 2024-09-21T10:24:39.571 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.571 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.571 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:39.571 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.763425+0000", 2024-09-21T10:24:39.571 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.105786238 2024-09-21T10:24:39.571 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.571 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.571 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:39.571 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.763437+0000", 2024-09-21T10:24:39.571 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2276000000000001e-05 2024-09-21T10:24:39.571 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.571 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.571 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:39.571 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.763495+0000", 2024-09-21T10:24:39.572 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7927999999999998e-05 2024-09-21T10:24:39.572 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.572 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.572 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:39.572 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.763645+0000", 2024-09-21T10:24:39.572 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000149476 2024-09-21T10:24:39.572 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.572 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.572 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:24:39.572 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.763795+0000", 2024-09-21T10:24:39.572 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000150087 2024-09-21T10:24:39.572 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.572 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.572 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.572 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.767850+0000", 2024-09-21T10:24:39.573 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0040555410000000002 2024-09-21T10:24:39.573 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.573 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.573 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.573 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.767885+0000", 2024-09-21T10:24:39.573 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4826000000000002e-05 2024-09-21T10:24:39.573 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.573 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.573 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:39.573 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.769522+0000", 2024-09-21T10:24:39.573 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0016363829999999999 2024-09-21T10:24:39.573 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.573 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.573 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:39.573 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.769569+0000", 2024-09-21T10:24:39.574 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6894000000000002e-05 2024-09-21T10:24:39.574 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.574 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.574 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:39.574 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.769648+0000", 2024-09-21T10:24:39.574 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9491000000000002e-05 2024-09-21T10:24:39.574 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.574 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.574 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.574 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.574 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.574 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.47186.0:28317 14.e e416/411 14:7d196cb4:::benchmark_data_smithi107_312686_object28316:head v 416'1782, mlcod=416'1779)", 2024-09-21T10:24:39.574 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:23:08.657695+0000", 2024-09-21T10:24:39.574 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 90.802458400999996, 2024-09-21T10:24:39.575 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.108974742, 2024-09-21T10:24:39.575 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:39.575 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:39.575 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:39.575 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.575 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:39.575 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.657695+0000", 2024-09-21T10:24:39.575 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.575 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.575 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.575 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:39.575 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.657695+0000", 2024-09-21T10:24:39.575 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.575 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.575 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.576 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:39.576 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.657695+0000", 2024-09-21T10:24:39.576 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:24:39.576 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.576 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.576 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:39.576 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.657809+0000", 2024-09-21T10:24:39.576 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000114273 2024-09-21T10:24:39.576 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.576 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.576 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:39.576 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.763428+0000", 2024-09-21T10:24:39.576 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.10561899700000001 2024-09-21T10:24:39.576 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.576 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.577 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:39.577 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.763436+0000", 2024-09-21T10:24:39.577 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.1380000000000007e-06 2024-09-21T10:24:39.577 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.577 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.577 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:39.577 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.763494+0000", 2024-09-21T10:24:39.577 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7465e-05 2024-09-21T10:24:39.577 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.577 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.577 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:39.577 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.763515+0000", 2024-09-21T10:24:39.577 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1772000000000001e-05 2024-09-21T10:24:39.577 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.577 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.578 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:39.578 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.766638+0000", 2024-09-21T10:24:39.578 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0031227669999999998 2024-09-21T10:24:39.578 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.578 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.578 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:39.578 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.766670+0000", 2024-09-21T10:24:39.578 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1853999999999997e-05 2024-09-21T10:24:39.578 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.578 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.578 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.578 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.578 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.578 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:32919 14.2 14:427e62e5:::benchmark_data_smithi107_312686_object32918:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-21T10:24:39.579 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:23:11.296297+0000", 2024-09-21T10:24:39.579 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 88.163856581999994, 2024-09-21T10:24:39.579 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.131622454, 2024-09-21T10:24:39.579 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:24:39.579 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:39.579 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:24:39.579 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:24:39.579 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:24:39.579 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32919 2024-09-21T10:24:39.579 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.579 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:24:39.579 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.579 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:24:39.579 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.296297+0000", 2024-09-21T10:24:39.579 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.580 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.580 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.580 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:24:39.580 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.296297+0000", 2024-09-21T10:24:39.580 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:24:39.580 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.580 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.580 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:24:39.580 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.296296+0000", 2024-09-21T10:24:39.580 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:24:39.580 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.580 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.580 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:24:39.580 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.296421+0000", 2024-09-21T10:24:39.580 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012501100000000001 2024-09-21T10:24:39.581 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.581 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.581 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:24:39.581 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.296432+0000", 2024-09-21T10:24:39.581 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1276e-05 2024-09-21T10:24:39.581 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.581 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.581 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:24:39.581 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.296438+0000", 2024-09-21T10:24:39.581 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0120000000000002e-06 2024-09-21T10:24:39.581 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.581 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.581 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:24:39.581 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.296471+0000", 2024-09-21T10:24:39.581 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2566e-05 2024-09-21T10:24:39.582 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.582 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.582 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:24:39.582 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.296572+0000", 2024-09-21T10:24:39.582 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000100957 2024-09-21T10:24:39.582 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.582 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.582 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:24:39.582 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.296681+0000", 2024-09-21T10:24:39.582 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000108874 2024-09-21T10:24:39.582 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.582 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.582 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:24:39.582 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.297778+0000", 2024-09-21T10:24:39.582 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001097338 2024-09-21T10:24:39.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.583 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.583 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.583 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.427397+0000", 2024-09-21T10:24:39.583 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.129619022 2024-09-21T10:24:39.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.583 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.583 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:39.583 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.427838+0000", 2024-09-21T10:24:39.583 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00044068699999999999 2024-09-21T10:24:39.583 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.583 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.583 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:24:39.583 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.427875+0000", 2024-09-21T10:24:39.584 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7305000000000003e-05 2024-09-21T10:24:39.584 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:24:39.584 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:24:39.584 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:24:39.584 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.427919+0000", 2024-09-21T10:24:39.584 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4286000000000003e-05 2024-09-21T10:24:39.584 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.584 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.584 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.584 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:24:39.584 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:24:39.584 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:24:39.585 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:39.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:39.612 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:39.686 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:39.711 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:39.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:39.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:39.865 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:39.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:39.967 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:39.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.990 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:39.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:40.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:40.127 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:40.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:40.228 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:40.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.272 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:40.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:40.332 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:24:40.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:40.410 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:40.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:40.432 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:40.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:40.571 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:40.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:40.611 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:24:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:24:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:24:40.669 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:24:40.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:40.720 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:40.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:40.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:40.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:40 smithi107 ceph-mon[102221]: pgmap v1500: 109 pgs: 109 active+clean; 6.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 0 B/s wr, 857 op/s 2024-09-21T10:24:40.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:40 smithi107 ceph-mon[105022]: pgmap v1500: 109 pgs: 109 active+clean; 6.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 0 B/s wr, 857 op/s 2024-09-21T10:24:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.872 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:40.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:40.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:40 smithi129 ceph-mon[88471]: pgmap v1500: 109 pgs: 109 active+clean; 6.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 0 B/s wr, 857 op/s 2024-09-21T10:24:40.973 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:41.001 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:41.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:41.103 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:41.136 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:41.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:41.238 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:41.271 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:41.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:41.373 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:41.401 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:41.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:41.503 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:41.531 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:41.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:41.586 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:24:41.633 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:41.657 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:41.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:41.759 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:41.798 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:41.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:41.899 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:41.924 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:41.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:42.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:42.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.067 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:42.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:42.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:24:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:24:41] "GET /metrics HTTP/1.1" 200 36776 "" "Prometheus/2.51.0" 2024-09-21T10:24:42.169 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:42.193 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:42.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:42.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:42.347 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:42.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:42.449 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:42.478 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:42.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:42.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:24:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:24:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:42.624 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:42.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:24:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:42.725 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:42.750 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:42.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:42.851 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:42.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.880 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:42.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:42.981 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:43.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.031 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:43.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:43.097 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:43.133 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:43.181 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:43.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:43.283 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:43.311 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:43.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:43.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:43 smithi107 ceph-mon[102221]: pgmap v1501: 109 pgs: 109 active+clean; 6.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 0 B/s wr, 830 op/s 2024-09-21T10:24:43.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:43 smithi107 ceph-mon[105022]: pgmap v1501: 109 pgs: 109 active+clean; 6.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 0 B/s wr, 830 op/s 2024-09-21T10:24:43.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:43.444 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:43.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:43.521 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:43 smithi129 ceph-mon[88471]: pgmap v1501: 109 pgs: 109 active+clean; 6.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 0 B/s wr, 830 op/s 2024-09-21T10:24:43.546 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:43.576 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:43.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:43.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:43.721 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:43.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:43.822 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:43.867 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:43.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:43.945 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:24:43.945 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:24:43.945 INFO:teuthology.orchestra.run.smithi129.stdout: "num_ops": 0 2024-09-21T10:24:43.945 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:24:43.954 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-09-21T10:24:43.955 INFO:teuthology.orchestra.run.smithi107.stderr:0 2024-09-21T10:24:43.955 INFO:teuthology.orchestra.run.smithi107.stderr:1 2024-09-21T10:24:43.955 INFO:teuthology.orchestra.run.smithi107.stderr:2 2024-09-21T10:24:43.955 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2024-09-21T10:24:43.955 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2024-09-21T10:24:43.955 INFO:teuthology.orchestra.run.smithi107.stderr:bench [] [] [] [] 2024-09-21T10:24:43.955 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:24:43.955 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2024-09-21T10:24:43.955 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2024-09-21T10:24:43.955 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2024-09-21T10:24:43.956 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-09-21T10:24:43.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:44.008 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:44.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:44.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:44.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.152 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:44.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:44.212 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:24:44.253 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:44.263 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:44 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3855180139' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:44.263 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:44 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:44.264 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:44 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3855180139' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:44.264 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:44 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:44.279 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:44.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:44.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:44.421 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:44.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:44.523 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:44.548 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:44.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:44.600 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:44 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3855180139' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:44.600 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:44 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:44.635 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:24:44.650 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:44.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.715 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:44.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:44.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:44.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.861 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:44.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:44.962 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:44.988 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:24:44.999 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:45.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:45.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:45.143 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:45.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:45.244 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:45.281 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:45 smithi107 ceph-mon[102221]: pgmap v1502: 109 pgs: 109 active+clean; 6.3 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 0 B/s wr, 999 op/s 2024-09-21T10:24:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:45 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:45 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:24:45.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:45 smithi107 ceph-mon[102221]: osdmap e427: 8 total, 8 up, 8 in 2024-09-21T10:24:45.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:45 smithi107 ceph-mon[105022]: pgmap v1502: 109 pgs: 109 active+clean; 6.3 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 0 B/s wr, 999 op/s 2024-09-21T10:24:45.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:45 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:45.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:45 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:24:45.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:45 smithi107 ceph-mon[105022]: osdmap e427: 8 total, 8 up, 8 in 2024-09-21T10:24:45.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:45.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:45 smithi129 ceph-mon[88471]: pgmap v1502: 109 pgs: 109 active+clean; 6.3 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 0 B/s wr, 999 op/s 2024-09-21T10:24:45.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:45 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:45.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:45 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:24:45.420 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:45 smithi129 ceph-mon[88471]: osdmap e427: 8 total, 8 up, 8 in 2024-09-21T10:24:45.426 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:45.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:45.531 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:45.559 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:45.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:45.661 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:45.695 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:45.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:45.797 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:45.820 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:45.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:45.922 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:45.949 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:45.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:46.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:46.062 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:24:46.063 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:24:46.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-21T10:24:46.092 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:46.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:46.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:46.244 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:46.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:46.346 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:46.387 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:46.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:46.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:46.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.534 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:46.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:46.635 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:46.675 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:46.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:46.776 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:46.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.812 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:46.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:46.845 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:46.895 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:24:46.895 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:24:46.895 INFO:teuthology.orchestra.run.smithi129.stdout: "complaint_time": 30, 2024-09-21T10:24:46.895 INFO:teuthology.orchestra.run.smithi129.stdout: "num_blocked_ops": 0 2024-09-21T10:24:46.895 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:24:46.914 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:46.966 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:46.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:47.067 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:47.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:24:47.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.098 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:47.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:47.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:47.249 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:47.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:47.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:47 smithi107 ceph-mon[102221]: pgmap v1504: 109 pgs: 109 active+clean; 6.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 0 B/s wr, 843 op/s 2024-09-21T10:24:47.300 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:47 smithi107 ceph-mon[105022]: pgmap v1504: 109 pgs: 109 active+clean; 6.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 0 B/s wr, 843 op/s 2024-09-21T10:24:47.352 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:47.380 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:24:47.381 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:24:47.411 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:47.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:47.438 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:24:47.439 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:24:47.470 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:24:47.470 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:24:47.471 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:47 smithi129 ceph-mon[88471]: pgmap v1504: 109 pgs: 109 active+clean; 6.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 0 B/s wr, 843 op/s 2024-09-21T10:24:47.507 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:24:47.507 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:24:47.512 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:47.536 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:47.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:47.541 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:24:47.541 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:24:47.573 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:24:47.573 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:24:47.603 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:24:47.603 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:24:47.631 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:24:47.631 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:24:47.638 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:47.659 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:47.760 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:47.782 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:47.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:47.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:47.916 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:47.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:48.018 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:48.042 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:48.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:48.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:48.184 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:48.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:48.285 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:48.315 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:48.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:48.410 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:24:48.417 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:48.481 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:48.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:48.557 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:48 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1877413421' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:48.558 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:48 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1877413421' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:48.582 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:48.604 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:48.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:48.635 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:48.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:48 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1877413421' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:48.705 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:48.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.738 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:48.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:48.740 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:24:48.839 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:48.870 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:48.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:48.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:49.004 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:49.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:49.105 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:49.128 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:49.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:49.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:49.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.278 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:49.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:49.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:49 smithi107 ceph-mon[102221]: pgmap v1505: 109 pgs: 109 active+clean; 6.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 0 B/s wr, 842 op/s 2024-09-21T10:24:49.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:49 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:24:49.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:49 smithi107 ceph-mon[105022]: pgmap v1505: 109 pgs: 109 active+clean; 6.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 0 B/s wr, 842 op/s 2024-09-21T10:24:49.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:49 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:24:49.379 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:49.405 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:49.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:49.506 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:49.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.554 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:49.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:49.655 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:49 smithi129 ceph-mon[88471]: pgmap v1505: 109 pgs: 109 active+clean; 6.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 0 B/s wr, 842 op/s 2024-09-21T10:24:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:49 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:24:49.679 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:49.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:49.783 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:49.826 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:49.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:49.929 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:49.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.973 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:49.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:50.074 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:50.102 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:50.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:50.204 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:50.230 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:50.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:50.332 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:50.361 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:50.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:50.365 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:24:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:24:50.217Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:24:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-09-21T10:24:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:0 2024-09-21T10:24:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:1 2024-09-21T10:24:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:2 2024-09-21T10:24:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2024-09-21T10:24:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2024-09-21T10:24:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:bench [] [] [] [] 2024-09-21T10:24:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:24:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2024-09-21T10:24:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2024-09-21T10:24:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2024-09-21T10:24:50.434 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-09-21T10:24:50.463 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:50.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.497 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:50.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:50.597 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:24:50.597 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:50.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.640 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:50.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:50.646 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:24:50.646 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:24:50.740 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:50.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.766 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:50.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:50.867 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:50.890 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:50.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:50.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:51.035 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:51.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:51.136 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:51.168 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:51.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:51.236 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:51 smithi107 ceph-mon[102221]: pgmap v1506: 109 pgs: 109 active+clean; 6.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 814 op/s 2024-09-21T10:24:51.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:51.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.317 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:51.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:51.351 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:24:51.352 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:24:51.352 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-21T10:24:51.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:51.487 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:51.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:51.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:51.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:51 smithi107 ceph-mon[105022]: pgmap v1506: 109 pgs: 109 active+clean; 6.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 814 op/s 2024-09-21T10:24:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.635 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:51.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:51.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:51 smithi129 ceph-mon[88471]: pgmap v1506: 109 pgs: 109 active+clean; 6.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 814 op/s 2024-09-21T10:24:51.669 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 10:24:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a[77168]: logger=cleanup t=2024-09-21T10:24:51.436154459Z level=info msg="Completed cleanup jobs" duration=13.67ms 2024-09-21T10:24:51.737 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:51.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.764 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:51.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:51.865 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:51.890 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:51.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:51.992 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:52.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:24:52.020 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:52.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:52.051 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:24:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:24:51] "GET /metrics HTTP/1.1" 200 36776 "" "Prometheus/2.51.0" 2024-09-21T10:24:52.122 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:52.150 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:52.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:52.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:52.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.368 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:52.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:52.469 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:52.515 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:52.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:52.616 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:52.641 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:52.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:52.743 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:52.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.813 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:52.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:52.914 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:52.965 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:52.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:53.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:53.099 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:53.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:53.200 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:53.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.252 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:53.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:53.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:53.370 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:53.376 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:53.419 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:53.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:53.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:53.527 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:53 smithi107 ceph-mon[102221]: pgmap v1507: 109 pgs: 109 active+clean; 6.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 814 op/s 2024-09-21T10:24:53.527 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:53 smithi107 ceph-mon[105022]: pgmap v1507: 109 pgs: 109 active+clean; 6.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 814 op/s 2024-09-21T10:24:53.563 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:53.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:53.613 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:53 smithi129 ceph-mon[88471]: pgmap v1507: 109 pgs: 109 active+clean; 6.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 814 op/s 2024-09-21T10:24:53.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:53.711 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:53.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:53.729 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:53.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:53.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.873 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:53.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:53.974 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:54.001 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:54.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:54.103 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:54.131 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:54.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:54.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:54.261 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:24:54.280 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:54.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:54.421 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:54.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:54.450 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:54 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1542357651' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:24:54.450 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:54 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:24:54.450 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:54 smithi107 ceph-mon[102221]: pgmap v1508: 109 pgs: 109 active+clean; 5.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 793 op/s 2024-09-21T10:24:54.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:54 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1542357651' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:24:54.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:54 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:24:54.450 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:54 smithi107 ceph-mon[105022]: pgmap v1508: 109 pgs: 109 active+clean; 5.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 793 op/s 2024-09-21T10:24:54.523 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:54.572 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:54.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:54.611 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:54 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1542357651' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:24:54.611 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:54 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:24:54.611 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:54 smithi129 ceph-mon[88471]: pgmap v1508: 109 pgs: 109 active+clean; 5.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 793 op/s 2024-09-21T10:24:54.674 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:54.697 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:54.798 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:54.822 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:54.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:54.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:54.967 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:54.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:55.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:55.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.107 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:55.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:55.208 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:55.240 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:55.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:55.341 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:55.367 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:55.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:55.468 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:55.489 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:55.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:55.548 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:55 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:55.548 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:55 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:24:55.548 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:55 smithi129 ceph-mon[88471]: osdmap e428: 8 total, 8 up, 8 in 2024-09-21T10:24:55.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:55.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:55 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:55.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:55 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:24:55.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:55 smithi107 ceph-mon[102221]: osdmap e428: 8 total, 8 up, 8 in 2024-09-21T10:24:55.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:55 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:55.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:55 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:24:55.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:55 smithi107 ceph-mon[105022]: osdmap e428: 8 total, 8 up, 8 in 2024-09-21T10:24:55.632 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:55.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:55.733 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:55.771 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:55.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:55.872 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:55.895 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:55.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:55.997 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:56.030 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:56.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:56.131 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:56.162 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:56.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:56.264 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:56.295 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:56.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:56.396 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:56.421 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:56.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:56.523 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:56.545 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:56.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:56.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:56.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.692 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:56.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:56.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:56.837 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:56.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:56.939 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:56.964 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:56.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:57.066 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:57.102 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:57.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:56 smithi129 ceph-mon[88471]: pgmap v1510: 109 pgs: 109 active+clean; 5.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 796 op/s 2024-09-21T10:24:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:24:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:24:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:57.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:57.238 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:57.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:57.340 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:57.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:56 smithi107 ceph-mon[102221]: pgmap v1510: 109 pgs: 109 active+clean; 5.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 796 op/s 2024-09-21T10:24:57.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:24:57.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:24:57.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:57.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:56 smithi107 ceph-mon[105022]: pgmap v1510: 109 pgs: 109 active+clean; 5.7 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 0 B/s wr, 796 op/s 2024-09-21T10:24:57.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:24:57.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:24:57.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:57.371 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:57.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:57.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:57.505 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:57.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:57.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:57.633 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:57.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:57.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:57.778 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:57.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:57.879 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:57.905 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:57.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:58.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:58.050 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:58.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:58.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.190 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:58.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:58.290 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:58.329 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:58.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:58.431 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:58.456 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:58.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:58.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:58.599 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:58.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:58.701 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:58.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.728 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:58.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:58.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:58.855 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:58.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:58.957 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:58.980 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:58.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:59.081 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:59.106 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:59.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:59.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:24:58 smithi129 ceph-mon[88471]: pgmap v1511: 109 pgs: 109 active+clean; 5.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 793 op/s 2024-09-21T10:24:59.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:59.250 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:59.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:59.351 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:59.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:24:58 smithi107 ceph-mon[102221]: pgmap v1511: 109 pgs: 109 active+clean; 5.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 793 op/s 2024-09-21T10:24:59.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:24:58 smithi107 ceph-mon[105022]: pgmap v1511: 109 pgs: 109 active+clean; 5.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 793 op/s 2024-09-21T10:24:59.381 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:59.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:59.483 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:59.507 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:59.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:59.609 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:59.630 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:59.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:59.731 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:59.772 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:24:59.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:59.874 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:59.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.903 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:24:59.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:00.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:00.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.043 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:00.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:00.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:00.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.177 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:00.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:00.278 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:00.314 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:00.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:00.416 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:00.456 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:00.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:00.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:00.619 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:00.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:25:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:25:00.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:25:00.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:00.759 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:00.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:00.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:00.903 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:01.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:01.042 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:01.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:01.082 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:00 smithi107 ceph-mon[102221]: pgmap v1512: 109 pgs: 109 active+clean; 5.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 800 op/s 2024-09-21T10:25:01.082 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:00 smithi107 ceph-mon[105022]: pgmap v1512: 109 pgs: 109 active+clean; 5.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 800 op/s 2024-09-21T10:25:01.144 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:01.169 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:00 smithi129 ceph-mon[88471]: pgmap v1512: 109 pgs: 109 active+clean; 5.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 800 op/s 2024-09-21T10:25:01.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:01.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:01.318 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:01.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:01.420 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:01.447 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:01.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:01.549 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:01.574 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:01.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:01.676 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:01.701 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:01.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:01.803 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:01.830 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:01.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:01.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:25:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:25:01] "GET /metrics HTTP/1.1" 200 36775 "" "Prometheus/2.51.0" 2024-09-21T10:25:01.932 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:01.958 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:01.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:02.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:02.094 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:02.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:02.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:02.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.248 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:02.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:02.349 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:02.377 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:02.479 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:02.503 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:02.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:02.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:02.641 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:02.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:02.743 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:02.778 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:02.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:02.880 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:02.913 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:02.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:03.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:03.061 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:03.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:03.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:02 smithi107 ceph-mon[102221]: pgmap v1513: 109 pgs: 109 active+clean; 5.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 800 op/s 2024-09-21T10:25:03.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:02 smithi107 ceph-mon[105022]: pgmap v1513: 109 pgs: 109 active+clean; 5.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 800 op/s 2024-09-21T10:25:03.163 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:03.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:02 smithi129 ceph-mon[88471]: pgmap v1513: 109 pgs: 109 active+clean; 5.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 800 op/s 2024-09-21T10:25:03.190 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:03.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:03.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:03.341 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:03.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:03.442 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:03.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.480 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:03.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:03.567 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-09-21T10:25:03.568 INFO:teuthology.orchestra.run.smithi107.stderr:0 2024-09-21T10:25:03.568 INFO:teuthology.orchestra.run.smithi107.stderr:1 2024-09-21T10:25:03.568 INFO:teuthology.orchestra.run.smithi107.stderr:2 2024-09-21T10:25:03.568 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2024-09-21T10:25:03.568 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2024-09-21T10:25:03.568 INFO:teuthology.orchestra.run.smithi107.stderr:bench [] [] [] [] 2024-09-21T10:25:03.568 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:25:03.568 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2024-09-21T10:25:03.568 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2024-09-21T10:25:03.568 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2024-09-21T10:25:03.568 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-09-21T10:25:03.581 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:03.607 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:25:03.607 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:25:03.608 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:03.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:03.710 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:03.756 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:03.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:03.897 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:03.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:03.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:04.041 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:25:04.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.042 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:04.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:04.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:04.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.183 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:04.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:04.284 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:04.309 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:04.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:04.411 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:04.431 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:04.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:04.533 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:04.556 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:04.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:04.658 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:04.680 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:04.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:04.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:04.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:25:04.832 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:04.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:04.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:04.977 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:04.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:05.029 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:25:05.029 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:25:05.029 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:25:05.029 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:25:05.029 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:25:05.030 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:25:05.030 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:25:05.030 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:25:05.030 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:25:05.030 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:25:05.030 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:25:05.031 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:25:05.031 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:25:05.031 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:25:05.031 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:25:05.031 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:25:05.079 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:05.106 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:05.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:05.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:04 smithi129 ceph-mon[88471]: pgmap v1514: 109 pgs: 109 active+clean; 5.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-21T10:25:05.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:04 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1735270695' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:05.207 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:05.231 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:05.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:05.333 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:05.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:04 smithi107 ceph-mon[102221]: pgmap v1514: 109 pgs: 109 active+clean; 5.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-21T10:25:05.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:04 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1735270695' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:05.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:04 smithi107 ceph-mon[105022]: pgmap v1514: 109 pgs: 109 active+clean; 5.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-21T10:25:05.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:04 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1735270695' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:05.365 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:05.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:05.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:05.499 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:05.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:05.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:05.651 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:05.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:05.752 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:05.781 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:05.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:05.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:05.917 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:05.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:06.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:06.055 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:06.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:06.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:06.199 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:06.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:06.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:06.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.346 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:06.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:06.447 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:06.473 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:06.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:06.575 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:06.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.602 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:06.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:06.703 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:06.728 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:06.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:06.830 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:06.863 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:06.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:06.964 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:07.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.019 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:07.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:07.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:06 smithi107 ceph-mon[102221]: pgmap v1515: 109 pgs: 109 active+clean; 5.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 730 op/s 2024-09-21T10:25:07.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:06 smithi107 ceph-mon[105022]: pgmap v1515: 109 pgs: 109 active+clean; 5.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 730 op/s 2024-09-21T10:25:07.120 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:07.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.162 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:07.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:07.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:06 smithi129 ceph-mon[88471]: pgmap v1515: 109 pgs: 109 active+clean; 5.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 730 op/s 2024-09-21T10:25:07.263 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:07.311 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:07.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:07.396 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:25:07.396 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:25:07.397 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-21T10:25:07.412 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:07.420 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:07.438 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:07.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:07.539 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:07.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.604 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:07.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:07.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:07.747 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:07.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:07.848 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:07.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.873 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:07.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:07.974 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:08.000 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:08.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:08.044 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:08.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:08.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.150 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:08.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:08.251 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:08.280 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:08.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:08.382 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:08.414 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:08.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:08.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:08.569 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:08.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:08.671 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:08.703 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:08.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:08.795 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:25:08.805 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:08.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.834 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:08.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:08.935 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:08.958 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:08.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:09.059 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:09.085 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:09.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:09.104 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:08 smithi107 ceph-mon[102221]: pgmap v1516: 109 pgs: 109 active+clean; 5.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 772 op/s 2024-09-21T10:25:09.108 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:08 smithi107 ceph-mon[105022]: pgmap v1516: 109 pgs: 109 active+clean; 5.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 772 op/s 2024-09-21T10:25:09.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:08 smithi129 ceph-mon[88471]: pgmap v1516: 109 pgs: 109 active+clean; 5.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 772 op/s 2024-09-21T10:25:09.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:09.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.223 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:09.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:09.324 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:09.347 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:09.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:09.448 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:09.477 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:09.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:09.579 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:09.603 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:09.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:09.705 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:09.730 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:09.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:09.832 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:09.865 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:09.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:09.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:10.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.005 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:10.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:10.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:10.140 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:10.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:10.242 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:10.268 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:10.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:10.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:10.413 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:10.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:10.515 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:10.542 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:10.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:10.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:10.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:25:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:25:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:25:10.694 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:10.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:10.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:10.834 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:10.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:10.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:11.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.001 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:11.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:11.010 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:10 smithi129 ceph-mon[88471]: pgmap v1517: 109 pgs: 109 active+clean; 5.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 946 op/s 2024-09-21T10:25:11.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:11.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.153 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:11.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:11.223 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:10 smithi107 ceph-mon[102221]: pgmap v1517: 109 pgs: 109 active+clean; 5.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 946 op/s 2024-09-21T10:25:11.223 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:10 smithi107 ceph-mon[105022]: pgmap v1517: 109 pgs: 109 active+clean; 5.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 946 op/s 2024-09-21T10:25:11.254 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:11.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.285 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:11.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:11.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:11.437 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:11.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:11.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:11.582 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:11.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:11.684 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:11.717 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:11.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:11.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:11.862 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:25:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:25:11] "GET /metrics HTTP/1.1" 200 36774 "" "Prometheus/2.51.0" 2024-09-21T10:25:11.863 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:11.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:11.965 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:11.988 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:11.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:12.090 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:12.115 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:12.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:12.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:12.217 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:12.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.253 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:12.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:12.354 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:12.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:12.378 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:12.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:12.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:12.516 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:12.520 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:12.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:12.524 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:12.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:12.670 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:12.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:12.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:12.811 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:12.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:12.913 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:12.930 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:25:12.955 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:12.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:13.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:13.109 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:13.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:13.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:12 smithi129 ceph-mon[88471]: pgmap v1518: 109 pgs: 109 active+clean; 5.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 692 op/s 2024-09-21T10:25:13.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:12 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4130297040' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:13.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:13.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.249 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:13.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:13.350 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:13.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:12 smithi107 ceph-mon[102221]: pgmap v1518: 109 pgs: 109 active+clean; 5.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 692 op/s 2024-09-21T10:25:13.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:12 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4130297040' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:13.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:12 smithi107 ceph-mon[105022]: pgmap v1518: 109 pgs: 109 active+clean; 5.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 692 op/s 2024-09-21T10:25:13.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:12 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4130297040' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:13.389 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:13.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:13.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:13.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.531 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:13.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:13.632 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:13.657 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:13.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:13.759 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:13.785 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:13.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:13.887 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:13.908 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:13.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:14.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:14.048 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:14.150 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:14.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.186 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:14.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:14.287 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:14.310 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:14.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:14.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:13 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:25:14.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:13 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:25:14.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:13 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4130297040' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:14.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:13 smithi107 ceph-mon[102221]: osdmap e429: 8 total, 8 up, 8 in 2024-09-21T10:25:14.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:13 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:25:14.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:13 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:25:14.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:13 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4130297040' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:14.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:13 smithi107 ceph-mon[105022]: osdmap e429: 8 total, 8 up, 8 in 2024-09-21T10:25:14.413 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:14.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:13 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:25:14.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:13 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:25:14.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:13 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4130297040' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:14.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:13 smithi129 ceph-mon[88471]: osdmap e429: 8 total, 8 up, 8 in 2024-09-21T10:25:14.455 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:14.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:14.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:14.588 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:14.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:14.690 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:14.711 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:14.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:14.813 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:14.834 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:14.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:14.935 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:14.964 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:14.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:15.066 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:15.092 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:15.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:15.194 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:15.205 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:14 smithi107 ceph-mon[102221]: pgmap v1520: 109 pgs: 109 active+clean; 4.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 816 op/s 2024-09-21T10:25:15.205 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:14 smithi107 ceph-mon[105022]: pgmap v1520: 109 pgs: 109 active+clean; 4.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 816 op/s 2024-09-21T10:25:15.222 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:15.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:15.324 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:15.348 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:15.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:15.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:14 smithi129 ceph-mon[88471]: pgmap v1520: 109 pgs: 109 active+clean; 4.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 816 op/s 2024-09-21T10:25:15.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:15.498 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:15.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:15.600 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:15.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.628 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:15.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:15.730 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:15.758 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:15.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:15.860 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:15.887 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:15.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:15.988 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:16.009 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:16.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:16.111 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:16.145 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:16.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:16.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:16.276 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:16.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:16.378 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:16.423 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:16.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:16.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:16.562 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:16.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:16.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:16.702 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:16.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:16.804 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:16.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.837 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:16.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:16.938 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:16.964 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:16.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:17.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:17.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.103 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:17.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:17.204 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:17.228 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:17.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:17.330 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:17.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:16 smithi107 ceph-mon[102221]: pgmap v1521: 109 pgs: 109 active+clean; 4.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 812 op/s 2024-09-21T10:25:17.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:16 smithi107 ceph-mon[105022]: pgmap v1521: 109 pgs: 109 active+clean; 4.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 812 op/s 2024-09-21T10:25:17.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.374 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:17.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:17.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:16 smithi129 ceph-mon[88471]: pgmap v1521: 109 pgs: 109 active+clean; 4.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 812 op/s 2024-09-21T10:25:17.474 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:17.512 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:17.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:17.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:17.665 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:17.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:17.767 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:17.812 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:17.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:17.914 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:17.942 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:17.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:18.044 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:18.070 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:18.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:18.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:18.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.212 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:18.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:18.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:18.348 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:18.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:18.449 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:18.475 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:18.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:18.577 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:18.599 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:18.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:18.700 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:18.723 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:18.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:18.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:18.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.867 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:18.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:18.968 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:18.993 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:18.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:19.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:19.116 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:18 smithi107 ceph-mon[102221]: pgmap v1522: 109 pgs: 109 active+clean; 4.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 809 op/s 2024-09-21T10:25:19.116 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:18 smithi107 ceph-mon[105022]: pgmap v1522: 109 pgs: 109 active+clean; 4.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 809 op/s 2024-09-21T10:25:19.132 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:25:19.132 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:25:19.132 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-09-21T10:25:19.132 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-09-21T10:25:19.133 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:25:19.153 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:19.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:19.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:19.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.291 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:19.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:19.392 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:19.417 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:19.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:19.418 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:18 smithi129 ceph-mon[88471]: pgmap v1522: 109 pgs: 109 active+clean; 4.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 809 op/s 2024-09-21T10:25:19.519 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:19.565 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:19.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:19.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:19.733 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:19.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:19.835 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:19.859 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:19.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:19.960 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:19.986 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:19.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:20.088 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:20.113 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:20.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:20.215 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:20.221 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-09-21T10:25:20.221 INFO:teuthology.orchestra.run.smithi107.stderr:0 2024-09-21T10:25:20.221 INFO:teuthology.orchestra.run.smithi107.stderr:1 2024-09-21T10:25:20.221 INFO:teuthology.orchestra.run.smithi107.stderr:2 2024-09-21T10:25:20.221 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2024-09-21T10:25:20.221 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2024-09-21T10:25:20.221 INFO:teuthology.orchestra.run.smithi107.stderr:bench [] [] [] [] 2024-09-21T10:25:20.221 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:25:20.221 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2024-09-21T10:25:20.221 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2024-09-21T10:25:20.221 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2024-09-21T10:25:20.222 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-09-21T10:25:20.243 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:20.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:20.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:25:20.344 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:20.368 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:20.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:20.470 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:20.473 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:25:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:25:20.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:25:20.496 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:20.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:20.598 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:20.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:20 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:25:20.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:20 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:25:20.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:20 smithi107 ceph-mon[102221]: pgmap v1523: 109 pgs: 109 active+clean; 4.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 803 op/s 2024-09-21T10:25:20.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:20 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:25:20.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:20 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:25:20.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:20 smithi107 ceph-mon[105022]: pgmap v1523: 109 pgs: 109 active+clean; 4.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 803 op/s 2024-09-21T10:25:20.648 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:20.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:20.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:25:20.750 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:20.775 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:20.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:20.877 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:20 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:25:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:20 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:25:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:20 smithi129 ceph-mon[88471]: pgmap v1523: 109 pgs: 109 active+clean; 4.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 803 op/s 2024-09-21T10:25:20.956 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:20.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:21.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:21.111 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:21.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:21.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:21.255 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:21.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:21.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:21.398 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:21.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:21.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:21.533 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:21.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:21.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:25:21.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:25:21.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:25:21.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:25:21.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:25:21.560 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:25:21.600 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:25:21.601 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:25:21.601 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-21T10:25:21.635 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:21.658 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:21.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:21.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:21.819 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:21.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:21.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:25:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:25:21] "GET /metrics HTTP/1.1" 200 36774 "" "Prometheus/2.51.0" 2024-09-21T10:25:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:25:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:25:21.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:25:21.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:21.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.990 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:21.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:22.024 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:22.084 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:22.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:22.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.156 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:22.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:22.257 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:22.293 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:22.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:22.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:22.440 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:22.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:22.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:22.579 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:22.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:22.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:22 smithi129 ceph-mon[88471]: pgmap v1524: 109 pgs: 109 active+clean; 4.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 803 op/s 2024-09-21T10:25:22.680 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:22.706 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:22.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:22.762 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:22 smithi107 ceph-mon[102221]: pgmap v1524: 109 pgs: 109 active+clean; 4.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 803 op/s 2024-09-21T10:25:22.762 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:22 smithi107 ceph-mon[105022]: pgmap v1524: 109 pgs: 109 active+clean; 4.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 803 op/s 2024-09-21T10:25:22.807 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:22.834 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:22.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:22.936 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:22.959 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:22.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:23.061 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:23.084 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:23.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:23.186 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:23.223 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:23.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:23.325 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:23.351 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:23.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:23.453 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:23.479 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:23.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:23.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:23.603 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:23.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:23.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:23.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:23.845 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:23.868 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:23.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:23.970 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:23.996 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:23.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:24.098 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:24.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.138 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:24.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:24.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:24.275 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:24.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:24.377 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:24.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.417 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:24.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:24.519 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:24.542 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:24.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:24.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:24.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.699 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:24.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:24.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:24.835 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:24.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:24.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:24 smithi129 ceph-mon[88471]: pgmap v1525: 109 pgs: 109 active+clean; 4.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 927 op/s 2024-09-21T10:25:24.936 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:24.959 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:24.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:25.060 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:25.092 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:25.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:25.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:24 smithi107 ceph-mon[102221]: pgmap v1525: 109 pgs: 109 active+clean; 4.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 927 op/s 2024-09-21T10:25:25.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:24 smithi107 ceph-mon[105022]: pgmap v1525: 109 pgs: 109 active+clean; 4.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 927 op/s 2024-09-21T10:25:25.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:25.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.232 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:25.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:25.333 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:25.359 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:25.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:25.461 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:25.495 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:25.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:25.597 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:25.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.627 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:25.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:25.728 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:25.750 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:25.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:25.852 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:25.877 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:25.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:25.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:26.017 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:26.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:26.119 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:26.160 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:26.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:26.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:26.299 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:26.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:26.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:26.450 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:26.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:26.548 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:25:26.548 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:25:26.553 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:26.578 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:26.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:26.581 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:25:26.582 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:25:26.617 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:25:26.617 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:25:26.660 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:25:26.661 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:25:26.679 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:26.690 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:25:26.690 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:25:26.705 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:26.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:26.718 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:25:26.718 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:25:26.747 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:25:26.748 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:25:26.780 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:25:26.780 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:25:26.807 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:26.829 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:26.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:26.931 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:26.957 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:26.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:27.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:27.106 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:27.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:27.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:27.255 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:27.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:27.357 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:27 smithi107 ceph-mon[102221]: pgmap v1526: 109 pgs: 109 active+clean; 4.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-21T10:25:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:25:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:27 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2178493640' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:27 smithi107 ceph-mon[105022]: pgmap v1526: 109 pgs: 109 active+clean; 4.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-21T10:25:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:25:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:27 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2178493640' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:27.381 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:27.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:27.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:27.518 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:27.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:27.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:27.668 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:27.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:27.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:27 smithi129 ceph-mon[88471]: pgmap v1526: 109 pgs: 109 active+clean; 4.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 738 op/s 2024-09-21T10:25:27.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:25:27.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:27.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:27 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2178493640' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:27.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:27.812 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:27.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.813 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:27.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:27.914 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:27.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.943 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:27.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:28.044 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:28.081 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:28.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:28.183 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:28.201 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:25:28.209 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:28.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:28.311 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:28.333 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:28.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:28.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:28.468 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:28.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:28.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:28.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.609 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:28.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:28.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:28 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1495778631' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:28.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:28 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:28.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:28 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1495778631' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:28.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:28 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:28.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:28 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1495778631' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:28.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:28 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:28.710 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:28.735 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:28.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:28.836 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:28.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.881 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:28.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:28.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:29.022 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:29.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:29.124 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:29.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.150 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:29.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:29.252 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:29.275 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:29.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:29.377 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:29.416 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:29.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:29.518 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:29.544 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:29.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:29.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:29 smithi107 ceph-mon[102221]: pgmap v1527: 109 pgs: 109 active+clean; 4.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 759 op/s 2024-09-21T10:25:29.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:29 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:29.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:29 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:25:29.578 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:29 smithi107 ceph-mon[102221]: osdmap e430: 8 total, 8 up, 8 in 2024-09-21T10:25:29.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:29 smithi107 ceph-mon[105022]: pgmap v1527: 109 pgs: 109 active+clean; 4.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 759 op/s 2024-09-21T10:25:29.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:29 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:29.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:29 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:25:29.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:29 smithi107 ceph-mon[105022]: osdmap e430: 8 total, 8 up, 8 in 2024-09-21T10:25:29.609 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:25:29.609 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:25:29.646 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:29 smithi129 ceph-mon[88471]: pgmap v1527: 109 pgs: 109 active+clean; 4.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 759 op/s 2024-09-21T10:25:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:29 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:29 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:25:29.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:29 smithi129 ceph-mon[88471]: osdmap e430: 8 total, 8 up, 8 in 2024-09-21T10:25:29.671 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:29.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:29.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:29.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.822 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:29.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:29.923 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:29.949 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:29.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:30.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:30.094 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:30.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:30.195 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:30.228 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:30.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:30.329 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:30.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:25:30.353 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:30.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:30.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:25:30.455 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:30.545 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:30.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:30.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:25:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:25:30.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:25:30.706 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:30.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:30.809 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:30.834 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:30.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:30.936 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:30.980 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:30.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:30.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:25:31.082 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:31.113 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:31.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:31.215 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:31.242 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:31.242 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:31.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:31.344 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:31.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:31 smithi107 ceph-mon[105022]: pgmap v1529: 109 pgs: 109 active+clean; 4.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 825 op/s 2024-09-21T10:25:31.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:31 smithi107 ceph-mon[102221]: pgmap v1529: 109 pgs: 109 active+clean; 4.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 825 op/s 2024-09-21T10:25:31.370 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:31.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:31.472 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:31.498 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:31.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:31.600 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:31.624 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:31.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:31 smithi129 ceph-mon[88471]: pgmap v1529: 109 pgs: 109 active+clean; 4.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 825 op/s 2024-09-21T10:25:31.726 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:31.752 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:31.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:31.853 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:31.906 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:31.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:31.988 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:25:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:25:31] "GET /metrics HTTP/1.1" 200 36776 "" "Prometheus/2.51.0" 2024-09-21T10:25:32.008 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:32.038 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:32.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:32.139 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:32.166 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:32.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:32.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:32.305 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:32.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:32.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:32.456 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:32.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:32.558 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:32.586 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:32.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:32.687 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:32.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.710 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:32.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:32.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:32.845 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:32.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:32.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:32.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.985 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:32.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:33.085 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:33.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.115 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:33.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:33.120 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-09-21T10:25:33.120 INFO:teuthology.orchestra.run.smithi107.stderr:0 2024-09-21T10:25:33.120 INFO:teuthology.orchestra.run.smithi107.stderr:1 2024-09-21T10:25:33.120 INFO:teuthology.orchestra.run.smithi107.stderr:2 2024-09-21T10:25:33.120 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2024-09-21T10:25:33.120 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2024-09-21T10:25:33.120 INFO:teuthology.orchestra.run.smithi107.stderr:bench [] [] [] [] 2024-09-21T10:25:33.121 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:25:33.121 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2024-09-21T10:25:33.121 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2024-09-21T10:25:33.121 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2024-09-21T10:25:33.121 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-09-21T10:25:33.216 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:33.245 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:33.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:33.346 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:33.393 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:33.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:33.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:33 smithi107 ceph-mon[102221]: pgmap v1530: 109 pgs: 109 active+clean; 4.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 825 op/s 2024-09-21T10:25:33.432 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:33 smithi107 ceph-mon[105022]: pgmap v1530: 109 pgs: 109 active+clean; 4.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 825 op/s 2024-09-21T10:25:33.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:33.534 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:33.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:33.635 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:33.658 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:33.658 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:33.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:33.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:33 smithi129 ceph-mon[88471]: pgmap v1530: 109 pgs: 109 active+clean; 4.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 825 op/s 2024-09-21T10:25:33.761 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:33.786 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:33.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:33.889 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:33.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.925 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:33.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:34.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:34.074 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:34.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:34.141 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:25:34.141 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:25:34.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-21T10:25:34.176 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:34.198 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:34.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:34.300 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:34.324 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:34.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:34.426 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:34.454 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:34.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:34.556 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:34.578 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:34.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:34.680 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:34.701 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:34.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:34.803 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:34.825 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:34.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:34.869 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:34.926 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:34.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.952 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:34.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:35.053 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:35.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.079 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:35.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:35.180 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:35.226 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:35.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:35.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:35.350 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:35 smithi107 ceph-mon[102221]: pgmap v1531: 109 pgs: 109 active+clean; 3.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 830 op/s 2024-09-21T10:25:35.351 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:35 smithi107 ceph-mon[105022]: pgmap v1531: 109 pgs: 109 active+clean; 3.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 830 op/s 2024-09-21T10:25:35.372 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:35.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:35.474 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:35.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.511 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:35.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:35.612 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:35.657 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:35.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:35.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:35 smithi129 ceph-mon[88471]: pgmap v1531: 109 pgs: 109 active+clean; 3.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 830 op/s 2024-09-21T10:25:35.759 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:35.785 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:35.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:35.787 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:35.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:35.946 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:35.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:36.047 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:36.081 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:36.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:36.183 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:36.207 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:36.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:36.309 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:36.336 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:36.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:36.437 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:36.460 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:36.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:36.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:36.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.601 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:36.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:36.702 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:36.725 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:36.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:36.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:36.865 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:36.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:36.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:37.004 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:37.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:37.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:37.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:37.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:37.151 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:37.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:37.252 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:37.278 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:37.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:37.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:37.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:37.456 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:37.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:37.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:37.558 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:37.586 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:37.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:37.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:37 smithi107 ceph-mon[102221]: pgmap v1532: 109 pgs: 109 active+clean; 3.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 832 op/s 2024-09-21T10:25:37.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:37 smithi107 ceph-mon[105022]: pgmap v1532: 109 pgs: 109 active+clean; 3.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 832 op/s 2024-09-21T10:25:37.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:37 smithi129 ceph-mon[88471]: pgmap v1532: 109 pgs: 109 active+clean; 3.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 832 op/s 2024-09-21T10:25:37.688 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:37.716 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:37.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:37.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:37.817 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:37.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:37.864 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:37.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:37.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:38.000 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:38.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:38.102 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:38.128 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:38.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:38.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:38.270 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:38.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:38.372 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:38.398 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:38.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:38.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:38.540 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:38.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:38.641 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:38.678 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:38.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:38.780 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:38.802 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:38.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:38.904 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:38.927 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:38.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:39.029 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:39.069 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:39.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:39.171 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:39.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.197 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:39.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:39.299 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:39.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.324 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:39.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:39.425 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:39.447 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:39.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:39.549 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:39.570 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:39.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:39.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:39 smithi107 ceph-mon[102221]: pgmap v1533: 109 pgs: 109 active+clean; 3.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 832 op/s 2024-09-21T10:25:39.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:39 smithi107 ceph-mon[105022]: pgmap v1533: 109 pgs: 109 active+clean; 3.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 832 op/s 2024-09-21T10:25:39.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:39 smithi129 ceph-mon[88471]: pgmap v1533: 109 pgs: 109 active+clean; 3.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 832 op/s 2024-09-21T10:25:39.672 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:39.699 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:39.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:39.803 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:39.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.840 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:39.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:39.941 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:39.965 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:39.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:40.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:40.108 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:40.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:40.210 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:40.235 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:40.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:40.337 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:40.362 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:40.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:40.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:40.512 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:40.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:40.570 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:25:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:25:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:25:40.614 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:40.641 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:40.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:40.743 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:40.770 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:40.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:40.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:40.899 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:40.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:41.000 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:41.046 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:41.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:41.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:41.186 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:41.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:41.288 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:41.309 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:41.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:41.411 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:41.450 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:41.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:41.552 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:41.575 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:41.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:41 smithi107 ceph-mon[102221]: pgmap v1534: 109 pgs: 109 active+clean; 3.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 967 op/s 2024-09-21T10:25:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:25:41.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:41.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:41 smithi107 ceph-mon[105022]: pgmap v1534: 109 pgs: 109 active+clean; 3.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 967 op/s 2024-09-21T10:25:41.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:25:41.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:41 smithi129 ceph-mon[88471]: pgmap v1534: 109 pgs: 109 active+clean; 3.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 967 op/s 2024-09-21T10:25:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:25:41.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:41.676 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:41.700 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:41.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:41.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:41.824 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:25:41.824 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:25:41.833 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:41.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:41.860 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:25:41.860 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:25:41.899 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:25:41.900 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:25:41.900 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:25:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:25:41] "GET /metrics HTTP/1.1" 200 36778 "" "Prometheus/2.51.0" 2024-09-21T10:25:41.934 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:41.937 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:25:41.937 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:25:41.962 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:41.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:41.968 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:25:41.968 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:25:42.000 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:25:42.000 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:25:42.043 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:25:42.043 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:25:42.064 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:42.082 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:25:42.082 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:25:42.090 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:42.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:42.149 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:25:42.149 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:25:42.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.149 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.47186.0:177443 14.b e430/411 14:de5e2143:::benchmark_data_smithi107_312686_object61652:head v 430'11033, mlcod=430'11031)", 2024-09-21T10:25:42.149 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:25:42.140372+0000", 2024-09-21T10:25:42.149 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0060613840000000004, 2024-09-21T10:25:42.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0060897549999999996, 2024-09-21T10:25:42.149 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:25:42.149 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:25:42.149 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:25:42.149 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:25:42.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:25:42.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.140372+0000", 2024-09-21T10:25:42.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:25:42.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:25:42.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.140372+0000", 2024-09-21T10:25:42.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:25:42.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:25:42.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.140373+0000", 2024-09-21T10:25:42.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0020000000000001e-06 2024-09-21T10:25:42.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:25:42.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.140381+0000", 2024-09-21T10:25:42.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4370000000000007e-06 2024-09-21T10:25:42.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:25:42.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.140397+0000", 2024-09-21T10:25:42.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6047000000000001e-05 2024-09-21T10:25:42.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:25:42.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.140405+0000", 2024-09-21T10:25:42.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8129999999999996e-06 2024-09-21T10:25:42.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:25:42.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.142894+0000", 2024-09-21T10:25:42.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0024885860000000001 2024-09-21T10:25:42.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:25:42.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.142913+0000", 2024-09-21T10:25:42.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9607e-05 2024-09-21T10:25:42.152 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:25:42.152 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:25:42.152 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:25:42.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.153 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.47186.0:177444 14.3 e430/411)", 2024-09-21T10:25:42.153 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:25:42.140491+0000", 2024-09-21T10:25:42.153 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0059419049999999999, 2024-09-21T10:25:42.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0060186900000000002, 2024-09-21T10:25:42.153 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:25:42.153 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:25:42.153 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2024-09-21T10:25:42.153 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:25:42.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.153 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:25:42.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.140491+0000", 2024-09-21T10:25:42.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:25:42.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:25:42.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.140491+0000", 2024-09-21T10:25:42.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:25:42.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:25:42.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.140492+0000", 2024-09-21T10:25:42.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6499999999999999e-07 2024-09-21T10:25:42.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:25:42.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.140497+0000", 2024-09-21T10:25:42.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.69e-06 2024-09-21T10:25:42.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:25:42.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.140507+0000", 2024-09-21T10:25:42.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0514e-05 2024-09-21T10:25:42.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:25:42.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.140513+0000", 2024-09-21T10:25:42.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0419999999999999e-06 2024-09-21T10:25:42.155 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:25:42.155 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:25:42.155 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:25:42.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.156 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.47186.0:177431 14.1 e430/411 14:8cf155aa:::benchmark_data_smithi107_312686_object61640:head v 430'10946, mlcod=430'10942)", 2024-09-21T10:25:42.156 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:25:42.144623+0000", 2024-09-21T10:25:42.156 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0018104530000000001, 2024-09-21T10:25:42.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00192694, 2024-09-21T10:25:42.156 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:25:42.156 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:25:42.156 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:25:42.156 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:25:42.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:25:42.156 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.144623+0000", 2024-09-21T10:25:42.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:25:42.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:25:42.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.144623+0000", 2024-09-21T10:25:42.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:25:42.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.157 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:25:42.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.144623+0000", 2024-09-21T10:25:42.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0499999999999999e-07 2024-09-21T10:25:42.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.157 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:25:42.157 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.144627+0000", 2024-09-21T10:25:42.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8330000000000004e-06 2024-09-21T10:25:42.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:25:42.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.144633+0000", 2024-09-21T10:25:42.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1090000000000003e-06 2024-09-21T10:25:42.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:25:42.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.144637+0000", 2024-09-21T10:25:42.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4120000000000003e-06 2024-09-21T10:25:42.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:25:42.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.144757+0000", 2024-09-21T10:25:42.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000119789 2024-09-21T10:25:42.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:25:42.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:25:42.159 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:25:42.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:42.144776+0000", 2024-09-21T10:25:42.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8586000000000001e-05 2024-09-21T10:25:42.159 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:25:42.159 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:25:42.159 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:25:42.159 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:25:42.159 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-09-21T10:25:42.159 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 3 2024-09-21T10:25:42.159 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:25:42.192 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:42.216 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:42.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:42.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:42.360 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:42.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:42.461 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:42.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.504 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:42.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:42.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:42.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:42 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3503125023' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:42.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:42 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3503125023' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:42.644 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:42 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3503125023' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:42.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:42.802 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:42.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:42.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:42.964 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:42.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:43.066 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:43.093 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:43.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:43.195 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:43.229 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:43.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:43.331 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:43.338 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:43 smithi107 ceph-mon[102221]: pgmap v1535: 109 pgs: 109 active+clean; 3.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 695 op/s 2024-09-21T10:25:43.339 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:43 smithi107 ceph-mon[105022]: pgmap v1535: 109 pgs: 109 active+clean; 3.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 695 op/s 2024-09-21T10:25:43.352 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:43.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:43.454 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:43.503 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:43.605 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:43.628 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:43.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:43 smithi129 ceph-mon[88471]: pgmap v1535: 109 pgs: 109 active+clean; 3.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 695 op/s 2024-09-21T10:25:43.731 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:43.768 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:43.869 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:43.896 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:43.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:43.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:44.043 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:44.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:44.145 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:44.169 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:44.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:44.270 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:44.298 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:44.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:44.400 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:44.428 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:44.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:44.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:44.575 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:44.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:44.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:44.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.712 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:44.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:44.813 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:44.837 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:44.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:44.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:44.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.997 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:44.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:45.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:25:45.098 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:45.121 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:45.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:45.222 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:45.243 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:45.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:45.283 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:25:45.346 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:45.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.401 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:45.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:45.499 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:45 smithi107 ceph-mon[102221]: pgmap v1536: 109 pgs: 109 active+clean; 3.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 957 op/s 2024-09-21T10:25:45.499 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:45 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/413227263' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:45.499 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:45 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:45.499 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:45 smithi107 ceph-mon[105022]: pgmap v1536: 109 pgs: 109 active+clean; 3.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 957 op/s 2024-09-21T10:25:45.499 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:45 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/413227263' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:45.499 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:45 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:45.503 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:45.529 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:45.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:45.593 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-09-21T10:25:45.593 INFO:teuthology.orchestra.run.smithi107.stderr:0 2024-09-21T10:25:45.593 INFO:teuthology.orchestra.run.smithi107.stderr:1 2024-09-21T10:25:45.593 INFO:teuthology.orchestra.run.smithi107.stderr:2 2024-09-21T10:25:45.593 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2024-09-21T10:25:45.593 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2024-09-21T10:25:45.593 INFO:teuthology.orchestra.run.smithi107.stderr:bench [] [] [] [] 2024-09-21T10:25:45.594 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:25:45.594 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2024-09-21T10:25:45.594 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2024-09-21T10:25:45.594 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2024-09-21T10:25:45.594 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-09-21T10:25:45.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:45 smithi129 ceph-mon[88471]: pgmap v1536: 109 pgs: 109 active+clean; 3.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 957 op/s 2024-09-21T10:25:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:45 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/413227263' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:45 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:45.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.671 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:45.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:45.772 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:45.797 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:45.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:45.899 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:45.924 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:45.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:46.026 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:46.049 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:46.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:46.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:46.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.199 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:46.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:46.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:46.338 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:46.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:46.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:46.480 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:46.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:46.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:46.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:25:46.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:46 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:46.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:46 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:46.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:46 smithi107 ceph-mon[102221]: osdmap e431: 8 total, 8 up, 8 in 2024-09-21T10:25:46.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:46 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:46.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:46 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:46.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:46 smithi107 ceph-mon[105022]: osdmap e431: 8 total, 8 up, 8 in 2024-09-21T10:25:46.620 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:46.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:46.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:46 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:46.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:46 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:46.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:46 smithi129 ceph-mon[88471]: osdmap e431: 8 total, 8 up, 8 in 2024-09-21T10:25:46.722 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:46.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.744 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:46.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:46.845 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:46.867 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:46.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:46.969 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:47.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.007 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:47.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:47.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:47.140 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:47.241 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:47.266 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:47.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:47.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.421 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:47.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:47.522 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:47.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:47 smithi107 ceph-mon[102221]: pgmap v1538: 109 pgs: 109 active+clean; 3.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 805 op/s 2024-09-21T10:25:47.524 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:25:47.524 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:25:47.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-21T10:25:47.525 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:47 smithi107 ceph-mon[105022]: pgmap v1538: 109 pgs: 109 active+clean; 3.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 805 op/s 2024-09-21T10:25:47.550 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:47.551 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:47.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:47.572 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:47.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:47 smithi129 ceph-mon[88471]: pgmap v1538: 109 pgs: 109 active+clean; 3.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 805 op/s 2024-09-21T10:25:47.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.705 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:47.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:47.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:47.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.867 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:47.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:47.968 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:47.992 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:47.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:48.093 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:48.117 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:48.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:48.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:48.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:25:48.240 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:48.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:48.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:48.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.377 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:48.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:48.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:48.524 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:48.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:48.569 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:48 smithi107 ceph-mon[102221]: pgmap v1539: 109 pgs: 109 active+clean; 3.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 802 op/s 2024-09-21T10:25:48.569 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:48 smithi107 ceph-mon[105022]: pgmap v1539: 109 pgs: 109 active+clean; 3.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 802 op/s 2024-09-21T10:25:48.627 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:48.650 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:48.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:48.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:48 smithi129 ceph-mon[88471]: pgmap v1539: 109 pgs: 109 active+clean; 3.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 802 op/s 2024-09-21T10:25:48.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:48.795 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:48.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:48.897 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:48.922 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:48.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:49.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:49.068 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:49.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:49.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:49.170 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:49.195 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:49.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:49.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:49.297 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:49.321 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:49.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:49.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:49.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:49.471 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:49.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:49.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:49.606 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:49.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:49.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:49.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:49.751 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:49.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:49.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:49.853 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:49.893 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:49.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:49.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:49.995 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:50.032 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:50.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:50.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:50.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.177 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:50.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:50.278 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:50.323 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:50.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:50.425 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:50.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.481 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:50.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:50.583 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:50.609 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:50.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:50.641 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:25:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:25:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:25:50.711 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:50.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.773 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:50.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:50.874 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:50.900 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:50.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:50.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:50 smithi129 ceph-mon[88471]: pgmap v1540: 109 pgs: 109 active+clean; 3.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 783 op/s 2024-09-21T10:25:51.001 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:51.016 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:50 smithi107 ceph-mon[102221]: pgmap v1540: 109 pgs: 109 active+clean; 3.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 783 op/s 2024-09-21T10:25:51.016 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:50 smithi107 ceph-mon[105022]: pgmap v1540: 109 pgs: 109 active+clean; 3.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 783 op/s 2024-09-21T10:25:51.028 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:51.050 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:51.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:51.083 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:25:51.083 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:25:51.119 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:25:51.119 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:25:51.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:51.156 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:25:51.157 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:25:51.190 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:25:51.190 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:25:51.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.198 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:51.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:51.223 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:25:51.223 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:25:51.253 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:25:51.253 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:25:51.284 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:25:51.284 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:25:51.299 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:51.316 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:25:51.316 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:25:51.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.327 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:51.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:51.427 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:51.456 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:51.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:51.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:51.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.595 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:51.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:51.696 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:51.722 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:51.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:51.824 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:51.853 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:51.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:51.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:51 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2836217863' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:51.954 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:51.981 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:51.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:52.083 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:52.090 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:51 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2836217863' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:52.090 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:51 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2836217863' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:52.091 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:25:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:25:51] "GET /metrics HTTP/1.1" 200 36778 "" "Prometheus/2.51.0" 2024-09-21T10:25:52.106 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:52.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:52.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:52.227 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:52.247 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:25:52.247 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:25:52.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.257 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:52.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:52.357 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:52.383 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:52.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:52.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:52.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.520 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:52.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:52.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:52.659 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:52.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:52.761 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:52.786 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:52.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:52.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:52 smithi107 ceph-mon[102221]: pgmap v1541: 109 pgs: 109 active+clean; 3.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 783 op/s 2024-09-21T10:25:52.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:52 smithi107 ceph-mon[105022]: pgmap v1541: 109 pgs: 109 active+clean; 3.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 783 op/s 2024-09-21T10:25:52.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:52.918 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:52.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:52 smithi129 ceph-mon[88471]: pgmap v1541: 109 pgs: 109 active+clean; 3.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 783 op/s 2024-09-21T10:25:52.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:53.020 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:53.042 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:53.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:53.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:25:53.144 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:53.168 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:53.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:53.270 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:53.291 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:53.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:53.393 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:53.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.417 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:53.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:53.518 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:53.544 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:53.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:53.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:53.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.690 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:53.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:53.791 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:53.815 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:53.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:53.917 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:53.943 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:53.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:54.045 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:54.070 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:54.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:54.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:54.203 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:54.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:54.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:54.354 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:54.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:54.455 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:54.481 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:54.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:54.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:54.615 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:54.717 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:54.740 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:54.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:54.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:54.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:54 smithi107 ceph-mon[102221]: pgmap v1542: 109 pgs: 109 active+clean; 2.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 804 op/s 2024-09-21T10:25:54.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:54 smithi107 ceph-mon[105022]: pgmap v1542: 109 pgs: 109 active+clean; 2.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 804 op/s 2024-09-21T10:25:54.875 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:54.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:54.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:54 smithi129 ceph-mon[88471]: pgmap v1542: 109 pgs: 109 active+clean; 2.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 804 op/s 2024-09-21T10:25:54.977 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:55.001 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:55.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:55.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:55.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.154 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:55.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:55.254 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:55.280 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:55.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:55.382 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:55.407 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:55.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:55.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:55.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.557 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:55.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:55.658 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:55.684 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:55.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:55.785 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:55.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:25:55.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.809 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:55.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:55.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:55.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.952 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:55.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:56.053 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:56.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.108 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:56.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:56.209 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:56.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.232 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:56.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:56.333 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:56.359 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:56.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:56.461 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:56.496 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:56.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:56.598 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:56.619 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:56.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:56.721 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:56.745 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:56.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:56.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:56.878 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:56.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:56 smithi129 ceph-mon[88471]: pgmap v1543: 109 pgs: 109 active+clean; 2.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 786 op/s 2024-09-21T10:25:56.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:56.980 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:57.003 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:57.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:57.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:57.108 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:56 smithi107 ceph-mon[102221]: pgmap v1543: 109 pgs: 109 active+clean; 2.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 786 op/s 2024-09-21T10:25:57.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:57.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:56 smithi107 ceph-mon[105022]: pgmap v1543: 109 pgs: 109 active+clean; 2.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 786 op/s 2024-09-21T10:25:57.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:57.145 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:57.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:57.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:57.290 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:57.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:57.392 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:57.413 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:57.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:57.514 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:57.540 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:57.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:57.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:57.664 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:57.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:57.765 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:57.787 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:57.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:57.889 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:57.923 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:57.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:58.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:58.070 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:58.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:58.172 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:58.202 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:58.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:58.304 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:58.329 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:58.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:58.332 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-09-21T10:25:58.332 INFO:teuthology.orchestra.run.smithi107.stderr:0 2024-09-21T10:25:58.332 INFO:teuthology.orchestra.run.smithi107.stderr:1 2024-09-21T10:25:58.332 INFO:teuthology.orchestra.run.smithi107.stderr:2 2024-09-21T10:25:58.332 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2024-09-21T10:25:58.332 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2024-09-21T10:25:58.333 INFO:teuthology.orchestra.run.smithi107.stderr:bench [] [] [] [] 2024-09-21T10:25:58.333 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:25:58.333 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2024-09-21T10:25:58.333 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2024-09-21T10:25:58.333 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2024-09-21T10:25:58.333 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-09-21T10:25:58.431 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:58.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.459 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:58.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:58.560 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:58.589 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:58.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:58.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:58.737 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:58.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:58.838 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:58.859 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:58.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:58.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:25:58 smithi129 ceph-mon[88471]: pgmap v1544: 109 pgs: 109 active+clean; 2.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 755 op/s 2024-09-21T10:25:58.961 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:59.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.008 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:59.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:59.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:25:58 smithi107 ceph-mon[102221]: pgmap v1544: 109 pgs: 109 active+clean; 2.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 755 op/s 2024-09-21T10:25:59.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:25:58 smithi107 ceph-mon[105022]: pgmap v1544: 109 pgs: 109 active+clean; 2.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 755 op/s 2024-09-21T10:25:59.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:59.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.145 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:59.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:59.246 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:59.270 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:59.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:59.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:59.407 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:59.409 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:59.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:59.511 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:59.537 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:59.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:59.639 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:59.662 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:59.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:59.765 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:59.790 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:25:59.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:59.892 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:59.930 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:25:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:59.998 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:25:59.999 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:25:59.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-21T10:26:00.032 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:00.060 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:00.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:00.162 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:00.189 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:00.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:00.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:00.347 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:00.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:00.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:00.489 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:00.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:00.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:00.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.636 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:00.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:26:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:26:00.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:26:00.737 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:00.760 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:00.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:00.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:00.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.908 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:00.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:00.967 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:00 smithi107 ceph-mon[102221]: pgmap v1545: 109 pgs: 109 active+clean; 2.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 919 op/s 2024-09-21T10:26:00.967 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:00 smithi107 ceph-mon[105022]: pgmap v1545: 109 pgs: 109 active+clean; 2.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 919 op/s 2024-09-21T10:26:01.009 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:01.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.040 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:01.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:01.141 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:01.166 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:01.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:01.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:00 smithi129 ceph-mon[88471]: pgmap v1545: 109 pgs: 109 active+clean; 2.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 919 op/s 2024-09-21T10:26:01.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:01.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.305 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:01.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:01.406 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:01.438 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:01.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:01.540 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:01.573 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:01.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:01.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:01.713 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:01.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:01.721 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:26:01.814 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:01.839 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:01.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:01.941 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:01.998 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:01.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:02.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:01 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4190173032' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:26:02.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:01 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:26:02.056 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:01 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4190173032' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:26:02.056 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:01 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:26:02.056 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:26:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:26:01] "GET /metrics HTTP/1.1" 200 36772 "" "Prometheus/2.51.0" 2024-09-21T10:26:02.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:02.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.139 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:02.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:01 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4190173032' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:26:02.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:01 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:26:02.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:02.281 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:02.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:02.382 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:02.412 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:02.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:02.513 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:02.536 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:02.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:02.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:02.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.681 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:02.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:02.782 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:02.807 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:02.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:02 smithi107 ceph-mon[102221]: pgmap v1546: 109 pgs: 109 active+clean; 2.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 670 op/s 2024-09-21T10:26:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:02 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:02 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:26:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:02 smithi107 ceph-mon[102221]: osdmap e432: 8 total, 8 up, 8 in 2024-09-21T10:26:02.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:02 smithi107 ceph-mon[105022]: pgmap v1546: 109 pgs: 109 active+clean; 2.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 670 op/s 2024-09-21T10:26:02.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:02 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:02.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:02 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:26:02.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:02 smithi107 ceph-mon[105022]: osdmap e432: 8 total, 8 up, 8 in 2024-09-21T10:26:02.908 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:02.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.938 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:02.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:03.039 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:03.069 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:03.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:03.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:02 smithi129 ceph-mon[88471]: pgmap v1546: 109 pgs: 109 active+clean; 2.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 670 op/s 2024-09-21T10:26:03.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:02 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:03.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:02 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:26:03.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:02 smithi129 ceph-mon[88471]: osdmap e432: 8 total, 8 up, 8 in 2024-09-21T10:26:03.170 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:03.181 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:03.194 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:03.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:03.224 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:26:03.224 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:26:03.234 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:03.296 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:03.318 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:03.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:03.420 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:03.468 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:03.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:03.570 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:03.597 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:03.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:03.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:03.747 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:03.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:03.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:03.888 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:03.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:03.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:04.034 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:04.136 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:04.163 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:04.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:04.264 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:04.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:26:04.293 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:04.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:04.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:04.444 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:04.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:04.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:04.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:26:04.623 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:04.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:04.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:04.778 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:04.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:04.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:04 smithi107 ceph-mon[102221]: pgmap v1548: 109 pgs: 109 active+clean; 2.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 794 op/s 2024-09-21T10:26:04.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:04 smithi107 ceph-mon[105022]: pgmap v1548: 109 pgs: 109 active+clean; 2.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 794 op/s 2024-09-21T10:26:04.880 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:04.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.906 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:04.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:05.007 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:05.031 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:05.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:05.133 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:05.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:04 smithi129 ceph-mon[88471]: pgmap v1548: 109 pgs: 109 active+clean; 2.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 794 op/s 2024-09-21T10:26:05.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.176 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:05.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:05.277 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:05.306 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:05.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:05.408 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:05.433 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:05.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:05.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:05.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.582 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:05.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:05.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:05.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.730 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:05.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:05.831 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:05.855 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:05.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:05.957 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:05.998 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:05.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:06.099 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:06.137 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:06.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:06.239 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:06.264 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:06.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:06.365 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:06.391 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:06.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:06.492 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:06.516 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:06.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:06.618 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:06.643 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:06.645 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:06.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:06.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:06.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.790 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:06.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:06.886 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:06 smithi107 ceph-mon[102221]: pgmap v1549: 109 pgs: 109 active+clean; 2.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 795 op/s 2024-09-21T10:26:06.886 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:06 smithi107 ceph-mon[105022]: pgmap v1549: 109 pgs: 109 active+clean; 2.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 795 op/s 2024-09-21T10:26:06.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:06.930 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:06.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:07.032 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:07.065 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:07.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:07.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:07.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:06 smithi129 ceph-mon[88471]: pgmap v1549: 109 pgs: 109 active+clean; 2.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 795 op/s 2024-09-21T10:26:07.212 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:07.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:07.313 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:07.339 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:07.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:07.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:07.476 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:07.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:07.533 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:07.578 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:07.605 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:07.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:07.706 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:07.762 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:07.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:07.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:07.901 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:07.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:08.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:08.039 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:08.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:08.141 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:08.168 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:08.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:08.270 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:08.296 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:08.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:08.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:08.439 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:08.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:08.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:08.585 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:08.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:08.689 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:08.712 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:08.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:08.808 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2024-09-21T10:26:08.808 INFO:teuthology.orchestra.run.smithi107.stderr:0 2024-09-21T10:26:08.808 INFO:teuthology.orchestra.run.smithi107.stderr:1 2024-09-21T10:26:08.808 INFO:teuthology.orchestra.run.smithi107.stderr:2 2024-09-21T10:26:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2024-09-21T10:26:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2024-09-21T10:26:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:bench [] [] [] [] 2024-09-21T10:26:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:26:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2024-09-21T10:26:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2024-09-21T10:26:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2024-09-21T10:26:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:Error EINVAL: invalid command 2024-09-21T10:26:08.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:08.852 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:08.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:08.954 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:08.985 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:08.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:09.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:09.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:08 smithi107 ceph-mon[102221]: pgmap v1550: 109 pgs: 109 active+clean; 2.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 806 op/s 2024-09-21T10:26:09.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:08 smithi107 ceph-mon[105022]: pgmap v1550: 109 pgs: 109 active+clean; 2.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 806 op/s 2024-09-21T10:26:09.119 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:09.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:09.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:08 smithi129 ceph-mon[88471]: pgmap v1550: 109 pgs: 109 active+clean; 2.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 806 op/s 2024-09-21T10:26:09.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:09.263 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:09.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:09.365 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:09.399 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:09.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:09.501 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:09.522 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:09.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:26:09.586 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:26:09.624 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:09.657 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:09.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:09.796 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:09.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:09.839 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:26:09.839 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:26:09.872 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:26:09.872 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:26:09.899 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:09.911 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:26:09.911 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:26:09.941 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:26:09.941 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:26:09.952 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:09.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:09.984 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:26:09.984 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:26:10.017 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:26:10.018 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:26:10.050 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:26:10.050 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:26:10.055 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:10.077 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:10.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:10.085 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:26:10.085 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:26:10.179 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:10.211 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:10.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:10.312 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:10.336 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:10.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:10.438 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:10.466 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:10.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:10.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:10.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.615 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:10.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:10.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:26:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:26:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:26:10.717 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:10.742 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:10.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:10.844 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:10.871 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:10.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:10.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:11.024 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:11.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:11.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:10 smithi107 ceph-mon[102221]: pgmap v1551: 109 pgs: 109 active+clean; 2.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-21T10:26:11.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:10 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1781493831' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:11.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:10 smithi107 ceph-mon[105022]: pgmap v1551: 109 pgs: 109 active+clean; 2.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-21T10:26:11.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:10 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1781493831' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:11.127 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:11.154 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:11.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:11.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:10 smithi129 ceph-mon[88471]: pgmap v1551: 109 pgs: 109 active+clean; 2.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-21T10:26:11.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:10 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1781493831' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:11.255 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:11.283 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:11.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:11.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:11.414 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:11.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:11.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:11.548 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:11.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:11.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:11.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.685 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:11.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:11.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:11.821 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:11.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:11.923 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:11.946 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:11.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:12.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:12.081 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:12.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:12.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:12.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:12.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:26:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:26:11] "GET /metrics HTTP/1.1" 200 36774 "" "Prometheus/2.51.0" 2024-09-21T10:26:12.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:12.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:12.218 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:12.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:12.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:12.359 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:12.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:12.461 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:12.485 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:12.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:12.586 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:12.629 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:12.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:12.731 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:12.769 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:12.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:12.871 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:12.894 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:12.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:12.995 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:13.025 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:13.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:13.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:12 smithi107 ceph-mon[102221]: pgmap v1552: 109 pgs: 109 active+clean; 2.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 841 op/s 2024-09-21T10:26:13.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:12 smithi107 ceph-mon[105022]: pgmap v1552: 109 pgs: 109 active+clean; 2.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 841 op/s 2024-09-21T10:26:13.127 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:13.154 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:13.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:13.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:12 smithi129 ceph-mon[88471]: pgmap v1552: 109 pgs: 109 active+clean; 2.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 841 op/s 2024-09-21T10:26:13.256 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:13.282 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:13.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:13.383 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:13.409 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:13.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:13.510 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:13.534 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:13.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:13.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:13.673 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:13.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:13.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:13.817 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:13.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:13.919 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:13.949 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:14.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:14.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.086 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:14.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:14.186 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:14.213 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:14.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:14.315 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:14.342 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:14.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:14.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:14.482 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:14.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:14.584 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:14.587 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 4, 6] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:26:14.587 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:26:14.587 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:26:14.587 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.009628 2024-09-21T10:26:14.587 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 5 0.009627691295640894 2024-09-21T10:26:14.611 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:14.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:14.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:14.751 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:14.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:14.852 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:14.875 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:14.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:14.905 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:14 smithi107 ceph-mon[102221]: pgmap v1553: 109 pgs: 109 active+clean; 1.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 946 op/s 2024-09-21T10:26:14.905 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:14 smithi107 ceph-mon[105022]: pgmap v1553: 109 pgs: 109 active+clean; 1.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 946 op/s 2024-09-21T10:26:14.977 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:15.005 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:15.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:15.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:15.156 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:15.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:15.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:14 smithi129 ceph-mon[88471]: pgmap v1553: 109 pgs: 109 active+clean; 1.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 946 op/s 2024-09-21T10:26:15.257 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:15.281 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:15.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:15.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:15.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.431 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:15.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:15.504 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:15.533 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:15.587 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:15.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:15.689 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:15.730 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:15.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:15.831 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:15.882 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:15.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:15.985 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:16.010 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:16.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:16.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:16.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.144 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:16.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:16.245 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:16.268 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:16.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:16.334 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:26:16.334 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:26:16.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.334 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47186.0:205252 14.0 14:0ca586c2:::benchmark_data_smithi107_312686_object89461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-09-21T10:26:16.334 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:26:16.289362+0000", 2024-09-21T10:26:16.334 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.021871541000000001, 2024-09-21T10:26:16.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.021901622999999999, 2024-09-21T10:26:16.334 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:26:16.334 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:26:16.334 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:26:16.334 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:26:16.335 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47186", 2024-09-21T10:26:16.335 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:26:16.335 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 205252 2024-09-21T10:26:16.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.335 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:26:16.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:26:16.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.289362+0000", 2024-09-21T10:26:16.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:26:16.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:26:16.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.289362+0000", 2024-09-21T10:26:16.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:26:16.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:26:16.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.289363+0000", 2024-09-21T10:26:16.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.714e-06 2024-09-21T10:26:16.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:26:16.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.289369+0000", 2024-09-21T10:26:16.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.874e-06 2024-09-21T10:26:16.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:26:16.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.289383+0000", 2024-09-21T10:26:16.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3414000000000001e-05 2024-09-21T10:26:16.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:26:16.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.289390+0000", 2024-09-21T10:26:16.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5009999999999998e-06 2024-09-21T10:26:16.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:26:16.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.289445+0000", 2024-09-21T10:26:16.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4824999999999998e-05 2024-09-21T10:26:16.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:26:16.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.289549+0000", 2024-09-21T10:26:16.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000104449 2024-09-21T10:26:16.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:26:16.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.289648+0000", 2024-09-21T10:26:16.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8541000000000004e-05 2024-09-21T10:26:16.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:16.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.294706+0000", 2024-09-21T10:26:16.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0050579989999999997 2024-09-21T10:26:16.338 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.338 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.338 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:26:16.338 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.300148+0000", 2024-09-21T10:26:16.338 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0054416680000000002 2024-09-21T10:26:16.338 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:26:16.338 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:26:16.339 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:26:16.339 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.339 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.339 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47186.0:205256 14.0 14:0a0f4113:::benchmark_data_smithi107_312686_object89465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-09-21T10:26:16.339 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:26:16.294489+0000", 2024-09-21T10:26:16.339 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.016744180000000001, 2024-09-21T10:26:16.339 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.016848676, 2024-09-21T10:26:16.339 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:26:16.339 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:26:16.339 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:26:16.339 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:26:16.339 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47186", 2024-09-21T10:26:16.339 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:26:16.339 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 205256 2024-09-21T10:26:16.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.340 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:26:16.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.340 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:26:16.340 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.294489+0000", 2024-09-21T10:26:16.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:26:16.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.340 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:26:16.340 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.294489+0000", 2024-09-21T10:26:16.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:26:16.340 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.340 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.340 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:26:16.340 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.294491+0000", 2024-09-21T10:26:16.340 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.598e-06 2024-09-21T10:26:16.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:26:16.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.294495+0000", 2024-09-21T10:26:16.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7400000000000004e-06 2024-09-21T10:26:16.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:26:16.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.294511+0000", 2024-09-21T10:26:16.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5597e-05 2024-09-21T10:26:16.341 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.341 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.341 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:26:16.341 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.294518+0000", 2024-09-21T10:26:16.341 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1799999999999999e-06 2024-09-21T10:26:16.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:26:16.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.294884+0000", 2024-09-21T10:26:16.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00036620499999999998 2024-09-21T10:26:16.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:26:16.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.295153+0000", 2024-09-21T10:26:16.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000268368 2024-09-21T10:26:16.342 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.342 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.342 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:26:16.342 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.295484+0000", 2024-09-21T10:26:16.342 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000331088 2024-09-21T10:26:16.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:16.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.304430+0000", 2024-09-21T10:26:16.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0089459889999999997 2024-09-21T10:26:16.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.343 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:26:16.343 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.304509+0000", 2024-09-21T10:26:16.343 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8966000000000003e-05 2024-09-21T10:26:16.343 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:26:16.343 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:26:16.343 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:26:16.343 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.343 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.344 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.47186.0:205246 14.c e432/411 14:3fe251bc:::benchmark_data_smithi107_312686_object89455:head v 432'12994, mlcod=432'12991)", 2024-09-21T10:26:16.344 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:26:16.300403+0000", 2024-09-21T10:26:16.344 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.010830213, 2024-09-21T10:26:16.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.011001179, 2024-09-21T10:26:16.344 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:26:16.344 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:26:16.344 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:16.344 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:26:16.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:26:16.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.300403+0000", 2024-09-21T10:26:16.344 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:26:16.344 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.344 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.344 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:26:16.344 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.300403+0000", 2024-09-21T10:26:16.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:26:16.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:26:16.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.300404+0000", 2024-09-21T10:26:16.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6299999999999993e-07 2024-09-21T10:26:16.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:26:16.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.300411+0000", 2024-09-21T10:26:16.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.635e-06 2024-09-21T10:26:16.345 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.345 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.345 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:26:16.345 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.300425+0000", 2024-09-21T10:26:16.345 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4245e-05 2024-09-21T10:26:16.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:26:16.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.300432+0000", 2024-09-21T10:26:16.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6630000000000004e-06 2024-09-21T10:26:16.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:26:16.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.300613+0000", 2024-09-21T10:26:16.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00018172800000000001 2024-09-21T10:26:16.346 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.346 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.346 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:26:16.346 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.300633+0000", 2024-09-21T10:26:16.346 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9707999999999998e-05 2024-09-21T10:26:16.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.347 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:26:16.347 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.304724+0000", 2024-09-21T10:26:16.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0040912199999999996 2024-09-21T10:26:16.347 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:26:16.347 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:26:16.347 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:26:16.347 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.347 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.347 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.47186.0:205253 14.8 e432/411 14:1cd44d45:::benchmark_data_smithi107_312686_object89462:head v 432'12850, mlcod=432'12848)", 2024-09-21T10:26:16.347 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:26:16.300455+0000", 2024-09-21T10:26:16.347 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.010777961000000001, 2024-09-21T10:26:16.347 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010999518999999999, 2024-09-21T10:26:16.347 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:26:16.348 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:26:16.348 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:26:16.348 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:26:16.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:26:16.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.300455+0000", 2024-09-21T10:26:16.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:26:16.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.348 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:26:16.348 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.300455+0000", 2024-09-21T10:26:16.348 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:26:16.348 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.348 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:26:16.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.300456+0000", 2024-09-21T10:26:16.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9299999999999999e-07 2024-09-21T10:26:16.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:26:16.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.300458+0000", 2024-09-21T10:26:16.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5490000000000001e-06 2024-09-21T10:26:16.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:26:16.349 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.300480+0000", 2024-09-21T10:26:16.349 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2138e-05 2024-09-21T10:26:16.349 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.349 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.349 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:26:16.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.300488+0000", 2024-09-21T10:26:16.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6210000000000004e-06 2024-09-21T10:26:16.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:26:16.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.304464+0000", 2024-09-21T10:26:16.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0039759740000000002 2024-09-21T10:26:16.350 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.350 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.350 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:26:16.350 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.304482+0000", 2024-09-21T10:26:16.350 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8028000000000001e-05 2024-09-21T10:26:16.350 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:26:16.350 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:26:16.350 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:26:16.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.351 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.351 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47186.0:205263 14.0 14:01cfa808:::benchmark_data_smithi107_312686_object89472:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-09-21T10:26:16.351 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:26:16.305136+0000", 2024-09-21T10:26:16.351 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0060969580000000004, 2024-09-21T10:26:16.351 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0063675770000000001, 2024-09-21T10:26:16.351 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:26:16.351 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:26:16.351 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:26:16.351 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:26:16.351 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47186", 2024-09-21T10:26:16.351 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:26:16.351 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 205263 2024-09-21T10:26:16.351 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.351 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:26:16.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:26:16.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.305136+0000", 2024-09-21T10:26:16.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:26:16.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:26:16.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.305136+0000", 2024-09-21T10:26:16.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:26:16.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:26:16.352 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.305138+0000", 2024-09-21T10:26:16.352 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.764e-06 2024-09-21T10:26:16.352 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.352 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.352 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:26:16.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.305143+0000", 2024-09-21T10:26:16.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7829999999999998e-06 2024-09-21T10:26:16.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:26:16.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.305155+0000", 2024-09-21T10:26:16.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2136e-05 2024-09-21T10:26:16.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:26:16.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.305162+0000", 2024-09-21T10:26:16.353 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1940000000000001e-06 2024-09-21T10:26:16.353 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.353 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.353 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:26:16.353 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.305220+0000", 2024-09-21T10:26:16.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7899000000000002e-05 2024-09-21T10:26:16.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:26:16.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.305389+0000", 2024-09-21T10:26:16.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016877299999999999 2024-09-21T10:26:16.354 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:16.354 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:16.354 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:26:16.354 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:16.305496+0000", 2024-09-21T10:26:16.354 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010697 2024-09-21T10:26:16.354 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:26:16.354 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:26:16.354 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:26:16.354 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:26:16.354 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-09-21T10:26:16.355 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 5 2024-09-21T10:26:16.355 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:26:16.369 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:16.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:26:16.391 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:16.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:16.493 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:16.535 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:16.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:16.636 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:16.660 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:16.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:16.762 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:16.784 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:16.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:16.886 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:16.915 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:16.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:17.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:17.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.056 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:17.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:17.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:16 smithi107 ceph-mon[102221]: pgmap v1554: 109 pgs: 109 active+clean; 1.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 777 op/s 2024-09-21T10:26:17.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:16 smithi107 ceph-mon[105022]: pgmap v1554: 109 pgs: 109 active+clean; 1.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 777 op/s 2024-09-21T10:26:17.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:16 smithi129 ceph-mon[88471]: pgmap v1554: 109 pgs: 109 active+clean; 1.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 777 op/s 2024-09-21T10:26:17.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.190 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:17.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:17.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:17.326 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:17.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:17.427 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:17.455 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:17.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:17.556 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:17.581 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:17.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:17.716 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:17.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:17.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:17.839 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:17.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:17.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:17.979 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:17.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:18.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:18.113 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:18.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:18.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:18.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:26:18.259 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:18.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:18.362 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:18.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.388 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:18.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:18.489 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:18.530 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:18.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:18.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:18.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.684 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:18.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:18.785 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:18.809 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:18.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:18.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:18 smithi107 ceph-mon[102221]: pgmap v1555: 109 pgs: 109 active+clean; 1.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 802 op/s 2024-09-21T10:26:18.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:18 smithi107 ceph-mon[105022]: pgmap v1555: 109 pgs: 109 active+clean; 1.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 802 op/s 2024-09-21T10:26:18.910 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:18.949 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:18.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:19.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:19.095 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:19.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:19.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:18 smithi129 ceph-mon[88471]: pgmap v1555: 109 pgs: 109 active+clean; 1.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 802 op/s 2024-09-21T10:26:19.197 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:19.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.228 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:19.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:19.328 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:19.353 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:19.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:19.454 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:19.476 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:19.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:19.578 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:19.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.625 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:19.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:19.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:19.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.781 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:19.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:19.882 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:19.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.909 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:19.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:20.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:20.068 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:20.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:20.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:20.217 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:20.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:20.258 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:20.270 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:20.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:20.362 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:20.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:20.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:20.517 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:20.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:20.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:20.658 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:20.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:26:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:26:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:26:20.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:20.778 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:26:20.780 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.5 primary-affinity to 0.00962769 (86302) 2024-09-21T10:26:20.821 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:20.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:20.848 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:20 smithi107 ceph-mon[102221]: pgmap v1556: 109 pgs: 109 active+clean; 1.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 916 op/s 2024-09-21T10:26:20.848 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:20 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/694222921' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:20.848 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:20 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3860915542' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.009627691295640894} : dispatch 2024-09-21T10:26:20.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:20 smithi107 ceph-mon[105022]: pgmap v1556: 109 pgs: 109 active+clean; 1.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 916 op/s 2024-09-21T10:26:20.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:20 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/694222921' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:20.848 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:20 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3860915542' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.009627691295640894} : dispatch 2024-09-21T10:26:20.923 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:20.955 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:20.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:21.057 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:21.082 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:21.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:21.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:20 smithi129 ceph-mon[88471]: pgmap v1556: 109 pgs: 109 active+clean; 1.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 916 op/s 2024-09-21T10:26:21.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:20 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/694222921' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:21.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:20 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3860915542' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.009627691295640894} : dispatch 2024-09-21T10:26:21.183 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:21.238 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:21.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:21.339 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:21.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.371 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:21.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:21.471 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:21.496 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:21.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:21.598 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:21.622 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:21.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:21.724 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:21.754 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:21.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:21.856 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:21.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:21 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:26:21.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:21 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:26:21.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:21 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/694222921' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:21.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:21 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3860915542' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.009627691295640894}]': finished 2024-09-21T10:26:21.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:21 smithi107 ceph-mon[102221]: osdmap e433: 8 total, 8 up, 8 in 2024-09-21T10:26:21.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:21 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:26:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:21 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:26:21.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:21 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:26:21.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:21 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/694222921' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:21.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:21 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3860915542' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.009627691295640894}]': finished 2024-09-21T10:26:21.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:21 smithi107 ceph-mon[105022]: osdmap e433: 8 total, 8 up, 8 in 2024-09-21T10:26:21.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:21 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:26:21.860 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:26:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:26:21] "GET /metrics HTTP/1.1" 200 36774 "" "Prometheus/2.51.0" 2024-09-21T10:26:21.884 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:21.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:21.986 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:22.017 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:22.018 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:21 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:26:22.018 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:21 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:26:22.018 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:21 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/694222921' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:22.018 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:21 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3860915542' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.009627691295640894}]': finished 2024-09-21T10:26:22.018 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:21 smithi129 ceph-mon[88471]: osdmap e433: 8 total, 8 up, 8 in 2024-09-21T10:26:22.018 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:21 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:26:22.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:22.119 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:22.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.168 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:22.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:22.269 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:22.293 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:22.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:22.394 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:22.432 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:22.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:22.533 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:22.564 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:22.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:22.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:22.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.709 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:22.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:22.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:22.850 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:22.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:22.952 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:22.976 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:22.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:23.066 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:22 smithi107 ceph-mon[102221]: pgmap v1558: 109 pgs: 109 active+clean; 1.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-21T10:26:23.066 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:22 smithi107 ceph-mon[105022]: pgmap v1558: 109 pgs: 109 active+clean; 1.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-21T10:26:23.078 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:23.102 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:23.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:23.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:22 smithi129 ceph-mon[88471]: pgmap v1558: 109 pgs: 109 active+clean; 1.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 819 op/s 2024-09-21T10:26:23.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:23.247 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:23.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:23.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:23.388 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:23.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:23.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:23.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.541 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:23.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:23.642 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:23.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:26:23.668 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:23.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:23.770 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:23.795 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:23.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:23.896 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:23.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.927 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:23.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:24.028 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:24.081 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:24.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:24.182 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:24.207 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:24.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:24.309 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:24.366 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:24.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:24.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:24.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.507 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:24.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:24.608 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:24.639 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:24.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:24.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:24.777 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:24.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:24.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:24.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.926 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:24.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:25.027 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:25.077 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:25.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:25.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:24 smithi107 ceph-mon[102221]: pgmap v1559: 109 pgs: 109 active+clean; 1.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 818 op/s 2024-09-21T10:26:25.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:24 smithi107 ceph-mon[105022]: pgmap v1559: 109 pgs: 109 active+clean; 1.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 818 op/s 2024-09-21T10:26:25.118 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:26:25.118 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:26:25.160 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:26:25.160 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:26:25.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:24 smithi129 ceph-mon[88471]: pgmap v1559: 109 pgs: 109 active+clean; 1.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 818 op/s 2024-09-21T10:26:25.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:25.203 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:26:25.203 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:26:25.214 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:25.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:25.238 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:26:25.239 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:26:25.271 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:26:25.271 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:26:25.312 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:26:25.313 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:26:25.313 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:26:25.313 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:26:25.316 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:25.337 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:25.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:25.351 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:26:25.351 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:26:25.389 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:26:25.389 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:26:25.438 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:25.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.475 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:25.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:25.576 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:25.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.602 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:25.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:25.702 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:25.740 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:25.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:25.796 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:25.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:25.884 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:25.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:25.986 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:26.008 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:26.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:26.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:25 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4066548858' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:26.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:25 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4066548858' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:26.109 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:26.140 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:26.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:25 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4066548858' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:26.243 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:26.278 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:26.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:26.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:26:26.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:26.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.442 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:26.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:26.543 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:26.603 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:26.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:26.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:26.747 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:26.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:26.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:26.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.897 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:26.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:26.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:27.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.046 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:27.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:27.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:27.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.192 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:27.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:27.293 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:27.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.350 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:27.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:27.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:27 smithi107 ceph-mon[102221]: pgmap v1560: 109 pgs: 109 active+clean; 1.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 814 op/s 2024-09-21T10:26:27.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:27.361 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:27.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:27 smithi107 ceph-mon[105022]: pgmap v1560: 109 pgs: 109 active+clean; 1.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 814 op/s 2024-09-21T10:26:27.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:27.361 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:27 smithi129 ceph-mon[88471]: pgmap v1560: 109 pgs: 109 active+clean; 1.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 814 op/s 2024-09-21T10:26:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:27.451 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:27.476 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:27.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:27.478 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 4, 6] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:26:27.478 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:26:27.479 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-21T10:26:27.479 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:26:27.479 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:26:27.479 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-21T10:26:27.578 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:27.611 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:27.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:27.712 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:27.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.755 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:27.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:27.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:27.886 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:27.893 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:27.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:27.995 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:28.022 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:28.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:28.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:28.168 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:28.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:28.191 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:28.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:28.316 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:28.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:28.418 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:28.447 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:28.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:28.549 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:28.572 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:28.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:28.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:28 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:28.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:28 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:28.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:28 smithi107 ceph-mon[102221]: pgmap v1561: 109 pgs: 109 active+clean; 1.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 799 op/s 2024-09-21T10:26:28.601 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:28 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:28.602 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:28 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:28.602 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:28 smithi107 ceph-mon[105022]: pgmap v1561: 109 pgs: 109 active+clean; 1.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 799 op/s 2024-09-21T10:26:28.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:28 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:28.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:28 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:28.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:28 smithi129 ceph-mon[88471]: pgmap v1561: 109 pgs: 109 active+clean; 1.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 799 op/s 2024-09-21T10:26:28.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:28.720 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:28.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:28.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:26:28.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:28.895 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:28.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:28.997 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:29.025 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:29.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:29.127 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:29.175 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:29.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:29.277 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:29.317 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:29.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:29.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:29.451 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:29.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:29.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:29.589 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:29.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:29.690 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:29.719 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:29.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:29.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:29.858 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:29.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:30.009 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:30.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:30.112 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:30.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.142 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:30.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:30.243 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:30.301 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:30.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:30.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:30.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.448 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:30.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:30.549 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:30.572 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:26:30.572 INFO:teuthology.orchestra.run.smithi129.stdout: "success": "" 2024-09-21T10:26:30.572 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:26:30.572 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:26:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:26:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:26:30.573 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:30.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:30.674 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:30.702 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:30.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:30.804 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:30.826 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:30.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:30.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:30 smithi129 ceph-mon[88471]: pgmap v1562: 109 pgs: 109 active+clean; 1.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 807 op/s 2024-09-21T10:26:30.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:30.992 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:30.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:31.035 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:30 smithi107 ceph-mon[102221]: pgmap v1562: 109 pgs: 109 active+clean; 1.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 807 op/s 2024-09-21T10:26:31.035 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:30 smithi107 ceph-mon[105022]: pgmap v1562: 109 pgs: 109 active+clean; 1.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 807 op/s 2024-09-21T10:26:31.094 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:31.119 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:31.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:31.220 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:31.244 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:31.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:31.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:31.386 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:31.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:31.420 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:26:31.420 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:26:31.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:31.531 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:31.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:31.632 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:31.641 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:31.658 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:31.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:31.692 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:26:31.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:31.806 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:31.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:31.908 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:31.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:31 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1534671679' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:26:31.933 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:31.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:32.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:32.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.068 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:32.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:32.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:31 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1534671679' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:26:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:26:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:26:31] "GET /metrics HTTP/1.1" 200 36774 "" "Prometheus/2.51.0" 2024-09-21T10:26:32.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:31 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1534671679' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:26:32.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:32.214 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:32.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:32.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:26:32.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:32.378 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:32.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:32.480 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:32.506 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:32.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:32.607 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:32.637 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:32.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:32.739 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:32.785 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:32.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:32.887 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:32.914 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:32.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:33.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:33.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.051 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:33.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:33.070 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:32 smithi107 ceph-mon[102221]: pgmap v1563: 109 pgs: 109 active+clean; 1.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 749 op/s 2024-09-21T10:26:33.070 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:32 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:33.070 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:32 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1534671679' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:26:33.070 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:32 smithi107 ceph-mon[102221]: osdmap e434: 8 total, 8 up, 8 in 2024-09-21T10:26:33.071 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:32 smithi107 ceph-mon[105022]: pgmap v1563: 109 pgs: 109 active+clean; 1.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 749 op/s 2024-09-21T10:26:33.071 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:32 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:33.071 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:32 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1534671679' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:26:33.071 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:32 smithi107 ceph-mon[105022]: osdmap e434: 8 total, 8 up, 8 in 2024-09-21T10:26:33.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:33.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:32 smithi129 ceph-mon[88471]: pgmap v1563: 109 pgs: 109 active+clean; 1.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 749 op/s 2024-09-21T10:26:33.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:32 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:33.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:32 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1534671679' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:26:33.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:32 smithi129 ceph-mon[88471]: osdmap e434: 8 total, 8 up, 8 in 2024-09-21T10:26:33.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.200 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:33.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:33.301 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:33.324 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:33.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:33.426 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:33.450 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:33.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:33.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:33.596 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:33.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:33.698 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:33.723 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:33.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:33.824 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:33.825 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:33.848 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:33.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:33.949 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:33.994 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:33.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:34.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:34.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.135 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:34.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:34.236 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:34.259 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:34.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:34.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:34.415 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:34.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:34.517 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:34.543 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:34.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:34.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:34.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.696 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:34.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:34.797 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:34.822 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:34.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:34.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:34 smithi107 ceph-mon[102221]: pgmap v1565: 109 pgs: 109 active+clean; 970 MiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 626 op/s 2024-09-21T10:26:34.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:34 smithi107 ceph-mon[105022]: pgmap v1565: 109 pgs: 109 active+clean; 970 MiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 626 op/s 2024-09-21T10:26:34.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:34.985 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:34.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:35.087 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:35.111 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:26:35.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.112 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:35.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:35.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:34 smithi129 ceph-mon[88471]: pgmap v1565: 109 pgs: 109 active+clean; 970 MiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 626 op/s 2024-09-21T10:26:35.213 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:35.237 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:35.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:35.339 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:35.361 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:35.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:35.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:35.526 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:35.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:35.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:35.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.672 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:35.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:35.773 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:35.799 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:35.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:35.901 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:35.927 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:35.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:36.029 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:36.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.052 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:36.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:36.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:35 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/870050683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:36.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:35 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/870050683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:36.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:36.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:35 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/870050683' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:36.194 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:36.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:36.296 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:36.324 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:36.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:36.426 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:36.456 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:36.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:36.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:36.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.595 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:36.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:36.690 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 7, 4, 6] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:26:36.690 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:26:36.690 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 5, 7, 4, 6] 2024-09-21T10:26:36.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd out 3 2024-09-21T10:26:36.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:36.776 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:36.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:36.878 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:36.905 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:36.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:37.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:37.047 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:37.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:37.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:36 smithi107 ceph-mon[102221]: pgmap v1566: 109 pgs: 109 active+clean; 778 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 805 op/s 2024-09-21T10:26:37.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:36 smithi107 ceph-mon[105022]: pgmap v1566: 109 pgs: 109 active+clean; 778 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 805 op/s 2024-09-21T10:26:37.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:37.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:36 smithi129 ceph-mon[88471]: pgmap v1566: 109 pgs: 109 active+clean; 778 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 805 op/s 2024-09-21T10:26:37.187 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:37.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:37.289 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:37.312 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:37.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:37.413 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:37.440 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:37.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:37.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:37.584 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:37.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:37.686 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:37.736 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:37.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:37.838 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:37.860 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:37.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:37.962 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:37.992 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:37.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:38.094 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:38.120 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:38.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:38.222 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:38.248 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:38.350 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:38.389 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:38.490 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:38.518 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:38.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:38.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:38.653 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:38.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:38.663 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:26:38.664 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:26:38.664 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:26:38.664 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:26:38.664 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:26:38.664 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:26:38.664 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:26:38.664 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:26:38.664 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:26:38.664 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:26:38.664 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:26:38.664 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:26:38.664 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:26:38.665 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:26:38.665 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:26:38.665 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:26:38.755 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:38.779 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:38.881 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:38.911 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:38.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:39.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.050 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:39.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:39.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:38 smithi107 ceph-mon[102221]: pgmap v1567: 109 pgs: 109 active+clean; 716 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 810 op/s 2024-09-21T10:26:39.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:38 smithi107 ceph-mon[105022]: pgmap v1567: 109 pgs: 109 active+clean; 716 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 810 op/s 2024-09-21T10:26:39.151 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:39.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:38 smithi129 ceph-mon[88471]: pgmap v1567: 109 pgs: 109 active+clean; 716 MiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 810 op/s 2024-09-21T10:26:39.177 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:39.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:39.278 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:39.315 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:39.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:39.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:39.449 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:39.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:39.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:39.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.590 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:39.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:39.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:39.722 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:39.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:39.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:39.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.845 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:39.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:39.946 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:39.972 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:39.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:40.073 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:40.096 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:40.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:40.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:40.224 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:40.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:40.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:40.367 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:40.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:40.469 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:40.495 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:40.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:40.597 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:40.636 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:40.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:40.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:26:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:26:40.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:26:40.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:40.803 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:40.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:40.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:40.941 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:40.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:41.042 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:41.061 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:40 smithi129 ceph-mon[88471]: pgmap v1568: 109 pgs: 109 active+clean; 558 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 811 op/s 2024-09-21T10:26:41.070 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:41.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:41.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:40 smithi107 ceph-mon[102221]: pgmap v1568: 109 pgs: 109 active+clean; 558 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 811 op/s 2024-09-21T10:26:41.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:40 smithi107 ceph-mon[105022]: pgmap v1568: 109 pgs: 109 active+clean; 558 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 811 op/s 2024-09-21T10:26:41.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:41.206 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:41.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:41.308 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:41.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.343 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:41.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:41.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:41.485 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:41.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:41.587 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:41.629 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:41.731 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:41.767 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:41.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:41.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:41.909 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:41.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:42.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:42.047 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:42.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:42.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:26:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:26:41] "GET /metrics HTTP/1.1" 200 36771 "" "Prometheus/2.51.0" 2024-09-21T10:26:42.149 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:42.177 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:42.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:42.279 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:42.302 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:42.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:42.404 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:42.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.478 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:42.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:42.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:42.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:42 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:42.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:42.540 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:42 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:42.580 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:42.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.609 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:42.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:42.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:42 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:42.710 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:42.731 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.776 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:42.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:42.877 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:42.922 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:42.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:43.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:43.068 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:43.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:43.170 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:43.196 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:43.196 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:43 smithi107 ceph-mon[102221]: pgmap v1569: 109 pgs: 109 active+clean; 527 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 806 op/s 2024-09-21T10:26:43.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:43.198 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:43 smithi107 ceph-mon[105022]: pgmap v1569: 109 pgs: 109 active+clean; 527 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 806 op/s 2024-09-21T10:26:43.290 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:26:43.290 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:26:43.290 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.290 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.47186.0:226939 14.3 e434/411 14:c3064abd:::benchmark_data_smithi107_312686_object111148:head v 434'14144, mlcod=434'14142)", 2024-09-21T10:26:43.291 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:26:43.213436+0000", 2024-09-21T10:26:43.291 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0019992260000000002, 2024-09-21T10:26:43.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020254460000000002, 2024-09-21T10:26:43.291 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:26:43.291 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:26:43.291 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:26:43.291 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:26:43.291 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.291 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:26:43.291 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213436+0000", 2024-09-21T10:26:43.291 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:26:43.291 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.292 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:26:43.292 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213436+0000", 2024-09-21T10:26:43.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:26:43.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.292 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:26:43.292 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213438+0000", 2024-09-21T10:26:43.292 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.294e-06 2024-09-21T10:26:43.292 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.292 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.292 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:26:43.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213447+0000", 2024-09-21T10:26:43.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.55e-06 2024-09-21T10:26:43.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:26:43.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213467+0000", 2024-09-21T10:26:43.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9885999999999999e-05 2024-09-21T10:26:43.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.293 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:26:43.293 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213476+0000", 2024-09-21T10:26:43.293 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6710000000000005e-06 2024-09-21T10:26:43.293 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.293 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:26:43.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213533+0000", 2024-09-21T10:26:43.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7333999999999997e-05 2024-09-21T10:26:43.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.294 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.294 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:26:43.294 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213554+0000", 2024-09-21T10:26:43.294 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0695000000000001e-05 2024-09-21T10:26:43.294 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:26:43.294 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:26:43.294 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:26:43.294 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.295 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.47186.0:226943 14.f e434/411 14:faa860a5:::benchmark_data_smithi107_312686_object111152:head v 434'13900, mlcod=434'13898)", 2024-09-21T10:26:43.295 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:26:43.213510+0000", 2024-09-21T10:26:43.295 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.001925997, 2024-09-21T10:26:43.295 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0020040280000000001, 2024-09-21T10:26:43.295 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:26:43.295 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:26:43.295 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-09-21T10:26:43.295 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:26:43.295 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.295 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:26:43.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213510+0000", 2024-09-21T10:26:43.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:26:43.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.296 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:26:43.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213510+0000", 2024-09-21T10:26:43.296 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:26:43.296 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.296 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.296 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:26:43.296 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213510+0000", 2024-09-21T10:26:43.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8100000000000003e-07 2024-09-21T10:26:43.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:26:43.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213514+0000", 2024-09-21T10:26:43.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0729999999999998e-06 2024-09-21T10:26:43.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.297 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.297 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:26:43.297 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213521+0000", 2024-09-21T10:26:43.297 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.888e-06 2024-09-21T10:26:43.297 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:26:43.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213526+0000", 2024-09-21T10:26:43.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4939999999999997e-06 2024-09-21T10:26:43.298 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.298 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:26:43.298 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213920+0000", 2024-09-21T10:26:43.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00039484900000000001 2024-09-21T10:26:43.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.299 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:26:43.299 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213937+0000", 2024-09-21T10:26:43.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6325e-05 2024-09-21T10:26:43.299 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:26:43.299 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:26:43.299 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:26:43.299 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.300 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47186.0:226940 14.1 14:8ccfe13f:::benchmark_data_smithi107_312686_object111149:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-21T10:26:43.300 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:26:43.213798+0000", 2024-09-21T10:26:43.300 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.0016374580000000001, 2024-09-21T10:26:43.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017656080000000001, 2024-09-21T10:26:43.300 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:26:43.300 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:26:43.300 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:26:43.300 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:26:43.300 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47186", 2024-09-21T10:26:43.300 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:26:43.300 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 226940 2024-09-21T10:26:43.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.301 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:26:43.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:26:43.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213798+0000", 2024-09-21T10:26:43.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:26:43.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:26:43.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213798+0000", 2024-09-21T10:26:43.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:26:43.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:26:43.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213800+0000", 2024-09-21T10:26:43.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.511e-06 2024-09-21T10:26:43.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:26:43.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213805+0000", 2024-09-21T10:26:43.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5310000000000001e-06 2024-09-21T10:26:43.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:26:43.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213817+0000", 2024-09-21T10:26:43.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1959e-05 2024-09-21T10:26:43.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:26:43.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213832+0000", 2024-09-21T10:26:43.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4712000000000001e-05 2024-09-21T10:26:43.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:26:43.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213882+0000", 2024-09-21T10:26:43.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9734999999999997e-05 2024-09-21T10:26:43.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.304 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:26:43.304 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213998+0000", 2024-09-21T10:26:43.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000115888 2024-09-21T10:26:43.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:26:43.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.214100+0000", 2024-09-21T10:26:43.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102638 2024-09-21T10:26:43.305 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:26:43.305 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:26:43.305 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:26:43.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.305 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_repop(client.47186.0:226941 14.4 e434/411)", 2024-09-21T10:26:43.305 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:26:43.213862+0000", 2024-09-21T10:26:43.305 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 0.001573666, 2024-09-21T10:26:43.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0017581999999999999, 2024-09-21T10:26:43.306 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:26:43.306 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:26:43.306 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "queued for pg", 2024-09-21T10:26:43.306 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:26:43.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:26:43.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213862+0000", 2024-09-21T10:26:43.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:26:43.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:26:43.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213862+0000", 2024-09-21T10:26:43.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:26:43.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:26:43.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213863+0000", 2024-09-21T10:26:43.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4099999999999998e-07 2024-09-21T10:26:43.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:26:43.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213867+0000", 2024-09-21T10:26:43.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1980000000000003e-06 2024-09-21T10:26:43.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:26:43.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213874+0000", 2024-09-21T10:26:43.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2710000000000003e-06 2024-09-21T10:26:43.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:26:43.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:26:43.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:26:43.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:43.213879+0000", 2024-09-21T10:26:43.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7690000000000004e-06 2024-09-21T10:26:43.308 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:26:43.309 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:26:43.309 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:26:43.309 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:26:43.309 INFO:teuthology.orchestra.run.smithi107.stdout: ], 2024-09-21T10:26:43.309 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 4 2024-09-21T10:26:43.309 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:26:43.309 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:43.335 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:43.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:43.436 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:43.476 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:43.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:43.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:43.620 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:43.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:43 smithi129 ceph-mon[88471]: pgmap v1569: 109 pgs: 109 active+clean; 527 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 806 op/s 2024-09-21T10:26:43.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:43.763 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:43.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:43.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:43.903 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:43.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:43.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:26:44.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:44.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.048 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:44.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:44.141 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:26:44.150 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:44.172 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:44.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:44.274 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:44.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.327 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:44.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:44.428 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:44.454 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:44.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:44.475 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:44.556 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:44.601 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:44.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:44.703 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:44.729 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:44.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:44.831 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:44.859 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:44.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:44.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:44.999 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:44.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:45.101 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:45.126 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:45.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:45.227 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:45.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.264 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:45.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:45.364 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:45.386 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:45.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:45.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:45.533 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:45.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:45.569 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:45 smithi107 ceph-mon[102221]: pgmap v1570: 109 pgs: 109 active+clean; 461 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-09-21T10:26:45.569 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:45 smithi107 ceph-mon[105022]: pgmap v1570: 109 pgs: 109 active+clean; 461 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-09-21T10:26:45.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:45 smithi129 ceph-mon[88471]: pgmap v1570: 109 pgs: 109 active+clean; 461 MiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 765 op/s 2024-09-21T10:26:45.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.688 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:45.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:45.789 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:45.815 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:45.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:45.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:45.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.983 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:45.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:46.084 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:46.124 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:46.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:46.227 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:46.266 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:46.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:46.368 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:46.395 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:46.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:46.497 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:46.521 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:46.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:46.563 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:46.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:46.678 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:46.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:46.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:46.822 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:46.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:46.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:46.958 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:46.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:47.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:47.113 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:47.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:47.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:47.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.253 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:47.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:47.339 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:47 smithi107 ceph-mon[102221]: pgmap v1571: 109 pgs: 109 active+clean; 271 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 930 op/s 2024-09-21T10:26:47.339 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:47 smithi107 ceph-mon[105022]: pgmap v1571: 109 pgs: 109 active+clean; 271 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 930 op/s 2024-09-21T10:26:47.354 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:47.380 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:47.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:47.481 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:47.503 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:47.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:47.605 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:47.632 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:47.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:47 smithi129 ceph-mon[88471]: pgmap v1571: 109 pgs: 109 active+clean; 271 MiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 930 op/s 2024-09-21T10:26:47.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:47.773 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:47.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:47.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:47.907 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:47.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:48.009 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:48.049 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:48.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:48.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:48.191 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:48.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:48.293 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:48.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.319 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:48.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:48.420 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:48.442 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:48.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:48.544 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:48.583 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:48.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:48.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:48.734 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:48.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:48.836 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:48.861 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:48.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:48.963 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:48.976 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Removed 115788 objects 2024-09-21T10:26:48.976 INFO:tasks.radosbench.radosbench.0.smithi107.stdout:Clean up completed and total clean up time :142.226 2024-09-21T10:26:48.985 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:48.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:49.039 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:49.087 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:49.108 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:49.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:49.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:49.219 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is set 2024-09-21T10:26:49.234 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:49.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:49.335 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:49.358 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:49.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:49.460 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:49.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.483 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:49.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:49.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:49 smithi107 ceph-mon[102221]: pgmap v1572: 109 pgs: 109 active+clean; 207 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 760 op/s 2024-09-21T10:26:49.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:49 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1055413540' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:49.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:49 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:49.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:49 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:49.559 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:49 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:49.559 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:49 smithi107 ceph-mon[105022]: pgmap v1572: 109 pgs: 109 active+clean; 207 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 760 op/s 2024-09-21T10:26:49.559 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:49 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1055413540' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:49.559 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:49 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:49.559 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:49 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:49.559 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:49 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:49.583 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:49.605 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:49.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:49 smithi129 ceph-mon[88471]: pgmap v1572: 109 pgs: 109 active+clean; 207 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 760 op/s 2024-09-21T10:26:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:49 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1055413540' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:49 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:49 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:49 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:49.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:49.727 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:49.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:49.829 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:49.851 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:49.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:49.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:49.975 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:49.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:50.076 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:50.077 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:49 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a[102198]: 2024-09-21T10:26:49.840+0000 7f3189124640 -1 mon.a@0(leader).osd e435 definitely_dead 0 2024-09-21T10:26:50.099 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:50.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:50.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:50.225 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:50.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:50.236 INFO:teuthology.orchestra.run.smithi107.stderr:marked out osd.3. 2024-09-21T10:26:50.326 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:50.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:50 smithi107 ceph-mon[105022]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:50.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:50 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:50.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:50 smithi107 ceph-mon[105022]: osdmap e435: 8 total, 8 up, 8 in 2024-09-21T10:26:50.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:50 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2208316307' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-21T10:26:50.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:50 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-21T10:26:50.329 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:50 smithi107 ceph-mon[105022]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-21T10:26:50.329 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:50 smithi107 ceph-mon[102221]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:50.329 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:50 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:50.329 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:50 smithi107 ceph-mon[102221]: osdmap e435: 8 total, 8 up, 8 in 2024-09-21T10:26:50.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:50 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2208316307' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-21T10:26:50.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:50 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-21T10:26:50.330 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:50 smithi107 ceph-mon[102221]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-21T10:26:50.348 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:50.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:50.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:50.472 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:50.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:50.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:50.596 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:50.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:26:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:26:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:26:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:50 smithi129 ceph-mon[88471]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:50 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:50 smithi129 ceph-mon[88471]: osdmap e435: 8 total, 8 up, 8 in 2024-09-21T10:26:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:50 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2208316307' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-21T10:26:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:50 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-09-21T10:26:50.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:50 smithi129 ceph-mon[88471]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-21T10:26:50.697 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:50.720 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:50.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:50.822 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:50.844 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:50.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:50.946 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:50.967 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:50.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:51.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:51.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.092 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:51.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:51.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:51.215 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:51.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:51.317 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:51.340 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:51.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:51.442 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:51.463 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:51.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:51.565 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:51.589 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:51.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:51.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:51 smithi107 ceph-mon[105022]: pgmap v1574: 109 pgs: 109 active+clean; 51 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 809 op/s 2024-09-21T10:26:51.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:51 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-21T10:26:51.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:51 smithi107 ceph-mon[105022]: osdmap e436: 8 total, 8 up, 7 in 2024-09-21T10:26:51.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:51 smithi107 ceph-mon[102221]: pgmap v1574: 109 pgs: 109 active+clean; 51 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 809 op/s 2024-09-21T10:26:51.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:51 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-21T10:26:51.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:51 smithi107 ceph-mon[102221]: osdmap e436: 8 total, 8 up, 7 in 2024-09-21T10:26:51.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:51 smithi129 ceph-mon[88471]: pgmap v1574: 109 pgs: 109 active+clean; 51 MiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 809 op/s 2024-09-21T10:26:51.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:51 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-21T10:26:51.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:51 smithi129 ceph-mon[88471]: osdmap e436: 8 total, 8 up, 7 in 2024-09-21T10:26:51.690 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:51.714 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:51.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:51.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:51.837 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:51.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:51.939 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:51.963 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:51.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:52.040 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:26:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:26:51] "GET /metrics HTTP/1.1" 200 36771 "" "Prometheus/2.51.0" 2024-09-21T10:26:52.066 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:52.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.092 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:52.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:52.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:26:52.193 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:52.216 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:52.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:52.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:52.342 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:52.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:52.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:52 smithi107 ceph-mon[102221]: osdmap e437: 8 total, 8 up, 7 in 2024-09-21T10:26:52.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:52 smithi107 ceph-mon[105022]: osdmap e437: 8 total, 8 up, 7 in 2024-09-21T10:26:52.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:52 smithi129 ceph-mon[88471]: osdmap e437: 8 total, 8 up, 7 in 2024-09-21T10:26:52.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:52.474 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:52.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:52.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:52.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.598 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:52.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:52.699 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:52.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.725 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:52.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:52.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:52.848 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:52.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:52.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:52.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.972 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:52.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:53.073 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:53.095 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:53.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:53.196 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:53.218 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:53.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:53.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:53.341 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:53.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:53.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:53.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.465 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:53.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:53.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:53.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.588 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:53.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:53.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:53 smithi107 ceph-mon[105022]: pgmap v1577: 109 pgs: 109 active+clean; 28 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 649 op/s 2024-09-21T10:26:53.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:53 smithi107 ceph-mon[105022]: 14.9 starting backfill to osd.0 from (0'0,0'0] MAX to 434'14556 2024-09-21T10:26:53.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:53 smithi107 ceph-mon[105022]: 14.a starting backfill to osd.4 from (0'0,0'0] MAX to 434'14552 2024-09-21T10:26:53.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:53 smithi107 ceph-mon[105022]: 14.1 starting backfill to osd.6 from (0'0,0'0] MAX to 434'14376 2024-09-21T10:26:53.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:53 smithi107 ceph-mon[105022]: 14.2 starting backfill to osd.5 from (0'0,0'0] MAX to 434'14466 2024-09-21T10:26:53.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:53 smithi107 ceph-mon[105022]: osdmap e438: 8 total, 8 up, 7 in 2024-09-21T10:26:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:53 smithi107 ceph-mon[105022]: 14.4 starting backfill to osd.5 from (0'0,0'0] MAX to 434'14464 2024-09-21T10:26:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:53 smithi107 ceph-mon[105022]: 14.f starting backfill to osd.5 from (0'0,0'0] MAX to 434'14168 2024-09-21T10:26:53.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:53 smithi107 ceph-mon[105022]: 14.3 starting backfill to osd.4 from (0'0,0'0] MAX to 434'14428 2024-09-21T10:26:53.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:53 smithi107 ceph-mon[102221]: pgmap v1577: 109 pgs: 109 active+clean; 28 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 649 op/s 2024-09-21T10:26:53.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:53 smithi107 ceph-mon[102221]: 14.9 starting backfill to osd.0 from (0'0,0'0] MAX to 434'14556 2024-09-21T10:26:53.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:53 smithi107 ceph-mon[102221]: 14.a starting backfill to osd.4 from (0'0,0'0] MAX to 434'14552 2024-09-21T10:26:53.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:53 smithi107 ceph-mon[102221]: 14.1 starting backfill to osd.6 from (0'0,0'0] MAX to 434'14376 2024-09-21T10:26:53.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:53 smithi107 ceph-mon[102221]: 14.2 starting backfill to osd.5 from (0'0,0'0] MAX to 434'14466 2024-09-21T10:26:53.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:53 smithi107 ceph-mon[102221]: osdmap e438: 8 total, 8 up, 7 in 2024-09-21T10:26:53.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:53 smithi107 ceph-mon[102221]: 14.4 starting backfill to osd.5 from (0'0,0'0] MAX to 434'14464 2024-09-21T10:26:53.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:53 smithi107 ceph-mon[102221]: 14.f starting backfill to osd.5 from (0'0,0'0] MAX to 434'14168 2024-09-21T10:26:53.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:53 smithi107 ceph-mon[102221]: 14.3 starting backfill to osd.4 from (0'0,0'0] MAX to 434'14428 2024-09-21T10:26:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:53 smithi129 ceph-mon[88471]: pgmap v1577: 109 pgs: 109 active+clean; 28 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 649 op/s 2024-09-21T10:26:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:53 smithi129 ceph-mon[88471]: 14.9 starting backfill to osd.0 from (0'0,0'0] MAX to 434'14556 2024-09-21T10:26:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:53 smithi129 ceph-mon[88471]: 14.a starting backfill to osd.4 from (0'0,0'0] MAX to 434'14552 2024-09-21T10:26:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:53 smithi129 ceph-mon[88471]: 14.1 starting backfill to osd.6 from (0'0,0'0] MAX to 434'14376 2024-09-21T10:26:53.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:53 smithi129 ceph-mon[88471]: 14.2 starting backfill to osd.5 from (0'0,0'0] MAX to 434'14466 2024-09-21T10:26:53.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:53 smithi129 ceph-mon[88471]: osdmap e438: 8 total, 8 up, 7 in 2024-09-21T10:26:53.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:53 smithi129 ceph-mon[88471]: 14.4 starting backfill to osd.5 from (0'0,0'0] MAX to 434'14464 2024-09-21T10:26:53.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:53 smithi129 ceph-mon[88471]: 14.f starting backfill to osd.5 from (0'0,0'0] MAX to 434'14168 2024-09-21T10:26:53.670 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:53 smithi129 ceph-mon[88471]: 14.3 starting backfill to osd.4 from (0'0,0'0] MAX to 434'14428 2024-09-21T10:26:53.689 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:53.711 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:53.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:53.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:53.835 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:53.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:53.937 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:53.962 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:53.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:54.064 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:54.089 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:54.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:54.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:54.216 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:54.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:54.318 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:54.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.341 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:54.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:54.442 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:54.467 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:54.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:54.569 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:54.592 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:54.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:54.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:54 smithi107 ceph-mon[105022]: osdmap e439: 8 total, 8 up, 7 in 2024-09-21T10:26:54.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:54 smithi107 ceph-mon[102221]: osdmap e439: 8 total, 8 up, 7 in 2024-09-21T10:26:54.631 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_9 2024-09-21T10:26:54.632 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_9 unique_pool_9 --yes-i-really-really-mean-it 2024-09-21T10:26:54.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:54 smithi129 ceph-mon[88471]: osdmap e439: 8 total, 8 up, 7 in 2024-09-21T10:26:54.694 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:54.717 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:54.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:54.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:54.840 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:54.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:54.942 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:54.965 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:54.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:55.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:55.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.090 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:55.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:55.191 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:55.216 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:55.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:55.317 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:55.339 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:55.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:55.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:55.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.463 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:55.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:55.547 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:55 smithi107 ceph-mon[102221]: pgmap v1580: 109 pgs: 1 active+remapped, 2 remapped+peering, 106 active+clean; 6.8 MiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 5.5 KiB/s rd, 12 KiB/s wr, 180 op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-21T10:26:55.547 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:55 smithi107 ceph-mon[102221]: osdmap e440: 8 total, 8 up, 7 in 2024-09-21T10:26:55.547 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:55 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:55.547 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:55 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:55.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:55 smithi107 ceph-mon[105022]: pgmap v1580: 109 pgs: 1 active+remapped, 2 remapped+peering, 106 active+clean; 6.8 MiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 5.5 KiB/s rd, 12 KiB/s wr, 180 op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-21T10:26:55.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:55 smithi107 ceph-mon[105022]: osdmap e440: 8 total, 8 up, 7 in 2024-09-21T10:26:55.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:55 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:55.547 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:55 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:55.549 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:26:55.549 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:26:55.565 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:55.566 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:26:55.566 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:26:55.592 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:55.593 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:26:55.593 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:26:55.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:55.609 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:26:55.609 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:26:55.624 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:26:55.624 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:26:55.640 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:26:55.640 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:26:55.655 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:26:55.656 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:26:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:55 smithi129 ceph-mon[88471]: pgmap v1580: 109 pgs: 1 active+remapped, 2 remapped+peering, 106 active+clean; 6.8 MiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 5.5 KiB/s rd, 12 KiB/s wr, 180 op/s; 8.7 KiB/s, 0 objects/s recovering 2024-09-21T10:26:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:55 smithi129 ceph-mon[88471]: osdmap e440: 8 total, 8 up, 7 in 2024-09-21T10:26:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:55 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:55 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:26:55.670 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:26:55.671 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:26:55.694 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:55.716 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:55.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:55.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:55.839 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:55.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:55.941 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:55.962 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:55.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:56.064 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:56.090 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:56.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:56.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:56.212 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:56.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:56.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:56.336 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:56.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:56.438 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:56.464 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:56.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:56.566 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:56.591 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:56.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:56 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/678651612' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:56 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/678651612' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:56 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/678651612' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:56.693 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:56.718 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:56.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:56.722 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 7, 4, 6] out_osds: [3] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:26:56.722 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:26:56.722 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:26:56.722 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.047844 2024-09-21T10:26:56.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 5 0.047844092920687364 2024-09-21T10:26:56.820 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:56.845 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:56.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:56.852 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:26:56.852 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:26:56.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:56.969 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:56.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:57.051 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:57.070 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:57.091 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:57.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:57.193 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:57.219 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:57.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:57.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:57.342 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:57.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:57.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:57.465 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:57.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:57.498 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:57 smithi107 ceph-mon[105022]: pgmap v1582: 109 pgs: 4 peering, 1 active+remapped, 2 remapped+peering, 102 active+clean; 297 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 10 KiB/s rd, 23 KiB/s wr, 105 op/s; 1.5 KiB/s, 0 objects/s recovering 2024-09-21T10:26:57.498 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:57 smithi107 ceph-mon[102221]: pgmap v1582: 109 pgs: 4 peering, 1 active+remapped, 2 remapped+peering, 102 active+clean; 297 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 10 KiB/s rd, 23 KiB/s wr, 105 op/s; 1.5 KiB/s, 0 objects/s recovering 2024-09-21T10:26:57.520 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:57.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:26:57.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:57.591 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:57.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:57 smithi129 ceph-mon[88471]: pgmap v1582: 109 pgs: 4 peering, 1 active+remapped, 2 remapped+peering, 102 active+clean; 297 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 10 KiB/s rd, 23 KiB/s wr, 105 op/s; 1.5 KiB/s, 0 objects/s recovering 2024-09-21T10:26:57.692 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:57.718 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:57.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:57.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:57.841 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:57.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:57.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:57.963 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:57.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:58.065 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:58.091 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:58.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:58.192 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:58.217 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:58.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:58.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:58.342 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:58.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:58.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:26:58.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:58.466 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:58.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:58.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:58.589 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:58.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:58.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:58.712 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:58.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:58.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:58.835 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:58.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:58.937 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:58.963 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:58.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:59.065 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:59.090 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:59.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:59.192 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:59.217 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:59.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:59.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:59.342 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:59.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:59.444 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:59.467 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:59.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:59.569 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:59.593 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:59.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:59.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:26:59 smithi107 ceph-mon[102221]: pgmap v1583: 109 pgs: 4 peering, 105 active+clean; 297 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 7.5 KiB/s rd, 16 KiB/s wr, 76 op/s; 1.1 KiB/s, 0 objects/s recovering 2024-09-21T10:26:59.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:26:59 smithi107 ceph-mon[105022]: pgmap v1583: 109 pgs: 4 peering, 105 active+clean; 297 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 7.5 KiB/s rd, 16 KiB/s wr, 76 op/s; 1.1 KiB/s, 0 objects/s recovering 2024-09-21T10:26:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:26:59 smithi129 ceph-mon[88471]: pgmap v1583: 109 pgs: 4 peering, 105 active+clean; 297 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 7.5 KiB/s rd, 16 KiB/s wr, 76 op/s; 1.1 KiB/s, 0 objects/s recovering 2024-09-21T10:26:59.694 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:59.715 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:59.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:59.817 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:59.840 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:26:59.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:59.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:59.964 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:26:59.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:00.065 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:00.090 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:00.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:00.193 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:00.218 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:00.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:00.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:00.340 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:00.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:00.442 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:00.468 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:00.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:00.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:00.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.593 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:00.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:27:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:27:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:27:00.693 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:00.719 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:00.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:00.820 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:00.846 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:00.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:00.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:00.969 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:00.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:01.070 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:01.096 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:01.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:01.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:01.219 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:01.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:01.244 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:01.245 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:01.249 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:01.321 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:01.348 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:01.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:01.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:01.471 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:01.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:01.507 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:01 smithi107 ceph-mon[102221]: pgmap v1584: 109 pgs: 109 active+clean; 297 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 6.1 KiB/s rd, 13 KiB/s wr, 25 op/s; 901 B/s, 0 objects/s recovering 2024-09-21T10:27:01.507 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:01 smithi107 ceph-mon[105022]: pgmap v1584: 109 pgs: 109 active+clean; 297 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 6.1 KiB/s rd, 13 KiB/s wr, 25 op/s; 901 B/s, 0 objects/s recovering 2024-09-21T10:27:01.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:01.595 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:01.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:01 smithi129 ceph-mon[88471]: pgmap v1584: 109 pgs: 109 active+clean; 297 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 6.1 KiB/s rd, 13 KiB/s wr, 25 op/s; 901 B/s, 0 objects/s recovering 2024-09-21T10:27:01.697 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:01.723 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:01.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:01.825 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:01.850 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:01.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:01.952 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:01.977 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:01.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:02.067 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:27:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:27:01] "GET /metrics HTTP/1.1" 200 36738 "" "Prometheus/2.51.0" 2024-09-21T10:27:02.078 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:02.103 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:02.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:02.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:02.227 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:02.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:02.285 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:27:02.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:02.349 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:02.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:02.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:02 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1470538505' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:27:02.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:02 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:27:02.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:02 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1470538505' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:27:02.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:02 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:27:02.452 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:02.473 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:02.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:02.575 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:02.601 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:02.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:02 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1470538505' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:27:02.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:02 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:27:02.702 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:02.728 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:02.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:02.829 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:02.856 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:02.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:02.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:02.979 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:03.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:03.103 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:03.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:03.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:03.226 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:03.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:03.283 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'unique_pool_9' removed 2024-09-21T10:27:03.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:03.362 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:03.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:03.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:03.472 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:03 smithi107 ceph-mon[105022]: pgmap v1585: 109 pgs: 109 active+clean; 297 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 2.7 KiB/s rd, 6.0 KiB/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:27:03.472 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:03 smithi107 ceph-mon[105022]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:03.472 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:03 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:27:03.472 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:03 smithi107 ceph-mon[105022]: osdmap e441: 8 total, 8 up, 7 in 2024-09-21T10:27:03.472 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:03 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2741348873' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:27:03.472 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:03 smithi107 ceph-mon[102221]: pgmap v1585: 109 pgs: 109 active+clean; 297 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 2.7 KiB/s rd, 6.0 KiB/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:27:03.473 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:03 smithi107 ceph-mon[102221]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:03.473 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:03 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:27:03.473 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:03 smithi107 ceph-mon[102221]: osdmap e441: 8 total, 8 up, 7 in 2024-09-21T10:27:03.473 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:03 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2741348873' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:27:03.487 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:03.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:03.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:03.611 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:03.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:03 smithi129 ceph-mon[88471]: pgmap v1585: 109 pgs: 109 active+clean; 297 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 2.7 KiB/s rd, 6.0 KiB/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:27:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:03 smithi129 ceph-mon[88471]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:03 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:27:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:03 smithi129 ceph-mon[88471]: osdmap e441: 8 total, 8 up, 7 in 2024-09-21T10:27:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:03 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2741348873' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:27:03.713 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:03.734 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:03.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:03.835 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:03.858 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:03.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:03.959 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:03.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.981 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:03.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:04.082 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:04.104 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:04.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:04.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:04.227 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:04.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:04.329 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:04.394 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:04.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:04.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:04.517 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:04.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:04.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:04 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2741348873' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:27:04.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:04 smithi107 ceph-mon[102221]: osdmap e442: 8 total, 8 up, 7 in 2024-09-21T10:27:04.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:04 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2741348873' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:27:04.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:04 smithi107 ceph-mon[105022]: osdmap e442: 8 total, 8 up, 7 in 2024-09-21T10:27:04.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:04.640 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:04.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:04 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2741348873' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_9", "pool2": "unique_pool_9", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:27:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:04 smithi129 ceph-mon[88471]: osdmap e442: 8 total, 8 up, 7 in 2024-09-21T10:27:04.742 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:04.763 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:04.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:04.865 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:04.887 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:04.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:04.988 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:05.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.011 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:05.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:05.112 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:05.133 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:05.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:05.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:05.257 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:05.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:05.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:05.380 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:05.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:05.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:05.504 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:05.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:05.550 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:05 smithi107 ceph-mon[102221]: pgmap v1588: 93 pgs: 93 active+clean; 297 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:05.550 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:05 smithi107 ceph-mon[102221]: 1.4 scrub starts 2024-09-21T10:27:05.551 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:05 smithi107 ceph-mon[105022]: pgmap v1588: 93 pgs: 93 active+clean; 297 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:05.551 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:05 smithi107 ceph-mon[105022]: 1.4 scrub starts 2024-09-21T10:27:05.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:05.626 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:05.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:05.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:27:05.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:05 smithi129 ceph-mon[88471]: pgmap v1588: 93 pgs: 93 active+clean; 297 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:05.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:05 smithi129 ceph-mon[88471]: 1.4 scrub starts 2024-09-21T10:27:05.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:05.750 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:05.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:05.852 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:05.876 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:05.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:05.978 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:06.001 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:06.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:06.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:06.124 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:06.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:06.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:06.248 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:06.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:06.350 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:06.371 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:06.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:06.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:06.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.495 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:06.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:06.596 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:06.621 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:06.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:06.641 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-21T10:27:06.641 INFO:teuthology.task.print:**** done end radosbench.yaml 2024-09-21T10:27:06.641 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-09-21T10:27:06.646 INFO:tasks.workunit:Pulling workunits from ref quincy 2024-09-21T10:27:06.646 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-21T10:27:06.646 DEBUG:teuthology.orchestra.run.smithi107:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-21T10:27:06.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.666 INFO:teuthology.orchestra.run.smithi107.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-21T10:27:06.666 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-21T10:27:06.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:06.748 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-21T10:27:06.748 DEBUG:teuthology.orchestra.run.smithi107:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-21T10:27:06.804 INFO:tasks.workunit:timeout=3h 2024-09-21T10:27:06.804 INFO:tasks.workunit:cleanup=True 2024-09-21T10:27:06.805 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch quincy https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2024-09-21T10:27:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:06.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:06.840 INFO:tasks.workunit.client.0.smithi107.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-21T10:27:06.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:06 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:27:06.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:06 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:27:06.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:06 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:27:06.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:06 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:27:06.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:06 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:27:06.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:06 smithi107 ceph-mon[102221]: pgmap v1589: 93 pgs: 93 active+clean; 297 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:06.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:06 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:27:06.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:06 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:27:06.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:06 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:27:06.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:06 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:27:06.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:06 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:27:06.860 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:06 smithi107 ceph-mon[105022]: pgmap v1589: 93 pgs: 93 active+clean; 297 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:06.910 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:06 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:27:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:06 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:27:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:06 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:27:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:06 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:27:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:06 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:27:06.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:06 smithi129 ceph-mon[88471]: pgmap v1589: 93 pgs: 93 active+clean; 297 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:06.931 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:06.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:07.033 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:07.059 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:07.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:07.161 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:07.187 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:07.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:07.288 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:07.313 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:07.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:07.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:07.436 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:07.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:07.538 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:07.564 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:07.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:07.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:07.688 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:07.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:07.790 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:07.816 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:07.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:07.918 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:07.939 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:07.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:08.041 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:08.067 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:08.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:08.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:07 smithi107 ceph-mon[102221]: 1.4 scrub starts 2024-09-21T10:27:08.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:07 smithi107 ceph-mon[102221]: 1.4 scrub ok 2024-09-21T10:27:08.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:07 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:27:08.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:07 smithi107 ceph-mon[105022]: 1.4 scrub starts 2024-09-21T10:27:08.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:07 smithi107 ceph-mon[105022]: 1.4 scrub ok 2024-09-21T10:27:08.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:07 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:27:08.139 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:08.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:08.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:07 smithi129 ceph-mon[88471]: 1.4 scrub starts 2024-09-21T10:27:08.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:07 smithi129 ceph-mon[88471]: 1.4 scrub ok 2024-09-21T10:27:08.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:07 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:27:08.190 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:08.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:08.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:08.313 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:08.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:08.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:08.437 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:08.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:08.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:08.559 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:08.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:08.661 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:08.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.688 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:08.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:08.789 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:08.814 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:08.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:08.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:08.938 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:08.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:09.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:08 smithi107 ceph-mon[102221]: pgmap v1590: 93 pgs: 93 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:09.030 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:08 smithi107 ceph-mon[105022]: pgmap v1590: 93 pgs: 93 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:09.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:09.060 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:09.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:09.162 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:09.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:08 smithi129 ceph-mon[88471]: pgmap v1590: 93 pgs: 93 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:09.188 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:09.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:09.280 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-09-21T10:27:09.280 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:27:09.289 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:09.311 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:09.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:09.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:09.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.434 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:09.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:09.535 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:09.561 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:09.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:09.662 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:09.684 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:09.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:09.786 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:09.811 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:09.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:09.913 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:09.938 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:10.040 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:10.065 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:10.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:10.166 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:10.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.193 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:10.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:10.294 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:10.319 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:10.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:10.420 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:10.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.447 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:10.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:10.456 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:27:10.514 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:27:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:27:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:27:10.549 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:10.572 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:10.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:10.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:10.697 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:10.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:10.762 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:27:10.798 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:10.819 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:10.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:10.921 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:10.943 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:10.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:10.944 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:10 smithi107 ceph-mon[105022]: pgmap v1591: 93 pgs: 93 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:10.944 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:10 smithi107 ceph-mon[102221]: pgmap v1591: 93 pgs: 93 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:11.044 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:11.066 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:11.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:11.146 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:27:11.146 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:27:11.155 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:27:11.155 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:27:11.169 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:11.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:10 smithi129 ceph-mon[88471]: pgmap v1591: 93 pgs: 93 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:11.172 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:27:11.172 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:27:11.190 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:11.191 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:27:11.191 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:27:11.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:11.209 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:27:11.209 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:27:11.225 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:27:11.225 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:27:11.241 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:27:11.241 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:27:11.256 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:27:11.256 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:27:11.293 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:11.313 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:11.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:11.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:11.436 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:11.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:11.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:11.558 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:11.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:11.659 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:11.681 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:11.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:11.783 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:11.804 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:11.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:11.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:27:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:27:11] "GET /metrics HTTP/1.1" 200 34371 "" "Prometheus/2.51.0" 2024-09-21T10:27:11.905 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:11.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.927 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:11.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:11.982 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:11 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2757116423' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:11.982 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:12.028 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:12.049 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:12.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:12.150 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:11 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2757116423' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:12.150 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:12.150 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:11 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2757116423' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:12.151 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:12.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:12.173 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:12.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:12.274 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:12.296 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:12.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:12.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:12.419 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:12.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:12.520 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:12.535 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:27:12.535 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:27:12.536 INFO:teuthology.orchestra.run.smithi129.stdout: "num_ops": 0 2024-09-21T10:27:12.536 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:27:12.547 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:12.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:12.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:12.670 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:12.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:12.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:12.792 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:12.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:12.876 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.5 primary-affinity to 0.0478441 (831352) 2024-09-21T10:27:12.880 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:27:12.894 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:12.915 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:12.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:13.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:13.037 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:13.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:13.139 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:13.161 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:13.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:12 smithi129 ceph-mon[88471]: pgmap v1592: 93 pgs: 93 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:13.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:12 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1495160627' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.047844092920687364} : dispatch 2024-09-21T10:27:13.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:12 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/490523967' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:27:13.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:12 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:27:13.263 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:13.284 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:13.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:13.324 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:27:13.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:12 smithi107 ceph-mon[102221]: pgmap v1592: 93 pgs: 93 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:13.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:12 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1495160627' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.047844092920687364} : dispatch 2024-09-21T10:27:13.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:12 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/490523967' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:27:13.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:12 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:27:13.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:12 smithi107 ceph-mon[105022]: pgmap v1592: 93 pgs: 93 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:13.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:12 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1495160627' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.047844092920687364} : dispatch 2024-09-21T10:27:13.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:12 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/490523967' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:27:13.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:12 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:27:13.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:13.407 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:13.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:13.508 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:13.530 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:13.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:13.607 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:27:13.632 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:13.652 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:13.712 DEBUG:teuthology.orchestra.run.smithi107:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-21T10:27:13.753 INFO:tasks.workunit.client.0.smithi107.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-21T10:27:13.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:13.755 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T10:27:13.755 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-21T10:27:13.774 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:13.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:13.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:13.899 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:13.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:13.994 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:13 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:27:13.994 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:13 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:27:13.994 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:13 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1495160627' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.047844092920687364}]': finished 2024-09-21T10:27:13.994 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:13 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:27:13.994 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:13 smithi129 ceph-mon[88471]: osdmap e443: 8 total, 8 up, 7 in 2024-09-21T10:27:14.001 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:14.023 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:14.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:14.125 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:14.146 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:14.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:14.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:14.266 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:13 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:27:14.266 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:13 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:27:14.266 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:13 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1495160627' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.047844092920687364}]': finished 2024-09-21T10:27:14.266 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:13 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:27:14.266 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:13 smithi107 ceph-mon[102221]: osdmap e443: 8 total, 8 up, 7 in 2024-09-21T10:27:14.266 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:13 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:27:14.266 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:13 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:27:14.266 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:13 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1495160627' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.047844092920687364}]': finished 2024-09-21T10:27:14.266 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:13 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:27:14.267 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:13 smithi107 ceph-mon[105022]: osdmap e443: 8 total, 8 up, 7 in 2024-09-21T10:27:14.272 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:14.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:14.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:14.395 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:14.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:14.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:14.519 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:14.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:14.642 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:14.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:14.743 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:14.765 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:14.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:14.867 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:14.889 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:14.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:14.991 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:15.012 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:15.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:15.059 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:14 smithi129 ceph-mon[88471]: pgmap v1594: 93 pgs: 93 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:15.059 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:14 smithi129 ceph-mon[88471]: osdmap e444: 8 total, 8 up, 7 in 2024-09-21T10:27:15.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:14 smithi107 ceph-mon[102221]: pgmap v1594: 93 pgs: 93 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:15.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:14 smithi107 ceph-mon[102221]: osdmap e444: 8 total, 8 up, 7 in 2024-09-21T10:27:15.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:14 smithi107 ceph-mon[105022]: pgmap v1594: 93 pgs: 93 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:15.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:14 smithi107 ceph-mon[105022]: osdmap e444: 8 total, 8 up, 7 in 2024-09-21T10:27:15.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:15.135 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:15.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:15.237 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:15.259 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:15.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:15.301 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:27:15.302 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:27:15.302 INFO:teuthology.orchestra.run.smithi129.stdout: "complaint_time": 30, 2024-09-21T10:27:15.302 INFO:teuthology.orchestra.run.smithi129.stdout: "num_blocked_ops": 0 2024-09-21T10:27:15.302 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:27:15.360 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:15.382 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:15.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:15.484 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:15.508 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:15.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:15.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:27:15.609 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:15.630 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:15.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:15.644 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-21T10:27:15.717 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-21T10:27:15.731 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:15.752 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:15.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:15.760 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T10:27:15.762 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T10:27:15.762 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-21T10:27:15.801 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:15.808 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T10:27:15.822 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T10:27:15.822 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-21T10:27:15.854 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:15.863 INFO:tasks.workunit:Running workunits matching cls/test_cls_rbd.sh on client.0... 2024-09-21T10:27:15.864 INFO:tasks.workunit:Running workunit cls/test_cls_rbd.sh... 2024-09-21T10:27:15.864 DEBUG:teuthology.orchestra.run.smithi107:workunit test cls/test_cls_rbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2024-09-21T10:27:15.875 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:15.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:15.964 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:27:15.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:15.997 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:15.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:16.098 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:16.113 INFO:tasks.workunit.client.0.smithi107.stdout:Running main() from gmock_main.cc 2024-09-21T10:27:16.113 INFO:tasks.workunit.client.0.smithi107.stdout:[==========] Running 70 tests from 1 test suite. 2024-09-21T10:27:16.113 INFO:tasks.workunit.client.0.smithi107.stdout:[----------] Global test environment set-up. 2024-09-21T10:27:16.113 INFO:tasks.workunit.client.0.smithi107.stdout:[----------] 70 tests from TestClsRbd 2024-09-21T10:27:16.120 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:16.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:16.210 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:27:16.221 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:16.244 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:16.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:16.346 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:16.367 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:16.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:16.469 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:16.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.491 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:16.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:16.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:16.613 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:16.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:16.715 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:16.738 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:16.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:16.839 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:16.861 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:16.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:16.963 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:16.984 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:16.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:17.014 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:16 smithi129 ceph-mon[88471]: pgmap v1596: 93 pgs: 93 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:17.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:17.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.108 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:17.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:17.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:16 smithi107 ceph-mon[105022]: pgmap v1596: 93 pgs: 93 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:17.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:16 smithi107 ceph-mon[102221]: pgmap v1596: 93 pgs: 93 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:17.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:27:17.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:17.238 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:17.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:17.339 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:17.361 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:17.462 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:17.484 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:17.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:17.586 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:17.606 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:17.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:17.690 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:27:17.691 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:27:17.691 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:27:17.691 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:27:17.705 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:27:17.705 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:27:17.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:17.724 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:27:17.724 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:27:17.728 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:17.731 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:17.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:17.740 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:27:17.740 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:27:17.757 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:27:17.757 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:27:17.773 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:27:17.773 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:27:17.788 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:27:17.788 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:27:17.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:17.851 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:27:17.851 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:27:17.854 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:17.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:17.905 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.get_all_features 2024-09-21T10:27:17.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:17.978 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:17.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:18.027 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:17 smithi107 ceph-mon[102221]: osdmap e445: 8 total, 8 up, 7 in 2024-09-21T10:27:18.027 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:17 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/862312379' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-346632-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:27:18.027 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:17 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-346632-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:27:18.027 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:17 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3447820987' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:18.027 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:17 smithi107 ceph-mon[105022]: osdmap e445: 8 total, 8 up, 7 in 2024-09-21T10:27:18.027 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:17 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/862312379' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-346632-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:27:18.027 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:17 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-346632-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:27:18.028 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:17 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3447820987' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:18.080 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:18.101 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.get_all_features (197 ms) 2024-09-21T10:27:18.101 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.copyup 2024-09-21T10:27:18.102 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:18.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:18.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:17 smithi129 ceph-mon[88471]: osdmap e445: 8 total, 8 up, 7 in 2024-09-21T10:27:18.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:17 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/862312379' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-346632-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:27:18.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:17 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-346632-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:27:18.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:17 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3447820987' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:18.203 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:18.225 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:18.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:18.299 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.copyup (197 ms) 2024-09-21T10:27:18.299 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.sparse_copyup 2024-09-21T10:27:18.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:18.338 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.sparse_copyup (40 ms) 2024-09-21T10:27:18.338 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.get_and_set_id 2024-09-21T10:27:18.345 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.get_and_set_id (6 ms) 2024-09-21T10:27:18.345 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.add_remove_child 2024-09-21T10:27:18.350 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:18.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:18.355 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.add_remove_child (11 ms) 2024-09-21T10:27:18.355 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.directory_methods 2024-09-21T10:27:18.381 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.directory_methods (25 ms) 2024-09-21T10:27:18.381 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.create 2024-09-21T10:27:18.408 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.create (28 ms) 2024-09-21T10:27:18.408 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.get_features 2024-09-21T10:27:18.410 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.get_features (2 ms) 2024-09-21T10:27:18.410 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.get_object_prefix 2024-09-21T10:27:18.411 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.get_object_prefix (1 ms) 2024-09-21T10:27:18.411 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.get_create_timestamp 2024-09-21T10:27:18.414 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.get_create_timestamp (3 ms) 2024-09-21T10:27:18.414 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.get_access_timestamp 2024-09-21T10:27:18.415 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.get_access_timestamp (1 ms) 2024-09-21T10:27:18.415 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.get_modify_timestamp 2024-09-21T10:27:18.417 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.get_modify_timestamp (1 ms) 2024-09-21T10:27:18.417 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.get_data_pool 2024-09-21T10:27:18.420 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.get_data_pool (4 ms) 2024-09-21T10:27:18.420 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.get_size 2024-09-21T10:27:18.424 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.get_size (4 ms) 2024-09-21T10:27:18.424 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.set_size 2024-09-21T10:27:18.428 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.set_size (4 ms) 2024-09-21T10:27:18.428 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.protection_status 2024-09-21T10:27:18.444 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.protection_status (16 ms) 2024-09-21T10:27:18.444 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.snapshot_limits 2024-09-21T10:27:18.452 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:18.452 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.snapshot_limits (8 ms) 2024-09-21T10:27:18.452 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.parents_v1 2024-09-21T10:27:18.474 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:18.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:18.498 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.parents_v1 (46 ms) 2024-09-21T10:27:18.498 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.parents_v2 2024-09-21T10:27:18.530 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.parents_v2 (31 ms) 2024-09-21T10:27:18.530 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.snapshots 2024-09-21T10:27:18.534 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:27:18.549 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.snapshots (20 ms) 2024-09-21T10:27:18.549 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.snapid_race 2024-09-21T10:27:18.557 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.snapid_race (8 ms) 2024-09-21T10:27:18.557 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.stripingv2 2024-09-21T10:27:18.565 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.stripingv2 (8 ms) 2024-09-21T10:27:18.565 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.object_map_save 2024-09-21T10:27:18.571 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.object_map_save (6 ms) 2024-09-21T10:27:18.571 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.object_map_resize 2024-09-21T10:27:18.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:18.579 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.object_map_resize (7 ms) 2024-09-21T10:27:18.579 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.object_map_update 2024-09-21T10:27:18.585 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.object_map_update (7 ms) 2024-09-21T10:27:18.586 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.object_map_load_enoent 2024-09-21T10:27:18.586 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.object_map_load_enoent (0 ms) 2024-09-21T10:27:18.586 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.object_map_snap_add 2024-09-21T10:27:18.592 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.object_map_snap_add (7 ms) 2024-09-21T10:27:18.592 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.object_map_snap_remove 2024-09-21T10:27:18.598 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:18.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:18.599 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.object_map_snap_remove (7 ms) 2024-09-21T10:27:18.599 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.flags 2024-09-21T10:27:18.603 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.flags (4 ms) 2024-09-21T10:27:18.604 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.metadata 2024-09-21T10:27:18.627 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.metadata (24 ms) 2024-09-21T10:27:18.627 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.set_features 2024-09-21T10:27:18.633 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.set_features (6 ms) 2024-09-21T10:27:18.634 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.mirror 2024-09-21T10:27:18.665 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.mirror (32 ms) 2024-09-21T10:27:18.665 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.mirror_image 2024-09-21T10:27:18.679 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.mirror_image (14 ms) 2024-09-21T10:27:18.679 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.mirror_image_status 2024-09-21T10:27:18.699 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:18.723 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:18.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:18.794 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:27:18.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:18.848 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:18.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:18.950 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:18.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.972 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:18.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:19.073 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:19.096 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:19.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:19.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:18 smithi129 ceph-mon[88471]: pgmap v1598: 125 pgs: 16 creating+peering, 16 unknown, 93 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:19.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:18 smithi129 ceph-mon[88471]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:27:19.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:18 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-346632-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:27:19.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:18 smithi129 ceph-mon[88471]: osdmap e446: 8 total, 8 up, 7 in 2024-09-21T10:27:19.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:19.224 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:19.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:19.290 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 7, 4, 6] out_osds: [3] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T10:27:19.291 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:27:19.291 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-09-21T10:27:19.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd in 3 2024-09-21T10:27:19.316 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:18 smithi107 ceph-mon[102221]: pgmap v1598: 125 pgs: 16 creating+peering, 16 unknown, 93 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:19.316 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:18 smithi107 ceph-mon[102221]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:27:19.316 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:18 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-346632-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:27:19.316 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:18 smithi107 ceph-mon[102221]: osdmap e446: 8 total, 8 up, 7 in 2024-09-21T10:27:19.317 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:18 smithi107 ceph-mon[105022]: pgmap v1598: 125 pgs: 16 creating+peering, 16 unknown, 93 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail 2024-09-21T10:27:19.317 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:18 smithi107 ceph-mon[105022]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:27:19.317 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:18 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi107-346632-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:27:19.317 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:18 smithi107 ceph-mon[105022]: osdmap e446: 8 total, 8 up, 7 in 2024-09-21T10:27:19.326 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:19.348 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:19.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:19.449 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:19.470 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:19.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:19.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:19.596 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:19.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:19.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:19.721 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:19.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:19.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:19.846 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:19.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:19.947 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:19.968 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:19.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:19.998 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:19 smithi129 ceph-mon[88471]: osdmap e447: 8 total, 8 up, 7 in 2024-09-21T10:27:20.070 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:20.092 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:20.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:20.172 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:20.193 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:20.214 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:20.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:20.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:19 smithi107 ceph-mon[102221]: osdmap e447: 8 total, 8 up, 7 in 2024-09-21T10:27:20.298 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:19 smithi107 ceph-mon[105022]: osdmap e447: 8 total, 8 up, 7 in 2024-09-21T10:27:20.316 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:20.320 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:27:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:27:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:27:20.340 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:20.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:20.473 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:20.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:20.575 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:20.578 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:27:20.578 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:27:20.601 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:20.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:20.702 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:20.723 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:20.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:20.825 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:20.847 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:20.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:20.948 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:20.970 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:20.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:21.072 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:21.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.093 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:21.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:21.133 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.mirror_image_status (2453 ms) 2024-09-21T10:27:21.133 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.mirror_image_map 2024-09-21T10:27:21.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:20 smithi129 ceph-mon[88471]: pgmap v1601: 125 pgs: 16 creating+peering, 109 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 107 KiB/s rd, 29 KiB/s wr, 144 op/s 2024-09-21T10:27:21.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:20 smithi129 ceph-mon[88471]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:27:21.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:20 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:27:21.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:21.214 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.mirror_image_map (81 ms) 2024-09-21T10:27:21.214 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.mirror_instances 2024-09-21T10:27:21.219 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:21.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:21.222 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.mirror_instances (9 ms) 2024-09-21T10:27:21.222 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.mirror_snapshot 2024-09-21T10:27:21.262 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.mirror_snapshot (40 ms) 2024-09-21T10:27:21.262 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.group_dir_list 2024-09-21T10:27:21.264 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.group_dir_list (2 ms) 2024-09-21T10:27:21.264 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.group_dir_add 2024-09-21T10:27:21.267 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.group_dir_add (2 ms) 2024-09-21T10:27:21.267 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.dir_add_already_existing 2024-09-21T10:27:21.270 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.dir_add_already_existing (4 ms) 2024-09-21T10:27:21.270 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.group_dir_rename 2024-09-21T10:27:21.273 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:20 smithi107 ceph-mon[102221]: pgmap v1601: 125 pgs: 16 creating+peering, 109 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 107 KiB/s rd, 29 KiB/s wr, 144 op/s 2024-09-21T10:27:21.273 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:20 smithi107 ceph-mon[102221]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:27:21.273 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:20 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:27:21.273 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:20 smithi107 ceph-mon[105022]: pgmap v1601: 125 pgs: 16 creating+peering, 109 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 107 KiB/s rd, 29 KiB/s wr, 144 op/s 2024-09-21T10:27:21.274 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:20 smithi107 ceph-mon[105022]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:27:21.274 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:20 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:27:21.275 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.group_dir_rename (5 ms) 2024-09-21T10:27:21.276 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.group_dir_remove 2024-09-21T10:27:21.279 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.group_dir_remove (3 ms) 2024-09-21T10:27:21.279 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.group_dir_remove_missing 2024-09-21T10:27:21.282 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.group_dir_remove_missing (4 ms) 2024-09-21T10:27:21.283 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.group_image_add 2024-09-21T10:27:21.285 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.group_image_add (2 ms) 2024-09-21T10:27:21.285 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.group_image_remove 2024-09-21T10:27:21.287 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:27:21.288 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.group_image_remove (4 ms) 2024-09-21T10:27:21.289 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.group_image_list 2024-09-21T10:27:21.291 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.group_image_list (3 ms) 2024-09-21T10:27:21.291 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.group_image_clean 2024-09-21T10:27:21.296 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.group_image_clean (5 ms) 2024-09-21T10:27:21.296 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.image_group_add 2024-09-21T10:27:21.298 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.image_group_add (2 ms) 2024-09-21T10:27:21.298 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.image_group_remove 2024-09-21T10:27:21.301 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.image_group_remove (3 ms) 2024-09-21T10:27:21.301 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.image_group_get 2024-09-21T10:27:21.304 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.image_group_get (3 ms) 2024-09-21T10:27:21.304 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_name 2024-09-21T10:27:21.305 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.group_snap_set_empty_name (1 ms) 2024-09-21T10:27:21.306 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_id 2024-09-21T10:27:21.307 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.group_snap_set_empty_id (2 ms) 2024-09-21T10:27:21.307 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_id 2024-09-21T10:27:21.311 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_id (3 ms) 2024-09-21T10:27:21.311 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_name 2024-09-21T10:27:21.313 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_name (3 ms) 2024-09-21T10:27:21.314 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.group_snap_set 2024-09-21T10:27:21.316 INFO:tasks.workunit.client.0.smithi107.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.7-1649-gfbd1a6ab/rpm/el9/BUILD/ceph-17.2.7-1649-gfbd1a6ab/src/test/cls_rbd/test_cls_rbd.cc:2681: Failure 2024-09-21T10:27:21.316 INFO:tasks.workunit.client.0.smithi107.stdout:Expected equality of these values: 2024-09-21T10:27:21.316 INFO:tasks.workunit.client.0.smithi107.stdout: 1U 2024-09-21T10:27:21.316 INFO:tasks.workunit.client.0.smithi107.stdout: Which is: 1 2024-09-21T10:27:21.316 INFO:tasks.workunit.client.0.smithi107.stdout: keys.size() 2024-09-21T10:27:21.316 INFO:tasks.workunit.client.0.smithi107.stdout: Which is: 3 2024-09-21T10:27:21.317 INFO:tasks.workunit.client.0.smithi107.stdout:[ FAILED ] TestClsRbd.group_snap_set (3 ms) 2024-09-21T10:27:21.317 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.group_snap_list 2024-09-21T10:27:21.319 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.group_snap_list (3 ms) 2024-09-21T10:27:21.319 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.group_snap_list_max_return 2024-09-21T10:27:21.321 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:21.338 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.group_snap_list_max_return (19 ms) 2024-09-21T10:27:21.338 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.group_snap_list_max_read 2024-09-21T10:27:21.352 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:21.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:21.445 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:27:21.454 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:21.513 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:21.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:21.538 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.group_snap_list_max_read (200 ms) 2024-09-21T10:27:21.538 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.group_snap_remove 2024-09-21T10:27:21.541 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.group_snap_remove (3 ms) 2024-09-21T10:27:21.541 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.group_snap_get_by_id 2024-09-21T10:27:21.543 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.group_snap_get_by_id (2 ms) 2024-09-21T10:27:21.543 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.trash_methods 2024-09-21T10:27:21.551 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.trash_methods (7 ms) 2024-09-21T10:27:21.551 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.op_features 2024-09-21T10:27:21.559 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.op_features (9 ms) 2024-09-21T10:27:21.559 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.clone_parent 2024-09-21T10:27:21.565 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:27:21.585 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.clone_parent (26 ms) 2024-09-21T10:27:21.585 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.clone_parent_ns 2024-09-21T10:27:21.594 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.clone_parent_ns (9 ms) 2024-09-21T10:27:21.594 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.clone_child 2024-09-21T10:27:21.607 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.clone_child (13 ms) 2024-09-21T10:27:21.607 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.namespace_methods 2024-09-21T10:27:21.614 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.namespace_methods (7 ms) 2024-09-21T10:27:21.614 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.migration 2024-09-21T10:27:21.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:21.625 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.migration (11 ms) 2024-09-21T10:27:21.625 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.migration_v1 2024-09-21T10:27:21.639 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:21.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:21.641 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.migration_v1 (15 ms) 2024-09-21T10:27:21.641 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.assert_snapc_seq 2024-09-21T10:27:21.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:21.762 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:21.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:21.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:27:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:27:21] "GET /metrics HTTP/1.1" 200 34371 "" "Prometheus/2.51.0" 2024-09-21T10:27:21.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:21.886 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:21.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:21.926 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is set 2024-09-21T10:27:21.954 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:21 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4234785070' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:27:21.954 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:21 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:27:21.988 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:22.010 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:22.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:22.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:22.134 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:22.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:22.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:22.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:21 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4234785070' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:27:22.241 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:21 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:27:22.241 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:21 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4234785070' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:27:22.241 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:21 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:27:22.257 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:22.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:22.358 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:22.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.381 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:22.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:22.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:22.503 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:22.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:22.605 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:22.626 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:22.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:22.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:22.749 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:22.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:22.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:22.872 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:22.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:22.926 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.assert_snapc_seq (1286 ms) 2024-09-21T10:27:22.927 INFO:tasks.workunit.client.0.smithi107.stdout:[ RUN ] TestClsRbd.sparsify 2024-09-21T10:27:22.955 INFO:tasks.workunit.client.0.smithi107.stdout:[ OK ] TestClsRbd.sparsify (29 ms) 2024-09-21T10:27:22.964 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:22 smithi129 ceph-mon[88471]: pgmap v1602: 125 pgs: 16 creating+peering, 109 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 102 KiB/s rd, 28 KiB/s wr, 138 op/s 2024-09-21T10:27:22.964 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:22 smithi129 ceph-mon[88471]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:22.964 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:22 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:27:22.964 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:22 smithi129 ceph-mon[88471]: osdmap e448: 8 total, 8 up, 7 in 2024-09-21T10:27:22.974 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:22.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.997 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:22.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:23.086 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:23.098 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:23.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:22 smithi107 ceph-mon[102221]: pgmap v1602: 125 pgs: 16 creating+peering, 109 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 102 KiB/s rd, 28 KiB/s wr, 138 op/s 2024-09-21T10:27:23.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:22 smithi107 ceph-mon[102221]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:23.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:22 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:27:23.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:22 smithi107 ceph-mon[102221]: osdmap e448: 8 total, 8 up, 7 in 2024-09-21T10:27:23.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:22 smithi107 ceph-mon[105022]: pgmap v1602: 125 pgs: 16 creating+peering, 109 active+clean; 279 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 102 KiB/s rd, 28 KiB/s wr, 138 op/s 2024-09-21T10:27:23.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:22 smithi107 ceph-mon[105022]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:23.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:22 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:27:23.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:22 smithi107 ceph-mon[105022]: osdmap e448: 8 total, 8 up, 7 in 2024-09-21T10:27:23.119 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:23.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:23.221 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:23.242 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:23.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:23.304 INFO:teuthology.orchestra.run.smithi129.stdout:ERROR: (22) Invalid argument 2024-09-21T10:27:23.304 INFO:teuthology.orchestra.run.smithi129.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:27:23.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:23.365 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:23.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:23.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:23.488 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:23.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:23.523 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:23 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a[102198]: 2024-09-21T10:27:23.246+0000 7f3189124640 -1 mon.a@0(leader).osd e449 definitely_dead 0 2024-09-21T10:27:23.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:23.612 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:23.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:23.713 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:23.735 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:23.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:23.837 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:23.858 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:23.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:23.930 INFO:teuthology.orchestra.run.smithi107.stderr:marked in osd.3. 2024-09-21T10:27:23.934 INFO:tasks.workunit.client.0.smithi107.stdout:[----------] 70 tests from TestClsRbd (5051 ms total) 2024-09-21T10:27:23.934 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-09-21T10:27:23.934 INFO:tasks.workunit.client.0.smithi107.stdout:[----------] Global test environment tear-down 2024-09-21T10:27:23.934 INFO:tasks.workunit.client.0.smithi107.stdout:[==========] 70 tests from 1 test suite ran. (7820 ms total) 2024-09-21T10:27:23.934 INFO:tasks.workunit.client.0.smithi107.stdout:[ PASSED ] 69 tests. 2024-09-21T10:27:23.934 INFO:tasks.workunit.client.0.smithi107.stdout:[ FAILED ] 1 test, listed below: 2024-09-21T10:27:23.934 INFO:tasks.workunit.client.0.smithi107.stdout:[ FAILED ] TestClsRbd.group_snap_set 2024-09-21T10:27:23.934 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-09-21T10:27:23.934 INFO:tasks.workunit.client.0.smithi107.stdout: 1 FAILED TEST 2024-09-21T10:27:23.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.945 INFO:tasks.workunit:Stopping ['cls/test_cls_rbd.sh'] on client.0... 2024-09-21T10:27:23.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-21T10:27:23.960 DEBUG:teuthology.orchestra.run.smithi129:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:23.982 INFO:teuthology.orchestra.run.smithi129.stderr:sudo: docker: command not found 2024-09-21T10:27:23.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi129 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:24.084 DEBUG:teuthology.orchestra.run.smithi107:> sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:24.106 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: docker: command not found 2024-09-21T10:27:24.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:24.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi107 with status 1: 'sudo docker kill -s 1 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:24.163 INFO:tasks.thrashosds:joining thrashosds 2024-09-21T10:27:24.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:23 smithi129 ceph-mon[88471]: osdmap e449: 8 total, 8 up, 7 in 2024-09-21T10:27:24.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:23 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2857179208' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-21T10:27:24.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:23 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-21T10:27:24.337 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:27:24.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:23 smithi107 ceph-mon[102221]: osdmap e449: 8 total, 8 up, 7 in 2024-09-21T10:27:24.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:23 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2857179208' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-21T10:27:24.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:23 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-21T10:27:24.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:23 smithi107 ceph-mon[105022]: osdmap e449: 8 total, 8 up, 7 in 2024-09-21T10:27:24.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:23 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2857179208' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-21T10:27:24.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:23 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["3"]} : dispatch 2024-09-21T10:27:24.583 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:27:24.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:27:24.961 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:24 smithi129 ceph-mon[88471]: pgmap v1605: 125 pgs: 125 active+clean; 280 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 1.8 MiB/s rd, 787 KiB/s wr, 321 op/s 2024-09-21T10:27:24.961 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:24 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-21T10:27:24.962 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:24 smithi129 ceph-mon[88471]: osdmap e450: 8 total, 8 up, 8 in 2024-09-21T10:27:25.337 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:24 smithi107 ceph-mon[102221]: pgmap v1605: 125 pgs: 125 active+clean; 280 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 1.8 MiB/s rd, 787 KiB/s wr, 321 op/s 2024-09-21T10:27:25.337 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:24 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-21T10:27:25.337 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:24 smithi107 ceph-mon[102221]: osdmap e450: 8 total, 8 up, 8 in 2024-09-21T10:27:25.337 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:24 smithi107 ceph-mon[105022]: pgmap v1605: 125 pgs: 125 active+clean; 280 KiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 1.8 MiB/s rd, 787 KiB/s wr, 321 op/s 2024-09-21T10:27:25.337 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:24 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-21T10:27:25.337 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:24 smithi107 ceph-mon[105022]: osdmap e450: 8 total, 8 up, 8 in 2024-09-21T10:27:25.362 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-09-21T10:27:25.841 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0: {} 2024-09-21T10:27:25.841 INFO:teuthology.orchestra.run.smithi107.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:27:25.847 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1: {} 2024-09-21T10:27:25.847 INFO:teuthology.orchestra.run.smithi107.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:27:25.863 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2: {} 2024-09-21T10:27:25.863 INFO:teuthology.orchestra.run.smithi107.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:27:25.884 INFO:teuthology.orchestra.run.smithi107.stdout:osd.3: {} 2024-09-21T10:27:25.884 INFO:teuthology.orchestra.run.smithi107.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:27:25.900 INFO:teuthology.orchestra.run.smithi107.stdout:osd.4: {} 2024-09-21T10:27:25.901 INFO:teuthology.orchestra.run.smithi107.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:27:25.913 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:25.917 INFO:teuthology.orchestra.run.smithi107.stdout:osd.5: {} 2024-09-21T10:27:25.917 INFO:teuthology.orchestra.run.smithi107.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:27:25.942 INFO:teuthology.orchestra.run.smithi107.stdout:osd.6: {} 2024-09-21T10:27:25.943 INFO:teuthology.orchestra.run.smithi107.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:27:25.959 INFO:teuthology.orchestra.run.smithi107.stdout:osd.7: {} 2024-09-21T10:27:25.959 INFO:teuthology.orchestra.run.smithi107.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:27:26.000 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:25 smithi129 ceph-mon[88471]: osdmap e451: 8 total, 8 up, 8 in 2024-09-21T10:27:26.001 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:25 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1751093168' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:26.185 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:25 smithi107 ceph-mon[102221]: osdmap e451: 8 total, 8 up, 8 in 2024-09-21T10:27:26.185 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:25 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1751093168' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:26.186 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:25 smithi107 ceph-mon[105022]: osdmap e451: 8 total, 8 up, 8 in 2024-09-21T10:27:26.186 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:25 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1751093168' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:26.240 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:27:26.240 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 20, 2024-09-21T10:27:26.240 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 600, 2024-09-21T10:27:26.240 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:27:26.240 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.241 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.37050.0:72869 13.c e373/367 13:36cbdb7c:::benchmark_data_smithi107_278655_object72868:head v 373'4510, mlcod=373'4508)", 2024-09-21T10:27:26.241 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.804929+0000", 2024-09-21T10:27:26.241 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 505.43193766500002, 2024-09-21T10:27:26.241 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.072342949000000004, 2024-09-21T10:27:26.241 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:26.241 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:26.241 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:26.241 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.241 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:26.241 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.804929+0000", 2024-09-21T10:27:26.241 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.241 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.241 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.242 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:26.242 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.804929+0000", 2024-09-21T10:27:26.242 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.242 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.242 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.242 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:26.242 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.804928+0000", 2024-09-21T10:27:26.242 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:26.242 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.242 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.242 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:26.242 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.806241+0000", 2024-09-21T10:27:26.242 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001313102 2024-09-21T10:27:26.242 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.242 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.243 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:26.243 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.806254+0000", 2024-09-21T10:27:26.243 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2791e-05 2024-09-21T10:27:26.243 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.243 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.243 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:26.243 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.806259+0000", 2024-09-21T10:27:26.243 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7570000000000001e-06 2024-09-21T10:27:26.243 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.243 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.243 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:26.243 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.806316+0000", 2024-09-21T10:27:26.243 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6900000000000001e-05 2024-09-21T10:27:26.243 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.244 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.244 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:26.244 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.806334+0000", 2024-09-21T10:27:26.244 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.7657999999999999e-05 2024-09-21T10:27:26.244 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.244 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.244 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:26.244 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.877152+0000", 2024-09-21T10:27:26.244 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.070817762000000006 2024-09-21T10:27:26.244 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.244 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.244 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:26.244 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.877272+0000", 2024-09-21T10:27:26.244 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000120127 2024-09-21T10:27:26.244 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.244 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.245 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.245 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.245 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.245 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.37050.0:72871 13.c e373/367 13:3d6a49e5:::benchmark_data_smithi107_278655_object72870:head v 373'4511, mlcod=373'4508)", 2024-09-21T10:27:26.245 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.807495+0000", 2024-09-21T10:27:26.245 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 505.42937126499999, 2024-09-21T10:27:26.245 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.069809948999999996, 2024-09-21T10:27:26.245 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:26.245 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:26.245 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:26.245 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.245 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:26.245 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.807495+0000", 2024-09-21T10:27:26.245 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.245 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.246 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.246 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:26.246 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.807495+0000", 2024-09-21T10:27:26.246 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.246 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.246 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.246 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:26.246 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.807494+0000", 2024-09-21T10:27:26.246 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:26.246 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.246 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.246 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:26.246 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.807584+0000", 2024-09-21T10:27:26.246 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.9843e-05 2024-09-21T10:27:26.247 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.247 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.247 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:26.247 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.807597+0000", 2024-09-21T10:27:26.247 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.259e-05 2024-09-21T10:27:26.247 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.247 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.247 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:26.247 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.807605+0000", 2024-09-21T10:27:26.247 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.5120000000000008e-06 2024-09-21T10:27:26.247 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.247 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.247 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:26.247 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.808879+0000", 2024-09-21T10:27:26.247 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0012737829999999999 2024-09-21T10:27:26.247 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.248 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.248 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:26.248 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.808898+0000", 2024-09-21T10:27:26.248 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.8774e-05 2024-09-21T10:27:26.248 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.248 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.248 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:26.248 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.877283+0000", 2024-09-21T10:27:26.248 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.068385647999999993 2024-09-21T10:27:26.248 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.248 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.248 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:26.248 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.877305+0000", 2024-09-21T10:27:26.248 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1739e-05 2024-09-21T10:27:26.249 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.249 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.249 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.249 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.249 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.249 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.37050.0:72874 13.c e373/367 13:34cb7899:::benchmark_data_smithi107_278655_object72873:head v 373'4512, mlcod=373'4508)", 2024-09-21T10:27:26.249 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.809956+0000", 2024-09-21T10:27:26.249 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 505.42691016700002, 2024-09-21T10:27:26.249 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.067378930000000004, 2024-09-21T10:27:26.249 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:26.249 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:26.249 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:26.249 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.250 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:26.250 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.809956+0000", 2024-09-21T10:27:26.250 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.250 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.250 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.250 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:26.250 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.809956+0000", 2024-09-21T10:27:26.250 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.250 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.250 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.250 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:26.250 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.809955+0000", 2024-09-21T10:27:26.250 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:26.250 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.250 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.809996+0000", 2024-09-21T10:27:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0368999999999999e-05 2024-09-21T10:27:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.810008+0000", 2024-09-21T10:27:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1809e-05 2024-09-21T10:27:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.810013+0000", 2024-09-21T10:27:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6659999999999996e-06 2024-09-21T10:27:26.251 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.814447+0000", 2024-09-21T10:27:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0044332299999999998 2024-09-21T10:27:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.814467+0000", 2024-09-21T10:27:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.0176000000000001e-05 2024-09-21T10:27:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.877308+0000", 2024-09-21T10:27:26.252 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.062841062000000003 2024-09-21T10:27:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.877335+0000", 2024-09-21T10:27:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7458e-05 2024-09-21T10:27:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.37050.0:72882 13.b e373/367 13:dc94c9ec:::benchmark_data_smithi107_278655_object72881:head v 373'4524, mlcod=373'4522)", 2024-09-21T10:27:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.815133+0000", 2024-09-21T10:27:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 505.421733485, 2024-09-21T10:27:26.253 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.066989959000000002, 2024-09-21T10:27:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.815133+0000", 2024-09-21T10:27:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.815133+0000", 2024-09-21T10:27:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.254 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.815132+0000", 2024-09-21T10:27:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.815182+0000", 2024-09-21T10:27:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0056999999999999e-05 2024-09-21T10:27:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.815194+0000", 2024-09-21T10:27:26.255 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1736000000000001e-05 2024-09-21T10:27:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.815199+0000", 2024-09-21T10:27:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5849999999999999e-06 2024-09-21T10:27:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.877198+0000", 2024-09-21T10:27:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.061998596000000003 2024-09-21T10:27:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:26.256 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.877247+0000", 2024-09-21T10:27:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9166000000000003e-05 2024-09-21T10:27:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.882077+0000", 2024-09-21T10:27:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0048300039999999997 2024-09-21T10:27:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.882123+0000", 2024-09-21T10:27:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5855000000000001e-05 2024-09-21T10:27:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.257 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.37050.0:72883 13.c e373/367 13:3285c635:::benchmark_data_smithi107_278655_object72882:head v 373'4513, mlcod=373'4508)", 2024-09-21T10:27:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.815238+0000", 2024-09-21T10:27:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 505.42162798800001, 2024-09-21T10:27:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.066226855000000001, 2024-09-21T10:27:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.815238+0000", 2024-09-21T10:27:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.258 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.815238+0000", 2024-09-21T10:27:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.815238+0000", 2024-09-21T10:27:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:27:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.815268+0000", 2024-09-21T10:27:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0504000000000001e-05 2024-09-21T10:27:26.259 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.877145+0000", 2024-09-21T10:27:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.061876392000000002 2024-09-21T10:27:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.877169+0000", 2024-09-21T10:27:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.4029999999999999e-05 2024-09-21T10:27:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.877358+0000", 2024-09-21T10:27:26.260 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00018900199999999999 2024-09-21T10:27:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.877378+0000", 2024-09-21T10:27:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.0228999999999999e-05 2024-09-21T10:27:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.881429+0000", 2024-09-21T10:27:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0040513220000000004 2024-09-21T10:27:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:26.261 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.881465+0000", 2024-09-21T10:27:26.262 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5871000000000003e-05 2024-09-21T10:27:26.262 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.262 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.262 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.262 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.262 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.262 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.37050.0:72884 13.e e373/367 13:79af2a0c:::benchmark_data_smithi107_278655_object72883:head v 373'4480, mlcod=373'4477)", 2024-09-21T10:27:26.262 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.815317+0000", 2024-09-21T10:27:26.262 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 505.42154935899998, 2024-09-21T10:27:26.262 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.076697507999999998, 2024-09-21T10:27:26.262 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:26.262 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:26.262 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:26.262 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.263 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:26.263 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.815317+0000", 2024-09-21T10:27:26.263 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.263 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.263 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.263 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:26.263 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.815317+0000", 2024-09-21T10:27:26.263 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.263 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.263 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.263 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:26.263 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.815316+0000", 2024-09-21T10:27:26.263 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:26.263 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.263 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.264 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:26.264 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.815341+0000", 2024-09-21T10:27:26.264 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.4769e-05 2024-09-21T10:27:26.264 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.264 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.264 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:26.264 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.877136+0000", 2024-09-21T10:27:26.264 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.061795029000000001 2024-09-21T10:27:26.264 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.264 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.264 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:26.264 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.877167+0000", 2024-09-21T10:27:26.264 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0843000000000003e-05 2024-09-21T10:27:26.265 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.265 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.265 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:26.265 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.877271+0000", 2024-09-21T10:27:26.265 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000104156 2024-09-21T10:27:26.265 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.265 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.265 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:26.265 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.877290+0000", 2024-09-21T10:27:26.265 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.9341999999999999e-05 2024-09-21T10:27:26.265 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.265 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.265 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:26.265 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.891912+0000", 2024-09-21T10:27:26.265 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.014621492 2024-09-21T10:27:26.266 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.266 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.266 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:26.266 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.892015+0000", 2024-09-21T10:27:26.266 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000102629 2024-09-21T10:27:26.266 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.266 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.266 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.266 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.266 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.266 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.37050.0:72878 13.6 e373/367 13:6262de9a:::benchmark_data_smithi107_278655_object72877:head v 373'4493, mlcod=373'4490)", 2024-09-21T10:27:26.266 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.816103+0000", 2024-09-21T10:27:26.266 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 505.42076378000002, 2024-09-21T10:27:26.267 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.078863886999999994, 2024-09-21T10:27:26.267 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:26.267 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:26.267 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:26.267 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.267 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:26.267 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.816103+0000", 2024-09-21T10:27:26.267 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.267 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.267 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.267 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:26.267 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.816103+0000", 2024-09-21T10:27:26.267 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.267 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.268 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.268 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:26.269 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.816097+0000", 2024-09-21T10:27:26.269 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999948 2024-09-21T10:27:26.269 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.269 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.269 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:26.269 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.816173+0000", 2024-09-21T10:27:26.269 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.5779999999999996e-05 2024-09-21T10:27:26.269 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.269 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.269 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:26.269 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.877141+0000", 2024-09-21T10:27:26.269 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.060967462 2024-09-21T10:27:26.269 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.270 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.270 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:26.270 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.877168+0000", 2024-09-21T10:27:26.270 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7453e-05 2024-09-21T10:27:26.270 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.270 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.270 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:26.270 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.877353+0000", 2024-09-21T10:27:26.270 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00018478799999999999 2024-09-21T10:27:26.270 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.270 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.270 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:26.270 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.877371+0000", 2024-09-21T10:27:26.270 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.7691999999999999e-05 2024-09-21T10:27:26.270 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.271 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.271 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:26.271 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.894903+0000", 2024-09-21T10:27:26.271 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.017532774000000001 2024-09-21T10:27:26.271 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.271 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.271 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:26.271 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.894967+0000", 2024-09-21T10:27:26.271 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3157000000000005e-05 2024-09-21T10:27:26.271 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.271 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.271 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.271 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.271 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.272 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.37050.0:174613 13.6 e386/367 13:644b7fdf:::benchmark_data_smithi107_278655_object46706:head v 386'10871, mlcod=386'10868)", 2024-09-21T10:27:26.272 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:46.294257+0000", 2024-09-21T10:27:26.272 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 399.94260939200001, 2024-09-21T10:27:26.272 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.17422923700000001, 2024-09-21T10:27:26.272 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:26.272 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:26.272 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:26.272 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.272 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:26.272 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.294257+0000", 2024-09-21T10:27:26.272 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.272 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.272 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.272 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:26.273 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.294257+0000", 2024-09-21T10:27:26.273 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.273 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.273 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.273 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:26.273 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.294256+0000", 2024-09-21T10:27:26.273 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:26.273 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.273 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.273 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:26.273 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.294265+0000", 2024-09-21T10:27:26.273 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.8519999999999993e-06 2024-09-21T10:27:26.273 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.273 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.273 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:26.274 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.294277+0000", 2024-09-21T10:27:26.274 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2398000000000001e-05 2024-09-21T10:27:26.274 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.274 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.274 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:26.274 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.294283+0000", 2024-09-21T10:27:26.274 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1410000000000002e-06 2024-09-21T10:27:26.274 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.274 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.274 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:26.274 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.294328+0000", 2024-09-21T10:27:26.274 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4558000000000003e-05 2024-09-21T10:27:26.274 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.274 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.274 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:26.275 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.294347+0000", 2024-09-21T10:27:26.275 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.8783999999999998e-05 2024-09-21T10:27:26.275 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.275 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.275 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:26.275 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468356+0000", 2024-09-21T10:27:26.275 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.17400921899999999 2024-09-21T10:27:26.275 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.275 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.275 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:26.275 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468486+0000", 2024-09-21T10:27:26.275 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000130395 2024-09-21T10:27:26.275 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.275 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.276 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.276 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.276 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.276 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.37050.0:174623 13.e e386/367 13:7f7be7eb:::benchmark_data_smithi107_278655_object46716:head v 386'10693, mlcod=386'10691)", 2024-09-21T10:27:26.276 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:46.300425+0000", 2024-09-21T10:27:26.276 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 399.936441081, 2024-09-21T10:27:26.276 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.168092769, 2024-09-21T10:27:26.276 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:26.276 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:26.276 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:26.276 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.276 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:26.276 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.300425+0000", 2024-09-21T10:27:26.276 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.277 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.277 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.277 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:26.277 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.300425+0000", 2024-09-21T10:27:26.277 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.277 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.277 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.277 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:26.277 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.300425+0000", 2024-09-21T10:27:26.277 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:27:26.277 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.277 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.277 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:26.277 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.300431+0000", 2024-09-21T10:27:26.277 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5429999999999999e-06 2024-09-21T10:27:26.278 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.278 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.278 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:26.278 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.300442+0000", 2024-09-21T10:27:26.278 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0458e-05 2024-09-21T10:27:26.278 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.278 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.278 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:26.278 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.300447+0000", 2024-09-21T10:27:26.278 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.378e-06 2024-09-21T10:27:26.278 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.278 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.278 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:26.278 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.300521+0000", 2024-09-21T10:27:26.279 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3446000000000001e-05 2024-09-21T10:27:26.279 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.279 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.279 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:26.279 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.300539+0000", 2024-09-21T10:27:26.279 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.8085e-05 2024-09-21T10:27:26.279 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.279 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.279 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:26.279 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468493+0000", 2024-09-21T10:27:26.279 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.167954518 2024-09-21T10:27:26.279 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.279 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.279 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:26.279 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468518+0000", 2024-09-21T10:27:26.280 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.5041e-05 2024-09-21T10:27:26.280 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.280 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.280 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.280 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.280 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.280 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.37050.0:174614 13.7 e386/381 13:e8d06524:::benchmark_data_smithi107_278655_object46707:head v 386'10950, mlcod=386'10947)", 2024-09-21T10:27:26.280 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:46.303073+0000", 2024-09-21T10:27:26.280 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 399.93379337200003, 2024-09-21T10:27:26.280 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.165326052, 2024-09-21T10:27:26.280 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:26.280 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:26.280 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:26.280 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.281 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:26.281 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303073+0000", 2024-09-21T10:27:26.281 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.281 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.281 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.281 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:26.281 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303073+0000", 2024-09-21T10:27:26.281 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.281 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.281 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.281 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:26.281 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303072+0000", 2024-09-21T10:27:26.281 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:27:26.281 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.281 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.282 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:26.282 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303079+0000", 2024-09-21T10:27:26.282 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.4420000000000001e-06 2024-09-21T10:27:26.282 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.282 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.282 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:26.282 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303091+0000", 2024-09-21T10:27:26.282 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.147e-05 2024-09-21T10:27:26.282 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.282 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.282 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:26.282 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303096+0000", 2024-09-21T10:27:26.283 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6869999999999999e-06 2024-09-21T10:27:26.283 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.283 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.283 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:26.283 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303157+0000", 2024-09-21T10:27:26.283 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0871e-05 2024-09-21T10:27:26.283 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.283 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.283 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:26.283 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303176+0000", 2024-09-21T10:27:26.283 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.8737e-05 2024-09-21T10:27:26.283 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.283 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.283 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:26.283 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468350+0000", 2024-09-21T10:27:26.283 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.165173704 2024-09-21T10:27:26.284 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.284 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.284 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:26.284 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468399+0000", 2024-09-21T10:27:26.284 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9401999999999999e-05 2024-09-21T10:27:26.284 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.284 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.284 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.284 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.284 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.284 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.37050.0:174621 13.5 e386/367 13:ab2f47f8:::benchmark_data_smithi107_278655_object46714:head v 386'10856, mlcod=386'10854)", 2024-09-21T10:27:26.284 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:46.303536+0000", 2024-09-21T10:27:26.284 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 399.93333076499999, 2024-09-21T10:27:26.284 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.168028229, 2024-09-21T10:27:26.284 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:26.285 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:26.285 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:26.285 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.285 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:26.285 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303536+0000", 2024-09-21T10:27:26.285 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.285 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.285 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.285 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:26.285 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303536+0000", 2024-09-21T10:27:26.285 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.285 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.285 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.285 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:26.286 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303535+0000", 2024-09-21T10:27:26.286 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:27:26.286 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.286 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.286 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:26.286 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303542+0000", 2024-09-21T10:27:26.286 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3610000000000003e-06 2024-09-21T10:27:26.286 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.286 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.286 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:26.286 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303553+0000", 2024-09-21T10:27:26.286 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0467e-05 2024-09-21T10:27:26.286 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.286 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.286 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:26.287 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303558+0000", 2024-09-21T10:27:26.287 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1850000000000003e-06 2024-09-21T10:27:26.287 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.287 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.287 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:26.287 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303595+0000", 2024-09-21T10:27:26.287 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7273e-05 2024-09-21T10:27:26.287 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.287 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.287 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:26.287 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.303612+0000", 2024-09-21T10:27:26.287 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.6565000000000001e-05 2024-09-21T10:27:26.287 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.287 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.287 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:26.288 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468372+0000", 2024-09-21T10:27:26.288 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.164760619 2024-09-21T10:27:26.288 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.288 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.288 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:26.288 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.471564+0000", 2024-09-21T10:27:26.288 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003191457 2024-09-21T10:27:26.288 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.288 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.288 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.288 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.288 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.288 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.37050.0:174630 13.4 e386/381 13:22268ddb:::benchmark_data_smithi107_278655_object46723:head v 386'10968, mlcod=386'10966)", 2024-09-21T10:27:26.288 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:46.304122+0000", 2024-09-21T10:27:26.289 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 399.932744275, 2024-09-21T10:27:26.289 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.175023032, 2024-09-21T10:27:26.289 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:26.289 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:26.289 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:26.289 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.289 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:26.289 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.304122+0000", 2024-09-21T10:27:26.289 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.289 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.289 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.289 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:26.289 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.304122+0000", 2024-09-21T10:27:26.289 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.290 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.290 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.290 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:26.290 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.304122+0000", 2024-09-21T10:27:26.290 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:27:26.290 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.290 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.290 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:26.290 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.304126+0000", 2024-09-21T10:27:26.290 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9389999999999996e-06 2024-09-21T10:27:26.290 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.290 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.290 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:26.290 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.304137+0000", 2024-09-21T10:27:26.291 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0522e-05 2024-09-21T10:27:26.291 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.291 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.291 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:26.291 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.304142+0000", 2024-09-21T10:27:26.291 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1499999999999998e-06 2024-09-21T10:27:26.291 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.291 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.291 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:26.291 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468378+0000", 2024-09-21T10:27:26.291 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.16423565200000001 2024-09-21T10:27:26.291 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.291 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.291 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:26.292 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468404+0000", 2024-09-21T10:27:26.292 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.6094000000000001e-05 2024-09-21T10:27:26.292 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.292 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.292 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:26.292 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.479084+0000", 2024-09-21T10:27:26.292 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.010679823 2024-09-21T10:27:26.292 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.292 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.292 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:26.292 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.479145+0000", 2024-09-21T10:27:26.292 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1499000000000002e-05 2024-09-21T10:27:26.292 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.292 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.293 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.293 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.293 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.293 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.37050.0:174633 13.2 e386/367 13:493179e2:::benchmark_data_smithi107_278655_object46726:head v 386'10818, mlcod=386'10816)", 2024-09-21T10:27:26.293 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:46.306435+0000", 2024-09-21T10:27:26.293 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 399.93043189899998, 2024-09-21T10:27:26.293 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.16794150799999999, 2024-09-21T10:27:26.293 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:26.293 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:26.293 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:26.293 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.293 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:26.293 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.306435+0000", 2024-09-21T10:27:26.293 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.294 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.294 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.294 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:26.294 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.306435+0000", 2024-09-21T10:27:26.294 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.294 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.294 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.294 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:26.294 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.306433+0000", 2024-09-21T10:27:26.294 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:26.294 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.294 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.294 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:26.294 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.306440+0000", 2024-09-21T10:27:26.295 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5039999999999999e-06 2024-09-21T10:27:26.295 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.295 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.295 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:26.295 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468393+0000", 2024-09-21T10:27:26.295 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.16195300200000001 2024-09-21T10:27:26.295 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.295 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.295 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:26.295 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468400+0000", 2024-09-21T10:27:26.295 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.1489999999999997e-06 2024-09-21T10:27:26.295 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.295 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.295 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:26.295 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468443+0000", 2024-09-21T10:27:26.296 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3337000000000002e-05 2024-09-21T10:27:26.296 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.296 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.296 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:26.296 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468462+0000", 2024-09-21T10:27:26.296 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.8746999999999999e-05 2024-09-21T10:27:26.296 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.296 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.296 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:26.296 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.474257+0000", 2024-09-21T10:27:26.296 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0057951770000000003 2024-09-21T10:27:26.296 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.296 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.296 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:26.297 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.474376+0000", 2024-09-21T10:27:26.297 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000118749 2024-09-21T10:27:26.297 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.297 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.297 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.297 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.297 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.297 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "MOSDPGPush(13.7 386/381 [PushOp(13:edaa4b58:::benchmark_data_smithi107_278655_object61423:head, version: 373'3864, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(13:edaa4b58:::benchmark_data_smithi107_278655_object61423:head@373'3864, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T10:27:26.297 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:46.308078+0000", 2024-09-21T10:27:26.297 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 399.92878894299997, 2024-09-21T10:27:26.297 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.160623403, 2024-09-21T10:27:26.297 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:26.297 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "started", 2024-09-21T10:27:26.297 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:26.297 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.298 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:26.298 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.308078+0000", 2024-09-21T10:27:26.298 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.298 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.298 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.298 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:26.298 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.308078+0000", 2024-09-21T10:27:26.298 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.298 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.298 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.298 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:26.298 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.308077+0000", 2024-09-21T10:27:26.298 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:26.298 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.298 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.299 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:26.299 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.308121+0000", 2024-09-21T10:27:26.299 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3968000000000003e-05 2024-09-21T10:27:26.299 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.299 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.299 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:26.299 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468344+0000", 2024-09-21T10:27:26.299 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.160223268 2024-09-21T10:27:26.299 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.299 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.299 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:26.299 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468365+0000", 2024-09-21T10:27:26.299 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.0803000000000001e-05 2024-09-21T10:27:26.300 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.300 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.300 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:26.300 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468474+0000", 2024-09-21T10:27:26.300 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010928400000000001 2024-09-21T10:27:26.300 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.300 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.300 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:26.300 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468498+0000", 2024-09-21T10:27:26.300 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.4221000000000001e-05 2024-09-21T10:27:26.300 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.300 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.300 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:26.300 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468701+0000", 2024-09-21T10:27:26.300 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00020275499999999999 2024-09-21T10:27:26.301 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.301 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.301 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.301 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.301 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.301 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.37050.0:174635 13.6 e386/367 13:6b286960:::benchmark_data_smithi107_278655_object46728:head v 386'10872, mlcod=386'10869)", 2024-09-21T10:27:26.301 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:46.312725+0000", 2024-09-21T10:27:26.301 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 399.92414188200001, 2024-09-21T10:27:26.301 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.15934837700000001, 2024-09-21T10:27:26.301 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:26.301 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:26.301 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:26.301 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.301 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:26.301 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.312725+0000", 2024-09-21T10:27:26.302 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.302 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.302 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.302 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:26.302 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.312725+0000", 2024-09-21T10:27:26.302 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.302 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.302 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.302 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:26.302 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.312723+0000", 2024-09-21T10:27:26.302 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:27:26.302 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.302 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.302 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:26.303 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.312733+0000", 2024-09-21T10:27:26.303 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.1560000000000001e-06 2024-09-21T10:27:26.303 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.303 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.303 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:26.303 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468348+0000", 2024-09-21T10:27:26.303 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.15561591299999999 2024-09-21T10:27:26.303 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.303 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.303 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:26.303 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468360+0000", 2024-09-21T10:27:26.303 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.208e-05 2024-09-21T10:27:26.303 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.303 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.304 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:26.304 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468542+0000", 2024-09-21T10:27:26.304 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000181836 2024-09-21T10:27:26.304 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.304 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.304 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:26.304 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.468561+0000", 2024-09-21T10:27:26.304 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.8504000000000001e-05 2024-09-21T10:27:26.304 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.304 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.304 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:26.304 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.471968+0000", 2024-09-21T10:27:26.304 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0034071029999999999 2024-09-21T10:27:26.304 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.304 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.305 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:26.305 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.472073+0000", 2024-09-21T10:27:26.305 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010501399999999999 2024-09-21T10:27:26.305 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.305 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.305 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.305 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.305 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.305 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.37050.0:201502 13.f e392/381 13:f68e24a4:::benchmark_data_smithi107_278655_object73595:head v 392'12598, mlcod=392'12595)", 2024-09-21T10:27:26.305 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:21:17.453994+0000", 2024-09-21T10:27:26.305 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 368.78287246100001, 2024-09-21T10:27:26.305 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.059373088999999997, 2024-09-21T10:27:26.305 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:26.305 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:26.306 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:26.306 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.306 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:26.306 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:17.453994+0000", 2024-09-21T10:27:26.306 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.306 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.306 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.306 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:26.306 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:17.453994+0000", 2024-09-21T10:27:26.306 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.306 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.306 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.306 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:26.306 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:17.453993+0000", 2024-09-21T10:27:26.307 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:26.307 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.307 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.307 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:26.307 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:17.453999+0000", 2024-09-21T10:27:26.307 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0519999999999998e-06 2024-09-21T10:27:26.307 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.307 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.307 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:26.307 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:17.454009+0000", 2024-09-21T10:27:26.307 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.679e-06 2024-09-21T10:27:26.307 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.307 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.307 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:26.308 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:17.454014+0000", 2024-09-21T10:27:26.308 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4140000000000002e-06 2024-09-21T10:27:26.308 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.308 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.308 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:26.308 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:17.454090+0000", 2024-09-21T10:27:26.308 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.6299999999999998e-05 2024-09-21T10:27:26.308 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.308 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.308 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:26.308 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:17.454110+0000", 2024-09-21T10:27:26.308 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.9468000000000001e-05 2024-09-21T10:27:26.308 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.308 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.309 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:26.309 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:17.513310+0000", 2024-09-21T10:27:26.309 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.059199996999999997 2024-09-21T10:27:26.309 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.309 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.309 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:26.309 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:17.513367+0000", 2024-09-21T10:27:26.309 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7240000000000001e-05 2024-09-21T10:27:26.309 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.309 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.309 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.309 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.309 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.309 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.37050.0:201503 13.f e392/381 13:fa409abc:::benchmark_data_smithi107_278655_object73596:head v 392'12599, mlcod=392'12596)", 2024-09-21T10:27:26.310 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:21:17.457953+0000", 2024-09-21T10:27:26.310 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 368.77891352500001, 2024-09-21T10:27:26.310 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.064352501000000006, 2024-09-21T10:27:26.310 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:26.310 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:26.310 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:26.310 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.310 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:26.310 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:17.457953+0000", 2024-09-21T10:27:26.310 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.310 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.310 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.310 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:26.311 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:17.457953+0000", 2024-09-21T10:27:26.311 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.311 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.311 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.311 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:26.311 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:17.457952+0000", 2024-09-21T10:27:26.311 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:26.311 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.311 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.311 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:26.311 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:17.457960+0000", 2024-09-21T10:27:26.311 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9589999999999995e-06 2024-09-21T10:27:26.311 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.311 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.312 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:26.312 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:17.491770+0000", 2024-09-21T10:27:26.312 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.033809657999999999 2024-09-21T10:27:26.312 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.312 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.312 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:26.312 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:17.491780+0000", 2024-09-21T10:27:26.312 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0239e-05 2024-09-21T10:27:26.312 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.312 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.312 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:26.312 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:17.492017+0000", 2024-09-21T10:27:26.312 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00023671099999999999 2024-09-21T10:27:26.312 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.312 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.313 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:26.313 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:17.492531+0000", 2024-09-21T10:27:26.313 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00051425400000000001 2024-09-21T10:27:26.313 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.313 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.313 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:26.313 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:17.522241+0000", 2024-09-21T10:27:26.313 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.029709666999999999 2024-09-21T10:27:26.313 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.313 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.313 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:26.313 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:17.522305+0000", 2024-09-21T10:27:26.313 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4828999999999999e-05 2024-09-21T10:27:26.313 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.314 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.314 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.314 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.314 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.314 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.47186.0:209415 14.7 e433/411 14:e350bf57:::benchmark_data_smithi107_312686_object93624:head v 433'13442, mlcod=433'13436)", 2024-09-21T10:27:26.314 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:26:21.640062+0000", 2024-09-21T10:27:26.314 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 64.596804896999998, 2024-09-21T10:27:26.314 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.034742193999999997, 2024-09-21T10:27:26.314 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:26.314 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:26.314 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:26.314 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.314 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:26.314 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:21.640062+0000", 2024-09-21T10:27:26.315 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.315 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.315 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.315 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:26.315 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:21.640062+0000", 2024-09-21T10:27:26.315 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.315 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.315 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.315 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:26.315 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:21.640061+0000", 2024-09-21T10:27:26.315 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:27:26.315 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.315 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.315 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:26.316 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:21.640066+0000", 2024-09-21T10:27:26.316 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1000000000000003e-06 2024-09-21T10:27:26.316 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.316 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.316 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:26.316 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:21.640074+0000", 2024-09-21T10:27:26.316 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.588e-06 2024-09-21T10:27:26.316 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.316 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.316 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:26.316 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:21.640077+0000", 2024-09-21T10:27:26.316 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8659999999999999e-06 2024-09-21T10:27:26.316 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.316 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.316 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:26.317 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:21.640111+0000", 2024-09-21T10:27:26.317 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3223999999999999e-05 2024-09-21T10:27:26.317 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.317 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.317 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:26.317 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:21.640124+0000", 2024-09-21T10:27:26.317 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3474e-05 2024-09-21T10:27:26.317 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.317 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.317 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:26.317 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:21.674764+0000", 2024-09-21T10:27:26.317 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.034639479000000001 2024-09-21T10:27:26.317 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.317 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.318 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:26.318 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:21.674804+0000", 2024-09-21T10:27:26.318 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0105000000000003e-05 2024-09-21T10:27:26.318 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.318 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.318 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.318 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.318 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.318 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.47186.0:209419 14.7 e433/411 14:eee6bac6:::benchmark_data_smithi107_312686_object93628:head v 433'13443, mlcod=433'13436)", 2024-09-21T10:27:26.318 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:26:21.644664+0000", 2024-09-21T10:27:26.318 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 64.592202040999993, 2024-09-21T10:27:26.318 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.030164758, 2024-09-21T10:27:26.318 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:26.318 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:26.319 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:26.319 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.319 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:26.319 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:21.644664+0000", 2024-09-21T10:27:26.319 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.319 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.319 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.319 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:26.319 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:21.644664+0000", 2024-09-21T10:27:26.319 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.319 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.319 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.319 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:26.319 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:21.644663+0000", 2024-09-21T10:27:26.320 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:26.320 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.320 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.320 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:26.320 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:21.644671+0000", 2024-09-21T10:27:26.320 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.0489999999999998e-06 2024-09-21T10:27:26.320 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.320 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.320 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:26.320 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:21.644683+0000", 2024-09-21T10:27:26.320 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.234e-05 2024-09-21T10:27:26.320 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.320 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.320 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:26.320 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:21.644689+0000", 2024-09-21T10:27:26.321 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.156e-06 2024-09-21T10:27:26.321 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.321 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.321 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:26.321 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:21.644731+0000", 2024-09-21T10:27:26.321 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1628999999999997e-05 2024-09-21T10:27:26.321 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.321 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.321 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:26.321 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:21.644749+0000", 2024-09-21T10:27:26.321 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.7870999999999999e-05 2024-09-21T10:27:26.321 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.321 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.321 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:26.321 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:21.674810+0000", 2024-09-21T10:27:26.322 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.030061528000000001 2024-09-21T10:27:26.322 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.322 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.322 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:26.322 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:21.674829+0000", 2024-09-21T10:27:26.322 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.9119000000000001e-05 2024-09-21T10:27:26.322 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.322 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.322 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.322 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.322 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.322 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_repop(client.47186.0:230895 14.d e434/411 14:bf976f5c:::benchmark_data_smithi107_312686_object115104:head v 434'14433, mlcod=434'14429)", 2024-09-21T10:27:26.322 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:26:48.079308+0000", 2024-09-21T10:27:26.323 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 38.157558272999999, 2024-09-21T10:27:26.323 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.023866077999999999, 2024-09-21T10:27:26.323 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:26.323 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:26.323 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:26.323 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.323 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:26.323 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:48.079308+0000", 2024-09-21T10:27:26.323 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.323 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.323 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.323 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:26.323 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:48.079308+0000", 2024-09-21T10:27:26.323 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:26.324 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.324 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.324 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:26.324 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:48.079307+0000", 2024-09-21T10:27:26.324 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:26.324 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.324 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.324 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:26.324 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:48.079314+0000", 2024-09-21T10:27:26.324 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9410000000000001e-06 2024-09-21T10:27:26.324 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.324 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.324 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:26.324 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:48.079327+0000", 2024-09-21T10:27:26.324 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2519e-05 2024-09-21T10:27:26.325 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.325 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.325 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:26.325 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:48.079333+0000", 2024-09-21T10:27:26.325 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6189999999999996e-06 2024-09-21T10:27:26.325 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.325 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.325 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:26.325 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:48.079392+0000", 2024-09-21T10:27:26.325 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8393999999999997e-05 2024-09-21T10:27:26.325 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.325 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.325 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:26.325 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:48.079417+0000", 2024-09-21T10:27:26.326 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.4797e-05 2024-09-21T10:27:26.326 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.326 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.326 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:26.326 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:48.103144+0000", 2024-09-21T10:27:26.326 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.023727270000000002 2024-09-21T10:27:26.326 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:26.326 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:26.326 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:26.326 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:26:48.103174+0000", 2024-09-21T10:27:26.326 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0440999999999999e-05 2024-09-21T10:27:26.326 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.326 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.326 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.327 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:26.327 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:26.327 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:27:27.067 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:26 smithi129 ceph-mon[88471]: pgmap v1608: 93 pgs: 9 peering, 84 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-21T10:27:27.067 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:27.093 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:27:27.337 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:27:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:26 smithi107 ceph-mon[102221]: pgmap v1608: 93 pgs: 9 peering, 84 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-21T10:27:27.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:26 smithi107 ceph-mon[105022]: pgmap v1608: 93 pgs: 9 peering, 84 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-21T10:27:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:28.950 INFO:teuthology.orchestra.run.smithi107.stderr:noscrub is unset 2024-09-21T10:27:29.046 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:28 smithi129 ceph-mon[88471]: pgmap v1609: 93 pgs: 9 peering, 84 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-21T10:27:29.046 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:28 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4193166011' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:27:29.046 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:28 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:27:29.057 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:27:29.057 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:27:29.057 INFO:teuthology.orchestra.run.smithi129.stdout: "num_ops": 0 2024-09-21T10:27:29.057 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:27:29.246 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:28 smithi107 ceph-mon[102221]: pgmap v1609: 93 pgs: 9 peering, 84 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-21T10:27:29.247 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:28 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4193166011' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:27:29.247 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:28 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:27:29.247 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:28 smithi107 ceph-mon[105022]: pgmap v1609: 93 pgs: 9 peering, 84 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-21T10:27:29.247 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:28 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4193166011' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:27:29.247 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:28 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:27:29.675 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:27:29.837 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:27:29.916 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:27:30.092 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:30.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:29 smithi107 ceph-mon[102221]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-21T10:27:30.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:29 smithi107 ceph-mon[102221]: Cluster is now healthy 2024-09-21T10:27:30.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:29 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:27:30.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:29 smithi107 ceph-mon[102221]: osdmap e452: 8 total, 8 up, 8 in 2024-09-21T10:27:30.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:29 smithi107 ceph-mon[105022]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-21T10:27:30.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:29 smithi107 ceph-mon[105022]: Cluster is now healthy 2024-09-21T10:27:30.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:29 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:27:30.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:29 smithi107 ceph-mon[105022]: osdmap e452: 8 total, 8 up, 8 in 2024-09-21T10:27:30.216 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:29 smithi129 ceph-mon[88471]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-21T10:27:30.216 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:29 smithi129 ceph-mon[88471]: Cluster is now healthy 2024-09-21T10:27:30.216 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:29 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:27:30.216 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:29 smithi129 ceph-mon[88471]: osdmap e452: 8 total, 8 up, 8 in 2024-09-21T10:27:30.363 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-21T10:27:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:27:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:27:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:27:31.166 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:30 smithi129 ceph-mon[88471]: pgmap v1611: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T10:27:31.223 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:31.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:30 smithi107 ceph-mon[102221]: pgmap v1611: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T10:27:31.307 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:30 smithi107 ceph-mon[105022]: pgmap v1611: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T10:27:31.511 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:27:31.511 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:27:31.511 INFO:teuthology.orchestra.run.smithi129.stdout: "complaint_time": 30, 2024-09-21T10:27:31.511 INFO:teuthology.orchestra.run.smithi129.stdout: "num_blocked_ops": 0 2024-09-21T10:27:31.511 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:27:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:27:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:27:31] "GET /metrics HTTP/1.1" 200 34379 "" "Prometheus/2.51.0" 2024-09-21T10:27:32.134 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:27:32.381 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:27:32.964 INFO:teuthology.orchestra.run.smithi107.stderr:nodeep-scrub is unset 2024-09-21T10:27:33.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:32 smithi129 ceph-mon[88471]: pgmap v1612: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-09-21T10:27:33.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:32 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/332942749' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:27:33.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:32 smithi107 ceph-mon[102221]: pgmap v1612: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-09-21T10:27:33.261 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:32 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/332942749' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:27:33.262 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:32 smithi107 ceph-mon[105022]: pgmap v1612: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-09-21T10:27:33.262 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:32 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/332942749' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:27:34.119 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:33 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/332942749' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:27:34.119 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:33 smithi129 ceph-mon[88471]: osdmap e453: 8 total, 8 up, 8 in 2024-09-21T10:27:34.205 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:33 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/332942749' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:27:34.205 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:33 smithi107 ceph-mon[102221]: osdmap e453: 8 total, 8 up, 8 in 2024-09-21T10:27:34.205 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:33 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/332942749' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:27:34.205 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:33 smithi107 ceph-mon[105022]: osdmap e453: 8 total, 8 up, 8 in 2024-09-21T10:27:34.216 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:27:34.216 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 20, 2024-09-21T10:27:34.216 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 600, 2024-09-21T10:27:34.217 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:27:34.217 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.217 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:23842 13.4 13:2ae185ed:::benchmark_data_smithi107_278655_object23841:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:27:34.217 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:29.864431+0000", 2024-09-21T10:27:34.217 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 544.34885943100005, 2024-09-21T10:27:34.217 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.168915237, 2024-09-21T10:27:34.217 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:34.217 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:34.217 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:34.217 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:34.217 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:34.217 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 23842 2024-09-21T10:27:34.218 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.218 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:34.218 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.218 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:34.218 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864431+0000", 2024-09-21T10:27:34.218 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.218 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.218 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.218 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:34.218 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864431+0000", 2024-09-21T10:27:34.218 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.218 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.218 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.218 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:34.219 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864430+0000", 2024-09-21T10:27:34.219 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:27:34.219 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.219 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.219 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:34.219 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864487+0000", 2024-09-21T10:27:34.219 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7219000000000002e-05 2024-09-21T10:27:34.219 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.219 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.219 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:34.219 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864499+0000", 2024-09-21T10:27:34.219 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2062000000000001e-05 2024-09-21T10:27:34.219 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.219 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.220 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:34.220 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864505+0000", 2024-09-21T10:27:34.220 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0100000000000001e-06 2024-09-21T10:27:34.220 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.220 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.220 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:34.220 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864555+0000", 2024-09-21T10:27:34.220 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9574999999999999e-05 2024-09-21T10:27:34.220 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.220 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.220 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:34.220 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864687+0000", 2024-09-21T10:27:34.220 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013213 2024-09-21T10:27:34.221 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.221 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.221 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:27:34.221 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.864915+0000", 2024-09-21T10:27:34.221 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00022785700000000001 2024-09-21T10:27:34.221 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.221 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.221 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:34.221 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.867465+0000", 2024-09-21T10:27:34.221 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0025501230000000001 2024-09-21T10:27:34.221 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.221 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.221 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.221 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.867521+0000", 2024-09-21T10:27:34.221 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5708000000000003e-05 2024-09-21T10:27:34.222 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.222 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.222 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.222 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:30.033258+0000", 2024-09-21T10:27:34.222 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.165737356 2024-09-21T10:27:34.222 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.222 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.222 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:34.222 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:30.033310+0000", 2024-09-21T10:27:34.222 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1539000000000002e-05 2024-09-21T10:27:34.222 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.222 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.223 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:34.223 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:30.033346+0000", 2024-09-21T10:27:34.223 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6936e-05 2024-09-21T10:27:34.223 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.223 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.223 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.223 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.223 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.223 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:23845 13.4 13:27245713:::benchmark_data_smithi107_278655_object23844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:27:34.223 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:29.866129+0000", 2024-09-21T10:27:34.223 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 544.34716165700002, 2024-09-21T10:27:34.223 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.16733213499999999, 2024-09-21T10:27:34.223 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:34.223 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:34.223 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:34.224 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:34.224 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:34.224 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 23845 2024-09-21T10:27:34.224 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.224 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:34.224 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.224 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:34.224 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866129+0000", 2024-09-21T10:27:34.224 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.224 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.224 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.224 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:34.224 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866129+0000", 2024-09-21T10:27:34.225 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.225 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.225 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.225 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:34.225 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866128+0000", 2024-09-21T10:27:34.225 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:27:34.225 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.225 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.225 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:34.225 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866189+0000", 2024-09-21T10:27:34.225 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1872999999999995e-05 2024-09-21T10:27:34.225 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.225 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.225 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:34.225 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866204+0000", 2024-09-21T10:27:34.226 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4739000000000001e-05 2024-09-21T10:27:34.226 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.226 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.226 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:34.226 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866211+0000", 2024-09-21T10:27:34.226 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2949999999999999e-06 2024-09-21T10:27:34.226 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.226 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.226 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:34.226 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866248+0000", 2024-09-21T10:27:34.226 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7704999999999999e-05 2024-09-21T10:27:34.226 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.226 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.226 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:34.227 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866354+0000", 2024-09-21T10:27:34.227 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000106022 2024-09-21T10:27:34.227 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.227 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.227 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:27:34.227 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.866575+0000", 2024-09-21T10:27:34.227 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00022120399999999999 2024-09-21T10:27:34.227 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.227 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.227 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.227 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.867933+0000", 2024-09-21T10:27:34.227 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0013572499999999999 2024-09-21T10:27:34.227 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.228 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.228 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:34.228 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:29.867987+0000", 2024-09-21T10:27:34.228 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3919999999999999e-05 2024-09-21T10:27:34.228 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.228 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.228 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.228 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:30.033382+0000", 2024-09-21T10:27:34.228 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.165395343 2024-09-21T10:27:34.228 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.228 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.228 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:34.228 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:30.033430+0000", 2024-09-21T10:27:34.228 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7997999999999997e-05 2024-09-21T10:27:34.228 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.229 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.229 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:34.229 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:30.033461+0000", 2024-09-21T10:27:34.229 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1149999999999998e-05 2024-09-21T10:27:34.229 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.229 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.229 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.229 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.229 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.229 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:26946 13.4 13:21264d98:::benchmark_data_smithi107_278655_object26945:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:27:34.229 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:31.498036+0000", 2024-09-21T10:27:34.229 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 542.71525475199996, 2024-09-21T10:27:34.229 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.30224518299999997, 2024-09-21T10:27:34.230 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:34.230 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:34.230 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:34.230 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:34.231 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:34.231 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 26946 2024-09-21T10:27:34.231 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.231 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:34.231 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.231 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:34.231 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498036+0000", 2024-09-21T10:27:34.231 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.231 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.231 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.231 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:34.231 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498036+0000", 2024-09-21T10:27:34.231 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.231 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.232 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.232 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:34.232 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498035+0000", 2024-09-21T10:27:34.232 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:27:34.232 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.232 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.232 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:34.232 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498090+0000", 2024-09-21T10:27:34.232 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5785000000000002e-05 2024-09-21T10:27:34.232 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.232 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.232 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:34.232 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498104+0000", 2024-09-21T10:27:34.232 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3871e-05 2024-09-21T10:27:34.233 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.233 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.233 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:34.233 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498111+0000", 2024-09-21T10:27:34.233 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4389999999999997e-06 2024-09-21T10:27:34.233 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.233 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.233 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:34.233 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498154+0000", 2024-09-21T10:27:34.233 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3238999999999997e-05 2024-09-21T10:27:34.233 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.233 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.233 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:34.234 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498237+0000", 2024-09-21T10:27:34.234 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.2571000000000006e-05 2024-09-21T10:27:34.234 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.234 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.234 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:27:34.234 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498343+0000", 2024-09-21T10:27:34.234 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000106498 2024-09-21T10:27:34.234 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.234 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.234 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:34.234 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.499830+0000", 2024-09-21T10:27:34.234 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0014867529999999999 2024-09-21T10:27:34.234 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.235 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.235 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.235 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.499999+0000", 2024-09-21T10:27:34.235 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00016944099999999999 2024-09-21T10:27:34.235 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.235 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.235 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.235 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.800209+0000", 2024-09-21T10:27:34.235 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.300209275 2024-09-21T10:27:34.235 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.235 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.235 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:34.235 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.800250+0000", 2024-09-21T10:27:34.235 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.189e-05 2024-09-21T10:27:34.236 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.236 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.236 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:34.236 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.800281+0000", 2024-09-21T10:27:34.236 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0618999999999999e-05 2024-09-21T10:27:34.236 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.236 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.236 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.236 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.236 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.236 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:72757 13.f 13:f96442f9:::benchmark_data_smithi107_278655_object72756:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:27:34.236 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.476101+0000", 2024-09-21T10:27:34.236 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 513.73718948400006, 2024-09-21T10:27:34.236 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.26259288400000003, 2024-09-21T10:27:34.236 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:34.236 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:34.237 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:34.237 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:34.237 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:34.237 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 72757 2024-09-21T10:27:34.237 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.237 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:34.237 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.237 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:34.237 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476101+0000", 2024-09-21T10:27:34.237 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.237 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.237 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.237 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:34.237 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476101+0000", 2024-09-21T10:27:34.237 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.238 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.238 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.238 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:34.238 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476100+0000", 2024-09-21T10:27:34.238 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:27:34.238 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.238 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.238 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:34.238 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476136+0000", 2024-09-21T10:27:34.238 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6081000000000002e-05 2024-09-21T10:27:34.238 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.238 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.238 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:34.239 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476151+0000", 2024-09-21T10:27:34.239 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4958e-05 2024-09-21T10:27:34.239 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.239 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.239 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:34.239 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476157+0000", 2024-09-21T10:27:34.239 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9150000000000001e-06 2024-09-21T10:27:34.239 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.239 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.239 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:34.239 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476242+0000", 2024-09-21T10:27:34.239 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.4896999999999997e-05 2024-09-21T10:27:34.239 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.239 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.239 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:34.240 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476468+0000", 2024-09-21T10:27:34.240 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00022605300000000001 2024-09-21T10:27:34.240 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.240 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.240 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:27:34.240 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.476602+0000", 2024-09-21T10:27:34.240 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013458 2024-09-21T10:27:34.240 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.240 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.240 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.240 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481355+0000", 2024-09-21T10:27:34.240 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0047528479999999996 2024-09-21T10:27:34.240 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.241 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.241 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:34.241 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485113+0000", 2024-09-21T10:27:34.241 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003757842 2024-09-21T10:27:34.241 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.241 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.241 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.241 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.738610+0000", 2024-09-21T10:27:34.241 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25349742400000003 2024-09-21T10:27:34.241 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.241 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.241 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:34.241 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.738650+0000", 2024-09-21T10:27:34.241 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0089999999999997e-05 2024-09-21T10:27:34.242 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.242 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.242 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:34.242 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.738694+0000", 2024-09-21T10:27:34.242 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3649000000000002e-05 2024-09-21T10:27:34.242 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.242 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.242 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.242 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.242 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.242 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:72765 13.f 13:f6d6e7fe:::benchmark_data_smithi107_278655_object72764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:27:34.242 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.481220+0000", 2024-09-21T10:27:34.242 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 513.73207087599997, 2024-09-21T10:27:34.242 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.26411038599999997, 2024-09-21T10:27:34.243 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:34.243 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:34.243 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:34.243 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:34.243 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:34.243 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 72765 2024-09-21T10:27:34.243 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.243 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:34.243 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.243 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:34.243 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481220+0000", 2024-09-21T10:27:34.243 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.243 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.243 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.244 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:34.244 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481220+0000", 2024-09-21T10:27:34.244 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.244 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.244 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.244 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:34.244 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481218+0000", 2024-09-21T10:27:34.244 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:27:34.244 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.244 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.244 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:34.244 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481277+0000", 2024-09-21T10:27:34.244 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8371000000000001e-05 2024-09-21T10:27:34.244 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.244 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.245 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:34.245 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481289+0000", 2024-09-21T10:27:34.245 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2197e-05 2024-09-21T10:27:34.245 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.245 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.245 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:34.245 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481295+0000", 2024-09-21T10:27:34.245 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8309999999999997e-06 2024-09-21T10:27:34.245 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.245 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.245 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:34.245 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481444+0000", 2024-09-21T10:27:34.245 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014953500000000001 2024-09-21T10:27:34.246 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.246 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.246 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:34.246 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.482468+0000", 2024-09-21T10:27:34.246 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0010236080000000001 2024-09-21T10:27:34.246 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.246 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.246 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:27:34.246 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.482570+0000", 2024-09-21T10:27:34.246 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000101711 2024-09-21T10:27:34.246 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.246 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.246 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:34.247 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.488060+0000", 2024-09-21T10:27:34.247 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0054900289999999996 2024-09-21T10:27:34.247 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.247 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.247 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.247 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489602+0000", 2024-09-21T10:27:34.247 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0015419990000000001 2024-09-21T10:27:34.247 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.247 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.247 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.247 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745244+0000", 2024-09-21T10:27:34.247 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25564244800000002 2024-09-21T10:27:34.247 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.247 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.248 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:34.248 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745299+0000", 2024-09-21T10:27:34.248 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4877999999999999e-05 2024-09-21T10:27:34.248 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.248 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.248 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:34.248 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745330+0000", 2024-09-21T10:27:34.248 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1177000000000003e-05 2024-09-21T10:27:34.248 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.248 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.248 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.248 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.248 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.248 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:72767 13.f 13:fba5e0d5:::benchmark_data_smithi107_278655_object72766:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:27:34.248 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.481760+0000", 2024-09-21T10:27:34.249 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 513.73153057599995, 2024-09-21T10:27:34.249 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.263929786, 2024-09-21T10:27:34.249 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:34.249 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:34.249 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:34.249 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:34.249 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:34.249 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 72767 2024-09-21T10:27:34.249 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.249 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:34.249 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.249 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:34.249 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481760+0000", 2024-09-21T10:27:34.249 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.250 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.250 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.250 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:34.250 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481760+0000", 2024-09-21T10:27:34.250 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.250 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.250 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.250 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:34.250 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481759+0000", 2024-09-21T10:27:34.250 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:34.250 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.250 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.250 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:34.250 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481815+0000", 2024-09-21T10:27:34.251 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6230000000000002e-05 2024-09-21T10:27:34.251 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.251 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.251 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:34.251 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481831+0000", 2024-09-21T10:27:34.251 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5613000000000001e-05 2024-09-21T10:27:34.251 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.251 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.251 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:34.251 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.481840+0000", 2024-09-21T10:27:34.251 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.9390000000000003e-06 2024-09-21T10:27:34.251 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.251 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.251 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:34.252 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485140+0000", 2024-09-21T10:27:34.252 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0033002909999999999 2024-09-21T10:27:34.252 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.252 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.252 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:34.252 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485285+0000", 2024-09-21T10:27:34.252 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014537099999999999 2024-09-21T10:27:34.252 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.252 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.252 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:27:34.252 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485433+0000", 2024-09-21T10:27:34.252 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014723399999999999 2024-09-21T10:27:34.252 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.252 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.252 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:34.253 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489134+0000", 2024-09-21T10:27:34.253 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0037009009999999999 2024-09-21T10:27:34.253 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.253 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.253 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.253 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489909+0000", 2024-09-21T10:27:34.253 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00077549400000000005 2024-09-21T10:27:34.253 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.253 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.253 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.253 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745617+0000", 2024-09-21T10:27:34.253 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.255707831 2024-09-21T10:27:34.253 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.253 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.253 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:34.254 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745660+0000", 2024-09-21T10:27:34.254 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2803999999999999e-05 2024-09-21T10:27:34.254 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.254 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.254 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:34.254 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745690+0000", 2024-09-21T10:27:34.254 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0079000000000001e-05 2024-09-21T10:27:34.254 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.254 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.254 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.254 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.254 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.254 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:72770 13.1 13:87104e38:::benchmark_data_smithi107_278655_object72769:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:27:34.254 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.485917+0000", 2024-09-21T10:27:34.254 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 513.72737371899996, 2024-09-21T10:27:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25941609300000001, 2024-09-21T10:27:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 72770 2024-09-21T10:27:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485917+0000", 2024-09-21T10:27:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.255 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485917+0000", 2024-09-21T10:27:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485915+0000", 2024-09-21T10:27:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:27:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485978+0000", 2024-09-21T10:27:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3101000000000005e-05 2024-09-21T10:27:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.256 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.485996+0000", 2024-09-21T10:27:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.7759000000000001e-05 2024-09-21T10:27:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.486004+0000", 2024-09-21T10:27:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.3239999999999997e-06 2024-09-21T10:27:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.486055+0000", 2024-09-21T10:27:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1193000000000003e-05 2024-09-21T10:27:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.257 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.486192+0000", 2024-09-21T10:27:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013693200000000001 2024-09-21T10:27:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,4", 2024-09-21T10:27:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.486328+0000", 2024-09-21T10:27:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013682700000000001 2024-09-21T10:27:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489305+0000", 2024-09-21T10:27:34.258 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002977001 2024-09-21T10:27:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489525+0000", 2024-09-21T10:27:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00021918799999999999 2024-09-21T10:27:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745228+0000", 2024-09-21T10:27:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25570367500000002 2024-09-21T10:27:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.259 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745295+0000", 2024-09-21T10:27:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7137000000000005e-05 2024-09-21T10:27:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745333+0000", 2024-09-21T10:27:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7518999999999998e-05 2024-09-21T10:27:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:72774 13.4 13:248222d3:::benchmark_data_smithi107_278655_object72773:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:27:34.260 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.489389+0000", 2024-09-21T10:27:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 513.72390135600006, 2024-09-21T10:27:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25593706900000002, 2024-09-21T10:27:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 72774 2024-09-21T10:27:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489389+0000", 2024-09-21T10:27:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.261 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489389+0000", 2024-09-21T10:27:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489388+0000", 2024-09-21T10:27:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:27:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489476+0000", 2024-09-21T10:27:34.262 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.8084999999999994e-05 2024-09-21T10:27:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489490+0000", 2024-09-21T10:27:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4139e-05 2024-09-21T10:27:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489498+0000", 2024-09-21T10:27:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.1899999999999995e-06 2024-09-21T10:27:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489554+0000", 2024-09-21T10:27:34.263 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5649999999999997e-05 2024-09-21T10:27:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489701+0000", 2024-09-21T10:27:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014734700000000001 2024-09-21T10:27:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:27:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.489837+0000", 2024-09-21T10:27:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013627299999999999 2024-09-21T10:27:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.492678+0000", 2024-09-21T10:27:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0028408890000000001 2024-09-21T10:27:34.264 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.265 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.265 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.265 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.492823+0000", 2024-09-21T10:27:34.265 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00014503199999999999 2024-09-21T10:27:34.265 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.265 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.265 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.265 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745232+0000", 2024-09-21T10:27:34.265 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25240888500000003 2024-09-21T10:27:34.265 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.265 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.265 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:34.265 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745285+0000", 2024-09-21T10:27:34.265 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3041000000000003e-05 2024-09-21T10:27:34.265 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.266 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.266 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:34.266 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.745326+0000", 2024-09-21T10:27:34.266 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.1245000000000002e-05 2024-09-21T10:27:34.266 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.266 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.266 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.266 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.266 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.266 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:18227 14.2 14:4ecdcd6d:::benchmark_data_smithi107_312686_object18226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-21T10:27:34.266 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:23:03.357929+0000", 2024-09-21T10:27:34.266 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 270.85536125499999, 2024-09-21T10:27:34.266 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21685865500000001, 2024-09-21T10:27:34.266 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:34.266 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:34.267 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:34.267 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:27:34.267 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:34.267 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 18227 2024-09-21T10:27:34.267 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.267 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:34.267 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.267 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:34.267 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.357929+0000", 2024-09-21T10:27:34.267 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.267 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.267 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.267 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:34.267 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.357929+0000", 2024-09-21T10:27:34.268 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.268 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.268 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.268 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:34.268 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.357928+0000", 2024-09-21T10:27:34.268 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:27:34.268 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.268 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.268 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:34.268 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.357966+0000", 2024-09-21T10:27:34.268 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7462999999999997e-05 2024-09-21T10:27:34.268 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.268 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.268 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:34.269 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.357980+0000", 2024-09-21T10:27:34.269 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4613999999999999e-05 2024-09-21T10:27:34.269 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.269 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.269 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:34.269 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.357986+0000", 2024-09-21T10:27:34.269 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7479999999999999e-06 2024-09-21T10:27:34.269 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.269 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.269 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:34.269 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.358093+0000", 2024-09-21T10:27:34.269 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000107163 2024-09-21T10:27:34.269 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.270 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.270 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:34.270 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.358206+0000", 2024-09-21T10:27:34.270 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000113193 2024-09-21T10:27:34.270 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.270 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.270 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:27:34.270 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.358336+0000", 2024-09-21T10:27:34.270 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012923200000000001 2024-09-21T10:27:34.270 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.270 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.270 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:34.271 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.359740+0000", 2024-09-21T10:27:34.271 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0014040840000000001 2024-09-21T10:27:34.271 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.271 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.271 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.271 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.360595+0000", 2024-09-21T10:27:34.271 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00085584699999999997 2024-09-21T10:27:34.271 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.271 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.271 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.271 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.574681+0000", 2024-09-21T10:27:34.271 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.214085942 2024-09-21T10:27:34.271 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.271 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.271 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:34.272 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.574741+0000", 2024-09-21T10:27:34.272 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.999e-05 2024-09-21T10:27:34.272 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.272 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.272 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:34.272 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.574788+0000", 2024-09-21T10:27:34.272 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6610999999999998e-05 2024-09-21T10:27:34.272 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.272 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.272 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.272 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.272 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.272 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:28305 14.2 14:436767ef:::benchmark_data_smithi107_312686_object28304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-21T10:27:34.272 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:23:08.650026+0000", 2024-09-21T10:27:34.272 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 265.56326509899998, 2024-09-21T10:27:34.273 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.114775101, 2024-09-21T10:27:34.273 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:34.273 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:34.273 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:34.273 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:27:34.273 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:34.273 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 28305 2024-09-21T10:27:34.273 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.273 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:34.273 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.273 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:34.273 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.650026+0000", 2024-09-21T10:27:34.273 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.273 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.273 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.274 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:34.274 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.650026+0000", 2024-09-21T10:27:34.274 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.274 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.274 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.274 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:34.274 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.650024+0000", 2024-09-21T10:27:34.274 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:27:34.274 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.274 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.274 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:34.274 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.650103+0000", 2024-09-21T10:27:34.274 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9327999999999997e-05 2024-09-21T10:27:34.274 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.275 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.275 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:34.275 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.650113+0000", 2024-09-21T10:27:34.275 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.004e-05 2024-09-21T10:27:34.275 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.275 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.275 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:34.275 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.650118+0000", 2024-09-21T10:27:34.275 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2920000000000003e-06 2024-09-21T10:27:34.275 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.275 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.275 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:34.275 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.650170+0000", 2024-09-21T10:27:34.276 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1579000000000002e-05 2024-09-21T10:27:34.276 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.276 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.276 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:34.276 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.650302+0000", 2024-09-21T10:27:34.276 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013227200000000001 2024-09-21T10:27:34.276 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.276 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.276 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:27:34.276 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.650449+0000", 2024-09-21T10:27:34.276 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000147031 2024-09-21T10:27:34.276 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.276 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.276 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:34.277 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.651586+0000", 2024-09-21T10:27:34.277 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001136749 2024-09-21T10:27:34.277 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.277 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.277 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.277 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.654653+0000", 2024-09-21T10:27:34.277 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00306676 2024-09-21T10:27:34.277 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.277 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.277 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.277 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.764721+0000", 2024-09-21T10:27:34.277 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.11006858799999999 2024-09-21T10:27:34.277 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.278 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.278 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:34.278 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.764764+0000", 2024-09-21T10:27:34.278 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2369e-05 2024-09-21T10:27:34.278 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.278 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.278 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:34.278 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.764801+0000", 2024-09-21T10:27:34.278 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6826999999999998e-05 2024-09-21T10:27:34.278 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.278 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.278 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.278 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.278 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.279 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:28320 14.2 14:44134065:::benchmark_data_smithi107_312686_object28319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-21T10:27:34.279 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:23:08.655811+0000", 2024-09-21T10:27:34.279 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 265.55747925700001, 2024-09-21T10:27:34.279 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.11174126500000001, 2024-09-21T10:27:34.279 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:34.279 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:34.279 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:34.279 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:27:34.279 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:34.279 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 28320 2024-09-21T10:27:34.279 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.279 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:34.279 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.279 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:34.280 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.655811+0000", 2024-09-21T10:27:34.280 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.280 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.280 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.280 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:34.280 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.655811+0000", 2024-09-21T10:27:34.280 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.280 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.280 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.280 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:34.280 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.655811+0000", 2024-09-21T10:27:34.280 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:27:34.280 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.280 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.280 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:34.281 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.655860+0000", 2024-09-21T10:27:34.281 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8943999999999998e-05 2024-09-21T10:27:34.281 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.281 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.281 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:34.281 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.655867+0000", 2024-09-21T10:27:34.281 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.221e-06 2024-09-21T10:27:34.281 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.281 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.281 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:34.281 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.655870+0000", 2024-09-21T10:27:34.281 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.915e-06 2024-09-21T10:27:34.281 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.281 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.281 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:34.281 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.657021+0000", 2024-09-21T10:27:34.282 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0011509409999999999 2024-09-21T10:27:34.282 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.282 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.282 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:34.282 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.657110+0000", 2024-09-21T10:27:34.282 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.9431999999999993e-05 2024-09-21T10:27:34.282 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.282 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.282 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:27:34.282 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.657197+0000", 2024-09-21T10:27:34.282 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6895000000000001e-05 2024-09-21T10:27:34.282 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.282 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.282 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.283 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.764698+0000", 2024-09-21T10:27:34.283 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.107500413 2024-09-21T10:27:34.283 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.283 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.283 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.283 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.764921+0000", 2024-09-21T10:27:34.283 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00022341400000000001 2024-09-21T10:27:34.283 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.283 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.283 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:34.283 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.767461+0000", 2024-09-21T10:27:34.284 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002540156 2024-09-21T10:27:34.284 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.284 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.284 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:34.284 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.767505+0000", 2024-09-21T10:27:34.284 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3649999999999997e-05 2024-09-21T10:27:34.284 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.285 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.285 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:34.285 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.767553+0000", 2024-09-21T10:27:34.285 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7670999999999998e-05 2024-09-21T10:27:34.285 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.285 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.285 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.285 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.285 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.285 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:28321 14.4 14:2c49af65:::benchmark_data_smithi107_312686_object28320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-21T10:27:34.285 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:23:08.657485+0000", 2024-09-21T10:27:34.285 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 265.55580594899999, 2024-09-21T10:27:34.285 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.112163366, 2024-09-21T10:27:34.285 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:34.286 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:34.286 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:34.286 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:27:34.286 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:34.286 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 28321 2024-09-21T10:27:34.286 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.286 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:34.286 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.286 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:34.286 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.657485+0000", 2024-09-21T10:27:34.286 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.286 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.286 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.286 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:34.286 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.657485+0000", 2024-09-21T10:27:34.287 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.287 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.287 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.287 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:34.287 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.657484+0000", 2024-09-21T10:27:34.287 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:34.287 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.287 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.287 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:34.287 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.657639+0000", 2024-09-21T10:27:34.287 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015507300000000001 2024-09-21T10:27:34.287 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.288 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.288 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:34.288 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.763425+0000", 2024-09-21T10:27:34.288 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.105786238 2024-09-21T10:27:34.288 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.288 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.288 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:34.288 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.763437+0000", 2024-09-21T10:27:34.288 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2276000000000001e-05 2024-09-21T10:27:34.288 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.288 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.288 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:34.288 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.763495+0000", 2024-09-21T10:27:34.288 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7927999999999998e-05 2024-09-21T10:27:34.288 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.289 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.289 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:34.289 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.763645+0000", 2024-09-21T10:27:34.289 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000149476 2024-09-21T10:27:34.289 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.289 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.289 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:27:34.289 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.763795+0000", 2024-09-21T10:27:34.289 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000150087 2024-09-21T10:27:34.289 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.289 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.289 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.289 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.767850+0000", 2024-09-21T10:27:34.289 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0040555410000000002 2024-09-21T10:27:34.290 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.290 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.290 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.290 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.767885+0000", 2024-09-21T10:27:34.290 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4826000000000002e-05 2024-09-21T10:27:34.290 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.290 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.290 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:34.290 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.769522+0000", 2024-09-21T10:27:34.290 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0016363829999999999 2024-09-21T10:27:34.290 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.290 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.290 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:34.290 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.769569+0000", 2024-09-21T10:27:34.291 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6894000000000002e-05 2024-09-21T10:27:34.291 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.291 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.291 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:34.291 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:08.769648+0000", 2024-09-21T10:27:34.291 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.9491000000000002e-05 2024-09-21T10:27:34.291 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.291 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.291 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.291 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.291 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.291 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:32919 14.2 14:427e62e5:::benchmark_data_smithi107_312686_object32918:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-21T10:27:34.291 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:23:11.296297+0000", 2024-09-21T10:27:34.291 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 262.91699364599998, 2024-09-21T10:27:34.291 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.131622454, 2024-09-21T10:27:34.292 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:34.292 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:34.292 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:34.292 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:27:34.292 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:34.292 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 32919 2024-09-21T10:27:34.292 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.292 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:34.292 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.292 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:34.292 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.296297+0000", 2024-09-21T10:27:34.292 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.292 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.292 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.292 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:34.293 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.296297+0000", 2024-09-21T10:27:34.293 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.293 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.293 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.293 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:34.293 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.296296+0000", 2024-09-21T10:27:34.293 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:34.293 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.293 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.293 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:34.293 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.296421+0000", 2024-09-21T10:27:34.293 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012501100000000001 2024-09-21T10:27:34.293 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.293 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.294 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:34.294 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.296432+0000", 2024-09-21T10:27:34.294 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1276e-05 2024-09-21T10:27:34.294 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.294 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.294 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:34.294 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.296438+0000", 2024-09-21T10:27:34.294 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0120000000000002e-06 2024-09-21T10:27:34.294 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.294 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.294 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:34.294 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.296471+0000", 2024-09-21T10:27:34.294 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2566e-05 2024-09-21T10:27:34.294 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.294 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.295 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:34.295 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.296572+0000", 2024-09-21T10:27:34.295 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000100957 2024-09-21T10:27:34.295 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.295 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.295 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:27:34.295 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.296681+0000", 2024-09-21T10:27:34.295 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000108874 2024-09-21T10:27:34.295 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.295 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.295 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:34.295 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.297778+0000", 2024-09-21T10:27:34.295 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001097338 2024-09-21T10:27:34.295 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.295 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.296 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.296 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.427397+0000", 2024-09-21T10:27:34.296 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.129619022 2024-09-21T10:27:34.296 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.296 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.296 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.296 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.427838+0000", 2024-09-21T10:27:34.296 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00044068699999999999 2024-09-21T10:27:34.296 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.296 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.296 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:34.296 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.427875+0000", 2024-09-21T10:27:34.296 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7305000000000003e-05 2024-09-21T10:27:34.296 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.297 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.297 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:34.297 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:11.427919+0000", 2024-09-21T10:27:34.297 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4286000000000003e-05 2024-09-21T10:27:34.297 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.297 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.297 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.297 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.297 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.297 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:128866 14.4 14:2e6905ff:::benchmark_data_smithi107_312686_object13075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-09-21T10:27:34.297 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:24:42.156294+0000", 2024-09-21T10:27:34.297 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 172.056996841, 2024-09-21T10:27:34.297 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.10921127899999999, 2024-09-21T10:27:34.297 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:34.297 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:34.297 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:34.298 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:27:34.298 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:34.298 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 128866 2024-09-21T10:27:34.298 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.298 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:34.298 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.298 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:34.298 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:24:42.156294+0000", 2024-09-21T10:27:34.298 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.298 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.298 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.298 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:34.298 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:24:42.156294+0000", 2024-09-21T10:27:34.299 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.299 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.299 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.299 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:34.299 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:24:42.156293+0000", 2024-09-21T10:27:34.299 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:34.299 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.299 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.299 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:34.299 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:24:42.156299+0000", 2024-09-21T10:27:34.299 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1299999999999998e-06 2024-09-21T10:27:34.299 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.299 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.299 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:34.299 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:24:42.156322+0000", 2024-09-21T10:27:34.300 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.3703e-05 2024-09-21T10:27:34.300 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.300 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.300 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:34.300 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:24:42.156329+0000", 2024-09-21T10:27:34.300 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6309999999999997e-06 2024-09-21T10:27:34.300 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.300 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.300 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:34.300 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:24:42.156395+0000", 2024-09-21T10:27:34.300 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.5993000000000003e-05 2024-09-21T10:27:34.300 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.300 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.300 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:34.300 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:24:42.156508+0000", 2024-09-21T10:27:34.301 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000112895 2024-09-21T10:27:34.301 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.301 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.301 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:27:34.301 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:24:42.156604+0000", 2024-09-21T10:27:34.301 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.5673000000000004e-05 2024-09-21T10:27:34.301 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.301 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.301 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:34.301 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:24:42.164080+0000", 2024-09-21T10:27:34.301 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.007476033 2024-09-21T10:27:34.301 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.301 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.301 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.301 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:24:42.170180+0000", 2024-09-21T10:27:34.301 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0061006339999999997 2024-09-21T10:27:34.302 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.302 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.302 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.302 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:24:42.265412+0000", 2024-09-21T10:27:34.302 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.095231952999999994 2024-09-21T10:27:34.302 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.302 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.302 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:34.302 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:24:42.265462+0000", 2024-09-21T10:27:34.302 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9740999999999997e-05 2024-09-21T10:27:34.302 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.302 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.302 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:34.302 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:24:42.265505+0000", 2024-09-21T10:27:34.302 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3078000000000003e-05 2024-09-21T10:27:34.303 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.303 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.303 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.303 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.303 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.303 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:155133 14.4 14:2219ced5:::benchmark_data_smithi107_312686_object39342:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-09-21T10:27:34.303 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:25:14.621447+0000", 2024-09-21T10:27:34.303 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 139.59184336499999, 2024-09-21T10:27:34.303 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.24242121699999999, 2024-09-21T10:27:34.303 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:34.303 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:34.303 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:34.303 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:27:34.303 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:34.303 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 155133 2024-09-21T10:27:34.304 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.304 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:34.304 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.304 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:34.304 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.621447+0000", 2024-09-21T10:27:34.304 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.304 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.304 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.304 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:34.304 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.621447+0000", 2024-09-21T10:27:34.304 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.304 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.304 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.304 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:34.304 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.621445+0000", 2024-09-21T10:27:34.305 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:27:34.305 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.305 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.305 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:34.305 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.621454+0000", 2024-09-21T10:27:34.305 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.0799999999999995e-06 2024-09-21T10:27:34.305 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.305 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.305 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:34.305 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.621471+0000", 2024-09-21T10:27:34.305 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.6957999999999999e-05 2024-09-21T10:27:34.305 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.305 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.305 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:34.305 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.621478+0000", 2024-09-21T10:27:34.306 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8369999999999999e-06 2024-09-21T10:27:34.306 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.306 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.306 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:34.306 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.621530+0000", 2024-09-21T10:27:34.306 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2154999999999998e-05 2024-09-21T10:27:34.306 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.306 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.306 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:34.306 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.621664+0000", 2024-09-21T10:27:34.306 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013410599999999999 2024-09-21T10:27:34.306 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.306 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.306 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:27:34.307 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.621778+0000", 2024-09-21T10:27:34.307 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011373499999999999 2024-09-21T10:27:34.307 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.307 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.307 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.307 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.628828+0000", 2024-09-21T10:27:34.307 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.007049619 2024-09-21T10:27:34.307 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.307 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.307 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:34.307 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.631656+0000", 2024-09-21T10:27:34.307 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0028282390000000002 2024-09-21T10:27:34.307 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.307 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.308 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.308 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.863811+0000", 2024-09-21T10:27:34.308 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23215490899999999 2024-09-21T10:27:34.308 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.308 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.308 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:34.308 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.863843+0000", 2024-09-21T10:27:34.308 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1798999999999998e-05 2024-09-21T10:27:34.308 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.308 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.308 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:34.308 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.863868+0000", 2024-09-21T10:27:34.308 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.5734000000000001e-05 2024-09-21T10:27:34.308 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.309 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.309 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.309 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.309 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.309 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:155135 14.4 14:23054d4f:::benchmark_data_smithi107_312686_object39344:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-09-21T10:27:34.309 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:25:14.622872+0000", 2024-09-21T10:27:34.309 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 139.590418567, 2024-09-21T10:27:34.309 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.24672826, 2024-09-21T10:27:34.309 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:34.309 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:34.309 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:34.309 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:27:34.309 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:34.309 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 155135 2024-09-21T10:27:34.309 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.310 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:34.310 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.310 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:34.310 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.622872+0000", 2024-09-21T10:27:34.310 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.310 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.310 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.310 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:34.310 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.622872+0000", 2024-09-21T10:27:34.310 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.310 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.310 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.310 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:34.310 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.622871+0000", 2024-09-21T10:27:34.310 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:34.311 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.311 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.311 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:34.311 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.622877+0000", 2024-09-21T10:27:34.311 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.251e-06 2024-09-21T10:27:34.311 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.311 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.311 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:34.311 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.622894+0000", 2024-09-21T10:27:34.311 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.6456999999999998e-05 2024-09-21T10:27:34.311 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.311 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.311 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:34.311 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.622900+0000", 2024-09-21T10:27:34.311 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0129999999999999e-06 2024-09-21T10:27:34.312 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.312 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.312 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:34.312 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.625587+0000", 2024-09-21T10:27:34.312 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002686917 2024-09-21T10:27:34.312 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.312 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.312 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:34.312 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.625703+0000", 2024-09-21T10:27:34.312 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000116056 2024-09-21T10:27:34.312 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.312 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.312 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:27:34.312 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.625823+0000", 2024-09-21T10:27:34.312 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012066300000000001 2024-09-21T10:27:34.313 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.313 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.313 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:34.313 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.631664+0000", 2024-09-21T10:27:34.313 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0058406370000000001 2024-09-21T10:27:34.313 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.313 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.313 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.313 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.632395+0000", 2024-09-21T10:27:34.313 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00073145999999999997 2024-09-21T10:27:34.313 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.313 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.313 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.313 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.869521+0000", 2024-09-21T10:27:34.314 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.237125587 2024-09-21T10:27:34.314 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.314 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.314 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:34.314 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.869571+0000", 2024-09-21T10:27:34.314 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9836000000000002e-05 2024-09-21T10:27:34.314 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.314 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.314 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:34.314 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.869600+0000", 2024-09-21T10:27:34.314 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.9522999999999998e-05 2024-09-21T10:27:34.314 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.314 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.314 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.315 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.315 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.315 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:155140 14.2 14:4831ead0:::benchmark_data_smithi107_312686_object39349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-09-21T10:27:34.315 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:25:14.626642+0000", 2024-09-21T10:27:34.315 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 139.58664830999999, 2024-09-21T10:27:34.315 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.232181416, 2024-09-21T10:27:34.315 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:34.315 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:34.315 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:34.315 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:27:34.315 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:34.315 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 155140 2024-09-21T10:27:34.315 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.315 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:34.315 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.316 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:34.316 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.626642+0000", 2024-09-21T10:27:34.316 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.316 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.316 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.316 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:34.316 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.626642+0000", 2024-09-21T10:27:34.316 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.316 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.316 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.316 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:34.316 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.626641+0000", 2024-09-21T10:27:34.316 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:27:34.316 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.316 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.317 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:34.317 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.626647+0000", 2024-09-21T10:27:34.317 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2929999999999997e-06 2024-09-21T10:27:34.317 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.317 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.317 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:34.317 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.626659+0000", 2024-09-21T10:27:34.317 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1554e-05 2024-09-21T10:27:34.317 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.317 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.317 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:34.317 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.626664+0000", 2024-09-21T10:27:34.317 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3299999999999998e-06 2024-09-21T10:27:34.317 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.317 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.318 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:34.318 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.626712+0000", 2024-09-21T10:27:34.318 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8180999999999998e-05 2024-09-21T10:27:34.318 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.318 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.318 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:34.318 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.626822+0000", 2024-09-21T10:27:34.318 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000109245 2024-09-21T10:27:34.318 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.318 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.318 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:27:34.318 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.626928+0000", 2024-09-21T10:27:34.318 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010623100000000001 2024-09-21T10:27:34.318 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.319 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.319 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:34.319 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.630647+0000", 2024-09-21T10:27:34.319 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.003719254 2024-09-21T10:27:34.319 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.319 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.319 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.319 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.631839+0000", 2024-09-21T10:27:34.319 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00119148 2024-09-21T10:27:34.319 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.319 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.319 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.319 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.858734+0000", 2024-09-21T10:27:34.319 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.22689510600000001 2024-09-21T10:27:34.319 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.320 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.320 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:34.320 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.858785+0000", 2024-09-21T10:27:34.320 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1181999999999999e-05 2024-09-21T10:27:34.320 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.320 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.320 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:34.320 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.858824+0000", 2024-09-21T10:27:34.320 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.8779999999999998e-05 2024-09-21T10:27:34.320 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.320 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.320 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.320 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.320 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.320 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:155141 14.4 14:208be84b:::benchmark_data_smithi107_312686_object39350:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-09-21T10:27:34.321 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:25:14.626755+0000", 2024-09-21T10:27:34.321 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 139.58653606600001, 2024-09-21T10:27:34.321 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.242946513, 2024-09-21T10:27:34.321 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:34.321 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:34.321 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:34.321 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:27:34.321 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:34.321 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 155141 2024-09-21T10:27:34.321 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.321 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:34.321 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.321 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:34.321 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.626755+0000", 2024-09-21T10:27:34.321 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.321 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.322 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.322 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:34.322 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.626755+0000", 2024-09-21T10:27:34.322 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.322 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.322 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.322 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:34.322 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.626754+0000", 2024-09-21T10:27:34.322 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:27:34.322 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.322 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.322 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:34.322 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.626758+0000", 2024-09-21T10:27:34.322 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6969999999999999e-06 2024-09-21T10:27:34.323 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.323 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.323 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:34.323 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.626765+0000", 2024-09-21T10:27:34.323 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2819999999999999e-06 2024-09-21T10:27:34.323 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.323 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.323 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:34.323 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.626769+0000", 2024-09-21T10:27:34.323 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2729999999999996e-06 2024-09-21T10:27:34.323 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.323 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.323 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:34.324 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.628858+0000", 2024-09-21T10:27:34.324 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0020885560000000001 2024-09-21T10:27:34.324 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.324 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.324 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:34.324 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.628958+0000", 2024-09-21T10:27:34.324 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.9887000000000001e-05 2024-09-21T10:27:34.324 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.324 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.324 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:27:34.324 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.629053+0000", 2024-09-21T10:27:34.324 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.5742000000000006e-05 2024-09-21T10:27:34.324 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.325 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.325 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:34.325 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.631785+0000", 2024-09-21T10:27:34.325 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0027319739999999999 2024-09-21T10:27:34.325 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.325 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.325 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.325 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.632474+0000", 2024-09-21T10:27:34.325 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00068839300000000003 2024-09-21T10:27:34.325 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.325 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.325 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.325 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.869637+0000", 2024-09-21T10:27:34.325 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23716302 2024-09-21T10:27:34.325 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.326 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.326 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:34.326 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.869667+0000", 2024-09-21T10:27:34.326 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0429e-05 2024-09-21T10:27:34.326 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.326 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.326 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:34.326 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.869701+0000", 2024-09-21T10:27:34.326 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.3878000000000003e-05 2024-09-21T10:27:34.326 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.326 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.326 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.326 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.327 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.327 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:155148 14.2 14:4de1c040:::benchmark_data_smithi107_312686_object39357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-09-21T10:27:34.327 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:25:14.634782+0000", 2024-09-21T10:27:34.327 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 139.578508384, 2024-09-21T10:27:34.327 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.22431262399999999, 2024-09-21T10:27:34.327 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:34.327 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:34.327 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:34.327 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:27:34.327 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:34.327 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 155148 2024-09-21T10:27:34.327 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.327 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:34.327 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.328 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:34.328 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634782+0000", 2024-09-21T10:27:34.328 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.328 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.328 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.328 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:34.328 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634782+0000", 2024-09-21T10:27:34.328 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.328 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.328 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.328 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:34.328 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634777+0000", 2024-09-21T10:27:34.328 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999943 2024-09-21T10:27:34.328 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.328 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.328 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:34.329 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634788+0000", 2024-09-21T10:27:34.329 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1168999999999999e-05 2024-09-21T10:27:34.329 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.329 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.329 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:34.329 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634802+0000", 2024-09-21T10:27:34.329 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4398e-05 2024-09-21T10:27:34.329 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.329 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.329 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:34.329 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634808+0000", 2024-09-21T10:27:34.329 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2330000000000003e-06 2024-09-21T10:27:34.329 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.329 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.329 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:34.330 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634835+0000", 2024-09-21T10:27:34.330 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.6947000000000001e-05 2024-09-21T10:27:34.330 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.330 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.330 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:34.330 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634912+0000", 2024-09-21T10:27:34.330 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.7052999999999996e-05 2024-09-21T10:27:34.330 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.330 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.330 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:27:34.330 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634991+0000", 2024-09-21T10:27:34.330 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.8944999999999997e-05 2024-09-21T10:27:34.330 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.330 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.331 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:34.331 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.637871+0000", 2024-09-21T10:27:34.331 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0028802060000000002 2024-09-21T10:27:34.331 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.331 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.331 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.331 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.638127+0000", 2024-09-21T10:27:34.331 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00025533000000000001 2024-09-21T10:27:34.331 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.331 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.331 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.331 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.859012+0000", 2024-09-21T10:27:34.331 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.22088513000000001 2024-09-21T10:27:34.332 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.332 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.332 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:34.332 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.859059+0000", 2024-09-21T10:27:34.332 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7259000000000003e-05 2024-09-21T10:27:34.332 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.332 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.332 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:34.332 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.859095+0000", 2024-09-21T10:27:34.332 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5692999999999999e-05 2024-09-21T10:27:34.332 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.332 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.332 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.332 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.333 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.333 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:155149 14.4 14:2826e1d7:::benchmark_data_smithi107_312686_object39358:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-09-21T10:27:34.333 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:25:14.634823+0000", 2024-09-21T10:27:34.333 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 139.57846717199999, 2024-09-21T10:27:34.333 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.23870488100000001, 2024-09-21T10:27:34.333 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:34.333 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:34.333 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:34.333 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:27:34.333 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:34.333 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 155149 2024-09-21T10:27:34.333 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.333 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:34.333 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.333 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:34.334 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634823+0000", 2024-09-21T10:27:34.334 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.334 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.334 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.334 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:34.334 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634823+0000", 2024-09-21T10:27:34.334 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:34.334 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.334 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.334 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:34.334 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634823+0000", 2024-09-21T10:27:34.334 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:27:34.334 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.334 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.335 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:34.335 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634826+0000", 2024-09-21T10:27:34.335 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.4230000000000001e-06 2024-09-21T10:27:34.335 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.335 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.335 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:34.335 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634833+0000", 2024-09-21T10:27:34.335 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.0500000000000003e-06 2024-09-21T10:27:34.335 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.335 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.335 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:34.335 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634838+0000", 2024-09-21T10:27:34.335 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.147e-06 2024-09-21T10:27:34.335 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.336 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.336 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:34.336 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634871+0000", 2024-09-21T10:27:34.336 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2966000000000003e-05 2024-09-21T10:27:34.336 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.336 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.336 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:34.336 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634952+0000", 2024-09-21T10:27:34.336 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.0579000000000002e-05 2024-09-21T10:27:34.336 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.336 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.336 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:27:34.336 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.635038+0000", 2024-09-21T10:27:34.336 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6322000000000005e-05 2024-09-21T10:27:34.337 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.337 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.337 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:34.337 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.637877+0000", 2024-09-21T10:27:34.337 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002838255 2024-09-21T10:27:34.337 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.337 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.337 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.337 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.638014+0000", 2024-09-21T10:27:34.337 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013714599999999999 2024-09-21T10:27:34.337 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.337 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.337 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:34.338 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.873470+0000", 2024-09-21T10:27:34.338 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.235456101 2024-09-21T10:27:34.338 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.338 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.338 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:34.338 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.873499+0000", 2024-09-21T10:27:34.338 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.8671999999999999e-05 2024-09-21T10:27:34.338 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:34.338 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:34.338 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:34.338 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.873528+0000", 2024-09-21T10:27:34.338 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.9821000000000001e-05 2024-09-21T10:27:34.338 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.338 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.339 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.339 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:34.339 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:34.339 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:27:34.460 INFO:teuthology.orchestra.run.smithi107.stdout:mon.a: {} 2024-09-21T10:27:34.460 INFO:teuthology.orchestra.run.smithi107.stdout:mon.b: {} 2024-09-21T10:27:34.460 INFO:teuthology.orchestra.run.smithi107.stdout:mon.c: {} 2024-09-21T10:27:34.460 INFO:teuthology.orchestra.run.smithi107.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:27:34.461 INFO:teuthology.orchestra.run.smithi107.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:27:34.461 INFO:teuthology.orchestra.run.smithi107.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:27:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:34 smithi107 ceph-mon[102221]: pgmap v1614: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 32 KiB/s, 0 objects/s recovering 2024-09-21T10:27:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:34 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1619734279' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:27:35.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:34 smithi107 ceph-mon[105022]: pgmap v1614: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 32 KiB/s, 0 objects/s recovering 2024-09-21T10:27:35.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:34 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1619734279' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:27:35.132 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:27:35.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T10:27:35.376 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:27:35.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:34 smithi129 ceph-mon[88471]: pgmap v1614: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 32 KiB/s, 0 objects/s recovering 2024-09-21T10:27:35.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:34 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1619734279' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:27:35.469 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:37.016 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:27:37.016 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:27:37.016 INFO:teuthology.orchestra.run.smithi129.stdout: "num_ops": 0 2024-09-21T10:27:37.016 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:27:37.307 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:36 smithi129 ceph-mon[88471]: pgmap v1615: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 32 KiB/s, 0 objects/s recovering 2024-09-21T10:27:37.308 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:36 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:27:37.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:36 smithi107 ceph-mon[102221]: pgmap v1615: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 32 KiB/s, 0 objects/s recovering 2024-09-21T10:27:37.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:36 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:27:37.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:36 smithi107 ceph-mon[105022]: pgmap v1615: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 32 KiB/s, 0 objects/s recovering 2024-09-21T10:27:37.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:36 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:27:37.636 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:27:37.881 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:27:37.983 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T10:27:38.252 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:37 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4190061355' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:27:38.306 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:37 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4190061355' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:27:38.306 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:37 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4190061355' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:27:38.737 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 1 1 2024-09-21T10:27:38.983 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:39.243 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:38 smithi129 ceph-mon[88471]: pgmap v1616: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 30 KiB/s, 0 objects/s recovering 2024-09-21T10:27:39.243 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:38 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4190061355' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:27:39.243 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:38 smithi129 ceph-mon[88471]: osdmap e454: 8 total, 8 up, 8 in 2024-09-21T10:27:39.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:38 smithi107 ceph-mon[102221]: pgmap v1616: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 30 KiB/s, 0 objects/s recovering 2024-09-21T10:27:39.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:38 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4190061355' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:27:39.244 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:38 smithi107 ceph-mon[102221]: osdmap e454: 8 total, 8 up, 8 in 2024-09-21T10:27:39.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:38 smithi107 ceph-mon[105022]: pgmap v1616: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 30 KiB/s, 0 objects/s recovering 2024-09-21T10:27:39.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:38 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4190061355' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:27:39.245 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:38 smithi107 ceph-mon[105022]: osdmap e454: 8 total, 8 up, 8 in 2024-09-21T10:27:39.594 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:27:39.594 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [], 2024-09-21T10:27:39.594 INFO:teuthology.orchestra.run.smithi129.stdout: "complaint_time": 30, 2024-09-21T10:27:39.595 INFO:teuthology.orchestra.run.smithi129.stdout: "num_blocked_ops": 0 2024-09-21T10:27:39.595 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:27:40.357 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:27:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:27:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:27:40.384 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:27:40.631 INFO:teuthology.orchestra.run.smithi129.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:27:40.987 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-21T10:27:41.111 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:40 smithi129 ceph-mon[88471]: pgmap v1618: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:41.111 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:40 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1993759300' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:27:41.339 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:40 smithi107 ceph-mon[102221]: pgmap v1618: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:41.340 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:40 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1993759300' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:27:41.340 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:40 smithi107 ceph-mon[105022]: pgmap v1618: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:41.340 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:40 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1993759300' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:27:41.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T10:27:41.994 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:27:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:27:41] "GET /metrics HTTP/1.1" 200 34369 "" "Prometheus/2.51.0" 2024-09-21T10:27:42.004 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:42.104 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:41 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1993759300' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:27:42.104 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:41 smithi129 ceph-mon[88471]: osdmap e455: 8 total, 8 up, 8 in 2024-09-21T10:27:42.104 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:27:42.104 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:42.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:41 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1993759300' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:27:42.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:41 smithi107 ceph-mon[102221]: osdmap e455: 8 total, 8 up, 8 in 2024-09-21T10:27:42.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:27:42.296 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:42.296 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:41 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1993759300' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:27:42.296 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:41 smithi107 ceph-mon[105022]: osdmap e455: 8 total, 8 up, 8 in 2024-09-21T10:27:42.296 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:27:42.296 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:42.378 INFO:teuthology.orchestra.run.smithi129.stdout:{ 2024-09-21T10:27:42.378 INFO:teuthology.orchestra.run.smithi129.stdout: "size": 20, 2024-09-21T10:27:42.379 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 600, 2024-09-21T10:27:42.379 INFO:teuthology.orchestra.run.smithi129.stdout: "ops": [ 2024-09-21T10:27:42.379 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.379 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:26945 13.0 13:0d6e0250:::benchmark_data_smithi107_278655_object26944:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:27:42.379 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:31.497184+0000", 2024-09-21T10:27:42.379 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 550.87741880900001, 2024-09-21T10:27:42.379 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.29837365799999999, 2024-09-21T10:27:42.379 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:42.379 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:42.379 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:42.379 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:42.379 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:42.379 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 26945 2024-09-21T10:27:42.380 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.380 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:42.380 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.380 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:42.380 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.497184+0000", 2024-09-21T10:27:42.380 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.380 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.380 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.380 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:42.380 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.497184+0000", 2024-09-21T10:27:42.380 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.380 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.380 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.381 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:42.381 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.497183+0000", 2024-09-21T10:27:42.381 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:27:42.381 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.381 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.381 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:42.381 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.497221+0000", 2024-09-21T10:27:42.381 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.7901000000000002e-05 2024-09-21T10:27:42.381 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.381 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.381 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:42.381 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.497235+0000", 2024-09-21T10:27:42.381 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3508e-05 2024-09-21T10:27:42.381 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.381 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.381 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:42.382 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.497241+0000", 2024-09-21T10:27:42.382 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.1040000000000003e-06 2024-09-21T10:27:42.382 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.382 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.382 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.382 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.497290+0000", 2024-09-21T10:27:42.382 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.8912999999999997e-05 2024-09-21T10:27:42.382 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.382 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.382 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:42.382 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.497385+0000", 2024-09-21T10:27:42.382 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.5767999999999995e-05 2024-09-21T10:27:42.382 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.382 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.383 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:27:42.383 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.497545+0000", 2024-09-21T10:27:42.383 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00015969700000000001 2024-09-21T10:27:42.383 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.383 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.383 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:42.383 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.498780+0000", 2024-09-21T10:27:42.383 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0012350200000000001 2024-09-21T10:27:42.383 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.383 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.383 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.383 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.499758+0000", 2024-09-21T10:27:42.383 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00097847899999999998 2024-09-21T10:27:42.383 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.384 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.384 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.384 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.795468+0000", 2024-09-21T10:27:42.384 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.29570952499999997 2024-09-21T10:27:42.384 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.384 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.384 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:42.384 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.795517+0000", 2024-09-21T10:27:42.384 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.9332000000000001e-05 2024-09-21T10:27:42.384 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.384 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.384 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:42.384 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.795558+0000", 2024-09-21T10:27:42.384 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0800000000000002e-05 2024-09-21T10:27:42.385 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.385 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.385 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.385 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.385 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.385 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:26977 13.0 13:080daddd:::benchmark_data_smithi107_278655_object26976:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-09-21T10:27:42.385 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:18:31.513658+0000", 2024-09-21T10:27:42.385 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 550.86094559200001, 2024-09-21T10:27:42.385 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.282703076, 2024-09-21T10:27:42.385 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:42.385 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:42.385 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:42.385 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:42.385 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:42.385 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 26977 2024-09-21T10:27:42.385 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.385 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:42.386 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.386 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:42.386 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.513658+0000", 2024-09-21T10:27:42.386 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.386 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.386 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.386 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:42.386 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.513658+0000", 2024-09-21T10:27:42.386 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.386 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.386 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.386 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:42.386 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.513656+0000", 2024-09-21T10:27:42.387 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:27:42.387 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.387 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.387 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:42.387 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.513692+0000", 2024-09-21T10:27:42.387 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6168999999999999e-05 2024-09-21T10:27:42.387 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.387 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.387 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:42.387 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.513706+0000", 2024-09-21T10:27:42.387 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3378e-05 2024-09-21T10:27:42.387 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.387 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.387 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:42.388 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.513712+0000", 2024-09-21T10:27:42.388 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0719999999999996e-06 2024-09-21T10:27:42.388 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.388 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.388 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.388 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.513762+0000", 2024-09-21T10:27:42.388 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0167999999999998e-05 2024-09-21T10:27:42.388 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.388 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.388 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:42.388 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.513876+0000", 2024-09-21T10:27:42.388 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000113838 2024-09-21T10:27:42.388 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.388 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.388 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:27:42.389 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.514002+0000", 2024-09-21T10:27:42.389 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012649199999999999 2024-09-21T10:27:42.389 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.389 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.389 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:42.389 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.515246+0000", 2024-09-21T10:27:42.389 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.001243798 2024-09-21T10:27:42.389 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.389 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.389 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.389 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.515347+0000", 2024-09-21T10:27:42.389 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010134899999999999 2024-09-21T10:27:42.389 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.389 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.389 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.389 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.796275+0000", 2024-09-21T10:27:42.390 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.28092745699999999 2024-09-21T10:27:42.390 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.390 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.390 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:42.390 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.796318+0000", 2024-09-21T10:27:42.390 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3501999999999998e-05 2024-09-21T10:27:42.390 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.390 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.390 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:42.390 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:18:31.796361+0000", 2024-09-21T10:27:42.390 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2580000000000002e-05 2024-09-21T10:27:42.390 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.390 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.390 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.390 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.391 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.391 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:72764 13.0 13:095f2978:::benchmark_data_smithi107_278655_object72763:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-21T10:27:42.391 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:00.480369+0000", 2024-09-21T10:27:42.391 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 521.89423478100002, 2024-09-21T10:27:42.391 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.27071356299999999, 2024-09-21T10:27:42.391 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:42.391 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:42.391 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:42.391 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:42.391 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:42.391 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 72764 2024-09-21T10:27:42.391 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.392 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:42.392 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.392 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:42.392 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.480369+0000", 2024-09-21T10:27:42.392 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.392 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.392 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.392 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:42.392 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.480369+0000", 2024-09-21T10:27:42.392 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.392 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.392 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.393 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:42.393 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.480367+0000", 2024-09-21T10:27:42.393 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:42.393 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.393 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.393 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:42.393 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.480412+0000", 2024-09-21T10:27:42.393 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4873000000000002e-05 2024-09-21T10:27:42.393 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.393 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.393 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:42.393 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.480426+0000", 2024-09-21T10:27:42.393 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.4146e-05 2024-09-21T10:27:42.394 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.394 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.394 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:42.394 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.480432+0000", 2024-09-21T10:27:42.394 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.5099999999999998e-06 2024-09-21T10:27:42.394 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.394 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.394 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.394 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.480504+0000", 2024-09-21T10:27:42.394 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2225999999999996e-05 2024-09-21T10:27:42.394 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.394 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.394 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:42.395 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.480618+0000", 2024-09-21T10:27:42.395 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000113651 2024-09-21T10:27:42.395 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.395 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.395 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:27:42.395 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.480747+0000", 2024-09-21T10:27:42.395 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000129059 2024-09-21T10:27:42.395 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.395 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.395 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.395 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.483671+0000", 2024-09-21T10:27:42.395 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0029244420000000002 2024-09-21T10:27:42.395 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.396 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.396 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:42.396 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.484638+0000", 2024-09-21T10:27:42.396 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00096638300000000002 2024-09-21T10:27:42.396 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.396 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.396 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.396 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.750988+0000", 2024-09-21T10:27:42.396 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.26635043600000002 2024-09-21T10:27:42.396 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.396 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.396 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:42.396 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.751039+0000", 2024-09-21T10:27:42.396 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1309e-05 2024-09-21T10:27:42.397 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.397 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.397 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:42.397 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:00.751082+0000", 2024-09-21T10:27:42.397 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.2599999999999999e-05 2024-09-21T10:27:42.397 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.397 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.397 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.397 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.397 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.397 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:134448 13.7 13:e615f15f:::benchmark_data_smithi107_278655_object6541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:27:42.397 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:58.699336+0000", 2024-09-21T10:27:42.397 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 463.675267523, 2024-09-21T10:27:42.397 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1137340400000002, 2024-09-21T10:27:42.398 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:42.398 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:27:42.398 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:42.398 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:42.398 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:42.398 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 134448 2024-09-21T10:27:42.398 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.398 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:42.398 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.398 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:42.398 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699336+0000", 2024-09-21T10:27:42.398 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.398 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.398 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.399 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:42.399 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699336+0000", 2024-09-21T10:27:42.399 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.399 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.399 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.399 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:42.399 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699335+0000", 2024-09-21T10:27:42.399 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:42.399 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.399 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.399 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:42.399 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699340+0000", 2024-09-21T10:27:42.399 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4450000000000004e-06 2024-09-21T10:27:42.399 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.400 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.400 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:42.400 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699355+0000", 2024-09-21T10:27:42.400 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5224e-05 2024-09-21T10:27:42.400 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.400 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.400 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:42.400 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699361+0000", 2024-09-21T10:27:42.400 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.8379999999999998e-06 2024-09-21T10:27:42.400 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.400 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.400 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.400 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.701910+0000", 2024-09-21T10:27:42.400 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0025482109999999999 2024-09-21T10:27:42.401 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.401 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.401 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:42.401 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.702024+0000", 2024-09-21T10:27:42.401 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000114465 2024-09-21T10:27:42.401 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.401 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.401 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:27:42.401 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.702137+0000", 2024-09-21T10:27:42.401 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011343199999999999 2024-09-21T10:27:42.401 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.401 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.401 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:42.402 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.710167+0000", 2024-09-21T10:27:42.402 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0080300509999999999 2024-09-21T10:27:42.402 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.402 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.402 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.402 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.715118+0000", 2024-09-21T10:27:42.402 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0049505210000000003 2024-09-21T10:27:42.402 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.402 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.402 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.402 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.737774+0000", 2024-09-21T10:27:42.402 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.022655709 2024-09-21T10:27:42.402 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.402 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.403 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:27:42.403 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.737782+0000", 2024-09-21T10:27:42.403 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.4109999999999996e-06 2024-09-21T10:27:42.403 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.403 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.403 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.403 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601581+0000", 2024-09-21T10:27:42.403 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.86379862200000002 2024-09-21T10:27:42.403 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.403 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.403 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:27:42.403 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601593+0000", 2024-09-21T10:27:42.403 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2381000000000001e-05 2024-09-21T10:27:42.403 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.403 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.404 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.404 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.813002+0000", 2024-09-21T10:27:42.404 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2114087659999999 2024-09-21T10:27:42.404 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.404 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.404 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:42.404 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.813070+0000", 2024-09-21T10:27:42.404 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7929000000000006e-05 2024-09-21T10:27:42.404 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.404 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.404 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.404 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.404 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.404 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:134449 13.7 13:ec92a00a:::benchmark_data_smithi107_278655_object6542:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:27:42.404 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:58.699371+0000", 2024-09-21T10:27:42.405 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 463.67523251900002, 2024-09-21T10:27:42.405 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1152291590000001, 2024-09-21T10:27:42.405 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:42.405 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:27:42.405 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:42.405 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:42.405 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:42.405 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 134449 2024-09-21T10:27:42.405 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.405 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:42.405 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.406 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:42.406 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699371+0000", 2024-09-21T10:27:42.407 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.407 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.407 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.407 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:42.407 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699371+0000", 2024-09-21T10:27:42.407 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.407 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.407 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.407 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:42.407 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699370+0000", 2024-09-21T10:27:42.407 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:27:42.407 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.407 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.407 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:42.408 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699373+0000", 2024-09-21T10:27:42.408 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.3379999999999999e-06 2024-09-21T10:27:42.408 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.408 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.408 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:42.408 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699378+0000", 2024-09-21T10:27:42.408 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.4750000000000001e-06 2024-09-21T10:27:42.408 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.408 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.408 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:42.408 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699381+0000", 2024-09-21T10:27:42.408 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.2200000000000001e-06 2024-09-21T10:27:42.408 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.408 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.408 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.409 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.705719+0000", 2024-09-21T10:27:42.409 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0063375430000000002 2024-09-21T10:27:42.409 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.409 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.409 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:42.409 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.705831+0000", 2024-09-21T10:27:42.409 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000111987 2024-09-21T10:27:42.409 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.409 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.409 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:27:42.409 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.705945+0000", 2024-09-21T10:27:42.409 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011391999999999999 2024-09-21T10:27:42.409 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.409 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.410 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.410 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.715146+0000", 2024-09-21T10:27:42.410 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0092007930000000005 2024-09-21T10:27:42.410 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.410 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.410 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:42.410 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.725273+0000", 2024-09-21T10:27:42.410 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.010127196999999999 2024-09-21T10:27:42.410 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.410 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.410 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.410 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.737806+0000", 2024-09-21T10:27:42.410 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.012532896 2024-09-21T10:27:42.410 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.411 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.411 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:27:42.411 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.737812+0000", 2024-09-21T10:27:42.411 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9540000000000001e-06 2024-09-21T10:27:42.411 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.411 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.411 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.411 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601625+0000", 2024-09-21T10:27:42.411 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.86381372599999995 2024-09-21T10:27:42.411 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.411 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.411 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:27:42.411 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601632+0000", 2024-09-21T10:27:42.411 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6529999999999997e-06 2024-09-21T10:27:42.412 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.412 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.412 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.412 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.814527+0000", 2024-09-21T10:27:42.412 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.21289533 2024-09-21T10:27:42.412 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.412 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.412 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:42.412 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.814600+0000", 2024-09-21T10:27:42.412 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.25e-05 2024-09-21T10:27:42.412 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.412 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.412 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.413 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.413 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.413 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:134451 13.7 13:e3630b82:::benchmark_data_smithi107_278655_object6544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e379)", 2024-09-21T10:27:42.413 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:58.699385+0000", 2024-09-21T10:27:42.413 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 463.67521827899998, 2024-09-21T10:27:42.413 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1294433669999999, 2024-09-21T10:27:42.413 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:42.413 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:27:42.413 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:42.413 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:42.413 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:42.413 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 134451 2024-09-21T10:27:42.413 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.413 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:42.414 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.414 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:42.414 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699385+0000", 2024-09-21T10:27:42.414 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.414 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.414 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.414 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:42.414 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699385+0000", 2024-09-21T10:27:42.414 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.414 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.414 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.414 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:42.415 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699385+0000", 2024-09-21T10:27:42.415 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:27:42.415 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.415 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.415 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:42.415 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699387+0000", 2024-09-21T10:27:42.415 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1859999999999999e-06 2024-09-21T10:27:42.415 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.415 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.415 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:42.415 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699392+0000", 2024-09-21T10:27:42.415 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7820000000000001e-06 2024-09-21T10:27:42.415 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.415 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.415 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:42.416 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.699395+0000", 2024-09-21T10:27:42.416 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0479999999999999e-06 2024-09-21T10:27:42.416 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.416 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.416 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.416 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.710241+0000", 2024-09-21T10:27:42.416 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.010846369999999999 2024-09-21T10:27:42.416 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.416 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.416 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:42.416 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.710514+0000", 2024-09-21T10:27:42.416 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00027252599999999999 2024-09-21T10:27:42.416 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.416 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.417 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:27:42.417 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.710874+0000", 2024-09-21T10:27:42.417 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00036023799999999998 2024-09-21T10:27:42.417 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.417 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.417 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:42.417 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.725278+0000", 2024-09-21T10:27:42.417 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.014403978 2024-09-21T10:27:42.417 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.417 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.417 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.417 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.737834+0000", 2024-09-21T10:27:42.417 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.012555851999999999 2024-09-21T10:27:42.417 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.417 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.417 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:27:42.418 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.737840+0000", 2024-09-21T10:27:42.418 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.421e-06 2024-09-21T10:27:42.418 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.418 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.418 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.418 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601657+0000", 2024-09-21T10:27:42.418 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.86381711699999997 2024-09-21T10:27:42.418 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.418 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.418 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:27:42.418 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601664+0000", 2024-09-21T10:27:42.418 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7390000000000002e-06 2024-09-21T10:27:42.418 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.418 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.419 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.419 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.826620+0000", 2024-09-21T10:27:42.419 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2249555560000001 2024-09-21T10:27:42.419 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.419 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.419 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:42.419 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.828828+0000", 2024-09-21T10:27:42.419 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0022088670000000002 2024-09-21T10:27:42.419 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.419 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.419 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.419 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.419 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.419 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:134459 13.f 13:f700217e:::benchmark_data_smithi107_278655_object6552:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:27:42.419 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:58.722388+0000", 2024-09-21T10:27:42.420 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 463.65221496599997, 2024-09-21T10:27:42.420 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1032898339999999, 2024-09-21T10:27:42.420 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:42.420 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:27:42.420 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:42.420 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:42.420 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:42.420 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 134459 2024-09-21T10:27:42.420 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.420 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:42.420 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.420 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:42.420 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722388+0000", 2024-09-21T10:27:42.420 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.421 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.421 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.421 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:42.421 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722388+0000", 2024-09-21T10:27:42.421 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.421 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.421 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.421 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:42.421 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722387+0000", 2024-09-21T10:27:42.421 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:42.421 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.421 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.421 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:42.421 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722393+0000", 2024-09-21T10:27:42.421 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7649999999999999e-06 2024-09-21T10:27:42.422 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.422 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.422 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:42.422 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722402+0000", 2024-09-21T10:27:42.422 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.2590000000000006e-06 2024-09-21T10:27:42.422 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.422 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.422 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:42.422 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722414+0000", 2024-09-21T10:27:42.422 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1579e-05 2024-09-21T10:27:42.422 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.422 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.422 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.422 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.738001+0000", 2024-09-21T10:27:42.423 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.015586598 2024-09-21T10:27:42.423 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.423 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.423 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:27:42.423 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.738008+0000", 2024-09-21T10:27:42.423 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2590000000000003e-06 2024-09-21T10:27:42.423 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.423 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.423 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.423 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601716+0000", 2024-09-21T10:27:42.423 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.86370839799999999 2024-09-21T10:27:42.423 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.423 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.423 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:27:42.424 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601723+0000", 2024-09-21T10:27:42.424 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8569999999999997e-06 2024-09-21T10:27:42.424 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.424 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.424 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.424 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.825551+0000", 2024-09-21T10:27:42.424 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.223827674 2024-09-21T10:27:42.424 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.424 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.424 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:42.424 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.825678+0000", 2024-09-21T10:27:42.424 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000127451 2024-09-21T10:27:42.424 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.424 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.424 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.425 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.425 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.425 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:134462 13.f 13:f29b039a:::benchmark_data_smithi107_278655_object6555:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:27:42.425 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:58.722420+0000", 2024-09-21T10:27:42.425 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 463.65218358300001, 2024-09-21T10:27:42.425 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1428640240000001, 2024-09-21T10:27:42.425 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:42.425 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "reached pg", 2024-09-21T10:27:42.425 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:42.425 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:42.425 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:42.425 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 134462 2024-09-21T10:27:42.425 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.425 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:42.425 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.425 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:42.426 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722420+0000", 2024-09-21T10:27:42.426 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.426 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.426 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.426 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:42.426 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722420+0000", 2024-09-21T10:27:42.426 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.426 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.426 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.426 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:42.426 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722419+0000", 2024-09-21T10:27:42.426 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:27:42.426 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.426 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.427 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:42.427 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722422+0000", 2024-09-21T10:27:42.427 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0989999999999999e-06 2024-09-21T10:27:42.427 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.427 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.427 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:42.427 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722429+0000", 2024-09-21T10:27:42.427 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.4989999999999999e-06 2024-09-21T10:27:42.427 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.427 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.427 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:42.427 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.722433+0000", 2024-09-21T10:27:42.427 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.9060000000000004e-06 2024-09-21T10:27:42.427 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.428 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.428 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.428 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.738055+0000", 2024-09-21T10:27:42.428 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.015622761000000001 2024-09-21T10:27:42.428 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.428 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.428 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:27:42.428 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.738061+0000", 2024-09-21T10:27:42.428 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7919999999999998e-06 2024-09-21T10:27:42.428 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.428 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.428 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.428 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601742+0000", 2024-09-21T10:27:42.428 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.86368112100000005 2024-09-21T10:27:42.428 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.429 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.429 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:27:42.429 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601748+0000", 2024-09-21T10:27:42.429 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0530000000000003e-06 2024-09-21T10:27:42.429 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.429 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.429 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.429 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.826378+0000", 2024-09-21T10:27:42.429 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2246290879999999 2024-09-21T10:27:42.429 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.429 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.429 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for missing object", 2024-09-21T10:27:42.429 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.826477+0000", 2024-09-21T10:27:42.429 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.9401999999999995e-05 2024-09-21T10:27:42.429 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.429 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.430 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.430 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.865163+0000", 2024-09-21T10:27:42.430 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.038685936999999997 2024-09-21T10:27:42.430 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.430 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.430 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:42.430 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.865284+0000", 2024-09-21T10:27:42.430 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000120866 2024-09-21T10:27:42.430 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.430 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.430 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.430 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.430 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.430 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:134465 13.7 13:e628427d:::benchmark_data_smithi107_278655_object6558:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:27:42.431 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:58.723761+0000", 2024-09-21T10:27:42.431 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 463.650842662, 2024-09-21T10:27:42.431 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.124022085, 2024-09-21T10:27:42.431 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:42.431 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:42.431 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:42.431 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:42.431 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:42.431 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 134465 2024-09-21T10:27:42.431 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.431 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:42.431 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.431 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:42.431 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.723761+0000", 2024-09-21T10:27:42.431 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.432 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.432 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.432 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:42.432 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.723761+0000", 2024-09-21T10:27:42.432 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.432 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.432 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.432 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:42.432 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.723760+0000", 2024-09-21T10:27:42.432 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:42.432 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.432 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.432 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:42.432 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.723764+0000", 2024-09-21T10:27:42.432 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3200000000000001e-06 2024-09-21T10:27:42.433 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.433 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.433 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:42.433 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.723773+0000", 2024-09-21T10:27:42.433 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.6489999999999997e-06 2024-09-21T10:27:42.433 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.433 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.433 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:42.433 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.723778+0000", 2024-09-21T10:27:42.433 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6239999999999999e-06 2024-09-21T10:27:42.433 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.433 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.433 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.433 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.738027+0000", 2024-09-21T10:27:42.434 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.01424893 2024-09-21T10:27:42.434 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.434 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.434 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:27:42.434 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.738034+0000", 2024-09-21T10:27:42.434 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2500000000000003e-06 2024-09-21T10:27:42.434 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.434 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.434 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.434 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601687+0000", 2024-09-21T10:27:42.434 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.86365340599999996 2024-09-21T10:27:42.434 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.434 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.434 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:27:42.434 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601694+0000", 2024-09-21T10:27:42.435 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.579e-06 2024-09-21T10:27:42.435 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.435 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.435 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.435 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.828858+0000", 2024-09-21T10:27:42.435 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.227164178 2024-09-21T10:27:42.435 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.435 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.435 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:42.435 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.828974+0000", 2024-09-21T10:27:42.435 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011628700000000001 2024-09-21T10:27:42.435 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.435 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.435 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 4,5,6", 2024-09-21T10:27:42.435 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.829556+0000", 2024-09-21T10:27:42.435 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00058166700000000001 2024-09-21T10:27:42.436 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.436 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.436 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:42.436 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.843383+0000", 2024-09-21T10:27:42.436 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.013827694 2024-09-21T10:27:42.436 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.436 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.436 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.436 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.847499+0000", 2024-09-21T10:27:42.436 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0041160839999999999 2024-09-21T10:27:42.436 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.436 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.436 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.436 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.847530+0000", 2024-09-21T10:27:42.437 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.0508999999999999e-05 2024-09-21T10:27:42.437 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.437 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.437 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.437 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.847556+0000", 2024-09-21T10:27:42.437 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.5749999999999999e-05 2024-09-21T10:27:42.437 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.437 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.437 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:42.437 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.847660+0000", 2024-09-21T10:27:42.437 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000104048 2024-09-21T10:27:42.437 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.437 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.437 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:42.437 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.847783+0000", 2024-09-21T10:27:42.437 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000122922 2024-09-21T10:27:42.438 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.438 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.438 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.438 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.438 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.438 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:134466 13.f 13:fdbed5d6:::benchmark_data_smithi107_278655_object6559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:27:42.438 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:58.735102+0000", 2024-09-21T10:27:42.438 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 463.63950134499999, 2024-09-21T10:27:42.438 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1198004730000002, 2024-09-21T10:27:42.438 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:42.438 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:42.438 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:42.438 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:42.438 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:42.439 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 134466 2024-09-21T10:27:42.439 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.439 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:42.439 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.439 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:42.439 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.735102+0000", 2024-09-21T10:27:42.439 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.439 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.439 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.439 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:42.439 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.735102+0000", 2024-09-21T10:27:42.439 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.439 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.439 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.439 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:42.440 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.735101+0000", 2024-09-21T10:27:42.440 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:42.440 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.440 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.440 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:42.440 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.735106+0000", 2024-09-21T10:27:42.440 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.7040000000000002e-06 2024-09-21T10:27:42.440 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.440 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.440 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:42.440 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.735115+0000", 2024-09-21T10:27:42.440 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.8260000000000005e-06 2024-09-21T10:27:42.440 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.440 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.440 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:42.440 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.735122+0000", 2024-09-21T10:27:42.441 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.64e-06 2024-09-21T10:27:42.441 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.441 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.441 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.441 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.738080+0000", 2024-09-21T10:27:42.441 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.002957889 2024-09-21T10:27:42.441 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.441 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.441 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:27:42.441 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.738086+0000", 2024-09-21T10:27:42.441 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.2160000000000003e-06 2024-09-21T10:27:42.441 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.441 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.441 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.441 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601769+0000", 2024-09-21T10:27:42.441 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.86368243499999997 2024-09-21T10:27:42.442 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.442 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.442 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:27:42.442 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.601775+0000", 2024-09-21T10:27:42.442 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6290000000000004e-06 2024-09-21T10:27:42.442 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.442 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.442 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.442 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.826712+0000", 2024-09-21T10:27:42.442 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2249370209999999 2024-09-21T10:27:42.442 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.442 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.442 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:42.442 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.826829+0000", 2024-09-21T10:27:42.443 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000116885 2024-09-21T10:27:42.443 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.443 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.443 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,5,6", 2024-09-21T10:27:42.443 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.827459+0000", 2024-09-21T10:27:42.443 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00062943700000000005 2024-09-21T10:27:42.443 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.443 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.443 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.443 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.839442+0000", 2024-09-21T10:27:42.443 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.011983053 2024-09-21T10:27:42.443 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.443 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.444 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.444 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.847816+0000", 2024-09-21T10:27:42.444 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0083745539999999993 2024-09-21T10:27:42.444 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.444 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.444 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.444 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.854585+0000", 2024-09-21T10:27:42.444 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0067689880000000001 2024-09-21T10:27:42.444 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.444 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.444 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:42.444 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.854652+0000", 2024-09-21T10:27:42.444 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6283000000000002e-05 2024-09-21T10:27:42.444 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.445 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.445 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:42.445 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.854688+0000", 2024-09-21T10:27:42.445 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.6705000000000002e-05 2024-09-21T10:27:42.445 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.445 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.445 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:42.445 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.854902+0000", 2024-09-21T10:27:42.445 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00021421 2024-09-21T10:27:42.445 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.445 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.445 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.445 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.445 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.445 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:134472 13.7 13:ea565dde:::benchmark_data_smithi107_278655_object6565:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-21T10:27:42.446 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:19:58.743620+0000", 2024-09-21T10:27:42.446 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 463.63098366600002, 2024-09-21T10:27:42.446 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.1110101480000001, 2024-09-21T10:27:42.446 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:42.446 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:42.446 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:42.446 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:42.446 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:42.446 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 134472 2024-09-21T10:27:42.446 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.446 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:42.446 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.446 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:42.446 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.743620+0000", 2024-09-21T10:27:42.447 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.447 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.447 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.447 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:42.447 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.743620+0000", 2024-09-21T10:27:42.447 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.447 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.447 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.447 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:42.447 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.743618+0000", 2024-09-21T10:27:42.447 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:27:42.447 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.447 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.447 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:42.448 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.743626+0000", 2024-09-21T10:27:42.448 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.34e-06 2024-09-21T10:27:42.448 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.448 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.448 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:42.448 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.743642+0000", 2024-09-21T10:27:42.448 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.6957e-05 2024-09-21T10:27:42.448 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.448 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.448 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:42.448 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.743650+0000", 2024-09-21T10:27:42.448 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.2119999999999997e-06 2024-09-21T10:27:42.448 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.448 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.448 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.449 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.743703+0000", 2024-09-21T10:27:42.449 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.2948999999999997e-05 2024-09-21T10:27:42.449 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.449 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.449 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:27:42.449 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:58.743716+0000", 2024-09-21T10:27:42.449 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.38e-05 2024-09-21T10:27:42.449 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.449 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.449 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.449 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.602148+0000", 2024-09-21T10:27:42.449 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.85843178200000003 2024-09-21T10:27:42.449 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.449 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.449 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for peered", 2024-09-21T10:27:42.450 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:19:59.602157+0000", 2024-09-21T10:27:42.450 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.5019999999999992e-06 2024-09-21T10:27:42.450 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.450 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.450 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.450 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.834845+0000", 2024-09-21T10:27:42.450 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2326882100000001 2024-09-21T10:27:42.450 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.450 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.450 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:42.450 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.834962+0000", 2024-09-21T10:27:42.450 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000116774 2024-09-21T10:27:42.450 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.450 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.451 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 4,5,6", 2024-09-21T10:27:42.451 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.835089+0000", 2024-09-21T10:27:42.451 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012683799999999999 2024-09-21T10:27:42.451 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.451 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.451 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.451 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.847843+0000", 2024-09-21T10:27:42.451 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.012754860999999999 2024-09-21T10:27:42.451 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.451 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.451 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.451 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.847868+0000", 2024-09-21T10:27:42.452 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.4922e-05 2024-09-21T10:27:42.452 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.452 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.452 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.452 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.847894+0000", 2024-09-21T10:27:42.452 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.5323000000000001e-05 2024-09-21T10:27:42.452 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.452 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.452 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:42.452 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.854458+0000", 2024-09-21T10:27:42.452 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0065641099999999997 2024-09-21T10:27:42.452 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.452 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.452 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:42.453 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.854517+0000", 2024-09-21T10:27:42.453 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9561000000000001e-05 2024-09-21T10:27:42.453 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.453 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.453 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:42.453 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:00.854630+0000", 2024-09-21T10:27:42.453 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000112475 2024-09-21T10:27:42.453 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.453 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.453 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.453 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.453 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.454 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:174977 13.f 13:f3eb7148:::benchmark_data_smithi107_278655_object47070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-21T10:27:42.454 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:46.819794+0000", 2024-09-21T10:27:42.454 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 415.55480968299997, 2024-09-21T10:27:42.454 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.37579683600000002, 2024-09-21T10:27:42.454 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:42.454 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:42.454 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:42.454 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:42.454 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:42.454 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 174977 2024-09-21T10:27:42.454 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.454 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:42.454 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.454 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:42.454 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.819794+0000", 2024-09-21T10:27:42.455 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.455 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.455 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.455 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:42.455 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.819794+0000", 2024-09-21T10:27:42.455 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.455 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.455 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.455 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:42.455 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.819792+0000", 2024-09-21T10:27:42.455 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:27:42.455 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.455 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.455 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:42.456 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.819799+0000", 2024-09-21T10:27:42.456 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.9580000000000001e-06 2024-09-21T10:27:42.456 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.456 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.456 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:42.456 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.819814+0000", 2024-09-21T10:27:42.456 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5211000000000001e-05 2024-09-21T10:27:42.456 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.456 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.456 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:42.456 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.819820+0000", 2024-09-21T10:27:42.456 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0480000000000004e-06 2024-09-21T10:27:42.456 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.456 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.456 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.457 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.819849+0000", 2024-09-21T10:27:42.457 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.8303999999999999e-05 2024-09-21T10:27:42.457 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.457 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.457 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:42.457 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.819947+0000", 2024-09-21T10:27:42.457 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.8627000000000003e-05 2024-09-21T10:27:42.457 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.457 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.457 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,5,6", 2024-09-21T10:27:42.457 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.820029+0000", 2024-09-21T10:27:42.457 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.1605000000000002e-05 2024-09-21T10:27:42.457 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.457 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.457 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.457 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.823532+0000", 2024-09-21T10:27:42.458 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0035031070000000001 2024-09-21T10:27:42.458 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.458 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.458 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.458 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.823560+0000", 2024-09-21T10:27:42.458 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.7467e-05 2024-09-21T10:27:42.458 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.458 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.458 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:42.458 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.823611+0000", 2024-09-21T10:27:42.458 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0985000000000001e-05 2024-09-21T10:27:42.458 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.458 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.458 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.458 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:47.195419+0000", 2024-09-21T10:27:42.459 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.37180849799999999 2024-09-21T10:27:42.459 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.459 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.459 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:42.459 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:47.195465+0000", 2024-09-21T10:27:42.459 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.6353999999999997e-05 2024-09-21T10:27:42.459 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.459 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.459 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:42.459 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:47.195590+0000", 2024-09-21T10:27:42.459 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000125084 2024-09-21T10:27:42.459 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.459 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.459 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.460 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.460 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.460 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:174979 13.f 13:f447ae81:::benchmark_data_smithi107_278655_object47072:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-09-21T10:27:42.460 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:20:46.839661+0000", 2024-09-21T10:27:42.460 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 415.53494180299998, 2024-09-21T10:27:42.460 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.35786738699999998, 2024-09-21T10:27:42.460 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:42.460 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:42.460 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:42.460 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:42.460 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:42.460 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 174979 2024-09-21T10:27:42.460 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.460 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:42.460 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.461 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:42.461 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.839661+0000", 2024-09-21T10:27:42.461 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.461 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.461 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.461 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:42.461 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.839661+0000", 2024-09-21T10:27:42.461 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.461 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.461 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.461 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:42.461 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.839660+0000", 2024-09-21T10:27:42.461 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:27:42.461 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.461 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.462 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:42.462 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.839668+0000", 2024-09-21T10:27:42.462 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.5660000000000001e-06 2024-09-21T10:27:42.462 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.462 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.462 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:42.462 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.839680+0000", 2024-09-21T10:27:42.462 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.2255999999999999e-05 2024-09-21T10:27:42.462 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.462 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.462 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:42.462 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.839686+0000", 2024-09-21T10:27:42.462 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6810000000000003e-06 2024-09-21T10:27:42.462 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.462 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.463 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.463 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.839726+0000", 2024-09-21T10:27:42.463 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0571999999999997e-05 2024-09-21T10:27:42.463 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.463 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.463 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:42.463 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.839844+0000", 2024-09-21T10:27:42.463 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000117799 2024-09-21T10:27:42.463 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.463 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.463 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 2,5,6", 2024-09-21T10:27:42.463 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.839971+0000", 2024-09-21T10:27:42.463 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012727999999999999 2024-09-21T10:27:42.463 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.464 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.464 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.464 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.843754+0000", 2024-09-21T10:27:42.464 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0037827870000000001 2024-09-21T10:27:42.464 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.464 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.464 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:42.464 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.843767+0000", 2024-09-21T10:27:42.464 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.3079e-05 2024-09-21T10:27:42.464 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.464 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.464 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.464 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:46.844117+0000", 2024-09-21T10:27:42.465 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00035028799999999998 2024-09-21T10:27:42.465 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.465 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.465 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.465 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:47.197221+0000", 2024-09-21T10:27:42.465 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.35310363 2024-09-21T10:27:42.465 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.465 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.465 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:42.465 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:47.197349+0000", 2024-09-21T10:27:42.465 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012828699999999999 2024-09-21T10:27:42.465 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.465 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.465 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:42.465 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:20:47.197529+0000", 2024-09-21T10:27:42.466 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000179648 2024-09-21T10:27:42.466 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.466 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.466 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.466 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.466 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.466 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:216952 13.0 13:081f7454:::benchmark_data_smithi107_278655_object89045:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-09-21T10:27:42.466 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:21:34.020771+0000", 2024-09-21T10:27:42.466 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 368.35383187899998, 2024-09-21T10:27:42.466 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.25804288800000003, 2024-09-21T10:27:42.466 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:42.466 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:42.466 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:42.466 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:42.467 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:42.467 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 216952 2024-09-21T10:27:42.467 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.467 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:42.467 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.467 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:42.467 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.020771+0000", 2024-09-21T10:27:42.467 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.467 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.467 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.467 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:42.467 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.020771+0000", 2024-09-21T10:27:42.467 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.467 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.467 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.468 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:42.468 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.020771+0000", 2024-09-21T10:27:42.468 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999995 2024-09-21T10:27:42.468 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.468 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.468 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:42.468 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.020775+0000", 2024-09-21T10:27:42.468 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.0929999999999996e-06 2024-09-21T10:27:42.468 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.468 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.468 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:42.468 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.020782+0000", 2024-09-21T10:27:42.468 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6499999999999999e-06 2024-09-21T10:27:42.468 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.468 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.469 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:42.469 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.020785+0000", 2024-09-21T10:27:42.469 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1970000000000001e-06 2024-09-21T10:27:42.469 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.469 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.469 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.469 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.020830+0000", 2024-09-21T10:27:42.469 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5386000000000002e-05 2024-09-21T10:27:42.469 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.469 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.469 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:42.469 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.020945+0000", 2024-09-21T10:27:42.469 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011482200000000001 2024-09-21T10:27:42.469 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.469 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.470 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:27:42.470 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.021053+0000", 2024-09-21T10:27:42.470 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010788199999999999 2024-09-21T10:27:42.470 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.470 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.470 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.470 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.024529+0000", 2024-09-21T10:27:42.470 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0034759909999999999 2024-09-21T10:27:42.470 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.470 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.470 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:42.470 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.024854+0000", 2024-09-21T10:27:42.470 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00032495100000000003 2024-09-21T10:27:42.470 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.471 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.471 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.471 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.278704+0000", 2024-09-21T10:27:42.471 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.253849671 2024-09-21T10:27:42.471 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.471 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.471 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:42.471 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.278770+0000", 2024-09-21T10:27:42.471 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6322999999999995e-05 2024-09-21T10:27:42.471 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.471 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.471 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:42.471 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:34.278814+0000", 2024-09-21T10:27:42.472 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.4468999999999997e-05 2024-09-21T10:27:42.472 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.472 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.472 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.472 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.472 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.472 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.37050.0:223963 13.0 13:0fa7f079:::benchmark_data_smithi107_278655_object96056:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e395)", 2024-09-21T10:27:42.472 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:21:41.097504+0000", 2024-09-21T10:27:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 361.27709943500003, 2024-09-21T10:27:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.196014301, 2024-09-21T10:27:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.37050", 2024-09-21T10:27:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 223963 2024-09-21T10:27:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:41.097504+0000", 2024-09-21T10:27:42.473 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.474 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.474 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.474 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:42.474 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:41.097504+0000", 2024-09-21T10:27:42.474 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.474 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.474 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.474 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:42.474 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:41.097502+0000", 2024-09-21T10:27:42.474 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:27:42.474 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.474 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.474 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:42.474 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:41.097508+0000", 2024-09-21T10:27:42.474 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.0140000000000004e-06 2024-09-21T10:27:42.475 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.475 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.475 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:42.475 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:41.097518+0000", 2024-09-21T10:27:42.475 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.0249e-05 2024-09-21T10:27:42.475 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.475 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.475 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:42.475 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:41.097524+0000", 2024-09-21T10:27:42.475 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3040000000000004e-06 2024-09-21T10:27:42.475 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.475 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.475 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.475 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:41.097567+0000", 2024-09-21T10:27:42.475 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3483999999999999e-05 2024-09-21T10:27:42.476 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.476 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.476 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:42.476 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:41.097662+0000", 2024-09-21T10:27:42.476 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.4866999999999998e-05 2024-09-21T10:27:42.476 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.476 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.476 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:27:42.476 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:41.097757+0000", 2024-09-21T10:27:42.476 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 9.5258000000000002e-05 2024-09-21T10:27:42.476 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.476 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.476 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:42.476 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:41.102285+0000", 2024-09-21T10:27:42.477 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0045277970000000001 2024-09-21T10:27:42.477 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.477 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.477 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.477 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:41.106737+0000", 2024-09-21T10:27:42.477 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0044519199999999998 2024-09-21T10:27:42.477 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.477 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.477 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.477 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:41.293421+0000", 2024-09-21T10:27:42.477 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.18668401900000001 2024-09-21T10:27:42.477 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.477 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.477 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:42.477 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:41.293475+0000", 2024-09-21T10:27:42.478 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.3603999999999997e-05 2024-09-21T10:27:42.478 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.478 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.478 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:42.478 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:21:41.293518+0000", 2024-09-21T10:27:42.478 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.3433999999999998e-05 2024-09-21T10:27:42.478 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.478 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.478 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.478 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.478 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.478 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:18235 14.f 14:f2217489:::benchmark_data_smithi107_312686_object18234:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-21T10:27:42.478 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:23:03.359300+0000", 2024-09-21T10:27:42.479 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 279.01530333300002, 2024-09-21T10:27:42.479 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21785349100000001, 2024-09-21T10:27:42.479 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:42.479 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:42.479 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:42.479 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:27:42.479 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:42.479 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 18235 2024-09-21T10:27:42.479 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.479 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:42.479 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.479 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:42.479 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.359300+0000", 2024-09-21T10:27:42.479 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.480 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.480 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.480 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:42.480 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.359300+0000", 2024-09-21T10:27:42.480 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.480 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.480 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.480 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:42.480 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.359299+0000", 2024-09-21T10:27:42.480 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:27:42.480 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.480 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.480 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:42.480 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.359427+0000", 2024-09-21T10:27:42.481 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000128348 2024-09-21T10:27:42.481 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.481 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.481 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:42.481 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.359444+0000", 2024-09-21T10:27:42.481 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.6878999999999999e-05 2024-09-21T10:27:42.481 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.481 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.481 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:42.481 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.359450+0000", 2024-09-21T10:27:42.481 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.6039999999999998e-06 2024-09-21T10:27:42.481 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.481 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.481 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.481 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.359636+0000", 2024-09-21T10:27:42.482 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00018609599999999999 2024-09-21T10:27:42.482 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.482 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.482 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:42.482 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.359746+0000", 2024-09-21T10:27:42.482 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010999500000000001 2024-09-21T10:27:42.482 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.482 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.482 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:27:42.482 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.359863+0000", 2024-09-21T10:27:42.482 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000116134 2024-09-21T10:27:42.482 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.482 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.483 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:42.483 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.360749+0000", 2024-09-21T10:27:42.483 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00088644500000000003 2024-09-21T10:27:42.483 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.483 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.483 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.483 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.362189+0000", 2024-09-21T10:27:42.483 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0014404800000000001 2024-09-21T10:27:42.483 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.483 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.483 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.483 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.577100+0000", 2024-09-21T10:27:42.483 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21491015599999999 2024-09-21T10:27:42.483 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.484 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.484 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:42.484 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.577131+0000", 2024-09-21T10:27:42.484 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1578999999999997e-05 2024-09-21T10:27:42.484 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.484 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.484 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:42.484 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.577153+0000", 2024-09-21T10:27:42.484 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.2226e-05 2024-09-21T10:27:42.484 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.484 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.484 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.485 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.485 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.485 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:18245 14.f 14:ff315b2c:::benchmark_data_smithi107_312686_object18244:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-21T10:27:42.485 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:23:03.363222+0000", 2024-09-21T10:27:42.485 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 279.01138140199998, 2024-09-21T10:27:42.485 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21401489700000001, 2024-09-21T10:27:42.485 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:42.485 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:42.485 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:42.485 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:27:42.485 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:42.485 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 18245 2024-09-21T10:27:42.485 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.485 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:42.485 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.486 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:42.486 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.363222+0000", 2024-09-21T10:27:42.486 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.486 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.486 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.486 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:42.486 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.363222+0000", 2024-09-21T10:27:42.486 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.486 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.486 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.486 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:42.486 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.363220+0000", 2024-09-21T10:27:42.486 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:27:42.486 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.487 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.487 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:42.487 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.363288+0000", 2024-09-21T10:27:42.487 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.8001000000000002e-05 2024-09-21T10:27:42.487 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.487 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.487 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:42.487 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.363304+0000", 2024-09-21T10:27:42.487 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.5353999999999999e-05 2024-09-21T10:27:42.487 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.487 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.487 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:42.487 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.363310+0000", 2024-09-21T10:27:42.487 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.3799999999999999e-06 2024-09-21T10:27:42.488 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.488 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.488 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.488 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.363356+0000", 2024-09-21T10:27:42.488 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4.5686000000000003e-05 2024-09-21T10:27:42.488 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.488 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.488 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:42.488 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.363494+0000", 2024-09-21T10:27:42.488 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000138314 2024-09-21T10:27:42.488 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.488 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.488 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:27:42.488 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.363636+0000", 2024-09-21T10:27:42.488 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.000142266 2024-09-21T10:27:42.489 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.489 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.489 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:42.489 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.364550+0000", 2024-09-21T10:27:42.489 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00091328999999999998 2024-09-21T10:27:42.489 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.489 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.489 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.489 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.365433+0000", 2024-09-21T10:27:42.489 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00088316899999999995 2024-09-21T10:27:42.489 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.489 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.489 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.489 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.577189+0000", 2024-09-21T10:27:42.489 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.21175611 2024-09-21T10:27:42.490 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.490 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.490 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:42.490 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.577217+0000", 2024-09-21T10:27:42.490 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.775e-05 2024-09-21T10:27:42.490 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.490 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.490 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:42.490 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.577237+0000", 2024-09-21T10:27:42.490 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.0049000000000001e-05 2024-09-21T10:27:42.490 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.490 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.491 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.491 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.491 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.491 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:18247 14.f 14:f1b43bb1:::benchmark_data_smithi107_312686_object18246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-21T10:27:42.491 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:23:03.364569+0000", 2024-09-21T10:27:42.491 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 279.01003408100001, 2024-09-21T10:27:42.491 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.212740032, 2024-09-21T10:27:42.491 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:42.491 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:42.491 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:42.491 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:27:42.491 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:42.491 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 18247 2024-09-21T10:27:42.491 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.492 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:42.492 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.492 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:42.492 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.364569+0000", 2024-09-21T10:27:42.492 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.492 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.492 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.492 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:42.492 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.364569+0000", 2024-09-21T10:27:42.492 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.492 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.492 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.492 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:42.492 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.364568+0000", 2024-09-21T10:27:42.492 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999986 2024-09-21T10:27:42.492 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.493 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.493 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:42.493 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.364639+0000", 2024-09-21T10:27:42.493 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.0838000000000002e-05 2024-09-21T10:27:42.493 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.493 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.493 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:42.493 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.364655+0000", 2024-09-21T10:27:42.493 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.6623e-05 2024-09-21T10:27:42.493 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.493 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.493 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:42.493 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.364662+0000", 2024-09-21T10:27:42.493 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 6.7320000000000001e-06 2024-09-21T10:27:42.493 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.494 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.494 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.494 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.364713+0000", 2024-09-21T10:27:42.494 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.0346000000000002e-05 2024-09-21T10:27:42.494 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.494 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.494 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:42.494 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.364835+0000", 2024-09-21T10:27:42.494 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012254 2024-09-21T10:27:42.494 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.494 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.494 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:27:42.494 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.364946+0000", 2024-09-21T10:27:42.494 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00011053900000000001 2024-09-21T10:27:42.495 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.495 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.495 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:42.495 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.365554+0000", 2024-09-21T10:27:42.495 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00060850599999999998 2024-09-21T10:27:42.495 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.495 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.495 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.495 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.365796+0000", 2024-09-21T10:27:42.495 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00024235700000000001 2024-09-21T10:27:42.495 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.495 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.495 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.495 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.577268+0000", 2024-09-21T10:27:42.495 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.211471986 2024-09-21T10:27:42.496 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.496 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.496 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:42.496 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.577292+0000", 2024-09-21T10:27:42.496 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.3751999999999999e-05 2024-09-21T10:27:42.496 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.496 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.496 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:42.496 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:23:03.577309+0000", 2024-09-21T10:27:42.496 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.7014e-05 2024-09-21T10:27:42.496 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.496 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.496 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.496 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.496 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.497 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:155136 14.f 14:f3e9009f:::benchmark_data_smithi107_312686_object39345:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-09-21T10:27:42.497 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:25:14.624345+0000", 2024-09-21T10:27:42.497 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 147.75025858199999, 2024-09-21T10:27:42.497 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.233893566, 2024-09-21T10:27:42.497 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:42.497 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:42.497 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:42.497 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:27:42.497 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:42.497 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 155136 2024-09-21T10:27:42.497 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.497 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:42.497 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.497 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:42.497 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.624345+0000", 2024-09-21T10:27:42.498 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.498 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.498 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.498 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:42.498 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.624345+0000", 2024-09-21T10:27:42.498 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.498 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.498 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.498 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:42.498 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.624343+0000", 2024-09-21T10:27:42.498 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.9999981 2024-09-21T10:27:42.498 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.498 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.498 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:42.499 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.624351+0000", 2024-09-21T10:27:42.499 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.7689999999999996e-06 2024-09-21T10:27:42.499 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.499 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.499 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:42.499 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.624369+0000", 2024-09-21T10:27:42.499 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.7900000000000001e-05 2024-09-21T10:27:42.499 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.499 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.499 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:42.499 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.624376+0000", 2024-09-21T10:27:42.499 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 7.729e-06 2024-09-21T10:27:42.499 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.499 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.500 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.500 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.624436+0000", 2024-09-21T10:27:42.500 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.9871000000000003e-05 2024-09-21T10:27:42.500 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.500 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.500 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:42.500 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.624566+0000", 2024-09-21T10:27:42.500 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00012945799999999999 2024-09-21T10:27:42.500 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.500 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.500 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:27:42.500 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.624701+0000", 2024-09-21T10:27:42.500 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00013534899999999999 2024-09-21T10:27:42.501 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.501 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.501 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:42.501 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.628078+0000", 2024-09-21T10:27:42.501 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0033772839999999999 2024-09-21T10:27:42.501 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.501 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.501 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.501 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.631020+0000", 2024-09-21T10:27:42.501 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0029415880000000002 2024-09-21T10:27:42.501 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.501 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.501 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.501 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.858175+0000", 2024-09-21T10:27:42.501 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.22715548499999999 2024-09-21T10:27:42.502 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.502 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.502 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:42.502 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.858211+0000", 2024-09-21T10:27:42.502 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.5933999999999998e-05 2024-09-21T10:27:42.502 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.502 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.502 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:42.502 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.858238+0000", 2024-09-21T10:27:42.502 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 2.6990000000000001e-05 2024-09-21T10:27:42.502 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.502 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.503 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.503 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.503 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.503 INFO:teuthology.orchestra.run.smithi129.stdout: "description": "osd_op(client.47186.0:155146 14.f 14:fce29f1e:::benchmark_data_smithi107_312686_object39355:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-09-21T10:27:42.503 INFO:teuthology.orchestra.run.smithi129.stdout: "initiated_at": "2024-09-21T10:25:14.634637+0000", 2024-09-21T10:27:42.503 INFO:teuthology.orchestra.run.smithi129.stdout: "age": 147.73996642500001, 2024-09-21T10:27:42.503 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.224235986, 2024-09-21T10:27:42.503 INFO:teuthology.orchestra.run.smithi129.stdout: "type_data": { 2024-09-21T10:27:42.503 INFO:teuthology.orchestra.run.smithi129.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:42.503 INFO:teuthology.orchestra.run.smithi129.stdout: "client_info": { 2024-09-21T10:27:42.503 INFO:teuthology.orchestra.run.smithi129.stdout: "client": "client.47186", 2024-09-21T10:27:42.503 INFO:teuthology.orchestra.run.smithi129.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:42.503 INFO:teuthology.orchestra.run.smithi129.stdout: "tid": 155146 2024-09-21T10:27:42.503 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.503 INFO:teuthology.orchestra.run.smithi129.stdout: "events": [ 2024-09-21T10:27:42.504 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.504 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "initiated", 2024-09-21T10:27:42.504 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634637+0000", 2024-09-21T10:27:42.504 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.504 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.504 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.504 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "throttled", 2024-09-21T10:27:42.504 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634637+0000", 2024-09-21T10:27:42.504 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0 2024-09-21T10:27:42.504 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.504 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.504 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "header_read", 2024-09-21T10:27:42.504 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634636+0000", 2024-09-21T10:27:42.504 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 4294967295.999999 2024-09-21T10:27:42.505 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.505 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.505 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "all_read", 2024-09-21T10:27:42.505 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634641+0000", 2024-09-21T10:27:42.505 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.7300000000000002e-06 2024-09-21T10:27:42.505 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.505 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.505 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "dispatched", 2024-09-21T10:27:42.505 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634653+0000", 2024-09-21T10:27:42.505 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.1745e-05 2024-09-21T10:27:42.505 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.505 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.505 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "queued_for_pg", 2024-09-21T10:27:42.505 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634659+0000", 2024-09-21T10:27:42.505 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.6239999999999999e-06 2024-09-21T10:27:42.506 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.506 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.506 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "reached_pg", 2024-09-21T10:27:42.506 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634690+0000", 2024-09-21T10:27:42.506 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.1072999999999999e-05 2024-09-21T10:27:42.506 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.506 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.506 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "started", 2024-09-21T10:27:42.506 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634774+0000", 2024-09-21T10:27:42.506 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 8.4433999999999998e-05 2024-09-21T10:27:42.506 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.506 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.507 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:27:42.507 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.634876+0000", 2024-09-21T10:27:42.507 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.00010207100000000001 2024-09-21T10:27:42.507 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.507 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.507 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.507 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.638012+0000", 2024-09-21T10:27:42.507 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.0031359270000000002 2024-09-21T10:27:42.507 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.507 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.507 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "op_commit", 2024-09-21T10:27:42.507 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.638028+0000", 2024-09-21T10:27:42.507 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 1.579e-05 2024-09-21T10:27:42.508 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.508 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.508 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:42.508 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.858783+0000", 2024-09-21T10:27:42.508 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 0.22075505200000001 2024-09-21T10:27:42.508 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.508 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.508 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "commit_sent", 2024-09-21T10:27:42.508 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.858835+0000", 2024-09-21T10:27:42.508 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 5.1490000000000003e-05 2024-09-21T10:27:42.508 INFO:teuthology.orchestra.run.smithi129.stdout: }, 2024-09-21T10:27:42.508 INFO:teuthology.orchestra.run.smithi129.stdout: { 2024-09-21T10:27:42.508 INFO:teuthology.orchestra.run.smithi129.stdout: "event": "done", 2024-09-21T10:27:42.508 INFO:teuthology.orchestra.run.smithi129.stdout: "time": "2024-09-21T10:25:14.858873+0000", 2024-09-21T10:27:42.508 INFO:teuthology.orchestra.run.smithi129.stdout: "duration": 3.82e-05 2024-09-21T10:27:42.509 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.509 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.509 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.509 INFO:teuthology.orchestra.run.smithi129.stdout: } 2024-09-21T10:27:42.509 INFO:teuthology.orchestra.run.smithi129.stdout: ] 2024-09-21T10:27:42.509 INFO:teuthology.orchestra.run.smithi129.stdout:} 2024-09-21T10:27:43.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:27:43.443 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:43 smithi107 ceph-mon[102221]: pgmap v1620: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:43.443 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:43 smithi107 ceph-mon[105022]: pgmap v1620: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:43.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:43 smithi129 ceph-mon[88471]: pgmap v1620: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:43.820 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:44.198 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T10:27:44.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:44 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3562158950' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:27:44.308 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:44 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3562158950' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:27:44.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:44 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3562158950' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:27:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:45 smithi107 ceph-mon[102221]: pgmap v1621: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:45 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3562158950' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:27:45.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:45 smithi107 ceph-mon[102221]: osdmap e456: 8 total, 8 up, 8 in 2024-09-21T10:27:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:45 smithi107 ceph-mon[105022]: pgmap v1621: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:45.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:45 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3562158950' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:27:45.610 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:45 smithi107 ceph-mon[105022]: osdmap e456: 8 total, 8 up, 8 in 2024-09-21T10:27:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:45 smithi129 ceph-mon[88471]: pgmap v1621: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:45 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3562158950' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:27:45.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:45 smithi129 ceph-mon[88471]: osdmap e456: 8 total, 8 up, 8 in 2024-09-21T10:27:45.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 2 1 2024-09-21T10:27:46.252 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:27:46.253 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:27:46.253 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-09-21T10:27:46.253 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:27:46.475 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:46.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:27:47.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:47 smithi107 ceph-mon[102221]: pgmap v1623: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:47.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:47 smithi107 ceph-mon[105022]: pgmap v1623: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:47.649 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:47.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:47 smithi129 ceph-mon[88471]: pgmap v1623: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:49.208 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-21T10:27:49.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:49 smithi107 ceph-mon[102221]: pgmap v1624: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:49.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:49 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3358808750' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:27:49.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:49 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:27:49.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:49 smithi107 ceph-mon[105022]: pgmap v1624: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:49.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:49 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3358808750' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:27:49.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:49 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:27:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:49 smithi129 ceph-mon[88471]: pgmap v1624: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:49 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3358808750' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:27:49.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:49 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:27:50.428 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:50 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:27:50.428 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:50 smithi107 ceph-mon[102221]: osdmap e457: 8 total, 8 up, 8 in 2024-09-21T10:27:50.428 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:50 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:27:50.429 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:50 smithi107 ceph-mon[105022]: osdmap e457: 8 total, 8 up, 8 in 2024-09-21T10:27:50.537 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:27:50.537 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-09-21T10:27:50.537 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-09-21T10:27:50.537 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-09-21T10:27:50.537 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:27:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:50 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:27:50.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:50 smithi129 ceph-mon[88471]: osdmap e457: 8 total, 8 up, 8 in 2024-09-21T10:27:50.670 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:27:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:27:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:27:51.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:27:51.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:51 smithi107 ceph-mon[102221]: pgmap v1626: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:51.497 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:51 smithi107 ceph-mon[105022]: pgmap v1626: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:51.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:51 smithi129 ceph-mon[88471]: pgmap v1626: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:51.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T10:27:51.926 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:52.048 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:27:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:27:51] "GET /metrics HTTP/1.1" 200 34369 "" "Prometheus/2.51.0" 2024-09-21T10:27:52.680 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:53.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:53 smithi107 ceph-mon[102221]: pgmap v1627: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:53.284 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:53 smithi107 ceph-mon[105022]: pgmap v1627: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:53.510 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-09-21T10:27:53.510 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-09-21T10:27:53.510 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-09-21T10:27:53.510 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-09-21T10:27:53.510 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.510 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:249514 13.5 13:ac1bf7a6:::benchmark_data_smithi107_278655_object121607:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e399)", 2024-09-21T10:27:53.511 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:22:07.100788+0000", 2024-09-21T10:27:53.511 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 346.394471157, 2024-09-21T10:27:53.511 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.061392304, 2024-09-21T10:27:53.511 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:27:53.511 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:27:53.511 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "reached pg", 2024-09-21T10:27:53.511 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:27:53.511 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:27:53.511 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:53.511 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 249514 2024-09-21T10:27:53.511 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.512 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:27:53.512 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.512 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:27:53.512 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.100788+0000", 2024-09-21T10:27:53.512 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.512 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.512 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.512 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:27:53.512 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.100788+0000", 2024-09-21T10:27:53.512 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.512 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.512 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.512 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:27:53.513 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.100790+0000", 2024-09-21T10:27:53.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9609999999999999e-06 2024-09-21T10:27:53.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:27:53.513 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.100797+0000", 2024-09-21T10:27:53.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2339999999999999e-06 2024-09-21T10:27:53.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.513 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:27:53.513 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.100810+0000", 2024-09-21T10:27:53.513 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3114e-05 2024-09-21T10:27:53.513 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.513 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:27:53.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.100818+0000", 2024-09-21T10:27:53.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6799999999999999e-06 2024-09-21T10:27:53.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.101374+0000", 2024-09-21T10:27:53.514 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00055542899999999997 2024-09-21T10:27:53.514 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.514 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.514 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op must wait for map", 2024-09-21T10:27:53.514 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.101382+0000", 2024-09-21T10:27:53.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7440000000000007e-06 2024-09-21T10:27:53.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.104016+0000", 2024-09-21T10:27:53.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0026347139999999998 2024-09-21T10:27:53.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.515 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.515 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:27:53.515 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.104029+0000", 2024-09-21T10:27:53.515 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2510999999999999e-05 2024-09-21T10:27:53.515 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.516 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.516 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.160290+0000", 2024-09-21T10:27:53.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0562612600000001 2024-09-21T10:27:53.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.516 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.516 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:27:53.516 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.162181+0000", 2024-09-21T10:27:53.516 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001890657 2024-09-21T10:27:53.516 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.516 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:27:53.516 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.516 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.517 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.517 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:249527 13.5 13:a508c654:::benchmark_data_smithi107_278655_object121620:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-09-21T10:27:53.517 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:22:07.114805+0000", 2024-09-21T10:27:53.517 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 346.38045426299999, 2024-09-21T10:27:53.517 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0590544689999999, 2024-09-21T10:27:53.517 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:27:53.517 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:27:53.517 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:53.517 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:27:53.517 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:27:53.517 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:53.517 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 249527 2024-09-21T10:27:53.517 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.518 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:27:53.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.518 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:27:53.518 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.114805+0000", 2024-09-21T10:27:53.518 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.518 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:27:53.518 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.114805+0000", 2024-09-21T10:27:53.518 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.518 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.518 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:27:53.519 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.114807+0000", 2024-09-21T10:27:53.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.875e-06 2024-09-21T10:27:53.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.519 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:27:53.519 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.114814+0000", 2024-09-21T10:27:53.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9959999999999996e-06 2024-09-21T10:27:53.519 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.519 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.519 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:27:53.519 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.114835+0000", 2024-09-21T10:27:53.519 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1148000000000001e-05 2024-09-21T10:27:53.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:27:53.520 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.114856+0000", 2024-09-21T10:27:53.520 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0716e-05 2024-09-21T10:27:53.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.520 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.114979+0000", 2024-09-21T10:27:53.520 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012298300000000001 2024-09-21T10:27:53.520 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.520 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.520 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:27:53.520 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.114990+0000", 2024-09-21T10:27:53.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0723999999999999e-05 2024-09-21T10:27:53.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.521 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.521 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.162747+0000", 2024-09-21T10:27:53.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.047756873 2024-09-21T10:27:53.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.521 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:27:53.521 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.163228+0000", 2024-09-21T10:27:53.521 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00048100999999999998 2024-09-21T10:27:53.521 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.521 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.521 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:27:53.522 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.163303+0000", 2024-09-21T10:27:53.522 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5537999999999994e-05 2024-09-21T10:27:53.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.522 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.522 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:27:53.522 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.173548+0000", 2024-09-21T10:27:53.522 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.010244593999999999 2024-09-21T10:27:53.522 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.522 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.522 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.173739+0000", 2024-09-21T10:27:53.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019120000000000001 2024-09-21T10:27:53.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.523 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.523 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.173786+0000", 2024-09-21T10:27:53.523 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7172999999999997e-05 2024-09-21T10:27:53.523 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.523 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.524 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:27:53.524 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.173816+0000", 2024-09-21T10:27:53.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9730000000000002e-05 2024-09-21T10:27:53.524 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.524 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.524 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:27:53.524 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.173860+0000", 2024-09-21T10:27:53.524 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3909000000000003e-05 2024-09-21T10:27:53.525 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.525 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:27:53.525 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.525 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.525 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.525 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:249528 13.5 13:a7e02809:::benchmark_data_smithi107_278655_object121621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-09-21T10:27:53.525 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:22:07.115911+0000", 2024-09-21T10:27:53.525 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 346.37934874799998, 2024-09-21T10:27:53.525 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0630208059999999, 2024-09-21T10:27:53.526 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:27:53.526 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:27:53.526 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:53.526 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:27:53.526 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:27:53.526 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:53.526 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 249528 2024-09-21T10:27:53.526 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.527 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:27:53.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.527 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:27:53.527 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.115911+0000", 2024-09-21T10:27:53.527 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.527 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.527 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.528 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:27:53.528 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.115911+0000", 2024-09-21T10:27:53.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.528 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.528 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:27:53.528 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.115912+0000", 2024-09-21T10:27:53.528 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.004e-06 2024-09-21T10:27:53.528 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:27:53.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.115917+0000", 2024-09-21T10:27:53.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8500000000000002e-06 2024-09-21T10:27:53.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.529 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.529 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:27:53.529 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.115933+0000", 2024-09-21T10:27:53.529 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5968999999999999e-05 2024-09-21T10:27:53.529 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:27:53.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.115941+0000", 2024-09-21T10:27:53.530 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1720000000000007e-06 2024-09-21T10:27:53.530 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.530 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.530 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.530 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.116004+0000", 2024-09-21T10:27:53.531 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2639999999999997e-05 2024-09-21T10:27:53.531 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.531 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.531 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:27:53.531 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.116013+0000", 2024-09-21T10:27:53.531 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3070000000000008e-06 2024-09-21T10:27:53.531 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.531 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.532 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.532 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.165887+0000", 2024-09-21T10:27:53.532 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0498742990000001 2024-09-21T10:27:53.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.532 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.532 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:27:53.532 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.165993+0000", 2024-09-21T10:27:53.532 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010578600000000001 2024-09-21T10:27:53.532 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.533 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.533 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:27:53.533 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.166097+0000", 2024-09-21T10:27:53.533 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103948 2024-09-21T10:27:53.533 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.533 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.533 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:27:53.533 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.173870+0000", 2024-09-21T10:27:53.533 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0077733660000000003 2024-09-21T10:27:53.534 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.534 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.534 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.534 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.178758+0000", 2024-09-21T10:27:53.534 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0048879170000000003 2024-09-21T10:27:53.534 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.534 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.534 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.534 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.178838+0000", 2024-09-21T10:27:53.535 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9728999999999995e-05 2024-09-21T10:27:53.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.535 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:27:53.535 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.178885+0000", 2024-09-21T10:27:53.535 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7515999999999997e-05 2024-09-21T10:27:53.535 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.535 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.536 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:27:53.536 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.178932+0000", 2024-09-21T10:27:53.536 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6303e-05 2024-09-21T10:27:53.536 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.536 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:27:53.536 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.536 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.536 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.537 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:249549 13.5 13:a14dcfc3:::benchmark_data_smithi107_278655_object121642:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-09-21T10:27:53.537 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:22:07.142498+0000", 2024-09-21T10:27:53.537 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 346.35276112899999, 2024-09-21T10:27:53.537 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0365948819999999, 2024-09-21T10:27:53.537 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:27:53.537 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:27:53.537 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:53.537 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:27:53.538 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:27:53.538 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:53.538 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 249549 2024-09-21T10:27:53.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.538 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:27:53.538 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.538 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:27:53.538 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142498+0000", 2024-09-21T10:27:53.538 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.538 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.538 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.538 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:27:53.538 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142498+0000", 2024-09-21T10:27:53.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.539 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:27:53.539 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142501+0000", 2024-09-21T10:27:53.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3870000000000002e-06 2024-09-21T10:27:53.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.539 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:27:53.539 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142507+0000", 2024-09-21T10:27:53.539 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4200000000000004e-06 2024-09-21T10:27:53.539 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.539 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.539 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:27:53.539 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142530+0000", 2024-09-21T10:27:53.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2259000000000002e-05 2024-09-21T10:27:53.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:27:53.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142539+0000", 2024-09-21T10:27:53.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9410000000000002e-06 2024-09-21T10:27:53.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.540 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142653+0000", 2024-09-21T10:27:53.540 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000113919 2024-09-21T10:27:53.540 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.540 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.540 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:27:53.541 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.142667+0000", 2024-09-21T10:27:53.541 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3321e-05 2024-09-21T10:27:53.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.541 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.541 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.168611+0000", 2024-09-21T10:27:53.541 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0259440360000001 2024-09-21T10:27:53.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.541 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:27:53.541 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.168709+0000", 2024-09-21T10:27:53.541 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8190999999999996e-05 2024-09-21T10:27:53.541 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.541 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.542 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:27:53.542 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.168841+0000", 2024-09-21T10:27:53.542 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000131703 2024-09-21T10:27:53.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.542 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:27:53.542 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.173877+0000", 2024-09-21T10:27:53.542 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.005035973 2024-09-21T10:27:53.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.542 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.542 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.542 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.178798+0000", 2024-09-21T10:27:53.542 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0049212250000000004 2024-09-21T10:27:53.542 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.543 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.543 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.178976+0000", 2024-09-21T10:27:53.543 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017834 2024-09-21T10:27:53.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.543 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:27:53.543 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179055+0000", 2024-09-21T10:27:53.543 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9271999999999996e-05 2024-09-21T10:27:53.543 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.543 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.543 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:27:53.543 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179093+0000", 2024-09-21T10:27:53.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7895000000000002e-05 2024-09-21T10:27:53.544 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.544 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:27:53.544 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.544 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.544 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.544 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:249560 13.5 13:a5a5c164:::benchmark_data_smithi107_278655_object121653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-09-21T10:27:53.544 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:22:07.152416+0000", 2024-09-21T10:27:53.544 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 346.34284331499998, 2024-09-21T10:27:53.544 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.026876795, 2024-09-21T10:27:53.544 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:27:53.544 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:27:53.544 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:53.544 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:27:53.545 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:27:53.545 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:53.545 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 249560 2024-09-21T10:27:53.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.545 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:27:53.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.545 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:27:53.545 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.152416+0000", 2024-09-21T10:27:53.545 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.545 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.545 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.546 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:27:53.546 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.152416+0000", 2024-09-21T10:27:53.546 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.546 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.546 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.546 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:27:53.546 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.152418+0000", 2024-09-21T10:27:53.546 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8050000000000001e-06 2024-09-21T10:27:53.546 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.546 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.546 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:27:53.546 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.152423+0000", 2024-09-21T10:27:53.547 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1139999999999997e-06 2024-09-21T10:27:53.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:27:53.547 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.152439+0000", 2024-09-21T10:27:53.547 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6259000000000001e-05 2024-09-21T10:27:53.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.547 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.547 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:27:53.547 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.152447+0000", 2024-09-21T10:27:53.547 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9230000000000002e-06 2024-09-21T10:27:53.547 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.152543+0000", 2024-09-21T10:27:53.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5638999999999997e-05 2024-09-21T10:27:53.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:27:53.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.152552+0000", 2024-09-21T10:27:53.548 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4199999999999996e-06 2024-09-21T10:27:53.548 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.548 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.548 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.548 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.171137+0000", 2024-09-21T10:27:53.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.018584741 2024-09-21T10:27:53.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.549 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:27:53.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.171235+0000", 2024-09-21T10:27:53.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7549999999999999e-05 2024-09-21T10:27:53.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.549 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:27:53.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.171302+0000", 2024-09-21T10:27:53.549 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7150999999999995e-05 2024-09-21T10:27:53.549 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.549 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.549 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:27:53.549 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.178720+0000", 2024-09-21T10:27:53.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0074180560000000001 2024-09-21T10:27:53.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.550 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.550 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179141+0000", 2024-09-21T10:27:53.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00042093000000000002 2024-09-21T10:27:53.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.550 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.550 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179216+0000", 2024-09-21T10:27:53.550 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.4732000000000002e-05 2024-09-21T10:27:53.550 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.550 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.551 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:27:53.551 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179260+0000", 2024-09-21T10:27:53.551 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3996999999999999e-05 2024-09-21T10:27:53.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.551 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:27:53.551 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179293+0000", 2024-09-21T10:27:53.551 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3478e-05 2024-09-21T10:27:53.551 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.551 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:27:53.551 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.551 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.551 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.551 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:249568 13.5 13:a205a8d5:::benchmark_data_smithi107_278655_object121661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-09-21T10:27:53.552 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:22:07.161674+0000", 2024-09-21T10:27:53.552 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 346.33358561300003, 2024-09-21T10:27:53.552 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.017720706, 2024-09-21T10:27:53.552 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:27:53.552 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:27:53.552 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:53.552 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:27:53.552 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:27:53.552 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:53.552 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 249568 2024-09-21T10:27:53.552 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.552 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:27:53.552 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.552 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:27:53.552 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.161674+0000", 2024-09-21T10:27:53.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.553 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:27:53.553 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.161674+0000", 2024-09-21T10:27:53.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.553 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:27:53.553 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.161676+0000", 2024-09-21T10:27:53.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.39e-06 2024-09-21T10:27:53.553 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.553 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.553 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:27:53.553 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.161685+0000", 2024-09-21T10:27:53.553 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0580000000000004e-06 2024-09-21T10:27:53.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:27:53.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.161706+0000", 2024-09-21T10:27:53.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1036e-05 2024-09-21T10:27:53.554 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.554 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.554 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:27:53.554 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.161716+0000", 2024-09-21T10:27:53.554 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8120000000000002e-06 2024-09-21T10:27:53.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.161907+0000", 2024-09-21T10:27:53.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00019061800000000001 2024-09-21T10:27:53.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:27:53.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.161931+0000", 2024-09-21T10:27:53.555 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3787000000000001e-05 2024-09-21T10:27:53.555 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.555 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.555 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.555 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.173906+0000", 2024-09-21T10:27:53.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0119751219999999 2024-09-21T10:27:53.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:27:53.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.173983+0000", 2024-09-21T10:27:53.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6921000000000004e-05 2024-09-21T10:27:53.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:27:53.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.174049+0000", 2024-09-21T10:27:53.556 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5901999999999993e-05 2024-09-21T10:27:53.556 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.556 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.556 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:27:53.556 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.178768+0000", 2024-09-21T10:27:53.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0047195259999999999 2024-09-21T10:27:53.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179184+0000", 2024-09-21T10:27:53.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041583100000000002 2024-09-21T10:27:53.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179325+0000", 2024-09-21T10:27:53.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014086099999999999 2024-09-21T10:27:53.557 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.557 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.557 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:27:53.557 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179365+0000", 2024-09-21T10:27:53.557 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0265999999999997e-05 2024-09-21T10:27:53.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.558 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:27:53.558 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179395+0000", 2024-09-21T10:27:53.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9575999999999999e-05 2024-09-21T10:27:53.558 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.558 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:27:53.558 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.558 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.558 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.558 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.37050.0:249570 13.5 13:ac098df3:::benchmark_data_smithi107_278655_object121663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-09-21T10:27:53.558 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:22:07.163153+0000", 2024-09-21T10:27:53.558 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 346.33210645600002, 2024-09-21T10:27:53.558 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0163782960000001, 2024-09-21T10:27:53.559 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:27:53.559 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:27:53.559 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:53.559 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:27:53.559 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.37050", 2024-09-21T10:27:53.559 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1085718982", 2024-09-21T10:27:53.559 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 249570 2024-09-21T10:27:53.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.559 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:27:53.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.559 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:27:53.559 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.163153+0000", 2024-09-21T10:27:53.559 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.559 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.559 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:27:53.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.163153+0000", 2024-09-21T10:27:53.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:27:53.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.163154+0000", 2024-09-21T10:27:53.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.322e-06 2024-09-21T10:27:53.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.560 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:27:53.560 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.163160+0000", 2024-09-21T10:27:53.560 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.699e-06 2024-09-21T10:27:53.560 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.560 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:27:53.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.163179+0000", 2024-09-21T10:27:53.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9046999999999999e-05 2024-09-21T10:27:53.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:27:53.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.163188+0000", 2024-09-21T10:27:53.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2730000000000002e-06 2024-09-21T10:27:53.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.561 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.561 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.163241+0000", 2024-09-21T10:27:53.561 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3659000000000002e-05 2024-09-21T10:27:53.561 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.561 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for peered", 2024-09-21T10:27:53.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:07.163248+0000", 2024-09-21T10:27:53.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5100000000000004e-06 2024-09-21T10:27:53.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.176127+0000", 2024-09-21T10:27:53.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0128796929999999 2024-09-21T10:27:53.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.562 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.562 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:27:53.562 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.176211+0000", 2024-09-21T10:27:53.562 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4028000000000003e-05 2024-09-21T10:27:53.562 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:27:53.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.176322+0000", 2024-09-21T10:27:53.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011076199999999999 2024-09-21T10:27:53.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:27:53.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179155+0000", 2024-09-21T10:27:53.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002832415 2024-09-21T10:27:53.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.563 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.563 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179427+0000", 2024-09-21T10:27:53.563 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000272286 2024-09-21T10:27:53.563 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.563 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179462+0000", 2024-09-21T10:27:53.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5499e-05 2024-09-21T10:27:53.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:27:53.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179503+0000", 2024-09-21T10:27:53.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0689999999999998e-05 2024-09-21T10:27:53.564 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.564 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.564 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:27:53.564 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:22:08.179531+0000", 2024-09-21T10:27:53.564 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8413000000000001e-05 2024-09-21T10:27:53.564 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.564 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:27:53.564 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.565 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.565 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47186.0:18231 14.6 14:6d110297:::benchmark_data_smithi107_312686_object18230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-21T10:27:53.565 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:23:03.358433+0000", 2024-09-21T10:27:53.565 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 290.13682676899998, 2024-09-21T10:27:53.565 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.21627909000000001, 2024-09-21T10:27:53.565 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:27:53.565 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:27:53.565 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:53.565 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:27:53.565 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47186", 2024-09-21T10:27:53.565 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:53.565 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 18231 2024-09-21T10:27:53.565 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.565 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:27:53.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:27:53.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.358433+0000", 2024-09-21T10:27:53.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.566 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.566 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:27:53.566 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.358433+0000", 2024-09-21T10:27:53.566 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.566 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:27:53.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.358434+0000", 2024-09-21T10:27:53.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.06e-07 2024-09-21T10:27:53.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:27:53.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.358456+0000", 2024-09-21T10:27:53.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2915999999999999e-05 2024-09-21T10:27:53.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.567 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.567 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:27:53.567 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.358466+0000", 2024-09-21T10:27:53.567 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0850000000000003e-06 2024-09-21T10:27:53.567 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:27:53.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.358471+0000", 2024-09-21T10:27:53.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.93e-06 2024-09-21T10:27:53.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.358542+0000", 2024-09-21T10:27:53.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0947999999999999e-05 2024-09-21T10:27:53.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.568 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.568 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:27:53.568 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.358668+0000", 2024-09-21T10:27:53.568 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012582399999999999 2024-09-21T10:27:53.568 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:27:53.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.358794+0000", 2024-09-21T10:27:53.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012602199999999999 2024-09-21T10:27:53.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.569 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:27:53.569 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.363404+0000", 2024-09-21T10:27:53.569 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0046101220000000004 2024-09-21T10:27:53.569 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.569 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.363467+0000", 2024-09-21T10:27:53.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2722e-05 2024-09-21T10:27:53.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.574603+0000", 2024-09-21T10:27:53.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.211135406 2024-09-21T10:27:53.570 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.570 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.570 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:27:53.570 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.574663+0000", 2024-09-21T10:27:53.570 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0659999999999999e-05 2024-09-21T10:27:53.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.571 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:27:53.571 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.574712+0000", 2024-09-21T10:27:53.571 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8748999999999997e-05 2024-09-21T10:27:53.571 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.571 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:27:53.571 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.571 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.571 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.571 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47186.0:18233 14.6 14:6d79f74c:::benchmark_data_smithi107_312686_object18232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-21T10:27:53.571 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:23:03.358492+0000", 2024-09-21T10:27:53.571 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 290.13676756900003, 2024-09-21T10:27:53.572 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.21923875600000001, 2024-09-21T10:27:53.572 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:27:53.572 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:27:53.572 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:53.573 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:27:53.573 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47186", 2024-09-21T10:27:53.573 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:53.573 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 18233 2024-09-21T10:27:53.573 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.573 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:27:53.573 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.573 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:27:53.573 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.358492+0000", 2024-09-21T10:27:53.573 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.574 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:27:53.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.358492+0000", 2024-09-21T10:27:53.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.574 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:27:53.574 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.358493+0000", 2024-09-21T10:27:53.574 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5899999999999996e-07 2024-09-21T10:27:53.574 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.574 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:27:53.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.358513+0000", 2024-09-21T10:27:53.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0557999999999999e-05 2024-09-21T10:27:53.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:27:53.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.358521+0000", 2024-09-21T10:27:53.575 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.875e-06 2024-09-21T10:27:53.575 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.575 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.575 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:27:53.575 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.358527+0000", 2024-09-21T10:27:53.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4879999999999998e-06 2024-09-21T10:27:53.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.360097+0000", 2024-09-21T10:27:53.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0015702769999999999 2024-09-21T10:27:53.576 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.576 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.576 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:27:53.576 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.360213+0000", 2024-09-21T10:27:53.576 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000116039 2024-09-21T10:27:53.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:27:53.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.360433+0000", 2024-09-21T10:27:53.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000219946 2024-09-21T10:27:53.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:27:53.577 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.363424+0000", 2024-09-21T10:27:53.577 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0029910869999999999 2024-09-21T10:27:53.577 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.577 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.577 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.364449+0000", 2024-09-21T10:27:53.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010246529999999999 2024-09-21T10:27:53.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.578 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.578 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.578 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.577637+0000", 2024-09-21T10:27:53.578 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.21318859300000001 2024-09-21T10:27:53.578 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.579 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.579 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:27:53.579 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.577689+0000", 2024-09-21T10:27:53.579 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1523000000000001e-05 2024-09-21T10:27:53.579 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.580 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.580 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:27:53.580 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.577731+0000", 2024-09-21T10:27:53.580 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2157999999999999e-05 2024-09-21T10:27:53.580 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.580 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:27:53.581 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.581 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.581 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.581 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47186.0:18236 14.6 14:67e28fd9:::benchmark_data_smithi107_312686_object18235:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-21T10:27:53.581 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:23:03.359345+0000", 2024-09-21T10:27:53.581 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 290.13591430899999, 2024-09-21T10:27:53.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.21851256999999999, 2024-09-21T10:27:53.582 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:27:53.582 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:27:53.582 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:53.582 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:27:53.582 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47186", 2024-09-21T10:27:53.583 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:53.583 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 18236 2024-09-21T10:27:53.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.583 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:27:53.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:27:53.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.359345+0000", 2024-09-21T10:27:53.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:27:53.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.359345+0000", 2024-09-21T10:27:53.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:27:53.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.359346+0000", 2024-09-21T10:27:53.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.11e-06 2024-09-21T10:27:53.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:27:53.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.359367+0000", 2024-09-21T10:27:53.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0503999999999999e-05 2024-09-21T10:27:53.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:27:53.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.359382+0000", 2024-09-21T10:27:53.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.519e-05 2024-09-21T10:27:53.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:27:53.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.359389+0000", 2024-09-21T10:27:53.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9909999999999996e-06 2024-09-21T10:27:53.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.361577+0000", 2024-09-21T10:27:53.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0021880379999999998 2024-09-21T10:27:53.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:27:53.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.361702+0000", 2024-09-21T10:27:53.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012506400000000001 2024-09-21T10:27:53.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:27:53.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.361911+0000", 2024-09-21T10:27:53.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000209187 2024-09-21T10:27:53.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:27:53.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.363433+0000", 2024-09-21T10:27:53.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001521349 2024-09-21T10:27:53.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.364512+0000", 2024-09-21T10:27:53.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0010796689999999999 2024-09-21T10:27:53.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.577775+0000", 2024-09-21T10:27:53.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.21326239699999999 2024-09-21T10:27:53.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:27:53.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.577815+0000", 2024-09-21T10:27:53.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0142000000000003e-05 2024-09-21T10:27:53.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:27:53.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.577858+0000", 2024-09-21T10:27:53.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2929000000000002e-05 2024-09-21T10:27:53.594 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.594 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:27:53.594 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.594 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47186.0:18240 14.6 14:6a3ee3ca:::benchmark_data_smithi107_312686_object18239:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-21T10:27:53.594 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:23:03.360589+0000", 2024-09-21T10:27:53.594 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 290.13467078299999, 2024-09-21T10:27:53.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.217934506, 2024-09-21T10:27:53.594 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:27:53.594 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:27:53.594 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:53.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:27:53.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47186", 2024-09-21T10:27:53.594 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:53.595 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 18240 2024-09-21T10:27:53.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.595 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:27:53.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:27:53.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.360589+0000", 2024-09-21T10:27:53.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:27:53.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.360589+0000", 2024-09-21T10:27:53.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:27:53.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.360590+0000", 2024-09-21T10:27:53.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.116e-06 2024-09-21T10:27:53.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:27:53.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.360612+0000", 2024-09-21T10:27:53.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2036999999999999e-05 2024-09-21T10:27:53.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:27:53.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.360623+0000", 2024-09-21T10:27:53.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0665999999999999e-05 2024-09-21T10:27:53.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:27:53.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.360630+0000", 2024-09-21T10:27:53.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9630000000000001e-06 2024-09-21T10:27:53.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.362532+0000", 2024-09-21T10:27:53.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0019022959999999999 2024-09-21T10:27:53.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:27:53.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.362645+0000", 2024-09-21T10:27:53.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000113335 2024-09-21T10:27:53.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:27:53.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.362787+0000", 2024-09-21T10:27:53.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014205700000000001 2024-09-21T10:27:53.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:27:53.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.363633+0000", 2024-09-21T10:27:53.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00084526999999999996 2024-09-21T10:27:53.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.364548+0000", 2024-09-21T10:27:53.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00091515499999999996 2024-09-21T10:27:53.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.578434+0000", 2024-09-21T10:27:53.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.21388615599999999 2024-09-21T10:27:53.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:27:53.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.578482+0000", 2024-09-21T10:27:53.599 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8052e-05 2024-09-21T10:27:53.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.599 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:27:53.599 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:23:03.578523+0000", 2024-09-21T10:27:53.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1403000000000003e-05 2024-09-21T10:27:53.600 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.600 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:27:53.600 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.600 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.600 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.600 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47186.0:155134 14.3 14:c2521e76:::benchmark_data_smithi107_312686_object39343:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-09-21T10:27:53.600 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:25:14.621403+0000", 2024-09-21T10:27:53.600 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 158.873856373, 2024-09-21T10:27:53.600 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.23700173699999999, 2024-09-21T10:27:53.600 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:27:53.600 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:27:53.601 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:53.601 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:27:53.601 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47186", 2024-09-21T10:27:53.601 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:53.601 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 155134 2024-09-21T10:27:53.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.601 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:27:53.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.601 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:27:53.601 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.621403+0000", 2024-09-21T10:27:53.601 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.601 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.601 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:27:53.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.621403+0000", 2024-09-21T10:27:53.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:27:53.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.621405+0000", 2024-09-21T10:27:53.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.358e-06 2024-09-21T10:27:53.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.602 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:27:53.602 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.621409+0000", 2024-09-21T10:27:53.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2719999999999999e-06 2024-09-21T10:27:53.602 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:27:53.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.621423+0000", 2024-09-21T10:27:53.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4255e-05 2024-09-21T10:27:53.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:27:53.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.621431+0000", 2024-09-21T10:27:53.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7759999999999997e-06 2024-09-21T10:27:53.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.621489+0000", 2024-09-21T10:27:53.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8227999999999999e-05 2024-09-21T10:27:53.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:27:53.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.621606+0000", 2024-09-21T10:27:53.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000116974 2024-09-21T10:27:53.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:27:53.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.621679+0000", 2024-09-21T10:27:53.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3393999999999996e-05 2024-09-21T10:27:53.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:27:53.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.628812+0000", 2024-09-21T10:27:53.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0071322740000000001 2024-09-21T10:27:53.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.631855+0000", 2024-09-21T10:27:53.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.003043273 2024-09-21T10:27:53.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.858241+0000", 2024-09-21T10:27:53.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.22638629199999999 2024-09-21T10:27:53.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:27:53.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.858352+0000", 2024-09-21T10:27:53.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011041599999999999 2024-09-21T10:27:53.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.606 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:27:53.606 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.858405+0000", 2024-09-21T10:27:53.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3224999999999999e-05 2024-09-21T10:27:53.606 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.606 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:27:53.606 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.606 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47186.0:155138 14.3 14:c19402c9:::benchmark_data_smithi107_312686_object39347:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-09-21T10:27:53.606 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:25:14.624621+0000", 2024-09-21T10:27:53.606 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 158.870638912, 2024-09-21T10:27:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.23433322500000001, 2024-09-21T10:27:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:27:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:27:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:27:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47186", 2024-09-21T10:27:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 155138 2024-09-21T10:27:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:27:53.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:27:53.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.624621+0000", 2024-09-21T10:27:53.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:27:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.624621+0000", 2024-09-21T10:27:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:27:53.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.624621+0000", 2024-09-21T10:27:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4400000000000001e-07 2024-09-21T10:27:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:27:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.624623+0000", 2024-09-21T10:27:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.979e-06 2024-09-21T10:27:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:27:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.624629+0000", 2024-09-21T10:27:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5729999999999998e-06 2024-09-21T10:27:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:27:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.624633+0000", 2024-09-21T10:27:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2350000000000001e-06 2024-09-21T10:27:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.624989+0000", 2024-09-21T10:27:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00035654700000000003 2024-09-21T10:27:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:27:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.625114+0000", 2024-09-21T10:27:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012436399999999999 2024-09-21T10:27:53.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:27:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.625228+0000", 2024-09-21T10:27:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011393299999999999 2024-09-21T10:27:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:27:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.629000+0000", 2024-09-21T10:27:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0037725559999999998 2024-09-21T10:27:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.631890+0000", 2024-09-21T10:27:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.002889389 2024-09-21T10:27:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.858808+0000", 2024-09-21T10:27:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.226918653 2024-09-21T10:27:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:27:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.858910+0000", 2024-09-21T10:27:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00010129299999999999 2024-09-21T10:27:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:27:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:14.858954+0000", 2024-09-21T10:27:53.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4258999999999998e-05 2024-09-21T10:27:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:27:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47186.0:182253 14.b 14:d83bcce8:::benchmark_data_smithi107_312686_object66462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-21T10:27:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:25:48.099660+0000", 2024-09-21T10:27:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 125.39559991500001, 2024-09-21T10:27:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.17959619900000001, 2024-09-21T10:27:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:27:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:27:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:27:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47186", 2024-09-21T10:27:53.614 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 182253 2024-09-21T10:27:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:27:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:27:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.099660+0000", 2024-09-21T10:27:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:27:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.099660+0000", 2024-09-21T10:27:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:27:53.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.099660+0000", 2024-09-21T10:27:53.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1500000000000005e-07 2024-09-21T10:27:53.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:27:53.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.099663+0000", 2024-09-21T10:27:53.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0000000000000001e-06 2024-09-21T10:27:53.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:27:53.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.099670+0000", 2024-09-21T10:27:53.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8750000000000002e-06 2024-09-21T10:27:53.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:27:53.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.099675+0000", 2024-09-21T10:27:53.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8300000000000003e-06 2024-09-21T10:27:53.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.111804+0000", 2024-09-21T10:27:53.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.012128880999999999 2024-09-21T10:27:53.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:27:53.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.111924+0000", 2024-09-21T10:27:53.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000120322 2024-09-21T10:27:53.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:27:53.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.112036+0000", 2024-09-21T10:27:53.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011150900000000001 2024-09-21T10:27:53.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:27:53.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.119749+0000", 2024-09-21T10:27:53.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00771362 2024-09-21T10:27:53.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.125662+0000", 2024-09-21T10:27:53.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0059130789999999999 2024-09-21T10:27:53.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.279125+0000", 2024-09-21T10:27:53.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.15346252899999999 2024-09-21T10:27:53.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:27:53.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.279167+0000", 2024-09-21T10:27:53.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2027000000000003e-05 2024-09-21T10:27:53.619 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.619 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.619 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:27:53.619 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.279256+0000", 2024-09-21T10:27:53.619 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.9011999999999994e-05 2024-09-21T10:27:53.619 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.619 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:27:53.620 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.620 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.620 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47186.0:182261 14.b 14:dcf67839:::benchmark_data_smithi107_312686_object66470:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-09-21T10:27:53.620 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:25:48.111666+0000", 2024-09-21T10:27:53.620 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 125.38359350899999, 2024-09-21T10:27:53.620 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16769410600000001, 2024-09-21T10:27:53.620 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:27:53.620 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:27:53.620 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:53.620 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:27:53.620 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47186", 2024-09-21T10:27:53.620 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:53.620 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 182261 2024-09-21T10:27:53.620 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.620 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:27:53.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:27:53.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.111666+0000", 2024-09-21T10:27:53.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:27:53.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.111666+0000", 2024-09-21T10:27:53.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.621 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.621 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.621 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:27:53.621 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.111667+0000", 2024-09-21T10:27:53.621 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0219999999999999e-06 2024-09-21T10:27:53.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:27:53.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.111671+0000", 2024-09-21T10:27:53.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.296e-06 2024-09-21T10:27:53.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:27:53.622 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.111682+0000", 2024-09-21T10:27:53.622 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1071e-05 2024-09-21T10:27:53.622 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.622 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.622 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:27:53.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.111689+0000", 2024-09-21T10:27:53.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5509999999999996e-06 2024-09-21T10:27:53.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.115803+0000", 2024-09-21T10:27:53.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0041142599999999998 2024-09-21T10:27:53.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.623 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:27:53.623 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.115945+0000", 2024-09-21T10:27:53.623 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000141958 2024-09-21T10:27:53.623 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.623 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:27:53.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.116070+0000", 2024-09-21T10:27:53.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012464799999999999 2024-09-21T10:27:53.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:27:53.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.119892+0000", 2024-09-21T10:27:53.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0038217490000000002 2024-09-21T10:27:53.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.624 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.624 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.624 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.128600+0000", 2024-09-21T10:27:53.624 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0087083249999999994 2024-09-21T10:27:53.624 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.279288+0000", 2024-09-21T10:27:53.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.15068814799999999 2024-09-21T10:27:53.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:27:53.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.279328+0000", 2024-09-21T10:27:53.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9443999999999997e-05 2024-09-21T10:27:53.625 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.625 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.625 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:27:53.625 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:25:48.279360+0000", 2024-09-21T10:27:53.625 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2634e-05 2024-09-21T10:27:53.625 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.625 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:27:53.626 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.626 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.626 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47186.0:206477 14.8 14:1bcea1c2:::benchmark_data_smithi107_312686_object90686:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-09-21T10:27:53.626 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:26:17.773288+0000", 2024-09-21T10:27:53.626 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 95.721972059999999, 2024-09-21T10:27:53.626 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.17661049600000001, 2024-09-21T10:27:53.626 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:27:53.626 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:27:53.626 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:53.626 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:27:53.626 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47186", 2024-09-21T10:27:53.626 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:53.626 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 206477 2024-09-21T10:27:53.626 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.627 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:27:53.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:27:53.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.773288+0000", 2024-09-21T10:27:53.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:27:53.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.773288+0000", 2024-09-21T10:27:53.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.627 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.627 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.627 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:27:53.627 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.773289+0000", 2024-09-21T10:27:53.627 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3939999999999999e-06 2024-09-21T10:27:53.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:27:53.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.773295+0000", 2024-09-21T10:27:53.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1319999999999999e-06 2024-09-21T10:27:53.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:27:53.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.773308+0000", 2024-09-21T10:27:53.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2434e-05 2024-09-21T10:27:53.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.628 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.628 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:27:53.628 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.773315+0000", 2024-09-21T10:27:53.628 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7149999999999998e-06 2024-09-21T10:27:53.628 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.778290+0000", 2024-09-21T10:27:53.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.004974577 2024-09-21T10:27:53.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:27:53.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.778531+0000", 2024-09-21T10:27:53.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00024149600000000001 2024-09-21T10:27:53.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.629 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:27:53.629 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.778628+0000", 2024-09-21T10:27:53.629 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6322000000000004e-05 2024-09-21T10:27:53.629 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.629 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.788085+0000", 2024-09-21T10:27:53.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0094574910000000002 2024-09-21T10:27:53.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:27:53.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.794247+0000", 2024-09-21T10:27:53.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0061617759999999999 2024-09-21T10:27:53.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.630 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.630 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.630 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.949808+0000", 2024-09-21T10:27:53.630 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.15556152000000001 2024-09-21T10:27:53.630 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:27:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.949859+0000", 2024-09-21T10:27:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1e-05 2024-09-21T10:27:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:27:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.949898+0000", 2024-09-21T10:27:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8639000000000001e-05 2024-09-21T10:27:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:27:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.631 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47186.0:206482 14.8 14:1b8dfa14:::benchmark_data_smithi107_312686_object90691:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-09-21T10:27:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:26:17.778776+0000", 2024-09-21T10:27:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 95.716483234999998, 2024-09-21T10:27:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.17123137499999999, 2024-09-21T10:27:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:27:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:27:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:27:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47186", 2024-09-21T10:27:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 206482 2024-09-21T10:27:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:27:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.632 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:27:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.778776+0000", 2024-09-21T10:27:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:27:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.778776+0000", 2024-09-21T10:27:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:27:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.778777+0000", 2024-09-21T10:27:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6199999999999996e-07 2024-09-21T10:27:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.633 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:27:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.778781+0000", 2024-09-21T10:27:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8149999999999999e-06 2024-09-21T10:27:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:27:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.778791+0000", 2024-09-21T10:27:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4749999999999999e-06 2024-09-21T10:27:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:27:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.778797+0000", 2024-09-21T10:27:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4760000000000003e-06 2024-09-21T10:27:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.634 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.788123+0000", 2024-09-21T10:27:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0093259989999999997 2024-09-21T10:27:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:27:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.788225+0000", 2024-09-21T10:27:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000101592 2024-09-21T10:27:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:27:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.788340+0000", 2024-09-21T10:27:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000115697 2024-09-21T10:27:53.635 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.794363+0000", 2024-09-21T10:27:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0060229230000000003 2024-09-21T10:27:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:27:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.794378+0000", 2024-09-21T10:27:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4812e-05 2024-09-21T10:27:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.949934+0000", 2024-09-21T10:27:53.636 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.15555556800000001 2024-09-21T10:27:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:27:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.949975+0000", 2024-09-21T10:27:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1189000000000002e-05 2024-09-21T10:27:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:27:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:17.950008+0000", 2024-09-21T10:27:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2966999999999998e-05 2024-09-21T10:27:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:27:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.637 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47186.0:224131 14.6 14:6b1786a7:::benchmark_data_smithi107_312686_object108340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-21T10:27:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:26:39.711208+0000", 2024-09-21T10:27:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 73.784051251999998, 2024-09-21T10:27:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.17028975399999999, 2024-09-21T10:27:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:27:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:27:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:27:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47186", 2024-09-21T10:27:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 224131 2024-09-21T10:27:53.638 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:27:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:27:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.711208+0000", 2024-09-21T10:27:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:27:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.711208+0000", 2024-09-21T10:27:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:27:53.639 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.711210+0000", 2024-09-21T10:27:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8929999999999999e-06 2024-09-21T10:27:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:27:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.711217+0000", 2024-09-21T10:27:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7499999999999997e-06 2024-09-21T10:27:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:27:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.711233+0000", 2024-09-21T10:27:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5869999999999999e-05 2024-09-21T10:27:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.640 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:27:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.711241+0000", 2024-09-21T10:27:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.507e-06 2024-09-21T10:27:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.712219+0000", 2024-09-21T10:27:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000977606 2024-09-21T10:27:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:27:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.712314+0000", 2024-09-21T10:27:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4977999999999998e-05 2024-09-21T10:27:53.641 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:27:53.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.712412+0000", 2024-09-21T10:27:53.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7677999999999996e-05 2024-09-21T10:27:53.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.720757+0000", 2024-09-21T10:27:53.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0083450659999999999 2024-09-21T10:27:53.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.642 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.642 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:27:53.642 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.721167+0000", 2024-09-21T10:27:53.642 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00041049499999999998 2024-09-21T10:27:53.642 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.881361+0000", 2024-09-21T10:27:53.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16019360099999999 2024-09-21T10:27:53.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:27:53.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.881459+0000", 2024-09-21T10:27:53.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8376999999999997e-05 2024-09-21T10:27:53.643 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.643 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.643 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:27:53.643 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.881498+0000", 2024-09-21T10:27:53.643 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8933000000000001e-05 2024-09-21T10:27:53.643 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.643 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:27:53.644 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.644 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.644 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "osd_op(client.47186.0:224149 14.6 14:63ca92a1:::benchmark_data_smithi107_312686_object108358:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-09-21T10:27:53.644 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-09-21T10:26:39.724305+0000", 2024-09-21T10:27:53.644 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 73.770954654999997, 2024-09-21T10:27:53.644 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16896325800000001, 2024-09-21T10:27:53.644 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-09-21T10:27:53.644 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-09-21T10:27:53.644 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:27:53.644 INFO:teuthology.orchestra.run.smithi107.stdout: "client_info": { 2024-09-21T10:27:53.644 INFO:teuthology.orchestra.run.smithi107.stdout: "client": "client.47186", 2024-09-21T10:27:53.644 INFO:teuthology.orchestra.run.smithi107.stdout: "client_addr": "172.21.15.107:0/1278938894", 2024-09-21T10:27:53.644 INFO:teuthology.orchestra.run.smithi107.stdout: "tid": 224149 2024-09-21T10:27:53.644 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.644 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-09-21T10:27:53.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-09-21T10:27:53.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.724305+0000", 2024-09-21T10:27:53.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-09-21T10:27:53.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.724305+0000", 2024-09-21T10:27:53.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-09-21T10:27:53.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.645 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.645 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-09-21T10:27:53.645 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.724306+0000", 2024-09-21T10:27:53.645 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.052e-06 2024-09-21T10:27:53.645 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-09-21T10:27:53.646 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.724310+0000", 2024-09-21T10:27:53.646 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4680000000000003e-06 2024-09-21T10:27:53.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-09-21T10:27:53.646 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.724329+0000", 2024-09-21T10:27:53.646 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8317000000000001e-05 2024-09-21T10:27:53.646 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.646 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.646 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-09-21T10:27:53.646 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.724337+0000", 2024-09-21T10:27:53.646 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.8809999999999996e-06 2024-09-21T10:27:53.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-09-21T10:27:53.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.724875+0000", 2024-09-21T10:27:53.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00053818500000000005 2024-09-21T10:27:53.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-09-21T10:27:53.647 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.724978+0000", 2024-09-21T10:27:53.647 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103268 2024-09-21T10:27:53.647 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.647 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.647 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:27:53.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.725082+0000", 2024-09-21T10:27:53.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000103539 2024-09-21T10:27:53.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "op_commit", 2024-09-21T10:27:53.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.728743+0000", 2024-09-21T10:27:53.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.0036615549999999999 2024-09-21T10:27:53.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.648 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.648 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.729625+0000", 2024-09-21T10:27:53.648 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00088165599999999998 2024-09-21T10:27:53.648 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.648 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:53.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.893176+0000", 2024-09-21T10:27:53.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.16355097199999999 2024-09-21T10:27:53.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "commit_sent", 2024-09-21T10:27:53.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.893227+0000", 2024-09-21T10:27:53.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.1289999999999999e-05 2024-09-21T10:27:53.649 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-09-21T10:27:53.649 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-09-21T10:27:53.649 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-09-21T10:27:53.649 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-09-21T10:26:39.893268+0000", 2024-09-21T10:27:53.649 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1075000000000002e-05 2024-09-21T10:27:53.650 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.650 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:27:53.650 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.650 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-09-21T10:27:53.650 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-09-21T10:27:53.650 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-09-21T10:27:53.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:53 smithi129 ceph-mon[88471]: pgmap v1627: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:55.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:55 smithi107 ceph-mon[102221]: pgmap v1628: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:55.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:55 smithi107 ceph-mon[105022]: pgmap v1628: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:55.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:55 smithi129 ceph-mon[88471]: pgmap v1628: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:56.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:56.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:56.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:57.234 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T10:27:57.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:57 smithi107 ceph-mon[102221]: pgmap v1629: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:57.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:57 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/512249247' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:27:57.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:57 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:27:57.545 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:57 smithi107 ceph-mon[105022]: pgmap v1629: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:57.545 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:57 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/512249247' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:27:57.545 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:57 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:27:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:57 smithi129 ceph-mon[88471]: pgmap v1629: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:57 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/512249247' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:27:57.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:57 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:27:57.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 3 1 2024-09-21T10:27:58.222 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:58.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:58 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:27:58.497 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:58 smithi107 ceph-mon[102221]: osdmap e458: 8 total, 8 up, 8 in 2024-09-21T10:27:58.497 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:58 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:27:58.498 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:58 smithi107 ceph-mon[105022]: osdmap e458: 8 total, 8 up, 8 in 2024-09-21T10:27:58.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:58 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:27:58.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:58 smithi129 ceph-mon[88471]: osdmap e458: 8 total, 8 up, 8 in 2024-09-21T10:27:59.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:27:59 smithi107 ceph-mon[102221]: pgmap v1631: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:59.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:27:59 smithi107 ceph-mon[105022]: pgmap v1631: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:27:59.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:27:59 smithi129 ceph-mon[88471]: pgmap v1631: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:00.253 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-21T10:28:00.254 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:00 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/959046294' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:28:00.254 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:00 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:28:00.254 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:00 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/959046294' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:28:00.254 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:00 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:28:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:00 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/959046294' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:28:00.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:00 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:28:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:28:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:28:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:28:01.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T10:28:01.396 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:01.517 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:01 smithi107 ceph-mon[102221]: pgmap v1632: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:01.518 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:01 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:28:01.518 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:01 smithi107 ceph-mon[102221]: osdmap e459: 8 total, 8 up, 8 in 2024-09-21T10:28:01.518 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:01 smithi107 ceph-mon[105022]: pgmap v1632: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:01.518 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:01 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:28:01.518 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:01 smithi107 ceph-mon[105022]: osdmap e459: 8 total, 8 up, 8 in 2024-09-21T10:28:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:01 smithi129 ceph-mon[88471]: pgmap v1632: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:01 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:28:01.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:01 smithi129 ceph-mon[88471]: osdmap e459: 8 total, 8 up, 8 in 2024-09-21T10:28:02.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:28:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:28:01] "GET /metrics HTTP/1.1" 200 34369 "" "Prometheus/2.51.0" 2024-09-21T10:28:03.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:03 smithi107 ceph-mon[102221]: pgmap v1634: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:03.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:03 smithi107 ceph-mon[105022]: pgmap v1634: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:03.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:03 smithi129 ceph-mon[88471]: pgmap v1634: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:04.260 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T10:28:04.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:04 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3976081689' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:28:04.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:04 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3976081689' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:28:04.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:04 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3976081689' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:28:05.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 4 1 2024-09-21T10:28:05.295 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:05.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:05 smithi107 ceph-mon[102221]: pgmap v1635: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:05.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:05 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3976081689' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:28:05.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:05 smithi107 ceph-mon[102221]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T10:28:05.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:05 smithi107 ceph-mon[105022]: pgmap v1635: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:05.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:05 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3976081689' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:28:05.360 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:05 smithi107 ceph-mon[105022]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T10:28:05.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:05 smithi129 ceph-mon[88471]: pgmap v1635: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:05.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:05 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3976081689' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:28:05.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:05 smithi129 ceph-mon[88471]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T10:28:06.486 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:06 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:28:06.486 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:06 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:28:06.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:06 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:28:07.356 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:07 smithi107 ceph-mon[102221]: pgmap v1637: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:07.356 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:07 smithi107 ceph-mon[105022]: pgmap v1637: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:07.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:07 smithi129 ceph-mon[88471]: pgmap v1637: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:08.271 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-21T10:28:08.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:08 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3711257260' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:28:08.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:08 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3711257260' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:28:08.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:08 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3711257260' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:28:09.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T10:28:09.367 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:09.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:09 smithi107 ceph-mon[102221]: pgmap v1638: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:09.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:09 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3711257260' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:28:09.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:09 smithi107 ceph-mon[102221]: osdmap e461: 8 total, 8 up, 8 in 2024-09-21T10:28:09.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:09 smithi107 ceph-mon[105022]: pgmap v1638: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:09.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:09 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3711257260' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:28:09.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:09 smithi107 ceph-mon[105022]: osdmap e461: 8 total, 8 up, 8 in 2024-09-21T10:28:09.645 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:09 smithi129 ceph-mon[88471]: pgmap v1638: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:09.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:09 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3711257260' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:28:09.646 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:09 smithi129 ceph-mon[88471]: osdmap e461: 8 total, 8 up, 8 in 2024-09-21T10:28:10.585 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:10 smithi107 ceph-mon[102221]: osdmap e462: 8 total, 8 up, 8 in 2024-09-21T10:28:10.585 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:10 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:28:10.585 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:10 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:28:10.585 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:10 smithi107 ceph-mon[105022]: osdmap e462: 8 total, 8 up, 8 in 2024-09-21T10:28:10.585 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:10 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:28:10.585 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:10 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:28:10.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:10 smithi129 ceph-mon[88471]: osdmap e462: 8 total, 8 up, 8 in 2024-09-21T10:28:10.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:10 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:28:10.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:10 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:28:10.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:28:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:28:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:28:11.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:11 smithi129 ceph-mon[88471]: pgmap v1641: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:11.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:11.548 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:11 smithi107 ceph-mon[102221]: pgmap v1641: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:11.548 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:11.548 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:11 smithi107 ceph-mon[105022]: pgmap v1641: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:11.549 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:12.077 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:28:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:28:11] "GET /metrics HTTP/1.1" 200 34368 "" "Prometheus/2.51.0" 2024-09-21T10:28:12.293 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T10:28:12.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:12 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3344809804' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:28:12.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:12 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3344809804' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:28:12.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:12 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3344809804' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:28:13.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 5 1 2024-09-21T10:28:13.259 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:13.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:13 smithi107 ceph-mon[105022]: pgmap v1642: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:13.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:13 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3344809804' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:28:13.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:13 smithi107 ceph-mon[105022]: osdmap e463: 8 total, 8 up, 8 in 2024-09-21T10:28:13.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:28:13.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:28:13.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:28:13.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:28:13.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:28:13.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:13 smithi107 ceph-mon[102221]: pgmap v1642: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:13.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:13 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3344809804' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:28:13.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:13 smithi107 ceph-mon[102221]: osdmap e463: 8 total, 8 up, 8 in 2024-09-21T10:28:13.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:28:13.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:28:13.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:28:13.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:28:13.610 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:28:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:13 smithi129 ceph-mon[88471]: pgmap v1642: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:13 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3344809804' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:28:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:13 smithi129 ceph-mon[88471]: osdmap e463: 8 total, 8 up, 8 in 2024-09-21T10:28:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:13 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:28:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:13 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:28:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:13 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:28:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:13 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:28:13.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:13 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:28:14.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:14 smithi107 ceph-mon[105022]: pgmap v1644: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:28:14.359 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:14 smithi107 ceph-mon[102221]: pgmap v1644: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:28:14.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:14 smithi129 ceph-mon[88471]: pgmap v1644: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:28:15.354 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-21T10:28:15.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:15 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/436548561' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:28:15.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:15 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/436548561' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:28:15.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:15 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/436548561' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:28:15.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T10:28:16.202 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:16.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:16 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/436548561' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:28:16.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:16 smithi107 ceph-mon[102221]: osdmap e464: 8 total, 8 up, 8 in 2024-09-21T10:28:16.577 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:16 smithi107 ceph-mon[102221]: pgmap v1646: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:28:16.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:16 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/436548561' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:28:16.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:16 smithi107 ceph-mon[105022]: osdmap e464: 8 total, 8 up, 8 in 2024-09-21T10:28:16.578 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:16 smithi107 ceph-mon[105022]: pgmap v1646: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:28:16.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:16 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/436548561' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:28:16.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:16 smithi129 ceph-mon[88471]: osdmap e464: 8 total, 8 up, 8 in 2024-09-21T10:28:16.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:16 smithi129 ceph-mon[88471]: pgmap v1646: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:28:17.664 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:17 smithi107 ceph-mon[102221]: osdmap e465: 8 total, 8 up, 8 in 2024-09-21T10:28:17.664 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:17 smithi107 ceph-mon[105022]: osdmap e465: 8 total, 8 up, 8 in 2024-09-21T10:28:17.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:17 smithi129 ceph-mon[88471]: osdmap e465: 8 total, 8 up, 8 in 2024-09-21T10:28:18.372 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T10:28:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:18 smithi129 ceph-mon[88471]: pgmap v1648: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:18 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1361779201' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:28:18.688 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:18 smithi107 ceph-mon[102221]: pgmap v1648: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:18.688 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:18 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1361779201' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:28:18.688 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:18 smithi107 ceph-mon[105022]: pgmap v1648: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:18.688 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:18 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1361779201' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:28:18.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 6 1 2024-09-21T10:28:19.236 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:19.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:19 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1361779201' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:28:19.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:19 smithi107 ceph-mon[102221]: osdmap e466: 8 total, 8 up, 8 in 2024-09-21T10:28:19.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:19 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1361779201' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:28:19.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:19 smithi107 ceph-mon[105022]: osdmap e466: 8 total, 8 up, 8 in 2024-09-21T10:28:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:19 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1361779201' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:28:19.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:19 smithi129 ceph-mon[88471]: osdmap e466: 8 total, 8 up, 8 in 2024-09-21T10:28:20.639 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:20 smithi107 ceph-mon[102221]: pgmap v1650: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:20.640 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:20 smithi107 ceph-mon[105022]: pgmap v1650: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:20.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:20 smithi129 ceph-mon[88471]: pgmap v1650: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:20.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:28:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:28:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:28:21.387 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-21T10:28:21.669 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:21 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2285461281' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:28:21.669 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:21 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2285461281' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:28:21.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:21 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2285461281' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:28:21.996 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:28:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:28:21] "GET /metrics HTTP/1.1" 200 34368 "" "Prometheus/2.51.0" 2024-09-21T10:28:22.028 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T10:28:22.283 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:22.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:22 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2285461281' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:28:22.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:22 smithi129 ceph-mon[88471]: osdmap e467: 8 total, 8 up, 8 in 2024-09-21T10:28:22.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:22 smithi129 ceph-mon[88471]: pgmap v1652: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:22.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:22 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2285461281' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:28:22.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:22 smithi107 ceph-mon[102221]: osdmap e467: 8 total, 8 up, 8 in 2024-09-21T10:28:22.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:22 smithi107 ceph-mon[102221]: pgmap v1652: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:22.673 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:22 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2285461281' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:28:22.673 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:22 smithi107 ceph-mon[105022]: osdmap e467: 8 total, 8 up, 8 in 2024-09-21T10:28:22.673 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:22 smithi107 ceph-mon[105022]: pgmap v1652: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:24.707 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T10:28:25.002 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:24 smithi107 ceph-mon[102221]: pgmap v1653: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:25.002 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:24 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3547698594' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:28:25.002 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:24 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:28:25.003 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:24 smithi107 ceph-mon[105022]: pgmap v1653: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:25.003 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:24 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3547698594' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:28:25.003 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:24 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:28:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:24 smithi129 ceph-mon[88471]: pgmap v1653: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:24 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3547698594' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:28:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:24 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:28:25.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 7 1 2024-09-21T10:28:25.553 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:25.964 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:25 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:28:25.964 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:25 smithi107 ceph-mon[102221]: osdmap e468: 8 total, 8 up, 8 in 2024-09-21T10:28:25.964 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:25 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:28:25.965 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:25 smithi107 ceph-mon[105022]: osdmap e468: 8 total, 8 up, 8 in 2024-09-21T10:28:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:25 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:28:26.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:25 smithi129 ceph-mon[88471]: osdmap e468: 8 total, 8 up, 8 in 2024-09-21T10:28:27.004 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:26 smithi107 ceph-mon[102221]: pgmap v1655: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:27.005 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:26 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:27.005 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:26 smithi107 ceph-mon[105022]: pgmap v1655: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:27.005 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:26 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:26 smithi129 ceph-mon[88471]: pgmap v1655: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:27.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:26 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:27.719 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-21T10:28:28.017 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:27 smithi107 ceph-mon[102221]: 1.c scrub starts 2024-09-21T10:28:28.017 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:27 smithi107 ceph-mon[102221]: 1.c scrub ok 2024-09-21T10:28:28.018 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:27 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2980355750' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:28:28.018 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:27 smithi107 ceph-mon[105022]: 1.c scrub starts 2024-09-21T10:28:28.018 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:27 smithi107 ceph-mon[105022]: 1.c scrub ok 2024-09-21T10:28:28.018 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:27 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2980355750' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:28:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:27 smithi129 ceph-mon[88471]: 1.c scrub starts 2024-09-21T10:28:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:27 smithi129 ceph-mon[88471]: 1.c scrub ok 2024-09-21T10:28:28.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:27 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2980355750' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:28:28.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T10:28:28.645 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:29.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:28 smithi107 ceph-mon[102221]: pgmap v1656: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:29.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:28 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2980355750' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:28:29.029 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:28 smithi107 ceph-mon[102221]: osdmap e469: 8 total, 8 up, 8 in 2024-09-21T10:28:29.029 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:28 smithi107 ceph-mon[105022]: pgmap v1656: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:29.029 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:28 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2980355750' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:28:29.029 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:28 smithi107 ceph-mon[105022]: osdmap e469: 8 total, 8 up, 8 in 2024-09-21T10:28:29.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:28 smithi129 ceph-mon[88471]: pgmap v1656: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:29.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:28 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2980355750' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:28:29.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:28 smithi129 ceph-mon[88471]: osdmap e469: 8 total, 8 up, 8 in 2024-09-21T10:28:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:28:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:28:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:28:30.723 INFO:teuthology.orchestra.run.smithi107.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T10:28:31.024 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:30 smithi107 ceph-mon[102221]: pgmap v1658: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:31.024 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:30 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/138546403' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:28:31.024 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:30 smithi107 ceph-mon[105022]: pgmap v1658: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:31.025 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:30 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/138546403' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:28:31.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:30 smithi129 ceph-mon[88471]: pgmap v1658: 93 pgs: 93 active+clean; 279 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:31.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:30 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/138546403' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:28:31.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 0 1 2024-09-21T10:28:31.901 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:32.022 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:31 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/138546403' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:28:32.022 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:31 smithi107 ceph-mon[102221]: osdmap e470: 8 total, 8 up, 8 in 2024-09-21T10:28:32.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:31 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/138546403' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:28:32.022 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:31 smithi107 ceph-mon[105022]: osdmap e470: 8 total, 8 up, 8 in 2024-09-21T10:28:32.023 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:28:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:28:31] "GET /metrics HTTP/1.1" 200 34369 "" "Prometheus/2.51.0" 2024-09-21T10:28:32.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:31 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/138546403' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:28:32.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:31 smithi129 ceph-mon[88471]: osdmap e470: 8 total, 8 up, 8 in 2024-09-21T10:28:33.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:32 smithi107 ceph-mon[102221]: pgmap v1660: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:33.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:32 smithi107 ceph-mon[105022]: pgmap v1660: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:33.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:32 smithi129 ceph-mon[88471]: pgmap v1660: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:34.748 INFO:teuthology.orchestra.run.smithi107.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-21T10:28:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:34 smithi107 ceph-mon[102221]: pgmap v1661: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:34 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/729736965' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:28:35.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:34 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:28:35.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:34 smithi107 ceph-mon[105022]: pgmap v1661: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:35.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:34 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/729736965' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:28:35.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:34 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:28:35.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:34 smithi129 ceph-mon[88471]: pgmap v1661: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:35.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:34 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/729736965' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:28:35.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:34 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:28:35.486 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-21T10:28:35.486 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-21T10:28:35.486 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-21T10:28:35.486 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-21T10:28:35.486 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T10:28:35.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T10:28:35.737 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:36.101 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:35 smithi107 ceph-mon[102221]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:28:36.101 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:35 smithi107 ceph-mon[102221]: osdmap e471: 8 total, 8 up, 8 in 2024-09-21T10:28:36.101 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:35 smithi107 ceph-mon[105022]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:28:36.101 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:35 smithi107 ceph-mon[105022]: osdmap e471: 8 total, 8 up, 8 in 2024-09-21T10:28:36.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:35 smithi129 ceph-mon[88471]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:28:36.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:35 smithi129 ceph-mon[88471]: osdmap e471: 8 total, 8 up, 8 in 2024-09-21T10:28:36.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:36 smithi107 ceph-mon[102221]: pgmap v1663: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:36.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:36 smithi107 ceph-mon[105022]: pgmap v1663: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:37.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:36 smithi129 ceph-mon[88471]: pgmap v1663: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:28:37.632 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":471,"fsid":"86c49514-77fc-11ef-baf6-efdc52797490","created":"2024-09-21T09:34:03.635940+0000","modified":"2024-09-21T10:28:34.734255+0000","last_up_change":"2024-09-21T10:09:22.801234+0000","last_in_change":"2024-09-21T10:27:23.920335+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":24,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:37:08.948207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"397","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"171","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8999999761581421,"score_stable":1.8999999761581421,"optimal_score":1,"raw_score_acting":1.8999999761581421,"raw_score_stable":1.8999999761581421,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:39:33.152023+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:40:47.823974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:40:48.313916+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"635c1eb0-29b1-4fdd-bd47-881ef05e49bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":81,"last_clean_end":201,"up_from":245,"up_thru":450,"down_at":202,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1818143286},{"type":"v1","addr":"172.21.15.107:6803","nonce":1818143286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1818143286},{"type":"v1","addr":"172.21.15.107:6805","nonce":1818143286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1818143286},{"type":"v1","addr":"172.21.15.107:6809","nonce":1818143286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1818143286},{"type":"v1","addr":"172.21.15.107:6807","nonce":1818143286}]},"public_addr":"172.21.15.107:6803/1818143286","cluster_addr":"172.21.15.107:6805/1818143286","heartbeat_back_addr":"172.21.15.107:6809/1818143286","heartbeat_front_addr":"172.21.15.107:6807/1818143286","state":["exists","up"]},{"osd":1,"uuid":"a39b6a18-939d-4999-89b2-1126c1b2d1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":91,"up_from":96,"up_thru":450,"down_at":92,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2988029448},{"type":"v1","addr":"172.21.15.107:6811","nonce":2988029448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2988029448},{"type":"v1","addr":"172.21.15.107:6813","nonce":2988029448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2988029448},{"type":"v1","addr":"172.21.15.107:6817","nonce":2988029448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2988029448},{"type":"v1","addr":"172.21.15.107:6815","nonce":2988029448}]},"public_addr":"172.21.15.107:6811/2988029448","cluster_addr":"172.21.15.107:6813/2988029448","heartbeat_back_addr":"172.21.15.107:6817/2988029448","heartbeat_front_addr":"172.21.15.107:6815/2988029448","state":["exists","up"]},{"osd":2,"uuid":"b01c5f1b-77e4-462f-80d0-fd1d3afb786b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":101,"up_from":105,"up_thru":450,"down_at":102,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1583228557},{"type":"v1","addr":"172.21.15.107:6819","nonce":1583228557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":1583228557},{"type":"v1","addr":"172.21.15.107:6821","nonce":1583228557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1583228557},{"type":"v1","addr":"172.21.15.107:6825","nonce":1583228557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":1583228557},{"type":"v1","addr":"172.21.15.107:6823","nonce":1583228557}]},"public_addr":"172.21.15.107:6819/1583228557","cluster_addr":"172.21.15.107:6821/1583228557","heartbeat_back_addr":"172.21.15.107:6825/1583228557","heartbeat_front_addr":"172.21.15.107:6823/1583228557","state":["exists","up"]},{"osd":3,"uuid":"5135d269-b6c4-4a9d-bcf9-ffe19bc5c9ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":122,"up_from":126,"up_thru":450,"down_at":123,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":860985313},{"type":"v1","addr":"172.21.15.107:6827","nonce":860985313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":860985313},{"type":"v1","addr":"172.21.15.107:6829","nonce":860985313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":860985313},{"type":"v1","addr":"172.21.15.107:6833","nonce":860985313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":860985313},{"type":"v1","addr":"172.21.15.107:6831","nonce":860985313}]},"public_addr":"172.21.15.107:6827/860985313","cluster_addr":"172.21.15.107:6829/860985313","heartbeat_back_addr":"172.21.15.107:6833/860985313","heartbeat_front_addr":"172.21.15.107:6831/860985313","state":["exists","up"]},{"osd":4,"uuid":"23242972-e341-4773-809f-299aabeacd56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":461,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6801","nonce":4101249335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6803","nonce":4101249335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6807","nonce":4101249335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6805","nonce":4101249335}]},"public_addr":"172.21.15.129:6801/4101249335","cluster_addr":"172.21.15.129:6803/4101249335","heartbeat_back_addr":"172.21.15.129:6807/4101249335","heartbeat_front_addr":"172.21.15.129:6805/4101249335","state":["exists","up"]},{"osd":5,"uuid":"f9da25cd-1680-4d97-a9d1-98dded10766f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":464,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6809","nonce":2733195233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6811","nonce":2733195233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6815","nonce":2733195233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6813","nonce":2733195233}]},"public_addr":"172.21.15.129:6809/2733195233","cluster_addr":"172.21.15.129:6811/2733195233","heartbeat_back_addr":"172.21.15.129:6815/2733195233","heartbeat_front_addr":"172.21.15.129:6813/2733195233","state":["exists","up"]},{"osd":6,"uuid":"d6d2bd99-0178-409c-a548-cbc30b8b76fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":450,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6817","nonce":3400496608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6819","nonce":3400496608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6823","nonce":3400496608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6821","nonce":3400496608}]},"public_addr":"172.21.15.129:6817/3400496608","cluster_addr":"172.21.15.129:6819/3400496608","heartbeat_back_addr":"172.21.15.129:6823/3400496608","heartbeat_front_addr":"172.21.15.129:6821/3400496608","state":["exists","up"]},{"osd":7,"uuid":"2c0a9723-0328-4134-b5d4-cf307839018a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":48,"last_clean_end":151,"up_from":175,"up_thru":450,"down_at":152,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":702981256},{"type":"v1","addr":"172.21.15.129:6825","nonce":702981256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":702981256},{"type":"v1","addr":"172.21.15.129:6827","nonce":702981256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":702981256},{"type":"v1","addr":"172.21.15.129:6831","nonce":702981256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":702981256},{"type":"v1","addr":"172.21.15.129:6829","nonce":702981256}]},"public_addr":"172.21.15.129:6825/702981256","cluster_addr":"172.21.15.129:6827/702981256","heartbeat_back_addr":"172.21.15.129:6831/702981256","heartbeat_front_addr":"172.21.15.129:6829/702981256","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:36:18.300841+0000","dead_epoch":201},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:36:42.334925+0000","dead_epoch":91},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:06.141467+0000","dead_epoch":101},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:30.344630+0000","dead_epoch":122},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:51.661657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:38:14.654068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:38:38.089961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:39:01.119878+0000","dead_epoch":151}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.0","mappings":[{"from":1,"to":7}]},{"pgid":"3.15","mappings":[{"from":7,"to":2}]},{"pgid":"3.19","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/3807929340":"2024-09-22T09:46:25.797234+0000","172.21.15.107:6800/1580746173":"2024-09-22T09:46:25.797234+0000","172.21.15.107:0/3989069453":"2024-09-22T09:46:25.797234+0000","172.21.15.107:0/4262440050":"2024-09-22T09:46:25.797234+0000","172.21.15.107:0/1261754960":"2024-09-22T09:46:25.797234+0000","172.21.15.107:0/1942117255":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/627572584":"2024-09-22T09:46:25.797234+0000","172.21.15.107:0/51270168":"2024-09-22T09:43:44.074024+0000","172.21.15.107:0/3025710247":"2024-09-22T09:43:44.074024+0000","172.21.15.107:6801/4208383830":"2024-09-22T09:34:56.643317+0000","172.21.15.129:0/306463122":"2024-09-22T09:45:31.607526+0000","172.21.15.107:6800/4208383830":"2024-09-22T09:34:56.643317+0000","172.21.15.107:6800/4055614859":"2024-09-22T09:43:44.074024+0000","172.21.15.107:0/4103644615":"2024-09-22T09:39:30.975484+0000","172.21.15.107:6801/1580746173":"2024-09-22T09:46:25.797234+0000","172.21.15.107:6801/2716631102":"2024-09-22T09:39:30.975484+0000","172.21.15.107:6800/2716631102":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/637981220":"2024-09-22T09:34:34.697433+0000","172.21.15.107:6801/1686228827":"2024-09-22T09:34:34.697433+0000","172.21.15.129:6835/2445466552":"2024-09-22T09:45:31.607526+0000","172.21.15.129:6834/2445466552":"2024-09-22T09:45:31.607526+0000","172.21.15.107:0/2526044436":"2024-09-22T09:39:30.975484+0000","172.21.15.129:0/3127898221":"2024-09-22T09:45:31.607526+0000","172.21.15.107:6800/1686228827":"2024-09-22T09:34:34.697433+0000","172.21.15.129:0/4112718236":"2024-09-22T09:45:31.607526+0000","172.21.15.107:0/2156530655":"2024-09-22T09:39:30.975484+0000","172.21.15.107:6801/4055614859":"2024-09-22T09:43:44.074024+0000","172.21.15.129:0/2239836304":"2024-09-22T09:45:31.607526+0000","172.21.15.107:0/1526680749":"2024-09-22T09:43:44.074024+0000","172.21.15.107:0/837896416":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/3424783814":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/2701254891":"2024-09-22T09:43:44.074024+0000","172.21.15.107:0/782384657":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/2492859118":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/218684017":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/2945318176":"2024-09-22T09:46:25.797234+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T10:28:37.979 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:37 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/4192513035' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:28:37.979 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:37 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/4192513035' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:28:38.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:37 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/4192513035' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:28:38.315 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T10:28:38.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T10:28:38.560 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:38.949 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:38 smithi107 ceph-mon[102221]: pgmap v1664: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:38.950 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:38 smithi107 ceph-mon[105022]: pgmap v1664: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:39.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:38 smithi129 ceph-mon[88471]: pgmap v1664: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:40.304 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:28:40.305 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":471,"fsid":"86c49514-77fc-11ef-baf6-efdc52797490","created":"2024-09-21T09:34:03.635940+0000","modified":"2024-09-21T10:28:34.734255+0000","last_up_change":"2024-09-21T10:09:22.801234+0000","last_in_change":"2024-09-21T10:27:23.920335+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":24,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:37:08.948207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"397","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"171","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8999999761581421,"score_stable":1.8999999761581421,"optimal_score":1,"raw_score_acting":1.8999999761581421,"raw_score_stable":1.8999999761581421,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:39:33.152023+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":57,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:40:47.823974+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:40:48.313916+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"635c1eb0-29b1-4fdd-bd47-881ef05e49bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":81,"last_clean_end":201,"up_from":245,"up_thru":450,"down_at":202,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1818143286},{"type":"v1","addr":"172.21.15.107:6803","nonce":1818143286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1818143286},{"type":"v1","addr":"172.21.15.107:6805","nonce":1818143286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1818143286},{"type":"v1","addr":"172.21.15.107:6809","nonce":1818143286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1818143286},{"type":"v1","addr":"172.21.15.107:6807","nonce":1818143286}]},"public_addr":"172.21.15.107:6803/1818143286","cluster_addr":"172.21.15.107:6805/1818143286","heartbeat_back_addr":"172.21.15.107:6809/1818143286","heartbeat_front_addr":"172.21.15.107:6807/1818143286","state":["exists","up"]},{"osd":1,"uuid":"a39b6a18-939d-4999-89b2-1126c1b2d1f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":91,"up_from":96,"up_thru":450,"down_at":92,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2988029448},{"type":"v1","addr":"172.21.15.107:6811","nonce":2988029448}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2988029448},{"type":"v1","addr":"172.21.15.107:6813","nonce":2988029448}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2988029448},{"type":"v1","addr":"172.21.15.107:6817","nonce":2988029448}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2988029448},{"type":"v1","addr":"172.21.15.107:6815","nonce":2988029448}]},"public_addr":"172.21.15.107:6811/2988029448","cluster_addr":"172.21.15.107:6813/2988029448","heartbeat_back_addr":"172.21.15.107:6817/2988029448","heartbeat_front_addr":"172.21.15.107:6815/2988029448","state":["exists","up"]},{"osd":2,"uuid":"b01c5f1b-77e4-462f-80d0-fd1d3afb786b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":101,"up_from":105,"up_thru":450,"down_at":102,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":1583228557},{"type":"v1","addr":"172.21.15.107:6819","nonce":1583228557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":1583228557},{"type":"v1","addr":"172.21.15.107:6821","nonce":1583228557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1583228557},{"type":"v1","addr":"172.21.15.107:6825","nonce":1583228557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":1583228557},{"type":"v1","addr":"172.21.15.107:6823","nonce":1583228557}]},"public_addr":"172.21.15.107:6819/1583228557","cluster_addr":"172.21.15.107:6821/1583228557","heartbeat_back_addr":"172.21.15.107:6825/1583228557","heartbeat_front_addr":"172.21.15.107:6823/1583228557","state":["exists","up"]},{"osd":3,"uuid":"5135d269-b6c4-4a9d-bcf9-ffe19bc5c9ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":122,"up_from":126,"up_thru":450,"down_at":123,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":860985313},{"type":"v1","addr":"172.21.15.107:6827","nonce":860985313}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":860985313},{"type":"v1","addr":"172.21.15.107:6829","nonce":860985313}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":860985313},{"type":"v1","addr":"172.21.15.107:6833","nonce":860985313}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":860985313},{"type":"v1","addr":"172.21.15.107:6831","nonce":860985313}]},"public_addr":"172.21.15.107:6827/860985313","cluster_addr":"172.21.15.107:6829/860985313","heartbeat_back_addr":"172.21.15.107:6833/860985313","heartbeat_front_addr":"172.21.15.107:6831/860985313","state":["exists","up"]},{"osd":4,"uuid":"23242972-e341-4773-809f-299aabeacd56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":461,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6800","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6801","nonce":4101249335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6802","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6803","nonce":4101249335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6806","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6807","nonce":4101249335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6804","nonce":4101249335},{"type":"v1","addr":"172.21.15.129:6805","nonce":4101249335}]},"public_addr":"172.21.15.129:6801/4101249335","cluster_addr":"172.21.15.129:6803/4101249335","heartbeat_back_addr":"172.21.15.129:6807/4101249335","heartbeat_front_addr":"172.21.15.129:6805/4101249335","state":["exists","up"]},{"osd":5,"uuid":"f9da25cd-1680-4d97-a9d1-98dded10766f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":464,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6808","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6809","nonce":2733195233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6810","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6811","nonce":2733195233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6814","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6815","nonce":2733195233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6812","nonce":2733195233},{"type":"v1","addr":"172.21.15.129:6813","nonce":2733195233}]},"public_addr":"172.21.15.129:6809/2733195233","cluster_addr":"172.21.15.129:6811/2733195233","heartbeat_back_addr":"172.21.15.129:6815/2733195233","heartbeat_front_addr":"172.21.15.129:6813/2733195233","state":["exists","up"]},{"osd":6,"uuid":"d6d2bd99-0178-409c-a548-cbc30b8b76fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":450,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6816","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6817","nonce":3400496608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6818","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6819","nonce":3400496608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6822","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6823","nonce":3400496608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6820","nonce":3400496608},{"type":"v1","addr":"172.21.15.129:6821","nonce":3400496608}]},"public_addr":"172.21.15.129:6817/3400496608","cluster_addr":"172.21.15.129:6819/3400496608","heartbeat_back_addr":"172.21.15.129:6823/3400496608","heartbeat_front_addr":"172.21.15.129:6821/3400496608","state":["exists","up"]},{"osd":7,"uuid":"2c0a9723-0328-4134-b5d4-cf307839018a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":48,"last_clean_end":151,"up_from":175,"up_thru":450,"down_at":152,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6824","nonce":702981256},{"type":"v1","addr":"172.21.15.129:6825","nonce":702981256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6826","nonce":702981256},{"type":"v1","addr":"172.21.15.129:6827","nonce":702981256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6830","nonce":702981256},{"type":"v1","addr":"172.21.15.129:6831","nonce":702981256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.129:6828","nonce":702981256},{"type":"v1","addr":"172.21.15.129:6829","nonce":702981256}]},"public_addr":"172.21.15.129:6825/702981256","cluster_addr":"172.21.15.129:6827/702981256","heartbeat_back_addr":"172.21.15.129:6831/702981256","heartbeat_front_addr":"172.21.15.129:6829/702981256","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:36:18.300841+0000","dead_epoch":201},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:36:42.334925+0000","dead_epoch":91},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:06.141467+0000","dead_epoch":101},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:30.344630+0000","dead_epoch":122},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:37:51.661657+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:38:14.654068+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:38:38.089961+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:39:01.119878+0000","dead_epoch":151}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.0","mappings":[{"from":1,"to":7}]},{"pgid":"3.15","mappings":[{"from":7,"to":2}]},{"pgid":"3.19","mappings":[{"from":1,"to":2}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:0/3807929340":"2024-09-22T09:46:25.797234+0000","172.21.15.107:6800/1580746173":"2024-09-22T09:46:25.797234+0000","172.21.15.107:0/3989069453":"2024-09-22T09:46:25.797234+0000","172.21.15.107:0/4262440050":"2024-09-22T09:46:25.797234+0000","172.21.15.107:0/1261754960":"2024-09-22T09:46:25.797234+0000","172.21.15.107:0/1942117255":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/627572584":"2024-09-22T09:46:25.797234+0000","172.21.15.107:0/51270168":"2024-09-22T09:43:44.074024+0000","172.21.15.107:0/3025710247":"2024-09-22T09:43:44.074024+0000","172.21.15.107:6801/4208383830":"2024-09-22T09:34:56.643317+0000","172.21.15.129:0/306463122":"2024-09-22T09:45:31.607526+0000","172.21.15.107:6800/4208383830":"2024-09-22T09:34:56.643317+0000","172.21.15.107:6800/4055614859":"2024-09-22T09:43:44.074024+0000","172.21.15.107:0/4103644615":"2024-09-22T09:39:30.975484+0000","172.21.15.107:6801/1580746173":"2024-09-22T09:46:25.797234+0000","172.21.15.107:6801/2716631102":"2024-09-22T09:39:30.975484+0000","172.21.15.107:6800/2716631102":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/637981220":"2024-09-22T09:34:34.697433+0000","172.21.15.107:6801/1686228827":"2024-09-22T09:34:34.697433+0000","172.21.15.129:6835/2445466552":"2024-09-22T09:45:31.607526+0000","172.21.15.129:6834/2445466552":"2024-09-22T09:45:31.607526+0000","172.21.15.107:0/2526044436":"2024-09-22T09:39:30.975484+0000","172.21.15.129:0/3127898221":"2024-09-22T09:45:31.607526+0000","172.21.15.107:6800/1686228827":"2024-09-22T09:34:34.697433+0000","172.21.15.129:0/4112718236":"2024-09-22T09:45:31.607526+0000","172.21.15.107:0/2156530655":"2024-09-22T09:39:30.975484+0000","172.21.15.107:6801/4055614859":"2024-09-22T09:43:44.074024+0000","172.21.15.129:0/2239836304":"2024-09-22T09:45:31.607526+0000","172.21.15.107:0/1526680749":"2024-09-22T09:43:44.074024+0000","172.21.15.107:0/837896416":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/3424783814":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/2701254891":"2024-09-22T09:43:44.074024+0000","172.21.15.107:0/782384657":"2024-09-22T09:34:56.643317+0000","172.21.15.107:0/2492859118":"2024-09-22T09:34:34.697433+0000","172.21.15.107:0/218684017":"2024-09-22T09:39:30.975484+0000","172.21.15.107:0/2945318176":"2024-09-22T09:46:25.797234+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":15,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T10:28:40.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:28:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:28:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:28:41.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:40 smithi107 ceph-mon[102221]: pgmap v1665: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:41.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:40 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/2278264157' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:28:41.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:40 smithi107 ceph-mon[105022]: pgmap v1665: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:41.095 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:40 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/2278264157' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:28:41.122 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T10:28:41.122 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T10:28:41.122 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T10:28:41.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T10:28:41.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T10:28:41.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T10:28:41.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T10:28:41.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T10:28:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:40 smithi129 ceph-mon[88471]: pgmap v1665: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:41.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:40 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/2278264157' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:28:41.565 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:41.990 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:41 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:41.990 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:41 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:41.990 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:28:41 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:28:41] "GET /metrics HTTP/1.1" 200 34362 "" "Prometheus/2.51.0" 2024-09-21T10:28:42.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:41 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:42.334 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:42.334 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:42.335 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:42.336 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:42.341 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:42.342 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:42.346 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:43.097 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:42 smithi107 ceph-mon[102221]: pgmap v1666: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:43.097 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:42 smithi107 ceph-mon[105022]: pgmap v1666: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:43.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:42 smithi129 ceph-mon[88471]: pgmap v1666: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:44.958 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:44 smithi107 ceph-mon[105022]: pgmap v1667: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:44.959 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:44 smithi107 ceph-mon[102221]: pgmap v1667: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:45.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:44 smithi129 ceph-mon[88471]: pgmap v1667: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:46.569 INFO:teuthology.orchestra.run.smithi107.stdout:450971566530 2024-09-21T10:28:46.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T10:28:46.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:46 smithi107 ceph-mon[102221]: pgmap v1668: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:46.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:46 smithi107 ceph-mon[105022]: pgmap v1668: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:47.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:46 smithi129 ceph-mon[88471]: pgmap v1668: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:48.937 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:48 smithi107 ceph-mon[102221]: pgmap v1669: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:48.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:48 smithi107 ceph-mon[105022]: pgmap v1669: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:49.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:48 smithi129 ceph-mon[88471]: pgmap v1669: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:50.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:28:50 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:28:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:28:50.986 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:50 smithi107 ceph-mon[105022]: pgmap v1670: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:50.986 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:50 smithi107 ceph-mon[102221]: pgmap v1670: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:51.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:50 smithi129 ceph-mon[88471]: pgmap v1670: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:51.810 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:28:51 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:28:51] "GET /metrics HTTP/1.1" 200 34362 "" "Prometheus/2.51.0" 2024-09-21T10:28:53.053 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:52 smithi107 ceph-mon[102221]: pgmap v1671: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:53.053 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:52 smithi107 ceph-mon[105022]: pgmap v1671: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:53.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:52 smithi129 ceph-mon[88471]: pgmap v1671: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:54.058 INFO:teuthology.orchestra.run.smithi107.stdout:541165879690 2024-09-21T10:28:54.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T10:28:54.502 INFO:teuthology.orchestra.run.smithi107.stdout:412316860876 2024-09-21T10:28:54.502 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T10:28:55.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:54 smithi107 ceph-mon[102221]: pgmap v1672: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:55.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:54 smithi107 ceph-mon[105022]: pgmap v1672: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:55.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:54 smithi129 ceph-mon[88471]: pgmap v1672: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:56.786 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:56 smithi107 ceph-mon[102221]: pgmap v1673: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:56.786 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:56 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:57.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:56 smithi107 ceph-mon[105022]: pgmap v1673: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:57.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:56 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:56 smithi129 ceph-mon[88471]: pgmap v1673: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:57.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:56 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:57.319 INFO:teuthology.orchestra.run.smithi107.stdout:154618823267 2024-09-21T10:28:57.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T10:28:57.798 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:57.798 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:57.799 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:59.088 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:28:58 smithi107 ceph-mon[102221]: pgmap v1674: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:59.088 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:28:58 smithi107 ceph-mon[105022]: pgmap v1674: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:59.113 INFO:teuthology.orchestra.run.smithi107.stdout:1052266987756 2024-09-21T10:28:59.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T10:28:59.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:28:58 smithi129 ceph-mon[88471]: pgmap v1674: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:28:59.598 INFO:teuthology.orchestra.run.smithi107.stdout:128849019497 2024-09-21T10:28:59.598 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T10:29:00.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:29:00 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:29:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:29:00.858 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:00 smithi107 ceph-mon[102221]: pgmap v1675: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:00.858 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:00 smithi107 ceph-mon[105022]: pgmap v1675: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:01.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:00 smithi129 ceph-mon[88471]: pgmap v1675: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:01.802 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:01 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:29:01] "GET /metrics HTTP/1.1" 200 34362 "" "Prometheus/2.51.0" 2024-09-21T10:29:02.626 INFO:teuthology.orchestra.run.smithi107.stdout:180388627039 2024-09-21T10:29:02.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T10:29:02.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:02 smithi107 ceph-mon[102221]: pgmap v1676: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:02.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:02 smithi107 ceph-mon[105022]: pgmap v1676: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:03.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:02 smithi129 ceph-mon[88471]: pgmap v1676: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:03.416 INFO:teuthology.orchestra.run.smithi107.stdout:751619277106 2024-09-21T10:29:03.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T10:29:03.802 INFO:teuthology.orchestra.run.smithi107.stdout:541165879693 2024-09-21T10:29:05.021 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:04 smithi107 ceph-mon[102221]: pgmap v1677: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:05.021 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:04 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/823749993' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:29:05.021 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:04 smithi107 ceph-mon[105022]: pgmap v1677: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:05.021 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:04 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/823749993' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:29:05.090 INFO:teuthology.orchestra.run.smithi107.stdout:412316860878 2024-09-21T10:29:05.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:04 smithi129 ceph-mon[88471]: pgmap v1677: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:05.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:04 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/823749993' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:29:05.958 INFO:tasks.cephadm.ceph_manager.ceph:need seq 412316860876 got 412316860878 for osd.1 2024-09-21T10:29:05.958 DEBUG:teuthology.parallel:result is None 2024-09-21T10:29:06.055 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:05 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/3173875674' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T10:29:06.056 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:05 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/3173875674' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T10:29:06.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:05 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/3173875674' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T10:29:06.327 INFO:teuthology.orchestra.run.smithi107.stdout:450971566534 2024-09-21T10:29:06.668 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:06.689 INFO:tasks.cephadm.ceph_manager.ceph:need seq 541165879690 got 541165879693 for osd.3 2024-09-21T10:29:06.690 DEBUG:teuthology.parallel:result is None 2024-09-21T10:29:07.015 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:06 smithi107 ceph-mon[102221]: pgmap v1678: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:07.015 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:06 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/621188797' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:29:07.015 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:06 smithi107 ceph-mon[105022]: pgmap v1678: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:07.015 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:06 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/621188797' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:29:07.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:06 smithi129 ceph-mon[88471]: pgmap v1678: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:07.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:06 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/621188797' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:29:07.435 INFO:tasks.cephadm.ceph_manager.ceph:need seq 450971566530 got 450971566534 for osd.2 2024-09-21T10:29:07.435 DEBUG:teuthology.parallel:result is None 2024-09-21T10:29:08.594 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:08.595 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:08.596 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:08.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:08 smithi107 ceph-mon[102221]: pgmap v1679: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:08.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:08 smithi107 ceph-mon[105022]: pgmap v1679: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:09.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:08 smithi129 ceph-mon[88471]: pgmap v1679: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:09.433 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:10.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:29:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:29:10.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:29:10.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:10 smithi107 ceph-mon[102221]: pgmap v1680: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:10.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:10 smithi107 ceph-mon[105022]: pgmap v1680: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:11.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:10 smithi129 ceph-mon[88471]: pgmap v1680: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:12.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:11 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:12.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:11 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:29:11] "GET /metrics HTTP/1.1" 200 34359 "" "Prometheus/2.51.0" 2024-09-21T10:29:12.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:11 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:12.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:11 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:12.877 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:12 smithi107 ceph-mon[102221]: pgmap v1681: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:12.878 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:12 smithi107 ceph-mon[105022]: pgmap v1681: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:13.168 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:12 smithi129 ceph-mon[88471]: pgmap v1681: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:13.273 INFO:teuthology.orchestra.run.smithi107.stdout:154618823271 2024-09-21T10:29:13.936 INFO:teuthology.orchestra.run.smithi107.stdout:180388627042 2024-09-21T10:29:13.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:29:13.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:13.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:13.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:13.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:13 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:13.937 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:13 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1430484997' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:29:13.937 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:29:13.937 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:13.938 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:13.938 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:13.938 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:13 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:13.938 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:13 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1430484997' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:29:14.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:13 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:29:14.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:13 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:14.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:13 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:14.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:13 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:14.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:13 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:14.170 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:13 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1430484997' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:29:14.196 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618823267 got 154618823271 for osd.5 2024-09-21T10:29:14.197 DEBUG:teuthology.parallel:result is None 2024-09-21T10:29:14.992 INFO:teuthology.orchestra.run.smithi107.stdout:751619277109 2024-09-21T10:29:15.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:14 smithi107 ceph-mon[102221]: pgmap v1682: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:15.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:14 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/228388839' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:29:15.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:14 smithi107 ceph-mon[105022]: pgmap v1682: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:15.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:14 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/228388839' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:29:15.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:14 smithi129 ceph-mon[88471]: pgmap v1682: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:15.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:14 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/228388839' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:29:15.186 INFO:teuthology.orchestra.run.smithi107.stdout:128849019501 2024-09-21T10:29:16.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:15 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1150395714' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:29:16.109 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:15 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/516239971' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T10:29:16.109 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:15 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1150395714' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:29:16.110 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:15 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/516239971' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T10:29:16.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:15 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1150395714' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:29:16.142 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:15 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/516239971' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T10:29:16.317 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849019497 got 128849019501 for osd.4 2024-09-21T10:29:16.317 DEBUG:teuthology.parallel:result is None 2024-09-21T10:29:16.827 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388627039 got 180388627042 for osd.6 2024-09-21T10:29:16.827 DEBUG:teuthology.parallel:result is None 2024-09-21T10:29:17.074 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:16 smithi107 ceph-mon[102221]: pgmap v1683: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:17.074 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:16 smithi107 ceph-mon[105022]: pgmap v1683: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:17.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:16 smithi129 ceph-mon[88471]: pgmap v1683: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:17.604 INFO:tasks.cephadm.ceph_manager.ceph:need seq 751619277106 got 751619277109 for osd.7 2024-09-21T10:29:17.604 DEBUG:teuthology.parallel:result is None 2024-09-21T10:29:18.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:18 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:18.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:18 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:18.524 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:18 smithi107 ceph-mon[105022]: pgmap v1684: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:18.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:18 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:18.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:18 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:18.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:18 smithi107 ceph-mon[102221]: pgmap v1684: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:18 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:18 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:18.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:18 smithi129 ceph-mon[88471]: pgmap v1684: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:19.656 INFO:teuthology.orchestra.run.smithi107.stdout:1052266987760 2024-09-21T10:29:19.859 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:19 smithi107 ceph-mon[105022]: from='client.? 172.21.15.107:0/1474448816' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:29:19.859 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:19 smithi107 ceph-mon[102221]: from='client.? 172.21.15.107:0/1474448816' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:29:19.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:19 smithi129 ceph-mon[88471]: from='client.? 172.21.15.107:0/1474448816' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:29:20.423 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1052266987756 got 1052266987760 for osd.0 2024-09-21T10:29:20.424 DEBUG:teuthology.parallel:result is None 2024-09-21T10:29:20.424 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T10:29:20.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:29:20.562 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:29:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:29:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:29:20.697 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:20.822 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:20 smithi107 ceph-mon[102221]: pgmap v1685: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:20.822 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:20 smithi107 ceph-mon[105022]: pgmap v1685: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:20.919 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:20 smithi129 ceph-mon[88471]: pgmap v1685: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:21.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:21 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:29:21] "GET /metrics HTTP/1.1" 200 34359 "" "Prometheus/2.51.0" 2024-09-21T10:29:22.712 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:22 smithi107 ceph-mon[102221]: pgmap v1686: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:23.025 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:22 smithi107 ceph-mon[105022]: pgmap v1686: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:23.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:22 smithi129 ceph-mon[88471]: pgmap v1686: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:23.223 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:29:23.224 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:29:24.073 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:23 smithi107 ceph-mon[102221]: from='client.47696 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:29:24.074 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:23 smithi107 ceph-mon[105022]: from='client.47696 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:29:24.098 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1686,"stamp":"2024-09-21T10:29:21.656618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":279608,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2159,"ondisk_log_size":2159,"up":279,"acting":279,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":279,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10258892,"kb_used_data":52780,"kb_used_omap":43,"kb_used_meta":10205972,"kb_avail":739702324,"statfs":{"total":767960285184,"available":757455179776,"internally_reserved":0,"allocated":54046720,"data_stored":31840830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44223,"internal_metadata":10450916161},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002856"},"pg_stats":[{"pgid":"1.11","version":"77'100","reported_seq":480,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.841841+0000","last_change":"2024-09-21T10:27:24.964892+0000","last_active":"2024-09-21T10:28:35.841841+0000","last_peered":"2024-09-21T10:28:35.841841+0000","last_clean":"2024-09-21T10:28:35.841841+0000","last_became_active":"2024-09-21T10:27:24.964235+0000","last_became_peered":"2024-09-21T10:27:24.964235+0000","last_unstale":"2024-09-21T10:28:35.841841+0000","last_undegraded":"2024-09-21T10:28:35.841841+0000","last_fullsized":"2024-09-21T10:28:35.841841+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:53.631923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:53.631923+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:00:17.393822+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744799+0000","last_change":"2024-09-21T10:27:24.961323+0000","last_active":"2024-09-21T10:28:34.744799+0000","last_peered":"2024-09-21T10:28:34.744799+0000","last_clean":"2024-09-21T10:28:34.744799+0000","last_became_active":"2024-09-21T10:27:24.960792+0000","last_became_peered":"2024-09-21T10:27:24.960792+0000","last_unstale":"2024-09-21T10:28:34.744799+0000","last_undegraded":"2024-09-21T10:28:34.744799+0000","last_fullsized":"2024-09-21T10:28:34.744799+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:56.559965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":356,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.724067+0000","last_change":"2024-09-21T10:14:30.717695+0000","last_active":"2024-09-21T10:28:30.724067+0000","last_peered":"2024-09-21T10:28:30.724067+0000","last_clean":"2024-09-21T10:28:30.724067+0000","last_became_active":"2024-09-21T10:14:30.705952+0000","last_became_peered":"2024-09-21T10:14:30.705952+0000","last_unstale":"2024-09-21T10:28:30.724067+0000","last_undegraded":"2024-09-21T10:28:30.724067+0000","last_fullsized":"2024-09-21T10:28:30.724067+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:08:18.863430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'100","reported_seq":166,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.503997+0000","last_change":"2024-09-21T10:28:16.369894+0000","last_active":"2024-09-21T10:28:31.503997+0000","last_peered":"2024-09-21T10:28:31.503997+0000","last_clean":"2024-09-21T10:28:31.503997+0000","last_became_active":"2024-09-21T10:28:16.369532+0000","last_became_peered":"2024-09-21T10:28:16.369532+0000","last_unstale":"2024-09-21T10:28:31.503997+0000","last_undegraded":"2024-09-21T10:28:31.503997+0000","last_fullsized":"2024-09-21T10:28:31.503997+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.798577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.798577+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:31:55.319583+0000","scrub_duration":0.0023622640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":471,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504520+0000","last_change":"2024-09-21T10:28:16.368292+0000","last_active":"2024-09-21T10:28:31.504520+0000","last_peered":"2024-09-21T10:28:31.504520+0000","last_clean":"2024-09-21T10:28:31.504520+0000","last_became_active":"2024-09-21T10:28:16.367188+0000","last_became_peered":"2024-09-21T10:28:16.367188+0000","last_unstale":"2024-09-21T10:28:31.504520+0000","last_undegraded":"2024-09-21T10:28:31.504520+0000","last_fullsized":"2024-09-21T10:28:31.504520+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:46.535906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":737,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.841877+0000","last_change":"2024-09-21T10:27:24.953070+0000","last_active":"2024-09-21T10:28:35.841877+0000","last_peered":"2024-09-21T10:28:35.841877+0000","last_clean":"2024-09-21T10:28:35.841877+0000","last_became_active":"2024-09-21T10:27:24.952728+0000","last_became_peered":"2024-09-21T10:27:24.952728+0000","last_unstale":"2024-09-21T10:28:35.841877+0000","last_undegraded":"2024-09-21T10:28:35.841877+0000","last_fullsized":"2024-09-21T10:28:35.841877+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:53.123836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'100","reported_seq":481,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842637+0000","last_change":"2024-09-21T10:27:24.955481+0000","last_active":"2024-09-21T10:28:35.842637+0000","last_peered":"2024-09-21T10:28:35.842637+0000","last_clean":"2024-09-21T10:28:35.842637+0000","last_became_active":"2024-09-21T10:27:24.955114+0000","last_became_peered":"2024-09-21T10:27:24.955114+0000","last_unstale":"2024-09-21T10:28:35.842637+0000","last_undegraded":"2024-09-21T10:28:35.842637+0000","last_fullsized":"2024-09-21T10:28:35.842637+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.433227+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:54.414681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":463,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842278+0000","last_change":"2024-09-21T10:27:24.956694+0000","last_active":"2024-09-21T10:28:35.842278+0000","last_peered":"2024-09-21T10:28:35.842278+0000","last_clean":"2024-09-21T10:28:35.842278+0000","last_became_active":"2024-09-21T10:27:24.956151+0000","last_became_peered":"2024-09-21T10:27:24.956151+0000","last_unstale":"2024-09-21T10:28:35.842278+0000","last_undegraded":"2024-09-21T10:28:35.842278+0000","last_fullsized":"2024-09-21T10:28:35.842278+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:43.175235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":372,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.723785+0000","last_change":"2024-09-21T10:22:19.646202+0000","last_active":"2024-09-21T10:28:30.723785+0000","last_peered":"2024-09-21T10:28:30.723785+0000","last_clean":"2024-09-21T10:28:30.723785+0000","last_became_active":"2024-09-21T10:22:19.645690+0000","last_became_peered":"2024-09-21T10:22:19.645690+0000","last_unstale":"2024-09-21T10:28:30.723785+0000","last_undegraded":"2024-09-21T10:28:30.723785+0000","last_fullsized":"2024-09-21T10:28:30.723785+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:44:01.509115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"77'100","reported_seq":76,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.503997+0000","last_change":"2024-09-21T10:28:16.370122+0000","last_active":"2024-09-21T10:28:31.503997+0000","last_peered":"2024-09-21T10:28:31.503997+0000","last_clean":"2024-09-21T10:28:31.503997+0000","last_became_active":"2024-09-21T10:28:16.369549+0000","last_became_peered":"2024-09-21T10:28:16.369549+0000","last_unstale":"2024-09-21T10:28:31.503997+0000","last_undegraded":"2024-09-21T10:28:31.503997+0000","last_fullsized":"2024-09-21T10:28:31.503997+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:15:02.841968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:15:02.841968+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:10.403124+0000","scrub_duration":0.002846973,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744797+0000","last_change":"2024-09-21T10:27:24.963982+0000","last_active":"2024-09-21T10:28:34.744797+0000","last_peered":"2024-09-21T10:28:34.744797+0000","last_clean":"2024-09-21T10:28:34.744797+0000","last_became_active":"2024-09-21T10:27:24.963354+0000","last_became_peered":"2024-09-21T10:27:24.963354+0000","last_unstale":"2024-09-21T10:28:34.744797+0000","last_undegraded":"2024-09-21T10:28:34.744797+0000","last_fullsized":"2024-09-21T10:28:34.744797+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:34.465081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":88,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504414+0000","last_change":"2024-09-21T10:22:19.662950+0000","last_active":"2024-09-21T10:28:31.504414+0000","last_peered":"2024-09-21T10:28:31.504414+0000","last_clean":"2024-09-21T10:28:31.504414+0000","last_became_active":"2024-09-21T10:22:19.649590+0000","last_became_peered":"2024-09-21T10:22:19.649590+0000","last_unstale":"2024-09-21T10:28:31.504414+0000","last_undegraded":"2024-09-21T10:28:31.504414+0000","last_fullsized":"2024-09-21T10:28:31.504414+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:38:15.512148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"77'100","reported_seq":364,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.724464+0000","last_change":"2024-09-21T10:14:30.708275+0000","last_active":"2024-09-21T10:28:30.724464+0000","last_peered":"2024-09-21T10:28:30.724464+0000","last_clean":"2024-09-21T10:28:30.724464+0000","last_became_active":"2024-09-21T10:14:30.707763+0000","last_became_peered":"2024-09-21T10:14:30.707763+0000","last_unstale":"2024-09-21T10:28:30.724464+0000","last_undegraded":"2024-09-21T10:28:30.724464+0000","last_fullsized":"2024-09-21T10:28:30.724464+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:55.259385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:55.259385+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:41.322447+0000","scrub_duration":0.0015667260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":399,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.724360+0000","last_change":"2024-09-21T10:14:30.720250+0000","last_active":"2024-09-21T10:28:30.724360+0000","last_peered":"2024-09-21T10:28:30.724360+0000","last_clean":"2024-09-21T10:28:30.724360+0000","last_became_active":"2024-09-21T10:14:30.707036+0000","last_became_peered":"2024-09-21T10:14:30.707036+0000","last_unstale":"2024-09-21T10:28:30.724360+0000","last_undegraded":"2024-09-21T10:28:30.724360+0000","last_fullsized":"2024-09-21T10:28:30.724360+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:17.438534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":404,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504145+0000","last_change":"2024-09-21T10:28:16.369325+0000","last_active":"2024-09-21T10:28:31.504145+0000","last_peered":"2024-09-21T10:28:31.504145+0000","last_clean":"2024-09-21T10:28:31.504145+0000","last_became_active":"2024-09-21T10:28:16.369029+0000","last_became_peered":"2024-09-21T10:28:16.369029+0000","last_unstale":"2024-09-21T10:28:31.504145+0000","last_undegraded":"2024-09-21T10:28:31.504145+0000","last_fullsized":"2024-09-21T10:28:31.504145+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:09:57.105557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"77'100","reported_seq":64,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744908+0000","last_change":"2024-09-21T10:28:26.837137+0000","last_active":"2024-09-21T10:28:34.744908+0000","last_peered":"2024-09-21T10:28:34.744908+0000","last_clean":"2024-09-21T10:28:34.744908+0000","last_became_active":"2024-09-21T10:27:24.954327+0000","last_became_peered":"2024-09-21T10:27:24.954327+0000","last_unstale":"2024-09-21T10:28:34.744908+0000","last_undegraded":"2024-09-21T10:28:34.744908+0000","last_fullsized":"2024-09-21T10:28:34.744908+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:28:26.837085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:28:26.837085+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:08.156692+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":378,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.724429+0000","last_change":"2024-09-21T10:18:12.643154+0000","last_active":"2024-09-21T10:28:30.724429+0000","last_peered":"2024-09-21T10:28:30.724429+0000","last_clean":"2024-09-21T10:28:30.724429+0000","last_became_active":"2024-09-21T10:18:12.636287+0000","last_became_peered":"2024-09-21T10:18:12.636287+0000","last_unstale":"2024-09-21T10:28:30.724429+0000","last_undegraded":"2024-09-21T10:28:30.724429+0000","last_fullsized":"2024-09-21T10:28:30.724429+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:45:31.008309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":261,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933426+0000","last_change":"2024-09-21T10:28:09.280335+0000","last_active":"2024-09-21T10:28:30.933426+0000","last_peered":"2024-09-21T10:28:30.933426+0000","last_clean":"2024-09-21T10:28:30.933426+0000","last_became_active":"2024-09-21T10:28:09.280073+0000","last_became_peered":"2024-09-21T10:28:09.280073+0000","last_unstale":"2024-09-21T10:28:30.933426+0000","last_undegraded":"2024-09-21T10:28:30.933426+0000","last_fullsized":"2024-09-21T10:28:30.933426+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:52.156080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"77'100","reported_seq":78,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504993+0000","last_change":"2024-09-21T10:22:19.657392+0000","last_active":"2024-09-21T10:28:31.504993+0000","last_peered":"2024-09-21T10:28:31.504993+0000","last_clean":"2024-09-21T10:28:31.504993+0000","last_became_active":"2024-09-21T10:22:19.650821+0000","last_became_peered":"2024-09-21T10:22:19.650821+0000","last_unstale":"2024-09-21T10:28:31.504993+0000","last_undegraded":"2024-09-21T10:28:31.504993+0000","last_fullsized":"2024-09-21T10:28:31.504993+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.289859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.289859+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:54:14.510032+0000","scrub_duration":0.001251809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":378,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.724265+0000","last_change":"2024-09-21T10:22:19.664692+0000","last_active":"2024-09-21T10:28:30.724265+0000","last_peered":"2024-09-21T10:28:30.724265+0000","last_clean":"2024-09-21T10:28:30.724265+0000","last_became_active":"2024-09-21T10:22:19.647696+0000","last_became_peered":"2024-09-21T10:22:19.647696+0000","last_unstale":"2024-09-21T10:28:30.724265+0000","last_undegraded":"2024-09-21T10:28:30.724265+0000","last_fullsized":"2024-09-21T10:28:30.724265+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:57:54.264752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":78,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504827+0000","last_change":"2024-09-21T10:22:19.652364+0000","last_active":"2024-09-21T10:28:31.504827+0000","last_peered":"2024-09-21T10:28:31.504827+0000","last_clean":"2024-09-21T10:28:31.504827+0000","last_became_active":"2024-09-21T10:22:19.651586+0000","last_became_peered":"2024-09-21T10:22:19.651586+0000","last_unstale":"2024-09-21T10:28:31.504827+0000","last_undegraded":"2024-09-21T10:28:31.504827+0000","last_fullsized":"2024-09-21T10:28:31.504827+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:03.593157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"77'100","reported_seq":71,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933904+0000","last_change":"2024-09-21T10:28:09.283812+0000","last_active":"2024-09-21T10:28:30.933904+0000","last_peered":"2024-09-21T10:28:30.933904+0000","last_clean":"2024-09-21T10:28:30.933904+0000","last_became_active":"2024-09-21T10:28:09.283516+0000","last_became_peered":"2024-09-21T10:28:09.283516+0000","last_unstale":"2024-09-21T10:28:30.933904+0000","last_undegraded":"2024-09-21T10:28:30.933904+0000","last_fullsized":"2024-09-21T10:28:30.933904+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:18:27.085525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:18:27.085525+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:05.245886+0000","scrub_duration":0.0036485279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":495,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504593+0000","last_change":"2024-09-21T10:28:16.370043+0000","last_active":"2024-09-21T10:28:31.504593+0000","last_peered":"2024-09-21T10:28:31.504593+0000","last_clean":"2024-09-21T10:28:31.504593+0000","last_became_active":"2024-09-21T10:28:16.369162+0000","last_became_peered":"2024-09-21T10:28:16.369162+0000","last_unstale":"2024-09-21T10:28:31.504593+0000","last_undegraded":"2024-09-21T10:28:31.504593+0000","last_fullsized":"2024-09-21T10:28:31.504593+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:18.580934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":710,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842048+0000","last_change":"2024-09-21T10:14:30.711105+0000","last_active":"2024-09-21T10:28:35.842048+0000","last_peered":"2024-09-21T10:28:35.842048+0000","last_clean":"2024-09-21T10:28:35.842048+0000","last_became_active":"2024-09-21T10:14:30.710759+0000","last_became_peered":"2024-09-21T10:14:30.710759+0000","last_unstale":"2024-09-21T10:28:35.842048+0000","last_undegraded":"2024-09-21T10:28:35.842048+0000","last_fullsized":"2024-09-21T10:28:35.842048+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:40.690042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":364,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.723970+0000","last_change":"2024-09-21T10:27:24.952212+0000","last_active":"2024-09-21T10:28:30.723970+0000","last_peered":"2024-09-21T10:28:30.723970+0000","last_clean":"2024-09-21T10:28:30.723970+0000","last_became_active":"2024-09-21T10:27:24.951883+0000","last_became_peered":"2024-09-21T10:27:24.951883+0000","last_unstale":"2024-09-21T10:28:30.723970+0000","last_undegraded":"2024-09-21T10:28:30.723970+0000","last_fullsized":"2024-09-21T10:28:30.723970+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.355912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.355912+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:19:22.848110+0000","scrub_duration":0.0017980030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":56,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744211+0000","last_change":"2024-09-21T10:27:24.968357+0000","last_active":"2024-09-21T10:28:34.744211+0000","last_peered":"2024-09-21T10:28:34.744211+0000","last_clean":"2024-09-21T10:28:34.744211+0000","last_became_active":"2024-09-21T10:27:24.954336+0000","last_became_peered":"2024-09-21T10:27:24.954336+0000","last_unstale":"2024-09-21T10:28:34.744211+0000","last_undegraded":"2024-09-21T10:28:34.744211+0000","last_fullsized":"2024-09-21T10:28:34.744211+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:05.711937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":277,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933671+0000","last_change":"2024-09-21T10:28:09.283069+0000","last_active":"2024-09-21T10:28:30.933671+0000","last_peered":"2024-09-21T10:28:30.933671+0000","last_clean":"2024-09-21T10:28:30.933671+0000","last_became_active":"2024-09-21T10:28:09.282668+0000","last_became_peered":"2024-09-21T10:28:09.282668+0000","last_unstale":"2024-09-21T10:28:30.933671+0000","last_undegraded":"2024-09-21T10:28:30.933671+0000","last_fullsized":"2024-09-21T10:28:30.933671+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:48:42.023717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":149,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.503872+0000","last_change":"2024-09-21T10:28:16.367871+0000","last_active":"2024-09-21T10:28:31.503872+0000","last_peered":"2024-09-21T10:28:31.503872+0000","last_clean":"2024-09-21T10:28:31.503872+0000","last_became_active":"2024-09-21T10:28:16.367530+0000","last_became_peered":"2024-09-21T10:28:16.367530+0000","last_unstale":"2024-09-21T10:28:31.503872+0000","last_undegraded":"2024-09-21T10:28:31.503872+0000","last_fullsized":"2024-09-21T10:28:31.503872+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:02.152981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:02.152981+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:44:55.561812+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":78,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504591+0000","last_change":"2024-09-21T10:22:19.652182+0000","last_active":"2024-09-21T10:28:31.504591+0000","last_peered":"2024-09-21T10:28:31.504591+0000","last_clean":"2024-09-21T10:28:31.504591+0000","last_became_active":"2024-09-21T10:22:19.651703+0000","last_became_peered":"2024-09-21T10:22:19.651703+0000","last_unstale":"2024-09-21T10:28:31.504591+0000","last_undegraded":"2024-09-21T10:28:31.504591+0000","last_fullsized":"2024-09-21T10:28:31.504591+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:08.752662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":420,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933741+0000","last_change":"2024-09-21T10:28:09.282543+0000","last_active":"2024-09-21T10:28:30.933741+0000","last_peered":"2024-09-21T10:28:30.933741+0000","last_clean":"2024-09-21T10:28:30.933741+0000","last_became_active":"2024-09-21T10:28:09.281807+0000","last_became_peered":"2024-09-21T10:28:09.281807+0000","last_unstale":"2024-09-21T10:28:30.933741+0000","last_undegraded":"2024-09-21T10:28:30.933741+0000","last_fullsized":"2024-09-21T10:28:30.933741+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:55:53.868908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"77'100","reported_seq":457,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842534+0000","last_change":"2024-09-21T10:14:05.474100+0000","last_active":"2024-09-21T10:28:35.842534+0000","last_peered":"2024-09-21T10:28:35.842534+0000","last_clean":"2024-09-21T10:28:35.842534+0000","last_became_active":"2024-09-21T10:14:05.473336+0000","last_became_peered":"2024-09-21T10:14:05.473336+0000","last_unstale":"2024-09-21T10:28:35.842534+0000","last_undegraded":"2024-09-21T10:28:35.842534+0000","last_fullsized":"2024-09-21T10:28:35.842534+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.540323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.540323+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:06:10.963352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":512,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933243+0000","last_change":"2024-09-21T10:28:09.282222+0000","last_active":"2024-09-21T10:28:30.933243+0000","last_peered":"2024-09-21T10:28:30.933243+0000","last_clean":"2024-09-21T10:28:30.933243+0000","last_became_active":"2024-09-21T10:28:09.281541+0000","last_became_peered":"2024-09-21T10:28:09.281541+0000","last_unstale":"2024-09-21T10:28:30.933243+0000","last_undegraded":"2024-09-21T10:28:30.933243+0000","last_fullsized":"2024-09-21T10:28:30.933243+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:08:25.537544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":290,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.934021+0000","last_change":"2024-09-21T10:28:09.282671+0000","last_active":"2024-09-21T10:28:30.934021+0000","last_peered":"2024-09-21T10:28:30.934021+0000","last_clean":"2024-09-21T10:28:30.934021+0000","last_became_active":"2024-09-21T10:28:09.282124+0000","last_became_peered":"2024-09-21T10:28:09.282124+0000","last_unstale":"2024-09-21T10:28:30.934021+0000","last_undegraded":"2024-09-21T10:28:30.934021+0000","last_fullsized":"2024-09-21T10:28:30.934021+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:35:54.562251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":67,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504241+0000","last_change":"2024-09-21T10:28:16.368890+0000","last_active":"2024-09-21T10:28:31.504241+0000","last_peered":"2024-09-21T10:28:31.504241+0000","last_clean":"2024-09-21T10:28:31.504241+0000","last_became_active":"2024-09-21T10:28:16.368573+0000","last_became_peered":"2024-09-21T10:28:16.368573+0000","last_unstale":"2024-09-21T10:28:31.504241+0000","last_undegraded":"2024-09-21T10:28:31.504241+0000","last_fullsized":"2024-09-21T10:28:31.504241+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:20:00.237668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:20:00.237668+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:32:59.606457+0000","scrub_duration":0.001567564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744208+0000","last_change":"2024-09-21T10:27:24.950027+0000","last_active":"2024-09-21T10:28:34.744208+0000","last_peered":"2024-09-21T10:28:34.744208+0000","last_clean":"2024-09-21T10:28:34.744208+0000","last_became_active":"2024-09-21T10:27:24.949461+0000","last_became_peered":"2024-09-21T10:27:24.949461+0000","last_unstale":"2024-09-21T10:28:34.744208+0000","last_undegraded":"2024-09-21T10:28:34.744208+0000","last_fullsized":"2024-09-21T10:28:34.744208+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:47.019081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":677,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842495+0000","last_change":"2024-09-21T10:27:24.951877+0000","last_active":"2024-09-21T10:28:35.842495+0000","last_peered":"2024-09-21T10:28:35.842495+0000","last_clean":"2024-09-21T10:28:35.842495+0000","last_became_active":"2024-09-21T10:27:24.951534+0000","last_became_peered":"2024-09-21T10:27:24.951534+0000","last_unstale":"2024-09-21T10:28:35.842495+0000","last_undegraded":"2024-09-21T10:28:35.842495+0000","last_fullsized":"2024-09-21T10:28:35.842495+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:47.963923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":381,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.723865+0000","last_change":"2024-09-21T10:22:19.655993+0000","last_active":"2024-09-21T10:28:30.723865+0000","last_peered":"2024-09-21T10:28:30.723865+0000","last_clean":"2024-09-21T10:28:30.723865+0000","last_became_active":"2024-09-21T10:22:19.655178+0000","last_became_peered":"2024-09-21T10:22:19.655178+0000","last_unstale":"2024-09-21T10:28:30.723865+0000","last_undegraded":"2024-09-21T10:28:30.723865+0000","last_fullsized":"2024-09-21T10:28:30.723865+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:47.393146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:47.393146+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:48.668712+0000","scrub_duration":0.001601086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":148,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504334+0000","last_change":"2024-09-21T10:28:16.367821+0000","last_active":"2024-09-21T10:28:31.504334+0000","last_peered":"2024-09-21T10:28:31.504334+0000","last_clean":"2024-09-21T10:28:31.504334+0000","last_became_active":"2024-09-21T10:28:16.367445+0000","last_became_peered":"2024-09-21T10:28:16.367445+0000","last_unstale":"2024-09-21T10:28:31.504334+0000","last_undegraded":"2024-09-21T10:28:31.504334+0000","last_fullsized":"2024-09-21T10:28:31.504334+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:49.941634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744295+0000","last_change":"2024-09-21T10:27:24.959198+0000","last_active":"2024-09-21T10:28:34.744295+0000","last_peered":"2024-09-21T10:28:34.744295+0000","last_clean":"2024-09-21T10:28:34.744295+0000","last_became_active":"2024-09-21T10:27:24.953113+0000","last_became_peered":"2024-09-21T10:27:24.953113+0000","last_unstale":"2024-09-21T10:28:34.744295+0000","last_undegraded":"2024-09-21T10:28:34.744295+0000","last_fullsized":"2024-09-21T10:28:34.744295+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:29:10.801832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":78,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504673+0000","last_change":"2024-09-21T10:22:19.651173+0000","last_active":"2024-09-21T10:28:31.504673+0000","last_peered":"2024-09-21T10:28:31.504673+0000","last_clean":"2024-09-21T10:28:31.504673+0000","last_became_active":"2024-09-21T10:22:19.649673+0000","last_became_peered":"2024-09-21T10:22:19.649673+0000","last_unstale":"2024-09-21T10:28:31.504673+0000","last_undegraded":"2024-09-21T10:28:31.504673+0000","last_fullsized":"2024-09-21T10:28:31.504673+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:28:37.605825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":482,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842467+0000","last_change":"2024-09-21T10:22:19.651858+0000","last_active":"2024-09-21T10:28:35.842467+0000","last_peered":"2024-09-21T10:28:35.842467+0000","last_clean":"2024-09-21T10:28:35.842467+0000","last_became_active":"2024-09-21T10:22:19.651510+0000","last_became_peered":"2024-09-21T10:22:19.651510+0000","last_unstale":"2024-09-21T10:28:35.842467+0000","last_undegraded":"2024-09-21T10:28:35.842467+0000","last_fullsized":"2024-09-21T10:28:35.842467+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.550551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.550551+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:01:42.501915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":335,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.723841+0000","last_change":"2024-09-21T10:09:23.838944+0000","last_active":"2024-09-21T10:28:30.723841+0000","last_peered":"2024-09-21T10:28:30.723841+0000","last_clean":"2024-09-21T10:28:30.723841+0000","last_became_active":"2024-09-21T10:09:23.837886+0000","last_became_peered":"2024-09-21T10:09:23.837886+0000","last_unstale":"2024-09-21T10:28:30.723841+0000","last_undegraded":"2024-09-21T10:28:30.723841+0000","last_fullsized":"2024-09-21T10:28:30.723841+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:16.928195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":702,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.841753+0000","last_change":"2024-09-21T10:27:24.953262+0000","last_active":"2024-09-21T10:28:35.841753+0000","last_peered":"2024-09-21T10:28:35.841753+0000","last_clean":"2024-09-21T10:28:35.841753+0000","last_became_active":"2024-09-21T10:27:24.952939+0000","last_became_peered":"2024-09-21T10:27:24.952939+0000","last_unstale":"2024-09-21T10:28:35.841753+0000","last_undegraded":"2024-09-21T10:28:35.841753+0000","last_fullsized":"2024-09-21T10:28:35.841753+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:49:46.284356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":449,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842196+0000","last_change":"2024-09-21T10:09:49.923044+0000","last_active":"2024-09-21T10:28:35.842196+0000","last_peered":"2024-09-21T10:28:35.842196+0000","last_clean":"2024-09-21T10:28:35.842196+0000","last_became_active":"2024-09-21T10:09:49.922711+0000","last_became_peered":"2024-09-21T10:09:49.922711+0000","last_unstale":"2024-09-21T10:28:35.842196+0000","last_undegraded":"2024-09-21T10:28:35.842196+0000","last_fullsized":"2024-09-21T10:28:35.842196+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:14.090084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":385,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.723981+0000","last_change":"2024-09-21T10:18:12.636851+0000","last_active":"2024-09-21T10:28:30.723981+0000","last_peered":"2024-09-21T10:28:30.723981+0000","last_clean":"2024-09-21T10:28:30.723981+0000","last_became_active":"2024-09-21T10:18:12.636513+0000","last_became_peered":"2024-09-21T10:18:12.636513+0000","last_unstale":"2024-09-21T10:28:30.723981+0000","last_undegraded":"2024-09-21T10:28:30.723981+0000","last_fullsized":"2024-09-21T10:28:30.723981+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:52.361137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:52.361137+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:21.025735+0000","scrub_duration":0.001255358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":857,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.746248+0000","last_change":"2024-09-21T10:27:24.955527+0000","last_active":"2024-09-21T10:28:34.746248+0000","last_peered":"2024-09-21T10:28:34.746248+0000","last_clean":"2024-09-21T10:28:34.746248+0000","last_became_active":"2024-09-21T10:27:24.955242+0000","last_became_peered":"2024-09-21T10:27:24.955242+0000","last_unstale":"2024-09-21T10:28:34.746248+0000","last_undegraded":"2024-09-21T10:28:34.746248+0000","last_fullsized":"2024-09-21T10:28:34.746248+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:10.360228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":239,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.723748+0000","last_change":"2024-09-21T10:22:19.647946+0000","last_active":"2024-09-21T10:28:30.723748+0000","last_peered":"2024-09-21T10:28:30.723748+0000","last_clean":"2024-09-21T10:28:30.723748+0000","last_became_active":"2024-09-21T10:22:19.647057+0000","last_became_peered":"2024-09-21T10:22:19.647057+0000","last_unstale":"2024-09-21T10:28:30.723748+0000","last_undegraded":"2024-09-21T10:28:30.723748+0000","last_fullsized":"2024-09-21T10:28:30.723748+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:06.839922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":836,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.746647+0000","last_change":"2024-09-21T10:10:57.395073+0000","last_active":"2024-09-21T10:28:34.746647+0000","last_peered":"2024-09-21T10:28:34.746647+0000","last_clean":"2024-09-21T10:28:34.746647+0000","last_became_active":"2024-09-21T10:10:57.394674+0000","last_became_peered":"2024-09-21T10:10:57.394674+0000","last_unstale":"2024-09-21T10:28:34.746647+0000","last_undegraded":"2024-09-21T10:28:34.746647+0000","last_fullsized":"2024-09-21T10:28:34.746647+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:48:48.912649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":298,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.503914+0000","last_change":"2024-09-21T10:28:16.367768+0000","last_active":"2024-09-21T10:28:31.503914+0000","last_peered":"2024-09-21T10:28:31.503914+0000","last_clean":"2024-09-21T10:28:31.503914+0000","last_became_active":"2024-09-21T10:28:16.367348+0000","last_became_peered":"2024-09-21T10:28:16.367348+0000","last_unstale":"2024-09-21T10:28:31.503914+0000","last_undegraded":"2024-09-21T10:28:31.503914+0000","last_fullsized":"2024-09-21T10:28:31.503914+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:04.043917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:04.043917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:53:29.741459+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":215,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504644+0000","last_change":"2024-09-21T10:28:16.370152+0000","last_active":"2024-09-21T10:28:31.504644+0000","last_peered":"2024-09-21T10:28:31.504644+0000","last_clean":"2024-09-21T10:28:31.504644+0000","last_became_active":"2024-09-21T10:28:16.369380+0000","last_became_peered":"2024-09-21T10:28:16.369380+0000","last_unstale":"2024-09-21T10:28:31.504644+0000","last_undegraded":"2024-09-21T10:28:31.504644+0000","last_fullsized":"2024-09-21T10:28:31.504644+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:59:24.527440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091939799999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":207,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933810+0000","last_change":"2024-09-21T10:28:09.539529+0000","last_active":"2024-09-21T10:28:30.933810+0000","last_peered":"2024-09-21T10:28:30.933810+0000","last_clean":"2024-09-21T10:28:30.933810+0000","last_became_active":"2024-09-21T10:28:09.539155+0000","last_became_peered":"2024-09-21T10:28:09.539155+0000","last_unstale":"2024-09-21T10:28:30.933810+0000","last_undegraded":"2024-09-21T10:28:30.933810+0000","last_fullsized":"2024-09-21T10:28:30.933810+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:34:06.730590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":683,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842106+0000","last_change":"2024-09-21T10:27:24.953701+0000","last_active":"2024-09-21T10:28:35.842106+0000","last_peered":"2024-09-21T10:28:35.842106+0000","last_clean":"2024-09-21T10:28:35.842106+0000","last_became_active":"2024-09-21T10:27:24.953371+0000","last_became_peered":"2024-09-21T10:27:24.953371+0000","last_unstale":"2024-09-21T10:28:35.842106+0000","last_undegraded":"2024-09-21T10:28:35.842106+0000","last_fullsized":"2024-09-21T10:28:35.842106+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:57:47.932977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":98,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504903+0000","last_change":"2024-09-21T10:27:24.949698+0000","last_active":"2024-09-21T10:28:31.504903+0000","last_peered":"2024-09-21T10:28:31.504903+0000","last_clean":"2024-09-21T10:28:31.504903+0000","last_became_active":"2024-09-21T10:27:24.949367+0000","last_became_peered":"2024-09-21T10:27:24.949367+0000","last_unstale":"2024-09-21T10:28:31.504903+0000","last_undegraded":"2024-09-21T10:28:31.504903+0000","last_fullsized":"2024-09-21T10:28:31.504903+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:21:47.185890+0000","last_deep_scrub":"77'100","last_deep_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_clean_scrub_stamp":"2024-09-21T10:21:47.185890+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:07.437186+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":189,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504432+0000","last_change":"2024-09-21T10:28:16.369315+0000","last_active":"2024-09-21T10:28:31.504432+0000","last_peered":"2024-09-21T10:28:31.504432+0000","last_clean":"2024-09-21T10:28:31.504432+0000","last_became_active":"2024-09-21T10:28:16.368614+0000","last_became_peered":"2024-09-21T10:28:16.368614+0000","last_unstale":"2024-09-21T10:28:31.504432+0000","last_undegraded":"2024-09-21T10:28:31.504432+0000","last_fullsized":"2024-09-21T10:28:31.504432+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:41:50.431319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":725,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842290+0000","last_change":"2024-09-21T10:14:05.475970+0000","last_active":"2024-09-21T10:28:35.842290+0000","last_peered":"2024-09-21T10:28:35.842290+0000","last_clean":"2024-09-21T10:28:35.842290+0000","last_became_active":"2024-09-21T10:14:05.475331+0000","last_became_peered":"2024-09-21T10:14:05.475331+0000","last_unstale":"2024-09-21T10:28:35.842290+0000","last_undegraded":"2024-09-21T10:28:35.842290+0000","last_fullsized":"2024-09-21T10:28:35.842290+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:17.069736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":661,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842460+0000","last_change":"2024-09-21T10:18:13.039172+0000","last_active":"2024-09-21T10:28:35.842460+0000","last_peered":"2024-09-21T10:28:35.842460+0000","last_clean":"2024-09-21T10:28:35.842460+0000","last_became_active":"2024-09-21T10:18:13.038756+0000","last_became_peered":"2024-09-21T10:18:13.038756+0000","last_unstale":"2024-09-21T10:28:35.842460+0000","last_undegraded":"2024-09-21T10:28:35.842460+0000","last_fullsized":"2024-09-21T10:28:35.842460+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:06.668926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":98,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504943+0000","last_change":"2024-09-21T10:27:24.955530+0000","last_active":"2024-09-21T10:28:31.504943+0000","last_peered":"2024-09-21T10:28:31.504943+0000","last_clean":"2024-09-21T10:28:31.504943+0000","last_became_active":"2024-09-21T10:27:24.955183+0000","last_became_peered":"2024-09-21T10:27:24.955183+0000","last_unstale":"2024-09-21T10:28:31.504943+0000","last_undegraded":"2024-09-21T10:28:31.504943+0000","last_fullsized":"2024-09-21T10:28:31.504943+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:20:01.397007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:20:01.397007+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:46:26.990160+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":657,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842202+0000","last_change":"2024-09-21T10:14:05.478152+0000","last_active":"2024-09-21T10:28:35.842202+0000","last_peered":"2024-09-21T10:28:35.842202+0000","last_clean":"2024-09-21T10:28:35.842202+0000","last_became_active":"2024-09-21T10:14:05.477547+0000","last_became_peered":"2024-09-21T10:14:05.477547+0000","last_unstale":"2024-09-21T10:28:35.842202+0000","last_undegraded":"2024-09-21T10:28:35.842202+0000","last_fullsized":"2024-09-21T10:28:35.842202+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:34.646046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":365,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.724291+0000","last_change":"2024-09-21T10:14:30.708090+0000","last_active":"2024-09-21T10:28:30.724291+0000","last_peered":"2024-09-21T10:28:30.724291+0000","last_clean":"2024-09-21T10:28:30.724291+0000","last_became_active":"2024-09-21T10:14:30.707266+0000","last_became_peered":"2024-09-21T10:14:30.707266+0000","last_unstale":"2024-09-21T10:28:30.724291+0000","last_undegraded":"2024-09-21T10:28:30.724291+0000","last_fullsized":"2024-09-21T10:28:30.724291+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:38.474537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":719,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842267+0000","last_change":"2024-09-21T10:10:57.397251+0000","last_active":"2024-09-21T10:28:35.842267+0000","last_peered":"2024-09-21T10:28:35.842267+0000","last_clean":"2024-09-21T10:28:35.842267+0000","last_became_active":"2024-09-21T10:10:57.396271+0000","last_became_peered":"2024-09-21T10:10:57.396271+0000","last_unstale":"2024-09-21T10:28:35.842267+0000","last_undegraded":"2024-09-21T10:28:35.842267+0000","last_fullsized":"2024-09-21T10:28:35.842267+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:17.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":135,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933926+0000","last_change":"2024-09-21T10:28:09.283444+0000","last_active":"2024-09-21T10:28:30.933926+0000","last_peered":"2024-09-21T10:28:30.933926+0000","last_clean":"2024-09-21T10:28:30.933926+0000","last_became_active":"2024-09-21T10:28:09.282358+0000","last_became_peered":"2024-09-21T10:28:09.282358+0000","last_unstale":"2024-09-21T10:28:30.933926+0000","last_undegraded":"2024-09-21T10:28:30.933926+0000","last_fullsized":"2024-09-21T10:28:30.933926+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:59.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:59.099852+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:00.002498+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":56,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744607+0000","last_change":"2024-09-21T10:27:24.970670+0000","last_active":"2024-09-21T10:28:34.744607+0000","last_peered":"2024-09-21T10:28:34.744607+0000","last_clean":"2024-09-21T10:28:34.744607+0000","last_became_active":"2024-09-21T10:27:24.958254+0000","last_became_peered":"2024-09-21T10:27:24.958254+0000","last_unstale":"2024-09-21T10:28:34.744607+0000","last_undegraded":"2024-09-21T10:28:34.744607+0000","last_fullsized":"2024-09-21T10:28:34.744607+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:00:23.704262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":678,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842552+0000","last_change":"2024-09-21T10:22:19.644169+0000","last_active":"2024-09-21T10:28:35.842552+0000","last_peered":"2024-09-21T10:28:35.842552+0000","last_clean":"2024-09-21T10:28:35.842552+0000","last_became_active":"2024-09-21T10:22:19.643917+0000","last_became_peered":"2024-09-21T10:22:19.643917+0000","last_unstale":"2024-09-21T10:28:35.842552+0000","last_undegraded":"2024-09-21T10:28:35.842552+0000","last_fullsized":"2024-09-21T10:28:35.842552+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:28.976858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744102+0000","last_change":"2024-09-21T10:27:24.950768+0000","last_active":"2024-09-21T10:28:34.744102+0000","last_peered":"2024-09-21T10:28:34.744102+0000","last_clean":"2024-09-21T10:28:34.744102+0000","last_became_active":"2024-09-21T10:27:24.950491+0000","last_became_peered":"2024-09-21T10:27:24.950491+0000","last_unstale":"2024-09-21T10:28:34.744102+0000","last_undegraded":"2024-09-21T10:28:34.744102+0000","last_fullsized":"2024-09-21T10:28:34.744102+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:10:20.907674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744613+0000","last_change":"2024-09-21T10:27:24.963317+0000","last_active":"2024-09-21T10:28:34.744613+0000","last_peered":"2024-09-21T10:28:34.744613+0000","last_clean":"2024-09-21T10:28:34.744613+0000","last_became_active":"2024-09-21T10:27:24.961251+0000","last_became_peered":"2024-09-21T10:27:24.961251+0000","last_unstale":"2024-09-21T10:28:34.744613+0000","last_undegraded":"2024-09-21T10:28:34.744613+0000","last_fullsized":"2024-09-21T10:28:34.744613+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:27:06.813004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:27:06.813004+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:11:28.686047+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":464,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842323+0000","last_change":"2024-09-21T10:14:30.718942+0000","last_active":"2024-09-21T10:28:35.842323+0000","last_peered":"2024-09-21T10:28:35.842323+0000","last_clean":"2024-09-21T10:28:35.842323+0000","last_became_active":"2024-09-21T10:14:30.709129+0000","last_became_peered":"2024-09-21T10:14:30.709129+0000","last_unstale":"2024-09-21T10:28:35.842323+0000","last_undegraded":"2024-09-21T10:28:35.842323+0000","last_fullsized":"2024-09-21T10:28:35.842323+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:18.171064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":78,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504811+0000","last_change":"2024-09-21T10:22:19.646015+0000","last_active":"2024-09-21T10:28:31.504811+0000","last_peered":"2024-09-21T10:28:31.504811+0000","last_clean":"2024-09-21T10:28:31.504811+0000","last_became_active":"2024-09-21T10:22:19.645608+0000","last_became_peered":"2024-09-21T10:22:19.645608+0000","last_unstale":"2024-09-21T10:28:31.504811+0000","last_undegraded":"2024-09-21T10:28:31.504811+0000","last_fullsized":"2024-09-21T10:28:31.504811+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:14.920346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":135,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933367+0000","last_change":"2024-09-21T10:28:09.282860+0000","last_active":"2024-09-21T10:28:30.933367+0000","last_peered":"2024-09-21T10:28:30.933367+0000","last_clean":"2024-09-21T10:28:30.933367+0000","last_became_active":"2024-09-21T10:28:09.282554+0000","last_became_peered":"2024-09-21T10:28:09.282554+0000","last_unstale":"2024-09-21T10:28:30.933367+0000","last_undegraded":"2024-09-21T10:28:30.933367+0000","last_fullsized":"2024-09-21T10:28:30.933367+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:22:33.734605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":135,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933888+0000","last_change":"2024-09-21T10:28:09.282465+0000","last_active":"2024-09-21T10:28:30.933888+0000","last_peered":"2024-09-21T10:28:30.933888+0000","last_clean":"2024-09-21T10:28:30.933888+0000","last_became_active":"2024-09-21T10:28:09.282190+0000","last_became_peered":"2024-09-21T10:28:09.282190+0000","last_unstale":"2024-09-21T10:28:30.933888+0000","last_undegraded":"2024-09-21T10:28:30.933888+0000","last_fullsized":"2024-09-21T10:28:30.933888+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:08:53.735076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":155,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933533+0000","last_change":"2024-09-21T10:28:09.283643+0000","last_active":"2024-09-21T10:28:30.933533+0000","last_peered":"2024-09-21T10:28:30.933533+0000","last_clean":"2024-09-21T10:28:30.933533+0000","last_became_active":"2024-09-21T10:28:09.283028+0000","last_became_peered":"2024-09-21T10:28:09.283028+0000","last_unstale":"2024-09-21T10:28:30.933533+0000","last_undegraded":"2024-09-21T10:28:30.933533+0000","last_fullsized":"2024-09-21T10:28:30.933533+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:45:25.893238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14232,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":729,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.841987+0000","last_change":"2024-09-21T10:22:19.644482+0000","last_active":"2024-09-21T10:28:35.841987+0000","last_peered":"2024-09-21T10:28:35.841987+0000","last_clean":"2024-09-21T10:28:35.841987+0000","last_became_active":"2024-09-21T10:22:19.644080+0000","last_became_peered":"2024-09-21T10:22:19.644080+0000","last_unstale":"2024-09-21T10:28:35.841987+0000","last_undegraded":"2024-09-21T10:28:35.841987+0000","last_fullsized":"2024-09-21T10:28:35.841987+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:42:37.848739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":484,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.746406+0000","last_change":"2024-09-21T10:27:24.966613+0000","last_active":"2024-09-21T10:28:34.746406+0000","last_peered":"2024-09-21T10:28:34.746406+0000","last_clean":"2024-09-21T10:28:34.746406+0000","last_became_active":"2024-09-21T10:27:24.954961+0000","last_became_peered":"2024-09-21T10:27:24.954961+0000","last_unstale":"2024-09-21T10:28:34.746406+0000","last_undegraded":"2024-09-21T10:28:34.746406+0000","last_fullsized":"2024-09-21T10:28:34.746406+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:15.918530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'100","reported_seq":610,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.746313+0000","last_change":"2024-09-21T10:18:12.633959+0000","last_active":"2024-09-21T10:28:34.746313+0000","last_peered":"2024-09-21T10:28:34.746313+0000","last_clean":"2024-09-21T10:28:34.746313+0000","last_became_active":"2024-09-21T10:18:12.633591+0000","last_became_peered":"2024-09-21T10:18:12.633591+0000","last_unstale":"2024-09-21T10:28:34.746313+0000","last_undegraded":"2024-09-21T10:28:34.746313+0000","last_fullsized":"2024-09-21T10:28:34.746313+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.014993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.014993+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:29:16.343472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":231,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504312+0000","last_change":"2024-09-21T10:28:16.363279+0000","last_active":"2024-09-21T10:28:31.504312+0000","last_peered":"2024-09-21T10:28:31.504312+0000","last_clean":"2024-09-21T10:28:31.504312+0000","last_became_active":"2024-09-21T10:28:16.362944+0000","last_became_peered":"2024-09-21T10:28:16.362944+0000","last_unstale":"2024-09-21T10:28:31.504312+0000","last_undegraded":"2024-09-21T10:28:31.504312+0000","last_fullsized":"2024-09-21T10:28:31.504312+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:39:17.368924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744423+0000","last_change":"2024-09-21T10:27:24.954422+0000","last_active":"2024-09-21T10:28:34.744423+0000","last_peered":"2024-09-21T10:28:34.744423+0000","last_clean":"2024-09-21T10:28:34.744423+0000","last_became_active":"2024-09-21T10:27:24.953876+0000","last_became_peered":"2024-09-21T10:27:24.953876+0000","last_unstale":"2024-09-21T10:28:34.744423+0000","last_undegraded":"2024-09-21T10:28:34.744423+0000","last_fullsized":"2024-09-21T10:28:34.744423+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:20.203783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":707,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842245+0000","last_change":"2024-09-21T10:27:24.955590+0000","last_active":"2024-09-21T10:28:35.842245+0000","last_peered":"2024-09-21T10:28:35.842245+0000","last_clean":"2024-09-21T10:28:35.842245+0000","last_became_active":"2024-09-21T10:27:24.955293+0000","last_became_peered":"2024-09-21T10:27:24.955293+0000","last_unstale":"2024-09-21T10:28:35.842245+0000","last_undegraded":"2024-09-21T10:28:35.842245+0000","last_fullsized":"2024-09-21T10:28:35.842245+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:36:15.305320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744179+0000","last_change":"2024-09-21T10:27:24.960830+0000","last_active":"2024-09-21T10:28:34.744179+0000","last_peered":"2024-09-21T10:28:34.744179+0000","last_clean":"2024-09-21T10:28:34.744179+0000","last_became_active":"2024-09-21T10:27:24.958919+0000","last_became_peered":"2024-09-21T10:27:24.958919+0000","last_unstale":"2024-09-21T10:28:34.744179+0000","last_undegraded":"2024-09-21T10:28:34.744179+0000","last_fullsized":"2024-09-21T10:28:34.744179+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:07:10.637181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":78,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.505009+0000","last_change":"2024-09-21T10:22:19.649280+0000","last_active":"2024-09-21T10:28:31.505009+0000","last_peered":"2024-09-21T10:28:31.505009+0000","last_clean":"2024-09-21T10:28:31.505009+0000","last_became_active":"2024-09-21T10:22:19.648950+0000","last_became_peered":"2024-09-21T10:22:19.648950+0000","last_unstale":"2024-09-21T10:28:31.505009+0000","last_undegraded":"2024-09-21T10:28:31.505009+0000","last_fullsized":"2024-09-21T10:28:31.505009+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:06:32.778203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744048+0000","last_change":"2024-09-21T10:27:24.961341+0000","last_active":"2024-09-21T10:28:34.744048+0000","last_peered":"2024-09-21T10:28:34.744048+0000","last_clean":"2024-09-21T10:28:34.744048+0000","last_became_active":"2024-09-21T10:27:24.961017+0000","last_became_peered":"2024-09-21T10:27:24.961017+0000","last_unstale":"2024-09-21T10:28:34.744048+0000","last_undegraded":"2024-09-21T10:28:34.744048+0000","last_fullsized":"2024-09-21T10:28:34.744048+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:21:55.827890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":740,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842413+0000","last_change":"2024-09-21T10:27:24.956462+0000","last_active":"2024-09-21T10:28:35.842413+0000","last_peered":"2024-09-21T10:28:35.842413+0000","last_clean":"2024-09-21T10:28:35.842413+0000","last_became_active":"2024-09-21T10:27:24.955291+0000","last_became_peered":"2024-09-21T10:27:24.955291+0000","last_unstale":"2024-09-21T10:28:35.842413+0000","last_undegraded":"2024-09-21T10:28:35.842413+0000","last_fullsized":"2024-09-21T10:28:35.842413+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:25:24.916725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":460,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.745918+0000","last_change":"2024-09-21T10:27:24.955413+0000","last_active":"2024-09-21T10:28:34.745918+0000","last_peered":"2024-09-21T10:28:34.745918+0000","last_clean":"2024-09-21T10:28:34.745918+0000","last_became_active":"2024-09-21T10:27:24.954856+0000","last_became_peered":"2024-09-21T10:27:24.954856+0000","last_unstale":"2024-09-21T10:28:34.745918+0000","last_undegraded":"2024-09-21T10:28:34.745918+0000","last_fullsized":"2024-09-21T10:28:34.745918+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:29.711141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744584+0000","last_change":"2024-09-21T10:27:24.960707+0000","last_active":"2024-09-21T10:28:34.744584+0000","last_peered":"2024-09-21T10:28:34.744584+0000","last_clean":"2024-09-21T10:28:34.744584+0000","last_became_active":"2024-09-21T10:27:24.958341+0000","last_became_peered":"2024-09-21T10:27:24.958341+0000","last_unstale":"2024-09-21T10:28:34.744584+0000","last_undegraded":"2024-09-21T10:28:34.744584+0000","last_fullsized":"2024-09-21T10:28:34.744584+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:01.897844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":125,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933471+0000","last_change":"2024-09-21T10:28:09.277786+0000","last_active":"2024-09-21T10:28:30.933471+0000","last_peered":"2024-09-21T10:28:30.933471+0000","last_clean":"2024-09-21T10:28:30.933471+0000","last_became_active":"2024-09-21T10:28:09.277389+0000","last_became_peered":"2024-09-21T10:28:09.277389+0000","last_unstale":"2024-09-21T10:28:30.933471+0000","last_undegraded":"2024-09-21T10:28:30.933471+0000","last_fullsized":"2024-09-21T10:28:30.933471+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:46.492879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":845,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.746510+0000","last_change":"2024-09-21T10:27:24.959812+0000","last_active":"2024-09-21T10:28:34.746510+0000","last_peered":"2024-09-21T10:28:34.746510+0000","last_clean":"2024-09-21T10:28:34.746510+0000","last_became_active":"2024-09-21T10:27:24.959509+0000","last_became_peered":"2024-09-21T10:27:24.959509+0000","last_unstale":"2024-09-21T10:28:34.746510+0000","last_undegraded":"2024-09-21T10:28:34.746510+0000","last_fullsized":"2024-09-21T10:28:34.746510+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:12.076554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":455,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842023+0000","last_change":"2024-09-21T10:14:30.702414+0000","last_active":"2024-09-21T10:28:35.842023+0000","last_peered":"2024-09-21T10:28:35.842023+0000","last_clean":"2024-09-21T10:28:35.842023+0000","last_became_active":"2024-09-21T10:14:30.702009+0000","last_became_peered":"2024-09-21T10:14:30.702009+0000","last_unstale":"2024-09-21T10:28:35.842023+0000","last_undegraded":"2024-09-21T10:28:35.842023+0000","last_fullsized":"2024-09-21T10:28:35.842023+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:50:48.396082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":451,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933491+0000","last_change":"2024-09-21T10:28:09.281257+0000","last_active":"2024-09-21T10:28:30.933491+0000","last_peered":"2024-09-21T10:28:30.933491+0000","last_clean":"2024-09-21T10:28:30.933491+0000","last_became_active":"2024-09-21T10:28:09.280924+0000","last_became_peered":"2024-09-21T10:28:09.280924+0000","last_unstale":"2024-09-21T10:28:30.933491+0000","last_undegraded":"2024-09-21T10:28:30.933491+0000","last_fullsized":"2024-09-21T10:28:30.933491+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:20.820089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":203,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504681+0000","last_change":"2024-09-21T10:28:16.368226+0000","last_active":"2024-09-21T10:28:31.504681+0000","last_peered":"2024-09-21T10:28:31.504681+0000","last_clean":"2024-09-21T10:28:31.504681+0000","last_became_active":"2024-09-21T10:28:16.367849+0000","last_became_peered":"2024-09-21T10:28:16.367849+0000","last_unstale":"2024-09-21T10:28:31.504681+0000","last_undegraded":"2024-09-21T10:28:31.504681+0000","last_fullsized":"2024-09-21T10:28:31.504681+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:22:36.268062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":542,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933378+0000","last_change":"2024-09-21T10:28:09.539278+0000","last_active":"2024-09-21T10:28:30.933378+0000","last_peered":"2024-09-21T10:28:30.933378+0000","last_clean":"2024-09-21T10:28:30.933378+0000","last_became_active":"2024-09-21T10:28:09.538995+0000","last_became_peered":"2024-09-21T10:28:09.538995+0000","last_unstale":"2024-09-21T10:28:30.933378+0000","last_undegraded":"2024-09-21T10:28:30.933378+0000","last_fullsized":"2024-09-21T10:28:30.933378+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:12.839257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":186,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504601+0000","last_change":"2024-09-21T10:28:16.369626+0000","last_active":"2024-09-21T10:28:31.504601+0000","last_peered":"2024-09-21T10:28:31.504601+0000","last_clean":"2024-09-21T10:28:31.504601+0000","last_became_active":"2024-09-21T10:28:16.369258+0000","last_became_peered":"2024-09-21T10:28:16.369258+0000","last_unstale":"2024-09-21T10:28:31.504601+0000","last_undegraded":"2024-09-21T10:28:31.504601+0000","last_fullsized":"2024-09-21T10:28:31.504601+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:25:44.405131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744420+0000","last_change":"2024-09-21T10:27:24.963337+0000","last_active":"2024-09-21T10:28:34.744420+0000","last_peered":"2024-09-21T10:28:34.744420+0000","last_clean":"2024-09-21T10:28:34.744420+0000","last_became_active":"2024-09-21T10:27:24.963022+0000","last_became_peered":"2024-09-21T10:27:24.963022+0000","last_unstale":"2024-09-21T10:28:34.744420+0000","last_undegraded":"2024-09-21T10:28:34.744420+0000","last_fullsized":"2024-09-21T10:28:34.744420+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:53:16.737509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744674+0000","last_change":"2024-09-21T10:27:24.961470+0000","last_active":"2024-09-21T10:28:34.744674+0000","last_peered":"2024-09-21T10:28:34.744674+0000","last_clean":"2024-09-21T10:28:34.744674+0000","last_became_active":"2024-09-21T10:27:24.960726+0000","last_became_peered":"2024-09-21T10:27:24.960726+0000","last_unstale":"2024-09-21T10:28:34.744674+0000","last_undegraded":"2024-09-21T10:28:34.744674+0000","last_fullsized":"2024-09-21T10:28:34.744674+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:45:57.713375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":135,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933182+0000","last_change":"2024-09-21T10:28:09.282833+0000","last_active":"2024-09-21T10:28:30.933182+0000","last_peered":"2024-09-21T10:28:30.933182+0000","last_clean":"2024-09-21T10:28:30.933182+0000","last_became_active":"2024-09-21T10:28:09.282255+0000","last_became_peered":"2024-09-21T10:28:09.282255+0000","last_unstale":"2024-09-21T10:28:30.933182+0000","last_undegraded":"2024-09-21T10:28:30.933182+0000","last_fullsized":"2024-09-21T10:28:30.933182+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:32:38.604486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":469,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842091+0000","last_change":"2024-09-21T10:10:57.401588+0000","last_active":"2024-09-21T10:28:35.842091+0000","last_peered":"2024-09-21T10:28:35.842091+0000","last_clean":"2024-09-21T10:28:35.842091+0000","last_became_active":"2024-09-21T10:10:57.397724+0000","last_became_peered":"2024-09-21T10:10:57.397724+0000","last_unstale":"2024-09-21T10:28:35.842091+0000","last_undegraded":"2024-09-21T10:28:35.842091+0000","last_fullsized":"2024-09-21T10:28:35.842091+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:47.843630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":16920,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26580,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388627044,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1498176,"kb_used_data":6708,"kb_used_omap":1,"kb_used_meta":1491454,"kb_avail":92246976,"statfs":{"total":95995035648,"available":94460903424,"internally_reserved":0,"allocated":6868992,"data_stored":4089257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1527249240},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019502,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1504964,"kb_used_data":6392,"kb_used_omap":1,"kb_used_meta":1498558,"kb_avail":92240188,"statfs":{"total":95995035648,"available":94453952512,"internally_reserved":0,"allocated":6545408,"data_stored":3761618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1534523736},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":245,"seq":1052266987761,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":648052,"kb_used_data":6696,"kb_used_omap":8,"kb_used_meta":641335,"kb_avail":93097100,"statfs":{"total":95995035648,"available":95331430400,"internally_reserved":0,"allocated":6856704,"data_stored":4085044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8940,"internal_metadata":656727316},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":175,"seq":751619277110,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":947844,"kb_used_data":6392,"kb_used_omap":12,"kb_used_meta":941427,"kb_avail":92797308,"statfs":{"total":95995035648,"available":95024443392,"internally_reserved":0,"allocated":6545408,"data_stored":3761614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12845,"internal_metadata":964021715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823273,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1788660,"kb_used_data":6632,"kb_used_omap":1,"kb_used_meta":1782014,"kb_avail":91956492,"statfs":{"total":95995035648,"available":94163447808,"internally_reserved":0,"allocated":6791168,"data_stored":4019461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1824782680},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879697,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1220468,"kb_used_data":6952,"kb_used_omap":4,"kb_used_meta":1213499,"kb_avail":92524684,"statfs":{"total":95995035648,"available":94745276416,"internally_reserved":0,"allocated":7118848,"data_stored":4347222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":1242623420},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566537,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1267196,"kb_used_data":6640,"kb_used_omap":6,"kb_used_meta":1260537,"kb_avail":92477956,"statfs":{"total":95995035648,"available":94697426944,"internally_reserved":0,"allocated":6799360,"data_stored":4020154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":1290790311},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860882,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1383532,"kb_used_data":6368,"kb_used_omap":5,"kb_used_meta":1377146,"kb_avail":92361620,"statfs":{"total":95995035648,"available":94578298880,"internally_reserved":0,"allocated":6520832,"data_stored":3756460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":1410197743},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7236,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9976,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:29:24.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 86c49514-77fc-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:29:24.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:23 smithi129 ceph-mon[88471]: from='client.47696 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:29:24.689 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:25.031 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:24 smithi107 ceph-mon[102221]: pgmap v1687: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:25.031 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:24 smithi107 ceph-mon[105022]: pgmap v1687: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:25.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:24 smithi129 ceph-mon[88471]: pgmap v1687: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:27.213 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T10:29:27.214 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T10:29:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:27 smithi107 ceph-mon[105022]: pgmap v1688: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:29:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:29:27.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:27 smithi107 ceph-mon[105022]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:27.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:27 smithi107 ceph-mon[102221]: pgmap v1688: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:27.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:27.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:27.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:27.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:29:27.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:29:27.360 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:27 smithi107 ceph-mon[102221]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:27 smithi129 ceph-mon[88471]: pgmap v1688: 93 pgs: 93 active+clean; 582 KiB data, 9.8 GiB used, 705 GiB / 715 GiB avail 2024-09-21T10:29:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:29:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:29:27.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:27 smithi129 ceph-mon[88471]: from='mgr.25219 172.21.15.107:0/1559417756' entity='mgr.y' 2024-09-21T10:29:28.020 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":1688,"stamp":"2024-09-21T10:29:25.657508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":279608,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2159,"ondisk_log_size":2159,"up":279,"acting":279,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":279,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10258892,"kb_used_data":52780,"kb_used_omap":43,"kb_used_meta":10205972,"kb_avail":739702324,"statfs":{"total":767960285184,"available":757455179776,"internally_reserved":0,"allocated":54046720,"data_stored":31840830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44223,"internal_metadata":10450916161},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002832"},"pg_stats":[{"pgid":"1.11","version":"77'100","reported_seq":480,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.841841+0000","last_change":"2024-09-21T10:27:24.964892+0000","last_active":"2024-09-21T10:28:35.841841+0000","last_peered":"2024-09-21T10:28:35.841841+0000","last_clean":"2024-09-21T10:28:35.841841+0000","last_became_active":"2024-09-21T10:27:24.964235+0000","last_became_peered":"2024-09-21T10:27:24.964235+0000","last_unstale":"2024-09-21T10:28:35.841841+0000","last_undegraded":"2024-09-21T10:28:35.841841+0000","last_fullsized":"2024-09-21T10:28:35.841841+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:53.631923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:53.631923+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:00:17.393822+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744799+0000","last_change":"2024-09-21T10:27:24.961323+0000","last_active":"2024-09-21T10:28:34.744799+0000","last_peered":"2024-09-21T10:28:34.744799+0000","last_clean":"2024-09-21T10:28:34.744799+0000","last_became_active":"2024-09-21T10:27:24.960792+0000","last_became_peered":"2024-09-21T10:27:24.960792+0000","last_unstale":"2024-09-21T10:28:34.744799+0000","last_undegraded":"2024-09-21T10:28:34.744799+0000","last_fullsized":"2024-09-21T10:28:34.744799+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:17:56.559965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"63'2","reported_seq":356,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.724067+0000","last_change":"2024-09-21T10:14:30.717695+0000","last_active":"2024-09-21T10:28:30.724067+0000","last_peered":"2024-09-21T10:28:30.724067+0000","last_clean":"2024-09-21T10:28:30.724067+0000","last_became_active":"2024-09-21T10:14:30.705952+0000","last_became_peered":"2024-09-21T10:14:30.705952+0000","last_unstale":"2024-09-21T10:28:30.724067+0000","last_undegraded":"2024-09-21T10:28:30.724067+0000","last_fullsized":"2024-09-21T10:28:30.724067+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:08:18.863430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"77'100","reported_seq":166,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.503997+0000","last_change":"2024-09-21T10:28:16.369894+0000","last_active":"2024-09-21T10:28:31.503997+0000","last_peered":"2024-09-21T10:28:31.503997+0000","last_clean":"2024-09-21T10:28:31.503997+0000","last_became_active":"2024-09-21T10:28:16.369532+0000","last_became_peered":"2024-09-21T10:28:16.369532+0000","last_unstale":"2024-09-21T10:28:31.503997+0000","last_undegraded":"2024-09-21T10:28:31.503997+0000","last_fullsized":"2024-09-21T10:28:31.503997+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.798577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.798577+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:31:55.319583+0000","scrub_duration":0.0023622640000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":471,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504520+0000","last_change":"2024-09-21T10:28:16.368292+0000","last_active":"2024-09-21T10:28:31.504520+0000","last_peered":"2024-09-21T10:28:31.504520+0000","last_clean":"2024-09-21T10:28:31.504520+0000","last_became_active":"2024-09-21T10:28:16.367188+0000","last_became_peered":"2024-09-21T10:28:16.367188+0000","last_unstale":"2024-09-21T10:28:31.504520+0000","last_undegraded":"2024-09-21T10:28:31.504520+0000","last_fullsized":"2024-09-21T10:28:31.504520+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:46.535906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":737,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.841877+0000","last_change":"2024-09-21T10:27:24.953070+0000","last_active":"2024-09-21T10:28:35.841877+0000","last_peered":"2024-09-21T10:28:35.841877+0000","last_clean":"2024-09-21T10:28:35.841877+0000","last_became_active":"2024-09-21T10:27:24.952728+0000","last_became_peered":"2024-09-21T10:27:24.952728+0000","last_unstale":"2024-09-21T10:28:35.841877+0000","last_undegraded":"2024-09-21T10:28:35.841877+0000","last_fullsized":"2024-09-21T10:28:35.841877+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:53.123836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"77'100","reported_seq":481,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842637+0000","last_change":"2024-09-21T10:27:24.955481+0000","last_active":"2024-09-21T10:28:35.842637+0000","last_peered":"2024-09-21T10:28:35.842637+0000","last_clean":"2024-09-21T10:28:35.842637+0000","last_became_active":"2024-09-21T10:27:24.955114+0000","last_became_peered":"2024-09-21T10:27:24.955114+0000","last_unstale":"2024-09-21T10:28:35.842637+0000","last_undegraded":"2024-09-21T10:28:35.842637+0000","last_fullsized":"2024-09-21T10:28:35.842637+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:50.433227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:50.433227+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:54.414681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":463,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842278+0000","last_change":"2024-09-21T10:27:24.956694+0000","last_active":"2024-09-21T10:28:35.842278+0000","last_peered":"2024-09-21T10:28:35.842278+0000","last_clean":"2024-09-21T10:28:35.842278+0000","last_became_active":"2024-09-21T10:27:24.956151+0000","last_became_peered":"2024-09-21T10:27:24.956151+0000","last_unstale":"2024-09-21T10:28:35.842278+0000","last_undegraded":"2024-09-21T10:28:35.842278+0000","last_fullsized":"2024-09-21T10:28:35.842278+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:43.175235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":372,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.723785+0000","last_change":"2024-09-21T10:22:19.646202+0000","last_active":"2024-09-21T10:28:30.723785+0000","last_peered":"2024-09-21T10:28:30.723785+0000","last_clean":"2024-09-21T10:28:30.723785+0000","last_became_active":"2024-09-21T10:22:19.645690+0000","last_became_peered":"2024-09-21T10:22:19.645690+0000","last_unstale":"2024-09-21T10:28:30.723785+0000","last_undegraded":"2024-09-21T10:28:30.723785+0000","last_fullsized":"2024-09-21T10:28:30.723785+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:44:01.509115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"77'100","reported_seq":76,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.503997+0000","last_change":"2024-09-21T10:28:16.370122+0000","last_active":"2024-09-21T10:28:31.503997+0000","last_peered":"2024-09-21T10:28:31.503997+0000","last_clean":"2024-09-21T10:28:31.503997+0000","last_became_active":"2024-09-21T10:28:16.369549+0000","last_became_peered":"2024-09-21T10:28:16.369549+0000","last_unstale":"2024-09-21T10:28:31.503997+0000","last_undegraded":"2024-09-21T10:28:31.503997+0000","last_fullsized":"2024-09-21T10:28:31.503997+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:15:02.841968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:15:02.841968+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:10.403124+0000","scrub_duration":0.002846973,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744797+0000","last_change":"2024-09-21T10:27:24.963982+0000","last_active":"2024-09-21T10:28:34.744797+0000","last_peered":"2024-09-21T10:28:34.744797+0000","last_clean":"2024-09-21T10:28:34.744797+0000","last_became_active":"2024-09-21T10:27:24.963354+0000","last_became_peered":"2024-09-21T10:27:24.963354+0000","last_unstale":"2024-09-21T10:28:34.744797+0000","last_undegraded":"2024-09-21T10:28:34.744797+0000","last_fullsized":"2024-09-21T10:28:34.744797+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:00:34.465081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"63'8","reported_seq":88,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504414+0000","last_change":"2024-09-21T10:22:19.662950+0000","last_active":"2024-09-21T10:28:31.504414+0000","last_peered":"2024-09-21T10:28:31.504414+0000","last_clean":"2024-09-21T10:28:31.504414+0000","last_became_active":"2024-09-21T10:22:19.649590+0000","last_became_peered":"2024-09-21T10:22:19.649590+0000","last_unstale":"2024-09-21T10:28:31.504414+0000","last_undegraded":"2024-09-21T10:28:31.504414+0000","last_fullsized":"2024-09-21T10:28:31.504414+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:38:15.512148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"77'100","reported_seq":364,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.724464+0000","last_change":"2024-09-21T10:14:30.708275+0000","last_active":"2024-09-21T10:28:30.724464+0000","last_peered":"2024-09-21T10:28:30.724464+0000","last_clean":"2024-09-21T10:28:30.724464+0000","last_became_active":"2024-09-21T10:14:30.707763+0000","last_became_peered":"2024-09-21T10:14:30.707763+0000","last_unstale":"2024-09-21T10:28:30.724464+0000","last_undegraded":"2024-09-21T10:28:30.724464+0000","last_fullsized":"2024-09-21T10:28:30.724464+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:55.259385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:55.259385+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:30:41.322447+0000","scrub_duration":0.0015667260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"63'1","reported_seq":399,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.724360+0000","last_change":"2024-09-21T10:14:30.720250+0000","last_active":"2024-09-21T10:28:30.724360+0000","last_peered":"2024-09-21T10:28:30.724360+0000","last_clean":"2024-09-21T10:28:30.724360+0000","last_became_active":"2024-09-21T10:14:30.707036+0000","last_became_peered":"2024-09-21T10:14:30.707036+0000","last_unstale":"2024-09-21T10:28:30.724360+0000","last_undegraded":"2024-09-21T10:28:30.724360+0000","last_fullsized":"2024-09-21T10:28:30.724360+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:17.438534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":404,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504145+0000","last_change":"2024-09-21T10:28:16.369325+0000","last_active":"2024-09-21T10:28:31.504145+0000","last_peered":"2024-09-21T10:28:31.504145+0000","last_clean":"2024-09-21T10:28:31.504145+0000","last_became_active":"2024-09-21T10:28:16.369029+0000","last_became_peered":"2024-09-21T10:28:16.369029+0000","last_unstale":"2024-09-21T10:28:31.504145+0000","last_undegraded":"2024-09-21T10:28:31.504145+0000","last_fullsized":"2024-09-21T10:28:31.504145+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:09:57.105557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"77'100","reported_seq":64,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744908+0000","last_change":"2024-09-21T10:28:26.837137+0000","last_active":"2024-09-21T10:28:34.744908+0000","last_peered":"2024-09-21T10:28:34.744908+0000","last_clean":"2024-09-21T10:28:34.744908+0000","last_became_active":"2024-09-21T10:27:24.954327+0000","last_became_peered":"2024-09-21T10:27:24.954327+0000","last_unstale":"2024-09-21T10:28:34.744908+0000","last_undegraded":"2024-09-21T10:28:34.744908+0000","last_fullsized":"2024-09-21T10:28:34.744908+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:28:26.837085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:28:26.837085+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:49:08.156692+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"63'2","reported_seq":378,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.724429+0000","last_change":"2024-09-21T10:18:12.643154+0000","last_active":"2024-09-21T10:28:30.724429+0000","last_peered":"2024-09-21T10:28:30.724429+0000","last_clean":"2024-09-21T10:28:30.724429+0000","last_became_active":"2024-09-21T10:18:12.636287+0000","last_became_peered":"2024-09-21T10:18:12.636287+0000","last_unstale":"2024-09-21T10:28:30.724429+0000","last_undegraded":"2024-09-21T10:28:30.724429+0000","last_fullsized":"2024-09-21T10:28:30.724429+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:45:31.008309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":261,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933426+0000","last_change":"2024-09-21T10:28:09.280335+0000","last_active":"2024-09-21T10:28:30.933426+0000","last_peered":"2024-09-21T10:28:30.933426+0000","last_clean":"2024-09-21T10:28:30.933426+0000","last_became_active":"2024-09-21T10:28:09.280073+0000","last_became_peered":"2024-09-21T10:28:09.280073+0000","last_unstale":"2024-09-21T10:28:30.933426+0000","last_undegraded":"2024-09-21T10:28:30.933426+0000","last_fullsized":"2024-09-21T10:28:30.933426+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:52.156080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"77'100","reported_seq":78,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504993+0000","last_change":"2024-09-21T10:22:19.657392+0000","last_active":"2024-09-21T10:28:31.504993+0000","last_peered":"2024-09-21T10:28:31.504993+0000","last_clean":"2024-09-21T10:28:31.504993+0000","last_became_active":"2024-09-21T10:22:19.650821+0000","last_became_peered":"2024-09-21T10:22:19.650821+0000","last_unstale":"2024-09-21T10:28:31.504993+0000","last_undegraded":"2024-09-21T10:28:31.504993+0000","last_fullsized":"2024-09-21T10:28:31.504993+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.289859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.289859+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:54:14.510032+0000","scrub_duration":0.001251809,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"63'1","reported_seq":378,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.724265+0000","last_change":"2024-09-21T10:22:19.664692+0000","last_active":"2024-09-21T10:28:30.724265+0000","last_peered":"2024-09-21T10:28:30.724265+0000","last_clean":"2024-09-21T10:28:30.724265+0000","last_became_active":"2024-09-21T10:22:19.647696+0000","last_became_peered":"2024-09-21T10:22:19.647696+0000","last_unstale":"2024-09-21T10:28:30.724265+0000","last_undegraded":"2024-09-21T10:28:30.724265+0000","last_fullsized":"2024-09-21T10:28:30.724265+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:57:54.264752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":78,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504827+0000","last_change":"2024-09-21T10:22:19.652364+0000","last_active":"2024-09-21T10:28:31.504827+0000","last_peered":"2024-09-21T10:28:31.504827+0000","last_clean":"2024-09-21T10:28:31.504827+0000","last_became_active":"2024-09-21T10:22:19.651586+0000","last_became_peered":"2024-09-21T10:22:19.651586+0000","last_unstale":"2024-09-21T10:28:31.504827+0000","last_undegraded":"2024-09-21T10:28:31.504827+0000","last_fullsized":"2024-09-21T10:28:31.504827+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:50:03.593157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"77'100","reported_seq":71,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933904+0000","last_change":"2024-09-21T10:28:09.283812+0000","last_active":"2024-09-21T10:28:30.933904+0000","last_peered":"2024-09-21T10:28:30.933904+0000","last_clean":"2024-09-21T10:28:30.933904+0000","last_became_active":"2024-09-21T10:28:09.283516+0000","last_became_peered":"2024-09-21T10:28:09.283516+0000","last_unstale":"2024-09-21T10:28:30.933904+0000","last_undegraded":"2024-09-21T10:28:30.933904+0000","last_fullsized":"2024-09-21T10:28:30.933904+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:18:27.085525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:18:27.085525+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:05.245886+0000","scrub_duration":0.0036485279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":495,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504593+0000","last_change":"2024-09-21T10:28:16.370043+0000","last_active":"2024-09-21T10:28:31.504593+0000","last_peered":"2024-09-21T10:28:31.504593+0000","last_clean":"2024-09-21T10:28:31.504593+0000","last_became_active":"2024-09-21T10:28:16.369162+0000","last_became_peered":"2024-09-21T10:28:16.369162+0000","last_unstale":"2024-09-21T10:28:31.504593+0000","last_undegraded":"2024-09-21T10:28:31.504593+0000","last_fullsized":"2024-09-21T10:28:31.504593+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:18.580934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":710,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842048+0000","last_change":"2024-09-21T10:14:30.711105+0000","last_active":"2024-09-21T10:28:35.842048+0000","last_peered":"2024-09-21T10:28:35.842048+0000","last_clean":"2024-09-21T10:28:35.842048+0000","last_became_active":"2024-09-21T10:14:30.710759+0000","last_became_peered":"2024-09-21T10:14:30.710759+0000","last_unstale":"2024-09-21T10:28:35.842048+0000","last_undegraded":"2024-09-21T10:28:35.842048+0000","last_fullsized":"2024-09-21T10:28:35.842048+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:40.690042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"77'100","reported_seq":364,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.723970+0000","last_change":"2024-09-21T10:27:24.952212+0000","last_active":"2024-09-21T10:28:30.723970+0000","last_peered":"2024-09-21T10:28:30.723970+0000","last_clean":"2024-09-21T10:28:30.723970+0000","last_became_active":"2024-09-21T10:27:24.951883+0000","last_became_peered":"2024-09-21T10:27:24.951883+0000","last_unstale":"2024-09-21T10:28:30.723970+0000","last_undegraded":"2024-09-21T10:28:30.723970+0000","last_fullsized":"2024-09-21T10:28:30.723970+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.355912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.355912+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:19:22.848110+0000","scrub_duration":0.0017980030000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"63'2","reported_seq":56,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744211+0000","last_change":"2024-09-21T10:27:24.968357+0000","last_active":"2024-09-21T10:28:34.744211+0000","last_peered":"2024-09-21T10:28:34.744211+0000","last_clean":"2024-09-21T10:28:34.744211+0000","last_became_active":"2024-09-21T10:27:24.954336+0000","last_became_peered":"2024-09-21T10:27:24.954336+0000","last_unstale":"2024-09-21T10:28:34.744211+0000","last_undegraded":"2024-09-21T10:28:34.744211+0000","last_fullsized":"2024-09-21T10:28:34.744211+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:05.711937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":277,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933671+0000","last_change":"2024-09-21T10:28:09.283069+0000","last_active":"2024-09-21T10:28:30.933671+0000","last_peered":"2024-09-21T10:28:30.933671+0000","last_clean":"2024-09-21T10:28:30.933671+0000","last_became_active":"2024-09-21T10:28:09.282668+0000","last_became_peered":"2024-09-21T10:28:09.282668+0000","last_unstale":"2024-09-21T10:28:30.933671+0000","last_undegraded":"2024-09-21T10:28:30.933671+0000","last_fullsized":"2024-09-21T10:28:30.933671+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:48:42.023717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"77'100","reported_seq":149,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.503872+0000","last_change":"2024-09-21T10:28:16.367871+0000","last_active":"2024-09-21T10:28:31.503872+0000","last_peered":"2024-09-21T10:28:31.503872+0000","last_clean":"2024-09-21T10:28:31.503872+0000","last_became_active":"2024-09-21T10:28:16.367530+0000","last_became_peered":"2024-09-21T10:28:16.367530+0000","last_unstale":"2024-09-21T10:28:31.503872+0000","last_undegraded":"2024-09-21T10:28:31.503872+0000","last_fullsized":"2024-09-21T10:28:31.503872+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:02.152981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:02.152981+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:44:55.561812+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":78,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504591+0000","last_change":"2024-09-21T10:22:19.652182+0000","last_active":"2024-09-21T10:28:31.504591+0000","last_peered":"2024-09-21T10:28:31.504591+0000","last_clean":"2024-09-21T10:28:31.504591+0000","last_became_active":"2024-09-21T10:22:19.651703+0000","last_became_peered":"2024-09-21T10:22:19.651703+0000","last_unstale":"2024-09-21T10:28:31.504591+0000","last_undegraded":"2024-09-21T10:28:31.504591+0000","last_fullsized":"2024-09-21T10:28:31.504591+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:15:08.752662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":420,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933741+0000","last_change":"2024-09-21T10:28:09.282543+0000","last_active":"2024-09-21T10:28:30.933741+0000","last_peered":"2024-09-21T10:28:30.933741+0000","last_clean":"2024-09-21T10:28:30.933741+0000","last_became_active":"2024-09-21T10:28:09.281807+0000","last_became_peered":"2024-09-21T10:28:09.281807+0000","last_unstale":"2024-09-21T10:28:30.933741+0000","last_undegraded":"2024-09-21T10:28:30.933741+0000","last_fullsized":"2024-09-21T10:28:30.933741+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:55:53.868908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"77'100","reported_seq":457,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842534+0000","last_change":"2024-09-21T10:14:05.474100+0000","last_active":"2024-09-21T10:28:35.842534+0000","last_peered":"2024-09-21T10:28:35.842534+0000","last_clean":"2024-09-21T10:28:35.842534+0000","last_became_active":"2024-09-21T10:14:05.473336+0000","last_became_peered":"2024-09-21T10:14:05.473336+0000","last_unstale":"2024-09-21T10:28:35.842534+0000","last_undegraded":"2024-09-21T10:28:35.842534+0000","last_fullsized":"2024-09-21T10:28:35.842534+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:53.540323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:53.540323+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:06:10.963352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":512,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933243+0000","last_change":"2024-09-21T10:28:09.282222+0000","last_active":"2024-09-21T10:28:30.933243+0000","last_peered":"2024-09-21T10:28:30.933243+0000","last_clean":"2024-09-21T10:28:30.933243+0000","last_became_active":"2024-09-21T10:28:09.281541+0000","last_became_peered":"2024-09-21T10:28:09.281541+0000","last_unstale":"2024-09-21T10:28:30.933243+0000","last_undegraded":"2024-09-21T10:28:30.933243+0000","last_fullsized":"2024-09-21T10:28:30.933243+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:08:25.537544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":290,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.934021+0000","last_change":"2024-09-21T10:28:09.282671+0000","last_active":"2024-09-21T10:28:30.934021+0000","last_peered":"2024-09-21T10:28:30.934021+0000","last_clean":"2024-09-21T10:28:30.934021+0000","last_became_active":"2024-09-21T10:28:09.282124+0000","last_became_peered":"2024-09-21T10:28:09.282124+0000","last_unstale":"2024-09-21T10:28:30.934021+0000","last_undegraded":"2024-09-21T10:28:30.934021+0000","last_fullsized":"2024-09-21T10:28:30.934021+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:35:54.562251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"77'100","reported_seq":67,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504241+0000","last_change":"2024-09-21T10:28:16.368890+0000","last_active":"2024-09-21T10:28:31.504241+0000","last_peered":"2024-09-21T10:28:31.504241+0000","last_clean":"2024-09-21T10:28:31.504241+0000","last_became_active":"2024-09-21T10:28:16.368573+0000","last_became_peered":"2024-09-21T10:28:16.368573+0000","last_unstale":"2024-09-21T10:28:31.504241+0000","last_undegraded":"2024-09-21T10:28:31.504241+0000","last_fullsized":"2024-09-21T10:28:31.504241+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:20:00.237668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:20:00.237668+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:32:59.606457+0000","scrub_duration":0.001567564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744208+0000","last_change":"2024-09-21T10:27:24.950027+0000","last_active":"2024-09-21T10:28:34.744208+0000","last_peered":"2024-09-21T10:28:34.744208+0000","last_clean":"2024-09-21T10:28:34.744208+0000","last_became_active":"2024-09-21T10:27:24.949461+0000","last_became_peered":"2024-09-21T10:27:24.949461+0000","last_unstale":"2024-09-21T10:28:34.744208+0000","last_undegraded":"2024-09-21T10:28:34.744208+0000","last_fullsized":"2024-09-21T10:28:34.744208+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:57:47.019081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":677,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842495+0000","last_change":"2024-09-21T10:27:24.951877+0000","last_active":"2024-09-21T10:28:35.842495+0000","last_peered":"2024-09-21T10:28:35.842495+0000","last_clean":"2024-09-21T10:28:35.842495+0000","last_became_active":"2024-09-21T10:27:24.951534+0000","last_became_peered":"2024-09-21T10:27:24.951534+0000","last_unstale":"2024-09-21T10:28:35.842495+0000","last_undegraded":"2024-09-21T10:28:35.842495+0000","last_fullsized":"2024-09-21T10:28:35.842495+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:27:47.963923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"77'100","reported_seq":381,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.723865+0000","last_change":"2024-09-21T10:22:19.655993+0000","last_active":"2024-09-21T10:28:30.723865+0000","last_peered":"2024-09-21T10:28:30.723865+0000","last_clean":"2024-09-21T10:28:30.723865+0000","last_became_active":"2024-09-21T10:22:19.655178+0000","last_became_peered":"2024-09-21T10:22:19.655178+0000","last_unstale":"2024-09-21T10:28:30.723865+0000","last_undegraded":"2024-09-21T10:28:30.723865+0000","last_fullsized":"2024-09-21T10:28:30.723865+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:47.393146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:47.393146+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:07:48.668712+0000","scrub_duration":0.001601086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"55'1","reported_seq":148,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504334+0000","last_change":"2024-09-21T10:28:16.367821+0000","last_active":"2024-09-21T10:28:31.504334+0000","last_peered":"2024-09-21T10:28:31.504334+0000","last_clean":"2024-09-21T10:28:31.504334+0000","last_became_active":"2024-09-21T10:28:16.367445+0000","last_became_peered":"2024-09-21T10:28:16.367445+0000","last_unstale":"2024-09-21T10:28:31.504334+0000","last_undegraded":"2024-09-21T10:28:31.504334+0000","last_fullsized":"2024-09-21T10:28:31.504334+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:49.941634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744295+0000","last_change":"2024-09-21T10:27:24.959198+0000","last_active":"2024-09-21T10:28:34.744295+0000","last_peered":"2024-09-21T10:28:34.744295+0000","last_clean":"2024-09-21T10:28:34.744295+0000","last_became_active":"2024-09-21T10:27:24.953113+0000","last_became_peered":"2024-09-21T10:27:24.953113+0000","last_unstale":"2024-09-21T10:28:34.744295+0000","last_undegraded":"2024-09-21T10:28:34.744295+0000","last_fullsized":"2024-09-21T10:28:34.744295+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:29:10.801832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":78,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504673+0000","last_change":"2024-09-21T10:22:19.651173+0000","last_active":"2024-09-21T10:28:31.504673+0000","last_peered":"2024-09-21T10:28:31.504673+0000","last_clean":"2024-09-21T10:28:31.504673+0000","last_became_active":"2024-09-21T10:22:19.649673+0000","last_became_peered":"2024-09-21T10:22:19.649673+0000","last_unstale":"2024-09-21T10:28:31.504673+0000","last_undegraded":"2024-09-21T10:28:31.504673+0000","last_fullsized":"2024-09-21T10:28:31.504673+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:28:37.605825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"77'100","reported_seq":482,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842467+0000","last_change":"2024-09-21T10:22:19.651858+0000","last_active":"2024-09-21T10:28:35.842467+0000","last_peered":"2024-09-21T10:28:35.842467+0000","last_clean":"2024-09-21T10:28:35.842467+0000","last_became_active":"2024-09-21T10:22:19.651510+0000","last_became_peered":"2024-09-21T10:22:19.651510+0000","last_unstale":"2024-09-21T10:28:35.842467+0000","last_undegraded":"2024-09-21T10:28:35.842467+0000","last_fullsized":"2024-09-21T10:28:35.842467+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:32.550551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:32.550551+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:01:42.501915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":335,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.723841+0000","last_change":"2024-09-21T10:09:23.838944+0000","last_active":"2024-09-21T10:28:30.723841+0000","last_peered":"2024-09-21T10:28:30.723841+0000","last_clean":"2024-09-21T10:28:30.723841+0000","last_became_active":"2024-09-21T10:09:23.837886+0000","last_became_peered":"2024-09-21T10:09:23.837886+0000","last_unstale":"2024-09-21T10:28:30.723841+0000","last_undegraded":"2024-09-21T10:28:30.723841+0000","last_fullsized":"2024-09-21T10:28:30.723841+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:52:16.928195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030773699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":702,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.841753+0000","last_change":"2024-09-21T10:27:24.953262+0000","last_active":"2024-09-21T10:28:35.841753+0000","last_peered":"2024-09-21T10:28:35.841753+0000","last_clean":"2024-09-21T10:28:35.841753+0000","last_became_active":"2024-09-21T10:27:24.952939+0000","last_became_peered":"2024-09-21T10:27:24.952939+0000","last_unstale":"2024-09-21T10:28:35.841753+0000","last_undegraded":"2024-09-21T10:28:35.841753+0000","last_fullsized":"2024-09-21T10:28:35.841753+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:49:46.284356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":450,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842359+0000","last_change":"2024-09-21T10:09:49.923044+0000","last_active":"2024-09-21T10:28:35.842359+0000","last_peered":"2024-09-21T10:28:35.842359+0000","last_clean":"2024-09-21T10:28:35.842359+0000","last_became_active":"2024-09-21T10:09:49.922711+0000","last_became_peered":"2024-09-21T10:09:49.922711+0000","last_unstale":"2024-09-21T10:28:35.842359+0000","last_undegraded":"2024-09-21T10:28:35.842359+0000","last_fullsized":"2024-09-21T10:28:35.842359+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:57:14.090084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"77'100","reported_seq":385,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.723981+0000","last_change":"2024-09-21T10:18:12.636851+0000","last_active":"2024-09-21T10:28:30.723981+0000","last_peered":"2024-09-21T10:28:30.723981+0000","last_clean":"2024-09-21T10:28:30.723981+0000","last_became_active":"2024-09-21T10:18:12.636513+0000","last_became_peered":"2024-09-21T10:18:12.636513+0000","last_unstale":"2024-09-21T10:28:30.723981+0000","last_undegraded":"2024-09-21T10:28:30.723981+0000","last_fullsized":"2024-09-21T10:28:30.723981+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:52.361137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:52.361137+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:21.025735+0000","scrub_duration":0.001255358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":857,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.746248+0000","last_change":"2024-09-21T10:27:24.955527+0000","last_active":"2024-09-21T10:28:34.746248+0000","last_peered":"2024-09-21T10:28:34.746248+0000","last_clean":"2024-09-21T10:28:34.746248+0000","last_became_active":"2024-09-21T10:27:24.955242+0000","last_became_peered":"2024-09-21T10:27:24.955242+0000","last_unstale":"2024-09-21T10:28:34.746248+0000","last_undegraded":"2024-09-21T10:28:34.746248+0000","last_fullsized":"2024-09-21T10:28:34.746248+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:10.360228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026741699999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":239,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.723748+0000","last_change":"2024-09-21T10:22:19.647946+0000","last_active":"2024-09-21T10:28:30.723748+0000","last_peered":"2024-09-21T10:28:30.723748+0000","last_clean":"2024-09-21T10:28:30.723748+0000","last_became_active":"2024-09-21T10:22:19.647057+0000","last_became_peered":"2024-09-21T10:22:19.647057+0000","last_unstale":"2024-09-21T10:28:30.723748+0000","last_undegraded":"2024-09-21T10:28:30.723748+0000","last_fullsized":"2024-09-21T10:28:30.723748+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:06.839922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":836,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.746647+0000","last_change":"2024-09-21T10:10:57.395073+0000","last_active":"2024-09-21T10:28:34.746647+0000","last_peered":"2024-09-21T10:28:34.746647+0000","last_clean":"2024-09-21T10:28:34.746647+0000","last_became_active":"2024-09-21T10:10:57.394674+0000","last_became_peered":"2024-09-21T10:10:57.394674+0000","last_unstale":"2024-09-21T10:28:34.746647+0000","last_undegraded":"2024-09-21T10:28:34.746647+0000","last_fullsized":"2024-09-21T10:28:34.746647+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:48:48.912649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"77'100","reported_seq":298,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.503914+0000","last_change":"2024-09-21T10:28:16.367768+0000","last_active":"2024-09-21T10:28:31.503914+0000","last_peered":"2024-09-21T10:28:31.503914+0000","last_clean":"2024-09-21T10:28:31.503914+0000","last_became_active":"2024-09-21T10:28:16.367348+0000","last_became_peered":"2024-09-21T10:28:16.367348+0000","last_unstale":"2024-09-21T10:28:31.503914+0000","last_undegraded":"2024-09-21T10:28:31.503914+0000","last_fullsized":"2024-09-21T10:28:31.503914+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:09:04.043917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:09:04.043917+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:53:29.741459+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"57'2","reported_seq":215,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504644+0000","last_change":"2024-09-21T10:28:16.370152+0000","last_active":"2024-09-21T10:28:31.504644+0000","last_peered":"2024-09-21T10:28:31.504644+0000","last_clean":"2024-09-21T10:28:31.504644+0000","last_became_active":"2024-09-21T10:28:16.369380+0000","last_became_peered":"2024-09-21T10:28:16.369380+0000","last_unstale":"2024-09-21T10:28:31.504644+0000","last_undegraded":"2024-09-21T10:28:31.504644+0000","last_fullsized":"2024-09-21T10:28:31.504644+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:59:24.527440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091939799999999998,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":207,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933810+0000","last_change":"2024-09-21T10:28:09.539529+0000","last_active":"2024-09-21T10:28:30.933810+0000","last_peered":"2024-09-21T10:28:30.933810+0000","last_clean":"2024-09-21T10:28:30.933810+0000","last_became_active":"2024-09-21T10:28:09.539155+0000","last_became_peered":"2024-09-21T10:28:09.539155+0000","last_unstale":"2024-09-21T10:28:30.933810+0000","last_undegraded":"2024-09-21T10:28:30.933810+0000","last_fullsized":"2024-09-21T10:28:30.933810+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:34:06.730590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":683,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842106+0000","last_change":"2024-09-21T10:27:24.953701+0000","last_active":"2024-09-21T10:28:35.842106+0000","last_peered":"2024-09-21T10:28:35.842106+0000","last_clean":"2024-09-21T10:28:35.842106+0000","last_became_active":"2024-09-21T10:27:24.953371+0000","last_became_peered":"2024-09-21T10:27:24.953371+0000","last_unstale":"2024-09-21T10:28:35.842106+0000","last_undegraded":"2024-09-21T10:28:35.842106+0000","last_fullsized":"2024-09-21T10:28:35.842106+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:57:47.932977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"77'100","reported_seq":98,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504903+0000","last_change":"2024-09-21T10:27:24.949698+0000","last_active":"2024-09-21T10:28:31.504903+0000","last_peered":"2024-09-21T10:28:31.504903+0000","last_clean":"2024-09-21T10:28:31.504903+0000","last_became_active":"2024-09-21T10:27:24.949367+0000","last_became_peered":"2024-09-21T10:27:24.949367+0000","last_unstale":"2024-09-21T10:28:31.504903+0000","last_undegraded":"2024-09-21T10:28:31.504903+0000","last_fullsized":"2024-09-21T10:28:31.504903+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:21:47.185890+0000","last_deep_scrub":"77'100","last_deep_scrub_stamp":"2024-09-21T10:04:45.242073+0000","last_clean_scrub_stamp":"2024-09-21T10:21:47.185890+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:07.437186+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":189,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504432+0000","last_change":"2024-09-21T10:28:16.369315+0000","last_active":"2024-09-21T10:28:31.504432+0000","last_peered":"2024-09-21T10:28:31.504432+0000","last_clean":"2024-09-21T10:28:31.504432+0000","last_became_active":"2024-09-21T10:28:16.368614+0000","last_became_peered":"2024-09-21T10:28:16.368614+0000","last_unstale":"2024-09-21T10:28:31.504432+0000","last_undegraded":"2024-09-21T10:28:31.504432+0000","last_fullsized":"2024-09-21T10:28:31.504432+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:41:50.431319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":725,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842290+0000","last_change":"2024-09-21T10:14:05.475970+0000","last_active":"2024-09-21T10:28:35.842290+0000","last_peered":"2024-09-21T10:28:35.842290+0000","last_clean":"2024-09-21T10:28:35.842290+0000","last_became_active":"2024-09-21T10:14:05.475331+0000","last_became_peered":"2024-09-21T10:14:05.475331+0000","last_unstale":"2024-09-21T10:28:35.842290+0000","last_undegraded":"2024-09-21T10:28:35.842290+0000","last_fullsized":"2024-09-21T10:28:35.842290+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:17.069736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021667999999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":661,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842460+0000","last_change":"2024-09-21T10:18:13.039172+0000","last_active":"2024-09-21T10:28:35.842460+0000","last_peered":"2024-09-21T10:28:35.842460+0000","last_clean":"2024-09-21T10:28:35.842460+0000","last_became_active":"2024-09-21T10:18:13.038756+0000","last_became_peered":"2024-09-21T10:18:13.038756+0000","last_unstale":"2024-09-21T10:28:35.842460+0000","last_undegraded":"2024-09-21T10:28:35.842460+0000","last_fullsized":"2024-09-21T10:28:35.842460+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:06.668926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"77'100","reported_seq":98,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504943+0000","last_change":"2024-09-21T10:27:24.955530+0000","last_active":"2024-09-21T10:28:31.504943+0000","last_peered":"2024-09-21T10:28:31.504943+0000","last_clean":"2024-09-21T10:28:31.504943+0000","last_became_active":"2024-09-21T10:27:24.955183+0000","last_became_peered":"2024-09-21T10:27:24.955183+0000","last_unstale":"2024-09-21T10:28:31.504943+0000","last_undegraded":"2024-09-21T10:28:31.504943+0000","last_fullsized":"2024-09-21T10:28:31.504943+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:20:01.397007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:20:01.397007+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:46:26.990160+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":657,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842202+0000","last_change":"2024-09-21T10:14:05.478152+0000","last_active":"2024-09-21T10:28:35.842202+0000","last_peered":"2024-09-21T10:28:35.842202+0000","last_clean":"2024-09-21T10:28:35.842202+0000","last_became_active":"2024-09-21T10:14:05.477547+0000","last_became_peered":"2024-09-21T10:14:05.477547+0000","last_unstale":"2024-09-21T10:28:35.842202+0000","last_undegraded":"2024-09-21T10:28:35.842202+0000","last_fullsized":"2024-09-21T10:28:35.842202+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:34.646046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":365,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.724291+0000","last_change":"2024-09-21T10:14:30.708090+0000","last_active":"2024-09-21T10:28:30.724291+0000","last_peered":"2024-09-21T10:28:30.724291+0000","last_clean":"2024-09-21T10:28:30.724291+0000","last_became_active":"2024-09-21T10:14:30.707266+0000","last_became_peered":"2024-09-21T10:14:30.707266+0000","last_unstale":"2024-09-21T10:28:30.724291+0000","last_undegraded":"2024-09-21T10:28:30.724291+0000","last_fullsized":"2024-09-21T10:28:30.724291+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:38.474537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052090399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":719,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842267+0000","last_change":"2024-09-21T10:10:57.397251+0000","last_active":"2024-09-21T10:28:35.842267+0000","last_peered":"2024-09-21T10:28:35.842267+0000","last_clean":"2024-09-21T10:28:35.842267+0000","last_became_active":"2024-09-21T10:10:57.396271+0000","last_became_peered":"2024-09-21T10:10:57.396271+0000","last_unstale":"2024-09-21T10:28:35.842267+0000","last_undegraded":"2024-09-21T10:28:35.842267+0000","last_fullsized":"2024-09-21T10:28:35.842267+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:17.155371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"77'100","reported_seq":135,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933926+0000","last_change":"2024-09-21T10:28:09.283444+0000","last_active":"2024-09-21T10:28:30.933926+0000","last_peered":"2024-09-21T10:28:30.933926+0000","last_clean":"2024-09-21T10:28:30.933926+0000","last_became_active":"2024-09-21T10:28:09.282358+0000","last_became_peered":"2024-09-21T10:28:09.282358+0000","last_unstale":"2024-09-21T10:28:30.933926+0000","last_undegraded":"2024-09-21T10:28:30.933926+0000","last_fullsized":"2024-09-21T10:28:30.933926+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:08:59.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:08:59.099852+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:00.002498+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"63'2","reported_seq":56,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744607+0000","last_change":"2024-09-21T10:27:24.970670+0000","last_active":"2024-09-21T10:28:34.744607+0000","last_peered":"2024-09-21T10:28:34.744607+0000","last_clean":"2024-09-21T10:28:34.744607+0000","last_became_active":"2024-09-21T10:27:24.958254+0000","last_became_peered":"2024-09-21T10:27:24.958254+0000","last_unstale":"2024-09-21T10:28:34.744607+0000","last_undegraded":"2024-09-21T10:28:34.744607+0000","last_fullsized":"2024-09-21T10:28:34.744607+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:00:23.704262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":678,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842552+0000","last_change":"2024-09-21T10:22:19.644169+0000","last_active":"2024-09-21T10:28:35.842552+0000","last_peered":"2024-09-21T10:28:35.842552+0000","last_clean":"2024-09-21T10:28:35.842552+0000","last_became_active":"2024-09-21T10:22:19.643917+0000","last_became_peered":"2024-09-21T10:22:19.643917+0000","last_unstale":"2024-09-21T10:28:35.842552+0000","last_undegraded":"2024-09-21T10:28:35.842552+0000","last_fullsized":"2024-09-21T10:28:35.842552+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:28.976858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":53,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744148+0000","last_change":"2024-09-21T10:27:24.950768+0000","last_active":"2024-09-21T10:28:34.744148+0000","last_peered":"2024-09-21T10:28:34.744148+0000","last_clean":"2024-09-21T10:28:34.744148+0000","last_became_active":"2024-09-21T10:27:24.950491+0000","last_became_peered":"2024-09-21T10:27:24.950491+0000","last_unstale":"2024-09-21T10:28:34.744148+0000","last_undegraded":"2024-09-21T10:28:34.744148+0000","last_fullsized":"2024-09-21T10:28:34.744148+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:10:20.907674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"77'100","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744613+0000","last_change":"2024-09-21T10:27:24.963317+0000","last_active":"2024-09-21T10:28:34.744613+0000","last_peered":"2024-09-21T10:28:34.744613+0000","last_clean":"2024-09-21T10:28:34.744613+0000","last_became_active":"2024-09-21T10:27:24.961251+0000","last_became_peered":"2024-09-21T10:27:24.961251+0000","last_unstale":"2024-09-21T10:28:34.744613+0000","last_undegraded":"2024-09-21T10:28:34.744613+0000","last_fullsized":"2024-09-21T10:28:34.744613+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:27:06.813004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:27:06.813004+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:11:28.686047+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"63'1","reported_seq":465,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842365+0000","last_change":"2024-09-21T10:14:30.718942+0000","last_active":"2024-09-21T10:28:35.842365+0000","last_peered":"2024-09-21T10:28:35.842365+0000","last_clean":"2024-09-21T10:28:35.842365+0000","last_became_active":"2024-09-21T10:14:30.709129+0000","last_became_peered":"2024-09-21T10:14:30.709129+0000","last_unstale":"2024-09-21T10:28:35.842365+0000","last_undegraded":"2024-09-21T10:28:35.842365+0000","last_fullsized":"2024-09-21T10:28:35.842365+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:18.171064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":78,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504811+0000","last_change":"2024-09-21T10:22:19.646015+0000","last_active":"2024-09-21T10:28:31.504811+0000","last_peered":"2024-09-21T10:28:31.504811+0000","last_clean":"2024-09-21T10:28:31.504811+0000","last_became_active":"2024-09-21T10:22:19.645608+0000","last_became_peered":"2024-09-21T10:22:19.645608+0000","last_unstale":"2024-09-21T10:28:31.504811+0000","last_undegraded":"2024-09-21T10:28:31.504811+0000","last_fullsized":"2024-09-21T10:28:31.504811+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:39:33.886658+0000","last_clean_scrub_stamp":"2024-09-21T09:39:33.886658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:14.920346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039884099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":135,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933367+0000","last_change":"2024-09-21T10:28:09.282860+0000","last_active":"2024-09-21T10:28:30.933367+0000","last_peered":"2024-09-21T10:28:30.933367+0000","last_clean":"2024-09-21T10:28:30.933367+0000","last_became_active":"2024-09-21T10:28:09.282554+0000","last_became_peered":"2024-09-21T10:28:09.282554+0000","last_unstale":"2024-09-21T10:28:30.933367+0000","last_undegraded":"2024-09-21T10:28:30.933367+0000","last_fullsized":"2024-09-21T10:28:30.933367+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:22:33.734605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":135,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933888+0000","last_change":"2024-09-21T10:28:09.282465+0000","last_active":"2024-09-21T10:28:30.933888+0000","last_peered":"2024-09-21T10:28:30.933888+0000","last_clean":"2024-09-21T10:28:30.933888+0000","last_became_active":"2024-09-21T10:28:09.282190+0000","last_became_peered":"2024-09-21T10:28:09.282190+0000","last_unstale":"2024-09-21T10:28:30.933888+0000","last_undegraded":"2024-09-21T10:28:30.933888+0000","last_fullsized":"2024-09-21T10:28:30.933888+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:08:53.735076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"77'23","reported_seq":155,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933533+0000","last_change":"2024-09-21T10:28:09.283643+0000","last_active":"2024-09-21T10:28:30.933533+0000","last_peered":"2024-09-21T10:28:30.933533+0000","last_clean":"2024-09-21T10:28:30.933533+0000","last_became_active":"2024-09-21T10:28:09.283028+0000","last_became_peered":"2024-09-21T10:28:09.283028+0000","last_unstale":"2024-09-21T10:28:30.933533+0000","last_undegraded":"2024-09-21T10:28:30.933533+0000","last_fullsized":"2024-09-21T10:28:30.933533+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:45:25.893238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":14232,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":729,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.841987+0000","last_change":"2024-09-21T10:22:19.644482+0000","last_active":"2024-09-21T10:28:35.841987+0000","last_peered":"2024-09-21T10:28:35.841987+0000","last_clean":"2024-09-21T10:28:35.841987+0000","last_became_active":"2024-09-21T10:22:19.644080+0000","last_became_peered":"2024-09-21T10:22:19.644080+0000","last_unstale":"2024-09-21T10:28:35.841987+0000","last_undegraded":"2024-09-21T10:28:35.841987+0000","last_fullsized":"2024-09-21T10:28:35.841987+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:42:37.848739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"63'3","reported_seq":484,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.746406+0000","last_change":"2024-09-21T10:27:24.966613+0000","last_active":"2024-09-21T10:28:34.746406+0000","last_peered":"2024-09-21T10:28:34.746406+0000","last_clean":"2024-09-21T10:28:34.746406+0000","last_became_active":"2024-09-21T10:27:24.954961+0000","last_became_peered":"2024-09-21T10:27:24.954961+0000","last_unstale":"2024-09-21T10:28:34.746406+0000","last_undegraded":"2024-09-21T10:28:34.746406+0000","last_fullsized":"2024-09-21T10:28:34.746406+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:13:15.918530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":450,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.14","version":"77'100","reported_seq":610,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.746313+0000","last_change":"2024-09-21T10:18:12.633959+0000","last_active":"2024-09-21T10:28:34.746313+0000","last_peered":"2024-09-21T10:28:34.746313+0000","last_clean":"2024-09-21T10:28:34.746313+0000","last_became_active":"2024-09-21T10:18:12.633591+0000","last_became_peered":"2024-09-21T10:18:12.633591+0000","last_unstale":"2024-09-21T10:28:34.746313+0000","last_undegraded":"2024-09-21T10:28:34.746313+0000","last_fullsized":"2024-09-21T10:28:34.746313+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":171,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'100","last_scrub_stamp":"2024-09-21T10:04:43.014993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:37:09.076441+0000","last_clean_scrub_stamp":"2024-09-21T10:04:43.014993+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:29:16.343472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":231,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504312+0000","last_change":"2024-09-21T10:28:16.363279+0000","last_active":"2024-09-21T10:28:31.504312+0000","last_peered":"2024-09-21T10:28:31.504312+0000","last_clean":"2024-09-21T10:28:31.504312+0000","last_became_active":"2024-09-21T10:28:16.362944+0000","last_became_peered":"2024-09-21T10:28:16.362944+0000","last_unstale":"2024-09-21T10:28:31.504312+0000","last_undegraded":"2024-09-21T10:28:31.504312+0000","last_fullsized":"2024-09-21T10:28:31.504312+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:39:17.368924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744423+0000","last_change":"2024-09-21T10:27:24.954422+0000","last_active":"2024-09-21T10:28:34.744423+0000","last_peered":"2024-09-21T10:28:34.744423+0000","last_clean":"2024-09-21T10:28:34.744423+0000","last_became_active":"2024-09-21T10:27:24.953876+0000","last_became_peered":"2024-09-21T10:27:24.953876+0000","last_unstale":"2024-09-21T10:28:34.744423+0000","last_undegraded":"2024-09-21T10:28:34.744423+0000","last_fullsized":"2024-09-21T10:28:34.744423+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:20.203783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":707,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842245+0000","last_change":"2024-09-21T10:27:24.955590+0000","last_active":"2024-09-21T10:28:35.842245+0000","last_peered":"2024-09-21T10:28:35.842245+0000","last_clean":"2024-09-21T10:28:35.842245+0000","last_became_active":"2024-09-21T10:27:24.955293+0000","last_became_peered":"2024-09-21T10:27:24.955293+0000","last_unstale":"2024-09-21T10:28:35.842245+0000","last_undegraded":"2024-09-21T10:28:35.842245+0000","last_fullsized":"2024-09-21T10:28:35.842245+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:36:15.305320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744179+0000","last_change":"2024-09-21T10:27:24.960830+0000","last_active":"2024-09-21T10:28:34.744179+0000","last_peered":"2024-09-21T10:28:34.744179+0000","last_clean":"2024-09-21T10:28:34.744179+0000","last_became_active":"2024-09-21T10:27:24.958919+0000","last_became_peered":"2024-09-21T10:27:24.958919+0000","last_unstale":"2024-09-21T10:28:34.744179+0000","last_undegraded":"2024-09-21T10:28:34.744179+0000","last_fullsized":"2024-09-21T10:28:34.744179+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:07:10.637181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":78,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.505009+0000","last_change":"2024-09-21T10:22:19.649280+0000","last_active":"2024-09-21T10:28:31.505009+0000","last_peered":"2024-09-21T10:28:31.505009+0000","last_clean":"2024-09-21T10:28:31.505009+0000","last_became_active":"2024-09-21T10:22:19.648950+0000","last_became_peered":"2024-09-21T10:22:19.648950+0000","last_unstale":"2024-09-21T10:28:31.505009+0000","last_undegraded":"2024-09-21T10:28:31.505009+0000","last_fullsized":"2024-09-21T10:28:31.505009+0000","mapping_epoch":402,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":403,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:06:32.778203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744048+0000","last_change":"2024-09-21T10:27:24.961341+0000","last_active":"2024-09-21T10:28:34.744048+0000","last_peered":"2024-09-21T10:28:34.744048+0000","last_clean":"2024-09-21T10:28:34.744048+0000","last_became_active":"2024-09-21T10:27:24.961017+0000","last_became_peered":"2024-09-21T10:27:24.961017+0000","last_unstale":"2024-09-21T10:28:34.744048+0000","last_undegraded":"2024-09-21T10:28:34.744048+0000","last_fullsized":"2024-09-21T10:28:34.744048+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:21:55.827890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":741,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842447+0000","last_change":"2024-09-21T10:27:24.956462+0000","last_active":"2024-09-21T10:28:35.842447+0000","last_peered":"2024-09-21T10:28:35.842447+0000","last_clean":"2024-09-21T10:28:35.842447+0000","last_became_active":"2024-09-21T10:27:24.955291+0000","last_became_peered":"2024-09-21T10:27:24.955291+0000","last_unstale":"2024-09-21T10:28:35.842447+0000","last_undegraded":"2024-09-21T10:28:35.842447+0000","last_fullsized":"2024-09-21T10:28:35.842447+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:25:24.916725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":460,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.745918+0000","last_change":"2024-09-21T10:27:24.955413+0000","last_active":"2024-09-21T10:28:34.745918+0000","last_peered":"2024-09-21T10:28:34.745918+0000","last_clean":"2024-09-21T10:28:34.745918+0000","last_became_active":"2024-09-21T10:27:24.954856+0000","last_became_peered":"2024-09-21T10:27:24.954856+0000","last_unstale":"2024-09-21T10:28:34.745918+0000","last_undegraded":"2024-09-21T10:28:34.745918+0000","last_fullsized":"2024-09-21T10:28:34.745918+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:29.711141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744584+0000","last_change":"2024-09-21T10:27:24.960707+0000","last_active":"2024-09-21T10:28:34.744584+0000","last_peered":"2024-09-21T10:28:34.744584+0000","last_clean":"2024-09-21T10:28:34.744584+0000","last_became_active":"2024-09-21T10:27:24.958341+0000","last_became_peered":"2024-09-21T10:27:24.958341+0000","last_unstale":"2024-09-21T10:28:34.744584+0000","last_undegraded":"2024-09-21T10:28:34.744584+0000","last_fullsized":"2024-09-21T10:28:34.744584+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:01.897844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"63'1","reported_seq":125,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933471+0000","last_change":"2024-09-21T10:28:09.277786+0000","last_active":"2024-09-21T10:28:30.933471+0000","last_peered":"2024-09-21T10:28:30.933471+0000","last_clean":"2024-09-21T10:28:30.933471+0000","last_became_active":"2024-09-21T10:28:09.277389+0000","last_became_peered":"2024-09-21T10:28:09.277389+0000","last_unstale":"2024-09-21T10:28:30.933471+0000","last_undegraded":"2024-09-21T10:28:30.933471+0000","last_fullsized":"2024-09-21T10:28:30.933471+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:46.492879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":845,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.746510+0000","last_change":"2024-09-21T10:27:24.959812+0000","last_active":"2024-09-21T10:28:34.746510+0000","last_peered":"2024-09-21T10:28:34.746510+0000","last_clean":"2024-09-21T10:28:34.746510+0000","last_became_active":"2024-09-21T10:27:24.959509+0000","last_became_peered":"2024-09-21T10:27:24.959509+0000","last_unstale":"2024-09-21T10:28:34.746510+0000","last_undegraded":"2024-09-21T10:28:34.746510+0000","last_fullsized":"2024-09-21T10:28:34.746510+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:12.076554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"63'1","reported_seq":455,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842023+0000","last_change":"2024-09-21T10:14:30.702414+0000","last_active":"2024-09-21T10:28:35.842023+0000","last_peered":"2024-09-21T10:28:35.842023+0000","last_clean":"2024-09-21T10:28:35.842023+0000","last_became_active":"2024-09-21T10:14:30.702009+0000","last_became_peered":"2024-09-21T10:14:30.702009+0000","last_unstale":"2024-09-21T10:28:35.842023+0000","last_undegraded":"2024-09-21T10:28:35.842023+0000","last_fullsized":"2024-09-21T10:28:35.842023+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":304,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:50:48.396082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":451,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933491+0000","last_change":"2024-09-21T10:28:09.281257+0000","last_active":"2024-09-21T10:28:30.933491+0000","last_peered":"2024-09-21T10:28:30.933491+0000","last_clean":"2024-09-21T10:28:30.933491+0000","last_became_active":"2024-09-21T10:28:09.280924+0000","last_became_peered":"2024-09-21T10:28:09.280924+0000","last_unstale":"2024-09-21T10:28:30.933491+0000","last_undegraded":"2024-09-21T10:28:30.933491+0000","last_fullsized":"2024-09-21T10:28:30.933491+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:20.820089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"63'2","reported_seq":203,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504681+0000","last_change":"2024-09-21T10:28:16.368226+0000","last_active":"2024-09-21T10:28:31.504681+0000","last_peered":"2024-09-21T10:28:31.504681+0000","last_clean":"2024-09-21T10:28:31.504681+0000","last_became_active":"2024-09-21T10:28:16.367849+0000","last_became_peered":"2024-09-21T10:28:16.367849+0000","last_unstale":"2024-09-21T10:28:31.504681+0000","last_undegraded":"2024-09-21T10:28:31.504681+0000","last_fullsized":"2024-09-21T10:28:31.504681+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:22:36.268062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":542,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933378+0000","last_change":"2024-09-21T10:28:09.539278+0000","last_active":"2024-09-21T10:28:30.933378+0000","last_peered":"2024-09-21T10:28:30.933378+0000","last_clean":"2024-09-21T10:28:30.933378+0000","last_became_active":"2024-09-21T10:28:09.538995+0000","last_became_peered":"2024-09-21T10:28:09.538995+0000","last_unstale":"2024-09-21T10:28:30.933378+0000","last_undegraded":"2024-09-21T10:28:30.933378+0000","last_fullsized":"2024-09-21T10:28:30.933378+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:22:12.839257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"63'2","reported_seq":186,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:31.504601+0000","last_change":"2024-09-21T10:28:16.369626+0000","last_active":"2024-09-21T10:28:31.504601+0000","last_peered":"2024-09-21T10:28:31.504601+0000","last_clean":"2024-09-21T10:28:31.504601+0000","last_became_active":"2024-09-21T10:28:16.369258+0000","last_became_peered":"2024-09-21T10:28:16.369258+0000","last_unstale":"2024-09-21T10:28:31.504601+0000","last_undegraded":"2024-09-21T10:28:31.504601+0000","last_fullsized":"2024-09-21T10:28:31.504601+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:25:44.405131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":52,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744420+0000","last_change":"2024-09-21T10:27:24.963337+0000","last_active":"2024-09-21T10:28:34.744420+0000","last_peered":"2024-09-21T10:28:34.744420+0000","last_clean":"2024-09-21T10:28:34.744420+0000","last_became_active":"2024-09-21T10:27:24.963022+0000","last_became_peered":"2024-09-21T10:27:24.963022+0000","last_unstale":"2024-09-21T10:28:34.744420+0000","last_undegraded":"2024-09-21T10:28:34.744420+0000","last_fullsized":"2024-09-21T10:28:34.744420+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:53:16.737509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"63'1","reported_seq":53,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:34.744728+0000","last_change":"2024-09-21T10:27:24.961470+0000","last_active":"2024-09-21T10:28:34.744728+0000","last_peered":"2024-09-21T10:28:34.744728+0000","last_clean":"2024-09-21T10:28:34.744728+0000","last_became_active":"2024-09-21T10:27:24.960726+0000","last_became_peered":"2024-09-21T10:27:24.960726+0000","last_unstale":"2024-09-21T10:28:34.744728+0000","last_undegraded":"2024-09-21T10:28:34.744728+0000","last_fullsized":"2024-09-21T10:28:34.744728+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:45:57.713375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":135,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-21T10:28:30.933182+0000","last_change":"2024-09-21T10:28:09.282833+0000","last_active":"2024-09-21T10:28:30.933182+0000","last_peered":"2024-09-21T10:28:30.933182+0000","last_clean":"2024-09-21T10:28:30.933182+0000","last_became_active":"2024-09-21T10:28:09.282255+0000","last_became_peered":"2024-09-21T10:28:09.282255+0000","last_unstale":"2024-09-21T10:28:30.933182+0000","last_undegraded":"2024-09-21T10:28:30.933182+0000","last_fullsized":"2024-09-21T10:28:30.933182+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:49.072967+0000","last_clean_scrub_stamp":"2024-09-21T09:40:49.072967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:32:38.604486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"63'4","reported_seq":469,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-21T10:28:35.842091+0000","last_change":"2024-09-21T10:10:57.401588+0000","last_active":"2024-09-21T10:28:35.842091+0000","last_peered":"2024-09-21T10:28:35.842091+0000","last_clean":"2024-09-21T10:28:35.842091+0000","last_became_active":"2024-09-21T10:10:57.397724+0000","last_became_peered":"2024-09-21T10:10:57.397724+0000","last_unstale":"2024-09-21T10:28:35.842091+0000","last_undegraded":"2024-09-21T10:28:35.842091+0000","last_fullsized":"2024-09-21T10:28:35.842091+0000","mapping_epoch":266,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":267,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:40:48.067317+0000","last_clean_scrub_stamp":"2024-09-21T09:40:48.067317+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:10:47.843630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":16920,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26580,"internal_metadata":0},"log_size":56,"ondisk_log_size":56,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":42,"seq":180388627045,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1498176,"kb_used_data":6708,"kb_used_omap":1,"kb_used_meta":1491454,"kb_avail":92246976,"statfs":{"total":95995035648,"available":94460903424,"internally_reserved":0,"allocated":6868992,"data_stored":4089257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1527249240},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019503,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1504964,"kb_used_data":6392,"kb_used_omap":1,"kb_used_meta":1498558,"kb_avail":92240188,"statfs":{"total":95995035648,"available":94453952512,"internally_reserved":0,"allocated":6545408,"data_stored":3761618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1534523736},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":245,"seq":1052266987762,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":648052,"kb_used_data":6696,"kb_used_omap":8,"kb_used_meta":641335,"kb_avail":93097100,"statfs":{"total":95995035648,"available":95331430400,"internally_reserved":0,"allocated":6856704,"data_stored":4085044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8940,"internal_metadata":656727316},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":175,"seq":751619277111,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":947844,"kb_used_data":6392,"kb_used_omap":12,"kb_used_meta":941427,"kb_avail":92797308,"statfs":{"total":95995035648,"available":95024443392,"internally_reserved":0,"allocated":6545408,"data_stored":3761614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12845,"internal_metadata":964021715},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618823274,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1788660,"kb_used_data":6632,"kb_used_omap":1,"kb_used_meta":1782014,"kb_avail":91956492,"statfs":{"total":95995035648,"available":94163447808,"internally_reserved":0,"allocated":6791168,"data_stored":4019461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1824782680},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":126,"seq":541165879698,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1220468,"kb_used_data":6952,"kb_used_omap":4,"kb_used_meta":1213499,"kb_avail":92524684,"statfs":{"total":95995035648,"available":94745276416,"internally_reserved":0,"allocated":7118848,"data_stored":4347222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4676,"internal_metadata":1242623420},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":105,"seq":450971566538,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1267196,"kb_used_data":6640,"kb_used_omap":6,"kb_used_meta":1260537,"kb_avail":92477956,"statfs":{"total":95995035648,"available":94697426944,"internally_reserved":0,"allocated":6799360,"data_stored":4020154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6745,"internal_metadata":1290790311},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":96,"seq":412316860883,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1383532,"kb_used_data":6368,"kb_used_omap":5,"kb_used_meta":1377146,"kb_avail":92361620,"statfs":{"total":95995035648,"available":94578298880,"internally_reserved":0,"allocated":6520832,"data_stored":3756460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5905,"internal_metadata":1410197743},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7236,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4201,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2195,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2972,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9976,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:29:28.024 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T10:29:28.024 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi107 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-09-21T10:29:28.184 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=692619d76c8e406e8b1ce7d25e95e5b3 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi107 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-09-21T10:29:28.187 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-21T10:29:28.197 INFO:tasks.cephadm:Teardown begin 2024-09-21T10:29:28.197 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T10:29:28.239 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T10:29:28.296 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-21T10:29:28.296 DEBUG:teuthology.orchestra.run.smithi107:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T10:29:28.311 DEBUG:teuthology.orchestra.run.smithi129:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T10:29:28.329 INFO:tasks.cephadm:Stopping all daemons... 2024-09-21T10:29:28.329 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-21T10:29:28.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.a 2024-09-21T10:29:28.358 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:28 smithi107 ceph-mon[102221]: from='client.47702 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:29:28.359 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:28 smithi107 ceph-mon[105022]: from='client.47702 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:29:28.419 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:28 smithi129 ceph-mon[88471]: from='client.47702 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:29:28.669 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:28 smithi107 systemd[1]: Stopping Ceph mon.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:29:29.012 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:28 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a[102198]: 2024-09-21T10:29:28.665+0000 7f318e92f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:29:29.012 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:28 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a[102198]: 2024-09-21T10:29:28.665+0000 7f318e92f640 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-09-21T10:29:29.012 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:28 smithi107 podman[358044]: 2024-09-21 10:29:28.698055453 +0000 UTC m=+0.148901658 container died 34ae11d9ef1bef09d439cd3adc6611bf21acd49bced5be16110e17da397cdd3b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a, org.label-schema.license=GPLv2, RELEASE=main-62da54b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-09-21T10:29:29.012 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:28 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:10:29:28] ENGINE Bus STOPPING 2024-09-21T10:29:29.012 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:28 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:10:29:28] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:29:29.013 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:28 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:10:29:28] ENGINE Bus STOPPED 2024-09-21T10:29:29.013 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:28 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:10:29:28] ENGINE Bus STARTING 2024-09-21T10:29:29.013 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:28 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:10:29:28] ENGINE Serving on http://:::9283 2024-09-21T10:29:29.013 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:28 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:10:29:28] ENGINE Bus STARTED 2024-09-21T10:29:29.344 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:29 smithi107 podman[358044]: 2024-09-21 10:29:29.115461025 +0000 UTC m=+0.566307230 container cleanup 34ae11d9ef1bef09d439cd3adc6611bf21acd49bced5be16110e17da397cdd3b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-21T10:29:29.344 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:29 smithi107 bash[358044]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a 2024-09-21T10:29:29.587 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.a.service' 2024-09-21T10:29:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:29 smithi107 podman[358062]: 2024-09-21 10:29:29.342668892 +0000 UTC m=+0.642953196 container remove 34ae11d9ef1bef09d439cd3adc6611bf21acd49bced5be16110e17da397cdd3b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=main-62da54b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:29:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:29 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-21T10:29:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:29 smithi107 systemd[1]: Stopped Ceph mon.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T10:29:29.609 INFO:journalctl@ceph.mon.a.smithi107.stdout:Sep 21 10:29:29 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.a.service: Consumed 39.916s CPU time. 2024-09-21T10:29:29.646 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:29:29.646 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-21T10:29:29.647 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-21T10:29:29.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.c 2024-09-21T10:29:29.957 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:29 smithi107 systemd[1]: Stopping Ceph mon.c for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:29:30.286 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:29 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:10:29:29] ENGINE Bus STOPPING 2024-09-21T10:29:30.286 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:29 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-c[104975]: 2024-09-21T10:29:29.954+0000 7f367202d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:29:30.286 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:29 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-c[104975]: 2024-09-21T10:29:29.954+0000 7f367202d640 -1 mon.c@2(peon) e4 *** Got Signal Terminated *** 2024-09-21T10:29:30.286 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:30 smithi107 podman[358189]: 2024-09-21 10:29:30.027646593 +0000 UTC m=+0.165643042 container died 04f7b77f15006980802ffeb5bdcc25491298cdb8dcae3a9f8a0733805698feae (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-c, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-21T10:29:30.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:30 smithi107 podman[358189]: 2024-09-21 10:29:30.395688681 +0000 UTC m=+0.533685128 container cleanup 04f7b77f15006980802ffeb5bdcc25491298cdb8dcae3a9f8a0733805698feae (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-c, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-62da54b, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-21T10:29:30.609 INFO:journalctl@ceph.mon.c.smithi107.stdout:Sep 21 10:29:30 smithi107 bash[358189]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-c 2024-09-21T10:29:30.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:30 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:10:29:30] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:29:30.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:30 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:10:29:30] ENGINE Bus STOPPED 2024-09-21T10:29:30.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:30 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:10:29:30] ENGINE Bus STARTING 2024-09-21T10:29:30.610 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:30 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:10:29:30] ENGINE Serving on http://:::9283 2024-09-21T10:29:30.610 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:30 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: [21/Sep/2024:10:29:30] ENGINE Bus STARTED 2024-09-21T10:29:30.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:29:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:29:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:29:30.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.c.service' 2024-09-21T10:29:30.906 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:29:30.906 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-21T10:29:30.907 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-21T10:29:30.907 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.b 2024-09-21T10:29:31.297 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:30 smithi129 systemd[1]: Stopping Ceph mon.b for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:29:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:31 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-b[88449]: 2024-09-21T10:29:31.294+0000 7fa03a7e4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:29:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:31 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-b[88449]: 2024-09-21T10:29:31.294+0000 7fa03a7e4640 -1 mon.b@1(peon) e4 *** Got Signal Terminated *** 2024-09-21T10:29:31.669 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:31 smithi129 podman[237169]: 2024-09-21 10:29:31.531091193 +0000 UTC m=+0.431704347 container died b01038fb71fdf70d509f346955879d8e3df264048d40a1f71577033c05f29130 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-b, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-21T10:29:32.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:31 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: ::ffff:172.21.15.129 - - [21/Sep/2024:10:29:31] "GET /metrics HTTP/1.1" 200 34361 "" "Prometheus/2.51.0" 2024-09-21T10:29:32.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:32 smithi129 podman[237169]: 2024-09-21 10:29:32.02639125 +0000 UTC m=+0.927004339 container cleanup b01038fb71fdf70d509f346955879d8e3df264048d40a1f71577033c05f29130 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-b, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True) 2024-09-21T10:29:32.169 INFO:journalctl@ceph.mon.b.smithi129.stdout:Sep 21 10:29:32 smithi129 bash[237169]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-mon-b 2024-09-21T10:29:32.490 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@mon.b.service' 2024-09-21T10:29:32.527 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:29:32.527 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-21T10:29:32.527 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-21T10:29:32.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-86c49514-77fc-11ef-baf6-efdc52797490@mgr.y 2024-09-21T10:29:32.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:32 smithi107 systemd[1]: Stopping Ceph mgr.y for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:29:33.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T10:29:32.843+0000 7efe13cb5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:29:33.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:32 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y[93185]: 2024-09-21T10:29:32.843+0000 7efe13cb5640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-21T10:29:33.109 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:32 smithi107 podman[358332]: 2024-09-21 10:29:32.886726965 +0000 UTC m=+0.182116921 container died 849409c95398c5ab98493bea5e35bdfdbe0cda89ae5fb02dd29fb6a58b1146bd (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:29:33.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:33 smithi107 podman[358332]: 2024-09-21 10:29:33.427066613 +0000 UTC m=+0.722456596 container cleanup 849409c95398c5ab98493bea5e35bdfdbe0cda89ae5fb02dd29fb6a58b1146bd (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-09-21T10:29:33.609 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Sep 21 10:29:33 smithi107 bash[358332]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-y 2024-09-21T10:29:33.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@mgr.y.service' 2024-09-21T10:29:33.937 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:29:33.938 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-21T10:29:33.938 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-21T10:29:33.938 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-86c49514-77fc-11ef-baf6-efdc52797490@mgr.x 2024-09-21T10:29:34.275 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 10:29:33 smithi129 systemd[1]: Stopping Ceph mgr.x for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:29:34.587 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 10:29:34 smithi129 podman[237303]: 2024-09-21 10:29:34.273292196 +0000 UTC m=+0.144166827 container died 59e80c9df333c233436a37c607a3d0cdf0fd857c162c517f6b7a64f32fed6d8a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:29:34.919 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 10:29:34 smithi129 podman[237303]: 2024-09-21 10:29:34.774039233 +0000 UTC m=+0.644913853 container cleanup 59e80c9df333c233436a37c607a3d0cdf0fd857c162c517f6b7a64f32fed6d8a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=main-62da54b) 2024-09-21T10:29:34.919 INFO:journalctl@ceph.mgr.x.smithi129.stdout:Sep 21 10:29:34 smithi129 bash[237303]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-mgr-x 2024-09-21T10:29:35.182 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@mgr.x.service' 2024-09-21T10:29:35.216 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:29:35.217 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-21T10:29:35.217 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-21T10:29:35.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.0 2024-09-21T10:29:35.518 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:29:35 smithi107 systemd[1]: Stopping Ceph osd.0 for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:29:35.859 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:29:35 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0[212700]: 2024-09-21T10:29:35.516+0000 7f597fd08640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:29:35.859 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:29:35 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0[212700]: 2024-09-21T10:29:35.516+0000 7f597fd08640 -1 osd.0 471 *** Got signal Terminated *** 2024-09-21T10:29:35.859 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:29:35 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0[212700]: 2024-09-21T10:29:35.516+0000 7f597fd08640 -1 osd.0 471 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:29:40.669 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:29:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:29:40.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"86c49514-77fc-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:29:41.326 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:29:41 smithi107 podman[358463]: 2024-09-21 10:29:41.067928896 +0000 UTC m=+5.666028035 container died 621c0348f44478d35411c06235dd833382407e8c0e4767c3222485f1bed4f670 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-09-21T10:29:41.580 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:29:41 smithi107 podman[358463]: 2024-09-21 10:29:41.556928463 +0000 UTC m=+6.155027601 container cleanup 621c0348f44478d35411c06235dd833382407e8c0e4767c3222485f1bed4f670 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:29:41.580 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:29:41 smithi107 bash[358463]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0 2024-09-21T10:29:42.109 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:29:41 smithi107 podman[358487]: 2024-09-21 10:29:41.809023245 +0000 UTC m=+0.741042344 container remove 621c0348f44478d35411c06235dd833382407e8c0e4767c3222485f1bed4f670 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-62da54b, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-21T10:29:42.377 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:29:42 smithi107 podman[358582]: 2024-09-21 10:29:42.025234034 +0000 UTC m=+0.015831661 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:29:42.377 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:29:42 smithi107 podman[358582]: 2024-09-21 10:29:42.160049348 +0000 UTC m=+0.150647022 container create 51687b3f7c9fa8f62a5b9b87c1641321cb51c2d74860ca8d5fde47fc766afc71 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-21T10:29:42.377 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:29:42 smithi107 podman[358582]: 2024-09-21 10:29:42.370723812 +0000 UTC m=+0.361321427 container init 51687b3f7c9fa8f62a5b9b87c1641321cb51c2d74860ca8d5fde47fc766afc71 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-21T10:29:42.377 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:29:42 smithi107 podman[358582]: 2024-09-21 10:29:42.375502059 +0000 UTC m=+0.366099670 container start 51687b3f7c9fa8f62a5b9b87c1641321cb51c2d74860ca8d5fde47fc766afc71 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-21T10:29:42.859 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:29:42 smithi107 podman[358582]: 2024-09-21 10:29:42.487341825 +0000 UTC m=+0.477939450 container attach 51687b3f7c9fa8f62a5b9b87c1641321cb51c2d74860ca8d5fde47fc766afc71 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-62da54b, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-21T10:29:43.734 INFO:journalctl@ceph.osd.0.smithi107.stdout:Sep 21 10:29:43 smithi107 podman[358582]: 2024-09-21 10:29:43.394006255 +0000 UTC m=+1.384603869 container died 51687b3f7c9fa8f62a5b9b87c1641321cb51c2d74860ca8d5fde47fc766afc71 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-0-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-21T10:29:44.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.0.service' 2024-09-21T10:29:44.044 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:29:44.045 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-21T10:29:44.045 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-21T10:29:44.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.1 2024-09-21T10:29:44.359 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 10:29:44 smithi107 systemd[1]: Stopping Ceph osd.1 for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:29:44.859 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 10:29:44 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1[118184]: 2024-09-21T10:29:44.387+0000 7f63ff8f2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:29:44.859 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 10:29:44 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1[118184]: 2024-09-21T10:29:44.387+0000 7f63ff8f2640 -1 osd.1 471 *** Got signal Terminated *** 2024-09-21T10:29:44.859 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 10:29:44 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1[118184]: 2024-09-21T10:29:44.387+0000 7f63ff8f2640 -1 osd.1 471 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:29:50.109 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 10:29:49 smithi107 podman[358714]: 2024-09-21 10:29:49.849515299 +0000 UTC m=+5.585178339 container died c2b756a1ca12591146d0427a1da635e48ad1342f299b90003a2581d91c4281eb (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-21T10:29:50.859 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 10:29:50 smithi107 podman[358714]: 2024-09-21 10:29:50.419907507 +0000 UTC m=+6.155570546 container cleanup c2b756a1ca12591146d0427a1da635e48ad1342f299b90003a2581d91c4281eb (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-21T10:29:50.859 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 10:29:50 smithi107 bash[358714]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1 2024-09-21T10:29:50.859 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 10:29:50 smithi107 podman[358735]: 2024-09-21 10:29:50.630152463 +0000 UTC m=+0.778553699 container remove c2b756a1ca12591146d0427a1da635e48ad1342f299b90003a2581d91c4281eb (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1, ceph=True, io.buildah.version=1.37.2, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-21T10:29:51.327 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 10:29:51 smithi107 podman[358828]: 2024-09-21 10:29:50.94488894 +0000 UTC m=+0.015652035 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:29:51.327 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 10:29:51 smithi107 podman[358828]: 2024-09-21 10:29:51.081251413 +0000 UTC m=+0.152014487 container create df710e2597a12fffe5016deb0495438ab82cfa20c8875d66c53fc194878d748e (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-21T10:29:51.327 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 10:29:51 smithi107 podman[358828]: 2024-09-21 10:29:51.325567653 +0000 UTC m=+0.396330735 container init df710e2597a12fffe5016deb0495438ab82cfa20c8875d66c53fc194878d748e (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:29:51.609 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 10:29:51 smithi107 podman[358828]: 2024-09-21 10:29:51.331153989 +0000 UTC m=+0.401917063 container start df710e2597a12fffe5016deb0495438ab82cfa20c8875d66c53fc194878d748e (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-21T10:29:51.609 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 10:29:51 smithi107 podman[358828]: 2024-09-21 10:29:51.408914439 +0000 UTC m=+0.479677515 container attach df710e2597a12fffe5016deb0495438ab82cfa20c8875d66c53fc194878d748e (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-deactivate, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-21T10:29:51.609 INFO:journalctl@ceph.osd.1.smithi107.stdout:Sep 21 10:29:51 smithi107 podman[358828]: 2024-09-21 10:29:51.528990551 +0000 UTC m=+0.599753638 container died df710e2597a12fffe5016deb0495438ab82cfa20c8875d66c53fc194878d748e (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-21T10:29:51.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:29:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:29:51.495Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.107:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.107:8765: connect: connection refused" 2024-09-21T10:29:51.919 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:29:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:29:51.501Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.107:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.107:8765: connect: connection refused" 2024-09-21T10:29:51.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:29:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:29:51.501Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.107:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.107:8765: connect: connection refused" 2024-09-21T10:29:51.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:29:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:29:51.501Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.107:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.107:8765: connect: connection refused" 2024-09-21T10:29:51.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:29:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:29:51.501Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.107:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.107:8765: connect: connection refused" 2024-09-21T10:29:51.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:29:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:29:51.501Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=smb msg="Unable to refresh target groups" err="Get \"http://172.21.15.107:8765/sd/prometheus/sd-config?service=smb\": dial tcp 172.21.15.107:8765: connect: connection refused" 2024-09-21T10:29:51.920 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:29:51 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:29:51.501Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.107:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.107:8765: connect: connection refused" 2024-09-21T10:29:52.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.1.service' 2024-09-21T10:29:52.283 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:29:52.283 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-21T10:29:52.284 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-21T10:29:52.284 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.2 2024-09-21T10:29:52.517 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 10:29:52 smithi107 systemd[1]: Stopping Ceph osd.2 for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:29:52.859 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 10:29:52 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2[124527]: 2024-09-21T10:29:52.651+0000 7fb98f20b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:29:52.859 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 10:29:52 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2[124527]: 2024-09-21T10:29:52.651+0000 7fb98f20b640 -1 osd.2 471 *** Got signal Terminated *** 2024-09-21T10:29:52.860 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 10:29:52 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2[124527]: 2024-09-21T10:29:52.651+0000 7fb98f20b640 -1 osd.2 471 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:29:58.410 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 10:29:58 smithi107 podman[358956]: 2024-09-21 10:29:58.12186716 +0000 UTC m=+5.593145076 container died 4e385b5d2d2818f498c6b13d6b6ac379e03f8b2be82eeb8237d17416971d11a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2, RELEASE=main-62da54b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-21T10:29:58.859 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 10:29:58 smithi107 podman[358956]: 2024-09-21 10:29:58.591687992 +0000 UTC m=+6.062965952 container cleanup 4e385b5d2d2818f498c6b13d6b6ac379e03f8b2be82eeb8237d17416971d11a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:29:58.859 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 10:29:58 smithi107 bash[358956]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2 2024-09-21T10:29:59.175 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 10:29:58 smithi107 podman[358987]: 2024-09-21 10:29:58.868031399 +0000 UTC m=+0.745725949 container remove 4e385b5d2d2818f498c6b13d6b6ac379e03f8b2be82eeb8237d17416971d11a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-21T10:29:59.465 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 10:29:59 smithi107 podman[359086]: 2024-09-21 10:29:59.077260139 +0000 UTC m=+0.015802389 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:29:59.465 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 10:29:59 smithi107 podman[359086]: 2024-09-21 10:29:59.193973555 +0000 UTC m=+0.132515791 container create 876de24daa48949e3725d99c8f7f334cf4bcf3e11cae877975a92e37ddd97923 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2) 2024-09-21T10:29:59.465 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 10:29:59 smithi107 podman[359086]: 2024-09-21 10:29:59.388422108 +0000 UTC m=+0.326964346 container init 876de24daa48949e3725d99c8f7f334cf4bcf3e11cae877975a92e37ddd97923 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-21T10:29:59.465 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 10:29:59 smithi107 podman[359086]: 2024-09-21 10:29:59.393375693 +0000 UTC m=+0.331917934 container start 876de24daa48949e3725d99c8f7f334cf4bcf3e11cae877975a92e37ddd97923 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-deactivate, RELEASE=main-62da54b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-21T10:29:59.465 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 10:29:59 smithi107 podman[359086]: 2024-09-21 10:29:59.463294006 +0000 UTC m=+0.401836255 container attach 876de24daa48949e3725d99c8f7f334cf4bcf3e11cae877975a92e37ddd97923 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-62da54b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-09-21T10:29:59.859 INFO:journalctl@ceph.osd.2.smithi107.stdout:Sep 21 10:29:59 smithi107 podman[359086]: 2024-09-21 10:29:59.589828616 +0000 UTC m=+0.528370864 container died 876de24daa48949e3725d99c8f7f334cf4bcf3e11cae877975a92e37ddd97923 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-2-deactivate, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:30:00.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.2.service' 2024-09-21T10:30:00.381 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:30:00.381 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-21T10:30:00.381 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-21T10:30:00.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.3 2024-09-21T10:30:00.609 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 10:30:00 smithi107 systemd[1]: Stopping Ceph osd.3 for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:30:01.109 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 10:30:00 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3[136895]: 2024-09-21T10:30:00.734+0000 7fe157467640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:30:01.109 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 10:30:00 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3[136895]: 2024-09-21T10:30:00.734+0000 7fe157467640 -1 osd.3 471 *** Got signal Terminated *** 2024-09-21T10:30:01.109 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 10:30:00 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3[136895]: 2024-09-21T10:30:00.734+0000 7fe157467640 -1 osd.3 471 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:30:02.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:02 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:02.160+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:02.609 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 10:30:02 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3[136895]: 2024-09-21T10:30:02.311+0000 7fe14fa74640 -1 osd.3 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.215630+0000 front 2024-09-21T10:29:39.215737+0000 (oldest deadline 2024-09-21T10:30:01.515746+0000) 2024-09-21T10:30:03.609 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 10:30:03 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3[136895]: 2024-09-21T10:30:03.327+0000 7fe14fa74640 -1 osd.3 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.215630+0000 front 2024-09-21T10:29:39.215737+0000 (oldest deadline 2024-09-21T10:30:01.515746+0000) 2024-09-21T10:30:03.669 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:03 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:03.171+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:03.669 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:03 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:03.170+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:04.401 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:04 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:04.130+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:04.402 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:04 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:04.172+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:04.402 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:04 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4[38016]: 2024-09-21T10:30:04.189+0000 7efe94c7b640 -1 osd.4 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.090991+0000 front 2024-09-21T10:29:39.090957+0000 (oldest deadline 2024-09-21T10:30:03.790698+0000) 2024-09-21T10:30:04.608 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 10:30:04 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3[136895]: 2024-09-21T10:30:04.330+0000 7fe14fa74640 -1 osd.3 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.215630+0000 front 2024-09-21T10:29:39.215737+0000 (oldest deadline 2024-09-21T10:30:01.515746+0000) 2024-09-21T10:30:04.669 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:04 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:04.399+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.717617+0000 front 2024-09-21T10:29:39.717780+0000 (oldest deadline 2024-09-21T10:30:03.817746+0000) 2024-09-21T10:30:05.419 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:05 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:05.374+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.717617+0000 front 2024-09-21T10:29:39.717780+0000 (oldest deadline 2024-09-21T10:30:03.817746+0000) 2024-09-21T10:30:05.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:05 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:05.158+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:05.420 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:05 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4[38016]: 2024-09-21T10:30:05.186+0000 7efe94c7b640 -1 osd.4 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.090991+0000 front 2024-09-21T10:29:39.090957+0000 (oldest deadline 2024-09-21T10:30:03.790698+0000) 2024-09-21T10:30:05.420 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:05 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:05.151+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:05.609 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 10:30:05 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3[136895]: 2024-09-21T10:30:05.293+0000 7fe14fa74640 -1 osd.3 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.215630+0000 front 2024-09-21T10:29:39.215737+0000 (oldest deadline 2024-09-21T10:30:01.515746+0000) 2024-09-21T10:30:06.413 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:06 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:06.115+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:06.413 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:06 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4[38016]: 2024-09-21T10:30:06.193+0000 7efe94c7b640 -1 osd.4 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.090991+0000 front 2024-09-21T10:29:39.090957+0000 (oldest deadline 2024-09-21T10:30:03.790698+0000) 2024-09-21T10:30:06.413 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:06 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:06.177+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:06.609 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 10:30:06 smithi107 podman[359216]: 2024-09-21 10:30:06.338624501 +0000 UTC m=+5.767317055 container died 687d481b0402fe04ad0e09f4019d97ba7beb5a2bf62e84b64610a90eef23e58d (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-21T10:30:06.669 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:06 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:06.410+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.717617+0000 front 2024-09-21T10:29:39.717780+0000 (oldest deadline 2024-09-21T10:30:03.817746+0000) 2024-09-21T10:30:07.002 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 10:30:06 smithi107 podman[359216]: 2024-09-21 10:30:06.773906003 +0000 UTC m=+6.202598551 container cleanup 687d481b0402fe04ad0e09f4019d97ba7beb5a2bf62e84b64610a90eef23e58d (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-21T10:30:07.002 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 10:30:06 smithi107 bash[359216]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3 2024-09-21T10:30:07.359 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 10:30:07 smithi107 podman[359236]: 2024-09-21 10:30:07.000261856 +0000 UTC m=+0.660178520 container remove 687d481b0402fe04ad0e09f4019d97ba7beb5a2bf62e84b64610a90eef23e58d (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:30:07.386 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:07 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4[38016]: 2024-09-21T10:30:07.185+0000 7efe94c7b640 -1 osd.4 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.090991+0000 front 2024-09-21T10:29:39.090957+0000 (oldest deadline 2024-09-21T10:30:03.790698+0000) 2024-09-21T10:30:07.386 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:07 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:07.206+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:07.387 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:07 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:07.107+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:07.387 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:07 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:07.383+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.717617+0000 front 2024-09-21T10:29:39.717780+0000 (oldest deadline 2024-09-21T10:30:03.817746+0000) 2024-09-21T10:30:07.697 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 10:30:07 smithi107 podman[359329]: 2024-09-21 10:30:07.305571019 +0000 UTC m=+0.015993774 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:30:07.697 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 10:30:07 smithi107 podman[359329]: 2024-09-21 10:30:07.417875549 +0000 UTC m=+0.128298296 container create d59f958ddd8e8ee124560ffb6961932e6926be8a2cc5cf7fe9281d64853901c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-21T10:30:07.697 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 10:30:07 smithi107 podman[359329]: 2024-09-21 10:30:07.628918387 +0000 UTC m=+0.339341140 container init d59f958ddd8e8ee124560ffb6961932e6926be8a2cc5cf7fe9281d64853901c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=main-62da54b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-21T10:30:07.697 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 10:30:07 smithi107 podman[359329]: 2024-09-21 10:30:07.633845558 +0000 UTC m=+0.344268301 container start d59f958ddd8e8ee124560ffb6961932e6926be8a2cc5cf7fe9281d64853901c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-62da54b, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-21T10:30:07.697 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 10:30:07 smithi107 podman[359329]: 2024-09-21 10:30:07.695381976 +0000 UTC m=+0.405804728 container attach d59f958ddd8e8ee124560ffb6961932e6926be8a2cc5cf7fe9281d64853901c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-21T10:30:08.109 INFO:journalctl@ceph.osd.3.smithi107.stdout:Sep 21 10:30:07 smithi107 podman[359329]: 2024-09-21 10:30:07.834250584 +0000 UTC m=+0.544673340 container died d59f958ddd8e8ee124560ffb6961932e6926be8a2cc5cf7fe9281d64853901c9 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:30:08.385 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:08 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4[38016]: 2024-09-21T10:30:08.214+0000 7efe94c7b640 -1 osd.4 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.090991+0000 front 2024-09-21T10:29:39.090957+0000 (oldest deadline 2024-09-21T10:30:03.790698+0000) 2024-09-21T10:30:08.385 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:08 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:08.184+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:08.386 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:08 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:08.106+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:08.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.3.service' 2024-09-21T10:30:08.508 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:30:08.509 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-21T10:30:08.509 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-21T10:30:08.509 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.4 2024-09-21T10:30:08.669 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:08 smithi129 systemd[1]: Stopping Ceph osd.4 for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:30:08.669 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:08 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:08.382+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.717617+0000 front 2024-09-21T10:29:39.717780+0000 (oldest deadline 2024-09-21T10:30:03.817746+0000) 2024-09-21T10:30:09.145 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:08 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4[38016]: 2024-09-21T10:30:08.811+0000 7efe9b669640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:30:09.145 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:08 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4[38016]: 2024-09-21T10:30:08.811+0000 7efe9b669640 -1 osd.4 471 *** Got signal Terminated *** 2024-09-21T10:30:09.145 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:08 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4[38016]: 2024-09-21T10:30:08.811+0000 7efe9b669640 -1 osd.4 471 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:30:09.419 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:09 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:09.387+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.717617+0000 front 2024-09-21T10:29:39.717780+0000 (oldest deadline 2024-09-21T10:30:03.817746+0000) 2024-09-21T10:30:09.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:09 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:09.232+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:09.420 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:09 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4[38016]: 2024-09-21T10:30:09.196+0000 7efe94c7b640 -1 osd.4 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.090991+0000 front 2024-09-21T10:29:39.090957+0000 (oldest deadline 2024-09-21T10:30:03.790698+0000) 2024-09-21T10:30:09.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:09 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:09.142+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:10.384 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:10.108+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:10.385 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4[38016]: 2024-09-21T10:30:10.200+0000 7efe94c7b640 -1 osd.4 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.090991+0000 front 2024-09-21T10:29:39.090957+0000 (oldest deadline 2024-09-21T10:30:03.790698+0000) 2024-09-21T10:30:10.385 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:10.272+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:10.669 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:10.381+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.717617+0000 front 2024-09-21T10:29:39.717780+0000 (oldest deadline 2024-09-21T10:30:03.817746+0000) 2024-09-21T10:30:10.669 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:10 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:10.381+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.518672+0000 front 2024-09-21T10:29:48.518738+0000 (oldest deadline 2024-09-21T10:30:10.218505+0000) 2024-09-21T10:30:11.411 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:11 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4[38016]: 2024-09-21T10:30:11.185+0000 7efe94c7b640 -1 osd.4 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.090991+0000 front 2024-09-21T10:29:39.090957+0000 (oldest deadline 2024-09-21T10:30:03.790698+0000) 2024-09-21T10:30:11.412 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:11 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:11.266+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:11.412 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:11 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:11.266+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.952620+0000 front 2024-09-21T10:29:48.952588+0000 (oldest deadline 2024-09-21T10:30:11.252456+0000) 2024-09-21T10:30:11.412 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:11 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:11.133+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:11.669 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:11 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:11.408+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.717617+0000 front 2024-09-21T10:29:39.717780+0000 (oldest deadline 2024-09-21T10:30:03.817746+0000) 2024-09-21T10:30:11.669 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:11 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:11.408+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.518672+0000 front 2024-09-21T10:29:48.518738+0000 (oldest deadline 2024-09-21T10:30:10.218505+0000) 2024-09-21T10:30:12.408 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:12 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4[38016]: 2024-09-21T10:30:12.161+0000 7efe94c7b640 -1 osd.4 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.090991+0000 front 2024-09-21T10:29:39.090957+0000 (oldest deadline 2024-09-21T10:30:03.790698+0000) 2024-09-21T10:30:12.408 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:12 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4[38016]: 2024-09-21T10:30:12.161+0000 7efe94c7b640 -1 osd.4 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:45.392067+0000 front 2024-09-21T10:29:45.392091+0000 (oldest deadline 2024-09-21T10:30:11.292025+0000) 2024-09-21T10:30:12.409 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:12 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:12.238+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:12.409 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:12 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:12.238+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.952620+0000 front 2024-09-21T10:29:48.952588+0000 (oldest deadline 2024-09-21T10:30:11.252456+0000) 2024-09-21T10:30:12.409 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:12 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:12.111+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:12.409 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:12 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:12.111+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:12.669 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:12 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:12.405+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.717617+0000 front 2024-09-21T10:29:39.717780+0000 (oldest deadline 2024-09-21T10:30:03.817746+0000) 2024-09-21T10:30:12.669 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:12 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:12.405+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.518672+0000 front 2024-09-21T10:29:48.518738+0000 (oldest deadline 2024-09-21T10:30:10.218505+0000) 2024-09-21T10:30:13.379 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:13 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4[38016]: 2024-09-21T10:30:13.129+0000 7efe94c7b640 -1 osd.4 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.090991+0000 front 2024-09-21T10:29:39.090957+0000 (oldest deadline 2024-09-21T10:30:03.790698+0000) 2024-09-21T10:30:13.380 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:13 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4[38016]: 2024-09-21T10:30:13.129+0000 7efe94c7b640 -1 osd.4 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:45.392067+0000 front 2024-09-21T10:29:45.392091+0000 (oldest deadline 2024-09-21T10:30:11.292025+0000) 2024-09-21T10:30:13.380 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:13 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:13.229+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:13.380 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:13 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:13.229+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.952620+0000 front 2024-09-21T10:29:48.952588+0000 (oldest deadline 2024-09-21T10:30:11.252456+0000) 2024-09-21T10:30:13.380 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:13 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:13.086+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:13.380 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:13 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:13.086+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:13.669 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:13 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:13.377+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.717617+0000 front 2024-09-21T10:29:39.717780+0000 (oldest deadline 2024-09-21T10:30:03.817746+0000) 2024-09-21T10:30:13.669 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:13 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:13.377+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.518672+0000 front 2024-09-21T10:29:48.518738+0000 (oldest deadline 2024-09-21T10:30:10.218505+0000) 2024-09-21T10:30:14.377 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:14 smithi129 podman[237453]: 2024-09-21 10:30:14.256072463 +0000 UTC m=+5.548689388 container died 7c6ca82e1ac1786c1b1ef84a2a2965ae7fa4e899931ad9c7f9332a05057d9664 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4, RELEASE=quincy-fbd1a6a, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:30:14.377 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:14 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:14.229+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:14.378 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:14 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:14.229+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.952620+0000 front 2024-09-21T10:29:48.952588+0000 (oldest deadline 2024-09-21T10:30:11.252456+0000) 2024-09-21T10:30:14.378 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:14 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:14.068+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:14.378 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:14 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:14.068+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:14.669 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:14 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:14.374+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.717617+0000 front 2024-09-21T10:29:39.717780+0000 (oldest deadline 2024-09-21T10:30:03.817746+0000) 2024-09-21T10:30:14.669 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:14 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:14.374+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.518672+0000 front 2024-09-21T10:29:48.518738+0000 (oldest deadline 2024-09-21T10:30:10.218505+0000) 2024-09-21T10:30:15.060 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:14 smithi129 podman[237453]: 2024-09-21 10:30:14.693973 +0000 UTC m=+5.986589964 container cleanup 7c6ca82e1ac1786c1b1ef84a2a2965ae7fa4e899931ad9c7f9332a05057d9664 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=quincy-fbd1a6a, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:30:15.060 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:14 smithi129 bash[237453]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4 2024-09-21T10:30:15.060 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:14 smithi129 podman[237487]: 2024-09-21 10:30:14.937581357 +0000 UTC m=+0.676200682 container remove 7c6ca82e1ac1786c1b1ef84a2a2965ae7fa4e899931ad9c7f9332a05057d9664 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-21T10:30:15.348 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:15.203+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:15.348 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:15.203+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.952620+0000 front 2024-09-21T10:29:48.952588+0000 (oldest deadline 2024-09-21T10:30:11.252456+0000) 2024-09-21T10:30:15.348 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:15.057+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:15.348 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:15.057+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:15.650 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:15.398+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.717617+0000 front 2024-09-21T10:29:39.717780+0000 (oldest deadline 2024-09-21T10:30:03.817746+0000) 2024-09-21T10:30:15.650 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:15 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:15.398+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.518672+0000 front 2024-09-21T10:29:48.518738+0000 (oldest deadline 2024-09-21T10:30:10.218505+0000) 2024-09-21T10:30:15.651 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:15 smithi129 podman[237582]: 2024-09-21 10:30:15.249357011 +0000 UTC m=+0.016569465 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:30:15.651 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:15 smithi129 podman[237582]: 2024-09-21 10:30:15.346734005 +0000 UTC m=+0.113946459 container create 2a6953a4d729e1808f5c91dfc9821fb7a4d93de2e759e67a2ae35635f4f31847 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=quincy-fbd1a6a, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-21T10:30:15.651 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:15 smithi129 podman[237582]: 2024-09-21 10:30:15.557577284 +0000 UTC m=+0.324789740 container init 2a6953a4d729e1808f5c91dfc9821fb7a4d93de2e759e67a2ae35635f4f31847 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-21T10:30:15.651 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:15 smithi129 podman[237582]: 2024-09-21 10:30:15.562516198 +0000 UTC m=+0.329728653 container start 2a6953a4d729e1808f5c91dfc9821fb7a4d93de2e759e67a2ae35635f4f31847 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a) 2024-09-21T10:30:15.651 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:15 smithi129 podman[237582]: 2024-09-21 10:30:15.649069164 +0000 UTC m=+0.416281623 container attach 2a6953a4d729e1808f5c91dfc9821fb7a4d93de2e759e67a2ae35635f4f31847 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4-deactivate, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-21T10:30:16.169 INFO:journalctl@ceph.osd.4.smithi129.stdout:Sep 21 10:30:15 smithi129 podman[237582]: 2024-09-21 10:30:15.902607283 +0000 UTC m=+0.669819740 container died 2a6953a4d729e1808f5c91dfc9821fb7a4d93de2e759e67a2ae35635f4f31847 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-4-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-21T10:30:16.169 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:16 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:16.095+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:16.170 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:16 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:16.095+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:16.479 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:16 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:16.182+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:16.479 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:16 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:16.182+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.952620+0000 front 2024-09-21T10:29:48.952588+0000 (oldest deadline 2024-09-21T10:30:11.252456+0000) 2024-09-21T10:30:16.479 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:16 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:16.367+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.717617+0000 front 2024-09-21T10:29:39.717780+0000 (oldest deadline 2024-09-21T10:30:03.817746+0000) 2024-09-21T10:30:16.479 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:16 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:16.367+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.518672+0000 front 2024-09-21T10:29:48.518738+0000 (oldest deadline 2024-09-21T10:30:10.218505+0000) 2024-09-21T10:30:16.493 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.4.service' 2024-09-21T10:30:16.531 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:30:16.531 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-21T10:30:16.532 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-21T10:30:16.532 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.5 2024-09-21T10:30:16.892 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:16 smithi129 systemd[1]: Stopping Ceph osd.5 for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:30:17.169 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:17.139+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:17.169 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:17.139+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:17.169 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:16 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:16.889+0000 7f17ef2e0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:30:17.170 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:16 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:16.889+0000 7f17ef2e0640 -1 osd.5 471 *** Got signal Terminated *** 2024-09-21T10:30:17.170 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:16 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:16.890+0000 7f17ef2e0640 -1 osd.5 471 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:30:17.669 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:17.177+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:17.669 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:17.177+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.952620+0000 front 2024-09-21T10:29:48.952588+0000 (oldest deadline 2024-09-21T10:30:11.252456+0000) 2024-09-21T10:30:17.669 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:17.349+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.717617+0000 front 2024-09-21T10:29:39.717780+0000 (oldest deadline 2024-09-21T10:30:03.817746+0000) 2024-09-21T10:30:17.669 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:17 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:17.349+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.518672+0000 front 2024-09-21T10:29:48.518738+0000 (oldest deadline 2024-09-21T10:30:10.218505+0000) 2024-09-21T10:30:18.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:18 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:18.182+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:18.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:18 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:18.182+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:18.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:18 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:18.140+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:18.420 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:18 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:18.140+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.952620+0000 front 2024-09-21T10:29:48.952588+0000 (oldest deadline 2024-09-21T10:30:11.252456+0000) 2024-09-21T10:30:18.420 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:18 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:18.304+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.717617+0000 front 2024-09-21T10:29:39.717780+0000 (oldest deadline 2024-09-21T10:30:03.817746+0000) 2024-09-21T10:30:18.420 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:18 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:18.304+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.518672+0000 front 2024-09-21T10:29:48.518738+0000 (oldest deadline 2024-09-21T10:30:10.218505+0000) 2024-09-21T10:30:19.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:19 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:19.146+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:19.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:19 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:19.146+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:19.419 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:19 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:19.338+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.717617+0000 front 2024-09-21T10:29:39.717780+0000 (oldest deadline 2024-09-21T10:30:03.817746+0000) 2024-09-21T10:30:19.420 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:19 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:19.338+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.518672+0000 front 2024-09-21T10:29:48.518738+0000 (oldest deadline 2024-09-21T10:30:10.218505+0000) 2024-09-21T10:30:19.420 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:19 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:19.093+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:19.420 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:19 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:19.093+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.952620+0000 front 2024-09-21T10:29:48.952588+0000 (oldest deadline 2024-09-21T10:30:11.252456+0000) 2024-09-21T10:30:20.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:20.104+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:20.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:20.104+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.952620+0000 front 2024-09-21T10:29:48.952588+0000 (oldest deadline 2024-09-21T10:30:11.252456+0000) 2024-09-21T10:30:20.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:20.129+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:20.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:20.129+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:20.420 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:20.290+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.717617+0000 front 2024-09-21T10:29:39.717780+0000 (oldest deadline 2024-09-21T10:30:03.817746+0000) 2024-09-21T10:30:20.420 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:20.290+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.518672+0000 front 2024-09-21T10:29:48.518738+0000 (oldest deadline 2024-09-21T10:30:10.218505+0000) 2024-09-21T10:30:20.420 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:20 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:20.290+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:54.919629+0000 front 2024-09-21T10:29:54.919560+0000 (oldest deadline 2024-09-21T10:30:19.619472+0000) 2024-09-21T10:30:21.321 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:21 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:21.068+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:21.321 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:21 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:21.068+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.952620+0000 front 2024-09-21T10:29:48.952588+0000 (oldest deadline 2024-09-21T10:30:11.252456+0000) 2024-09-21T10:30:21.321 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:21 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:21.068+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:56.553295+0000 front 2024-09-21T10:29:56.553372+0000 (oldest deadline 2024-09-21T10:30:20.653231+0000) 2024-09-21T10:30:21.322 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:21 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:21.140+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:21.322 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:21 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:21.140+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:21.669 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:21 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:21.318+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:39.717617+0000 front 2024-09-21T10:29:39.717780+0000 (oldest deadline 2024-09-21T10:30:03.817746+0000) 2024-09-21T10:30:21.669 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:21 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:21.318+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.518672+0000 front 2024-09-21T10:29:48.518738+0000 (oldest deadline 2024-09-21T10:30:10.218505+0000) 2024-09-21T10:30:21.669 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:21 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5[44412]: 2024-09-21T10:30:21.318+0000 7f17ea8f6640 -1 osd.5 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:54.919629+0000 front 2024-09-21T10:29:54.919560+0000 (oldest deadline 2024-09-21T10:30:19.619472+0000) 2024-09-21T10:30:22.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:22 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:22.125+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:22.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:22 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:22.125+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:22.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:22 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:22.056+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:22.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:22 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:22.056+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.952620+0000 front 2024-09-21T10:29:48.952588+0000 (oldest deadline 2024-09-21T10:30:11.252456+0000) 2024-09-21T10:30:22.420 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:22 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:22.056+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:56.553295+0000 front 2024-09-21T10:29:56.553372+0000 (oldest deadline 2024-09-21T10:30:20.653231+0000) 2024-09-21T10:30:22.850 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:22 smithi129 podman[237708]: 2024-09-21 10:30:22.548551179 +0000 UTC m=+5.798612780 container died 9ae131a2ef13ea7ef6031d7c67094b89c37230c30bef338bd2145ce46f273bf7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:30:23.169 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:22 smithi129 podman[237708]: 2024-09-21 10:30:22.92210475 +0000 UTC m=+6.172166340 container cleanup 9ae131a2ef13ea7ef6031d7c67094b89c37230c30bef338bd2145ce46f273bf7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-21T10:30:23.169 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:22 smithi129 bash[237708]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5 2024-09-21T10:30:23.169 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:23 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:23.009+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:23.170 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:23 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:23.009+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.952620+0000 front 2024-09-21T10:29:48.952588+0000 (oldest deadline 2024-09-21T10:30:11.252456+0000) 2024-09-21T10:30:23.170 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:23 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:23.009+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:56.553295+0000 front 2024-09-21T10:29:56.553372+0000 (oldest deadline 2024-09-21T10:30:20.653231+0000) 2024-09-21T10:30:23.669 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:23 smithi129 podman[237729]: 2024-09-21 10:30:23.264854026 +0000 UTC m=+0.707797254 container remove 9ae131a2ef13ea7ef6031d7c67094b89c37230c30bef338bd2145ce46f273bf7 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:30:23.669 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:23 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:23.170+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:23.669 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:23 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:23.170+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:23.669 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:23 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:23.170+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:54.214106+0000 front 2024-09-21T10:29:54.214185+0000 (oldest deadline 2024-09-21T10:30:22.414337+0000) 2024-09-21T10:30:23.998 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:23 smithi129 podman[237821]: 2024-09-21 10:30:23.683129797 +0000 UTC m=+0.099081206 container create 475e433ea0ca0ba15cf786edfefce632c62b6544cdc59031b5fc21b978baa307 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:30:24.036 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:23 smithi129 podman[237821]: 2024-09-21 10:30:23.602604416 +0000 UTC m=+0.018555840 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:30:24.036 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:23 smithi129 podman[237821]: 2024-09-21 10:30:23.910304117 +0000 UTC m=+0.326255525 container init 475e433ea0ca0ba15cf786edfefce632c62b6544cdc59031b5fc21b978baa307 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=quincy-fbd1a6a, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:30:24.036 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:23 smithi129 podman[237821]: 2024-09-21 10:30:23.915333795 +0000 UTC m=+0.331285220 container start 475e433ea0ca0ba15cf786edfefce632c62b6544cdc59031b5fc21b978baa307 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-21T10:30:24.399 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:23 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:23.992+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:24.400 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:23 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:23.992+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.952620+0000 front 2024-09-21T10:29:48.952588+0000 (oldest deadline 2024-09-21T10:30:11.252456+0000) 2024-09-21T10:30:24.400 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:23 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:23.992+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:56.553295+0000 front 2024-09-21T10:29:56.553372+0000 (oldest deadline 2024-09-21T10:30:20.653231+0000) 2024-09-21T10:30:24.401 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:24 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:24.184+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:24.401 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:24 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:24.184+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:24.402 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:24 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:24.184+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:54.214106+0000 front 2024-09-21T10:29:54.214185+0000 (oldest deadline 2024-09-21T10:30:22.414337+0000) 2024-09-21T10:30:24.402 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:24 smithi129 podman[237821]: 2024-09-21 10:30:24.019083971 +0000 UTC m=+0.435035399 container attach 475e433ea0ca0ba15cf786edfefce632c62b6544cdc59031b5fc21b978baa307 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-fbd1a6a) 2024-09-21T10:30:24.402 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:24 smithi129 podman[237821]: 2024-09-21 10:30:24.096166163 +0000 UTC m=+0.512117581 container died 475e433ea0ca0ba15cf786edfefce632c62b6544cdc59031b5fc21b978baa307 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a) 2024-09-21T10:30:24.651 INFO:journalctl@ceph.osd.5.smithi129.stdout:Sep 21 10:30:24 smithi129 podman[237864]: 2024-09-21 10:30:24.647602313 +0000 UTC m=+0.545160114 container remove 475e433ea0ca0ba15cf786edfefce632c62b6544cdc59031b5fc21b978baa307 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:30:24.666 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.5.service' 2024-09-21T10:30:24.704 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:30:24.704 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-21T10:30:24.704 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-21T10:30:24.704 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.6 2024-09-21T10:30:24.919 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:24 smithi129 systemd[1]: Stopping Ceph osd.6 for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:30:25.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:25.150+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:25.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:25.150+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:25.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:25.150+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:54.214106+0000 front 2024-09-21T10:29:54.214185+0000 (oldest deadline 2024-09-21T10:30:22.414337+0000) 2024-09-21T10:30:25.420 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:24 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:24.966+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:25.420 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:24 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:24.966+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.952620+0000 front 2024-09-21T10:29:48.952588+0000 (oldest deadline 2024-09-21T10:30:11.252456+0000) 2024-09-21T10:30:25.420 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:24 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:24.966+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:56.553295+0000 front 2024-09-21T10:29:56.553372+0000 (oldest deadline 2024-09-21T10:30:20.653231+0000) 2024-09-21T10:30:25.420 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:25.018+0000 7f9f2752e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:30:25.420 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:25.018+0000 7f9f2752e640 -1 osd.6 471 *** Got signal Terminated *** 2024-09-21T10:30:25.420 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:25 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:25.018+0000 7f9f2752e640 -1 osd.6 471 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:30:26.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:26.016+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:26.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:26.016+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.952620+0000 front 2024-09-21T10:29:48.952588+0000 (oldest deadline 2024-09-21T10:30:11.252456+0000) 2024-09-21T10:30:26.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:26.016+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:56.553295+0000 front 2024-09-21T10:29:56.553372+0000 (oldest deadline 2024-09-21T10:30:20.653231+0000) 2024-09-21T10:30:26.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:26.148+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:26.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:26.148+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:26.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:26 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:26.148+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:54.214106+0000 front 2024-09-21T10:29:54.214185+0000 (oldest deadline 2024-09-21T10:30:22.414337+0000) 2024-09-21T10:30:27.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:27 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:27.062+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:27.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:27 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:27.062+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.952620+0000 front 2024-09-21T10:29:48.952588+0000 (oldest deadline 2024-09-21T10:30:11.252456+0000) 2024-09-21T10:30:27.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:27 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:27.062+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:56.553295+0000 front 2024-09-21T10:29:56.553372+0000 (oldest deadline 2024-09-21T10:30:20.653231+0000) 2024-09-21T10:30:27.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:27 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:27.116+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:27.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:27 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:27.116+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:27.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:27 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:27.116+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:54.214106+0000 front 2024-09-21T10:29:54.214185+0000 (oldest deadline 2024-09-21T10:30:22.414337+0000) 2024-09-21T10:30:28.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:28 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:28.014+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:28.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:28 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:28.014+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.952620+0000 front 2024-09-21T10:29:48.952588+0000 (oldest deadline 2024-09-21T10:30:11.252456+0000) 2024-09-21T10:30:28.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:28 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:28.014+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:56.553295+0000 front 2024-09-21T10:29:56.553372+0000 (oldest deadline 2024-09-21T10:30:20.653231+0000) 2024-09-21T10:30:28.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:28 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:28.156+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:28.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:28 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:28.156+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:28.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:28 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:28.156+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:54.214106+0000 front 2024-09-21T10:29:54.214185+0000 (oldest deadline 2024-09-21T10:30:22.414337+0000) 2024-09-21T10:30:28.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:28 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:28.156+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6830 osd.3 since back 2024-09-21T10:30:04.715129+0000 front 2024-09-21T10:30:04.715179+0000 (oldest deadline 2024-09-21T10:30:28.115404+0000) 2024-09-21T10:30:29.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:28 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:28.979+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:29.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:28 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:28.979+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.952620+0000 front 2024-09-21T10:29:48.952588+0000 (oldest deadline 2024-09-21T10:30:11.252456+0000) 2024-09-21T10:30:29.420 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:28 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:28.979+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:56.553295+0000 front 2024-09-21T10:29:56.553372+0000 (oldest deadline 2024-09-21T10:30:20.653231+0000) 2024-09-21T10:30:29.420 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:28 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:28.979+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6830 osd.3 since back 2024-09-21T10:30:02.954189+0000 front 2024-09-21T10:30:02.954259+0000 (oldest deadline 2024-09-21T10:30:28.854212+0000) 2024-09-21T10:30:29.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:29 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:29.168+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:29.421 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:29 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:29.168+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:29.421 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:29 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:29.168+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:54.214106+0000 front 2024-09-21T10:29:54.214185+0000 (oldest deadline 2024-09-21T10:30:22.414337+0000) 2024-09-21T10:30:29.421 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:29 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:29.168+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6830 osd.3 since back 2024-09-21T10:30:04.715129+0000 front 2024-09-21T10:30:04.715179+0000 (oldest deadline 2024-09-21T10:30:28.115404+0000) 2024-09-21T10:30:30.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:30.014+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:38.451056+0000 front 2024-09-21T10:29:38.451077+0000 (oldest deadline 2024-09-21T10:30:01.350921+0000) 2024-09-21T10:30:30.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:30.014+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:48.952620+0000 front 2024-09-21T10:29:48.952588+0000 (oldest deadline 2024-09-21T10:30:11.252456+0000) 2024-09-21T10:30:30.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:30.014+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:56.553295+0000 front 2024-09-21T10:29:56.553372+0000 (oldest deadline 2024-09-21T10:30:20.653231+0000) 2024-09-21T10:30:30.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6[50756]: 2024-09-21T10:30:30.014+0000 7f9f22b44640 -1 osd.6 471 heartbeat_check: no reply from 172.21.15.107:6830 osd.3 since back 2024-09-21T10:30:02.954189+0000 front 2024-09-21T10:30:02.954259+0000 (oldest deadline 2024-09-21T10:30:28.854212+0000) 2024-09-21T10:30:30.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:30.155+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:30.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:30.155+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:30.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:30.155+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:54.214106+0000 front 2024-09-21T10:29:54.214185+0000 (oldest deadline 2024-09-21T10:30:22.414337+0000) 2024-09-21T10:30:30.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:30 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:30.155+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6830 osd.3 since back 2024-09-21T10:30:04.715129+0000 front 2024-09-21T10:30:04.715179+0000 (oldest deadline 2024-09-21T10:30:28.115404+0000) 2024-09-21T10:30:30.795 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:30 smithi129 podman[237948]: 2024-09-21 10:30:30.507936769 +0000 UTC m=+5.584152646 container died a5ccfa3e24c20a9e59b307a136c3d130d120877586cc03256556eb25c7203e65 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-21T10:30:31.145 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:30 smithi129 podman[237948]: 2024-09-21 10:30:30.891704101 +0000 UTC m=+5.967919980 container cleanup a5ccfa3e24c20a9e59b307a136c3d130d120877586cc03256556eb25c7203e65 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=quincy-fbd1a6a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:30:31.145 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:30 smithi129 bash[237948]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6 2024-09-21T10:30:31.419 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:31 smithi129 podman[237971]: 2024-09-21 10:30:31.143367842 +0000 UTC m=+0.632903561 container remove a5ccfa3e24c20a9e59b307a136c3d130d120877586cc03256556eb25c7203e65 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:30:31.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:31 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:31.173+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:31.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:31 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:31.173+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:31.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:31 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:31.173+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:54.214106+0000 front 2024-09-21T10:29:54.214185+0000 (oldest deadline 2024-09-21T10:30:22.414337+0000) 2024-09-21T10:30:31.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:31 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:31.173+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6830 osd.3 since back 2024-09-21T10:30:04.715129+0000 front 2024-09-21T10:30:04.715179+0000 (oldest deadline 2024-09-21T10:30:28.115404+0000) 2024-09-21T10:30:31.831 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:31 smithi129 podman[238065]: 2024-09-21 10:30:31.40545654 +0000 UTC m=+0.016220466 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:30:31.831 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:31 smithi129 podman[238065]: 2024-09-21 10:30:31.511047646 +0000 UTC m=+0.121811566 container create 97bcc2fb74103d4217db8a1566a9ddff5cc9eb456d3e236d283b709599d49c39 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=quincy-fbd1a6a, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-21T10:30:31.831 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:31 smithi129 podman[238065]: 2024-09-21 10:30:31.713533403 +0000 UTC m=+0.324297328 container init 97bcc2fb74103d4217db8a1566a9ddff5cc9eb456d3e236d283b709599d49c39 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-21T10:30:31.832 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:31 smithi129 podman[238065]: 2024-09-21 10:30:31.71830119 +0000 UTC m=+0.329065116 container start 97bcc2fb74103d4217db8a1566a9ddff5cc9eb456d3e236d283b709599d49c39 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=quincy-fbd1a6a) 2024-09-21T10:30:31.832 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:31 smithi129 podman[238065]: 2024-09-21 10:30:31.829909526 +0000 UTC m=+0.440673447 container attach 97bcc2fb74103d4217db8a1566a9ddff5cc9eb456d3e236d283b709599d49c39 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-21T10:30:32.138 INFO:journalctl@ceph.osd.6.smithi129.stdout:Sep 21 10:30:31 smithi129 podman[238065]: 2024-09-21 10:30:31.89490467 +0000 UTC m=+0.505668591 container died 97bcc2fb74103d4217db8a1566a9ddff5cc9eb456d3e236d283b709599d49c39 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-6-deactivate, maintainer=Guillaume Abrioux , RELEASE=quincy-fbd1a6a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:30:32.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:32 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:32.136+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:32.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:32 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:32.136+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:32.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:32 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:32.136+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:54.214106+0000 front 2024-09-21T10:29:54.214185+0000 (oldest deadline 2024-09-21T10:30:22.414337+0000) 2024-09-21T10:30:32.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:32 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:32.136+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6830 osd.3 since back 2024-09-21T10:30:04.715129+0000 front 2024-09-21T10:30:04.715179+0000 (oldest deadline 2024-09-21T10:30:28.115404+0000) 2024-09-21T10:30:32.467 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.6.service' 2024-09-21T10:30:32.505 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:30:32.506 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-21T10:30:32.506 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-21T10:30:32.506 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.7 2024-09-21T10:30:32.840 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:32 smithi129 systemd[1]: Stopping Ceph osd.7 for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:30:33.133 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:32 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:32.838+0000 7f6160c5d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:30:33.133 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:32 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:32.838+0000 7f6160c5d640 -1 osd.7 471 *** Got signal Terminated *** 2024-09-21T10:30:33.134 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:32 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:32.838+0000 7f6160c5d640 -1 osd.7 471 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:30:33.134 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:33 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:33.131+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:33.134 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:33 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:33.131+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:33.134 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:33 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:33.131+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:54.214106+0000 front 2024-09-21T10:29:54.214185+0000 (oldest deadline 2024-09-21T10:30:22.414337+0000) 2024-09-21T10:30:33.134 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:33 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:33.131+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6830 osd.3 since back 2024-09-21T10:30:04.715129+0000 front 2024-09-21T10:30:04.715179+0000 (oldest deadline 2024-09-21T10:30:28.115404+0000) 2024-09-21T10:30:34.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:34 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:34.152+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:34.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:34 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:34.152+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:34.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:34 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:34.152+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:54.214106+0000 front 2024-09-21T10:29:54.214185+0000 (oldest deadline 2024-09-21T10:30:22.414337+0000) 2024-09-21T10:30:34.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:34 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:34.152+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6830 osd.3 since back 2024-09-21T10:30:04.715129+0000 front 2024-09-21T10:30:04.715179+0000 (oldest deadline 2024-09-21T10:30:28.115404+0000) 2024-09-21T10:30:35.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:35 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:35.114+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:35.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:35 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:35.114+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:35.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:35 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:35.114+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:54.214106+0000 front 2024-09-21T10:29:54.214185+0000 (oldest deadline 2024-09-21T10:30:22.414337+0000) 2024-09-21T10:30:35.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:35 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:35.114+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6830 osd.3 since back 2024-09-21T10:30:04.715129+0000 front 2024-09-21T10:30:04.715179+0000 (oldest deadline 2024-09-21T10:30:28.115404+0000) 2024-09-21T10:30:35.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:35 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:35.114+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.129:6804 osd.4 since back 2024-09-21T10:30:12.716605+0000 front 2024-09-21T10:30:12.716580+0000 (oldest deadline 2024-09-21T10:30:35.016628+0000) 2024-09-21T10:30:36.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:36 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:36.139+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:36.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:36 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:36.139+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:36.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:36 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:36.139+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:54.214106+0000 front 2024-09-21T10:29:54.214185+0000 (oldest deadline 2024-09-21T10:30:22.414337+0000) 2024-09-21T10:30:36.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:36 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:36.139+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6830 osd.3 since back 2024-09-21T10:30:04.715129+0000 front 2024-09-21T10:30:04.715179+0000 (oldest deadline 2024-09-21T10:30:28.115404+0000) 2024-09-21T10:30:36.420 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:36 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:36.139+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.129:6804 osd.4 since back 2024-09-21T10:30:12.716605+0000 front 2024-09-21T10:30:12.716580+0000 (oldest deadline 2024-09-21T10:30:35.016628+0000) 2024-09-21T10:30:37.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:37 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:37.112+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6806 osd.0 since back 2024-09-21T10:29:37.911779+0000 front 2024-09-21T10:29:37.911825+0000 (oldest deadline 2024-09-21T10:30:02.611655+0000) 2024-09-21T10:30:37.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:37 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:37.112+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6814 osd.1 since back 2024-09-21T10:29:47.912631+0000 front 2024-09-21T10:29:47.912732+0000 (oldest deadline 2024-09-21T10:30:12.012674+0000) 2024-09-21T10:30:37.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:37 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:37.112+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6822 osd.2 since back 2024-09-21T10:29:54.214106+0000 front 2024-09-21T10:29:54.214185+0000 (oldest deadline 2024-09-21T10:30:22.414337+0000) 2024-09-21T10:30:37.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:37 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:37.112+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.107:6830 osd.3 since back 2024-09-21T10:30:04.715129+0000 front 2024-09-21T10:30:04.715179+0000 (oldest deadline 2024-09-21T10:30:28.115404+0000) 2024-09-21T10:30:37.419 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:37 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7[112798]: 2024-09-21T10:30:37.112+0000 7f615b271640 -1 osd.7 471 heartbeat_check: no reply from 172.21.15.129:6804 osd.4 since back 2024-09-21T10:30:12.716605+0000 front 2024-09-21T10:30:12.716580+0000 (oldest deadline 2024-09-21T10:30:35.016628+0000) 2024-09-21T10:30:38.622 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:38 smithi129 podman[238192]: 2024-09-21 10:30:38.34773639 +0000 UTC m=+5.618275840 container died 0c874265bc9e1d7ff6f45c9ced6c6a97bf293d1e305476f344d830c03fc5f31d (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-21T10:30:38.919 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:38 smithi129 podman[238192]: 2024-09-21 10:30:38.740319195 +0000 UTC m=+6.010858635 container cleanup 0c874265bc9e1d7ff6f45c9ced6c6a97bf293d1e305476f344d830c03fc5f31d (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:30:38.919 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:38 smithi129 bash[238192]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7 2024-09-21T10:30:39.304 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:38 smithi129 podman[238216]: 2024-09-21 10:30:38.995710309 +0000 UTC m=+0.642864839 container remove 0c874265bc9e1d7ff6f45c9ced6c6a97bf293d1e305476f344d830c03fc5f31d (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-21T10:30:39.633 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:39 smithi129 podman[238311]: 2024-09-21 10:30:39.205161448 +0000 UTC m=+0.015870770 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:30:39.633 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:39 smithi129 podman[238311]: 2024-09-21 10:30:39.327035407 +0000 UTC m=+0.137744763 container create a14ccf9897975e6167ffad2a0681dc0a16691588eba3fa82d04e81e0d9c9c654 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-21T10:30:39.633 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:39 smithi129 podman[238311]: 2024-09-21 10:30:39.540488108 +0000 UTC m=+0.351197427 container init a14ccf9897975e6167ffad2a0681dc0a16691588eba3fa82d04e81e0d9c9c654 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-09-21T10:30:39.634 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:39 smithi129 podman[238311]: 2024-09-21 10:30:39.545740783 +0000 UTC m=+0.356450101 container start a14ccf9897975e6167ffad2a0681dc0a16691588eba3fa82d04e81e0d9c9c654 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a, org.label-schema.schema-version=1.0) 2024-09-21T10:30:39.634 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:39 smithi129 podman[238311]: 2024-09-21 10:30:39.631916946 +0000 UTC m=+0.442626265 container attach a14ccf9897975e6167ffad2a0681dc0a16691588eba3fa82d04e81e0d9c9c654 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-21T10:30:39.919 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:39 smithi129 podman[238311]: 2024-09-21 10:30:39.724523232 +0000 UTC m=+0.535232542 container died a14ccf9897975e6167ffad2a0681dc0a16691588eba3fa82d04e81e0d9c9c654 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:30:40.261 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:40 smithi129 podman[238354]: 2024-09-21 10:30:40.243644192 +0000 UTC m=+0.513516185 container remove a14ccf9897975e6167ffad2a0681dc0a16691588eba3fa82d04e81e0d9c9c654 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-osd-7-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a) 2024-09-21T10:30:40.261 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:40 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.7.service: Deactivated successfully. 2024-09-21T10:30:40.262 INFO:journalctl@ceph.osd.7.smithi129.stdout:Sep 21 10:30:40 smithi129 systemd[1]: Stopped Ceph osd.7 for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T10:30:40.264 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@osd.7.service' 2024-09-21T10:30:40.303 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:30:40.303 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-21T10:30:40.303 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-21T10:30:40.304 DEBUG:teuthology.orchestra.run.smithi129:> sudo systemctl stop ceph-86c49514-77fc-11ef-baf6-efdc52797490@prometheus.a 2024-09-21T10:30:40.625 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:30:40 smithi129 systemd[1]: Stopping Ceph prometheus.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:30:40.895 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:30:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:30:40.624Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T10:30:40.895 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:30:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:30:40.624Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-21T10:30:40.895 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:30:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:30:40.624Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-21T10:30:40.895 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:30:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:30:40.624Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T10:30:40.895 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:30:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:30:40.624Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-21T10:30:40.895 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:30:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:30:40.624Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-21T10:30:40.896 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:30:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:30:40.624Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T10:30:40.896 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:30:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:30:40.624Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-21T10:30:40.896 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:30:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:30:40.624Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-21T10:30:40.896 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:30:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:30:40.632Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T10:30:40.896 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:30:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:30:40.632Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-21T10:30:40.896 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:30:40 smithi129 ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a[80159]: ts=2024-09-21T10:30:40.632Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-21T10:30:40.896 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:30:40 smithi129 podman[238437]: 2024-09-21 10:30:40.644730476 +0000 UTC m=+0.117314009 container died 24d10324469fb7098ed95c535553a37e4f1530cd9bb6cbabb0221add82cc91e8 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T10:30:41.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:30:40 smithi129 podman[238437]: 2024-09-21 10:30:40.996528442 +0000 UTC m=+0.469111972 container cleanup 24d10324469fb7098ed95c535553a37e4f1530cd9bb6cbabb0221add82cc91e8 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T10:30:41.169 INFO:journalctl@ceph.prometheus.a.smithi129.stdout:Sep 21 10:30:40 smithi129 bash[238437]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T10:30:41.410 DEBUG:teuthology.orchestra.run.smithi129:> sudo pkill -f 'journalctl -f -n 0 -u ceph-86c49514-77fc-11ef-baf6-efdc52797490@prometheus.a.service' 2024-09-21T10:30:41.445 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:30:41.445 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-21T10:30:41.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 86c49514-77fc-11ef-baf6-efdc52797490 --force --keep-logs 2024-09-21T10:30:43.109 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 10:30:42 smithi107 systemd[1]: Stopping Ceph alertmanager.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:30:43.609 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 10:30:43 smithi107 ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a[91145]: ts=2024-09-21T10:30:43.249Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T10:30:43.609 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 10:30:43 smithi107 podman[359757]: 2024-09-21 10:30:43.300507594 +0000 UTC m=+0.162952795 container died 6fa4ebf1f69dc6a718c10a6f65067b5566505bfac921cba11285301c58e901c7 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T10:30:44.030 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 10:30:43 smithi107 podman[359757]: 2024-09-21 10:30:43.802399772 +0000 UTC m=+0.664844960 container cleanup 6fa4ebf1f69dc6a718c10a6f65067b5566505bfac921cba11285301c58e901c7 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T10:30:44.030 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 10:30:43 smithi107 bash[359757]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T10:30:44.354 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 10:30:44 smithi107 podman[359774]: 2024-09-21 10:30:44.028782328 +0000 UTC m=+0.726472537 container remove 6fa4ebf1f69dc6a718c10a6f65067b5566505bfac921cba11285301c58e901c7 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T10:30:44.354 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 10:30:44 smithi107 podman[359774]: 2024-09-21 10:30:44.121646518 +0000 UTC m=+0.819336725 volume remove b5fffda4bb334deca3d69d6b2dba52b4fce6f6cbaeb0564ca9a4209071007a44 2024-09-21T10:30:44.813 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 10:30:44 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T10:30:44.813 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 10:30:44 smithi107 systemd[1]: Stopped Ceph alertmanager.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T10:30:44.813 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Sep 21 10:30:44 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@alertmanager.a.service: Consumed 3.987s CPU time. 2024-09-21T10:30:44.814 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 10:30:44 smithi107 systemd[1]: Stopping Ceph node-exporter.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:30:45.108 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 10:30:44 smithi107 podman[359894]: 2024-09-21 10:30:44.811617619 +0000 UTC m=+0.118757736 container died 2444eabb7c7e00cc120684747c3c88e9900cb82db45b8588bc340dc25f7a957e (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T10:30:45.609 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 10:30:45 smithi107 podman[359894]: 2024-09-21 10:30:45.282546947 +0000 UTC m=+0.589687051 container cleanup 2444eabb7c7e00cc120684747c3c88e9900cb82db45b8588bc340dc25f7a957e (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T10:30:45.609 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 10:30:45 smithi107 bash[359894]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T10:30:45.988 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 10:30:45 smithi107 podman[359911]: 2024-09-21 10:30:45.734126266 +0000 UTC m=+0.926515846 container remove 2444eabb7c7e00cc120684747c3c88e9900cb82db45b8588bc340dc25f7a957e (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T10:30:45.988 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 10:30:45 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T10:30:45.988 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 10:30:45 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-21T10:30:45.988 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 10:30:45 smithi107 systemd[1]: Stopped Ceph node-exporter.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T10:30:45.989 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Sep 21 10:30:45 smithi107 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@node-exporter.a.service: Consumed 9.147s CPU time. 2024-09-21T10:30:54.920 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 86c49514-77fc-11ef-baf6-efdc52797490 --force --keep-logs 2024-09-21T10:30:56.669 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 10:30:56 smithi129 systemd[1]: Stopping Ceph grafana.a for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:30:56.992 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 10:30:56 smithi129 podman[238872]: 2024-09-21 10:30:56.713722984 +0000 UTC m=+0.118681374 container died 72cc8ff672f5005bafc273cf66b2b7719280a74d029925698dac30d555e4c837 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a, architecture=x86_64, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, distribution-scope=public, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, name=ubi8, release=239.1651231664, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, io.buildah.version=1.32.0, description=Ceph Grafana Container) 2024-09-21T10:30:57.361 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 10:30:57 smithi129 podman[238872]: 2024-09-21 10:30:57.083833347 +0000 UTC m=+0.488791744 container cleanup 72cc8ff672f5005bafc273cf66b2b7719280a74d029925698dac30d555e4c837 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, release=239.1651231664, io.openshift.expose-services=, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner , vcs-type=git, io.buildah.version=1.32.0, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-09-21T10:30:57.361 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 10:30:57 smithi129 bash[238872]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a 2024-09-21T10:30:57.669 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 10:30:57 smithi129 podman[238891]: 2024-09-21 10:30:57.359883348 +0000 UTC m=+0.644498032 container remove 72cc8ff672f5005bafc273cf66b2b7719280a74d029925698dac30d555e4c837 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-grafana-a, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, io.openshift.expose-services=, vendor=Red Hat, Inc., version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, release=239.1651231664, name=ubi8, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, distribution-scope=public) 2024-09-21T10:30:57.669 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 10:30:57 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T10:30:57.669 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 10:30:57 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@grafana.a.service: Failed with result 'exit-code'. 2024-09-21T10:30:57.669 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 10:30:57 smithi129 systemd[1]: Stopped Ceph grafana.a for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T10:30:57.670 INFO:journalctl@ceph.grafana.a.smithi129.stdout:Sep 21 10:30:57 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@grafana.a.service: Consumed 31.517s CPU time. 2024-09-21T10:30:58.169 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 10:30:57 smithi129 systemd[1]: Stopping Ceph node-exporter.b for 86c49514-77fc-11ef-baf6-efdc52797490... 2024-09-21T10:30:58.169 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 10:30:57 smithi129 podman[239004]: 2024-09-21 10:30:57.925591485 +0000 UTC m=+0.106685416 container died ce5db8669c3d7532df39b68d9012c444911b23f5cd55f97a28b1f65338e6f5bf (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T10:30:58.456 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 10:30:58 smithi129 podman[239004]: 2024-09-21 10:30:58.254172531 +0000 UTC m=+0.435266465 container cleanup ce5db8669c3d7532df39b68d9012c444911b23f5cd55f97a28b1f65338e6f5bf (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T10:30:58.457 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 10:30:58 smithi129 bash[239004]: ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b 2024-09-21T10:30:58.707 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 10:30:58 smithi129 podman[239021]: 2024-09-21 10:30:58.455430586 +0000 UTC m=+0.524075744 container remove ce5db8669c3d7532df39b68d9012c444911b23f5cd55f97a28b1f65338e6f5bf (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-86c49514-77fc-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T10:30:58.707 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 10:30:58 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-21T10:30:58.707 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 10:30:58 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-21T10:30:58.707 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 10:30:58 smithi129 systemd[1]: Stopped Ceph node-exporter.b for 86c49514-77fc-11ef-baf6-efdc52797490. 2024-09-21T10:30:58.708 INFO:journalctl@ceph.node-exporter.b.smithi129.stdout:Sep 21 10:30:58 smithi129 systemd[1]: ceph-86c49514-77fc-11ef-baf6-efdc52797490@node-exporter.b.service: Consumed 8.404s CPU time. 2024-09-21T10:31:04.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T10:31:04.593 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T10:31:04.620 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-21T10:31:04.622 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/crash to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913651/remote/smithi107/crash 2024-09-21T10:31:04.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/crash -- . 2024-09-21T10:31:04.675 INFO:teuthology.orchestra.run.smithi107.stderr:tar: /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T10:31:04.675 INFO:teuthology.orchestra.run.smithi107.stderr:tar: Error is not recoverable: exiting now 2024-09-21T10:31:04.678 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/crash to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913651/remote/smithi129/crash 2024-09-21T10:31:04.678 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/crash -- . 2024-09-21T10:31:04.716 INFO:teuthology.orchestra.run.smithi129.stderr:tar: /var/lib/ceph/86c49514-77fc-11ef-baf6-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T10:31:04.717 INFO:teuthology.orchestra.run.smithi129.stderr:tar: Error is not recoverable: exiting now 2024-09-21T10:31:04.718 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-21T10:31:04.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v OSD_DOWN | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v MON_DOWN | egrep -v 'out of quorum' | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v OSDMAP_FLAGS | egrep -v PG_AVAILABILITY | head -n 1 2024-09-21T10:31:04.807 INFO:teuthology.orchestra.run.smithi107.stdout:2024-09-21T09:50:00.002925+0000 mon.a (mon.0) 318 : cluster [WRN] pg 6.d is stuck undersized for 62s, current state active+recovering+undersized+remapped, last acting [5,1] 2024-09-21T10:31:04.808 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-09-21T10:31:04.808 INFO:tasks.cephadm:Compressing logs... 2024-09-21T10:31:04.808 DEBUG:teuthology.orchestra.run.smithi107:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T10:31:04.810 DEBUG:teuthology.orchestra.run.smithi129:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T10:31:04.848 INFO:teuthology.orchestra.run.smithi107.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-21T10:31:04.848 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.0.log 2024-09-21T10:31:04.848 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.1.log 2024-09-21T10:31:04.849 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-volume.log 2024-09-21T10:31:04.850 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.0.log: /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph.audit.log 2024-09-21T10:31:04.850 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mon.c.log 2024-09-21T10:31:04.850 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph.cephadm.log 2024-09-21T10:31:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mon.a.log 2024-09-21T10:31:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.2.log 2024-09-21T10:31:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mgr.y.log 2024-09-21T10:31:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mds.foo.smithi107.sedied.log 2024-09-21T10:31:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph.log 2024-09-21T10:31:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.3.log 2024-09-21T10:31:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mds.foo.smithi107.sedied.log: 82.5% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mds.foo.smithi107.sedied.log.gz 2024-09-21T10:31:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T10:31:04.862 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.3.log: /var/log/ceph/cephadm.log: 92.6% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph.log.gz 2024-09-21T10:31:04.869 INFO:teuthology.orchestra.run.smithi129.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-21T10:31:04.871 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.7.log 2024-09-21T10:31:04.871 INFO:teuthology.orchestra.run.smithi129.stderr:gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-volume.log 2024-09-21T10:31:04.871 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.4.log 2024-09-21T10:31:04.874 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph.audit.log 2024-09-21T10:31:04.874 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph.cephadm.log 2024-09-21T10:31:04.874 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mds.foo.smithi129.zdzmqr.log 2024-09-21T10:31:04.874 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.6.log 2024-09-21T10:31:04.874 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mds.foo.smithi129.zdzmqr.log: gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.5.log 2024-09-21T10:31:04.874 INFO:teuthology.orchestra.run.smithi129.stderr: 62.2% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mds.foo.smithi129.zdzmqr.log.gz 2024-09-21T10:31:04.874 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph.log 2024-09-21T10:31:04.875 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mgr.x.log 2024-09-21T10:31:04.875 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mon.b.log 2024-09-21T10:31:04.875 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T10:31:05.147 INFO:teuthology.orchestra.run.smithi107.stderr: 95.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T10:31:05.167 INFO:teuthology.orchestra.run.smithi107.stderr: 91.2% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph.cephadm.log.gz 2024-09-21T10:31:05.186 INFO:teuthology.orchestra.run.smithi129.stderr:/var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mon.b.log: /var/log/ceph/cephadm.log: 91.2% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-volume.log.gz 2024-09-21T10:31:05.209 INFO:teuthology.orchestra.run.smithi129.stderr: 89.9% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph.audit.log.gz 2024-09-21T10:31:05.210 INFO:teuthology.orchestra.run.smithi129.stderr: 84.6% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph.cephadm.log.gz 2024-09-21T10:31:05.256 INFO:teuthology.orchestra.run.smithi129.stderr: 86.8% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph.log.gz 2024-09-21T10:31:05.603 INFO:teuthology.orchestra.run.smithi107.stderr: 91.0% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-volume.log.gz 2024-09-21T10:31:05.608 INFO:teuthology.orchestra.run.smithi129.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T10:31:06.257 INFO:teuthology.orchestra.run.smithi107.stderr: 93.9% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph.audit.log.gz 2024-09-21T10:31:07.108 INFO:teuthology.orchestra.run.smithi129.stderr: 90.0% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mgr.x.log.gz 2024-09-21T10:31:11.995 INFO:teuthology.orchestra.run.smithi107.stderr: 89.4% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mgr.y.log.gz 2024-09-21T10:31:17.846 INFO:teuthology.orchestra.run.smithi129.stderr: 92.0% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mon.b.log.gz 2024-09-21T10:31:26.512 INFO:teuthology.orchestra.run.smithi107.stderr: 91.4% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mon.c.log.gz 2024-09-21T10:32:03.165 INFO:teuthology.orchestra.run.smithi107.stderr: 90.2% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-mon.a.log.gz 2024-09-21T10:50:04.928 INFO:teuthology.orchestra.run.smithi129.stderr: 93.5% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.4.log.gz 2024-09-21T10:54:39.358 INFO:teuthology.orchestra.run.smithi129.stderr: 93.6% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.6.log.gz 2024-09-21T10:57:00.548 INFO:teuthology.orchestra.run.smithi129.stderr: 92.8% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.7.log.gz 2024-09-21T10:57:11.787 INFO:teuthology.orchestra.run.smithi129.stderr: 93.3% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.5.log.gz 2024-09-21T10:57:11.790 INFO:teuthology.orchestra.run.smithi129.stderr: 2024-09-21T10:57:11.790 INFO:teuthology.orchestra.run.smithi129.stderr:real 26m6.965s 2024-09-21T10:57:11.791 INFO:teuthology.orchestra.run.smithi129.stderr:user 8m14.008s 2024-09-21T10:57:11.791 INFO:teuthology.orchestra.run.smithi129.stderr:sys 0m27.280s 2024-09-21T11:03:43.880 INFO:teuthology.orchestra.run.smithi107.stderr: 93.5% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.1.log.gz 2024-09-21T11:04:59.622 INFO:teuthology.orchestra.run.smithi107.stderr: 93.2% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.0.log.gz 2024-09-21T11:05:07.353 INFO:teuthology.orchestra.run.smithi107.stderr: 93.4% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.2.log.gz 2024-09-21T11:05:32.381 INFO:teuthology.orchestra.run.smithi107.stderr: 93.2% -- replaced with /var/log/ceph/86c49514-77fc-11ef-baf6-efdc52797490/ceph-osd.3.log.gz 2024-09-21T11:05:32.407 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-09-21T11:05:32.407 INFO:teuthology.orchestra.run.smithi107.stderr:real 34m27.582s 2024-09-21T11:05:32.407 INFO:teuthology.orchestra.run.smithi107.stderr:user 11m15.106s 2024-09-21T11:05:32.407 INFO:teuthology.orchestra.run.smithi107.stderr:sys 0m34.054s 2024-09-21T11:05:32.408 INFO:tasks.cephadm:Archiving logs... 2024-09-21T11:05:32.408 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913651/remote/smithi107/log 2024-09-21T11:05:32.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T11:07:10.757 DEBUG:teuthology.misc:Transferring archived files from smithi129:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913651/remote/smithi129/log 2024-09-21T11:07:10.758 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T11:08:22.484 INFO:tasks.cephadm:Removing cluster... 2024-09-21T11:08:22.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 86c49514-77fc-11ef-baf6-efdc52797490 --force 2024-09-21T11:08:24.047 DEBUG:teuthology.orchestra.run.smithi129:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 86c49514-77fc-11ef-baf6-efdc52797490 --force 2024-09-21T11:08:25.191 INFO:tasks.cephadm:Removing cephadm ... 2024-09-21T11:08:25.191 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T11:08:25.210 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T11:08:25.226 INFO:tasks.cephadm:Teardown complete 2024-09-21T11:08:25.226 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T11:08:25.241 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T11:08:25.242 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T11:08:25.253 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T11:08:25.344 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T11:08:25.354 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T11:08:25.354 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T11:08:25.357 DEBUG:teuthology.orchestra.run.smithi129:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T11:08:25.371 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-09-21T11:08:25.371 INFO:teuthology.orchestra.run.smithi129.stderr:bash: line 1: ntpq: command not found 2024-09-21T11:08:25.401 INFO:teuthology.orchestra.run.smithi129.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T11:08:25.401 INFO:teuthology.orchestra.run.smithi129.stdout:=============================================================================== 2024-09-21T11:08:25.402 INFO:teuthology.orchestra.run.smithi129.stdout:^* hv01.front.sepia.ceph.com 3 6 377 22 -254us[ -245us] +/- 64ms 2024-09-21T11:08:25.402 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 21 +224us[ +224us] +/- 57ms 2024-09-21T11:08:25.402 INFO:teuthology.orchestra.run.smithi129.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 +75us[ +75us] +/- 74ms 2024-09-21T11:08:25.402 INFO:teuthology.orchestra.run.smithi129.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T11:08:25.486 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T11:08:25.486 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-09-21T11:08:25.486 INFO:teuthology.orchestra.run.smithi107.stdout:^* hv01.front.sepia.ceph.com 3 6 377 20 -276us[ -274us] +/- 64ms 2024-09-21T11:08:25.486 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 22 +218us[ +220us] +/- 57ms 2024-09-21T11:08:25.486 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 +70us[ +70us] +/- 74ms 2024-09-21T11:08:25.486 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T11:08:25.497 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T11:08:25.509 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T11:08:25.509 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T11:08:25.529 DEBUG:teuthology.orchestra.run.smithi107:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T11:08:25.627 DEBUG:teuthology.orchestra.run.smithi129:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T11:08:25.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T11:08:25.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:08:25.759 DEBUG:teuthology.orchestra.run.smithi129:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T11:08:25.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:08:25.787 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T11:08:25.799 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T11:08:25.838 INFO:teuthology.task.internal:Duration was 6071.866793 seconds 2024-09-21T11:08:25.839 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T11:08:25.872 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T11:08:25.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T11:08:25.875 DEBUG:teuthology.orchestra.run.smithi129:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T11:08:25.926 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T11:08:25.935 INFO:teuthology.orchestra.run.smithi129.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T11:08:26.468 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T11:08:26.469 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2024-09-21T11:08:26.469 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T11:08:26.500 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi129.front.sepia.ceph.com 2024-09-21T11:08:26.500 DEBUG:teuthology.orchestra.run.smithi129:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T11:08:26.525 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T11:08:26.525 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T11:08:26.542 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T11:08:26.592 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T11:08:26.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T11:08:26.625 DEBUG:teuthology.orchestra.run.smithi129:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T11:08:28.286 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T11:08:28.302 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T11:08:28.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T11:08:28.329 DEBUG:teuthology.orchestra.run.smithi129:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T11:08:28.356 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T11:08:28.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T11:08:28.376 DEBUG:teuthology.orchestra.run.smithi129:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T11:08:28.432 INFO:teuthology.orchestra.run.smithi129.stdout:kernel.core_pattern = core 2024-09-21T11:08:28.436 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2024-09-21T11:08:28.487 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T11:08:28.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:08:28.503 DEBUG:teuthology.orchestra.run.smithi129:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T11:08:28.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:08:28.544 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T11:08:28.555 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T11:08:28.556 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913651/remote/smithi107 2024-09-21T11:08:28.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T11:08:28.625 DEBUG:teuthology.misc:Transferring archived files from smithi129:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913651/remote/smithi129 2024-09-21T11:08:28.625 DEBUG:teuthology.orchestra.run.smithi129:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T11:08:28.670 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T11:08:28.670 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T11:08:28.673 DEBUG:teuthology.orchestra.run.smithi129:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T11:08:28.716 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T11:08:28.728 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T11:08:28.728 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T11:08:28.781 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T11:08:28.781 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T11:08:28.784 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T11:08:28.798 INFO:teuthology.orchestra.run.smithi129.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 11:08 /home/ubuntu/cephtest 2024-09-21T11:08:28.812 INFO:teuthology.orchestra.run.smithi107.stdout: 265613 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 21 11:08 /home/ubuntu/cephtest 2024-09-21T11:08:28.812 INFO:teuthology.orchestra.run.smithi107.stdout: 532061 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 21 10:27 /home/ubuntu/cephtest/mnt.0 2024-09-21T11:08:28.812 INFO:teuthology.orchestra.run.smithi107.stdout: 532062 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 21 10:27 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T11:08:28.813 INFO:teuthology.orchestra.run.smithi107.stdout: 532022 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 10:27 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-21T11:08:28.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:08:28.814 INFO:teuthology.orchestra.run.smithi107.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-21T11:08:28.814 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi107 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-21T11:08:28.820 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T11:08:28.832 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T11:08:28.856 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi107 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-21T11:08:28.863 INFO:teuthology.run:Summary data: description: upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} duration: 6071.866793394089 failure_reason: 'Command failed (workunit test cls/test_cls_rbd.sh) on smithi107 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=692619d76c8e406e8b1ce7d25e95e5b3 status: fail success: false 2024-09-21T11:08:28.863 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T11:08:28.954 INFO:teuthology.run:FAIL